When sqlite queries fail for no reason

If you have worked with QtSql, you might have hit the dreaded “Parameter count mismatch” for your perfectly valid SQL query. The problem is excruciatingly hard to debug because the query itself works perfectly fine with the sqlite3 tool.

Here’s the solution: Compile Qt with -system-sqlite.

The problem: Qt uses it’s own sqlite3 headers under src/3rdparty by default which are completely out of date. Qt 4.7 has sqlite3 header from 2009-10-14 version 3.6.18. Almost 2 years old and current sqlite version is 3.7.7! That’s like using Qt 4.5.3 in 2011 🙂 FTS3/4 table queries fail consistently when using Qt’s own headers.

I have opened QTBUG-21040.


On WebKit and WebKit2

Ever heard of WebKit2 and wondering what it means from a Qt perspective? Here’s an attempt to explain QtWebKit and QtWebKit2 in simple terms. I make no attempt to be completely technically correct, it’s meant to be able to explain terminology to the WebKit uninitiated.

In WebKit lingo, “WebCore” is the thing that takes of parsing/layouting/rendering of various css/svg/html documents, providing DOM bindings etc. “JavaScriptCore” implements JavaScript support and is also referred to as SFX (Squirrel fish Extreme). JavaScriptCore can be used as a stand alone JavaScript engine and has no dependencies on WebCore. WebCore uses JavaScriptCore to support JavaScript in web pages. WebCore also contains support for NPAPI plugins (like flash). “WebKit” uses the WebCore to build a platform/toolkit specific API. For example, the Qt “WebKit” port provides QWebElement which exposes the WebCore’s DOM. By definition, WebKit is platform/toolkit/port specific. The Qt port is simply called QtWebKit.

The QtWebKit port is released periodically independent of Qt releases. These ports have the number QtWebKit 2.0, QtWebKit 2.1, QtWebKit 2.2 etc. QtWebKit 2.0 is identical to what was shipped with Qt 4.7.0. QtWebKit 2.1, intended to be mobile friendly, is not part of any shipping Qt release. QtWebKit 2.2, which will be shipped as part of the upcoming Qt 4.8.0, is yet to be released.

Now for WebKit2. The first and most important thing you should know about WebKit2 (even before you know what WebKit2 is) is that WebKit2 is NEITHER AN UPGRADE NOR A NEWER RELEASE of WebKit. It is a parallel port that can happily co-exist with “WebKit”. Let me reiterate: Stop trying to think of WebKit2 as WebKit version 2 :-) Think of it as a completely different API from existing WebKit.

WebKit is the traditional in-process renderer. If you create 100 web pages, they all reside in one process. If one page causes a crash, it brings everything down. WebKit2 provides a system and an API to make it possible to render a page in a separate process. The process management is taken care of by WebKit2. The actual rendering of the page happens using WebCore. WebKit2, therefore, spawns out processes, renders pages in these processes and makes the end result available to the application. It provides mechanisms deliver events from the application to the rendering process. The Qt port of WebKit2 is simply called QtWebKit2. QtWebKit2 is what is used in the N9 browser.

White-space has never been more important. QtWebKit 2.x is a completely different beast from QtWebKit2. QtWebKit 2.x is plain old QtWebKit releases. QtWebKit2 is Qt’s port of WebKit2. This unfortunate naming is a result of Apple announcing WebKit2 shortly after the Qt guys deciding to call their releases QtWebKit 2.x.

WebKit2 and Chromium are similar in their goal. Chromium does not use WebKit2 and probably never will. The Chromium code was intended for the chromium browser specifically. The WebKit2 code was designed upfront to be an API. This difference in motivation resulted in different implementations. See this page for more details.

Because of the multi-process nature of QtWebKit2, many APIs that existed in QtWebKit simply don’t exist anymore. WebKit2 design lends itself to an asynchronous API compared to WebKit where most API was synchronous. For example, DOM introspection of web pages using QWebElement is not possible since the web page’s DOM resides in another process.

QtWebKit2 has a hard dependency on Qt5 and is very much a moving target like Qt5. QtWebKit will probably not work well with Qt5, we have to wait and see.

Current status: Nokia’s Qt WebKit team has decided to focus on QtWebKit2. They have decided to pass on maintainership of QtWebKit to someone else. At the time of writing, there is no publicly announced appointed maintainer to QtWebKit.

Update: Mentioned about QtWebKit 2.x releases based on Jocelyn’s comment.


Qt/Caca Lighthouse Plugin


At the Qt Contributors Summit, Johannes‘ showed me his Qt/Caca Lighthouse plugin. Caca is a graphics library to output text instead of pixels. So this plugin lets you run Qt programs on the console :-)

His code needed some love, so I forked it and cleaned it up. Caca does not provide a event fd and so we have to keep polling caca for events. Since this wasn’t ideal, I moved the event handling to a separate thread and blocked for events. Unfortunately, I found that the caca library is not thread-safe and rendering and processing events in separate threads makes caca crash at randomly. So, I ended up moving the rendering to the event processing thread and having to resurrect the 20ms event timer again :-( The cool thing though is that now Qt renders to QImage in the main ui thread and hands it off to caca. Caca opens a X connection (or similar), converts the image into text, displays a window and handles events in another thread. With some refactoring and thanks to QMetaObject::invokeMethod, the threaded and non-threaded rendering are pretty much the same and can be switched using an environment variable (THREADED_CACA=1).

Animated tiles:


If you want to hack further, code is on gitorious. (Caca doesn’t seem to deliver gpm events with ncurses, so that would be a nice fix to have)

Update: Welcome slashdot readers


Qt Contributors’ Summit

Up until 3 hours back, I wasn’t intending to attend QCS since my visa was expected to arrive late. We have 15 working days waiting period here just to get to the visa interview phase. My tickets were booked for the 19th June to Berlin nevertheless for a Qt Media Hub hackfest. I attended the visa interview yesterday and mentioned that it would be great if they could give me visa for the 16th Jun. They wanted rebooked hotel and plane booking in under 20 mins and a german invite (The english invite left them wondering why a certain Alexandra from Oslo is inviting me to a conference in Germany)! The only internet parlor I found nearby was hopeless and I returned back home.

Incredibly, the German consulate was kind enough to not only send me my visa a couple of hours of back but they also gave me a visa starting from 16th June! With very many things working to my favor (like just a 100 EUR rescheduling fee for the ticket), I am very happy to say I will arrive on 16th afternoon!

Qt Contributors' summit

Qt Contributors' summit


Business as usual

Disclaimer: I am an ex-troll and my company does a lot of Qt business with Nokia

I have been reading the largely negative comments in the blogs by Aron and Daniel about the future of Qt. Sigh, the anonymous internet has made it all too simple for people to post abusive comments and suggest conspiracy theories.

I empathize with the anger; my own business relies on the Qt ecosystem. However, this decision appears to be quite logical to me. On one hand, you have Symbian. Nobody in their right mind would want a Symbian future, let alone pitch it as the competitor for Android or iOS. If you think that line requires justification, you shouldn’t be reading this article, move along :) Second, MeeGo. I am going to speculate here since I have not seen the actual harmattan/MeeGo UI. So, let’s say we have something like the Intel MeeGo tablet shown at MWC. Shocking, no? They are “working” on Copy&Paste, zooming is slow, opening the app causes lots of flicker, the scrolling looks laggy and the presenter is defensive. Continuing my speculation, assuming Nokia’s UX is in a similar state, what would you as a CEO do? I mean, this is the state _today_, imagine 5 months back. I would just drop MeeGo and try shopping for the software elsewhere.

I believe that’s what has happened. Nokia had to make a tough call because MeeGo doesn’t appear to be shippable anytime soon. Some people are of the opinion that choosing an alternate OS is pointless because by the time one adapts to the new OS, one can clean up MeeGo. I think one reason here could be that Elop&Co simply lost confidence in their developers after seeing the state of MeeGo. Another reason probably is that Maemo has always been a research project inside Nokia. MeeGo was announced exactly a year back at MWC 2010. I don’t think Nokia internally ever took it out of the ‘research project’ mode. Yes, they had a little flirt with it trying to make it the main platform but they quickly seem to have discovered it’s not ready (Ballmer mentions that they started talks back in Nov 2010).

So, they now have to choose between Android and WP7. I simply don’t see how Nokia can compete with the existing vendors who have a big head start with Android phones. Back here in India, everybody and their great grandmother are _shipping_ Android phones. Motorola, LG, Samsung, HTC, Videocon (yes, that washing machine company), Dell, Sony, Acer, Micromax, Olive (yes, 10, that I know of!) are already shipping Android phones across all price ranges. With the upcoming cricket world cup and IPL, I expect lot more Android phones to be advertised. A Nokia android phone will be indistinguishable in this crowd. So, personally, I would go with WP7 too as there is some hope for differentiation. With WP7, Nokia can hopefully put pressure on MS (who wants this whole thing to succeed badly for their own future) to deliver on the software.

All this talk of conspiracy theories is quite baseless. Elop cannot make unilateral decisions, that’s not how companies work. He obviously requires the support of the directors. If the share holders think this is a bad decision, they can fire the board of directors but I don’t see this happening. If my speculation about MeeGo’s current state is correct, all it takes is to show the share holders the current MeeGo prototype and that would be the end of discussion.

It’s also a good decision for Nokia for not considering Qt port on WP7. Heck, Qt/Symbian local compilation support on Linux/Mac isn’t there (for what 2 years?) with Nokia having complete control over all the software layers – the toolkit, IDE, OS. Qt on WP7 is a massive massive investment. It is probably a worthwhile undertaking that project after Nokia/WP7 is successful.

FWIW, we all have to be happy that Nokia has been open about this even before it has done anything about it. I, for one, totally appreciate their Openness. So, before you pour out your hate for Nokia, please remember that this is just business as usual. At the end of the day, they have to pay salaries. They have had to disappoint us developers for their own survival. If you are going to argue that MeeGo was truly groundbreaking and what not – please put your money on MeeGo, start your own company and ship MeeGo devices instead of pointing fingers at Nokia.

Where does this leave Qt? Qt has taken a bit big hit because of this decision. This decision means that Qt has suddenly become lot less relevant. I expect the MeeGo phone to be only as successful as the n900. I don’t expect MeeGo or Qt to die inside Nokia until WP7 phones are wildly popular. One single phone, that has been delayed over and over again and that has been sidelined into research does not give me a lot of confidence. Personally, I was hoping for this uber-awesome device for which I can build and _sell_ great applications.

My view is that Qt’s future lies outside Nokia. The Qt fanboy I am, I will do everything I can to keep Qt going. Qt has a very good future in the embedded space (settop boxes, IVI etc). Many companies I met at CES this year were committed to using Qt. For them to continue using Qt, the open governance model simply has to happen. Now. Qt has to be seen as a toolkit that has constant progress with an active community. Ports of Android, iOS can then become part of main stream. If this does not happen soon, future companies are just going to switch over to Android for their devices.


qjsonparser: Parse and stringify JSON with Qt

To my knowledge, there are 3 Qt based JSON parsers out there – QJson, JsonQt and this. QJson uses bison for parsing and JsonQt is hand-written. I have used QJson before and it works perfectly fine.

If you are like me, you will sense an opportunity here to write a qlalr based parser :-) So, here it is – qjsonparser. The grammar is from RFC4627. It behaves very much like QJson – it returns a QVariant for the parsed JSON, uses QVariantMap for objects and QVariantList for arrays. Unlike the others, code is meant to be compiled in place (instead of a library). So, there’s just 3 files overall that you need to drop into your code (README).

If you haven’t used qlalr before:
1. qlalr generates reentrant parsers out of the box. With flex/bison in C mode, one needs to do all sorts of stuff to create a reentrant parser. (QJson uses bison in c++ mode, so it’s reentrant)
2. Complete control over shift/reduce. AFAIK, parsing incrementally using bison isn’t possible easily. With qlalr, you have to write the equivalent of yyparse() yourself and this gives a great deal of control over the shift/reduce steps.
3. It’s completely undocumented. With bison, you don’t really need to understand how LALR parsers work. qlalr, on the other hand, will make you pull out your compiler book. Most of the yyparse() equivalent code that I mentioned can be copy/pasted. But if you are averse to copy/pasting seemingly obfuscated/random code, you absolutely have to understand how LALR parsers work before touching qlalr. Which is why you shouldn’t believe those posts which say that you will be at home with qlalr if you have used bison before. And oh, I don’t intend to spoil the fun of using it by documenting it :-) .
4. qlalr is used by QXmlStreamReader and the old QtScript code. I think it’s the best parser generator for Qt projects.

PS: Currently, I use a hand made lexer, but you can use lexgen which is a Qt friendly scanner. I would have used it but it would add to the number of files. lexgen is used in the Qt’s CSS scanner. The CSS parser, however, is hand made since qlalr didn’t exist then.

UPDATE: This project has moved to http://gitorious.org/qjsonparser/qjsonparser


WebKit repositories

Here’s a cheat sheet for all the WebKit repositories.

SVN: http://svn.webkit.org/repository/webkit. This is always the latest and greatest WebKit code since SVN is the version control used for WebKit code. To checkout trunk, you would do ‘svn co http://svn.webkit.org/repository/webkit/trunk’.

Git mirror at webkit.org: git://git.webkit.org/WebKit.git. This git repository is a mirror of the svn repository. If you work with the Qt/WebKit port, you should _not_ choose this (for reasons explained below).

Git mirror at Gitorious.org: git@gitorious.org:webkit/webkit.git. This git repository is a mirror of the svn repository. The difference is that the author/committer names in the commits are “cleaned” up. For example, in the webkit.org’s repo it might say ‘girish@forwardbias.in@268f45cc-cd09-0410-ab3c-d52691b4dbfc’. In gitorious’ repo, this is cleaned up as ‘girish@forwardbias.in. As a result of this cleanup, the repository sha1’s of gitorious and webkit.org _don’t_ match. This mirror is run by Tor Arne. If it is not up to date for some reason, you now know whom to poke.

qtwebkit at gitorious.org: git://gitorious.org/+qtwebkit-developers/webkit/qtwebkit.git. This is the QtWebKit repository. The releases are essentially branches created from the gitorious.org mirror with patches/hot fixes applied. These patches may not have yet landed upstream. And the commits might be in a different order. Since, qtwebkit is branch off gitorious.org, QtWebKit developers would find working with the gitorious.org easier. If you fetch gitorious.org mirror often, you will have lesser objects to fetch to keep qtwebkit uptodate (as opposed to webkit.org git mirror which has nothing in common with qtwebkit).

My usual git-svn workflow:
git fetch webkit-gitorious (get git-svn commits)
git update-ref refs/remotes/trunk webkit-gitorious/master (change the trunk)
git svn rebase (makes git-svn update the svn<->sha1 mapping)



I find this script (notify-me) most useful when I build webkit 🙂

Just ‘make && notify-me’. You will get a tray notification when the build is complete. Code is not mine, it’s ripped from some site.

#!/usr/bin/env python
“””This is a python 2.5 script that creates a notification using dbus.”””
import dbus
item = (‘org.freedesktop.Notifications’)
path = (‘/org/freedesktop/Notifications’)
interface = (‘org.freedesktop.Notifications’)

icon = ”
array = ”
hint = ”
time = 10000 # Use seconds x 1000
app_name = (‘Test Application’)
title = (‘Whatever it is, is done’)
body = (‘Its all over.’)

bus = dbus.SessionBus()
notif = bus.get_object(item, path)
notify = dbus.Interface(notif, interface)
notify.Notify(app_name, 0, icon, title, body, array, hint, time)


AC for plugins

I have been away from WebKit development for a while. There were some pressing matters in Maemo6 to attend to.

One of the things I am working on is the Flash support for Maemo6. And I am finding that the abstractions of QGV, meegotouch, Qt/WebKit are “adding up” resulting in much slower fps than using just plain Qt.

Some time back, Simon explained to me the concept of Accelerate Compositing. WebKit has no AC support for NPAPI plugins but by doing so we can remove the Qt/WebKit overhead when updating frames. Also, it allows us to implement Flash transparency correctly in QGV.

So, yesterday evening, I started hacking on Accelerated Composition (AC) for NPAPI plugins. With help from Noam, I am happy to report that Flash is rendering with AC 🙂 Expect it to land sometime next week. Here’s a screenie with the fps.

(And oh, that’s the first screenshot in this blog 🙂

UPDATE: You can find the initial code at http://gitorious.org/~girish/webkit/girishs-webkit/commits/plugins_ac_35524. https://bugs.webkit.org/show_bug.cgi?id=35524 is the bugzilla entry.