Postings tagged with MeeGo Input Methods
Real users, real feedback
We released Maliit 0.80.7 on Friday. Over these last days, I am doubly proud about our project. Not only did the N9's virtual keyboard get astonishing reviews across the board, but what's even better: We managed to keep this software open-source. In our communities, there will always be those who focus too much on technical aspects. I remember the technical struggles we had even within MeeGo! But now we get feedback from real users who couldn't care less about what Qt or MeeGo Touch is, and to be honest, that's a refreshing change.
Being here at Qt's Developer Days 2011, it feels great to get such feedback directly, from first-time users of the Nokia N9. Especially the fine haptic feedback and the keyboard's accuracy gets noticed.
I also had the possibility to see a Japanese input method — running on the N9 and powered by Maliit. Seeing how well this plugin already integrates with the platform, I feel that our architecture yet again has been justified. I am looking forward to see more Maliit plugins, and more platforms using Maliit!
Brave New World
Today, we managed to get out the first release of Maliit as an independent project. As far as possible without API break, we started to use our own Maliit naming convention for installation paths, library names, etc.
During that cleanup, Jan Arne did a tremendous job improving our build infrastructure. Instead of Qt's .prf files, Maliit now uses pkg-config everywhere. Plugin developers don't have to think about install paths - they can simply read it from the .pc files, for instance:
pkg-config --variable pluginsdir maliit-plugins-0.80
The version number at the end indicates another nice improvement: versioned libraries and plugin interfaces, which means several Maliit versions can be installed in parallel (although you never want to run more than one server per session).
To install the Maliit framework into a custom directory (say, $HOME/install), simply use:
qmake -r M_IM_PREFIX=$HOME/install
Afterwards, set your PKG_CONFIG_PATH to $HOME/install/lib/pkgconfig and compile the Maliit plugins. Quite honestly, it has never been easier to develop input methods for Maliit than with this release.
Input methods and Wayland in Qt5
I was attending the Qt Contributors' Summit 2011. During the key note, it was promised that everything is up for discussion so I took my chance to discuss about improving input methods support for Qt5.
After some initial discussions with Kristian Høgsberg (Wayland, of course) and Jørgen Lind (who works on Qt Lighhouse), I also addressed Wayland. It became clear that one needs some kind of input method interface directly in Wayland. Kristian immediately started with a small prototype, in order to explain better how a Wayland compositor can provide a much better window management policy than what we currently have with Maliit and X11.
I think the session itself was really successful. I was surprised at the strong interest in this topic.
It became apparent that we should do something about Qt's input context API. For instance, add more input methods hints, come up with a better interface that describes the focus widget, preedit handling, orientation support and so on.
Now we only need to agree on how to make it happen :-)
Decent examples, at last!
Jon spent a lot of time in the last weeks improving our documentation for Maliit. However, while documentation is good (and necessary), Jon thinks that examples are better. I can only agree with that.
I am planning to use Jon's example material for an input method workshop at one of the upcoming MeeGo Freedays (German only) here in Berlin, just to see how useful it is for newcomers.
Maliit starts to feel more and more like a real open-source project, and I am proud of that. Just compare our wiki from end of February with the current, information-packed version, or take a look at the steady traffic on our own mailing list (started only in March this year). Or perhaps just try googling it!
I am really happy to eventually see real contributions and input method plugins from others, which makes me think that we are on a good way.
But this is still only the beginning, and more interesting news will follow soon.
Maliit Session at MeeGo Spring Conference 2011
I was attending the MeeGo Spring Conference as an invited speaker, talking about Maliit - the MeeGo Input Methods - and how to develop custom input method plugins for it.
We had more people going to the talk in Dublin than to this one, but at least the slides and the recording are available online now.
Writing QML-based input methods for Maliit
This week I pushed a simple QML-based virtual keyboard to our MeeGo Keyboard repository. It's functional, but don't expect too much. This functionality will most likely never arrive in MeeGo 1.2, sorry folks.
Getting Started
Let's continue with the good parts instead: The required C++ code was kept to an absolute minimum and in fact it's only needed to wrap the QML resources into a Qt/C++ plugin that can be loaded by our framework. If you want to write your own QML-based input method plugin for Maliit, then this is the required interface:
class MyQmlBasedInputMethodPlugin
: public QObject, public MInputMethodQuickPlugin
{
Q_OBJECT
Q_INTERFACES(MInputMethodQuickPlugin MInputMethodPlugin)
public:
MyQmlBasedInputMethodPlugin();
//! \reimpl
virtual QString name() const;
virtual QString qmlFileName() const;
//! \reimpl_end
};
The plugin's name is used for a specific GConf key. Once set to that name, the framework will load this plugin as the currently active one:
$ gconftool-2 -s /meegotouch/inputmethods/plugins/onscreen -t string "WhateverNameYouCameUpWith"
qmlFileName returns the (absolute) file name (or URL) to your main QML file. The framework will set up the required QML environment and try to load this file.
QML components can interface with the application through a set of properties and slots, tied to a root context named MInputMethodQuick. It exposes the following read-only Qt properties:
- int screenWidth,
- int screenHeight,
- int appOrientation (0, 90, 180, 270 with 0 being the typical landscape orientation and with 270 being the typical portrait orientation).
Additionally, QML components can invoke the following Qt slots:
- setInputMethodArea(const QRect &area)
- sendPreedit(const QString &text)
- sendCommit(const QString &text)
- pluginSwitchRequired(int switchDirection)
Please refer to the header file of MInputMethodQuick to access the full documentation (sorry, not available in MeeGo API docs).
Also, big thanks to Kimmo Surakka and Miika Silfverberg for all their help with this side project.
Self Compositing
Any QML plugin using MInputMethodQuickPlugin will support self compositing out of the box. That is, instead of letting the system compositor blend the application window and virtual keyboard window together into one, the virtual keyboard bypasses the system compositor and redirects the application's window to itself. This usually improves raw FPS but also cuts down on latency. Try running the MeeGo Input Methods service like so:
$ meego-im-uiserver -use-self-composition
This should result in a noticable speedup for your QML plugin. Self compositing is tricky, as it requires a well-behaving window manager (and believe it or not, mcompositor is the only one available getting it nearly 100% right). Usually an additional parameter combination of -manual-redirection, -bypass-wm-hint and -software will do the trick. For MeeGo 1.3, we want to activate self compositing by default, so it's good to know that your QML-based plugin will support this useful feature for free.
A QML-based virtual keyboard for Fremantle
As part of the exercise I also packaged MeeGo Keyboard Quick for Fremantle - it's available from Maemo's extras-devel repository. Be warned though, this is a rough tech demo and as such, fairly useless. A nice advantage of it is that it allowed me to package a MeeGo Input Methods version for Fremantle that is completely free of any MTF/libmeegotouch components.
How to testdrive:
- Install meego-keyboard-quick (after you convinced yourself that you really know what you do - activating extras-devel can brick your N900!),
- Install meego-im-demos
-
Open a terminal, start the MeeGo Input Methods service:
$ meego-im-uiserver -bypass-wm-hint -software -graphicssystem raster -
Open a second terminal, to run the meego-im-demos-plainqt application:
$ export QT_IM_MODULE=MInputContext
$ meego-im-demo-plainqt -fullscreen -software -graphicssystem raster
Instead of the test application, one should be able to use it in any other Qt application, as long as the MeeGo Input Methods service is running and the QT_IM_MODULE environment variable is set correctly.
To fight wrong expectations: I won't have the time to polish or to maintain this port, and from my first tests it would need undefined amounts of integration work to get it to run properly in Fremantle, starting with proper GTK+/Hildon support. If you are, however, interested in furthering this port, then please don't hesitate to contact us.
Painting Strategy in MeeGo's Virtual Keyboard
I've never been happy with the conclusion in the influential blog post "Qt Graphics and Performance — The Cost of Convenience" by Gunnar Sletta: If you want performance, downgrade QGraphicsView to a mere canvas with a single QGraphicsItem. It defeats the whole purpose of decomposing the problem into many small QGraphicsItems and is therefore entirely counter-intuitive. One might be quick to ask what Qt Graphics View is good for (and one might find the answer here), but instead I would like to present an alternate solution to Gunnar's which reaches the same performance but embraces the very nature of Qt Graphics View.
Flashback
Gunnar took the example of a virtual keyboard (which doesn't really do anything, it just sits there and paints itself). And as it happens, I've been working on a real virtual keyboard. At one point, our painting strategy was exactly following the advice of Gunnar, but once new requirements piled up, it became increasingly harder to deal with that painting strategy.
The new requirements were custom key sizes, flexible layouts, custom font sizes for each label, custom font colours, custom key background and custom key labels. And all that dynamically during runtime please.
In MeeGo 1.3, applications will be able to make us of that. For example, when writing an email, the application can temporarily turn the enter key on the virtual keyboard into a "Send" button. Should the user enter an invalid email address, the application might choose to disable the "Send" button until the mistake has been fixed.
Reinventing QGraphicsItems - or not?
As the required logic for keys becomes more complex, so does the data structure representing keys. The paint method we used in the single keyboard item was about to become a real mess. Especially after I added geometry to keys, I knew that I was, in fact, reimplementing QGraphicsItems. But I could not reuse them because of performance reasons! Or was it possible that Gunnar's blog post pointed into the wrong direction?
So I thought about my dilemma and figured out that the actual constraint is rather the amount of paint event calls per frame, and eventually, how much screen area needs updating. With the single QGraphicsItem approach, every change triggers a scheduled repaint for the whole keyboard - even if it's only the label position in one key.
I then realized that only pressed or customized keys would need to be actively painted, and there we had it: An alternate painting strategy that blits the idle keyboard view into a single QGraphicsItem but lets active keys paint themselves as overlays. The rest of our team was sceptical about the idea, so Viacheslav Sobolev - a colleague of mine during my 11 months in Helsinki - had to first improve the performance benchmarks before the team would accept the fact that indeed we managed to keep the same performance, and even improve it for Qt's raster engine.
The Strategy
Even if the general idea of overlay items is very simple, I summarized the following steps that one needs to take care of:
- Aggressive caching - only update when absolutely needed. Analyse what changes and encapsulate change into QGraphicsItems.
- Keys keep changing. Therefore, model keys as QGraphicsItems. Qt Graphics View will not propagate paint events to hidden items, so we hide all keys by default.
- Allow the keyboard item to query a pixmap version from each key. Therefore, the keys act as painting delegates for the keyboard. However, they can also actively paint themselves when activated.
- Paint the idle state of the keyboard into a single QGraphicsItem and avoid calling update on it. Use QGraphicsItem::DeviceCoordinateCache as its caching strategy. The idle state itself can be described as the keyboard's background together will all keys in idle state (that is, neither pressed nor customized).
- Activate keys if pressed or customized, which lets themselves paint as overlays over the static keyboard item.
- Caveat#1: Make sure your implementation can paint the idle state of the keyboard at all times. There's no need to waste more memory with manual caching here - just make sure you can query pixmaps from a key's idle state (even if the key is not idle at that particular moment). The documentation regarding QGraphicsItem caching modes is incorrect.
- Caveat#2: Handle mouse and touch events on the parent keyboard item and propagate state changes to the keys. Most of the keys will usually be inactive, so they cannot receive such events directly.
As a bonus, you can find the basics of the strategy in a modified benchmark of the original blog post. I added two new options "-overlayitems" and "-fullscreen". The latter is useful for N900 where it helps to get rid of system compositor noise.
Is MeeGo Keyboard conquering GNOME3?
You'd probably like me more if I had spend my time testing the upcoming release of MeeGo Tablet UX on this shiny new WeTab (I ♥ Openismus), but Jon couldn't resist to test-drive GNOME3 first. Well, and since we like our virtual keyboard so much, we just had to make it work of course.
Jon also wrote about the features shown in the video.
New features in MeeGo Keyboard
This week we published new features for the MeeGo Keyboard, including accent popups (activated through long-press on certain keys) and support for Chinese Input Methods. The latter still requires an IM engine that supports Cangjie for example to be really useful (not provided by MeeGo Input Methods).
Ubuntu users can easily test it out, as I updated the packages in the MeeGo Input Methods PPA. I also enabled all language layouts by default for those packages, as we yet have no real UI for the Desktop to control such settings. You will have to install the additional layouts package manually though (package is called meego-keyboard-zh-layouts).
One can switch to another language by swiping to the left or right (on the keyboard itself), which is admittedly a bit stressful with a mouse (and nigh impossible with touchpads!). It works quite nicely though when using touch screens. I am sorry for the strange transition animation, it apparently needs some adjustments for the desktop.
We'll also have some GNOME3-related news soon, so stay tuned!
Customizing the MeeGo Keyboard
The look & feel of the MeeGo Keyboard (comes with the MeeGo Input Methods) can be easily adjusted to your needs. In the last post I already gave a brief example on how to modify the graphical assets. In fact, you will find that the look is mostly defined by the graphics contained in that one SVG file. The elements' id's are referenced from the keyboard's CSS file, so it is a good idea to just leave them as-is, unless you really need more graphical elements. The CSS file is installed to a slightly different folder than the SVG file:
/usr/share/themes/base/meegotouch/libmeego-keyboard/style/libmeego-keyboard.css
If that looks a bit complex then it's probably because it is. When hit by new design requirements we didn't always manage to find a clean solution in time. But at least it works and there's even documentation available.
Most containers are named MImAbstractKeyAreaStyle - they describe the style of the widgets that we use to represent keyboard layouts. You won't find containers that describe the actual keys though (am sorry for that). Other than the CSS syntax, those containers don't share much with regular CSS attributes - MeeGo Touch allows you to define your own custom attributes, and that's exactly what I did, to quite some extent.
The whole keyboard responds to touch events or mouse clicks. That is, the reactive areas of the keys cover the whole keyboard. A single key's reactive area is usually aligned with its bounding rectangle, unless spacers are involved. Spacers expand to fill the remaining width in a row, thus allowing flexible alignment of keys. Usability studies showed us that people expect a spacer's area to be consumed by the reactive area of the neighboring keys. The diagram to the left visualizes how reactive areas are constructed from key margins, keyboard paddings and spacers. Keyboard paddings override key margins, e.g, the top margin of key is replaced by the top padding of the keyboard, if that key is located in the top-most row.
For each keyboard layout, the amount of keys defines its style mode (only works if the amount of keys is in range of 10-15 or 30-45 and if sync-style-mode-with-key-count is enabled). These keyboard layouts can get a fully customized styling on their own, e.g., different paddings, key sizes, images, etc. This helps with pixel-perfect designs, but can also be a lot of work.
We avoided to blend CSS attributes with the keyboard layout files, in order to keep the latter reusable for any kind of device. The basic idea is that - for a new form factor - only the CSS file would need to be adjusted. However, there are certain XML attributes in the layout files that can be controlled from the CSS file. For example: A row of keys can carry a height attribute (small, medium, large, x-large, xx-large) which is then mapped to key-height-* CSS attribute, where the concrete height of each group is eventually specified. Thus, the declarative nature of the XML files is preserved.
I hope this post made it less daunting to go and create your own MeeGo Keyboard design. I would love to see some concepts that could potentially replace the current design!
Updated MeeGo Input Methods packages
I published new versions of the MeeGo Input Methods (framework and keyboard plugin) on the project's Launchpad PPA. It contains the API/ABI break that was announced in the beginning of this month.
At first I thought I would do this packaging only for others but I have to admit that I start to enjoy using my host system for input method development - it's really easier than having to use the MeeGo SDK or scratchbox even.
I also added some patches so that the on-screen keyboard docks itself to the bottom of the screen. For MeeGo Touch applications, focus widget relocation will work on the desktop now. If that's not enough for you then how about a slightly translucent keyboard background? For this to come true, you only need to modify the keyboard's SVG file, located at:
/usr/share/themes/base/meegotouch/svg/meegotouch-virtual-keyboard.svg
Search for this section:
<g id="meegotouch-keyboard-background">
<g>
<rect x="100" y="100" width="64" height="64"/>
</g>
</g>
and add a opacity style property to the rect element like so:
<rect style="opacity:0.5;" x="100" y="100" width="64" height="64"/>
You need to restart the meego-im-uiserver process to see the effects though. I will explain the other theming possibilities in another blog post.
The packaging problems I encountered in my last blog post are also solved: A Launchpad PPA keeps all packages in one repository, so one needs to explicitly mention the Ubuntu version in the package names, otherwise they can easily supersede each other.
MeeGo Input Methods for your desktop
I've been working on the MeeGo Input Methods project (codename "Maliit") for nearly a year now. The project provides a pluggable framework for input methods. It comes with a reference plug-in for a multi-touch-capable virtual keyboard.
It had bothered me that, even though our source code was available at gitorious.org, there were nearly no contributions from the outside. I attribute that to the difficulties when it comes to compiling all required components, but also to the lack of perceived openness.
We now offer packages for Ubuntu, through the Openismus PPA for MeeGo Input Methods, thanks to the packaging efforts of Jon. This is an offer for those interested in developing input methods for MeeGo. It's not targeting end-users (yet?) - a virtual keyboard might not be useful on a desktop, unless you have a touchscreen.
We also have a public wiki and if you wish you can visit us in #meego-inputmethods @ freenode.net.
