Handwriting Recognition

Here is the video of the iLiad application in action, which I presented during my diploma presentation. It’s without sound but all important things can be understand anyway. The video is in Ogg Vorbis video format that can be watched with, e.g. VLC.

Video

Description

First, the two areas of the window are shown. 1) The writing area where the user writes in with the pen of the iLiad and 2) the action area where all results are shown and the actions are performed.

After that the user writes down the topic of the session “Music Facts”. To send the text to the web service, which performes the handwriting recognition (HWR) and the ontology-based information extraction (OBIE) the user pushes the send button. A vertical dashed line next to the written text signals that the web service returned a result. Now, the user can click on the text and sees the result in the action area. Each column represents one word, each row represents one word alternative. To change the order of word alternatives the correct oneis just clicked and thus, swaped to the top. The top path in the lattice is the text the user chooses to be correct. “Music Facts” should be the current topic, so the as topic button is pressed.

The user writes some further notes about music. You might notice that he/she does not need to wait for the web service to handle one note, instead the user can continue to write while the data is sent in background.

The result of the first note shows a found instance by highlighting the background of the word alternative “Madonna”. This is also true for the last note.  Notice, that the phrase “Breakfast Club” was not correctly recognized. To correct this mistake the word alternative “Club” is clicked. Hence, the instance Breakfast Club is now found.

A last thing to mention is the reduced result list. Below the word alternative “Madonna”, “Breakfast” and “Club” there are no further alternatives. That is because the application discarded all alternatives after instances were found. Thus, we created a benefit for the user that has less things to check.

The paper that was submitted  to ICFHR was accepted this week. We are all very happy about it. After the camera ready version is submitted I will upload it here.

Exactly one week after the presentation I keep my promise and upload my thesis and also the presentation.

Diploma thesis

Presentation

Presentation with comments

Have a lot of fun with it :)

I submitted my diploma thesis just today. As said I will also publish it on this page, after my presentation.

As a little “appetizer” here is the graphical user interface (GUI) of the final iLiad application. It is taken from the thesis. Sorry for the missing use case description. They are described in detail in the thesis.

Figure 1: GUI directly after the start of the application

The first figure shows iLiad’s graphical user interface (GUI). Topmost is the area the user writes on with the stylus (area 1). Below, all results are shown (area 2). The area next to it contains the action buttons that perform one of the three possible actions (area 3). Finally, on the right side of the action buttons there is the send button, which sends the written content to the FusionService (web service that handles the fusion of handwriting recognition (HWR) and ontology-based information extraction (OBIE)).

Figure 2: iLiad GUI after two notes have been written

After the description of the common areas, Figure 2 depicts the GUI after writing 2 notes. In the following there is a description about the system’s use cases and how they were applied:

• B.2: handle input text: Each stroke is visualized in the writing area as required by use case B.2. When the user finishes the text he or she pushes the send button, which transfers the data to the Bridge (an application that handles the communication between iLiad and the FusionService). After the FusionService returned the result, the written text is highlighted by a vertical line on the left side of it, as shown for the text “Chris Martin is the lead singer of Coldplay”. This indicates that the result is available. After clicking on the text a rectangle visualizes that it is selected.
• B.3: correct result data: Selecting an annotation shows the wordlist in the result area of the window. This is depicted for the “Madonna was a former member of Breakfast Club”. Each column shows one word, each row one word alternative. By clicking on an alternative, the user is able to select this as the correct one, which moves it to the top of the list.
• B.4: show result data: All found instances within the top path of the list are highlighted by gray background color. Note that the list of alternatives is reduced whenever instances were found, e.g., the first word Madonna has only one alternative, although the HWR process returned 4 more (Madonia, Madonnn, Madonnu, Madounn).
• B.5: process result data, B.1: choose research topic: After rectifying the result the user chooses the action to perform by pushing one of the three action buttons (commit, discard, and as topic), which corresponds to use case B.5 in case of commit and discard, or to use case B.1 in case of as topic.
• C.5: store data: If the user chooses to commit the data the top path of the wordlist is stored on the FusionService in a TriG file, a dialect of RDF (The specification of the file format can be found on: http://www4.wiwiss.fu-berlin.de/bizer/TriG/).

I did not yet introduce the algorithm that was developed for combining the results of handwriting recognition (HWR) and ontology-based information extraction (OBIE). Here is a short overview.

A major challenge during the design of an appropriate algorithm was to find one that does not only take the best result of the HWR into account. Imagine, the recognizer returns a list of words, which contain typically more than one alternative per word. Since there is no correct text (the HWR does not know if the results are correct or not), up to k alternatives per word had to be considered. One might argue that it is easy, because the OBIE process just needs to receive every word alternative, but there is a problem in that. Labels in the used ontology can consist of more than one word, e.g., Super Trooper, a song from ABBA consists of two words.

There are now two parameters: k, the number of considered alternatives, and n, the lenght of the labels. The designed algorithm uses an n-gram approach. To fully describe the proposed method would not be appropriate in this blog. Instead I am going to publish my thesis after I finished, there is a detailed description of all relevant parts.

For the moment I can write a little bit of the experiment’s results. Our goal was to improve the performance of recognized instances in the handwritten text. This was measured by means of precision, recall, and f-measure. There is a good description of these at http://en.wikipedia.org/wiki/Precision_and_recall. The f-measure values of our algorithm, using different parameter configurations, were compared to the naive approach of just taking the best HWR result (i.e., taking the best alternative of each word) for the instance recognition.

The results showed an improvement of 1.76% using our algorithm (f-measure = 59.68%). The problem is, this result is only true for the experiments that used a general British lexicon for the HWR. When using an extended lexicon that was created by adding every label of the ontology to the dictionary, the results are worse. Here the simple approach performed with an f-measure of 69.67% what was 1.81% better than the complicated one. Note that the f-measure improved by about 10% by extending the lexicon of the HWR.

I have to confess that these results disappointed me a lot, since they proofed that all the work I did could not improve the result. Anyway, I did some research that failed, even that kind of result is a good result. At least do we know now that the HWR performs that good, that there is no necessitiy to make it complicated.