Handwriting recognition

Today I prepared data for the evaluation of my webservice. I took some texts from http://www.bbc.co.uk/music and extracted rdf data of this service as well as from dbpedia (http://dbpedia.org/).

In the next two weeks I want to let 15 persons write five texts each and addidionally some notes. To evaluate the results will be very interesting to me. I’m really excited, how the results will look like and how I can (hopefully) improve them by changing the algorithm inside my webservice.

When I had a call to my supervisor the last time, we talked about  a publication again. He told me that there is a conference for handwriting recognition (see http://www.isical.ac.in/~icfhr2010/index.html). If I had first results of my evaluation I could write a paper and submit it there. That would be really great.  Especially because this is a specialized conference. Deadline for submitting is February, 15th. So I have to hurry up, to get this done. Of course that means a lot more work to be done, but it could help me in future.

This is an example of how the re-ranking of word alternatives work.

The written sentence is: “Do you know Michael Phelps?”

Hwr returns the following words - some with alternatives (each row is another alternative, e.g. “Michael” is an alternative for “Michel”), ordered by confidence:

Every alternative has a confidence value between 0 and 1 (1 means hwr is sure).

Now each alternative is sent to iDocument. When it found something, this information is saved. Maybe the user knows a “Michael” but no “Michel”. iDocument will find this out and might return information about “Michael”. E.g. that it is a name of a person. This information has a belief value, meaning how sure iDocument is, that this is right.

iDocument might also find more than one information. “Michael” could also be the name of a project in a company the user knows. Every “information-block” has a belief value.

Summing-up:
x: number of alternatives for each word
hc: hwr confidence - each word has a confidence value from the hwr
n: number of information for an alternative, found by iDocument
ib: iDocument belief - each information has a belief value

Now how can I do a re-ranking of the alternatives? I use this formula to calculate a coefficient for each alternative that is used for sorting:

Hence alternatives will be rated higher the more information iDocument will find. The final ranking might be the following:

The last few days I was modifying the interface of my webservice. That became necessary, because iDocument returns a lot more information than I expected. But how to find out, what is important to the user and what isn’t? The goal is to do as much automatic as possible. I don’t want the user to interact more than necessary.

Currently the webserver sends every word that was recognized by the hwr to iDocument. After that it does a re-ranking of the alternatives. Therefore it takes the confidence values the hwr calculated for each word  and the belief values of iDocument. In the next post I will show an example. But first I want to write about another problem: There is much information that need to be visualized on the iLiad. For now I don’t have a clue, how to do this, because there is only a greyscale display and the device is very slow.

The implementation of my webservice proceeds very well. Today I finished the communication between my webservice and the handwriting recognition (hwr) MyScript (VisionObjects). There were some problems with the results this returned, but these are all solved now.

Next step will be the communication with iDocument. I got to know the Api of it yesterday and today. This shouldn’t be too hard, but the Api is in an early state, so it can change almost every day. Another problem is, that I need some things that don’t exist right now and it takes some time for the developer to implement it. I am the first user of the iDocument Api and that’s great. I can take (a very small) part in the developing process :)

After handling the Api I have to decide how iDocument gets the results of the handwriting recognition and after that the hardest part will begin. I must find out what might be the best way to combine the results of hwr and iDocument.

I spent a lot of time in the web, searching for existing solutions for handwriting recognition. I didn’t find very much information. There is some software that converts handwritten text into digital text e.g. from Microsoft or VisionObjects.  Furthermore there is a paper that describes a process of annotating a text on a palm and editing these annotations on your desktop computer (Marcel Götze et al., ViDio – Virtual Digital Annotations).

The good thing for me and my subject is, that I didn’t find anything about improving the recognition performance of handwritten annotations by using ontological knowledge. Therefore I can write about this topic as the first one ever :)