Japanese learning tool with HTML5 interface

Web. 16:34, 2011.06.20.

I always find useful those language learning tools that are aimed to train a single specific skill among those needed to master a foreign language (one tool one task).
I was looking for something that could help me to build some knowledge about daily sentences, both in meaning and in pronunciation, showing kanjis while providing also them as kana. Like i did with playkanji.com i decided to build an user interface in JavaScript to fulfill the need: teacher.playkanji.com.

Japanese suits very well to software development since it isn't really necessary to have your program translate a sentence (very hard) to help the user with additional info. Chasen provides some help in parsing sentence structure while Edict is the most famous digital japanese dictionary.

My web application is really HTML5 dependent, it makes use of some local storage functions as well as audio tags. You are supposed to read and listen the sentence clicking on the play ♪ button, the if necessary some info (meaning and pronunciation) of displayed kanjis is provided clicking on kanji help (help 漢字). While typing your translation in the grey box it will be parsed and evaluated in real time. It will be compared to the reference translation, and matching word will be highlighted in blue. For those interested in the internals of it I'm using the Levenshtein distance to highlight your words in a blue shade getting darker while they are getting closer to the correct word.
One limitation of my batch of data is that dictionary references that you will find clicking on help won't exatcly match the reference translation, since it would require large amount of manual work to fix. Consider them as a hint and help yourself with some synonyms while messing with Levenshtein output.

HTML5 local storage capabilities let me save the study history within your browser, so there's no need for login, password and so on: just fire up teacher.playkanji.com and enjoy.
Clicking clear this result and clear all history in the right upper side of the page will lead you to reset the current result and the whole history respectively.

Drop me a line if you spot a bug or a bad translation, the app has not been tested thoroughly and i'm still drilling through sentences. Currently supports only Firefox and Chrome, don't even bother trying on IE. Opera and safari untested.

Linux kernel programming exercises 1

Linux. 18:33, 2011.04.17.

This is the first post of a series (i hope) about linux kernel development.
Despite there are excellent books [1][2] that describe throughly both kernel and modules, there aren't many sources that provide programming exercises in the form task-hint-solution.

Purpose of this series of posts is to compensate for that lack. Source of exercises will be available on github and commented here. Please note that modules/patches described here won't always be useful as they are just intended to increase linux kernel hacking experience.

Task

The first proposed task is to develop a kernel module aimed to check the ARP table to detect when something fishy is going on. Requisites:

• Solution has to be a kernel module, as it can be done without harming the kernel tree. ARP checking can be easily done in user space reading /proc/net/arp but that won't help in undestanding how the linux kernel works.


• run checks at regular intervals of time without any busy waiting.


• Checks should detect duplicated entries in the ARP table. That's an acceptable approssimation to detect arp spoofing.

Hints

Some way to efficiently parse the kernel source tree is mandatory, oneliner below does the trick:

find . -regex ".*\\.\([ch]\)" -exec grep -Hn $1 {} \;

Symbols accessible from modules are explicitly exported by following macros:

• EXPORT_SYMBOL(..)


• EXPORT_SYMBOL_GPL(..)

so running the previous line of code within linux-2.6.38.2/net/ipv4 results in a lot of matches, which grepped again for "arp" lead to a reasonable amount of lines.

./arp.c:119:EXPORT_SYMBOL(clip_tbl_hook);

./arp.c:202:EXPORT_SYMBOL(arp_tbl);

./arp.c:515:EXPORT_SYMBOL(arp_find);

./arp.c:718:EXPORT_SYMBOL(arp_create);

./arp.c:728:EXPORT_SYMBOL(arp_xmit);

./arp.c:754:EXPORT_SYMBOL(arp_send);

./arp.c:1160:EXPORT_SYMBOL(arp_invalidate);

./netfilter/arp_tables.c:61:EXPORT_SYMBOL_GPL(arpt_alloc_initial_table);

./netfilter/arp_tables.c:1900:EXPORT_SYMBOL(arpt_register_table);

./netfilter/arp_tables.c:1901:EXPORT_SYMBOL(arpt_unregister_table);

./netfilter/arp_tables.c:1902:EXPORT_SYMBOL(arpt_do_table);

arp_tbl looks like a good place to start from.

Solution (full code)

arp_tbl is an instance of a more general table struct neigh_table, which is used to keep track of associations between network and data link layers. Likely there's a set of functions ready to parse it laying somewhere in the kernel tree. net/core/neighbour.c can be found with the same combination of find and grep cited above. The function needed is neigh_for_each, which is also conveniently exported to modules. Function usage can be learnt by ./decnet/dn_neigh.c:535:. Note that nr_neigh_for_each is not the same.

neigh_for_each requires a callback function that will be called once for every entry in the table. Optional argument is not required in this case.

void neigh_handler(struct neighbour * n, void * null)

{

  struct neigh_list_t *tmp;

  int found = 0;

  char hbuffer[HBUFFERLEN];



  /* search */

  list_for_each_entry(tmp, &neigh_list.list, list) {

    if(memcmp(n->ha,tmp->ha,n->dev->addr_len)==0) {

      format_hwaddr(n->ha, n->dev->addr_len, hbuffer);

      printk(KERN_ALERT "duplicated entry: %s\n", hbuffer);

      found = 1;

    }

  }

  

  /* add an entry */

  if(!found) {

    struct neigh_list_t * new_entry = (struct neigh_list_t *) kmalloc(sizeof(struct neigh_list_t), GFP_KERNEL);

    memcpy(new_entry->ha,n->ha,n->dev->addr_len);

    memcpy(new_entry->primary_key,n->primary_key,sizeof(u8 *));

    list_add(&(new_entry->list), &(neigh_list.list));

  }  

}

Standard kernel data structures have been used to keep track of previous entries. A list is filled with IP and MAC addresses and then compared to each new neighbour entry. When a MAC address is not found in the current list it will be added at the end of the loop. When a MAC has already be seen instead, "duplicated entry: XX:XX:XX:XX:XX:XX" is printed and visible through dmesg.

To run this check every N seconds kernel workqueues have been chosen.

static struct workqueue_struct * workq;

static DECLARE_DELAYED_WORK(work, arp_tbl_check);

...

workq = create_singlethread_workqueue("arp_tbl_check_wq");

queue_delayed_work(workq, &work, HZ * 5);

For this case a simple single-thread single-work queue would suffice. HZ * 5 sets the next run at approximately 5 seconds of jiffies away. Grab full code.

Vehicle classification

Motion. 17:56, 2011.01.27.

I improved my ruby motion recognition library, RMotion, to provide object classification features.
Currently it has been tested on vehicles, expecially on cars shapes. In the following demo video it is configured to recognize cars (red label) and filter out everything else (blue label). Cars which are within the camera only by an half are not classified as matching, and that's the intended behaviour since their shape isn't looking like a car one.

Video is deliberately slowed down, since here in front of my window people drive insanely fast! As usual i'll write some documentation and usage examples and post them here.