marathon day 21: a long way to go

Data Collection

• total of 117 unique signs (60 2-handed signs + 57 2-handed signs)
• 3 signers per sign; on average 2 samples per sign per signer
• approximately 700 isolated sign samples
• 88 out of 700 isolated sign samples edited
• 63 grouped sign samples
• 37 out of 68 grouped sign samples edited

This is far from complete. In addition to editing the existing samples I also need to annotate and re-sample (resize) them appropriately. Analysis should reveal the direction(s) to take the research; this implies a new list will be made designed to capture/answer the questions raised during the analysis. One question already raised is this: what is the complexity of one-handed signs vs. two-handed signs?

Papers and Presentations

• draft Chapter 1: Introduction almost done
• draft Chapter 2: Review of Related Lit almost done
• draft Chapter 3: Research Problem started
• draft Chapter 4: Methodology started
• draft Chapter 5: Results has nothing yet
• elevator pitch presentation half-way done; needs more images/pictures
• proposal defense presentation not even close

The goal is to produce a draft that can be edited and tweaked for three purposes: (a) a mid-term report; (b) paper for publication; and (c) thesis proposal.

minor updates

• changed "research tips" label to "research"
• added "marathon" label to all research marathon posts
• added title to marathon day 4 post

marathon day 18: small steps

FSL video progress

• 90 out of 127 signs (9 out of 13 groups) recorded
• 2 out of 3 signers recorded
• 90 out of 328 recorded samples processed

Proposal/Thesis Paper writing

• configured LaTeX templates
• draft Chapter 1: Introduction
• draft Chapter 2: Related Literature
• draft Chapter 3: Research Problem

Underestimating the Problem

update: minor edits for clarification

Challenges in Sign Language Recogniton

1. Multiple types of events
2. Multiple channels
3. Multiple interpretations
4. Sign language linguistics

Multiple types of events
From an earlier post, I mentioned that sign language has several things going on. We need to look at the hand shapes, the arm motion with respect to the body, the body posture, and facial expressions; each one a different problem. For example, recognizing facial expressions require a different sort of processing than tracking arm motions.

Multiple channels
We can think of a channel as "one thing to keep an eye on". In sign language we have at least three channels: the left arm, the right arm and the face. Thankfully, we only have a limited set of combinations with types of events. For example, hands are found on the ends of the arms. If we can track the arms, we can easily find the hands. In general, we will never find a head on the end of an arm.

Multiple Interpretations
This is where things get messy. When you see a gesture, it could be one of three things: (a) a lexical(?) sign, (b) a (cultural) gesture or (c) visual action. (I am not sure if lexical is the correct term to use.)

• lexical signs: This is what you would normally think of as a sign - a gesture + hand shape + facial expression that has a specific meaning. You could think of it as words or phrases in sign language.
• gestures: This is a gesture that has specific meaning to a particular culture. For example showing someone an extended middle to insult them. This gesture is known and used by both signers and non-signers alike.
• visual action: Like playing Charades, the motion of the hands, arms, head, and body act out something related to the meaning or idea being conveyed.

For example, let's say the signer taps the side of their head with their index finger. Does that mean (a) the sign for THINKING; or does it mean (b) "he's crazy"; or is it (c) acting out "something long and thin poked my head"?

Sign Language linguistics
On top of all of that, we have sign language linguistics. How can we tell when one sign ends and another begins? Unfortunately, we don't have a clear model for Filipino Sign Language at the sentence and discourse levels, which means that automatic, real-time translation is not possible at the moment.

marathon day 15: I lost my nose

It is time to re-evaluate our progress. What have we accomplished? What have we learned?

Accomplishments

1. Created a screencast of the prototype of the visualization tool for demonstrations.
2. Incremental improvements in the visualization tool.
3. Standardized the mencoder options for my data files. mencoder is part of the MPlayer project.
4. Scheduled additional recording sessions with the Deaf next week.
5. Edited two groups of FSL video recordings. I am looking into automating this with video segmentation.
6. Started working on the problem of metrics. How do I measure similarities between (recorded) signs?

Non-Accomplishments

1. Re-organized the files under my thesis directory. Includes updates to the back-up scripts.
2. Added to the goals for the marathon: video segmentation module
3. Added to the goals for the marathon: hand shape recognition
4. Added to the goals for the marathon: hand tracking
5. Constantly underestimate the scope of sign language complexity.