Underestimating the Problem

update: minor edits for clarification

Challenges in Sign Language Recogniton

1. Multiple types of events
2. Multiple channels
3. Multiple interpretations
4. Sign language linguistics

Multiple types of events
From an earlier post, I mentioned that sign language has several things going on. We need to look at the hand shapes, the arm motion with respect to the body, the body posture, and facial expressions; each one a different problem. For example, recognizing facial expressions require a different sort of processing than tracking arm motions.

Multiple channels
We can think of a channel as "one thing to keep an eye on". In sign language we have at least three channels: the left arm, the right arm and the face. Thankfully, we only have a limited set of combinations with types of events. For example, hands are found on the ends of the arms. If we can track the arms, we can easily find the hands. In general, we will never find a head on the end of an arm.

Multiple Interpretations
This is where things get messy. When you see a gesture, it could be one of three things: (a) a lexical(?) sign, (b) a (cultural) gesture or (c) visual action. (I am not sure if lexical is the correct term to use.)

• lexical signs: This is what you would normally think of as a sign - a gesture + hand shape + facial expression that has a specific meaning. You could think of it as words or phrases in sign language.
• gestures: This is a gesture that has specific meaning to a particular culture. For example showing someone an extended middle to insult them. This gesture is known and used by both signers and non-signers alike.
• visual action: Like playing Charades, the motion of the hands, arms, head, and body act out something related to the meaning or idea being conveyed.

For example, let's say the signer taps the side of their head with their index finger. Does that mean (a) the sign for THINKING; or does it mean (b) "he's crazy"; or is it (c) acting out "something long and thin poked my head"?

Sign Language linguistics
On top of all of that, we have sign language linguistics. How can we tell when one sign ends and another begins? Unfortunately, we don't have a clear model for Filipino Sign Language at the sentence and discourse levels, which means that automatic, real-time translation is not possible at the moment.