Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Messing around with gestures

Discussion in 'Made With Unity' started by Charles Hinshaw, Sep 19, 2008.

  1. Charles Hinshaw

    Charles Hinshaw

    Joined:
    Feb 6, 2008
    Posts:
    1,070
    This evening, I decided to take a brief break to write some code -- I've been pretty lazy about learning C#, and thought that an interesting project might be to write a tool for gesture recognition.

    I built the recognition part -- the end goal will be to have a functioning tool that allows custom gestures to be recorded in the editor. As it is, I had to encode them by hand, so I only did "A", "B" and "C".

    It isn't much to look at, but I wanted to share. When I complete everything, I'll release the tool. That said, it isn't a priority, I had just finished what I was working on for the night and wasn't tired.

    Oh, and since it will probably be asked -- no, iPhone wasn't even a thought while doing this. Neither was recognizing letters, either. I had a conversation a few days ago with a friend about a use for a gesture system in a very specific context and I guess it has been on my brain a little bit since then.

    Instructions are on an image on the document -- this is a web player.
     

    Attached Files:

  2. attilam

    attilam

    Joined:
    Jun 2, 2007
    Posts:
    86
    Works nicely here!

    What sort of algorithm did you use for this?

    I made a very simple one after reading an article a long time ago. That one was based on fitting all the points of the drawing into a 3x3 matrix, then creating a "path" within the matrix that described how the symbol was drawn in time.

    Worked alright, but of course it had some problems and wasn't too sophisticated in recognizing everything :)
     
  3. bigkahuna

    bigkahuna

    Joined:
    Apr 30, 2006
    Posts:
    5,434
    Pretty cool! Works nicely here also. Folks who use tablets might like to see something like this integrated into the editor I bet.
     
  4. Charles Hinshaw

    Charles Hinshaw

    Joined:
    Feb 6, 2008
    Posts:
    1,070
    The basic algorithm is:

    - 3x3 matrix of directions
    - ignore time and capture a string of relative moves
    - remove repetition from the captured string
    - calculate levenshtein distance from known symbols

    I think that there are also special cases that should be able to be flagged on a known symbol, like starting and ending quadrant and height:width ratio. That would prevent confusion when two very different symbols have a similar sequence. For example, right now, if you draw the right leg of an A with a slight curve going in and back out, it can be mistaken for a C since removing the left leg is the shortest path to a known symbol and it can generally be done with a cost lower than my threshold. Defining that the C has to start in the upper-right quadrant would remove that confusion.
     
  5. jeremyace

    jeremyace

    Joined:
    Oct 12, 2005
    Posts:
    1,661
    Some people have had success using ANNs (Artificial Neural Networks) for the recognition.

    Then you can have the user actually teach custom gestures to the system, far more complex gestures, etc.

    Might be worth checking out if you are interested.

    -Jeremy
     
  6. Joe ByDesign

    Joe ByDesign

    Joined:
    Oct 13, 2005
    Posts:
    841
    Just saw this... Nice scene Charles; cool tech! :)

    Am curious, have you used this in a project as of yet? Would like to see the application.
     
  7. Dakta

    Dakta

    Joined:
    Apr 8, 2008
    Posts:
    492
    Thanks for the bump! I almost forgot abut this.


    Any uses for it yet?
     
  8. rawrman2381238

    rawrman2381238

    Joined:
    Jan 28, 2008
    Posts:
    122
    awesome! any chance of sharing the code?
     
  9. pavees

    pavees

    Joined:
    Jan 1, 2009
    Posts:
    116
    really looks cool...:) i have a request just guide me on getting the circle detection or round gesture. Or if possible give me some sample coding to attain it.
     
  10. Randy-Edmonds

    Randy-Edmonds

    Joined:
    Oct 10, 2005
    Posts:
    1,122
    Really cool.. thanks for sharing the basic algorithm. I was wondering how this type of thing could be done on the iPhone and now it seems fairly easy. awesome!

    By the way, I found this on the internets...
    Levenshtein Distance Algorithm: C# Implementation
    http://www.merriampark.com/ldcsharp.htm
     
  11. Sandor

    Sandor

    Joined:
    Apr 17, 2009
    Posts:
    91
    Hey a very nice one! Reminds me on the $1 geture recogniser:

    http://depts.washington.edu/aimgroup/proj/dollar/

    This one is very impressive. It is orientation independent, a very good recog-rate and on plus: there is a C# and a javascript implementation there...

    Cheers,
     
  12. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
  13. Sandor

    Sandor

    Joined:
    Apr 17, 2009
    Posts:
    91
    Hey Mads,

    sounds very interresting. But i am having troubles running your example project. I am getting this errors: "Assets/code/gesture.js(26,32): BCE0005: Unknown identifier: 'MouseLook'." Also the slots in your script GO are missing the scripts... Would be nice to see the scene running...

    Cheers,
     
  14. pavees

    pavees

    Joined:
    Jan 1, 2009
    Posts:
    116
    Mee too getting a lot of error in many script files. Pls can you reffer a proper error free code (or sample).
     
  15. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
  16. Sandor

    Sandor

    Joined:
    Apr 17, 2009
    Posts:
    91
    Hey Mads,

    thank you very much for uploading the new project. At the first look it is working very nicely - cool job. I will have a more extensive look the next few days... I'll try to bring that on our MultiTouch table... Will keep you upt-to-date...

    Cheers Thanks again,
     
  17. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
    Great Sandor

    Looking forward to hear, and maybe see, what I can do on a touch table.

    Please keep me updated.. sounds very interesting.
     
  18. pavees

    pavees

    Joined:
    Jan 1, 2009
    Posts:
    116
    hei, all of u there, I am very happy to say that i have completed my task to get the the circle detection, or the round gesture detection.

    the gesture files or example attached to this post, i was not able to run it bcos of some error and so. any way thanks to all who supported..
     
  19. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
    awesome!
    Would very much like to see what you guys are using my recognizer for. I just used it as an interaction method for magic spells in a casual game.

    pavees - is it for a game?

    Sandor - any luck with the multi-touch table?
     
  20. pavees

    pavees

    Joined:
    Jan 1, 2009
    Posts:
    116
    yap its for a game, an animation based game and those animations works on the basis of the gestures. I was not able to use ur gesture detection method as i found many errors in ther.
     
  21. ibmeruu

    ibmeruu

    Joined:
    Apr 21, 2010
    Posts:
    18
    When you save a new template by holding down control key where is it saved to? After I save one it doesn't go back to recognize mode.

    Thank you.

    Ps this is awesome!
     
  22. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
    Thanks, nice to know someone can use it!
    In the current version, it only saves runtime.
    If you search in this forum for how to write to a file, you could save the gestures and reload them each time.
     
  23. ibmeruu

    ibmeruu

    Joined:
    Apr 21, 2010
    Posts:
    18
    Im ok with saving at runtime only, but after I save a template it doesn't go back to recognize mode.

    Am I doing something wrong?

    and saving to a file is a good idea but the next step.

    Thanks again.
     
  24. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
    As I write in the wiki page:
    The user right clicks (held down) in order to draw a gesture in the 3D scene. When released the recognizer starts. The gesture will be recorded and added to the gesture template bank if the user hold down CTRL while right clicking.

    So when you have saved a template, you should be in recognition by default. Then just use right click method again.
     
  25. viezel

    viezel

    Joined:
    May 16, 2009
    Posts:
    19
  26. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    That's great, but I'm not sure they have been translated into c# Unity code as they appear to be missing monobehavior on 2 of the classes? Do you just mean generic c#?

    Also, does the sample JS project on the wiki work at all?
    I had a quick test on Unity3 and it fails to do anything.

    EDIT: Working now. Looks like gesturerecognizer was missing from the plane object. Also fps script is missing.

    Thanks for this!
     
    Last edited: Oct 1, 2010