Motion Capture in conjunction with Unity

Discussion in 'Unity Gossip' started by Myhijim, Jul 12, 2013.

  1. Myhijim

    Myhijim

    Member

    Joined:
    Jun 15, 2012
    Messages:
    1,046
    Hey everyone, recently my team and I have started looking towards motion capture for our animations and found that the Xbox 360 kinect works as quite a decent motion capture system alongside Brekel(free). This post will revolve highly around that software.

    Now our problem is most certainly from an animation/rigging perspective, so we have a few questions.
    1. Do you use any motion capture within your games? What program do you use?

    2. Is motion capture most useful in terms of cutscenes over actual gameplay animations? Is using motion captures for ingame animations feesable?

    3. How would one go about transferring said animations from the default Kinect rig to one of a more complex nature?



    Thank you for taking the time to look at this thread, we are highly considering motion captures as they seem extremely fluent from tests and should cut the workflow strain of the animation department by a fair amount



    Helpful sources

    Brekel :
    http://www.brekel.com/?page_id=155

    Awesome Working Example of Brekel in Animation :
  2. Justei

    Justei

    Member

    Joined:
    Aug 31, 2012
    Messages:
    115
    Something that people don't realize when it comes to MoCap is that it's not something that will lower your amount of work by a lot.

    Actually having worked with it quite a lot, especially with markerless mocap, it's not something that cuts down the amount of work or time spent on animations. Actually in some cases it might take longer to mocap something than it will doing it manually. This might not be the case with everything, but in a lot of them, it will.

    The only reason I ever use our MoCap studio is if we need a specific animation that is hard to produce, for example for complex cutscenes. Where doing it manually doesn't make sense to me, otherwise I'll definitely get something manually made. Just as a heads-up.

    This software however is slightly better than the one we use, so it might do a better job, but from my personal experiences doing it manually is faster and less work. Simply because you cut out the whole acting - tracking - fixing the animation in MB part of the process.
  3. Myhijim

    Myhijim

    Member

    Joined:
    Jun 15, 2012
    Messages:
    1,046
    Thank you for such a fast reply it is greatly appreciated!

    Maybe in terms of cutting down time that is only really relevant to our current skill level and it does seem much more fluid than anyone in my team could currently make as we are all hobbyists.

    Most importantly here, did you transfer from a simple rig to a more expansive rig? If so, how did you achieve this. I mean the kinect rig I believe has about 13 bones, and does not include fingers, which we do require, would it be possible to transfer this data from this rig to a rig that incorporates fingers and manually edit it from there?


    This software is free and so far I like what it has shown me in terms of simplicity, it is just getting those animations to game ready.
    Last edited: Jul 12, 2013
  4. Justei

    Justei

    Member

    Joined:
    Aug 31, 2012
    Messages:
    115
    That should not be a problem if you use some software like Motion Builder, I haven't been using it for some time, but if I remember correctly it wasn't a big problem at all, you just had to re-target and it worked fine.

    However I'm sure someone more qualified could answer that specific question better. I only know basics and the general workflow we use for Mocap, I don't do it myself unless there really is no choice :).
  5. Gnimmel

    Gnimmel

    Member

    Joined:
    Apr 21, 2010
    Messages:
    218
    Motion builder is the standard for fixing motion capture, but if you want to use any other 3D software, it's easy enough to create a custom rig. Basically make any animation rig you need with fingers etc and then connect it to the kinect rig. By connect I mean constrain all your controls so they follow the kinect rig, things like parent constraint the ik foot control to the kinect rigs foot etc. You can make a real animation rig, splineIK spine controls, IK/FK switching etc as long as you can connect all these controls back to the kinect rig. In the example of IK/FK switching, just make sure both controls are connected, so the IK hand can be parent constrained to the kinect hand and the FK arm controls are orient constrained to the kinect rig. Remember you need these connected, not parented because in the end the animation curves need to be in the same space as the animation rig. Once you have that set up, any motion you capture and put on the kinect rig will drive the animation rig. From here you need a way to transfer that driven animation onto a clean version of the animation rig. Depending on your software you could script something or perhaps simply bake it out and save the new animations curves. Once it's all been transferred, you have a motion capture motion on a real animation rig which you can now fix and add all the extra's like finger animation etc. I've done this many times in maya through script to automate the export of animation data, but a similar method should be possible in most 3D animation software.
  6. lmbarns

    lmbarns

    Member

    Joined:
    Jul 14, 2011
    Messages:
    1,173
    I was thinking of buying iclone and their plugin but am glad I didn't as this seems to work fairly well. Only problem I'm having is in 3ds max the model/rig plays the animation perfectly but when I export to fbx and open in Unity the limbs are all messed up. Tried exporting to blender, then export to fbx and the animation didn't come through. Also i recorded as bvh then switch to bip in 3ds max before applying to my model...wonder if I need a different model without fingers or something...


    edit:: spent all day at work messing around with mocap as well as modeling/texturing with the kinect via "Skanect". Pretty neat, can't wait for the new sensor to come out....
    Last edited: Jul 12, 2013
  7. lmbarns

    lmbarns

    Member

    Joined:
    Jul 14, 2011
    Messages:
    1,173
    I pointed my kinect at my co-worker and generated this 3d textured model lol:
    $coreyModel.png $coreyModel2.png

    I'm sure it would be better if I had a tripod or something rather than waving it by hand, and the new kinect controller will really make this amazing.

    But damn I can still just take the texture and tweak it after the auto texturemapping. Also Im just using the free trial of the program which only does low resolution objects but it's only $99 euro for the pro version....
  8. Forge Vault

    Forge Vault

    New Member

    Joined:
    Feb 18, 2012
    Messages:
    214
    Either way your going to need an experienced animator.

    In any case it doesn't hurt to have some of Preston Blair's diagrams handy - its pretty much the bible of animation(don't be fooled by the cartoony illustrations). http://animationresources.org/?p=2091
    The biped animation charts are further down the page, there are also front motion charts but oddly enough there missing from this page.
  9. Olafson

    Olafson

    Member

    Joined:
    Aug 7, 2012
    Messages:
    149
    We have been using Motion Capture data since quite some while now and we are more than satisfied. None of us have ever learned how to animate things by hand, so obviously we are pretty bad at it
    Using the Motion Capture data we recorded our self with IPIsoft we were able to achieve quite great results.
    Obviously, fingers will have to be animated by hand, but we can always hire a professional animator to do that for us later. Overall, using software like IPIsoft greatly reduces the work, time and most importantly the money that is needed to get great looking animations.

    IPIsoft provides you with a very easy and fast solution to remove any jitter etc. from the recorded animations and its exported animations and rig is more or less already usable in a game. It is perfect. In terms of time, we have been able to record all our animations in less than 3 days, and editing them took about 1 month. And we have a lot of animations.

    Here is our badly written, boring and short DevBlog about it: http://www.fsegames.eu/devblog4.html
  10. AaronC

    AaronC

    Member

    Joined:
    Mar 6, 2006
    Messages:
    3,203
    Mocap is part one of two, part two being the cleanup, retargeting and fine touches.

    If modelling humanoids I use a motionbuilder friendly skeleton to begin with at rigging stage but thus far mobu and mecanim dont seem to want to be friends, so I'm not going to bother too much with mecanim until I can figure out what the problem is. Motionbuilder is more useful as Unity indie mecanim doesnt have IK tools.

    Mocap would be essential in my view for cutscenes etc if you want realistic animation, but good software to cleanup is just as essential.
  11. Bael

    Bael

    Member

    Joined:
    Jul 21, 2008
    Messages:
    106
    To start, motion capture is a tool to assist animators, not to replace them. You should never be taking raw mocap data and plugging it into your game/cutscene. Mocap is a time saver (provided the quality is good), but you still need animators to clean up and enhance the motion for your particular use - this is especially true when using low-quality capture solutions. Think of it like a texture artist and photographs - you use the photos as a base to build upon, not as the end result.

    Mocap is most beneficial for cinema work, but you can use it for gameplay animations as well (and lots of studios do). But keep in mind, using mocap for gameplay animations requires much more pose tweaking and timing adjustment. Real life movement is slow and unresponsive, so unless you want a slow and unresponsive game you'll have to edit your mocap data.

    As for software, MotionBuilder is the only package you should need - it's the industry standard for working with motion capture data. With it you can clean up the data, re-target it to other skeletons, enhance the posing and timing, and basically anything else you'll want to do.

    Lastly I think it's necessary to discus quality. Brekel is horrible. Every motion in that video looks horrible. Everything from the waist up is jerky and unnatural, and everything below the waist is just flat out broken. Hyper-extended limbs, popping joints, overly smoothed rotations - basically everything you don't want in mocap it suffers from. It can't even track standing conversations properly, so forget about doing any fast actions with it. Even other similar packages like iPisoft, while being budget quality themselves, are head-and-shoulders above Brekel. Basically if you're going to use Brekel - invest in a good animator.

    Motion capture will always speed up the animation department, provided mocap is a fit for your project ('realistic' humanoid) - but you have to be smart in how you use it. Quality of the data, and the actors performance are the two most important aspects of motion-capture. If either one of those are sufficiently compromised, you're really no better off than having an animator hand-key it.
  12. Myhijim

    Myhijim

    Member

    Joined:
    Jun 15, 2012
    Messages:
    1,046
    Wow so many replies, thank you guys for all of the info it is greatly appreciated!

    That is extremely useful thank you than got rid of the majority of my doubts of motion capture. I will have to send this to my animators/riggers for them to set this up as I understand little to nothing of what you just said.


    I'm smelling a brilliant kickstarter reward right here, "Get a small model of yourself with ingame armour!" Hehe. That program looks great and it would definatley be a buy for me soon.


    Thanks for that, we do have 2 fairly experience animators.


    Motion capture was something I stumbled across that I suddenly realized I could use my kinect for it and that makes my life and my interest in this project so much greater.

    I am currently really considering dismissing Brekel and going with Ipi Soft as it seem more clean and well.... Just plain better. Thanks for the heads up!


    Ah, I am hoping we can twist mecanim our way but we will see, so far it is playing nice but it may not when we import these new animations.


    Thank you for this very comprehensive post, it has been very enlightening.

    By no means am I attempting to remove my animators, this is just a way to achieve much more realism in the game.

    I realize now that I have compared with Ipi Soft that Brekel is quite horrible and after testing has proven that also

    Dunno how great of an actor I will be though :p



    Thank you all for helping me out in this matter, it is greatly appreciated and this shows the quality of the help of the Unity Community.

    Cheers
    Myhijim
  13. AaronC

    AaronC

    Member

    Joined:
    Mar 6, 2006
    Messages:
    3,203
    You should google David Bain, bit of a doppleganger thing going on here.

    Cheers for the heads up on skanect :)
  14. Dan Fury

    Dan Fury

    New Member

    Joined:
    Jul 18, 2010
    Messages:
    158
    After working with iPisoft for 1 1/2 years, all I can say is that you will get great raw data out of it and then you have to spent 5 times the money and time on motionbuilder to make these animations truly useful. You can get a bunch of raw animations on the net, but there's a reason why e.g. companie like Mixamo ask 50 bucks for a cleaned up animation.
    I would first check their store and only if you can't use these animations invest the time and money into building your own, because it is a huge amount of work to cleanup and animate the rest that Kinect captures don't provide.
  15. DanoG

    DanoG

    Member

    Joined:
    Dec 13, 2013
    Messages:
    33
    Hi there Myhijim,

    Was just browsing the forums after posting some replies in regards to our new tool we released in the asset store. It may solve some of the problems you are encountering and give you some good results.

    http://forum.unity3d.com/threads/217012-Cinema-Mo-Cap-Released

    Hopefully this will help you guys out! The price is right, as it's a pretty low cost investment to get some good base animations for your projects.

    Cheers,

    Dan
  16. lazygunn

    lazygunn

    Member

    Joined:
    Jul 24, 2011
    Messages:
    1,974
    ipisoft is great fun for grabbing mocap, its not very accurate and you need to be careful about clothes youre wearing what youre standing on and what your hair's doing (mainly for girls). the innacuracy i can see definitely being the reason someone above said it was to be a basis not an end result. i dot really mind though, i like the fun. retargettng wise,ipi straight to unity seemed to zero odd stuff, running it through motionbuilder cleans it up a lot and ensures a very easy import to unity

    even then your capture might just not be working out, but mecanim has really good tools for getting the good stuff out of the capture - even all that carnegie mellon stock mocap that seems unsuable can be tweaked into something usable with mecanim, its a fun process too. i'd definitely say mecanim was an essential part of the process and has me in slight disagreement with an above comment implying there need be some awkward delay within animation game mocp. It's not true, least in my experiences with mechanim. Changing the snappiness of transitions is one thing, but you can do tones with the information it gives you about how likely a natural loop you can get, cut out extraneous movement, keeping orientation steady and so on

    Mecanims half the fun of getting your animation right I think. Had great fun with a project earlier in the year using some kinect mocap and unity
  17. eliteslayer

    eliteslayer

    Member

    Joined:
    May 12, 2013
    Messages:
    24
    How do i connect my kinect to the computer? most kinesb port so it can slide into the computer easily but my kinect came with a bundle and the thing to at the end of my kinect looks like a usb but it has a slanted corner making it impossible for me to connect my kinect to the computer. can anybody help me with that?
  18. AaronC

    AaronC

    Member

    Joined:
    Mar 6, 2006
    Messages:
    3,203
    The Gen 1 xbox version had an additional power supply, Like a USB extension with a power point. You need those if using kinect/iPiSoft
  19. IntegerSet Games

    IntegerSet Games

    New Member

    Joined:
    Dec 23, 2013
    Messages:
    1
    I've found motion capture to be a huge boon for my current project, but I wouldn't say it's necessarily easier or faster to use, but more that it allows a less expensive workflow for me, as I can't afford to be hiring a professional animator just yet.

    Right now my setup consists of two Kinects, my old PC, and my basement. I was originally using four PS Eyes, but due to the poor illumination, there were a lot of synchronisation issues. The Kinects solved this handily, probably thanks to the IR arrays. Finally a use for the Kinect, I suppose! I also get simple FBX data that goes right into Unity smoothly, but for obvious reasons I don't do that just yet.

    See, unfortunately, even though I get some really fantastic capture data as far as accuracy is concerned, head, hand, and finger tracking are either not support by my setup or not supported by the software just yet. This means that after I've captured everything, I need to bring it in to the 3D editor of my choice and manually add the head position, hand positions, and finger positions manually. As I said, I don't have an animator, but this is enough that I'm able to handle it myself, however slowly. Then I have to check the mesh itself and make sure there's no curious deformations or clipping issues.

    For the investment cost, I'd say it's a great savings for me, but my current game requires a lot of specific motions that I otherwise wouldn't be able to reproduce myself without them looking either really stiff or incredibly poorly timed.

    Would I recommend using motion capture? Maybe. If you have beginner-intermediate level skills with 3D animation, a really tight budget, and the space to set it up on your own? Sure. If you can afford to hire a professional animator and they don't explicitly recommend this to you? Absolutely not.
  20. CBosselmann

    CBosselmann

    Member

    Joined:
    Dec 9, 2012
    Messages:
    188
    Hahah yeap, thats uncanny. Bain's ears are much worse though and at least this guy has the sense not to wear a knitted cardigan.