Dynamic Music - Call for Feature Requests!

I noticed that there's no dynamic music system available on the Asset Store, so I've decided to make one! The system will be creative team-facing, so the iteration loop can stay fully with composers, implementers, and designers. Though I have an initial plan working, I'd like to find out what the community would like to see in such a tool.

Here is the initial list of high-level features:

• Dynamic mixing of "Tracks", which include stem/instrument audio and some configuration data
• Organization of Tracks into "Songs", which represent the full piece of music
• Tagging of Tracks with "Moods", which are implementer-created descriptors
• An interface for the implementer to hook up gameplay elements/events to Moods (and the intensity of those moods)
• A nice inspector to manage Songs, Tracks, and Moods

If this sounds like a system you would like to use, please let me know what you would like to see or how you envision yourself using it. If you don't think you'd be interested in this feature set, but have another idea of how a dynamic music system should work, I'm interested in hearing that too!

I've made a bit of progress on the inspector and dynamic mixing fronts, and eventually I'll set up a section of my blog for project updates. For now, I'll check this thread and post updates as progress happens.

Thanks!
-c

 

I wrote one of these last year for Folk Tale, exactly as you are describe, after seeing a presentation on audio from Mass Effect. Worked pretty well with in-memory samples on a single track, until I started to push it and then the sync fell apart ( I was probably using an update cycle that was too slow, as I was still learning Unity back then ). In reality using this in a real game didn't appear to be feasible. The section samples ( e.g. strings, percussion, lead ) often result in the total file sizes for your song being larger than a single soundtrack file. Further, it's not practical to keep soundtrack files in memory in a game unless you have 1 song per level and never change it. And you don't want to be decompressing from compressed memory in real-time. So that means streaming. Streaming has it's own problems when you introduce multiple streams in Unity, especially on lower end "minimum specification" PC's.

In the end I booted the whole lot out and opted for a simple cross-fading single streamed track approach. This was in Unity 3.3. The audio handling may well have improved since then. Of course you've still got the sucky garbage collection spike issue to address that will break your audio, or high CPU usage causing audio pops.

Hippocoder may well be able to provide some insight into how best to proceed given his audio app experience.

If you are going to push ahead with this, I'd recommend you stress test it as part of a full game, with lots of channels, and cross-fading songs, simulating how a proper game may use it. Then look at the profiler to make sure it's performing, and not consuming many ms or making allocations per frame. Remember, soundtracks should consume very few ms per frame as the rest of the game needs those. It's no good writing this if it consumes 10ms per frame.

I'll be following this thread with interest because if you can solve all the issues described, it'd be a very useful asset.

 

Hi there. I've actually been working on a generic one of these for a while now in my spare time. It's currently in the process of being redesigned from a non-streaming prototype to an approach that protects sync streaming.

If you're interested in building your own, I'd suggest you have a look at my slides from my presentation at the Vienna Game AI conference as it contains some detail on some of the challenges and design choices I've made working on my own system.
http://www.cratesmith.com/archives/652

Cheers!

 

Awesome! Thanks for the words of wisdom, and yes, performance is a major concern. I'm interested to see what roadblocks Unity throws up for working with multitrack music, as I've only done "standard" audio system implementations in it until now.

 

After putting in a little more time on this project, it seems like these issues can be adressed. At the very least, it will be the usual tradeoff of more tracks vs. better performance, but those of us who have worked in game audio are used to that.

Updates, in no particular order:
- Sync between tracks works, testing with 13 tracks going in and out in a blank project and in the AngryBots project.
- AudioSource pooling will, in theory, help with performance by limiting instantiations as well as not playing audio once it gets turned all the way down.
- Trying to eliminate doing things in Update(). The only remaining holdout is how I'm handling fades (suggestions?). Really wish there was an engine-level Fade( targetVol, time ) call, so it wasn't necessary to do it in script.
- Switching songs, changing intensity, changing moods are all in.
- The interface for defining songs is bare bones, but in.

Looks like I'll have to start scraping together the dough for a pro license so I can profile this thing.

P.S. - Cratesmith, I only see 3 slides there, but thanks!

 

Sync is a nightmare, but you can use co-routines to make sure you're firing out data regularly. But the only real way to mix is to mix it yourself from arrays using unity's sample buffer stuff. This way, you're guaranteed a well timed mix. This would be a bit of a disaster on mobile unless you target the current gen.

Thing is, FMod supports ALL this dynamic crap and even has a dedicated tool for it called Designer, and unity expressed interest in this a while back (rolling their own tool to be used in unity). However, I have not heard a peep out of them since. I wouldn't be surprised if it's something coming up in their minds though what with everyone asking for better audio for half a decade.

 

I'd take hippo's word on the sample buffer mixing as he knows what he's talking about there.

Expanding on the co-routine approach, one approach would be to make this loop at a high frequency to ensure you hit the sync, such as every 1/200th second or higher. The routine would need to be extremely fast with close to zero overhead.

A better approach that might work but requires the dev to specify the track bpm could be to fix the routine loop to trigger every four bars ( quick calculation based on bpm ) and force replay of all the instrument tracks at that point ( risk: delay in audio engine firing samples ). I'd test this approach first. One thing I found is that relying on the samples to auto-loop at the correct point is not an option, as musicians often provide loops of non-uniform duration, causing some instruments to loop sooner or later than others. 1ms out compounds quickly when looping and soon becomes audible to the trained ear.

 

The way I usually with sync issues with looped music is to periodically call a resync. Just pick one loop and set everything else to the same sample. As long as the loops aren't way off in terms of length it won't be noticeable, and it definitely won't accumulate. Trying to maintain perfect sync is not only unnecessary, it's crazy expensive.

I just dealt with one issue where instantiations were causing a frame to be a bit too long, and messing up my sync. Firing off a coroutine to resync in a few frames after the instantiation fixed it right up. I'll probably have to figure out a more robust solution when the game and the music system are both doing lots of instantiations, but it works for now.

And yeah, believe it or not this is the *second* game engine I've worked with that has FMOD hidden away from the people who would use it.

Thanks for the insight folks, it's much appreciated!

 

This can get as complicated as anything. Dynamic music really is an unexplored territory. If it's allowed to develop in the games industry we could eventually get music that changes instantly according to all manner of possible data of how the player plays and create a soundtrack that is just as convincing as a carefully crafted score to an action movie. The knowledge exists, but the processing power to use so much of the processor that is needed (about one core should suffice) isn't prioritized by the major game studios. :-|

I'm studying interactive sound design and previously I've studied algorithmic composition, I think what's needed for a big change in this aspect is tools that are approachable for people with a music making background, actually sequencing and using that material to build a dictionary etc. There's a lot of research on this and on what works and what doesn't so these aren't simply pipe dreams. I recommend reading a lot of computer music journal so as to not reinvent the wheel.

 

Hey Veqtor, you've essentially summed up why I'm making this. I also studied computer music.

And yes, the goal of this project is to make composer-accessible tools so little is lost in translation between content creation and implementation. Ultimately, I'd love to have a tool set that's as comfortable for a composer as a MIDI sequencer. That way you can write some music and audition what It'll sound like in the game right away.

What I'm *not* trying to do is replace FMOD Designer for those who have the means to use it on their projects. That's a much more mature tool set, and I would recommend using it if you're looking for something very high-performance. Nothing we can do at runtime on the scripting layer of Unity will compete, but I can try to make it as unobtrusive as possible. At the end of the day, it will come down to the team deciding if the system does what they need it to do and "making room" for it, just like they would with a large simulation, high-res textures, etc.

 

Hi,

I've got a working FMOD plugin (iOS only, but could be extended). Still looking to sell it / form a partnership. Supports FMOD Designer produced fev and fsb files.

More info here : http://forum.unity3d.com/threads/124373-Working-FMOD-plugin-for-iOS-who-is-interested

Cheers,

Gregzo

 

I've gone still another route.

I have a Maestro object that is solely responsible for audio control on the audio source attached to the game camera. I have a method that switches between two charts, at the same point where the previous chart left off. In and of itself, this is not impressive at all.

Where it gets interesting is how I've designed my music. In Thora, the project I'm using this in, there's atmospheric ambience that plays while the player is exploring a map. When the player gets close to an enemy, a transition occurs to that theme's aggressive mix! Of course, if the player gets away, the music returns to the atmospheric ambience again. (Much like Final Fantasy XIII-2, which is what gave me the idea...)

I would be interested in finding ways to combine all that into a single track, though...it is somewhat awkward to always be swapping between audio clips, if I could instead define start/end points in a chart to use in certain situations...

 

I worked out a simple solution: for dynamic music in Unity.... Its on the Asset Store. Still i m not a scripter so it might not be the Ideal way to do it, but it works.

 

What is your product lukasgmusic? Curious.