Adaptive/Interactive/Random soundtrack generation - the future of background music for video games

Hi Unity Devs!
At the moment, I'm working on an adaptive/interactive/random soundtrack generation script.
In the video below, the music is meant for an RPG style video game that takes place in the medieval ages with rather long "exploration" and "battle" sequences, not a fast paced action game.

The way it works is pretty simple. It detects the closest enemy or trigger from an Array and based on the distance, it either plays the soft theme or the battle theme. The switch between the two always happens at the end of a musical "phrase" or at the end of a melody to put it simply so that you don't have the fade in/ fade out effect but rather so it sounds like a real marching band that stops and then plays something else.

Every time the Battle sequence finishes, the script randomly reassigns different percussion loops or melodies so that next time it plays the Battle sequence, the samples are different.
For the moment, you only have one Battle soundtrack "structure" and one "Peace" soundtrack "structure", but I will add several, which will greatly enhance the feel that the soundtrack changes every time you enter a Battle.

UPDATE: I have released a web player for a new script that enables you to activate different "layers" of the interactive battle music!

(you'll have to wait a bit before hearing any sound, don't forget the samples need to load into memory)

For now, it simply works using keystrokes. Pressing a key will "activate" the loop, pressing the key right below will "stop" the loop from playing.
The "W" and "S" key determine whether you are in "peace" more or "battle" mode and will influence which drum loop / tambourine loop that is playing. This will give you more "granular" control over what sample is playing at what time during the gameplay (instead of using the "automatic" script that is featured in the video below)

Do you think this is useful? I really believe that this could be the future of video game soundtracks... It would finally end the "traditional" use of loops that fade in/fade out into each other, and you always end up hearing the exact same soundtrack...

You can also check out my Authentic Medieval Ages Audio Pack which includes the very first randomized soundtrack generation script I've created but without the "interactive" part.

Anyways, hope you like it!
Sincerely,
Marma

 

Any thoughts?

 

I think you forgot to include the option I've been looking for, which is layer based looping and fading..
What I'd like to see is a system where you can trigger loops, such as kick, snare, hihat, bass, piano, pads.. etc etc (or fade them out), and they are synced, so you don't have to worry about audio offset.
Off course this would require a lot more processing power since you would have to set up busses with compressors etc for each layer.

 

Thanks for your comment! Yes, I've looked that up too... There is a method in Unity to play a sample at a specific time of a "reference" sample already playing so that there would not be any offset.

http://answers.unity3d.com/questions/790707/how-to-make-two-audio-tracks-play-in-sync.html

This would mean that you would have a "base" layer of audio that would play continuously, and you can "add" other samples to it "as you go".

I've not included this method as a question in the poll because it's just a variation on the "loop layer fade in / fade out" system.
As a composer, what I was most worried about, is that the soundtrack sounds like actual musicians playing a real piece of music, rather than like a DJ that fades between tracks. That's especially important for orchestral music or medieval music, not so much for electronic music as fade in / fade out is part of the "standard" composition techniques for electronic music.

I could actually easily implement a system like the one you describe, but then you, as the developer, will end up being the "composer", and will be responsible for the "realism" of the soundtrack you decide to generate when you make certain samples come in or not.

If you're interested, I can try to write a script that uses the samples in the demo above and let you try it to see if you like it! ;-)

 

Haven't gotten to see the video with sound yet, need to get home from work first,
but actually, I would be willing to purchase a system that would work akin to how the abelton clip launcher screen works.
And with automatic fades timing it would be even better.

The difference of orchestral vs DJ, do you mean playing a random loop?
Wouldn't it be the same thing, I mean introducing high strings and perc at increased tension.

Also, would it include a some soft of tag, to specify be able to only select loops at a specific key.

 

Hi!
Well, if you think about it, when you listen to live music like a medieval marching band during a medieval battle or a bunch of medieval musicians playing in the street, the structure of a song and the order and timing in which instruments come in or stop, it is all based on measures so you cannot "start" or "finish" a melody at any moment, you have to let it run it's course.

That is why I have opted for a "pre composed" song structure that plays samples at a given time, with at the end of each musical sequence, the possibility to transition to another "mood". It's actually waaaaaaay easier to write a script according to what you are describing because all you have to do is to make samples run continuously in sync, and fade in/fade out the samples according to triggers...

The advantage of a "precomposed" system is that it actually sounds like a "real" piece of live music played by musicians. The interesting part is the "random" element: each sample can be replaced by another similar one but with a different melody, different drum loop and so you can have the same song "structure" but with different loops!

For the moment, there is only one song structure in the video, but I will add more, and then, I can also randomly change between the precomposed structures to add an even more "live band" effect and make sure that the player does not "feel" the repetition as much. In the end, I could program as many "precomposed" structures as I want, it just takes time, and the more there are, the more the player would feel that the soundtrack is continuously changing.

You can test the demo scene for my first random soundtrack generation script in a Unity Web Player here: http://goo.gl/CdhKmc
This one generates random soundtracks based on 6 predetermined song structures but they are not "interactive", they simply run from start to finish and then generate a new song once one is finished.

 

Marma, I watched your videos a week or so ago and was blown away. After noticing this post, I watched them again, this time with my son looking over my shoulder. We were both blown away.

I love the random tracks and I especially love the authentic medieval sounds. Our game is trying to be as authentic as possible in a number of "medieval" and tribal cultures so your music is perfect for us. I will definitely be using this for the test area and our big city.

I wonder if you will be making more types of music packs? I would love to see a pack of authentic middle eastern music as well as a tribal one, with drums and sounds you might hear in Africa or among Native American cultures. Any chance you might look at packs like this?

I am amazed by the amount of time you put into this pack, and your videos illustrate your passion for doing this. I am very impressed! Good job!

 

Hi Teila!
Thanks a lot for your message, it means a lot to me! I'm just an amateur composer and would like to compose music for a living some day so your support is veeery important for me...
I decided to do "authentic" medieval music precisely because no one seemed to care whether the music accurately reflected the period or not and thought there was a gap to fill... so if you think I've managed to do just that, I'm very happy indeed!
It took me ove 4 months to actually finish the audio pack available on Unity. It was a loooot of research, reading articles about medieval harmony, melody, instruments etc...
Right now, as you can see from the video above, I thought I would publish an extension for my Medieval Pack because I believe that "battle" music is sort of lacking. I have one "adaptive" piece that works for battles but it is not random so it can get repetitive after a while.

I posted this thread essentially so I could get feedback on how to make the script even better and more usable by Unity developers. I think it will take me another month to complete this script (medieval battle music) and once I'm done, well, I'll just see what I'll compose next!

I've posted a video with different music styles I am somewhat familiar with, check it out and let me know what you think!

Eastern european music shouldn't be too difficult... I actually play the cello in an amateur orchestra and we are playing Bartok's Romanian Dances for our next concert so I know what it sounds like. African music, I would have to research more... There's a lot of work there on percussion and rythms, just like Indian music, and there is also a lot of singing actually so I wonder how I'd manage that!
I'm not too bad at singing, but I sure don't sing like the guy that sings the opening of the Lion King!

In any case, I'll keep this thread alive and feed you with updates on how this battle extension is coming along!

 

Battle extensions! Yay! I missed that in your above post. I think that is a great idea.

Yeah, that singer in the Lion King song is amazing. lol The cultures in our game are not really African, but they are tribal with drums. I suppose a random drum, maybe with a few flutes in there and that deep Australian pipe (forgot the name but it is a delightful name...) would work fabulously for primitive cultures and it could be more generic, appealing to more people.

The other idea might be to make some sort of tool so we could add our own random music. I can find lots of Eastern and Tribal music out there, but it won't have your cool randomization effects. While not absolutely necessary, it sure makes things interesting in-game for the players.

I have done some research into medieval music myself, not as a musician but when writing lore and designing the instruments and music for our game. I think you did a fantastic job.

 

Great music! I enjoyed it, especially the pirate music. You are a very versatile musician.

 

I'm trying to understand exactly how it is supposed to work,
or rather I'm just posting my thoughts on how I think it works, or how I envision it would work.

so I assembled this little graphic to make it easier to talk about..


So, what I think you mean is that each layer has an intro/buildup and then it proceeds to play either red or green loop (provided there is only 2 options), and then repeats randomly either green or red. And when the layer stops it proceeds to an outro/release clip.

Am I understanding it correctly?

 

Yes, exactly!
It depends on the random script however.
In this random generation script, there is no need for intro and outro since the loops themselves can sound like the natural end of a phrase.

In the video in my first post, it works exactly as you pointed out, except that even the intros and outros are randomized!
Also, in your illustration, it says "loop infinite", but in that part, I actually have a predefined structure for the samples to play.
For instance, the drum loop will start on the first measure, then on the 4th measure, the piano will kick in, on the 12th measure, a different piano loop will start and I'll add another drum layer etc... so it sounds like an actual piece.

Here an illustration of how that would look like..

 

In that case, it would be very useful!

Are you planning on making a public script for this, or asset store?
And in that case, do you need any beta testers maybe
Also, have you thought about how to arrange a user interface so its understandable?

 

Ah, yes, my biggest wish!
I would love to make a dedicated script for this!
But it would be quite complex to code. For the moment, the scripts I have written are specifically tailored to the sample pack I composed. The reason for this is that unless you compose all samples to be universally combined in really ANY order, the precomposed song structures might work for one set of samples and not another.

Also, I would also love to make an interface, but I'm really just an amateur self-taught coder. I like it, but I have nooo clue as to how to design an interface. I know how I would go about it, but don't have the knowledge to see it through.

For the moment, what I'm trying to do, is to provide a "drag and drop" script that is as simple as possible and works as soon as you attach it to a game object. But I would really need to work with an experienced coder to create a universally usable script. It would also require the writing of detailed documentation for any music composer so that they understand how it works in order to compose samples in the right way...

But since no one is currently doing randomized AND adaptive/interactive music, I don't see it as very likely...

 

I was searching for something something like this a while ago,
when I was researching an idea I had.
The idea was an iPad app for preschoolers,
where you had a stage and could drag and drop animals into the stage and drag instruments on top of them,
and they would play music.
So .. if you gave the drumset to the rhino it would sound different than if you gave it to the butterfly...

 

Easily done... but again, in a tailored script...
I'll try to code a script with a new system this week for you to test and send you a private message if you don't mind...

 

That be great!
I think music in games, in general, need some TLC and not be treated like a static cinema experience.

 

Hi guys!
As promised, here is a web player for a script that enables you to activate different "layers" of the interactive battle music!
Try it and let me know what you think!
(you'll have to wait a bit before hearing any sound, don't forget the samples need to load into memory)

For now, it simply works using keystrokes. Pressing a key will "activate" the loop, pressing the key right below will "stop" the loop from playing.
The "W" and "S" key determine whether you are in "peace" more or "battle" mode and will influence which drum loop / tambourine loop that is playing.

PS: Kemonono, I tried to send you a private message but somehow it didn't work. It said "you are not authorised to start a conversation with Kemonono"..

 

Sounds fabulous!! I like it a lot. I have your free version in my game right now and am enjoying the music as I work. I will definitely be buying the full version.

 

Hi Teila!
Glad you can make a good use of it!
I'll also try to get this new Asset out soon since there aren't that many soundtracks suitable for battle in my original Medieval Pack.
Thanks to your feedback and Kemonono's I'll include the script in the demo above so that as developers, you will have the choice to use the "automatic" script or have much more control over what is playing when during the gameplay...
Thanks a lot!

 

Marma,

I purchased the full pack. There is one thing I would like to do but it doesn't seem to work. I would like to put the object with the random script on it near my inn and control the distance at which the sound can be heard. I noticed there is a max distance showing up when the game is in play mode but it is not available in edit mode. How can I set the distance? Is this something that needs to be done via script and if so, which script will allow me to change distance? Thanks.

 

Hi Teila!
Thanks a million! Your support means a lot to me!
And I stand by what I promised: "the future updates will be based on users' feedback!" (if of course it is within my powers... )
In the version you have, I did not include a "ready" code to change the volume as a function of distance.
BUT, you asked, and I looked into it, and here is the demo of the updated script!

I'll upload it as soon as possible as an update to the Audio Pack and the Music Pack but if you need it fast, send me a private message with your email and I'll email you the script.
In the meantime, if you could review my asset that would reaaaaally help me out a lot!

 

I can wait. Busy with shaders at the moment. I will be happy to write a review. Let me play with it for a day or so. I haven't really done anything with it so far other than put it in my game and listening to the music while I tested shaders and weather systems. It was very lovely!

Thank you for the distance update. That will help a lot.

 

OK! I'll let you know when the asset has been updated (it always takes a week or more) but I'll first tweak the script a bit to make sure it works 100%...

 

Hi everyone!
So here is the definite demo for the additional script that will change the volume of the random soundtrack generation depending on the distance of the player to a game object.

You might ask yourself why a "new" script is needed, and that's because there are a looot of different audio sources in the random soundtrack generation and therefore, you cannot use Unity 5's built in Audio Mixer or other built in functions.

The new script was uploaded today and submitted for review, hopefully it will be available sometime next week!

 

The Audio Pack is updated with the new "volume as a function of distance" random soundtrack generation script!
The update to the Music pack is pending review!

 

Sounds really cool!

How about having a difference between "internal" phrases and "transition" phrases, to make for a less abrupt but still natural (not fade-in/fade-out) transitions between states?

Another feature that could be cool is slowly ramping up the tension, by having elements/phrases that can fit in both settings. So, for example, you could be walking around in "peace" and get near a mountain cave which will replace the drums with "battle" drums and slowly ramp their volume up until you reach the actual battle where the bagpipes transition to the battle theme?

Anyhow - it sounds really cool, good luck!

 

cant listen to it since Im at work, but sounds fantastic. I'd love something similar to what Nidhogg had (which was also procedural)

 

The volume changes are fantastic! I love it. I put a game object in my tavern and now hear the music play when I walk past the tavern. Inside, the music will be louder, although I don't yet have an interior...but that will come later.

Thank you for the changes.

 

@Teila : you're welcome! Glad it works well!

@caladluin : I'll include that feature probably into the script that gives you full control over the loops playing.
I'll simply add a feature to control the volume of each "layer" and then you can be the "composer" for your game and tweak everything as you like...
I'll still include the random/interactive "automatic" script for the developers that don't want to spend time programming the entire soundtrack... Maybe I'll think of something in between fully flexible and fully automatic.

As far as the difference between the internal and transitional phrases, I don't quite understand what you mean... Do you mean specific melodies or drum loops that would be played during transition phases? I have already included transition samples for drums, but for melodies it's a bit harder to imagine how it would work...
When I'll have my new computer however, I'd like to move on to a whole other level with very short samples that can be arranged in any order to create an even more complex random soundtrack! We'll see if I can manage...

@ostrich160 : I played Nidhogg at a friends' and loved the game, but was too busy and stressed killing my opponent to listen to the music... How was the music put together?

 

Marma,

I am having issues with your asset. It seems to spike my Profiler and almost as much as my Camera Render. The debug.log seems to be the problem.



Oh, and this was taken in an area where you could not hear the sound so far enough away.

 

Hi Teila,
Yes, this is "normal"... well, sort of! I forgot to comment out a debug.log that helped me test whether the script was working well...
Just open the script in MonoDevelop and comment out or delete the line with "Debug.Log ("The sound is stopped " + i);"

If you're not too much into scripting, I'll update the Audio Pack again once I've made some other interesting addition and let you know when it's out!

 

I am not into scripting...LOL. I have two guys to do that for me but I would be perfectly happy to wait until you add the interesting stuff. Please do let me know when it is out!

I can easily comment out the debug.log though. That is just about my limit other than deleting stuff.

 

OK, let me know if you manage to comment it out and if it fixes the issue you're having!

 

Im not entirely sure, I just know its dynamic

 

I did comment it out. There were actually two places with debug.log not commented out and I found them. It did fix the problem to some extent. I am still playing around with it to figure out what is causing my performance decrease. Disabling the music asset did help but not sure yet if that is the issue. I will let you know if I figure it out.

 

OK good to know! Keep in mind that my asset does unfortunately require some computing power to work since it loads into memory a number of samples to work and has to play several layers of instruments (audiosources) to function. That is the flip side of having a randomized soundtrack that is not repetitive: it's great for the user to hear but it requires much more CPU and especially memory to run than simply playing a precomposed single soundtrack loop...

But since devices get more and more memory and computing power, it should not cause too many issues...

If you still experience major performance issues, there are several ways this can be fixed:
- I could "add" into the audio pack *.OGG type files instead of WAV (compressed files) which take up less space but require more computing power to be processed and played at the same time.
- You can also see if the option "Load Type" for all the samples in the Resources sub folders is ticked as "stream from disk" or "load into memory". This can also ease the load on memory/CPU although since multiple samples need to play at the same time for the script to work (drums, melody, accompaniement), some devices may have trouble streaming 3 or 4 samples at the same time from disk.

See here for the reference in the Unity Manual: http://docs.unity3d.com/Manual/class-AudioClip.html

 

Hi Marma,

I am still having issues. I changed all the music to Compressed in Memory (some of it was not compressed at all) and the quality to 1 just to see what would happen.

Unfortunately, it didn't change anything. If I start the game with the Random Music checked on in the inspector, my controller is very laggy, with choppy movements. My fps is okay.

But when I stop start the game with the Random Music unchecked in the inspector, movement is very smooth and nice. My fps is not really that much different, which is weird. Usually when I get choppy movement, it is for some obvious reason.

Obviously, I can't use the Random Music with choppy movement. My players won't like it.

I have a PC desktop, with plenty of power so I don't think it is my PC that is the problem.

So...any chance we can get the individual melodies as songs and loops? Or, I think you mentioned above, a way for us to put together the pieces and "compose" melodies for our game from your pieces and then place them into the game as individual songs. We would lose the random, which is very cool, but it simply doesn't work for me at the moment. I am not getting any errors so can't figure out why not but I am going to assume that it is conflicting with something, some unseen problem.

 

Hi teila!
I'll have another look at the script as soon as I can and try to optimize it..
Also, it might be that the preview in unity will use more processing power so can you try exporting your game/scene and run it?

I can of course render a few pieces from the random script in case other developers want to use them in games that require a lot of processing power for graphics..

I'll update the asset soon!

PS: I am planning to add a random script with church organs and since it only plays a single sample at a time, it should work without any problems..

I'll keep you posted!

 

Hey Marma,

I discovered something interesting. The random music works fine without the distance component so I think that might be the issue.

 

OK good to know!
Well, perhaps I will reduce the frequency at which the volume changes, since it updates the volume every time the player moves... More soon! ;-)

 

New update to my Medieval Audio Pack and the Music Pack: random medieval church organ music!

Let me know what you think!

 

Beautiful! Thanks for the update.

I also love your pictures in the video. I am one of those people who cannot resist going into old churches. While visiting Europe, I went into dozens of them...so gorgeous. Your pictures remind me of that experience.

I think this is one of my most trouble free assets! lol Of course, it is music, and not some complicated script for water or lighting.

Edit..that sounded bad! Yours is a very complex music script but it is easier for ME than some of the other stuff I have. Whew!

 

This is a wonderful concept! I'll be interested in seeing what the final product is like .

Will we be able to use the program to set our own conditions for change? Say, instead of the distance between player and enemy being a condition, I wanted the condition to be something like a value output from my game or the relative position in a story?

 

Hi Tiki!
Well, the script works on the basis of an Array of enemy positions and when an enemy is close, the battle script runs.

If you want to set other conditions for the script, it's all up to you really... The script for now changes the value of booleans to be true or false (battle mode or not) based on some parameter... You can easily create a small portion of code inside my script that changes the value of the booleans based on something specific in your game of course!

The trouble is, I can't really "pre program" it for you since it all depends on what parameters you want to feed into the script.
The distance between a player and an enemy is kind of "universal" to most games so that one I could implement myself...
If you have specific requests, I can look into it of course!

 

What if you were to set up an editor window that allows devs to pick from a handful of typical scripting options?

For instance, expose a float, integer, string, and enum with a requirement toggle, to decide which one you want to use.

[ ] Float
[x] Int
[ ] String
[ ] Enum

Then pair them with another enum holding keywords

[ ] Float - Custom-v-
[x] Int - EnemyDistance-v-
[ ] String - Custom-v-
[ ] Enum - Custom-v-

Custom would open up another couple of slots in the row.
[ ] Float - Custom-v- : [script] [value] [goal]
[x] Int - EnemyDistance -v- : [enemy] [player] [goal]
[ ] String - Custom-v- : [script] [value] [goal]
[ ] Enum - Custom-v- : [script] [value] [goal]

Something like this should leave you a lot of room to continue expanding options as people need them. Just a suggestion... I see a lot of potential for this program, I wish I was more musically inclined.

 

Hmmm, that sounds like a good idea, indeed! Thanks!
But I will have to take a few tutorials on how to create a user interface under Unity...
After all, I am a composer and I self-taught myself coding but I can't really say I'm a coder...

 

Learning the Editor interface isn't that complicated, not much different from the legacy GUI system I think you used for the web demo. I would suggest learning to make an "EditorWindow" in particular. I've found it can be useful to serialize information or generate asset files, but perhaps a more seasoned coder could give you more direction there as well.

I wish you luck!

 

OK thanks! I'll look into it!
Perhaps using the standard editor window with public booleans/variables can work as well... I'll figure it out!

By the way, my new script for interactive video game music based on triggers is out! You can download it for free here:
https://www.assetstore.unity3d.com/en/#!/content/23837

Try it out and don't forget to drop a review to let me know what you think!

 

Hi everyone!
Just want you to know I just released the PRO version of my Authentic Medieval Battle Music, which features the very first interactive AND random background music generation script for the Unity game engine!!!
Check it out here:
https://www.assetstore.unity3d.com/en/#!/content/40584