Seamless Loop-based Interactive/Adaptive Music Player/Sequencer - can it be done?

Discussion in 'Scripting' started by Unitony, Mar 24, 2011.

  1. Unitony

    Unitony

    New Member

    Joined:
    Jul 19, 2010
    Messages:
    4
    Hi,

    I want to use interactive music in my game and I'm currently building my own interactive loop-based sequencer in Unity using Javascript (something like http://www.nlnplayer.com/nln-player/index.php?&url=shortburst). The system should use sound files such as wav or ogg (not MOD). Since the music should adapt to the gameplay relatively quickly, I am using short loops (about 1.5 seconds per loop = 1 measure) that I want to play seamlessly in sequences. I am using Unity 3.3 btw.

    The problem I have is that I can't manage to get my loops to play seamlessly. I tried both .wav and .ogg files (I'm aware of loop problems with compressed formats, such as mp3). I've browsed this forum and looked at many examples on how to achieve such seamless playing of musical loops, but I have not yet found a working example. The three main approaches I found (including variations) and which I tried:

    1. on Update, check if the current clip !isPlaying, then change soundfile and play (http://answers.unity3d.com/questions/19925/pick-random-song) -> currently the most simple and best sounding option, but still 'gappy'

    2. play clip, then use Invoke() (or InvokeRepeat or yield etc. - tried them all) in combination with audio.clip.length to try and start a new clip right after the other -> very noticable, uneven gaps

    3. play a clip and loop it, then during play change the soundfile -> changing clip unfortunately stops play and you have to restart it manually again

    I tried various variations but none of these provided an unhearable transition between two different soundfiles. I was surprised that the first option (http://answers.unity3d.com/questions/19925/pick-random-song) worked as well as it did, since I found there was only a noticable gap when the random soundfile was different than the previous one - when the next random[soundfile] was the same as the previous one, there was a seamless transition.

    I also had a look at http://www.psneville.com/?p=874 but since I'm not that familiar with C# and because the code is slightly outdated (when I open the package in Unity 3.3 I get a lot of errors), I have not managed to find out if and if so, how this example really works.

    One more important detail: I am making an iPhone game (so I'd would really like my interactive music to work on iPhone). All of my findings with noticable transitions between two music sound files so far have been on PC, MAC and iPhone platforms though.

    So: is there anybody out there who has managed to create a working musical sequencer/loop-based music system in Unity with seamless transitions between music sound files? Or can anyone point me in the right direction? Or tell me that it cannot yet be done in Unity?

    Thanks!

    Tony
     
    Last edited: Mar 24, 2011
  2. Antitheory

    Antitheory

    New Member

    Joined:
    Nov 14, 2010
    Messages:
    549
    How are you playing you audio in the game?

    If you have an AudioSource component attached to a GameObject you simply check the "loop" box and it seamlessly loops.
     
  3. Antitheory

    Antitheory

    New Member

    Joined:
    Nov 14, 2010
    Messages:
    549
    Wait... I see that you are looking for a seamless transition between two different sound-files...

    For that I would just make sure to have every sound-file playing in a separate AudioSource... Well you wouldn't exactly need 100 sources if you have 100 different wavs, you would need as many sources as can be played at once.

    Essentially you should only switch the AudioClip on the source when it's not playing or it's volume is at zero, otherwise you will likely get the "gap" problem.

    Also you should fade in/out clips when they start/end over a very short period. If you use a tracker like Sony Acid you'll notice that it automatically does this for you.

    Lastly make sure that indeed your loops are seamless and that the DC offsets are normalized.
     
  4. Unitony

    Unitony

    New Member

    Joined:
    Jul 19, 2010
    Messages:
    4
    Hi,

    Thanks for your quick reply!

    Yes, I want to have a seamless transition between two different sound files. I have considered using your suggested 'layered' approach (if I understand your comment correctly). Did you mean it like sync-starting multiple (let's say 8 or so) sound files (using multiple audio sources) at exactly the same time with the volume on zero, with only one sound file/audio source audible by having it's volume on 1? And then when I want to change the sequence I'd do a really quick crossfade?

    I haven't tried this in Unity yet (I will, just to see how well it works). Unfortunately, the downside is that it means that ALL my music sound files must be exactly the same length (there goes my composition), and I have an x amount of soundfiles constantly playing (memory?) during gameplay. Even if it would sync perfectly (which I doubt, have you tried it yourself?), I'd really like to avoid such a possible kludge fix unless I really really have to.

    Isn't there any other way?
     
  5. Antitheory

    Antitheory

    New Member

    Joined:
    Nov 14, 2010
    Messages:
    549
    You really only need two sources if you only have a series of wav files that will be played in sequence. Basically you just cant switch the audio-clip on a single source and expect it to transition without a gap, your transitions need to happen between sources, not on the same source. dig?

    Meaning if you just have clips A-B-C-D and you want to play them in order you do this

    Source 1:A--------A...........C----------C
    Source 2:............B---------B..............D---------D

    Of course you need more sources if you want b and c to play at the same time

    Source1:A--------A .............D-------------D
    Source2:.............B-----------B
    Source3:.............C-----------C

    This should be seamless if timed correctly... I don't know how easy that would be though. My guess is that Unity's sound capabilities are not really geared toward precision timed audio stuff like a dedicated music-making program is. But at least try it out.
     
    Last edited: Mar 24, 2011
  6. Jesse Anders

    Jesse Anders

    New Member

    Joined:
    Apr 5, 2008
    Messages:
    2,857
    My guess is that Unity isn't particularly well-suited for this. I say 'guess' because I don't know that for sure, and I'd be happy to be proven wrong.

    I've implemented seamless track queuing in another context (outside of Unity), and in that implementation, audio data was managed at a low level, and when one track ended, a new track was ready to go on the very next sample, which I think is what's needed for truly seamless effects (especially with rhythmic music).

    I'm not sure how this would be accomplished with Unity outside of plug-ins (I don't know for a fact that it could be accomplished with plug-ins, but it seems likely that it could). All of the timing methods available (basically, the ones you already mentioned) are somewhat variable and have a rather course granularity, and therefore probably aren't of much use for something like this.

    Again though, I'd be happy to be proven wrong. (Also, I don't always peruse all the new features when a new version of Unity comes out, so there may be new functionality available that I'm not aware of.)
     
  7. aoakenfo

    aoakenfo

    New Member

    Joined:
    Jul 14, 2009
    Messages:
    35
    I doubt you'll ever get sample accurate timing from C#.

    The two options I see are:

    1. Use Unity as a frontend and send OSC messages to an external application like Max/MSP.
    2. Write a plugin, wrapping up CoreAudio/Midi (if you're on a Mac) and then make calls to it from Unity.
     
  8. se7en

    se7en

    New Member

    Joined:
    Dec 3, 2008
    Messages:
    226
    I made a completely interactive audio controller in my first iPhone game called Circuit_Strike.One - Geometry Wars/Asteroids meets Rez style. It was done entirely with clips from the same track so I created a timer and when a new sample was ready to be played it would wait for the next cue point. Pretty simple actually. It does not work with compressed audio because decompression causes lag. I just created short loops for the main beats and breaks then fired off other sounds long and short in layers depending on what the player was doing.

    It was a lot of fun and is definitely doable on an iPhone since this was done pre 3GS.

    shay
     
  9. Unitony

    Unitony

    New Member

    Joined:
    Jul 19, 2010
    Messages:
    4
    Hi,

    Can you share some insights on how you made your timer? Or if you could perhaps share some of your code? I tried a timer approach with Invoke() but in my case it didn't work perfectly, even with samples of exactly the same length. Did you perhaps use a co-routine or... ?

    Thanks!

    Tony
     
    Last edited: Mar 28, 2011
  10. Unitony

    Unitony

    New Member

    Joined:
    Jul 19, 2010
    Messages:
    4
    Hi again,

    I made an example of the solution suggested by Antitheory, using more than one audio clip to play the music clips. Here's the code:

    Code (csharp):
    1.  
    2.  
    3. var Clip: int; // keeps count of the current Clip (= sound file)
    4. var Block: int; // keeps count of the current Block (= sequence of sound files)
    5. var Intro: AudioClip[]; // this is a block, just a sequence of clips - make sure it contains 4 clips (by adding sound files in the Inspector)
    6. var Intensity1: AudioClip[]; // this is another block
    7. var Intensity2: AudioClip[]; // and another block
    8. var Music: Array = new Array(); // 'the composition' in blocks
    9.  
    10.  
    11. function Start() {
    12.     // in this particular script, make sure every Block-array (Intro, Intensity1/2) has exactly 4 sound clips
    13.     Music = [Intro, Intensity1, Intensity2];
    14.    
    15.     Clip = -1;  // init first Clip (off-by-1)
    16.     Block = 0;  // init first Block
    17.    
    18.     // start music
    19.     NextClip();
    20. }
    21.  
    22.  
    23. function NextClip() {
    24.     // update current clip with the next clip
    25.     Clip = Clip + 1;
    26.    
    27.     // play current clip
    28.     PlayClip(Music[Block][Clip],transform, 1.0f, 1.0f);
    29.    
    30.     // call NextClip-function on the exact moment when the current clip ends
    31.     Invoke("NextClip", Music[Block][Clip].length);
    32.    
    33.     //The following is the 'composition' - this is why you need exactly 4 clips and 3 blocks
    34.     // for a more interesting composition, or have the music adapt to the gameplay,
    35.     // add some kind of Clip/Block-picking function here:
    36.     if (Clip == 3)
    37.     {
    38.         Clip = -1;
    39.         Block = Block + 1;
    40.             if (Block == 3)
    41.             {
    42.                 Block = 0;
    43.             }
    44.     }
    45. }
    46.  
    47.  
    48.  
    49. function PlayClip(sound : AudioClip, tf : Transform, randomPitch : float, randomVolume : float) {
    50.     // create a temporary audio source and use it to play the clip, then destroy it.
    51.     var soundObject = new GameObject("Temporary Music Clip");
    52.     var soundSource: AudioSource = soundObject.AddComponent(AudioSource);
    53.     soundSource.clip = sound;
    54.     soundSource.pitch = randomPitch;
    55.     soundSource.volume = randomVolume;
    56.     soundObject.transform.parent = tf;
    57.     soundObject.transform.position = tf.position;
    58.     soundSource.priority = 1;
    59.     soundSource.Play();
    60.     Destroy(soundObject, sound.length + 0.1);
    61. }
    62.  
    63.  
    Drag this unto your camera (or some other object) and then add music clips/loops to the Block-arrays (in the Inspector - probably make sure your clips are set to 2D if you want background music). for this example to work make sure you fill all 3 blocks with 4 clips each.

    What it does is that it creates a temporary audio source through which it then plays a clip (a piece of music). Then Invoke() ) is used to create another temporary audio source on the exact moment the clip ends playing, to play a new clip, etc. etc.

    This is just a small test but I've found that it works relatively well on my computer - using both WAV (uncompressed) and OGG (compressed) sound files. I still have to test it on iPhone though. I sense that the timing is sometimes very slightly off, but only a tiny little bit - I do want to toy around with it some more to see if I can get it to work better.

    Suggestions, anyone? For instance using a co-routine instead of invoke or... ?

    Unitony
     
    Last edited: Mar 29, 2011
  11. tomnullpointer

    tomnullpointer

    New Member

    Joined:
    Sep 20, 2010
    Messages:
    123
    I posted this a while ago,

    I think it might help you
    http://forum.unity3d.com/threads/78003-Audio-stepsequencer-and-they-said-it-couldnt-be-done!
    I did quite a bit of research on this - using custom fmob libraries dll plugins, OSC to pd and back, all sorts etc, None of them held a decent tempo (you can tell pretty soon if a repeat is out of time)

    the link above shows the best solution i found). Its been pretty useful to a few other people too. So I thought Id relink it here.

    www.nullpointer.co.uk
     
    Last edited: Mar 29, 2011
  12. aoakenfo

    aoakenfo

    New Member

    Joined:
    Jul 14, 2009
    Messages:
    35
  13. thekingofclubs

    thekingofclubs

    Member

    Joined:
    Jul 15, 2014
    Messages:
    4
    I am trying to do a similar thing with my game. I want the music intensity to go up as the challenge grows. I had the slight "gap" when switching between tracks too. I tried setting the "Load type" of the tracks to Stream from disc because I thought the gap occurred because Unity takes time to load the track in the memory or something. But that did not help. What helped is using FixedUpdate(). This is what Unity says about FixedUpdate()

    FixedUpdate is often called more frequently than Update. It can be called multiple times per frame, if the frame rate is low and it may not be called between frames at all if the frame rate is high.

    So basically the gap occurs because the new track loads when the new frame begins because Update() is called only once per frame. FixedUpdate() works even during the loaded frame so the gap disappears. I am still slightly unsatisfied with the result but this has been the best outcome of all my experiments.
     
  14. ponx

    ponx

    Member

    Joined:
    Sep 13, 2010
    Messages:
    8