Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

A list of audio bugs (iOS mostly)

Discussion in 'Editor & General Support' started by gregzo, Aug 17, 2012.

  1. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi to all!

    To save others some time, here's a list of bugs I've submitted that were reproduced and aknowledged by Unity. Please, feel free to comment/add!

    1) When deving for iOS platforms, changing Edit -> ProjectSettings -> Audio -> DSPBufferSize will change the DSPBufferSize on the device when building, not in the editor, where it must be manualy set ( AudioSettings.SetDSPBufferSize). Problem : using AudioSettings.SetDSPBufferSize will simply kill all audio on iOS platforms for the scene it was called in. Work around : dummy scene just to set the buffer size.

    Bug status : Submitted, aknowledged and reproduced by Unity.

    2)Play(offset) inaccuracy : when the DSPBufferSize is set to best latency, Play(offset) will sometimes be 470-471 samples late on iOS devices.

    Bug status : Submitted, aknowledged and reproduced by Unity.

    3)AudioClip.SetData and AudioClip.GetData do not gracefuly handle exceptions and WILL make Unity hard crash if the array you pass is longer than the audio clip. [EDIT : Has been fixed in 3.5.6]

    Bug status : Submitted, acknowledged and reproduced. Seems too big not to be taken care of, but as the updates roll, it still isn't.

    I will try to post more, regarding quirks rather than bugs, when I have time.

    Cheers,

    Gregzo
     
    Last edited: Sep 29, 2012
  2. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    A quirk rather than a bug(iOS):

    When running through XCode, the appearance of the iOS keyboard will cause audio to stutter badly. Doesn't happen when a build is launched from the device, though...

    Bug status: Submitted, aknowledged and reproduced by Unity.
     
  3. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Another quirk, which might explain some bugs, regarding timeSamples. This post might help new users who are trying to get a better understanding of audio in Unity, so I will first detail a few findings regarding the under documented use of AudioSource.timeSamples.

    1)AudioSource.timeSamples returns the index of the first sample of the source's clip in the audio buffer. Hence, it will always return the index of a sample to be played.

    2) Because audio runs in a seperate thread, the following can be done to measure audio time more precisely (useful if you need milisecond accuracy):

    Code (csharp):
    1. var tempTimeSamples : int = audio.timeSamples;
    2. while(audio.timeSamples==tempTimeSamples)
    3. {
    4.    //do nothing at all, wait for the audio buffer to be renewed
    5. }
    6. //Now do stuff such as measuring time (Stopwatch advised)
    Edit: The busy loop above is discouraged on janm's reply at the bottom of this thread

    3) The quirk : since timeSamples returns the index of the first sample in the buffer, the following script should always return DSPBuffserSize's bufferSize :
    Code (csharp):
    1.  
    2. var tempTimeSamples : int = audio.timeSamples;
    3. while(audio.timeSamples==tempTimeSamples)
    4. {
    5.    //do nothing
    6. }
    7. if(audio.timeSamples>bufferSize) //quick hack to avoid checking just after the clip has looped
    8. {
    9.      Debug.Log(audio.timeSamples-tempTimeSamples);
    10. }
    Which it does in most circumstances, but not when DSPBufferSize is set to best latency on iOS, where instead of printing 256, the script will print 470-471. And, unsurprisingly, when the Play(offset) bug mentioned in my first post happens, audio is late by 470-471 samples. Hmmmm...

    Quirk status : Described in my submission of the above detailed Play(offset) bug.
     
    Last edited: Oct 2, 2012
  4. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    A minor quirk, but rather annoying for non pro-users:

    AudioReverbZone applies it0s reverb filters AFTER AudioListener.OnAudioFilterRead()

    Consequence : Saving AudioListener's audio data to wav will result in files which sound as dry as the Sahara...

    Quirk status : discussed on the forum with previous audio programmer Soren, who advised to fake reverb zones with filters. Filters are only available in Unity Pro.

    Related quirk : playing clips with a negative pitch (ie reverse) and an offset will bypass AudioReverbZones. (Yes, it's a pretty rare use case, but it was mine...)
     
  5. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    A small quirk I'd forgotten about but which may be the cause of quite a few headaches :
    Debg.Log(AudioSettings.outputSampleRate); will print 24000 instead of 44100 on iOS devices. Do not believe, or attempt to set it manualy...

    Bug status : submitted.

    Update : got an answer, and submitted a follow up problem. Here's the exchange of mails so far:


    From: Unity Account <accounts@unity3d.com>

    Hi,

    I've just got a response from developer. Quoting it:

    "This is actually correct behavior -- FMOD defaults to using a sample rate of 24000 even though the iPad hardware supports or even runs at a higher sampling rate. AudioSettings.outputSampleRate is a read/write property so it is possible to change the sample rate by explicitly setting it to something else than this default, but currently this is buggy, and I am planning to remove this functionality and change it so that you instead set the rate via project settings (since this avoids the need to reload sounds due to the FMOD reset that is caused by the sample rate change). However, that is a different issue (feel free to open a bug about that) and as regards this issue, I have to close it as "by design" because 24000 is correct in this case."


    If you have any further questions, feel free to ask us.

    Best regards,
    Zbignev
    QA Team

    ////////////////////////

    Hi,

    Thanks for the reply.

    The bug, then, lies elsewhere : Play(offset) calculates the offset time at 44.1 khz. So audio.Play(24000); on a AudioClip of 24 khz sample rate will not offset by 1 second, but by 24000/44100 second.

    Attached is a script which demonstrates this (when running on device, not in the editor).

    This is one of the reasons I thought AudioSettings.outputSampleRate was incorrect on iOS devices.


    Code (csharp):
    1.  
    2. #pragma strict
    3.  
    4. var clip : AudioClip;
    5.  
    6. var sources : AudioSource[];
    7.  
    8. private var sampleRate : int;
    9.  
    10. private var gain : float = .5;
    11.  
    12. function Start()
    13. {
    14.     sampleRate = AudioSettings.outputSampleRate;
    15.     print(sampleRate); // prints 24000 on the iPad
    16.    
    17.     sources = new AudioSource[2];
    18.     for(source in sources)
    19.     {
    20.         source = gameObject.AddComponent(AudioSource);
    21.         source.loop=true;
    22.     }
    23.    
    24.     clip = AudioClip.Create("newClip",sampleRate,1,sampleRate,true,false); //creating a one second clip whatever the sample rate
    25.    
    26.     var data : float[] = new float[clip.samples];
    27.    
    28.     var time : float;
    29.    
    30.     for(var i:int = 0; i<1024;i++) //filling data with a short sine wave (A440)
    31.     {
    32.         time = i;
    33.         time = time/sampleRate;
    34.         data[i]=gain*Mathf.Sin(440*time*Mathf.PI*2);
    35.     }
    36.    
    37.     clip.SetData(data,0);
    38.    
    39.     sources[0].clip = clip;
    40.     sources[1].clip = clip;
    41.     yield;
    42.     sources[0].Play();
    43.     sources[1].Play(clip.samples/2);//The bug : should play perfectly alternated sources (i.e. offset of .5s), whatever the sample rate. Instead, it seems the offset time is still calculated on a 44100 sample rate.
    44. }
    Regards

    G

    Edit - my bad for not reading the docs properly (or where they updated since I first read about Play(offset)?), as janm mentions in his reply at the end of this thread, they do mention the offset is always calculated at 44.1khz.
     
    Last edited: Oct 2, 2012
  6. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Just got a mail from Unity stating that bug no3 will be fixed in the next release.

    Bug detail : AudioClip.SetData and AudioClip.GetData do not gracefuly handle exceptions and WILL make Unity hard crash if the array you pass is longer than the audio clip.

    Buggy since 3.5 beta

    Already fixed in next release (4.0? or is there another 3.5.x down the road?)
     
  7. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,303
    There will be another point 3.5 release almost for sure...

    btw I recently too submitted a bug of this category..
    the FFT data (from GetSpectrumData) in the editor are somewhat different to those obtained when running on the iOS device - it seems they are bit of shifted in the buffer there.. )
    it's reproduced and confirmed as a bug by Unity.
     
  8. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,303
    another audio FFT kind of bug ( not yet submitted )
    the last value in returned FFT buffer is somewhat off - mostly it is too big compared to the rest of the buffer - say previous couple of hundreds or so;
    therefore if you try to find say pitch of the buffer you almost always end up at the end of it and hence always find the highest frequency, which is bollocks of course
    as a workaround I just ignore last value in returned spectrum data for now..
     
  9. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    @r618

    Thanks for your contribution to this thread! Do post example scripts if you have time, as well as any other quirks encountered. The goal here is to summarize all audio issues in order to save future users some pretty frustrating headaches.
     
    Last edited: Oct 2, 2012
  10. Aurore

    Aurore

    Director of Real-Time Learning

    Joined:
    Aug 1, 2012
    Posts:
    3,106
    Thanks for listing these bugs, now we can get less dupes!
     
  11. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
  12. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    What is FMOD? I was trying to figure out what is the default sampling rate on iPad / iPhones. I heard it was 44.1k, not 24k....
     
  13. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi jerotas,

    FMOD is the audio engine unity uses. And it is FMOD which defaults at 24000, not iOS devices, as you can read in post no5 of this thread!

    Please open a new thread if you have questions, to keep this one focused on bugs.

    Cheers,

    Gregzo
     
  14. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Another quirk rather than a bug, which I wouldn't mention if it was more clearly explained in the docs :

    OnAudioFilterRead(data:float[],channels:int), when placed on an AudioSource, returns the source's clip's data from the listener's POV. Consequence : stereo, distance adjusted but pre-reverbZone data on 3d mono clips.

    Also, it seems the float[] is recycled, so grabbing a reference to it on multiple AudioSources doesn't work, as I discovered trying to draw the wav form of simultaneously playing sources :
    Code (csharp):
    1.  
    2. //This script doesn't work if there are multiple instances of it :
    3. //refData will not reliably be the source's clip's data.
    4.  
    5. var refData : float[];
    6.  
    7. function OnAudioFilterRead(data:float[],channels:int)
    8. {
    9.      refData = data;
    10. }
    11.  
    12. function Update()
    13. {
    14.      DrawAudioData(refData);
    15. }
    The solution is to copy the data instead of grabbing a reference :
    Code (csharp):
    1.  
    2. var copiedData : float[];
    3.  
    4. function Awake()
    5. {
    6.      copiedData = new float[bufferSize*channels]; //allocate the array
    7. }
    8.  
    9. function OnAudioFilterRead(data:float[],channels:int)
    10. {
    11.      System.Array.Copy(data,copiedData,data.length);
    12. }
    13.  
    14. function Update()
    15. {
    16.      DrawAudioData(copiedData);
    17. }
    Hoping to save you a few headaches,

    Gregzo
     
  15. janm_unity3d

    janm_unity3d

    Unity Technologies

    Joined:
    Jun 12, 2012
    Posts:
    36
    Hi gregzo,

    it is true that we haven't been very active on the forums and the reason is that we have been busy getting familiar with the code so we don't break existing behavior while fixing bugs (hopefully!) For that reason we are not yet running 100% speed, but getting there, and the 4.0 release has for us been all about looking into existing bugs and fixing them. We also have been working on new features, but nothing that is ready to be announced yet. However, our aim is to add more high-level functionality to Unity that makes it easier to produce reusable assets and ease processes in the game that currently require massive amounts of scripting.

    The most effective way to reach us is to actually use the bugtracking system to report bugs. This way it will be reviewed by our testers, and if they can reproduce it and verify that this a valid use case of a given feature (I'm not talking about your reports here, but it does happen that some features are misunderstood or used in a wrong context) the severity level of the bug will rapidly be raised and will reach us immediately. I know this all sounds very formal, but it is better to have the bugs split up into separate cases that we can track and link (also for future reference) than having them on a thread in a forum.

    To address the bugs and quirks you mentioned (the numbers indicate the number of your post):

    1) Regarding Play(offset) -- the function unfortunately uses a reference rate of 44100 Hz regardless of what the sample rate of the system or what the sample rate of the clip is. I have a change that will most likely appear in 4.1 which still keeps Play(offset) as it is for backward compatibility reasons, but will notify you that the function is deprecated and that PlayDelayed(t) should be used instead, where t is the delay from now specified in seconds. This will not solve the problems with the system-dependent inaccuracy, but it makes much more sense to specify the time in seconds than in samples of some given reference. To solve the problem with inaccuracy we have another solution that makes much more sense for music applications. Let me just describe what the basic problem is: All the timing code is running on the main thread which controls the rendering of the graphics at a more or less fixed frame rate. All FMODs mixing however runs on a different thread, and it is only inbetween the frame updates that FMOD events get executed. This thread runs at a rate of the sample rate divided by the DSP block size, so anything in the range of 48000 / 1024 = 46 blocks processed per second. Long story short: It is not a practical solution to do the time synchronization using the render threads' clock. Instead, if you imagine there is a global time line counting every sample that is output, and you could request AudioSources to play their sounds in the future starting at a given point on this time line, you could get perfect synching, and because the play requests can be placed well ahead in time, FMOD will have plenty of time to open and prebuffer the sound without causing any locks that cause performance hits or other hickups in the system. This is exactly the feature that we have in the pipeline, but since it required some larger changes under the hood, expect that it will first be available in 4.1. The AudioSettings has a new dspTime property that gives you the time in seconds. Then, in the update you simply keep a look at the time and see if it's about time to schedule the next sound. The example below is taken from the documentation of the new feature and shows how to do stitching of loops that run at 140 BPM. As you can see, the time-keeping still happens on the render thread in the Update function, but since we are looking far ahead of time, we are not bound to requiring a high sample rate to run precisely anymore, since the placement happens on the dspTime time line. We got the impression at this year's Unite that there is a big demand for a feature like this for music applications, but the same system can be used for other stuff that needs sample-exact timing such as automatic weapons.

    Code (csharp):
    1.  
    2.     public var bpm = 140.0;
    3.     public var numBeatsPerSegment = 16;
    4.     public var clips = new AudioClip[2];
    5.  
    6.     private var nextEventTime:double;
    7.     private var flip = 0;
    8.     private var audioSources = new AudioSource[2];
    9.     private var running = false;
    10.  
    11.     function Start()
    12.     {
    13.         for(var i = 0; i < 2; i++)
    14.         {
    15.             var child:GameObject = new GameObject("Player");
    16.             child.transform.parent = gameObject.transform;
    17.             audioSources[i] = child.AddComponent("AudioSource");
    18.         }
    19.         nextEventTime = AudioSettings.dspTime + 2.0;
    20.         running = true;
    21.  
    22.     }
    23.     function Update()
    24.     {
    25.         if(!running)
    26.             return;
    27.         var time = AudioSettings.dspTime;
    28.         if(time + 1.0 > nextEventTime)
    29.         {
    30.             // We are now approx. 1 second before the time at which the sound should play, so we will schedule it now in order for the system to have enough time
    31.             // to prepare the playback at the specified time. This may involve opening buffering a streamed file and should therefore take any worst-case delay into account.
    32.             audioSources[flip].clip = clips[flip];
    33.             audioSources[flip].PlayScheduled(nextEventTime);
    34.             Debug.Log("Scheduled source " + flip + " to start at time " + nextEventTime);
    35.  
    36.             // Place the next event 16 beats from here at a rate of 140 beats per minute
    37.             // If the clip does not have fixed timing the code below may be replaced by: nextEventTime += clip.length
    38.             // but probably needs to be corrected
    39.             nextEventTime += (60.0 / bpm) * numBeatsPerSegment;
    40.  
    41.             // Flip between two audio sources so that the loading process of one does not interfere with the one that's playing out
    42.             flip = 1 - flip;
    43.         }
    44.     }
    45.  
    3) Yes, the issue here is related to the same delay issue you mentioned in 1) and comes down to the fact that we are working within a multithreaded environment. It is obviously more prevalent when DSP buffer size is reduced as the mixer is having to thrash more and could potentially do multiple frame mixes between last satisfying the expression in the while and the Debug.Log.
    As stated above, this issue can only fundamentally be solved by scheduling playback relative to some global clock value, which is hopefully part of some upcoming audio hotness.
    We would also advise you not to use busy-loops like in the code examples, because they prevent FMOD from updating its state (it needs frame update to do so). While this may have worked in your particular case, we cannot guarantee that this will stay like that -- in any case the scheduled play functionality will provide robust solutions for these cases.

    4) This is a good point, and something we will consider for the future, if you were able to submit a bug report for this, we could better track this behaviour change request.

    5) See the comment for post 1: The 44100 in the Play(offset) function is (as the documentation states) just a reference rate, regardless of what the system or AudioClip sample rate is. I don't like this, but it would break backward compatibility to change it, so it will be deprecated and replaced by PlayDelayed(t) where t is a value in seconds. In the future we will change everything to use time values specified in seconds so that it's not necessary to do the conversions all the time.

    7) @r618: This is due to the fact that the iOS device runs af 24 kHz, while the editor runs in 44.1 kHz or 48 kHz. So because of all the practical problems that happen when working with windowed FFT data, you will see spectral leakage, the effect of linear interpolation of the data at the different rate etc. causing all kinds of differences between the cases even though the overall shape of the FFT looks the same.

    8) @r618: We haven't gotten around to experiment much with the FFT in Unity yet, but what you describe sounds like a typical phenomenon about the FFT. The peaking value that you refer to could perhaps be the DC-value (or the amplitude at 0 Hz). At least when we tried audio.GetSpectrumData(plotData, 0, FFTWindow.BlackmanHarris) everything looked fine, so make sure your signal is free of any DC bias.

    14) OnAudioFilterRead quirks
    You are absolutely right -- the reason for this is that each AudioSource performs a mixdown of all the sounds it is playing (it can be many when it's one-shots) and only applies the effects on the mix. This also means that FMOD will apply the 3D distance-attenuation during the mixdown and therefore, if you apply a distortion filter on the AudioSource, that filter will be run on the whole mix of voices played by that AudioSource, thus you will hear the sound distort more and more as it gets closer/louder. I can see that this is not expected behavior if you think that the effects should be applied to the sounds individually, but the alternative would be to have all the AudioSource's effects multiply instantiated for each voice and run them all prior to the attenuation and panning that happens in the the mixdown, which is much more costly in terms of CPU and memory and will furthermore cut the effects as soon as the voice finishes playing. We will investigate if we can change anything to have the distance-based attenuation be applied onto the mix of the voices of the AudioSource rather than the individual voices, but we can't promise anything. Speaking about practical use cases, most of the effects such as lowpass, highpass, echo and chorus are linear, so scaling prior to the effect doesn't matter, but of course it is a problem when non-linear effects such as distortion are used.
    And yes, the data array is volatile, so you should never store any references to it. The idea is that you do all processing reading from the data array and writing back to it and that the time your processing takes should be as short as possible. If it takes too long (i.e. if you for instance do Debug.Log on every sample) the processing can't follow up and will lead to unstability that eventually causes a crash. This applies however to pretty much any realtime system.

    Cheers,

    Wayne, Jan
     
  16. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    @Wayne @Jan

    Many thanks for the very detailed reply! Unity 4.1 cannot come fast enough.

    Looking forward to try out the result of your work, and at your disposal should you need testers (not in 4.0 beta, as I only own the iOS Basic License).

    Regards,

    Gregzo

    P.S.: Your reply is exactly what I, and probably others, were expecting for some time : it's reassuring to know you're on top of things. Some users who have bought iOS pro because of the new audio functions might be upset that fixes won't come before 4.1, at quite a hefty price, though... Not your problem, but worth mentioning.
     
    Last edited: Oct 2, 2012
  17. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,303
    hm, I'm not familiar with signal processing and its lingo enough it seems
    I've used c/a zero value pitch detected (alongside energy of the buffer) as an indication that 'nothing is happening' in the streamed audio, thus handling possible loss of stream (and possibly restarting it)
    If the rightmost value / value at the maximal index on FFT buffer was not omitted, I got false positives - is this typical demonstration of DC bias ? If so, am I supposed to substract bias from the spectrum in order to get reasonable values ?
    Note: the stream was as 'normal' as it gets ) when streaming audio from the net, but I also had to ignore last FFT buffer value when playing local audio clip (i.e. mp3 file) since the algorithm was the same
     
  18. Malveka

    Malveka

    Joined:
    Nov 6, 2009
    Posts:
    191
    Wayne, Jan,

    Thanks for the detailed and information packed update! I really appreciate the fact that you took the time to post such a detailed and thoughtful reply. I'll be looking forward to bug fixes and enhancements in the upcoming 4.0 and 4.1 releases as you work your audio magic. 8^)

    Cheers,
    Mal
     
  19. BazLevelUp

    BazLevelUp

    Joined:
    Jun 27, 2012
    Posts:
    36
    Apparently, there are still issues regarding AudioClip.GetData() when called upon an AudioClip retrieve using WWW.GetAudioClip().

    If we look at this thread, it seems that the error message that happened before Unity 4 is now gone :)
    Still, I wasn't able to make it worked on Android ! :(

    What I do : With Unity Beta 4.0.0b11, after getting the url for the audio file using a file browser, I use it with a WWW object to retrieve an AudioClip. The clip is valid and I can play it. But when I called GetData(), the float array I got is filled with nothing but 0s :(

    Talked about in 2 topics : Jakovds and mine

    Report status : ongoing
     
  20. Aurore

    Aurore

    Director of Real-Time Learning

    Joined:
    Aug 1, 2012
    Posts:
    3,106
    Have you bug reported it? Post the number if you have/when you do. :)
     
  21. janm_unity3d

    janm_unity3d

    Unity Technologies

    Joined:
    Jun 12, 2012
    Posts:
    36
    Hi BazLevelUp, you cannot use GetData/SetData on a streamed sound. The reason for this is that since the sound is streamed, only a small part of it is in memory (and that is in compressed form even, so cannot be modified). There is no way we can support it, since this is by design in FMOD, but it should of course be mentioned in the documentation.
     
  22. Game-Whiz

    Game-Whiz

    Joined:
    Nov 10, 2011
    Posts:
    122
    Since this seems to be the thread of choice for audio issues, here goes mine :)

    I'm getting stutter on an hardware decoded mp3. My game has some performance hot spots, but these certainly shouldn't be an issue for a hardware decoded mp3. To test this, I made a simple project that only manages to run at 0.5 frames per second, and the hardware decoded mp3 played without a hiccup.

    In my game, things aren't so simple. I'm using NGUI and SmoothMoves, and I've matched the hickups to functions in these plugins that update geometry. Is there any relation between the updating of geometry and these hiccups in mp3? What else could be happening?

    Thanks for any help,

    Alex
     
  23. janm_unity3d

    janm_unity3d

    Unity Technologies

    Joined:
    Jun 12, 2012
    Posts:
    36
    Hi Alex,

    are you getting stutter with other file formats too or is it only MP3? Could you perhaps try with an uncompressed format like .wav? If it still stutters, it might be caused by interruptions happening otherwhere in the system. Garbage collection is a typical cause of such interruptions, and from what you write here, it sounds like the problem could actually be that the geometry plugins do lots of small allocations.

    Jan
     
  24. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    Looking forward to the new 4.1 Audio features, thanks for the heads up on them.

    Any chance of mp3 and/or ogg encode/decode from wave or float[] (from a mixdown of the main AudioListener) making it in?
    EditorUtility.ExtractOggFile seems to be a hook into FMOD's codec support, but can't access it outside editor.

    Use case is user generated recordings. Saving them as wav or float[] is huge space hog, being able to compress on platform would be good.
    All the 2rd part solutions I've seen are either expensive or GPL, which is not an option atm - and FMOD already supports this afaik?

    ty!
     
  25. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
  26. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    Is there any more info on this? I've been testing www.GetAudioClip() with a remote download and when I try and save the samples from it it looks to me like it only returns half the file. The size of the resultant clip is exactly half of the original file.

    ?

    ty!
     
  27. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
  28. wkaj

    wkaj

    Unity Technologies

    Joined:
    May 18, 2012
    Posts:
    50
    Hi gregzo,

    Can you give a reference to the bug you mentioned has been filed? I am having trouble finding it.

    cheers

    W
     
  29. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi wkaj,

    I just filed it again, got a confirmation e-mail immediately this time. Case nb 525163.

    I first filed it yesterday before going to sleep, and indeed did not get a confirmation e-mail.

    Cheers,

    Gregzo
     
  30. wkaj

    wkaj

    Unity Technologies

    Joined:
    May 18, 2012
    Posts:
    50
    Hi gregzo,

    We'll take a look at it thanks. The issue could be related to OS level microphone initialisation blocking everything. As a work around, I recommend starting the microphone perhaps when you start the scene and only process the output when required, that way you will avoid the startup and shutdown times.

    Cheers

    W
     
  31. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi and thanks for the tip, it's what I've been doing already. 1s looped clip for the mic, fed into float arrays when one wants to record.

    I'm by no means a Core Audio specialist, but my guess is that behind the scenes Microphone.Start has to start a new Audio Session, and that Unity's FMOD session pauses while this happens. Maybe FMOD related? I'm pretty sure that native apps which use the microphone don't simply leave it on non stop.

    I hope Jan and yourself are making good progress, and can't wait to try out the results of your work!

    One thing that I believe should be made extremely clear in future docs is the fact that on iOS, FMOD resamples everything at 24khz. Feeding it anything at a higher sample rate is simply a waste of resources, and I suppose causes a small performance hit (resampling). All my new iOS audio projects only use 24khz clips and I'm having much less synchronization quirks, even at "best latency" settings (buffersize of 256).

    Cheers,

    Gregzo
     
  32. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Question :

    Why is audio data stored as 32 bit floats? Wouldn't storing data as int16 save half the ram, and increase performance? There might be a good reason, if so, what is it? As it is, loading a 16 bit wav in a Unity AudioClip makes it take double the amount of memory it should...

    Cheers,

    Gregzo
     
  33. janm_unity3d

    janm_unity3d

    Unity Technologies

    Joined:
    Jun 12, 2012
    Posts:
    36
    Hi Gregzo,

    audio data that you load from assets isn't stored as 32 bit floats internally, unless that's actually the format it's stored in (in the baked asset -- not the source from which the asset was generated), but this would only be the case for some WAV/AIFF subtypes. The compressed files are always 16 bits internally. The conversion to 32 bit float first happens in GetData/SetData and it's mainly to make access easy and unified. However, when you create user sounds, it uses a 32 bit float internal representation.

    Jan
     
  34. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    Hi,
    I posted a thread on audio latency and synchronization on recording here http://forum.unity3d.com/threads/171785-Audio-Latency-on-recording?p=1174624#post1174624
    I'm not sure if it's a bug...or a feature. Any help appreciated.

    ty!

    ps: Would still love to know if we will have access to ogg file read/writing in 4.1? Be very handy!

    Edit: Scratch FMOD for runtime encoding to OGG
    http://www.fmod.org/forum/viewtopic.php?f=7&t=15702&p=52722#p52722

    Anyone know of a good cross platform solution that would work with unity for encoding to Ogg?
     
    Last edited: Feb 26, 2013
  35. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
  36. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi to all,

    Unity 4.1 is out, and as promised, some new audio functions have appeared. I couldn't miss the opportunity to tickle them a little, and so far, it's all working really well. So, congrats to Wayne and Jan for delivering as promised!

    The SetScheduledEndTime function is particularly useful, and saves a ton of potentially buggy code (to stop an audiosource on a precise sample before, I had to set the clip's data at 0 for a few thousand samples, and monitor via a coroutine when playback would enter the "silent zone" of the clip before stopping; restoring the vandalized clip took another SetData operation).

    Below is code I've used to test, it might be useful to audio newcomers. Jan, Wayne, do let me know if I'm not handling things properly!
    The script tests switching sources at any time between sources which play the same clip, the idea being that one should not hear anything amiss if the switch was sample accurate. Also included, a StopSourceOnSample function.

    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. public class NewAudioTests : MonoBehaviour {
    5.    
    6.     public AudioClip testClip;
    7.     public double testLatency = .1d; //how much time is given to schedule playback (press space to switch sources. If you hear clipping, try a higher testLatency value)
    8.     public int stopSample; //on which sample to stop (press enter to stop, again to play)
    9.    
    10.     AudioClip _createdClip;
    11.     AudioSource[] _sources;
    12.    
    13.     //ref for switching sources
    14.     AudioSource _currentSource;
    15.     AudioSource _otherSource;
    16.    
    17.     void Awake()
    18.     {
    19.         _sources = new AudioSource[2];
    20.        
    21.         for(int i=0;i<2;i++)
    22.         {
    23.             _sources[i] = gameObject.AddComponent(typeof(AudioSource)) as AudioSource;
    24.             _sources[i].loop = true;
    25.             _sources[i].clip = testClip;
    26.         }
    27.        
    28.         _currentSource = _sources[0]; //init refSources
    29.         _otherSource = _sources[1];
    30.        
    31.         _currentSource.Play();
    32.        
    33.     }
    34.    
    35.     void Update()
    36.     {
    37.         if(Input.GetKeyDown("space"))
    38.         {
    39.             SwitchSources(testLatency);
    40.         }
    41.        
    42.         if(Input.GetKeyDown ("enter"))
    43.         {
    44.             if(_currentSource.isPlaying)
    45.             {
    46.                 StopSourceOnSample(stopSample,_currentSource);
    47.             }
    48.             else
    49.             {
    50.                 _currentSource.timeSamples = 0;
    51.                 _currentSource.Play ();
    52.             }
    53.         }
    54.     }
    55.    
    56.     void SwitchSources(double deltaTime) //deltaTime : in how much time will the switch occur
    57.     {
    58.         int deltaSamples = (int)(deltaTime*testClip.frequency); //deltaTime in samples
    59.         double dspTime = AudioSettings.dspTime; //cache dspTime to be safe (not on the same thread)
    60.         int timeSamples = _currentSource.timeSamples; //idem
    61.        
    62.         _otherSource.timeSamples = (timeSamples+deltaSamples)%testClip.samples; //adjust the other sources playback position(timeSamples) abd make sure it wraps (%)
    63.         _otherSource.PlayScheduled(dspTime+deltaTime);
    64.         _currentSource.SetScheduledEndTime(dspTime+deltaTime); //this one deserves a kiss, thanks guys! Eliminates A LOT of ugly stopping coroutines I had to write to stop a source on a precise sample.
    65.        
    66.         AudioSource tmpSource = _currentSource; //Switching source refs
    67.         _currentSource = _otherSource;
    68.         _otherSource = tmpSource;
    69.     }
    70.    
    71.     void StopSourceOnSample(int stopSample,AudioSource source)
    72.     {
    73.         int timeSamples = source.timeSamples; //cache timeSamples and dsp time
    74.         double dspTime = AudioSettings.dspTime;
    75.        
    76.         int deltaSamples;
    77.         if(stopSample>timeSamples)
    78.         {
    79.             deltaSamples = stopSample - timeSamples;
    80.         }
    81.         else //clip will loop before we stop
    82.         {
    83.             deltaSamples = source.clip.samples - timeSamples + stopSample;
    84.         }
    85.        
    86.         double deltaTime = (double)deltaSamples/source.clip.frequency;
    87.  
    88.         source.SetScheduledEndTime(dspTime + deltaTime);
    89.     }
    90. }
    One minor gripe : the examples provided in the docs will repel inexperienced programmers. For example,in AudioSettings.dspTime's example, why include real time synthesis through OnAudioFilterRead? It's nice to have such an example, but the code provided is uncommented and will surely scare off newcomers.

    Anyway, many thanks for the much needed overhaul. I'll keep posting if I find issues (haven't tested on iOS yet).

    Cheers,

    Gregzo
     
  37. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi to all,

    Aside from being thrilled by the new audio functions, one bug remains:

    AudioSettings.outputSampleRate doesn't work in the editor (the rate is stuck at 44100 on mac, I guess 48000 on PC). This is quite annoying when developping iOS apps, which default at 24000, in particular when one uses OnAudioFilterRead for low latency stuff.

    I also had a strange quirk an hour ago when trying to debug outputSampleRate : all audio stopped going through (again, in the editor). The sources were playing (AudioSource.isPlaying ==true), but no sound could be heard. I relaunched, and rebooted, nothing. Had to re-install Unity. Can't reproduce that one, unfortunately...

    Cheers,

    Gregzo
     
  38. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    2,051
    As an addition to Gregzo's great work I figured I post a test project I made using the new 4.1 features, specifically the 'AudioSource.PlayScheduled'

    You can find a web demo here and a Unity package of the demo (minus the audio files) here.

    The demo itself is really nothing special, indeed its pretty damn horrible utilising Unity's default GUI, but the point wasn't to make something pretty, but to test out the functionality and robustness of PlaySchedule across many platforms (Win, Mac, iOS) using some old audio loops (not exactly great) I had from a project made over a decade ago in Shockwave.

    As such then its functionally very basic, but hopefully the source package might be of use to some.


    Web Demo Instructions
    Basically the sequencer is always in play mode and you can assign loops to the bars in any of the 6 tracks (an arbitrary chosen limit).

    A the top of the screen you have the Clip database, there are 4 different clip libraries to choose between, 'Disco', 'Rubber', 'Electro' 'Funky'. Choosing a different database of audio clips will erase all placed loops in the current tracks. Each clip library tests out different compression and memory settings, see the text info along side.

    Beneath the clip database is the actual sequencer. It displays 6 bars (each clip loop is 2 bars long though) and 6 tracks. To add a loop to a bar, simply click on the box (the ones that start with the word 'empty'). A pop-up will appear listing all the available sound loops for the selected database/library. Simply click on the audio clip you want to insert. You can also choose 'clear' to remove audio from that bar and are free to replace the audio clip by just selecting any loop.

    At the top left and right of the sequencer are the back and fwd buttons, these advance the display through the sequencer so you can make the song as long as you like. The sequencer itself simply loops from the first bar to the length of the track with the last bar in.

    At the end of each track is a 'clear' button, this will delete all loops from that track.

    You also have three sliders,
    • Volume - obviously sets the overall volume.
    • Queue Ahead - this is the amount of time before a clip is need, to call playSchedule() (i.e. current.dspTime + lookAhead > nextEventTime) So if there are pauses or glitches on playback you'd want to increase this time, in order to give Unity more time to decompress the sounds before playing them.
    • Pitch - testing out changing the pitch of the audio source and therefore the clips it plays, allows you to speed up or slow down the samples and apparent bpm.
     
  39. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    2,051
    As for bugs I'd really love for Unity to address three issues with audio for iOS

    1. The 24khz limit - That is such a horrible restriction. However I'd still like the ability to choose an output sample rate at run time, since for audio generation its helpful to have the option of doing high quality (say 44khz) vs low quality (24khz) for performance.

    2. Disable FMOD in some fashion to prevent it from taking control of the audio device to allow for other third party audio solutions

    3. Enable background audio, so that we could make use of new tech such as Audiobus, which is rapidly building up interest and has just been included in Apple's own GarageBand as well as being built into The Amazing Audio Engine for iOS which itself looks very cool.

    edit:
    I'm a little undecided about Gregzo point concerning user audio data being restricted to 32bit float. I fully agree that having 16 bit makes sense from a memory point of view and if you implemented a audio generator using fixed point math it might make sense in terms of performance too, but then i'm a bit lazy these days, so might just stick with floats, so perhaps being limited to 32 bit floats isn't so bad?
     
    Last edited: Mar 21, 2013
  40. Malandro

    Malandro

    Joined:
    Jul 22, 2012
    Posts:
    3
    Hi noisecrime,

    Thank you for the demo. I'm using playSchedule, I have some questions and maybe you can help me.

    - When you insert a clip on the absolute time-line using playSchedule you can't change the volume or pause it, right? To stop it you need to use SetScheduledEndTime()?

    - It could be possible to pause or stop the entire time-line?

    Thank you!
     
  41. janm_unity3d

    janm_unity3d

    Unity Technologies

    Joined:
    Jun 12, 2012
    Posts:
    36
    Hi gregzo and noisecrime,

    thanks for the kind words and for sharing the demos!

    You are right about the example chosen for AudioSettings.dspTime being too complex -- I wanted to show how this could be useful for tempo-synchronized effects such as stutter-delays, synchronized LFO-modulated filters etc, and therefore chose the simplest effect that would demonstrate this was a metronome, but I guess something that would just print out messages on the console would have been appropriate in this case. Perhaps it makes more sense to put these into demos in the asset store.

    Regarding AudioSettings.outputSampleRate, this seems to be mostly a problem of the Mac sound driver. Windows drivers seem more willing to accept sample rates not supported by the hardware, while on the Mac you can only set those that you actually see in the "Audio MIDI Setup" "Audio Devices" window. FMOD will not try to do the sample-rate conversion, so selecting a rate not supported will silence FMOD (a null-device will be used). I do realize this is a big inconvenience when developing for iOS, but it would require changes inside FMOD to fix this. So it is important to not have assumptions about the a specific sample rate in your code and always base the calculations on AudioSettings.outputSampleRate.

    The quirk you mention with the audio sources isPlaying being true is in fact caused by Unity using the FMOD null-device when initialization failed due to an unsupported sample rate. This way, the sound system will be silent, but will still behave in the same way as if it was using a real device.

    I have added a "Disable Audio" property in the project settings which will force Unity to use the null-device in standalone builds. Should be in an upcoming release. This is to support developers who want to use different sound systems than FMOD and avoid having FMOD allocate a physical output device in this case.

    Regarding sending audio to other applications, I think this is a case where you want (and need) to have your own high performance native audio system running that handles the integration with the inter-process audio APIs and will be able to run in the background even when Unity's frame updates are suspended while the app is not in the foreground.

    Jan
     
  42. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    Hi,

    Has anybody tried audio on android?
    http://forum.unity3d.com/threads/173234-Android-Audio-Issues

    Edit: Found the problem
    Leave AudioSettings.outputSampleRate as is. I was setting it to 44100 in my desktop build, and I'd left it in when I made my android build...and it causes all sorts of latency issues in addition to others audio hangups.

    Seems to work fine now.
     
    Last edited: Apr 24, 2013
  43. SimonG

    SimonG

    Joined:
    Apr 30, 2013
    Posts:
    2
    The bug preventing games from raising the FMOD mixer rate from 24 to 48 KHz (or other rates) on iOS was quietly fixed in the final Unity 3.5.6 release. Previously changes were ignored. Hence we are able to play our 48 KHz PC and console assets with crisp harmonics rather than a top octave of interpolation noise - this makes a big difference on headphones, and not much on the iOS speaker. 48 KHz mixing has been the default on PC since 1997 and standard on all Xboxes, PS2 and PS3 so it's convenient as well as better-sounding to use it on Unity3D too. If all your assets come from CD rather than DAT-rate recordings - especially the ones, like music, that don't move in relation to the listener - you may be better off with 44.1 KHz, or even 22.05 on slow hardware, to minimise artefacts of FMOD's simple sample rate converter but this is necessarily a compromise for a mix of assets and doesn't help much once Doppler introduces continuous pitch shift. Offline resampling, with a long sync FIR and anti-alias filter, is much the best way to match assets to the mixer rate you choose, and at least we can make the choice now!

    Unfortunately Android still has an annoying related bug. You can set the mixer to run at 48KHz when you start it up (understandably you can't change the rate with audio playing) but if you try this everything will play VERY SLOWLY and downpitched until you background your game and return it to foreground.

    I've not seen any other comments about this bug but given that AudioSettings.outputSampleRate = 48000; was simply ignored until recently it may be that few others have persisted with the feature. The encouraging sign is that when you return the game to foreground everything plays fine, at the highest rate and quality which suits the hardware. Unfortunately till the initial error is fixed it's not possible to use the feature in a game, as it will sound obviously wrong at startup.

    I have reported this to Wayne but not heard back from him yet. Meanwhile I'm very keen to hear if this has affected others, or if Unity 4.1 fixes it.
     
  44. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    @SimonG

    Thanks for letting us know, will test soon. Why hide such an important step forward?

    44.1khz for audio apps on iOS would be more convenient than 48 for one silly reason : Sonoma's AudioCopy only accepts 44.1.

    Cheers,

    Gregzo
     
  45. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051


    Thanks for the info. That must be what was happening also when I was setting it to 44.1K. I didn't try the background/foreground hack as I commented the setting of it to 44.1K out, but even though it works to smack the engine upside the head back into correct operation it's not an appropriate thing to impose on users to have to do to get it to work - UX wise.
     
    Last edited: Apr 30, 2013
  46. SimonG

    SimonG

    Joined:
    Apr 30, 2013
    Posts:
    2
    I've now reproduced the same problem on Unity 4.1.2f1, so I can confirm that this Android bug has not been addressed even in recent updates. :-(
     
  47. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi to all.

    Little issue with setting outputSampleRate on iOS : audio assets loaded from resources need to be loaded AFTER setting outputSampleRate, or they simply will disappear ( "trying to load assets not on disk" error ). I've setup a dummy InitScene just to set sample rate so that samples can be loaded in Awake in my main scene.

    It's great to finally have access to higher mixing rates. Why this important feature isn't mentioned in the docs, or in any release notes? And why isn't there an oprion in the editor's audio settings ( iOS default sampling rate, for example ) ?
     
  48. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi,

    Another bug, extremely annoying in my current circumstances:

    Since 4.2 ( maybe earlier ), setting AudioReverbZone.reverbPreset at runtime does not work.
    The new preset shows up in the inspector, but is not affecting the audio output.
    It's most probably very easy to solve : setting the preset in the inspector still works, only setting it by code malfunctions....

    Bug submitted, will update this post when I get news.

    Update : Silly fix, pseudo code:

    reverbZone.enabled = false;
    reverbZone.reverbPreset = newPreset;
    reverbZone.enabled = true;
     
    Last edited: Oct 11, 2013
  49. MHD-Salka

    MHD-Salka

    Joined:
    Mar 19, 2013
    Posts:
    72
    on android, changing dsp buffer from script causes huge amount of underruns, and the audio engine becomes extremely buggy, even when its set to something like (1024 ,4)
    also, audio settings set to "low latency" has almost zero effect, there is still huge amount of lag compared to iOS.
     
  50. AngelMixu_

    AngelMixu_

    Joined:
    Jul 12, 2013
    Posts:
    10
    Hi!
    I'm using Unity 4.2.1 for developing a game, and I need to mix some short clips into a full song.
    This is the code I'm using:
    I create an AudioClip, then I get the data of the full song, and of the short clips, then I try to mix the clips into the song data, and set to the created clip.

    At this point, songData is a big array of 0's.
    As I read, this issue was fixed, but don't know if it's not solved, or it's my fault.
    Do you have any idea of what's happening?