Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[Official] Audio Improvements

Discussion in 'Audio & Video' started by Erik-Juhl, Mar 10, 2016.

  1. Erik-Juhl

    Erik-Juhl

    Unity Technologies

    Joined:
    Jul 11, 2012
    Posts:
    59
    Hello!


    As you have probably seen in other threads, we like to check in on the direction we are taking with our technology to make sure we are best serving you. In this thread we would like to discuss the direction of Audio with you. Instead of restating the long term roadmap and short term projects I’d rather just step back and listen to what you all have experienced and what you all are needing. Knowing a bit about you also helps us better understand your feedback so please consider the following questions to get this discussion going.



    Who are you?


    What kind of game are you trying to build or would like to build?


    How does Audio fit into that? What use-cases do you have?


    What are the GOOD things about the Audio system that you like?


    What are the BAD things about the Audio system that you dislike?


    How can we make it BETTER?



    Thanks!


    Erik
     
  2. Baranger

    Baranger

    Joined:
    Mar 11, 2015
    Posts:
    1
    Hi,
    We are Audioplum studio ( http://audioplumstudio.com/) and we have worked on the realisation of audio and its integration in Unity on several projects, the latest one being A.B.C. Murders. After a year of getting professional with the available tools, we decided to create our own ones, which we published on the asset store (https://www.assetstore.unity3d.com/en/#!/content/45642) using our publisher's name, Baranger-Holsnyder ; we tried to fulfill all audio integration needs by "filling in the blanks" in Unity's audio, so to have a versatile and complete tool, made by sound designers for sound designers. It is all Unity is missing, and if Unity would be to make this part of itself, our plugin wouldn't exist anymore, but you would have all that an audio professional can ask of a game engine.

    Best regards,

    Alexandre Baranger ( Director/Audio Producer ) and Sacha Holsnyder ( Sound Designer/Developer )
     
  3. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    Where was this stated in the first place? The main roadmap doesn't have anything about Audio on it, and hasn't for a long time, even though this was mentioned at Unite Boston last fall.

    That out of the way, on to a response:

    Who are you?
    I'm the lead developer of Koreographer.

    What kind of game are you trying to build or would like to build?
    As a side project I'm working on a musical spaceship shooter where all interactions are based on features of the music (the game uses Koreographer).

    How does Audio fit into that? What use-cases do you have?
    We currently use a multi-layer music system that we built on top of Audio Sources. Koreographer watches the state of Audio playback and sends signals to the game about what's happening in them so the game can trigger events. The multiple layers allow the player to interact with the mix and overall experience. We also support pitch adjustments as speeding up/slowing down audio playback causes the game itself to speed up/slow down.

    What are the GOOD things about the Audio system that you like?
    This question is very difficult to answer because we tend to take for granted what we already have that works. As such this is a short-list of things I am thankful for:
    • File type support (thanks, FMOD Low-Level!).
    • File loading support seems to work flawlessly.
    • Basic AudioSource API is simple and intuitive.
    What are the BAD things about the Audio system that you dislike?
    • Recommendations about audio preparation are non-existent and lead to confusion for teams. In conversations with folks on the Audio Team, I've heard stuff like all audio imported into Unity should be uncompressed (wav/pcm). The documentation explains the reason behind this but does not recommend authoring in wav and allowing the system to deal with compression internally. This means that a lot of teams deliver audio to Unity as compressed (and, importantly, lossy) file types.
    • Documentation for AudioSource.time and AudioSource.timeSamples does not mention that the values returned are not the "actual time of the audio precisely" but, rather, "the position of the audio buffer "read" head and therefore can be identical in situations with high frame rates.
      • This note in AudioSource.time is only somewhat helpful: "Be aware that: On a compressed audio track position does not necessary reflect the actual time in the track. Compressed audio is represented as a set of so-called packets. The length of a packet depends on the compression settings and can quite often be 2-3 seconds per packet." Here are a few problems:
        • No mention of whether or not this affects the timeSamples version as well (a similar note does not appear in timeSamples).
        • No description of what the returned time actually represents. Does it equate to the point of the last sample that was buffered? The last sample that was decompressed?
        • Is there a way to mitigate this? Any recommended strategy for working around such a big gap?
        • As "Compressed audio" is cryptic shorthand for "MP3/Vorbis", does this affect both of these? Only some? Is the problem platform-specific?
        • This quote is followed up with a "See Also: timeSamples variable" on the same line as the warning, which makes them feel connected. I'm willing to bet that this is simply a related field and should exist in its own paragraph (as is done with the related message on the AudioSource.time doc).
    • Mod file issues that have existed with no word from the audio team for quite a while.
    • Very little metadata about what's going on with audio:
      • Did the audio loop?
      • Did the audio jump/seek?
    • Sample-specific timing is fairly impossible, it seems. You can schedule audio but you cannot pin a sample to a location along the absolute timeline (this would require filling a buffer with zeros up until the location specified by the "pin" at which point actual audio samples would begin filling).
    • When scheduling one AudioClip playback to match another, you have to time them both against the AudioDSP timeline. If you attempt to change the playback speed ("pitch") of the already-playing audio, you need to reschedule the one that is attempting to match the other. This is particularly annoying if you're doing a pitch-ramp.
    • Dealing with synchronization of multiple audio layers is susceptible to threading issues (unless you essentially rebuild using PCM callbacks what's happening at a lower level already) as adjustments to different AudioSources must happen serially on the Main thread.
    • No good control for AudioClip playback in the Editor. Basic playback is indeed possible but you do not get the same control over pitch, effects, etc. that you do on the game side. You can get this stuff at edit time but it requires adding a "no save" object to the scene and dealing with that complication.
    How can we make it BETTER?
    • When retrieving time (whether samples or seconds), add an option for this to be estimated. Currently it simply tells you "the audio buffer has read this far". The estimation would calculate how much of the buffered samples have likely made it to the speaker.
    • Improve documentation of what's going on with the AudioSource.time and AudioSource.timeSamples APIs.
    • Improve Audio Synchronization capabilities.
      • Allow one AudioSource to synchronize/schedule against another AudioSource.
      • Allow scheduled playback to be pinned to a specific location along the timeline. This should properly handle pins that occur in the past (to handle situations where someone attempts to pin something to now-time rather than slightly in the future). By this I mean that if I say "pin sample 50 to now" and the buffer read is already 100 samples ahead of what the DSP time was reported as "now" then the buffer will begin filling at sample 150, as though it had already handled buffering the previous 100 samples.
      • Provide an immutable struct with property updates that can be passed to the Audio thread, ensuring that changes to a multiple audio sources happen at the same time on the Audio thread. Something like "AdjustParametersSynced"...
    • Expand the Editor playback controls of AudioClips (getting AudioSource equivalence would be amazing). This would allow more interesting/useful edit-time tools to be developed.
    I hope this is helpful!
     
    Last edited: Mar 11, 2016
    Delumine likes this.
  4. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    SonicBloomEric, I agree with everything you said. However, I believe rescheduling is worse than you think. Referring to this:

    • When scheduling one AudioClip playback to match another, you have to time them both against the AudioDSP timeline. If you attempt to change the playback speed ("pitch") of the already-playing audio, you need toreschedule the one that is attempting to match the other. This is particularly annoying if you're doing a pitch-ramp.
    I have a thread stating that it is impossible to guarantee accuracy (for gapless) of the rescheduled clip based on the current position of the audio, due to the fact that audiosource.time is a float and the DSP one used to calculate schedule time is a double. Math precision gets lost and often tells you 3 slightly different times to schedule when asking 3 frames in a row! Here's my thread.

    http://forum.unity3d.com/threads/wh...always-slightly-different-every-frame.354579/

    Basically, if you change the pitch or position of a currently playing clip, and want to reschedule the next one to be gapless, it's a crapshoot. Sometimes there will be a slight gap. Sometimes there will be a slight overlap. Sometimes it will be gapless (or close enough to not be recognizable as non-gapless).

    Would also like to see:
    1) Ability to loop a clip X times instead of just no loop and infinite loop.
    2) Built in events on the Audio Source we can subscribe to telling us: did the audio stop / loop back to beginning.
    3) Ability to loop just part of the clip. Making intro/looped section possible in a single clip.
    4) To fix the math error rescheduling problem I stated above, would like a property in AudioSource to give the *always correct* AudioDSP end time of the current clip if it is not looped, based on its current position and pitch.

    These are all issues that many of the 6,000+ users of our Master Audio plugin encounter in some cases or are asking for.
     
    Last edited: Mar 10, 2016
    SonicBloomEric likes this.
  5. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    This would be great but we'd still have potential issues with scheduling reverb tails to apply to the sub-section loops. This would be handled if we could schedule an AudioSource to play along with the timeline of another AudioSource, and not just the Audio DSP timeline.

    One more for "How can we make it BETTER?":
    • AudioSource.time and all related time-in-seconds methods should have a version that uses double as the type for all the precision issues mentioned by @jerotas above [and more!].
    Hope this helps.
     
  6. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    What I like:
    1) API is easy.
    2) Audio Mixer in Unity 5 is really cool.

    Dislike:
    well the sheer number of dislikes (mostly things missing) is why we built Master Audio. The number of features not in "core Unity audio" that are commonly used and needed are staggering. I won't enumerate them here because that's a source of income for me.

    A notable one that our plugin does not cover and probably never will:
    1) Ability to do split-screen games with multiple Audio Listeners.

    Also, audio occlusion would be a great thing to add (we plan on adding it to our plugin regardless)
     
  7. Tazman

    Tazman

    Joined:
    Sep 27, 2011
    Posts:
    94
    Who are you?

    Developer of the Fabric audio toolset

    What are the GOOD things about the Audio system that you like?

    - Simple straight forward audio API
    - Good cross platform compression support
    - Audio Mixer is implemented well
    - Native Audio Plugin SDK
    - Audio clip preload/load in background options are very useful

    What are the BAD things about the Audio system that you dislike?

    - Threading issues, we have games that can sometime experience stalls when Fabric calls certain AudioSource functions. I believe this is probably due to FMOD's somewhat "messy" multi-threaded support.

    - As mentioned already scheduling of audio sources with variable pitch changes can be very tricky. In Fabric this is handled by using the Audio DSP timeline entirely (can't use audiosource.time due to the math errors jerotas mentioned) and re-calculating the next transition internally but it requires a small latency on first play... its not ideal but it works unless the audio thread is starved then things get out of sync since there is no mechanism to detect that.

    - No easy way to load audio clips that are not referenced by a script and are located outside a resources folder. Yes this can be done using the WWW class but its audio support is not very good.

    How can we make it BETTER?

    The suggestions by SonicBloomEric are spot on, on top of them I will add....

    - Ability to set channel gains on multi-channel audio clips.
    - Ability to query audio source's final gain (after attenuation is applied)
    - Allow to pass different types of parameters to the native audio plugins not just floats
    - Allow Native Audio Plugins for audio sources as well
    - Multi listener support, this is supported already by FMOD internally so in theory!! it should be easy to expose.
     
    SonicBloomEric likes this.
  8. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    280
    I'm the author of RhythmTool.

    For the stuff I'm working on I don't need a whole bunch of features. Just something that lets me play and analyze samples. In that regard, the current system is easy and straightforward to use. It's easy to analyze and manipulate AudioClips.

    The main problem I have come across is the limited support for loading audio files at runtime. Loading MP3 files is only supported on mobile platforms, for example. That's probably due to licensing etc., but it should be possible to use the OS's decoder, like Bass.dll does.

    Other than that I would really like to see a GetSpectrum method that would return the spectrum of a specified part of an AudioClip, and not just the spectrum of what's currently being played by an AudioSource. That would make it a lot easier to analyze spectrum data ahead of time. Right now you'd have to either play a muted version or roll your own FFT.
     
  9. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,350
    hi,

    I've pretty much used only the basic unity audio features (as in, drag & drop wav/ogg in and it works fine..)
    but if there is a wishlist, then playing c64 sid files would be nice!
     
  10. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    One little thing I came across just yesterday was that the workflow for the AudioMixer is pretty poor. Unless I've missed something extremely basic, it appears that you cannot audition your Mixer setup unless you're actively playing your game? If triggering/controlling a mix with the editor in Edit Mode isn't something that's already supported then it really should be.

    I imagine, for instance, that I should be able to play a sound effect through a bus in a looping manner and then play with Effect settings in real time to get a sense for what the changes are actually doing. Currently it seems that the best way to simulate this would be to build a test scene for this very purpose and turn on "Edit in Play Mode"... which seems like way too much work for something that should simply be built in.

    Based on this tutorial video it appears that the normal workflow is to create a dummy scene, add an AudioSource with Play On Awake and Looping enabled and then put the Audio Mixer panel in "Edit in Play Mode". This feels counterintuitive and actually goes against everything we're taught throughout the rest of Unity (don't make edits you care about in play mode). We don't have to be in Play Mode to create and edit animations, right? Why is audio any different here?
     
  11. Zicandar

    Zicandar

    Joined:
    Feb 10, 2014
    Posts:
    388
    Who are you?
    Developer of the ELIAS Adaptive Music Engine + A Unity Plugin for it.

    What kind of game are you trying to build or would like to build?
    A Music engine that allows for much better handling of music in games.

    What are the GOOD things about the Audio system that you like?
    It's easy to do stuff such as output a custom stream.
    Unity handles resampling between both sample rates and channels.
    OnAudioFilterRead saves the day for the most part! But removes the other benefits mentioned.

    What are the BAD things about the Audio system that you dislike?
    Missing a few parameters in C# for streamed sounds, because the default in FMOD seems to be 400ms buffer size, and I have been unable to find a way to change that for streamed audio, even if I set the length of the audio clip to be smaller then that. (In what case it will simply read smaller amounts at a time, until it has gotten the 400ms it wants.


    How can we make it BETTER?
    For streamed sounds, don't ask for 400ms of sound at a time. Either expose it as a parameter or simply put it to max the size in samples parameter passed in.
    It's only possible to get decoded data from an audio clip if it's decompressed on load. I would either like to be able to get the raw (ogg) data "streamed", and/or get raw data from streamed samples. (I can understand that this fix would require a bit of work).
    Allow OnAudioFilterRead to be run before channel and sample rate conversions are run. Currently the "filters" need to be able to handle a lot of different combinations.
    As mentioned by Tazman, allow audio sources to be created in the Native audio plugin.
    Allow this without a crazy hack: native audio plugins in unity 5.0
    Allow the Native audio plugins to use more then simply floats. (Perhaps the same list as FMOD supports, as the Data parameter type would allow more custom behavior).
     
  12. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    I think you may be looking for DSP Buffer Size stuff. Check out the Audio Manager for configuration settings and access to adjusting it.
    Oooh, seconded!
     
  13. Zicandar

    Zicandar

    Joined:
    Feb 10, 2014
    Posts:
    388
    Oh no, that won't make any difference! If you look at FMOD's code, you'll see that when creating a streamed sound you choose how large the stream buffer is supposed to be, and that is by default 400ms.
    Your suggestion matters for OnAudioFilterRead, but not for OnAudioRead... (Unless they have fixed this with a somewhat recent version of Unity of course).
     
    Last edited: Apr 8, 2016
  14. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    Ahh. This I could not speak to as I have not tried to use the OnAudioRead. I guess I'm used to using the PCMReaderCallback. And it is (was??) my understanding that the size of requested audio is dependent upon the DSP Buffer Size. I actually didn't even know that MonoBehaviour.OnAudioFilterRead was a thing...!

    What is the "OnAudioRead" method/function to which you are referring?
     
  15. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    1,035
    Who are you?
    Developer of "unofficial" Unity plugin of Superpowered audio sdk and Alive music engine(no reference it's an internal project)

    What kind of game are you trying to build or would like to build?
    It's a secret right now, but it will incorporate rhythm mini games

    How does Audio fit into that? What use-cases do you have?
    The alive engine play tracker modules with automation, note events, instruments and sample swap at runtime and more. It is mostly used to have a better feedback and immersion in the game.
    I am relying mostly on onAudioFilterRead and Resource async loading at runtime.
    I am also using the above plugin, but that's more of a pet project and out of necessity right now(see bug below)

    What are the GOOD things about the Audio system that you like?
    I love that it's simple and just works. Everything is easy to setup in seconds.


    What are the BAD things about the Audio system that you dislike?
    Too little access to low level things.
    Some on the top of my head:
    1-Cannot access audio files binary if the audio it's recognized by unity.
    Please allow reading recognized files too! Right now the only way it's to rename them to .bytes.
    So you either read them by yourself or you read them with Unity. Cannot have both.
    Also allow reading the audio data as it is as bytes or words, so we avoid doing back and forth between float and bytes.

    2-onAudioFilterRead has lots of bugs.
    The last I have reported is that it allocate memory for the buffer at every call.
    Another one is that if you create an audioclip (didn't report this one yet, sorry) and attach it to the audiosource where the onAudioFilterRead is, the mixer will be totally ignored. No mute, no bypass.
    Another I can't remember, probably was only with Unity 4.

    3-Please allow PCMReaderCallback(is that the onAudioRead you mention? ) to act exactly like onAudioFilterRead. Right now it prebuffer something at initialization, and it stop being called after the length you specified. It would be nice to have an infinite call so that you can have "virtual" audio clip that are actually generated by code elsewhere. With onAudioFilterRead you lose this abstraction.

    4-The native audio plugin is a nightmare. There is very little info on it, that I gave up.
    Plus the workflow is too lengthy, allow the code to be integrated and compiled in Unity where possible, just like iOS plugin that you can include m,mm,cpp,h, files in the project.


    With onAudioFilterRead you don't have to resample samplerates and channels. Maybe you did something you shouldn't? onAudioFilterRead it's quite picky and buggy.
    For example you must not create and attach an audioclip or it will bypass the mixer(while in Unity 4 you are required to create it or it will have issues).
    Anyway there is a bug in that method now that allocate memory for the buffer at every call, causing the GC to fire continuosly, so beware.
     
    Last edited: Apr 11, 2016
  16. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    I get around this by creating a "buffer" audio clip and set the AudioSource to looping. If I want no audio played back for a time I can either stop the AudioSource or simply pass zeros to the buffer. Have you tried something like this?
     
  17. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    1,035
    But does it get called at constant time like onAudioFilterRead?
    Somebody told me it doesn't. I need to play audio in real time.
     
  18. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    I have no idea how OnAudioFilterRead works just yet - never looked into it. The PCMReaderCallback is called from the Audio thread with the exception of the initial "Play" call - in that case it seems to fill up the initial playback buffers from the main thread before handing things to the audio thread. It doesn't get things "in real time" as it uses the buffers, of which the number and size can be configured in the Audio Manager. The lower the number & smaller the size of these buffers, the lower the latency. That said, you also increase the likelihood that you'll end up with voice starvation.

    I would have completely guessed that OnAudioFilterRead works the same way (or similar enough as makes little difference in the end - especially if it's called on the Main thread, rather than the Audio thread).
     
  19. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    Allow us to actually inspect the state of the AudioSource. It's really hard to understand why there isn't an "isPaused" flag (or, better yet, "AudioSource.playbackState", which could be "Playing | Paused | Stopped"). We have the ability to call Play, Pause, UnPause, and Stop. There is a difference between Play and UnPause and the one "isPlaying" flag simply isn't enough.
     
  20. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    1,035
    I investigated further and like I thought, PCMReadercallback is not tied to sample rate but it's file buffer based.
    If I input 2 sample as length, it ask me 4 sample (probably because it's the minimum for stereo), if I input 4096 samples it ask me 99% of the time 4096 and sometime a little less, can't understand the rationale but it's not tied to sample rate.

    onAudioFilterRead is the way to go for outputting a wave stream to the soundcard.

    So, I stand on my request. Could you guys implement a PCMReadercallback variant that act exactly like onAudioFilterRead, using samplerate and dsp buffer size? So we can have a virtual audioclip to pass around in the project?
     
  21. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    The PCMReaderCallback absolutely is sample rate based. Everything audio in Unity is as the entire audio system runs at a single Sample Rate at the end of the day. The Sample Rate is how many samples need to reach the sound driver in a given second to be sent to the speaker, right? That has no bearing on how many samples the system may ask you to provide at any given moment. It may have a buffer of 0.5 seconds (this means it's storing 22050 samples when your sample rate is set to 44100). As the driver takes some of those samples, the system now has less than its ideal 0.5 seconds of buffer and must go back to the file (AudioClip) to get more samples. In the case of the PCMReaderCallback, you are essentially creating a shunt into the system that lets you define the float values that the system receives as sample data.

    Take a look at AudioClip.Create again. You can set the frequency of the data - that is the same thing as "the sample rate at which the file was recorded." Keep in mind that the audio system may have a different sample rate than the audio file. To get things to match you run through a process known as resampling, something @Zicandar posted about above. Regardless, the frequency tells you how many samples-per-second your data was 'recorded' at, while the lengthSamples tells you how many samples the AudioClip contains. In this case, there is no actual "file" backing the system but you may have opened a file buffer elsewhere and be managing it yourself. To determine the length of the AudioClip in solar time, you can simply do "lengthSamples / frequency".
    Do you actually know what your DSPBufferSize is? I believe the default is 4096. Try making your AudioClip much bigger (like 1 second's worth of data - give it a frequency and a lengthSamples value of 44100). I'd be willing to bet that you still see that the data buffer in your PCMReaderCallback function has a length of 4096.

    As for why you get a callback requesting two samples (four because stereo), well, that may be because the file itself doesn't have more than that and the request for more samples may internally be "Mathf.Min(dspBufferSize, lengthSamples)".

    Keep in mind that the 4096 length is also looking for stereo data. This means that you should be feeding it 2048 samples for each channel and, I believe, this must be interleaved.

    One Major Question
    Did you set the "stream" parameter to true? If you do not, the AudioClip will read everything at the beginning and Unity will handle all of the audio sample buffering by itself. If you set the stream parameter to true, you will begin to receive lots of PCMReaderCallback calls, most of which come from the Audio thread.

    From the documentation:
     
    00christian00 likes this.
  22. Polkatuba

    Polkatuba

    Joined:
    Oct 31, 2014
    Posts:
    79
    How can we make it BETTER?

    Based on the last time checked:

    - I think the 5.1 channel support should go deeper. Currently there is no native way of setting what goes into center and LFE channels and what does not. In fact, currently ALL audio goes automatically into LFE which is certainly not desired. I think feature like this would crucial in order to make usable surround sound with Unity.

    - The reverb zones are separate from the mixer. You cannot make them to go into a certain mixer channel. It kind of ruins what you are trying to achieve with the mixer. Part of your audio goes thru mixer and some other part (the reverb zones) does not.

    Thanks
     
  23. Zicandar

    Zicandar

    Joined:
    Feb 10, 2014
    Posts:
    388
    This is acually already possible! Use the Streaming Assets folder. (Look at our plugin for an idea on how to do this: eliassoftware.com )
    Sorry yes, I meant the PCMReaderCallback and not onAudioRead, your correct.
    However you can get it to be an infitinte virtual audio clip, we do. To achieve this you need to set create the audio clip and set it to streamed. HOWEVER it will ask for samples in chunks, until it gets 400ms of audio, then wait 400ms, and then ask enough times to get 400ms of audio. THIS is my problem with it, and easy to verify.
    Well, they work differently, OnAudioFilterRead will always provide and expect Unitys output sample rate and channels, not the audio clips. Unlike GeneratePCMData, it will ask for what we specified and then convert it for us. This means that for us who create audio, we need to ourselves convert it to the expected sample rate and channel count.
    As mentioned earlier, it's probably tied to the 400ms "streaming buffer" that FMOD has by default. If using OnAudioFilterRead it will always ask for a multiple of the DSP buffer if I remember correctly.

    All this info is from one or two versions old Unity, but I have not seen any patch/release notes that indicate this has been fixed!
    Any suggestions on how to "nicely" work around this would be much appreciated!
     
    SonicBloomEric likes this.
  24. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,085
    Ahh, this would actually explain the variability in responsiveness that I seemed to get when I was playing around with it - sometimes things felt snappy and sometimes things would take a bit longer to "respond". I'm guessing this was just "being lucky that the change was requested just before the next round of sample-feeding" versus unlucky that it had to wait for the next set of buffers to request.

    For the record I have no idea if things changed in 5+. The last time I seriously looked into this I was developing on 4.5 (still the default platform for what we do - almost nothing important has changed for what I'm focusing on).
     
  25. Zicandar

    Zicandar

    Joined:
    Feb 10, 2014
    Posts:
    388
    Sadly it hasn't, at least not until 5.2.
    As I might have mentioned you can work around this by using OnAudioFilterRead instead, but then you have other issues as mentioned...
     
  26. Erik-Juhl

    Erik-Juhl

    Unity Technologies

    Joined:
    Jul 11, 2012
    Posts:
    59
    Thanks to everyone for taking time to give us all this great information. I'll try to summarize the feedback received so far below. If you have feedback for us that you don't see in this summary please let us know!


    The Good
    Simple and just works
    Everything is easy to setup in seconds
    Basic audiosource api is simple and intuitive
    File level support
    File loading works flawlessly
    Audio Mixer
    Cross platform compression support
    Native audio plugin sdk
    Audio clip preload/load in background
    Easy to analyze and manipulate audio clips
    Easy to output a custom stream
    Handles resampling between both sample rates and channels
    OnAudioFilterRead saves the day


    The Bad
    Documentation incomplete or otherwise lacking important detail
    No recommendations or best practices for audio preparation
    Mod file issues
    Very little metadata
    Sample-specific timing impossible
    Difficult scheduling and synchronization
    No good control for audioclip playback in the editor
    Threading issues causing stalls
    No easy way to load clips not referenced by s script and located outside resources folder
    Limited support for loadinf audio files at runtime
    Missing a few parameters in c# for streamed sounds (cant change buffer size)
    Too little access to low level things
    Cannot access audio files binary if not recognized
    OnAudioFilterRead has lots of bugs


    What could be Better

    Documentation
    Improve audio synchronization capabilities
    Expand playback controls of audioclips in editor
    Ability to loop a clip X times
    Built in events on the audio source
    Ability to loop just part of the clip
    All time related methods should have a version that uses double
    Multiple audio listeners
    Audio occlusion
    Set channel gain on multi-channel clips
    Query audio source's final gain
    Pass more than just floats to native audio plugins
    Allow native audio plugins for audio sources
    GetSpectrum method that would return the spectrum of a specified part of a clip
    Triggering/controlling a mix in edit mode
    Able to get the raw data streamed
    Allow OnAudioFilterRead to be run before channel and sample rate conversions are run.
     
  27. laggyluk

    laggyluk

    Joined:
    Oct 7, 2011
    Posts:
    32
    Hi
    Who are you?
    < that's me

    What kind of game are you trying to build or would like to build?
    working on a sort of music tracker/sequencer

    How does Audio fit into that? What use-cases do you have?
    I have working sample based instruments implemented, planing to add some sound generators.
    Had high hopes for new audio mixer and built in audio effects but it doesn't really give me any advantage. Each instrument user adds would have to have individual AudioMixerGroup created at runtime and audio API doesn't support that

    What are the GOOD things about the Audio system that you like?
    It works and has potential

    What are the BAD things about the Audio system that you dislike?
    As other mentioned before, working with threading/audio scheduling is sometimes painful.
    Limited API compared to editor

    How can we make it BETTER?
    Please make it possible to create and configure AudioMixerGroups at runtime,
     
  28. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    Agreey laggyluk, I think the newfangled mixer is awesome but pretty useless when you have to "expose" parameters to use them and can't really do anything from script.
     
  29. ReaktorDave

    ReaktorDave

    Joined:
    May 8, 2014
    Posts:
    139
    Who are you?
    I'm a sound designer (bachelor) and composer (one year at university left). In my free time, I work on sound design and music for games. I also code in C# to integrate and create the tools I need in Unity.

    What kind of game are you trying to build or would like to build?
    Mainly narrative-driven games ...

    How does Audio fit into that? What use-cases do you have?
    ... so I'm focused on creating film-like mixes and adaptive music.

    What are the GOOD things about the Audio system that you like?
    AudioSource.PlayScheduled() in combination with AudioSettings.dspTime is a great way to code custom music systems.
    The new mixer seems quite capable and I like that you can define a specific mixer group for each AudioSource. However, I didn't use the mixer that much atm, as the mix of my current game will be done next month or so.
    It's great to have tools like OnAudioFilterRead().

    What are the BAD things about the Audio system that you dislike?
    Playing an audiofile not from start but from a certain position is rather painful, as you have to use OnAudioFilterRead() for that.
    Panning is not customizable enough.
    Mixer is not accessible enough from script (just skimmed it's API).
    No native support for custom realtime-parameters and curve types, only for predefined parameters (volume, spatial, spread, reverb).
    When writing scripts using arrays/lists of audio files, you have to drag each audio file into the array/list manually.
    Accessing audiodata during assetimport is not possible.

    How can we make it BETTER?

    Support for importing wavpack (not necessarily for deploying, just to keep the repository small).

    AudioSource.PlayScheduled should be able to start from a custom point of the audio file. It would also be nice if you could define custom loop points for each audioclip.

    Once a sound is in the 3D-Space, I have no control about it's panning. I'd like to control parameters like center-strength (how strong the sound sticks to the center before panning to left/right), monitor angles (affecting when at what left/right/fron/rear-position and how strong a sound is panned to these channels), LFE-sends (currently, Unity just sends everything to the LFE ...) - and it would be good if these parameters could also be controlled/accessed/manipulated (so I can f.i. define two different center-strengths and depending on the distance to the AudioSource interpolate between these settings).

    I read over the API of the mixer and thought that the concept of exposing a parameter and by that loosing the ability to also control it with snapshots is quite strict. However, as I'm going to focus on mixing my current project next month, I'll be able to give more detailed feedback somewhere in June/July. Would that be too late?

    It would be awesome if the concept of realtime-parameters for volume, spatial, spread and reverb would be user-extensible to any parameter, target and destination. This way, for example I'd be able to adjust delay times if an AudioSource is far away.

    Placing audio assets into arrays/lists could be improved by supporting drag&drop of multiple selected items into arrays/lists.

    During Assetimport, it should be possible to access the audiodata (the actual decoded audio samples), regardless of compression and load type. An awesome feature would be if we could during assetimport directly add custom (even more awesome: timestamped) metadata to the audio file and if we could retrieve that metadata on a playing AudioSource from script. This would be extremely useful for precalculations like RMS or formant analysis and would save tons of CPU if not calculated during runtime.

    It would be killer if Unity audio had support for geometry based audio. I could imagine it like tagging levels with materials and defining how these materials would affect certain parameters of the reverb and eq or like how much nearing a wall affects delay times and volumes.
     
    Last edited: May 24, 2016
  30. YondernautsGames

    YondernautsGames

    Joined:
    Nov 24, 2014
    Posts:
    352
    Who are you?
    I'm a solo indie developer working mainly in mobile but looking to branch into PC and possibly console at some point in the future. I do everything but create the audio for my projects. I tend to buy that royalty free, except for my current project...

    What kind of game are you trying to build or would like to build?

    Working on an app contract that provides educational audio experiences for schools.

    How does Audio fit into that? What use-cases do you have?

    The people I'm contracting for provide me with audio files and I am creating an app to play and structure them. Most of my work for this project is UI based, creating a kind of guided MP3 player for teachers to control the audio the kids are listening to.

    What are the GOOD things about the Audio system that you like?

    The audio mixer is great. Recent improvements have made it much easier for me to sync multiple tracks and precisely time fades, etc. In fact, I had to write very little code to get all the functionality of a basic mp3 player (skipping within tracks, time stamps, etc). Pre-load vs streaming vs load in background is all quite flexible. Not messed with the positional audio yet as it's not really relevant to what I'm working on, but would like to experiment at some point soon.

    What are the BAD things about the Audio system that you dislike?

    • Forced conversion of audio. I don't seem to be able to provide files that are in a playable format without Unity messing with them. Only way to get the results I want is to provide uncompressed PCM audio, which leads me to...
    • FSBTool ERROR: Run out of memory! Half my source files can't be imported without compressing them ahead of import (and therefore causing quality issues due to prior issue). If the input file is over 400mb it hits me every time (that sounds big, but the resulting compressed file would be much smaller). This also doesn't seem to have any connection to the memory size on the machine in question. Seems to be hitting some kind of max usage constant in code.
    How can we make it BETTER?
    • Firstly, fix the memory crash preventing my importing uncompressed audio.
    • Improve either the documentation or the naming in the editor to make the difference between positional and simple stereo audio more obvious. I don't want any positional effects messing with the quality of the output and I think I've achieved that (by outputting directly to a mixer), but it's not obvious. I'm having to experiment a lot and take things on faith because it's not clearly explained.
    Thanks
     
  31. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    1,035
    Erik, doesn't Unity have a dedicated audio team? It has been 2 months I reported 2 confirmed bugs and I hardly ever see anything audio related on the Unity release notes.
    Are you doing a major overhaul of the audio framework or the same devs have to work on non-audio stuff? Otherwise I don't understand why the audio bugs( not just mines) stays there forever.
     
  32. DaveHoskins

    DaveHoskins

    Joined:
    Sep 9, 2013
    Posts:
    190
    I would personally like to see a native code call as a substitute to OnAudioFilterRead.
    It would be very useful.
     
  33. CrazyBits

    CrazyBits

    Joined:
    May 17, 2014
    Posts:
    3
    Who are you?
    Lone developer with few nice collaborations, working on PC/Console games. Not really an "audio guy", but I can share my experience.

    What kind of game are you trying to build or would like to build?
    Trespassers, a 2D coop run n' gun game with lots of enemies, upgrades, etc. I dont know in the future, but 3D games will be there too.

    How does Audio fit into that? What use-cases do you have?
    A really dynamic game needs really dynamic sound, avoiding repetitions on common and really frequent sounds (shots, hits, deaths, steps). Mostly stereo sound, due to the nature of the game.

    What are the GOOD things about the Audio system that you like?
    • Directly integrated in Unity, including profiling tools, so it is really easy to set up, and everything stays in the same position
    • It works
    • Free for all platforms
    • Drag & drop for importing files into Unity is there
    • Visual interface for the components related (AudioSource, special reverb areas ...) gives good control
    • Audio mixer: controlling channels is straightforward in most cases, and plain easy for basic needs
    What are the BAD things about the Audio system that you dislike?
    We have been working with newer versions of FMOD and, only in case we have really serious problems, we won't switch back, because:
    • The tool itself. FMOD Studio is a really good tool to author the audio, and its profiler works really well with Unity. So, the audio guy works in his enviroment without having to learn a new tool, limited in our opinion.
    • Audio clip VS event: this is crucial for speed of development and solves many issues. Say I want to fire my gun, so I want it to sound as the designer expected, just a line of code saying "gunFireEvent.Play();" should suffice in most cases. The event itself should be responsible for randomizing the sound accordingly (pitch, volume, clip, whatever). Yes, this could be created with custom scripts but, instead of reinventing the wheel, you got better tools for that (FMOD Studio). I recommend everyone checking the possibilities that a single event has (loop regions, transitions, multiple audio tracks mixed, randomized ...).
    How can we make it BETTER?
    • Converting the workflow from playing audio clips to playing events. I think this is the way to go, as it eases the development for both the sound designer and the programmer. And of course, this would break all the existing projects ;)
    • Tools: although good enough for some tasks, it is way behind other existing (free!) tools. Editing those events is the key.
    Thanks for asking, and hope this helps!
     
    Last edited: Jun 14, 2016
  34. jwinn

    jwinn

    Joined:
    Sep 1, 2012
    Posts:
    88
    Should this thread be pinned? It's getting lost and I think it's important enough to be featured so that it receives the most developer responses. If not then... *bump*
     
  35. willemsenzo

    willemsenzo

    Joined:
    Nov 15, 2012
    Posts:
    585
    How can we make it BETTER?

    Read the audio forums more because there is a wealth of good information out there. Sadly what I don't like is the fact that almost all of that information is given by members of the community, rather than Unity themselves. If there are issues nobody at Unity gives a damn and I have to rely on either myself or other people. For example I opened a thread because I'm experiencing problems with making OnAudioFilterRead work in the editor. Did not get a single reply, and thread got burried under dozens of other threads. Problem is still not solved, I have no clue other than using different audio libraries to do what I need to do. In my opinion that's not really sending the right signals to the community if you basically say: just use middleware and all your problems are gone. I guess that's just the flipside of the coin.
     
  36. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    How is it that middleware solves unity audio bugs? A lot of the middleware software doesn't change the audio engine, but builds on top of it (ours included), so bugs inherent in the foundation would still be unsolvable in most cases.
     
  37. willemsenzo

    willemsenzo

    Joined:
    Nov 15, 2012
    Posts:
    585
    I'm using NAudio and it doesn't intervene with Unity's audio engine. How middleware can solve Unity bugs? It can't, but by using different framework you can work around the limitations of the current audio engine, which is perfectly fine by me.
     
  38. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    Limitations sure, that does make sense. I thought you were talking about bugs.
     
    Last edited: Jul 25, 2016
  39. willemsenzo

    willemsenzo

    Joined:
    Nov 15, 2012
    Posts:
    585
    I was talking about how to make things better. In reality I don't like to use middleware if it's not absolutely needed, but if I don't see any other option I have to make a decision.
     
  40. HeyItsLollie

    HeyItsLollie

    Joined:
    May 16, 2015
    Posts:
    68
    Hope it's not too late to contribute to this thread!

    Who are you?

    A multimedia producer (art, animation, and music) making a transition into game development. Still building prototypes and learning the craft.

    What kind of game are you trying to build or would like to build?
    I'm mostly interested in platformers, console-style JRPGs, and rhythm games.

    How does Audio fit into that? What use-cases do you have?
    In the case of platformers and JRPGs, music is largely responsible for setting the mood of an area or scene. Sometimes this requires dynamic music that evolves as the player progresses, or layered music where elements of the song can be swapped with alternate layers/stripped away as the player performs different actions.

    For many rhythm games, music IS the game. Systems are required to keep the audio in sync with gameplay.

    What are the GOOD things about the Audio system that you like?
    I really like the Audio Mixer and its effects. As someone with prior experience with music and audio, I find Unity's mixer to be fairly intuitive and easy to use, though somewhat more basic. I hope that the mixer becomes more capable with the introduction of Unity's director sequencer, allowing for features like volume/effects automation.

    What are the BAD things about the Audio system that you dislike?
    As far as I can tell, Unity's support for tracker modules has always been fairly limited, almost as if it were abandoned soon after its introduction. Any benefit of using tracker modules for music is thrown out the window, as there's no programmable access to core features that actually make tracker modules a viable audio format.

    And, because Unity's support for tracker modules is so limited, the only real options are for developers to either create/plug in their own solution for playing tracker modules, or to render out audio streams instead. Preparing long audio streams for games can be a time-consuming process in itself, especially if the audio needs to loop with intro/ending segments, or if it needs to be layered in any way.

    How can we make it BETTER?
    Having researched Unity's support for tracker modules for a while, and in trying to find implementations or solutions by other Unity developers, I've come to the conclusion that it's floundering support is a result of its original implementation, which failed to provide enough to make the audio format viable for developers to use in the first place. Poor implementation discourages developers from using tracker modules, which turns into a lack of interest from other developers to pursue modules over other more-functional formats, which discourages Unity from furthering development to support tracker module formats. And that totally sucks!

    So, ways to improve tracker module support.
    • Programmable access specifically for tracker modules. Suggestions for this go as far back as 2010, when tracker module support was originally announced for Unity 3.0.
    • The ability to seek to specific patterns/beats/lines in a tracker module. Much like how audio playback can be set with AudioSource.time and AudioSource.timeSamples, tracker modules would greatly benefit from the ability to begin playback from (or seek to) a specific Pattern, Beat, or Line.
    • Looping is busted. Sadly, it's been busted for years, and it is a common (if quiet) reason for why developers interested in tracker modules forgo them for other audio formats.
    • Read-only access to tracker module info and parameters. Information like the song's default/current tempo, the number of patterns in a module, a specified pattern's length, the number of lines per beat, and channel panning/volume if available. Tracker modules tend to have fields for the title and author as well.
    • Give tracker modules per-channel access to Unity's audio mixer. One of the biggest benefits that tracker modules offer is that, unlike audio stream formats like WAV and OGG, tracker modules can have a variable number of channels — up to 32 channels for .MOD modules, and per-channel panning/volume controls for .XM and .IT modules. Giving tracker modules per-channel access to Unity's audio mixer would provide developers with total control over how their music is presented in-game, offering volume control and DSP effects to mix music live, bypassing the need to render and prepare layered music prior to adding it to a project, and without ever having to worry about individual channels falling out of sync — All while enjoying the benefits that Unity's own documentation touts: high-quality audio with low file sizes.
    I understand that any level of support for tracker modules is more likely to be a pipe-dream at this point, having gone so long since its implementation without even a mention. But I also strongly believe that Unity's original implementation, along with its failure to fix long-standing issues, never really allowed tracker modules to have much of a chance in the first place. Maybe this could change with a more robust integration.
     
    Last edited: Nov 18, 2016
  41. CarlosBCV

    CarlosBCV

    Joined:
    Jun 21, 2013
    Posts:
    4
    I am a programmer, currently developing a musical game.
    In general, the sound system of unity is not bad because it lets you expand it when you need it, but it has some IMPORTANT LACKS.

    THE MOST IMPORTANT, OnAudioFilterRead(float[], int), in unity documentation says "In this way you can use the filter as the audio clip, procedurally generating audio". The problem is that I have no way to synchronize it. I don't know when that buffer will be played.
    That could be fixed by simply adding a third parameter that will be the dsptime when it will be played:
    OnAudioFilterRead(float[] data, int cannels, double dsptime)
    This fix is the most important because without it, OnAudioFilterRead has no sense, because we cannot synchronize it to make our own filters or generators.

    Other thing that will be very useful is what other users said, that you could set the start position and the end position of the loop of an AudioSource.

    There is other thing, less important but a bit annoying. Currently, I have to create a GameObject with an AudioSource for each note that I want to play, will be great if I could play different notes with a single AudioSource, for example:
    AudioSource.PlayScheduledOneShot(double time, float volume, float pan, float pitch)
     
  42. Matexyu

    Matexyu

    Joined:
    Feb 22, 2017
    Posts:
    13
    How can we make it BETTER?
    I'm implementing a software synthesizer on Gear VR device.
    It is REALLY easy to hit CPU limits due to throttling (and possibly due to Samsung becoming paranoid about burning devices).

    I've tried using the Oculus Mobile Utilities to change OVRManager.cpuLevel and OVRManager.gpuLevel, but it's basically useless.

    The very same softsynth works flawlessly in non-Gear VR mode.

    So, in mobile VR mode CPU clock speed is quite slowed down, but at least the amount of cores is more than one.

    So, a very good improvement would be to have Multi-threaded Audio rendering.
    This would probably imply performing on multiple threads / cores:
    1. the OnAudioFilterRead() callbacks
    2. the native mixer channels DSP code.
    More details from Oculus here.
     
  43. LandePlage

    LandePlage

    Joined:
    May 21, 2016
    Posts:
    33
    Who are you?
    Freelance composer and sound designer for games and other projects using Unity.

    What kind of game are you trying to build or would like to build?
    I recently started working on a dynamic music system package.

    How does Audio fit into that? What use-cases do you have?
    I'm using several AudioSource components on top of each other to handle seamless looping within one audio file. The system supports both horizontal resequencing and vertical layering of tracks. To do this, I made my own metadata reader that just reads the bytes inside wave-files directly. I could then have Unity auto-import loop regions with intros and tails, and then a system that loops these and handles queuing of regions, etc.


    What are the GOOD things about the Audio system that you like?

    - It's really easy to get started quickly with fairly advanced stuff like reverb effects and positional audio.
    - The AudioMixer system is easy and powerful. Awesome.
    - Handling different file formats is a breeze. Unity doesn't care whether you throw an .ogg or a .wav at it.

    What are the BAD things about the Audio system that you dislike?
    - We only have AudioSource and AudioClip to work with, and these could be more fleshed out.
    - Different audio file formats have different metadata that can be very useful to read. Cue points can be used to define loop regions. Tempo information for playing musical stingers on the next beat. In fact, you could add any information you'd like and use it however you'd like in-game.

    How can we make it BETTER?

    - Add more metadata reading from audio files. Something like clip.metadata.tempo, or clip.metadata.cuepoint[0].
    - Add a function that returns a waveform texture.
    - Add functions to AudioSource that makes it easier to create dynamic music, with more granular control over scheduled looping. Native handling of "when playback reached sample X, go to sample Y", instead of having to manually check whether audioSource.time is over a given audioSettings.dspTime.
    - Use the suggestions of the other folks in this thread. Lots of good feedback here!
     
    SonicBloomEric likes this.
  44. Awan212

    Awan212

    Joined:
    Mar 1, 2024
    Posts:
    4
    Hey there!
    I'm excited to jump into this discussion about the direction of Audio in our technology. It's always refreshing to see a company actively seeking input from its community.

    So, who am I? Well, I'm an avid gamer with a penchant for immersive experiences. I love diving into worlds that feel alive and dynamic, where every sound contributes to the overall atmosphere.

    As for the kind of game I'd like to build or play, I'm drawn to RPGs and adventure games that transport me to fantastical realms or intriguing landscapes. Think sprawling open worlds filled with mysteries to unravel and quests to undertake.

    Now, let's talk Audio. In my ideal game, Audio is absolutely crucial. It's not just background noise; it's the heartbeat of the world. From the rustle of leaves in the wind to the distant rumble of an approaching storm, every sound should serve to deepen the immersion and pull the player further into the experience. And of course, let's not forget about epic soundtracks that elevate key moments and set the tone for the entire adventure.

    So, what's working well with the Audio system? I appreciate its versatility and the range of options it provides for creating dynamic soundscapes. The ability to layer different sounds and adjust their properties in real-time is a game-changer for crafting truly immersive environments.

    However, there are a few areas that could use improvement. Sometimes the transitions between different audio tracks or effects can feel a bit abrupt, pulling me out of the experience momentarily. And while the system offers a lot of flexibility, it can be a bit overwhelming to navigate, especially for newcomers.

    So, how can we make it better? One suggestion I have is to streamline the interface and provide more intuitive tools for managing audio assets. It would also be great to see more built-in options for creating dynamic soundscapes that react to player actions and environmental changes.

    Overall, I'm excited to see where the Audio system goes from here. With a few tweaks and enhancements, I have no doubt it will continue to enhance the gaming experience for players everywhere.

    Also, if anyone's interested in delving deeper into the world of immersive sound design, I recently stumbled upon a fantastic resource on Dua Qunoot that might provide some inspiration.

    Looking forward to hearing everyone's thoughts and ideas!
    Cheers!