Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Multi Audio - Split Screen Audio Plugin [Beta]

Discussion in 'Assets and Asset Store' started by Tinus, Jan 10, 2013.

  1. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Ever wanted to create a split-screen multiplayer game with Unity? I bet you have at some point, like many, and found out that Unity doesn't support multiple audio listeners by default. Ouch!

    The MultiAudio plugin helps you solve this problem. Without further ado, let's jump right into a little demonstration:



    Also check out this Webplayer Demonstration!

    Features

    Add an arbitrary* amount of Audio Listeners to your scenes. The most immediate application is for split screen gaming, but it can be used for any situation that requires more than one audible perspective.

    Add reverb and other audio filters.

    Very similar inspectors, scripting API and general workflow to Unity's regular audio components.

    *There are some limits of course, but you can certainly add more than one.

    Platforms

    It's built on top of Unity's audio system so it works on any platform.

    Performance

    A virtual sound heard by two listeners is realized by two real Unity Audio Sources. So if you have two listeners in your scene you can expect twice the performance demand compared to just having a single Audio Source. There's also some bookkeeping overhead, but it's marginal.

    Caveats

    Unity plays a maximum of 128 voices simultaneously. This number is shared for all listeners in your scene, so if you have two listeners, you can only have 64 simultaneously playing sources; 4 listeners means 32 sources, and so on. This means voice stealing and culling become more important! Implementations for these systems are currently not done, but they are on the todo list.

    Since Unity doesn't expose control over audio routing (send/receive, channels, busses) the current implementation of Audio Filters is less elegant than it could be. A separate filter component is needed for each real Audio Source. This is very wasteful, of course. Ideally you would allocate one filter, down-mix all sources that need to go through that filter, and filter them in one pass.

    Here's some feature requests that could use some votes:

    Audio Busses and Multiple Audio Listeners
    Audio Voice Culling
    Audio Polyphony Increase
    Additional Curve Parameters

    Beta Testing

    If you're interested in helping test this package please let me know! Beta testers will also get a discount towards their purchase. :)
     
    Last edited: Jan 21, 2013
  2. Adam-Buckner

    Adam-Buckner

    Joined:
    Jun 27, 2007
    Posts:
    5,664
    This is interesting.

    Thanks for doing this!
     
  3. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Thanks! I just hope I can reach all of the folks who tried to do split screen games with Unity before and ran into this issue. :)
     
  4. wkaj

    wkaj

    Unity Technologies

    Joined:
    May 18, 2012
    Posts:
    50
    Sub-mixing and routing are definitely a priority for future releases of Unity. We are currently working in improving this area.

    Cheers
     
  5. Oliver-Eberlei

    Oliver-Eberlei

    Joined:
    Jun 12, 2011
    Posts:
    110
    The timing is uncanny :) I just ran into this issue and found your thread. I'd be glad to help you out testing the system. I've send you a PM.

    Cheers
     
  6. POLYGAMe

    POLYGAMe

    Joined:
    Jan 20, 2009
    Posts:
    196
    I had this issue with Project Downforce. Definitely keen to beta test!
     
  7. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    @wkaj: That's good to hear! Is that something to expect for a 4.5 release, or is it too early to say?

    @Oliver: Message received, will respond in a bit.

    @POLYGAMe: Can you send me a private message with your email?
     
  8. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    I created a small Webplayer Demonstration so you can hear the MultiAudio plugin working in real time.

    The only thing holding up the beta release is an incompatibility issue with Prefabs, which will likely be resolved this week.

    Thanks to everyone who signed up as a tester so far! I think I have enough, but I could always use a couple more. Feel free to send me a PM if you're interested. :)
     
    Last edited: Jan 17, 2013
  9. wkaj

    wkaj

    Unity Technologies

    Joined:
    May 18, 2012
    Posts:
    50
    I'm interested in how you would ultimately handle effects on the listeners (hence master group effects). If you have multiple listeners, each with a set of unique effects applied to them, how would you imagine this should work aurally?
     
  10. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    @wkaj: As in: if Unity would natively support multiple listeners and per-listener effects, how should it sound?

    (Warning: big post coming up)

    Tools for Managing Multi-Listener Mixing

    First go for the naive approach, just superimpose the listening perspectives and send them to the stereo output. This will not work for aurally intense games, but there's also plenty games for which this should be fine.

    Next, introduce extra tools with which you can control the mix better. How you would use these depends on the game you're making, and how you think it should sound, but there's some possible common patterns. Of the top of my head:

    - Compressor/Limiter/Sidechaining effects on listeners and stereo outputs. It's hard enough to keep aurally dense Unity games from clipping, let alone when you add multiple listeners to the mix. A simple compressor at the end of the signal chain wouldn't be an elegant solution to this problem, but it would be a quick fix. I might try my hand at implementing a Compressor component using OnAudioFilterRead, actually.

    - High Dynamic Range audio. The newer entries in the Battlefield series use this to great effect. Again, it's no magic bullet that automatically cures your mix. But the HDR paradigm does seem like a powerful tool to help you organize dense and very dynamic mixes. Culling and prioritization are handled elegantly this way, which would no doubt come in handy for splitscreen setups as well. Caveats: DICE use custom software (not sure if FMOD could be juryrigged for this), and demands on audio asset creation become heavier.

    - Allow sound artists to prioritize sounds, and have some kind of mixer that manages sound priority between listeners. If player 2 really, really needs to hear this rocket that's about to hit him, perhaps you can duck the sound of player 1.

    Per-Listener Effects and General Audio Engine Thoughts

    Your question also started me musing about hacking together per-listener effects in the MultiAudio package, and whether it can be done:

    With the current audio implementation in Unity the options for this plugin aren't great.

    The pessimistic answer

    Can't be done. It's an inherent limitation to the way Unity audio currently works. Filter components cannot be added in such a way that only one virtual listener hears their effects. No actual sound passes through the virtual listeners, so the only place filters can be added is on the real listener.

    The failed answer

    Finally, I tried hacking OnAudioFilterRead to get buffers from other audio sources, downmix them, filter them, and play them back. The problem was that OnAudioFilterRead timing didn't sync up with the buffers gotten from the other sources. That, and even this down-mixing trick did work, the other sources would still be heard.

    The unfeasible answer

    You could, for each filter added to a virtual listener, add a real filter to each real source that it hears. This would work, but the implementation would be quite inelegant. Instead of using one filter instance to filter all affected source (which could even be optimized by first down-mixing those sources to a single buffer), you have to use many instances that all do exactly the same thing.

    It would actually make sense to do this if the current filters were affected by spatial properties (where they are in the scene with regards to audio sources), but they're simple stereo filters. An exception might be the Reverb Zone effect, but I suspect it just tweaks dry/wet ratio based on the specified falloff. The actual DSP doesn't need any spatial context to function.

    Another problem here is that all the effects in this supposed chain all get stacked on the objects they affect. Consider this example audio signal chain: source->lowpassfilter->distortionfilter->listener->chorusfilter->reverbfilter. If you simulate the following signal chain with this approach you had better be sure the filter objects were added in the right order. Now consider dynamic inserting and removing effects dynamically. Order is important here, but dynamically managing the ordering of components on a game object not something Unity does well, since it didn't make sense to do so before post processing and audio effects were introduced.

    This highlights an important problem: unity game objects and components are not a sufficient metaphor for audio signal chain definition. I realize Unity wants to use the component paradigm for everything, and I'm well aware that there are many parallels between game scene graphs and audio processing graphs. But there's also many differences.

    Sometimes you have tight links between the two (this object should play this sound), but many times you don't (this filter should effect all gunshot sounds for 3 seconds, starting now.)

    Audio functionality should be represented in the scene graph when it where it makes sense, but managed in another way when it doesn't.

    Anyway, this leads to some things Unity could do.

    The iffy answer


    * Unity audio filters expose a Filter(float[] samples, int channels) method.
    * Unity audio sources expose expose a version of GetOutputData(float[] samples, int channel) that gets you the next synchronous buffer to play. They also get the option to not produce audible sounds themselves. They become a type of Unit Generator.


    This way, in an AudioSource's OnAudioFilterRead you could then call GetOutputData on several other sources, manually down-mix them, then route the stereo result through one filter.

    This would work, but it's hacky

    * For design reasons: Unity's audio engine still has internal audio routing that you can't touch, but your hijacking it, patching things together in unexpected ways without its knowledge.
    * For performance reasons, audio processing is something you would want to do in C++. Not C#.


    The ideal answer

    Unity implements a native, much more flexible, scriptable audio routing system, and none of this is a problem anymore. :)

    Audio Scripting could be done through the currently available languages, and graphical editors akin to visual shader editors.


    It just occurred to me that the Fabric plugin by Tazman-Audio really highlights how much we desire better mixing control. They essentially implemented fake mixing. The interface acts as if you are mixing, and it does a whole lot of legwork under the hood to simulate the desired effect, but it doesn't actually downmix.


    Anyway, thank you so much for listening. I'm really curious to hear the thoughts you Unity guys have on all these things. :)

    Oh by the way, here's another thread I started with some grand speculation on game audio technology and workflow: Singularity - A flexible general purpose audio engine
     
  11. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Ok, the first Beta version is released! :D

    If you applied as a beta tester you should now find a PM sitting in your forum inbox containing all the details.

    Or if you would like to apply for the beta you can still do so. Just send me PM and I'll set you up!
     
  12. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Received good feedback from MMax.

    I use DotNet style property naming for all the code in the plugin, meaning the first letters of all names are capitalized. This makes converting existing code a less than trivial chore, and coding in Javascript without extensive auto-completion support is also more cumbersome.

    Should I change the property naming back to Unity style (e.g. audio.volume instead of audio.Volume)?

    Apart from that there's some good old bugs to iron out.

    Thanks MMax!
     
  13. wkaj

    wkaj

    Unity Technologies

    Joined:
    May 18, 2012
    Posts:
    50
    Hi Tinus,

    Thanks for the detailed feedback! Better and more flexible mixing is big priority at the moment and we hope to alleviate a lot of your concerns with upcoming features. Spatialization is still an interesting problem, perhaps disabling things like 3D panning and Doppler in multi-listener environments would be an acceptable compromise?

    Cheers
    W
     
  14. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Hey wkaj,

    That's good to hear! Again I'm very curious to hear anything about the roadmap; whether features will make it into 4.0, 4.5 or 5.0, but I understand if you can't comment on any of that. :)

    Regarding the disabling of 3D panning and Doppler, wouldn't that just mean trading one set of spatial sound properties for another? It might work for some specific sounds and functions, but in many cases I don't think you could easily make that compromise.


    Oh! Before I forget: please address the current AudioSource scripting interface.

    - Currently you cannot modify the volume rolloff curve through scripting, not even through reflection.

    Using SerializedProperties works of course, but that means foregoing the ability to make in-game changes to the rolloff curves. Admittedly there are not that many use cases where you would want to do this, but it does prevent you from dynamically adding a component and initializing the roloff curve property with a shared template value. For example, this puts limits on how the architecture of my MultiAudio plugin works and forces the framework to do a whole bunch of internal state management in the editor when adding/removing virtual audio components, where otherwise that state initialization could just be deferred until the game launches. In fact, 4/5th of the code is now purely there to do this state management, and it's still not robustly handling all possible use cases. Were it possible to modify the rolloff curve properties in the game runtime all of this code would be redundant.

    - Rolloff concept is not properly encapsulated

    Throughout the code common properties of what constitutes a rolloff become clear: A mode toggle (constant, linear, logarithmic, custom), different editor frontends for each mode, and an actual animation curve containing this data. You could encapsulate this in a serialized Rolloff datastructure and reuse it for Volume, Spread, Cuttoff Frequency and any other properties that you want to modulate based on distance.

    However, the implementation for these properties is different on a case-by-case basis. Some properties are separated between a float property to define a constant value, and a curve property to define linear, log or custom curves. Other properties don't have this float property, but just use an animationcurve with a single key in it to handle the constant value case (better).

    Rolloff properties for filters are edited in the AudioSource inspector too, which can make for even weirder code.

    Basically, I've seen the code for the AudioSource inspector, and it wasn't pretty.


    I've created my own rolloff encapsulation, with specific implementations where the details are different, and it cleans up the code so much! And while the inspector isn't yet as neat as Unity's own AudioSource, it does show how the encapsulated rolloff datatype becomes easily reusable:



    - Internal property naming is inconsistent

    Code (csharp):
    1.  
    2. m_audioClip
    3. m_PlayOnAwake
    4. m_Volume
    5. m_Pitch
    6. Loop
    7. Mute
    8. Priority
    9. DopplerLevel
    10. MinDistance
    11. MaxDistance
    12. Pan2D
    13. rolloffMode
    14. BypassEffects
    15. rolloffCustomCurve
    16. panLevelCustomCurve
    17. spreadCustomCurve
    18.  
    This means you can never just guess a property name when you want to access it through reflection or wrap it in a SerializedProperty (got these through DotPeek and a routine that iterates through and prints available members on a SerializedObject). I've noticed this consistency problem in several parts of the Unity codebase. Of course this appears to be private, under the hood stuff. But in my use case I'm accessing it as if its part of the public API, by necessity. Does Unity have a company-wide coding standards policy?

    Again, thanks for listening! If any of this seems a little harsh, that's only because I deeply care about Unity. :)
     
    Last edited: Feb 5, 2013
  15. Brad-Lee

    Brad-Lee

    Joined:
    May 26, 2013
    Posts:
    4
    I would be very interested to beta test this for you. I can see many cool applications for this that I want to try, such as emulating the 'cetera algorithm' for games!

    https://www.youtube.com/watch?v=pgeFdOayeaw <- cetera algorithm if anyone is unfamiliar
     
  16. OhiraKyou

    OhiraKyou

    Joined:
    Mar 27, 2012
    Posts:
    259
    If you're still working on this, I'd be very interested in giving it a shot. I'm a huge advocate of split screen on PC, and I'm developing a racing game with split screen support that would benefit significantly from more listeners. I'd certainly be interested in buying it when it's finished as well; it's something I would use often.
     
  17. northernthumb

    northernthumb

    Joined:
    Jan 2, 2013
    Posts:
    6
    This thread seems to have gone a bit quiet, but if you're still looking for Beta testers Tinus I'd be happy to help. We're also looking at split screen audio solutions, but there's not much point in re-inventing the wheel if your plug-in does what we need. Thanks.
     
  18. Autonoma

    Autonoma

    Joined:
    Jul 21, 2013
    Posts:
    92
    Tinus - sent you a PM. Didn't know if you got an email notification or not.
     
  19. epsilonstudios

    epsilonstudios

    Joined:
    Feb 20, 2012
    Posts:
    18
    Hi Tinus, I sent you a PM to get on this beta.
     
  20. CaoMengde777

    CaoMengde777

    Joined:
    Nov 5, 2013
    Posts:
    813
    how is this going? is there still any version available? im interested!
     
  21. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Hey CaoMengde777!

    Given the recent surge of interest in this plugin I'll be working on the beta next week. Send me a private message with your contact info, and you're in the beta. :)

    (Also note: I am finally receiving email notifications for PMs again. Yeah, oops!)
     
  22. epsilonstudios

    epsilonstudios

    Joined:
    Feb 20, 2012
    Posts:
    18
    Hey I sent you a private message, Not sure if you got it.
     
  23. emergki

    emergki

    Joined:
    Oct 15, 2007
    Posts:
    421
    Hi Tinus,

    I'm very interested in it.
     
  24. F.Salka

    F.Salka

    Joined:
    Dec 13, 2013
    Posts:
    43
    wait what?? unity can play a maximum of 128 voices simultaneously? are you sure about this? isn't the polyphony limit is 32? does this plug-in allows you to play more than 32 voices at the same time?

    I am definitely interested in this plug-in!
     
  25. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    This is great! Do you know if multiple audio listeners will be supported in Unity 5 without plugin? If not, I'm definitely interested to try this.
     
  26. 0tacun

    0tacun

    Joined:
    Jun 23, 2013
    Posts:
    245
    This plugin sounds very promising! How is the progress?
     
  27. uniphonic

    uniphonic

    Joined:
    Jun 24, 2012
    Posts:
    130
    Hmm... since I don't see any replies... If you're just looking for more voices, know that G-Audio ( http://www.g-audio-unity.com/ ) supports more than 32 voices, and as far as I know it's only limited by the hardware performance. I think the author said he's mixed over 100 piano samples simultaneously.

    I'm using G-Audio, and highly recommend it. It takes the audio capabilities of Unity to a whole new level, enabling things like generative music, and even granular synthesis. BTW, I have no affiliation.

    Cheers,
    Jacob
     
  28. seon

    seon

    Joined:
    Jan 10, 2007
    Posts:
    1,441
    Is anything happening with this? Is it finished or still in beta? How can we get access to it?
     
  29. Vectrex

    Vectrex

    Joined:
    Oct 31, 2009
    Posts:
    267
    Just bumping this as I think it's important. I mean, what console game doesn't have split screen multiplayer?
    Unity 5 I've heard doesn't support multiple listeners, even though it's a whole new audio system :|
     
    0tacun likes this.
  30. highlyinteractive

    highlyinteractive

    Joined:
    Sep 6, 2012
    Posts:
    116
    Is there any progress on this?

    I would really like to find a way to output sound to separate channels - surround or quadrophonic for example. Currently this is impossible in Unity, but it sounds like this plugin might be a solution.
     
  31. FireGreeks

    FireGreeks

    Joined:
    Jan 14, 2017
    Posts:
    5
    Hello there,

    I know I am REALLY late (like 6 years) but if you still have this package, I think it would help a great number of people (including me) and save them a lot and effort. Your idea was really smart, and it seemed perfectly executed. So what I am hoping for even if you, unfortunately, haven't finished polishing this asset (I couldn't care less if it's finished or not), it would be greatly appreciated if you would post a link to download this package.

    I REALLY hope from hearing from you @Tinus

    Have a great day.

    PS: If you don't see this post (which I hope you will 6 years later), I might steal your idea and try to implement it in my project ("couldn't be that hard"... I think I will fail miserably, :) yours just seemed perfect ;) )

    Cheers
     
  32. FireGreeks

    FireGreeks

    Joined:
    Jan 14, 2017
    Posts:
    5
    Nevermind, I got a (much simpler) version of your asset. No need to hassle. It took some time, but I managed. Thank you for the idea of virtual listeners and audioSources :)

    Cheers
     
    Tom507 likes this.
  33. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    437
    Glad to hear you figured out your own approach! Heh, I remember the code for this system existing of only about 3 crucial lines that did the transformation of virtual listener space to unity listener space, and the rest was bookkeeping.

    I'll see if I can find the original source, I must have it lying around somewhere.
     
  34. haxior

    haxior

    Joined:
    Sep 17, 2020
    Posts:
    3
    Hey Tinus, any luck finding the source code?