Procedural native audio synthesis with OnAudioFilterRead

I'm trying to use the Unity3.5 audio buffer access to integrate a synthesis engine in C to produce audio.
When I try to do it in the most efficient way, i.e. passing the audio buffer pointer, I get crashes in what looks like FMOD memory management.

Maybe I'm not allowed to pass arrays by reference like that, since the (hoped for) memory pinning seems to invoke some runtime code that is not allowed in the audio thread or so? I only have shaky knowledge about the Mono C# to C interface, p/invoke, marshaling, and all that.

the C# audio filter script on the audio source:

Code (csharp):

1. [DllImport ("synth-lib")]
2. private static extern void synth (IntPtr self, int nsamp, float[] data, int channels);
3.  
4. void OnAudioFilterRead (float[] data, int channels)  {
5.         synth(synth_struct, data.Length, data, channels);
6. }

BTW, passing null instead of data avoids the crash.

the C code prototype:

Code (csharp):

1. void synth (mysynth_t *self, int nsamp, float *data, int channels);

It also crashes when the synth routine does nothing at all.

more details and a crash log over on http://answers.unity3d.com/questions/206870/procedural-native-audio-synthesis-with-onaudiofilt.html
there is a sample project with bug report http://fogbugz.unity3d.com/default.asp?438781_9aqisqrtn9qbop5k

 

Just dropping a note to say I'm looking into this as well, but my knowledge of marshalling is equally shaky. Currently going through this: http://www.mono-project.com/Interop_with_Native_Libraries, but you've probably already found that.

Out of interest, what synthesis engine are you using? Something custom-made? I'm trying my hand at SuperCollider.

 

It seems that one copy is unavoidable: I made this work with the first method described in http://answers.unity3d.com/questions/34606/how-do-i-pass-arrays-from-c-to-c-in-unity-if-at-al.html

Code (csharp):

1.  
2. // C:
3. void synth (mysynth_t *self, int nsamp, float **data, int channels);
4.  
5. // C#:
6. [DllImport ("synth-lib")]
7. private static extern void synth (IntPtr self, int nsamp, ref float[] data, int channels);
8.  
9. void OnAudioFilterRead (float[] data, int channels)  {
10.     IntPtr buf   = IntPtr.Zero;
11.    
12.    synth(mysynth, data.Length, ref buf, channels);
13.  
14.     Marshal.Copy(buf, data, 0, data.Length);
15. }
16.  

C has to return a locally allocated buffer, that is then copied. That's ok for pure synthesis, but not for filters!

The second method described there with [In, Out] makes FMOD crash trying to invoke mono_array_to_lparray (see crash log below). However, this method does not crash outside of the audio thread.

It would be good to know from the audio devs if this is intentional, and if there is a more efficient way to pass audio to native code.

@Tinus: we're integrating a custom granular and corpus-based concatenative synth engine

The crashlog:

Code (csharp):

1.  
2. Thread 12 Crashed:
3. 0   libSystem.B.dylib               0x93c6c0ee __semwait_signal_nocancel + 10
4. 1   libSystem.B.dylib               0x93c6bfd2 nanosleep$NOCANCEL$UNIX2003 + 166
5. 2   libSystem.B.dylib               0x93ce6fb2 usleep$NOCANCEL$UNIX2003 + 61
6. 3   libSystem.B.dylib               0x93d086f0 abort + 105
7. 4   com.unity3d.UnityEditor3.x      0x0142cda3 LaunchBugReporter(BugReportMode) + 1139
8. 5   libmono.0.dylib                 0x028864b3 mono_chain_signal + 76
9. 6   libmono.0.dylib                 0x027be019 mono_sigsegv_signal_handler + 498
10. 7   libSystem.B.dylib               0x93c6505b _sigtramp + 43
11. 8   ???                             0xffffffff 0 + 4294967295
12. 9   libmono.0.dylib                 0x02986483 g_log + 41
13. 10  libmono.0.dylib                 0x028d5946 mono_array_to_lparray + 164
14. 11  ???                             0x07b9315c 0 + 129577308
15. 12  ???                             0x35f0e415 0 + 904979477
16. 13  ???                             0x35f0e1f2 0 + 904978930
17. 14  ???                             0x35f0e2c6 0 + 904979142
18. 15  libmono.0.dylib                 0x027c5140 mono_jit_runtime_invoke + 1149
19. 16  libmono.0.dylib                 0x0290d170 mono_runtime_invoke + 133
20. 17  com.unity3d.UnityEditor3.x      0x00678933 AudioCustomFilter::readCallback(FMOD_DSP_STATE*, float*, float*, unsigned int, int, int) + 291
21. 18  fmod_editor.dylib               0x02b0483b FMOD::DSPConnection::getInput(FMOD::DSP**) + 18131
22. 19  fmod_editor.dylib               0x02b044df FMOD::DSPConnection::getInput(FMOD::DSP**) + 17271
23. 20  fmod_editor.dylib               0x02b46df1 FMOD_Memory_Initialize + 47598
24. 21  fmod_editor.dylib               0x02b044df FMOD::DSPConnection::getInput(FMOD::DSP**) + 17271
25.  

 

Hi,
As long as you don't allocate new memory for the buffers in every call, then I doubt the Marshal.Copy will ever be a performance problem. Even if you have to do it twice for filtering. I mean a memcpy is nothing compared to the signal processing you're describing. Have you profiled it?

cheers,

 

Hi Soren, thanks for the reply.

Sure, one copy is not much. I was just wondering if there was a way to avoid it which I didn't know about. (E.g. protecting the C# array from being moved around for the time of the callback and passing the actual pointer into the C plugin --- but that would punch a hole into that "managed memory" hot air balloon =-)

Best...
...Diemo

 

Have a look at the Unity Port of C#Synth project too. Details http://forum.unity3d.com/threads/130104-UnitySynth-full-Xplatform-midi-synth?p=877768#post877768