iPhone microphone

Now that Unity iPhone 1.5 is out, is it possible to get mic input from within Unity? Is it only possible through a custom native plugin? If so, any pointers on where to start?

thanks.

 

The first question I would ask is what are you trying to do? Do you just want to record a short audio clip or are you after speech recognition and T-Pain effects?

 

I'm really just wanting to monitor the mic's input level... such that certain gameobjects in unity respond to how loud the user speaks/whistles/blows/whatever into the mic.

 

OK, so how about a customer object that opens and monitors the wave input stream. It could keep a small buffer that it would use to calculate the magnitude of the audio. The object would have init, shutdown and getMagnatude calls. Something like that?

 

I think I could use this Obj-C code (http://www.mobileorchard.com/tutorial-detecting-when-a-user-blows-into-the-mic/) and call it using Unity iPhone Pro's native plugin ability. But I was hoping there was an easier way.

 

I'd do the plug in and use that code.

 

Randy: give this a try. It's based on Dan's code. You can get the raw mic level, or the mic level with the low-pass filter. I haven't had time to test it, but it does compile.
C#:

Code (csharp):

1.  
2.     [DllImport ("__Internal")]
3.     public static extern void SB_startMicLevel();
4.  
5.     [DllImport ("__Internal")]
6.     public static extern void SB_stopMicLevel();
7.  
8.     [DllImport ("__Internal")]
9.     public static extern float SB_getMicLevel();
10.  
11.     [DllImport ("__Internal")]
12.     public static extern float SB_getMicLowPassFiltered();
13.  

Objective-C++:

Code (csharp):

1.  
2. //
3. //  SBMicLevel.mm
4. //  Unity-iPhone
5. //
6. //  Created by Rob Terrell on 8/26/09.
7. //  Copyright 2009 Stinkbot LLC. All rights reserved.
8. //
9.  
10. #import <Foundation/Foundation.h>
11. #import <AVFoundation/AVFoundation.h>
12. #import <CoreAudio/CoreAudioTypes.h>
13.  
14.  
15. AVAudioRecorder* recorder;
16. float lowPassResults;
17.  
18. extern "C" void SB_startMicLevel()
19. {
20.     NSURL *url = [NSURL fileURLWithPath:@"/dev/null"];
21.    
22.     NSDictionary *settings = [NSDictionary dictionaryWithObjectsAndKeys:
23.                               [NSNumber numberWithFloat: 44100.0],                 AVSampleRateKey,
24.                               [NSNumber numberWithInt: kAudioFormatAppleLossless], AVFormatIDKey,
25.                               [NSNumber numberWithInt: 1],                         AVNumberOfChannelsKey,
26.                               [NSNumber numberWithInt: AVAudioQualityMax],         AVEncoderAudioQualityKey,
27.                               nil];
28.    
29.     NSError *error;
30.    
31.     recorder = [[AVAudioRecorder alloc] initWithURL:url settings:settings error:&error];
32.    
33.     if (recorder) {
34.         [recorder prepareToRecord];
35.         recorder.meteringEnabled = YES;
36.         [recorder record];
37.     } else {
38.         NSLog([error description]);
39.     }
40. }
41.  
42.  
43. extern "C" float SB_getMicLevel()
44. {
45.     if (! recorder) SB_startMicLevel();
46.    
47.     if (recorder) {
48.         [recorder updateMeters];
49.         return  [recorder peakPowerForChannel:0];
50.     }
51.     return -1;
52. }
53.  
54.  
55. extern "C" float SB_getMicLowPassFiltered()
56. {
57.     if (recorder) {
58.         [recorder updateMeters];
59.         const double ALPHA = 0.05;
60.         float peakPowerForChannel = pow(10, (0.05 * [recorder peakPowerForChannel:0]));
61.         lowPassResults = ALPHA * peakPowerForChannel + (1.0 - ALPHA) * lowPassResults;
62.         return lowPassResults;
63.     }
64.     return -1; 
65. }
66.  
67.  
68. extern "C" void SB_stopMicLevel()
69. {
70.     [recorder release];
71.     recorder = nil;
72. }
73.  

 

I played around with the AVAudioRecorder, but it was always saying the output was around -120db regardless of what I was doing (this is on an iPhone with 3.0 installed).

Maybe I need to change the audio session category? I'm not sure the way Unity handles its audio or if it would conflict with that particular class.

 

I tried to build and run the code above, and I can get it to compile and build in Unity, but I get the following error during the linking:

Code (csharp):

1.  
2. Building target “Unity-iPhone” of project “Unity-iPhone” with configuration “Release” — (5 errors)
3.         cd "/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope"
4.     setenv IPHONEOS_DEPLOYMENT_TARGET 3.1
5.     setenv MACOSX_DEPLOYMENT_TARGET 10.5
6.     setenv PATH "/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin"
7.     /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/g++-4.2 -arch armv6 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.1.sdk "-L/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/build" "-L/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope" "-L/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/Libraries" "-F/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/build" -filelist "/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/build/Unity-iPhone.build/Release-iphoneos/Unity-iPhone.build/Objects-normal/armv6/sample.LinkFileList" -mmacosx-version-min=10.5 -Wl,-dead_strip -Wl,-S,-x -miphoneos-version-min=3.1 -framework Foundation -framework UIKit -framework OpenGLES -framework QuartzCore -framework OpenAL -liconv.2 -liPhone-lib -framework AudioToolbox -framework CFNetwork -framework MediaPlayer -framework CoreLocation -framework SystemConfiguration -o "/Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/build/sample.app/sample"
8. Undefined symbols:
9.   "_AVFormatIDKey", referenced from:
10.       _AVFormatIDKey$non_lazy_ptr in MicInput.o
11.   "_AVSampleRateKey", referenced from:
12.       _AVSampleRateKey$non_lazy_ptr in MicInput.o
13.   "_AVNumberOfChannelsKey", referenced from:
14.       _AVNumberOfChannelsKey$non_lazy_ptr in MicInput.o
15.   "_AVEncoderAudioQualityKey", referenced from:
16.       _AVEncoderAudioQualityKey$non_lazy_ptr in MicInput.o
17.   "_OBJC_CLASS_$_AVAudioRecorder", referenced from:
18.       __objc_classrefs__DATA@0 in MicInput.o
19. ld: symbol(s) not found
20. collect2: ld returned 1 exit status
21.           "_AVFormatIDKey", referenced from:
22.               _AVFormatIDKey$non_lazy_ptr in MicInput.o
23.           "_AVSampleRateKey", referenced from:
24.               _AVSampleRateKey$non_lazy_ptr in MicInput.o
25.           "_AVNumberOfChannelsKey", referenced from:
26.               _AVNumberOfChannelsKey$non_lazy_ptr in MicInput.o
27.           "_AVEncoderAudioQualityKey", referenced from:
28.               _AVEncoderAudioQualityKey$non_lazy_ptr in MicInput.o
29.           "_OBJC_CLASS_$_AVAudioRecorder", referenced from:
30.               __objc_classrefs__DATA@0 in MicInput.o
31.         ld: symbol(s) not found
32.         collect2: ld returned 1 exit status
33. Build failed (5 errors)

There is an EntryPointNotFoundException given in Unity using the code, but I also encountered that with a basic plugin that printed the number 5, and that program built and ran in the phone regardless of the exception, so I'm not sure that its related. (I made that basic plugin just to make sure that I knew how to complete the entire process). Any ideas on what's giving me the issue?

 

I put in the AVFoundation and CoreAudio frameworks into the project, and I was able to get the errors to go away, only to get it replaced by a new one:

Code (csharp):

1.  
2.  
3. ld: ldr 12-bit displacement out of range (5716 max +/-4096) in _AudioQueueAddPropertyListener$stub in _AudioQueueAddPropertyListener$stub from /Users/Dev1/Downloads/Penelope-1/Tutorial Project/Penelope/build/sample.app/sample
4.  
5.  

After some searching, this topic seems relevant:
http://forum.unity3d.com/viewtopic.php?p=202707

Has there been any solution to this issue? If not, are there any workarounds? I need to do some quick prototypes with mic input for my company. Thanks!

 

I'm currently a Unity iPhone Basic user...

Either thru the enhancement pack or not, can I do mic recording?

Will I need iPhone Advanced?
That's a big jump for one feature (small audio recording)

Help?


Sumner

 

I have Unity iPhone Advanced, who can tell me how to put mic plugin into it?
Pls help me. :cry: :cry: :cry:

 

Check out the plugin example project to get a feeling on how these things work and relate to other things.

 

Hi there, I've got the code from this thread working (thanks for the post btw bliprob!) - however, I hit the same -120 issue as Tetrad.

I discovered this was due to the AudioSession.category not being AVAudioSessionCategoryPlayAndRecord, as Unity (understandably) sets it to the best suited to the target iPhone hardware OS.

So, I set it to PlayAndRecord, and voila - microphone now working!

But, to keep Unity happy, I believe it should be set back to whatever Unity set it to in the first place... so I do that... and it crashes the app, instantly when I call my modified SB_stopMicLevel(). I can't get a debugger connection to stay up long enough to try and diagnose - does anyone have any ideas?

Here's my extra bit o' code... (I'm a n00b at Obj-C btw...)

As a file global scope variable...

Code (csharp):

1.  
2. NSString* audioCategory;
3.  

In SB_startMicLevel() - just before creating the recorder.

Code (csharp):

1.  
2.         AVAudioSession* avSession = [AVAudioSession sharedInstance] ;
3.         audioCategory = avSession.category ;
4.         [avSession
5.          setCategory: AVAudioSessionCategoryPlayAndRecord
6.          error: &error] ;
7.  

Inside SB_stopMicLevel()

Code (csharp):

1.  
2.         NSError *error ;
3.    
4.         [[AVAudioSession sharedInstance]
5.          setCategory: audioCategory
6.          error: &error] ;
7.  

[Edit] BTW - It also crashes if I don't reset the AudioSession.category...

 

hey, nice catch!

Going behind Unity's back with the audio session is a sure source of crashing, I've found. You should probably use the Unity call:

Code (csharp):

1.  
2. void UnitySetAudioSessionActive(bool active);

Call it with NO before you start recording, and YES when you're done. That might work without crashing.

 

I take no credit - Tetrad was on the case there before me

And as he suspected, Unity no likee...

Your catch bliprob is a better one yet - but sadly, it still dies. I'm setting it to false before altering the AudioSession and creating the recorder, and enabling it after restoring the session. I've also tried not restoring the session and enabling unity's API... but still no joy

I think the two things do not sit well together outside of Unity... we could do with a Unity API to get at the microphone. Has anyone else managed to get the Mic working with Unity?

 

Hi,

is there any solution for this problem? Or is it just not possible to record audio with unity?

Please help!

 

Hi, i just wanna use microphone to know when user blows. Is that possible from unity? I have Unity iphone advanced. Would it be available on Unity3?

 

I doubt it will be included in Unity 3. It's possible but you're going to have to dig into ObjectiveC and plug ins.

 

I´ll try to work with plugins. I have the C# code posted here on Unity. I should now include the xcode part on classes, right? Is it possible to run it from unity or it needs to be run on iphone/ipad?

 

Objective C plug in's only run on the device itself. Do some searching on this forum for iPhone plug in examples and be sure to read the Unity iPhone documentation about creating plug ins.

Once you've read the documentation, the first place to start would be to create a simple Objective C application so you can test the mic without involving Unity.

 

How about the plugin on this thread?

 

It might be a nice place to start but it doesn't look like anyone has gotten it going.

 

I have this code on Unity C# file:

Code (csharp):

1. using UnityEngine;
2. using System.Runtime.InteropServices;
3.  
4. public class PruebaAudio : MonoBehaviour {
5.    
6.     float NIVEL;
7.  
8.     // Use this for initialization
9.     void Start () {
10.        
11.         SB_startMicLevel();
12.        
13.        
14.        
15.        
16.        
17.    
18.     }
19.    
20.     // Update is called once per frame
21.     void Update () {
22.        
23.         Debug.Log(SB_getMicLevel().ToString());
24.    
25.     }
26.    
27.     [DllImport ("__Internal")]
28.    public static extern void SB_startMicLevel();
29.  
30.    [DllImport ("__Internal")]
31.    public static extern void SB_stopMicLevel();
32.  
33.    [DllImport ("__Internal")]
34.    public static extern float SB_getMicLevel();
35.  
36.    [DllImport ("__Internal")]
37.    public static extern float SB_getMicLowPassFiltered();
38. }

This:

Code (csharp):

1.  
2. //
3. //  SBMicLevel.mm
4. //  Unity-iPhone
5. //
6. //  Created by Rob Terrell on 8/26/09.
7. //  Copyright 2009 Stinkbot LLC. All rights reserved.
8. //
9.  
10. #import <Foundation/Foundation.h>
11. #import <AVFoundation/AVFoundation.h>
12. #import <CoreAudio/CoreAudioTypes.h>
13.  
14. @implementation SBMicLevel
15.  
16.  
17. AVAudioRecorder* recorder;
18. float lowPassResults;
19.  
20. extern "C" void SB_startMicLevel()
21. {
22.     NSURL *url = [NSURL fileURLWithPath:@"/dev/null"];
23.    
24.     NSDictionary *settings = [NSDictionary dictionaryWithObjectsAndKeys:
25.                               [NSNumber numberWithFloat: 44100.0],                 AVSampleRateKey,
26.                               [NSNumber numberWithInt: kAudioFormatAppleLossless], AVFormatIDKey,
27.                               [NSNumber numberWithInt: 1],                         AVNumberOfChannelsKey,
28.                               [NSNumber numberWithInt: AVAudioQualityMax],         AVEncoderAudioQualityKey,
29.                               nil];
30.    
31.     NSError *error;
32.    
33.     recorder = [[AVAudioRecorder alloc] initWithURL:url settings:settings error:&error];
34.    
35.     if (recorder) {
36.         [recorder prepareToRecord];
37.         recorder.meteringEnabled = YES;
38.         [recorder record];
39.     } else {
40.         NSLog([error description]);
41.     }
42. }
43.  
44.  
45. extern "C" float SB_getMicLevel()
46. {
47.     if (! recorder) SB_startMicLevel();
48.    
49.     if (recorder) {
50.         [recorder updateMeters];
51.         return  [recorder peakPowerForChannel:0];
52.     }
53.     return -1;
54. }
55.  
56.  
57. extern "C" float SB_getMicLowPassFiltered()
58. {
59.     if (recorder) {
60.         [recorder updateMeters];
61.         const double ALPHA = 0.05;
62.         float peakPowerForChannel = pow(10, (0.05 * [recorder peakPowerForChannel:0]));
63.         lowPassResults = ALPHA * peakPowerForChannel + (1.0 - ALPHA) * lowPassResults;
64.         return lowPassResults;
65.     }
66.     return -1;    
67. }
68.  
69.  
70. extern "C" void SB_stopMicLevel()
71. {
72.     [recorder release];
73.     recorder = nil;
74. }
75.  
76. @end

As SBMicLevel.mm

And this as SBMicLevel.h

Code (csharp):

1.  
2. //
3. //  SBMicLevel.mm
4. //  Unity-iPhone
5. //
6. //  Created by Rob Terrell on 8/26/09.
7. //  Copyright 2009 Stinkbot LLC. All rights reserved.
8. //
9.  
10. #import <Foundation/Foundation.h>
11. #import <AVFoundation/AVFoundation.h>
12. #import <CoreAudio/CoreAudioTypes.h>
13.  
14. @interface SBMicLevel : NSObject
15. {
16.     AVAudioRecorder* recorder;
17.     float lowPassResults;
18. }
19.  
20.  
21.  
22. - (void) SB_startMicLevel();
23.  
24.  
25.  
26. - (float) SB_getMicLevel();
27.  
28.  
29.  
30. - (float) SB_getMicLowPassFiltered();
31.  
32. - (void) SB_stopMicLevel();
33.  
34.  
35. @end

I don´t see any error, scene loads but there´s not any Debug on console. Where´s the problem? Where i´m wrong?

 

I've got your modifications to work

Where was your app crashing?

OS4 Unity 3 Pro

 

Hey. Any chance you could post msparrow's mods as you got them to work? I just tried the c#/c++ combination posted and no luck...maybe you can post the plugin binary and c# you used to invoke?

I'm trying to get very simple audio in working...

david

 

Would you care sharing the exact steps you followed to get it to work?
And can you answer if this works when developing on Windows as well?

 

I too just got this working.

1. Follow the code from Bitomule

2. if you dont want to view the readout in the XCode console then add a GUIText lable to your Unity interface and output the result there instead of to Debug.Log

3. In Xcode add the 3 frameworks (Foundation, CoreAudio, AVFoundation)

That will now compile (or should) but you will get a constant -120 reading

Edit SBMicLevel.mm and under the line

Code (csharp):

1. extern "C" void SB_startMicLevel() {

Add this

Code (csharp):

1.  
2. NSError *setCategoryError = nil;
3.    
4.     [[AVAudioSession sharedInstance]
5.      setCategory: AVAudioSessionCategoryPlayAndRecord
6.      error: &setCategoryError];
7.     if (setCategoryError) {
8.         NSLog([setCategoryError description]);
9.     }

{

Now build and run

I was getting a readout from 0 to -50 depending on how loud I shouted into the Mic.

Anim
Unity iPhone Basic

 

Would it be possible, in anyway, to compile this plug-in on a windows machine?

 

No, its an iOS plugin and as such requires XCode and the iOS SDK thus intel mac on osx 10.6

 

Possibly. You might be able to on a Hackintosh (A pc running OSX) look it up. But be aware its not supported by apple, you would probably need to jail break your iphone to install the app on it and it might still have issues as its mostly community managed not Apple.

 

and its illegal in addition to all these points as any hackintosh

 

I would have thought its only illegal if you get a pirate copy of Mac OSX rather than buying it?

 

Awesome guys, I got it working as a non-programmer, more or less!

It appears to get to 0 pretty quickly though - is that because of the code or the hardware? (yes, I love to ask stupid questions...)

 

@Anim Thanks for the snippet

 

Looks awesome but wont work for me :/
Can someone help?

Nothing comes in in the console after applicationDidBecomeActive() :/

 

My bad. After an hour or two of workrounds its ready to rock and roll! Love ya guys!