Blow detection (Using iOS Microphone)

Discussion in 'Developer Preview Archive' started by Logistic Win, Jan 5, 2012.

  1. Logistic Win

    Logistic Win

    Member

    Joined:
    Apr 4, 2009
    Messages:
    107
    Anyone out there have any working scripts to detect Microphone Input levels for iOS? I need to simply detect how hard the player is blowing in to the mic. Anyone? Anyone have a good idea?
  2. Logistic Win

    Logistic Win

    Member

    Joined:
    Apr 4, 2009
    Messages:
    107
  3. steego

    steego

    Member

    Joined:
    Jul 15, 2010
    Messages:
    180
    There's microphone recording in 3.5, but I don't know if it works with iphone (see Microphone class in docs). It also looks like you have to record a clip before you can analyze it. If that doesn't work for you I guess you'd have to make your own plugin to interface with the iphone microphone.

    Each sample in the recorded audio will correspond to the sound pressure level at a specific time, so just look for high values to detect a hard blow. If you need to look for a high sound pressure over time, keep in mind that sound pressure is a logarithmic scale (decibel).
  4. Logistic Win

    Logistic Win

    Member

    Joined:
    Apr 4, 2009
    Messages:
    107
    One of the new features in 3.5 is iPhone microphone support... has anyone done any work to detect audio input from Microphones before with Unity?
  5. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Since I read your last reply I have been look for the Microphone class in the docs (for about an hour), but cannot find it. Could you post a link please? Sorry, I feel like this should be simple.
  6. Vectrex

    Vectrex

    Member

    Joined:
    Oct 31, 2009
    Messages:
    149
    I notice that the Microphone functions have a set number of seconds of recording. What if you don't want to store anything but just stream and effect the microphone audio?
  7. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    847
  8. soren

    soren

    New Member

    Joined:
    Feb 18, 2008
    Messages:
    123
    From the top of my head:
    Code (csharp):
    1. {
    2.   clip.GetOutputData(...); <-- the input levels
    3. }
  9. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    In theory soren's example should work, however it's not so easy.

    Microphone.Start() records the input data to a clip (for a set amount of seconds), and it's AudioSource object (not the clip itself) can be analysed with the GetOutputData function. This is fine if you want to record for a while, and analyse later. However, I am looking to do real-time interaction with the microphone.

    I have tried start and finish recording between update calls (and to run GetOutputData in-between), but I am sure this is a bad approach.


    Also Pitch Poll looks fantastic (especially for only a slim $10), but I am just looking to read frequency, volume, and pitch and that's it. I don't really want ALL of the other great stuff that comes with it..
  10. soren

    soren

    New Member

    Joined:
    Feb 18, 2008
    Messages:
    123
    Right, you should use clip.GetData() instead. Or you could just play the AudioSource when GetPosition>0. Then you get it realtime with GetOutputData. (if you use audio.clip = Microphone.Start() and you play the AudioSource).
    Last edited: Jan 16, 2012
  11. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Ok, this is starting to make more sense. I have got this script so far... (I am not very good at audio, so forgive me for mistakes).

    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. public class MicHandle : MonoBehaviour {
    5.  
    6.   public GameObject display;        // GUIText for displaying results
    7.  
    8.   private int sampleCount = 1024;      // Sample Count.
    9.   private float refValue = 0.1f;    // RMS value for 0 dB.
    10.   private float threshold = 0.02f;  // Minimum amplitude to extract pitch (recieve anything)
    11.   private float rmsValue;           // Volume in RMS
    12.   private float dbValue;            // Volume in DB
    13.   private float pitchValue;         // Pitch - Hz (is this frequency?)
    14.  
    15.   private float[] samples;          // Samples
    16.   private float[] spectrum;         // Spectrum
    17.  
    18.   public void Start () {
    19.  
    20.     samples = new float[sampleCount];
    21.     spectrum = new float[sampleCount];
    22.  
    23.     // This starts the mic, for 999 seconds, recording at 48000 hz. I am unsure of how to avoid this hack.
    24.     audio.clip = Microphone.Start("Built-in Microphone", false, 999, 48000);
    25.     audio.Play();
    26.   }
    27.  
    28.   public void Update () {
    29.  
    30.     // Le big cheese doing its thing.
    31.     AnalyzeSound();
    32.  
    33.     if (display){
    34.       display.guiText.text = "RMS: " + rmsValue.ToString("F2") + " (" + dbValue.ToString("F1") + " dB)\n" + "Pitch: " + pitchValue.ToString("F0") + " Hz";
    35.     } else {
    36.       Debug.Log("Display broke somewhere");
    37.     }
    38.   }
    39.  
    40.   private void AnalyzeSound() {
    41.     audio.GetOutputData(samples, 0); // Get all of our samples from the mic.
    42.  
    43.     // Sums squared samples
    44.     float sum = 0;
    45.     for (int i = 0; i < sampleCount; i++){
    46.       sum += Mathf.Pow(samples[i], 2);
    47.     }
    48.  
    49.     rmsValue = Mathf.Sqrt(sum/sampleCount);          // RMS is the square root of the average value of the samples.
    50.     dbValue = 20*Mathf.Log10(rmsValue/refValue);  // dB
    51.  
    52.     // Clamp it to -160dB min
    53.     if (dbValue < -160) {
    54.       dbValue = -160;
    55.     }
    56.  
    57.     // Gets the sound spectrum.
    58.     audio.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris);
    59.     float maxV = 0;
    60.     int maxN = 0;
    61.  
    62.     // Find the highest sample.
    63.     for (int i = 0; i < sampleCount; i++){
    64.       if (spectrum[i] > maxV  spectrum[i] > threshold){
    65.         maxV = spectrum[i];
    66.         maxN = i; // maxN is the index of max
    67.       }
    68.     }
    69.  
    70.     // Pass the index to a float variable
    71.     float freqN = maxN;
    72.  
    73.     // Interpolate index using neighbours
    74.     if (maxN > 0  maxN < sampleCount - 1) {
    75.       float dL = spectrum[maxN-1] / spectrum[maxN];
    76.       float dR = spectrum[maxN+1] / spectrum[maxN];
    77.       freqN += 0.5f * (dR * dR - dL * dL);
    78.     }
    79.  
    80.     // Convert index to frequency
    81.     pitchValue = freqN * 24000 / sampleCount;
    82.   }
    83.  
    84. }
    I got the AnalyzeSound() function from here.

    I just gave this a few tests, and it seems to work ideally 4 out of 10 times (6 times no audio data is shown at all, maybe the mic doesn't turn on properly?). There must be something unstable in that code, but I am a little unsure of what it is.
  12. soren

    soren

    New Member

    Joined:
    Feb 18, 2008
    Messages:
    123
    You have to wait for Microphone.GetPosition > 0. Then it should work all the time.
  13. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Ah I see!

    I was getting stuck because calling audio.Play() every frame seems to stop the GetOutputData from working. I will have play around with things and post back any success / better code I have.

    Thanks dude :)
  14. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    So now I have a (slight) new problem.

    It is correct that waiting for Microphone.GetPosition > 0 ensures it never fails. However, this causes about a 1 second delay between the input and output of that clip. Is there some way I can Play() the clip part way through... say at the Microphone position?
  15. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Hello, hello, I have mostly success. Check it out, feedback welcome.

    The code can detect a blow, and won't trigger from a whistle, click, tap, hum etc. It does trigger if I drag my laptop over some material, like the couch, as the sound is very similar. Let me know what you think / tweak anything you see fit.

    An example project where you can blow up a sphere, and see some stats is ready to go if anyone wants it. Here's the code.

    Code (csharp):
    1.  
    2. using UnityEngine;
    3. using System.Collections;
    4. using System.Collections.Generic;
    5.  
    6. public class MicHandle : MonoBehaviour {
    7.  
    8.   private const int FREQUENCY = 48000;    // Wavelength, I think.
    9.   private const int SAMPLECOUNT = 1024;   // Sample Count.
    10.   private const float REFVALUE = 0.1f;    // RMS value for 0 dB.
    11.   private const float THRESHOLD = 0.02f;  // Minimum amplitude to extract pitch (recieve anything)
    12.   private const float ALPHA = 0.05f;      // The alpha for the low pass filter (I don't really understand this).
    13.  
    14.   public GameObject resultDisplay;   // GUIText for displaying results
    15.   public GameObject blowDisplay;     // GUIText for displaying blow or not blow.
    16.   public int recordedLength = 50;    // How many previous frames of sound are analyzed.
    17.   public int requiedBlowTime = 4;    // How long a blow must last to be classified as a blow (and not a sigh for instance).
    18.   public int clamp = 160;            // Used to clamp dB (I don't really understand this either).
    19.  
    20.   private float rmsValue;            // Volume in RMS
    21.   private float dbValue;             // Volume in DB
    22.   private float pitchValue;          // Pitch - Hz (is this frequency?)
    23.   private int blowingTime;           // How long each blow has lasted
    24.  
    25.   private float lowPassResults;      // Low Pass Filter result
    26.   private float peakPowerForChannel; //
    27.  
    28.   private float[] samples;           // Samples
    29.   private float[] spectrum;          // Spectrum
    30.   private List<float> dbValues;      // Used to average recent volume.
    31.   private List<float> pitchValues;   // Used to average recent pitch.
    32.  
    33.   public void Start () {
    34.     samples = new float[SAMPLECOUNT];
    35.     spectrum = new float[SAMPLECOUNT];
    36.     dbValues = new List<float>();
    37.     pitchValues = new List<float>();
    38.  
    39.     StartMicListener();
    40.   }
    41.  
    42.   public void Update () {
    43.  
    44.     // If the audio has stopped playing, this will restart the mic play the clip.
    45.     if (!audio.isPlaying) {
    46.       StartMicListener();
    47.     }
    48.  
    49.     // Gets volume and pitch values
    50.     AnalyzeSound();
    51.  
    52.     // Runs a series of algorithms to decide whether a blow is occuring.
    53.     DeriveBlow();
    54.  
    55.     // Update the meter display.
    56.     if (resultDisplay){
    57.       resultDisplay.guiText.text = "RMS: " + rmsValue.ToString("F2") + " (" + dbValue.ToString("F1") + " dB)\n" + "Pitch: " + pitchValue.ToString("F0") + " Hz";
    58.     }
    59.   }
    60.  
    61.   /// Starts the Mic, and plays the audio back in (near) real-time.
    62.   private void StartMicListener() {
    63.     audio.clip = Microphone.Start("Built-in Microphone", true, 999, FREQUENCY);
    64.     // HACK - Forces the function to wait until the microphone has started, before moving onto the play function.
    65.     while (!(Microphone.GetPosition("Built-in Microphone") > 0)) {
    66.     } audio.Play();
    67.   }
    68.  
    69.   /// Credits to aldonaletto for the function, http://goo.gl/VGwKt
    70.   /// Analyzes the sound, to get volume and pitch values.
    71.   private void AnalyzeSound() {
    72.  
    73.     // Get all of our samples from the mic.
    74.  
    75.     // Sums squared samples
    76.     float sum = 0;
    77.     for (int i = 0; i < SAMPLECOUNT; i++){
    78.       sum += Mathf.Pow(samples[i], 2);
    79.     }
    80.  
    81.     // RMS is the square root of the average value of the samples.
    82.     rmsValue = Mathf.Sqrt(sum / SAMPLECOUNT);
    83.     dbValue = 20 * Mathf.Log10(rmsValue / REFVALUE);
    84.  
    85.     // Clamp it to {clamp} min
    86.     if (dbValue < -clamp) {
    87.       dbValue = -clamp;
    88.     }
    89.  
    90.     // Gets the sound spectrum.
    91.     audio.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris);
    92.     float maxV = 0;
    93.     int maxN = 0;
    94.  
    95.     // Find the highest sample.
    96.     for (int i = 0; i < SAMPLECOUNT; i++){
    97.       if (spectrum[i] > maxV  spectrum[i] > THRESHOLD){
    98.         maxV = spectrum[i];
    99.         maxN = i; // maxN is the index of max
    100.       }
    101.     }
    102.  
    103.     // Pass the index to a float variable
    104.     float freqN = maxN;
    105.  
    106.     // Interpolate index using neighbours
    107.     if (maxN > 0  maxN < SAMPLECOUNT - 1) {
    108.       float dL = spectrum[maxN-1] / spectrum[maxN];
    109.       float dR = spectrum[maxN+1] / spectrum[maxN];
    110.       freqN += 0.5f * (dR * dR - dL * dL);
    111.     }
    112.  
    113.     // Convert index to frequency
    114.     pitchValue = freqN * 24000 / SAMPLECOUNT;
    115.   }
    116.  
    117.   private void DeriveBlow() {
    118.  
    119.     UpdateRecords(dbValue, dbValues);
    120.     UpdateRecords(pitchValue, pitchValues);
    121.  
    122.     // Find the average pitch in our records (used to decipher against whistles, clicks, etc).
    123.     float sumPitch = 0;
    124.     foreach (float num in pitchValues) {
    125.       sumPitch += num;
    126.     }
    127.     sumPitch /= pitchValues.Count;
    128.  
    129.     // Run our low pass filter.
    130.     lowPassResults = LowPassFilter(dbValue);
    131.  
    132.     // Decides whether this instance of the result could be a blow or not.
    133.     if (lowPassResults > -30  sumPitch == 0) {
    134.       blowingTime += 1;
    135.     } else {
    136.       blowingTime = 0;
    137.     }
    138.  
    139.     // Once enough successful blows have occured over the previous frames (requiredBlowTime), the blow is triggered.
    140.     // This example says "blowing", or "not blowing", and also blows up a sphere.
    141.     if (blowingTime > requiedBlowTime) {
    142.       blowDisplay.guiText.text = "Blowing";
    143.       GameObject.FindGameObjectWithTag("Meter").transform.localScale *= 1.012f;
    144.     } else {
    145.       blowDisplay.guiText.text = "Not blowing";
    146.       GameObject.FindGameObjectWithTag("Meter").transform.localScale *= 0.999f;
    147.     }
    148.   }
    149.  
    150.   // Updates a record, by removing the oldest entry and adding the newest value (val).
    151.   private void UpdateRecords(float val, List<float> record) {
    152.     if (record.Count > recordedLength) {
    153.       record.RemoveAt(0);
    154.     }
    155.     record.Add(val);
    156.   }
    157.  
    158.   /// Gives a result (I don't really understand this yet) based on the peak volume of the record
    159.   /// and the previous low pass results.
    160.   private float LowPassFilter(float peakVolume) {
    161.     return ALPHA * peakVolume + (1.0f - ALPHA) * lowPassResults;
    162.   }
    163. }
    164.  
    I will be posting a tutorial up once I get this code working nicely, over here http://goo.gl/YU0vj.
  16. soren

    soren

    New Member

    Joined:
    Feb 18, 2008
    Messages:
    123
    Cool!
    You should use AudioSettings.outputSampleRate to get the right frequency instead of hardcoding it (it's 24000 on iOS by default)
  17. Vectrex

    Vectrex

    Member

    Joined:
    Oct 31, 2009
    Messages:
    149
    Nice work. But does setting the Start function to 999 seconds actually record 999 seconds worth of audio into RAM? What happens if you want it to go indefinately?
  18. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Hey thanks, glad to see you guys like it :)

    I am rather concerned about this. Here is the problem:

    Microphone.Start takes four arguments - microphone name, loop recording (bool), record time in seconds, and frequency. If we set a looping record for 10 seconds, Unity will record as normal for however long (I tested this). However at the end of each period (10 seconds), the recording must be stopped, the clip deleted (ram freed up), a new clip made, and finally the recording begun again. This leds to a very short pause (about 1 / 20 of a second).

    So yes, the 999 seconds worth is ALL stored into the ram, which I know is silly :) the script is still at a testing stage. There is no way, within Unity, to just turn the Mic on and extract data. I am sure this will be added by the Unity team soon, but until then I might just use a 60 second clip (like below) and deal with the short pauses. I'm not sure... any ideas?

    Code (csharp):
    1.  Microphone.Start("Built-in Microphone", true, 60, AudioSettings.outputSampleRate); // Thanks Soren!
  19. soren

    soren

    New Member

    Joined:
    Feb 18, 2008
    Messages:
    123
    This doesn't seem right? Who's deleting the clip and make a new one?
    For your pitch detection you probably only need 0.5 s worth of data right? So why don't you just a 0.5s long buffer with Microphone.Start and set it to loop? A then perform your fourier transform on that data? (spectrum)
  20. aubergine

    aubergine

    Member

    Joined:
    Sep 12, 2009
    Messages:
    2,040
    a game called iBlow ?? No, thanks :p
  21. fanjules

    fanjules

    New Member

    Joined:
    Nov 9, 2011
    Messages:
    162
  22. Riro

    Riro

    New Member

    Joined:
    Jan 16, 2012
    Messages:
    14
    Thanks for the heads up Soren. I thought I remembered reading that at the end of the loop a new buffer was allocated. Perhaps I mis-understood. I'll give that a try anyway.

    @Aubergine that was definitely the plan.
    @I am da Bawss I will post it up when I am happy with the script, along with theory on the subject etc. Thanks for showing interest :)
    @fanjules Lol, who puts a moisture sensor into a microphone? :S
    Last edited: Jan 21, 2012
  23. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    847
    Hi,
    Thanks for the initial code as a starter!

    I have it up and working fine on windows desktop aok, but it keeps crashing on iOS deploy to an iPad2 with a Bad Exec.
    The code I am using is:
    /// Starts the Mic, and plays the audio back in (near) real-time.
    privatevoid StartMicListener ()
    {
    //audio.clip = Microphone.Start ("Built-in Microphone", true, 1, AudioSettings.outputSampleRate);
    //Main Camera (MicTest1) Output Sample rate: 24000
    //Name: iPhone audio input
    Debug.Log ("audio.clip = Microphone.Start: ");
    audio.clip = Microphone.Start (null, true, 1, AudioSettings.outputSampleRate);
    // HACK - Forces the function to wait until the microphone has started, before moving onto the play function.
    while (!(Microphone.GetPosition(null) > 0)) {
    Debug.Log ("Microphone.GetPosition(null)");
    }
    audio.Play ();
    }

    and the Xcode output from above up until Exec_Bad_access crash is:
    - Completed reload, in 0.300 seconds
    -> applicationDidBecomeActive()
    Main Camera (MicTest1) Output Sample rate: 24000

    (Filename: /Applications/buildAgent/work/b0bcff80449a48aa/Runtime/ExportGenerated/iPhonePlayer-armv7/UnityEngineDebug.cpp Line: 43)

    Name: iPhone audio input

    (Filename: /Applications/buildAgent/work/b0bcff80449a48aa/Runtime/ExportGenerated/iPhonePlayer-armv7/UnityEngineDebug.cpp Line: 43)

    audio.clip = Microphone.Start:

    (Filename: /Applications/buildAgent/work/b0bcff80449a48aa/Runtime/ExportGenerated/iPhonePlayer-armv7/UnityEngineDebug.cpp Line: 43)

    kill
    Current language: auto; currently c++
    quit

    So nothing is coming from:
    while (!(Microphone.GetPosition(null) > 0)) {
    Debug.Log ("Microphone.GetPosition(null)");
    }
    so it's either not liking that or straight to audio.Play (); and crash.

    Any clues?

    Thanks!
  24. AlteredReality

    AlteredReality

    Member

    Joined:
    Aug 15, 2011
    Messages:
    382
    The first thing that seems off to me is that you are passing null into the microphone functions, instead of the name of the device. I would suspect that could cause a potential crash. Is there a reason you are doing that? You should be able to get the list of device names from Microphone.devices. Have you tried that?
  25. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    847
    Hi Andy,. Ty for the feedback.
    According to the 3.5 manual, Microphone.start:
    static function Start (deviceName : String, loop : boolean, lengthSec : int, frequency : int) : AudioClip
    Parameters
    deviceName the name of the device. Passing null or an empty string will pick the default device. Get device names with Microphone.devices

    I tried listing the devices at the start already but on iOS I only get " iPhone audio input "
    which gives me the same result. I also tried
    "Built-in Microphone" (which works on pc/mac) to no result.
    I finally tried the null as suggested by the manual to see what the default would do - which should only be
    " iPhone audio input " I think.
    I'll take another look just to be sure though.
  26. Ungenious

    Ungenious

    New Member

    Joined:
    Jul 29, 2010
    Messages:
    2
    Is there a way to setup Microphone.Start( ) with a lengthSec of less than 1?
  27. Logistic Win

    Logistic Win

    Member

    Joined:
    Apr 4, 2009
    Messages:
    107
    So, has anyone gotten this work in progress to work without the 999 second limit (and of course without filling up the memory as well)?

    Also, I just want to thank you all for the help you have been giving... I have been pretty quite since I started the topic, but I want you all to know that I am very thankful for all you have been putting in to this.

    One last thing.... have any of you thought of any way to cancel out the output sound of the device so that the blow detection doesn't fire on the app's own audio out? That would be awesome.
  28. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    847
    audio.clip = Microphone.Start (null, true, 1, AudioSettings.outputSampleRate); is fine with a 1-sec (or whatever buffer).
    Make sure you also:
    audio.loop = true; //IMPORTANT!
    audio.playOnAwake = false;
    audio.mute = true;
    loop will ... loop! and stop the spiking you will get if a clip is recreated every buffer cycle
    Mute will...mute! the mic input to output
  29. CarlEmail

    CarlEmail

    Member

    Joined:
    Jun 30, 2006
    Messages:
    313
    Thanks sonicviz, that did the trick!
  30. Evil-Dog

    Evil-Dog

    Member

    Joined:
    Oct 4, 2011
    Messages:
    117
    Hi there! This is all amazing stuff and I thank you for it! However, I'm having an issue where I'm trying to get an immediate response from the microphone and determining if I'm in the right frequency range as soon as I speak but it seems to be delayed by 1 second and it keeps determining I'm in the frequency range (or not) 1 second after I'm done speaking. I use the same AnalyzeSound function as Riro to compute the pitch.

    Any thoughts on the problem of getting a frequency range that updates at the same time as you speak?
    Thank you!
  31. moghazy

    moghazy

    New Member

    Joined:
    Apr 3, 2011
    Messages:
    64
    it's not working on android it always crash .. it only work on iphone :(