Koreographer - Audio Driven Events, Animation, and Gameplay

Hmm... We will take a look at the documentation again for the next release. In the meantime, please feel free to ask if you have any specific questions. In general, you can get away with installing a single component at the root of the Fabric Hierarchy to get Koreographer support for every AudioClip used.

If that works for you, then great!

I'm not sure what to say to this. I know for a fact that a user is successfully using the Karaoke demo on an iPad 2 with zero issues. As for Android, Cinema Director would suffer the same issue and I am unsure of whether they have a configurable compensation system built in or not. Would love to know if they do, though!

Cool, thanks! Best of luck with your project!!

Hah! We usually find ourselves watching tutorial videos at at least 2x, which is why we built our videos the way we did: to skip the long boring sections and get to the meat. No need to watch someone code something when you can just as easily pause the video!

Yeah, by that we mean that we "guarantee that the integration should work with Fabric 2.2.2+". If you happen to come across an issue, please let us know! We're good friends with Taz - chatted with him a bunch at GDC today! - and are happy to get things working as quickly as possible!

Either way, good luck rockin' your project!

 

Great to hear, iPad2 is my lowest target, so I think I should be good. Thanks.

 

Hi @SonicBloomEric,
we have a project where users on a real world would sing along with a music playing in the game world. After that, a system would judge how well users sang. Is it doable with this plugin? If yes, how? If not, do you have any plan to add necessary features?

We have a very tight schedule.

 

Hi @kaiyum! Thanks for reaching out. Unfortunately Koreographer does not currently contain any pitch-matching algorithms or functionality. It is something we are evaluating but I would not expect anything soon.

I did a bit of poking around and found a few potential resources that you may find useful, though! See:

• Pitch Detector - An asset on the asset store that may get you what you need.
• Real time pitch detection - A Stack Overflow discussion with tons of fantastic research and links to open source solutions and algorithms that you may be able to leverage in your game.

I hope this is useful to you.

The one way that Koreographer can help you is with timing lyrics to appear on the screen, as well as managing the expected "pitch target" at any given moment as the audio plays. This side of synchronization is what Koreographer excels at.

Best of luck on your project!

 

Hi, any progress on offering a rhythm game demo? I recently bought the standard version and I would love insight into how to utilize Koreographer in that way. I've run through the quick start and users guide and have basic Koreographer functionality (events triggering properly) but I'm stalled out trying to get access to the tempo map information, which is what I believe I need?

How I assume I would use it to set up a basic beat-matching rhythm game:
I would compare the time of user input to the corresponding event time in the tempo map, +/- a buffer to allow for touches just prior and just after the beat.
Am I on the right track? How can I access the information if so, and if not, any help you could provide on how to actually accomplish this would be so helpful. Thanks so much.

 

Hi @JeffForKing! First off, thanks for picking up Koreographer! I'm glad you were able to quickly get the event triggering figured out!

As for the rhythm game demo, we don't have anything we're prepared to release yet but we will be releasing it with Koreographer 1.4, which is in the works. In the meantime, I'm more than happy to help you get things up and running without it!

You're on the right track here! The tempo map method will work if all you want to track are the major beats in a song. In that case you could just check at any time whether the user's input was +/- a certain percent of a beat from the current BeatTime. That would look something like this (inside a check against player input):

Code (CSharp):

1. float buffer = 0.1f;  // Percentage in beats.
2. float beatTime = Koreographer.GetBeatTime();
3. float curBeat = Mathf.Floor(beatTime);
4.  
5. if (beatTime - buffer <= curBeat ||       // On the late side.
6.     beatTime + buffer >= curBeat + 1f)    // On the early side.
7. {
8.     // Hit!
9. }

That said, for more flexibility, we recommend laying out your hits using a KoreographyTrack. This allows you to place KoreographyEvents exactly where you'd want the player to hit them, including sub-beats or syncopated beats! The code to do this looks something like this:

Code (CSharp):

1. // Fields in your custom behaviour for the inspector...
2. [EventID]
3. public string rhythmTrackID;
4. public Koreography rhythmKoreo;
5.  
6. int curCheckIdx = 0;
7. List<KoreographyEvent> rhythmEvents;
8. int bufferInSamples;
9.  
10. //...
11.  
12. // In Start() or somesuch
13. rhythmEvents = rhythmKoreo.GetTrackByID(rhythmTrackID).GetAllEvents();
14. bufferInSamples = Koreographer.GetSampleRate() * 0.1f; // The 0.1 is in seconds, not beats in this case.
15.  
16. //...
17.  
18. // In Update()
19. int curTime = Koreographer.GetSampleTime();
20.  
21. // In your input check (within Update()):
22. // Check that the current time is within a certain distance from a KoreographyEvent (assumes OneOff!).
23. if (Mathf.Abs(curTime - rhythmEvents[curCheckIdx].StartSample) <= bufferInSamples)
24. {
25.     // Hit!
26. }
27.  
28. // In Update, again:
29. if (curTime > rhythmEvents[curCheckIdx] .StartSample)
30. {
31.     curCheckIdx++;
32.     // Note that you want to check that curCheckIdx isn't out of bounds for rhythmEvents.
33.     //  If it is, you may just want to disable the component (or put the bounds check around
34.     //  all of the event checking logic because there are no longer valid beats to check
35.     //  anyway!  :)
36. }

There are a few things implied by these two samples:

1. The second version (based on KoreographyTrack) has a "buffer" in seconds. That will be consistent across all different songs you throw at it, regardless of BPM. The first one (based on the Tempo Map) will have a window that changes based on the BPM. There is a way to compensate for this that involves using BPM calculations but we honestly don't suggest using it: you start getting into floating point precision problems. And this is something we plan to address with the upcoming 1.4.
2. None of this shows how to do animations to indicate "an incoming beat" or the like. A lookahead, if you will. This is pretty trivial to do with the Tempo Map based solution, of course. For the KoreographyTrack version, it's also pretty simple. In that case you simply specify a range of time and keep two separate indices into the same list of tracks (rhythmEvents, in the example above). For each KoreographyEvent that falls in that range, calculate how far it's moved between the start and end and then update a visual representation in your scene to match!

I will also point out that in the second version above I showed how you could get a KoreographyTrack from a Koreography object (which you'd conceivably set in the Inspector). Another way to do it would be to simply specify the KoreographyTrack directly in the inspector. If you're building an entire system for this, then you likely have a "Game Controller" with a list of Koreography that are sent to a player (SimpleMusicPlayer or MultiMusicPlayer) for playback. Presumably you could also request the "current Koreography" from that Game Controller as well.

Does this all make sense? There's a lot to take in but I'm happy to dive into more detail or clarify wherever you may need it

You're very welcome!

 

Hello, I got Koreographer a few months ago on a Sale but I'm just starting to actually use it now... so far everything is great.

I'm just want to ask if there's a way to retrieve or get everytime a Beat occurs or something like that.

Let me explain, I basically want to execute something at each beat, I know the BPM of my song, and I need a float value that goes from 0 - 1 between each Beat.

So far I've already done it with code, calculating the current time and the lenght of each beat, and it kinda works... but, after a while, since it is not actually taking any info from the song itself, it's a different calculation, after some time, my beats are not sync with the song.

So I was wondering how could I use Koreographer to help me with that?.

Technically what I need is a Curve Value added between each Beat.. and perhaps an event at each beat.. something like this



But I'll have many songs, and each that song is about 3m long... so doing that by hand is gonna hurt.

Is there a way to do that automatically?

 

Hurray! Thanks for picking it up! Glad to hear that everything's great

I'm jumping around here a bit, but I figured I would point out that the Clone tool would be your friend for this ;D That said, read on!

Yes! I believe what you're looking for is this:

Code (CSharp):

1. float percentThroughBeat = Koreographer.GetBeatTimeNormalized();

Please let me know if this helps!

 

Yes That's what I needed, works like charm

Thanks for the quick reply.

I need to excecute a function in each beat... I can work with what I already have, just making a little logic, storing the last percent and if the last one is bigger than the current one, it means a new beat just restarted and I can call my function there.

But just asking if there's already a build in way to subscribe to an event for that?

 

Hurray! Glad that worked!

As for the OnBeat event callback, we do not currently provide one, no. We could add one, I think, if you think there would be utility. Generally the information is already available with the Beat Time APIs, as you've seen. Further, the Beat Time APIs are flexible in that you can get the Beat Time with subdivisions specified.

Out of curiosity, what information would you wish to receive in a callback? Would you only want a callback for main beats? Or subdivisions?

 

Thank you, that was absolutely the help I needed. It really does quickly/easily provide the basic mechanic for a rhythm game. So grateful for the quick and thorough response. Can't wait to delve deeper

 

Awesome! Super glad that this helped you out! Don't hesitate to reach out if you get stuck on something!

 

Hi Eric, I just bought the asset on Saturday and sent you a email using the Sonic Bloom web but still no answer, so better I ask it here and everybody can read your answer too.

1. Im not able to receive event data using casting, for example:

void PlayBit(KoreographyEvent evt)
{

Debug.Log((int) evt;
...
​

It show this error:

error CS0030: Cannot convert type `SonicBloom.Koreo.KoreographyEvent' to `int'
​

I was able to solve this using evt.GetIntValue()


2. I would need to edit the code becouse I need to record chords on the fly with the keyboard on the koreography editor, asigning events with specific values to specific keys on the keyboard, becouse doing it separately just tapping enter is really tedious.

Anyway the asset is great and it run nice on mobile. Im running the game on a BQ Aquaris M5 (CPU: Qualcomm® Snapdragon™ 615 Octa Core A53 1,5 GHz, GPU: Qualcomm® Adreno™ 405 550 MHz, MEMORY: 2 GB Ram)
at 60 FPS.

 

Sry I forgot another question:
- Is it possible to retrieve the song tempo in BPM instead of samples per bit?

 

Hi Jesús, thanks for the question! I was just working on a response to your questions, actually. And thanks for posting these questions here where others may benefit from the discussion!

I'm glad that you were able to solve this using the Payload.GetIntValue() method! That is one way. There are actually several ways to do it, as outlined in the Documentation (see the section called "Payload Handling" in the Koreographer User's Guide). The casting operators have not been overloaded but this is actually something we will look into adding! It's certainly a neat idea!

We've had one other request for this kind of feature and are interested in supporting it. However, we really aren't sure what the interface would look like. In the link that I just posted, I posit one solution but there are many others. In the end that user modified the Koreography Editor source code to add their own input mode with custom mappings to suit them (they have Koreographer Professional Edition). One issue we would face in implementing this generically is Keyboard Ghosting. Regardless, we are happy to look into adding support for the feature, although this likely wouldn't make it into the next release.

That said, Koreographer Professional Edition was built to optimize this kind of work using MIDI files. If you are working with a musician you should be able to request that they export a MIDI version of the file that you could simply import. If you wanted each note of the chord to appear in separate Koreography Tracks, you could export them as separate MIDI Tracks and then import them into Koreographer that way. This is by far the most precise, shortest way to get timing into the system. If you don't have a way to request the MIDI file from the composer you could chart out your events in MIDI in Reaper (or MIDIEditor, or Aria Maestosa), for instance, and then import them into Koreographer.

Alternatively, if your music is "regular" (in that it has a known chord progression), you could create your chord and then use either the Clone or Draw-and-then-Copy+Paste methods of Koreography Event generation to more quickly accomplish your goal

If none of the above solutions work for you, we're happy to provide you with access to the source code once we can verify your purchase of (or upgrade to) the Professional Edition.

Currently, there is no API for accessing the BPM of a song. We are working on overhauling the Tempo Section (tempo map) code for a future release that will address this. In the meantime, the math to convert the Samples Per Beat into Beats Per Minute is very straightforward. Once you have a Tempo Section reference and can grab the Samples Per Beat value, the code looks something like:

Code (CSharp):

1. public static float GetBPMFromSPB(float samplesPerBeat)
2. {
3.     float frequency = (float)Koreographer.GetSampleRate();
4.     return frequency / (samplesPerBeat * 60f);
5. }

Note that if you know the Sample Rate of the music/audio in your game you could easily hardcode that value rather than request it at runtime

Woah! That is really awesome to hear!! Way to go! Hopefully you'll be able to show off what you're making soon! We'd love to play it!

 

Back with another question, along the lines of what you were helping me with initially.

I see in the TempoSwitch script for the lights in the demo, you use a reference that changes every quarter beat, not events, to make changes to the lights. How can I use GetBeatTime to give me a whole note (or more generally change it), as I'd like to subdivide to get the smaller beats from there.
I thought of changing the bpm but I'm wondering if there's an out-of-the-box way be notified of each whole, half, quarter, eighth, etc notes, independent from events?

Does that question make sense?

 

Yes!

I turns out that, unless I've misinterpreted your question, you've got everything you need right there with Koroegrapher.GetBeatTime. If you look a little bit lower down in the TempoSwitch.Update method you'll see that we grab the current eighth note using the same method but with different parameters. From that method:

Code (CSharp):

1. // The Demo song has a quarter note as it's beat value.  This will get us the current
2. //  quarter note!
3. int curQuarterNote = Mathf.FloorToInt(Koreographer.GetBeatTime());
4.  
5. // ...
6.  
7. // The 'null' value asks Koreographer to look at the beat time of what it considers
8. // the current "Main" song. These demos use a basic player with a single song and
9. // define that as the Main song. Therefore there is no need to specify it. The
10. // '2' parameter, tells Koreographer to divide each beat into 2 equal parts. As the
11. // base beat value is 4, this will result in eighth notes.
12. int curEighthNote = Mathf.FloorToInt(Koreographer.GetBeatTime(null, 2));

Effectively, you can get what you want by specifying the subdivision you want when you use Koreographer.GetBeatTime:

Code (CSharp):

1. float quarterNote = Koreographer.GetBeatTime();
2. float quarterNote2 = Koreographer.GetBeatTime(null, 1);
3. float eighthNote = Koreographer.GetBeatTime(null, 2);
4. float twelfthNote = Koreographer.GetBeatTime(null, 3);
5. float sixteenthNote = Koreographer.GetBeatTime(null, 4);
6. // ... etc.

Now, to get half and whole notes if your music has a beat value of a quarter note, you'd probably want to do something like the following:

Code (CSharp):

1. // Assuming a beat value in the music of 4 (quarter note) and that
2. // you have a single tempo section.
3. float halfNote = Koreographer.GetBeatTime() / 2f;
4. float wholeNote = Koreographer.GetBeatTime() / 4f;

If you have a complex Tempo Map (multiple Tempo Sections defined) then you may need to interface with the Tempo Map directly (via a reference to your Koreography) to calculate these values.

Does this answer your question?

 

Ahhh, so GetBeatTime is returning the value between the prior beat (a whole number) and the upcoming beat (another whole number)?
So that's why, to make the GetBeatTime work in this context, you need to FloorToInt the result?

1. int curHalfNote=Mathf.FloorToInt(Koreographer.GetBeatTime()/2f);

I was struggling with this concept so much yesterday and today it's just like, oh. yeah. That makes total sense. Thank you so much for the quick reply!

 

GetBeatTime returns the current time-in-beats of the song. This means that if you're halfway through the 37th beat in the song, it will return 37.5. It works similarly to the built-in Time.time. By calling Mathf.FloorToInt, we can determine when the beat actually flipped to the next number (beat onset timing, roughly) and then do something with that. For the demo scene we simply switch the configuration of some spotlights (flip their configuration back and forth).

No problem at all! Glad to hear that it makes sense and you're unblocked! Best of luck as you continue pushing forward!

 

Is it possible to change the pitch of the Multi Music Player via code during runtime? If not, is it possible via a custom player? I've read the users guide section but it only mentions getting a pitch, not setting.

 

Yes! This is done by using the MultiMusicPlayer.Pitch property. This allows you to do something like the following:

Code (CSharp):

1. // Set in the inspector.
2. public MultiMusicPlayer multiMusicCom;
3.  
4. void Update()
5. {
6.     if (Input.GetKeyDown(KeyCode.UpArrow))
7.     {
8.         multiMusicCom.Pitch += 0.1f;
9.     }
10.     else if (Input.GetKeyDown(KeyCode.DownArrow))
11.     {
12.         multiMusicCom.Pitch -= 0.1f;
13.     }
14. }

Yup! The custom player would be one that you write yourself and involves overriding the VisorBase class in some way. This is a lot more involved than using the MultiMusicPlayer but also allows you to have full control over the music player. Fun fact: all audio/music players and integrations included with Koreographer override the VisorBase class in some way or another

This is something we plan to address with more complete API documentation (PDF/web rather than inline) in the future. Out of curiosity, what IDE do you use? We do provide the necessary XML files along with the DLLs to allow inline documentation (piggybacks on IntelliSense/MonoDevelop's equivalent). It would be good to know if this wasn't working for some users or was missed entirely

 

I'm using Xamarin Studio 5.10. And yeah, I tried earlier, but am not able to declare a MultiMusicPlayer. The auto-completion generally works well, and does contain some helpful documentation, but in this case it's the Unity classic The type or namespace name `MultiMusicPlayer' could not be found. Are you missing a using directive or an assembly reference?

Added some screenshots too, because why not-

1. using UnityEngine;
2. using System.Collections;
3. using SonicBloom.Koreo;
4. 
   
5. public class DeclareMusiPlayer : MonoBehaviour {
6. 
   
7. public Koreographer testKor;
8. public MultiMusicPlayer mus;
9. 
   
10. // Use this for initialization
11. void Start () {
13. }
15. // Update is called once per frame
16. void Update () {
18. }
19. }

 

Got it! For scripting purposes, the audio player classes are in the SonicBloom.Koreo.Players namespace! Try adding the following to the top of your script:

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using SonicBloom.Koreo;
4. using SonicBloom.Koreo.Players;    // ←ADD THIS! :D

For more information on the Namespaces check out the Scripting Koreographer section of the User's Guide (currently on page 33). We will look into doing a better job communicating these in the actual Music Player documentation sections going forward!

 

once again, the quick response time here is so appreciated. THANK YOU!

 

Of course! Happy to help! Glad we were able to quickly get you unblocked!

 

I have just started testing with Koreographer today and have run into a small issue.

I am finding events are intermittently firing when they are not supposed to. I have a basic track setup to fire off events as per the quick start guide and am firing events just off the main beats. Every now and again I will find beats will fire off in quick succession at random spots in the track (never the same). I debugged this by firing off a log from GetBeatTime using MathF to floor it. The vast majority of the beats come in at 7 or 8 but sometimes an event will fire twice. E.g. output will be 315, 323, 331, 331, 338.

I have coded around this for now but was wondering if there were some tips that may help resolve this.

 

Hopefully we'll be able to help you find a small fix ;D

Typically the only time that we see this kind of thing is if there are multiple OneOffs placed very close together or if you have perhaps accidentally stretched a OneOff into a Span. With multiple close OneOffs, you will always receive multiple events. With very short Spans you will only see multiple events triggered if a frame update happens to occur in between its Start and End sample position (one event for the start; another for the end).

Out of curiosity, could you clarify what the numbers you quote refer to? You mention "7 or 8" and then jump to the range 315-338. Are these all beats in a single song? Does it loop?

If you are okay with sending us a project, we would be happy to take a look at it and see if we can help you track down the issue. Feel free to reach out to support if you're interested in this route!

 

Firstly thanks very much for your reply. I will check tonight when I get home if there are any short spans I have put in there, sounds like a very likely culprit.

Sorry for the confusion there (I re-read and confused myself). The value I get back from GetBeatTime at the start of the track starts at 0, and then 7 and so on. Typically the difference I see between from one GetBeatTime to the next is either 7 or 8 (this is with using Mathf.Floor). However when I see the 2 events in extremely quick succession (maybe milliseconds between events) the difference between the 2 GetBeatTime values is a fraction of 1 (the value back will be the same twice if using MathF.Floor). Hope that makes it a bit clearer.

All the beats are in a single song, with only 1 event track, which is simply firing an event on each main beat (BPM around 130).

Keep in mind this is the first time I have used the software so it quite possibly is something I have done wrong, I will look at it more closely tonight.

P.S. Fantastic software I am having a lot of fun with it.

P.P.S. In simple speak what would I gain from going to Pro? Please keep in mind I am not using any of the compatible software like Sectr, Fabric etc at this stage. I noticed one of the features of pro is automatic event mapping, how does that work?

 

Ahh, understood! That makes sense. One thing that you can do is store a reference to the KoreographyEvent object that you're handed in the event callback. The next time the event comes through you could test whether that KoreographyEvent was the same as the one you stored during the last time (KoreographyEvents are C# objects so you can use the equality test to see if variables refer to the same instance or not!).

A simple test to see with whether or not you have a OneOff or a possibly-super-short span is to check the IsOneOff() method

Another thing to test is whether your callback comes on the same frame or not and, if not, if your doubling-up occurs in two successive frames. You can use the Time.frameCount method to get this.

No worries! Hopefully we'll help you get this figured out quickly!

Awesome! Thanks so much! Very excited to hear that

Beyond third-party integrations Koreographer Professional gets you:

• MIDI Converter: Import a MIDI file version of your music to get access to each and every note timing. This is especially useful if you are working with a composer.
• Audio Analysis: This feature is currently very early but we have big plans for it. The current functionality is limited to creating events with Payloads that describe the RMS of the waveform itself. This is good for making "speech" effects, etc. with pre-processed data rather than having to run an RMS algorithm at runtime. We have more planned for future versions.
• Source Code Access: We offer access to the source code for people who need it to implement some feature on their own that we either can't get to immediately or don't plan to support. Some users have used this to implement their own "annotation" input, for instance.

I hope that helps!

 

Thanks again for your very detailed reply. I did a bit of checking and I couldn't see any spans in there, so I just deleted the Koreography asset and made a new one, problem seems to be gone, thanks again

 

Glad to hear that everything is working! Please don't hesitate to reach out with questions if you get stuck again!

 

Ok, so if we didn't want to keep this null, what would the "clip name" be? I've tried names of seemingly everything but can't seem to specify which track's beat time I want to utilize. Tried audio clip names, tempo section names, event names, object names... can't seem to sort that out.

 

Koreographer's BeatTime functions are part of the Music Time API. The API requires that you use one of the music players included with Koreographer, a third party integration with the Music Time API configured, or implement the IKoreographedPlayer interface yourself and properly initialize the musicPlaybackController property of the Koreographer object in your scene.

Further, this only works with the name of an AudioClip that is both:

1. being played by the configured musicPlaybackController and
2. has a Koreography loaded into the Koreographer object (component).

Please be aware that both of the included Music Players (SimpleMusicPlayer & MultiMusicPlayer) handle the connections for you (both loading of Koreography, if specified, as well as connecting to the Koreographer singleton instance by initializing the musicPlaybackController property).

If you have a reference to either a SimpleMusicPlayer or a MultiMusicPlayer object in your scene you can simply call the GetCurrentClipName() method on them to retrieve the current "music" track.

It should be noted that the MultiMusicPlayer will always return the base layer's name, regardless of whether it's configured with Koreography or not (if no Koreography is specified in this layer, the Music Time APIs will not function). That said, if you happen to know the name of the other layers, they, too, will work properly.

If it is simply the null that you are trying to avoid, it should be noted that both string.Empty and "" will both function the same way.

Do you mind if I ask for a bit more detail on the precise nature of the problem you are trying to solve?

 

Sure, I'm happy to post code if it helps provide more clarity but basically I want to be able to have at least two music tracks, both of which I can change the pitch of to speed up, slow down, or reverse the playback independent from the other.

This is the answer to my question but, as I think you mention, it seems i can only use the string name in GetBeatTime if it corresponds to the first loaded Koreography in Koreographer-- which is basically what it returns anyway-- so it doesn't solve my larger problem.

 

Hey Eric, love Koreography but I have a quick best practices question for you. I'm using Koreography on background music to sync animations, and that's working great. However, I also have a narrator that I would like to have tracked for subtitles and RMS visualization. The narrator triggers after different game events. Currently I have many different audio files for each of the narrators phrases.

What's the best way to implement this? Can I do it inside of just one Koreography instance, or should I use two, one for music and one for voice? Should I combine all the voice into a single file, and just modify the start/end time of playing the file, although this would cause headaches when making changes? Do I create a separate Koreography file and track for each one of my many audio files, and swap out which one is loaded into the Koreography instance? Does that cause bloat or extra overhead?

Everything else is working great, and I'm sure I can figure out a solution, but I figured you might have some insight on the best way to accomplish the task without causing performance problems!

Thanks

 

Understood. I assume you're okay with the relative cacophony that will result from this?

I assume that you're using two SimpleMusicPlayers to control the two tracks separately. In this case, the second SimpleMusicPlayer to start up will take over the Music Player API duties from the first (a single Koreographer only supports a single Music Player at a time). The SimpleMusicPlayer also does not [currently] have an option to specify the Koreographer object to associate.

That said, you can emulate the Music Time API as long as you have a reference to the Koreography loaded into your SimpleMusicPlayer and the SimpleMusicPlayer reference itself. You would do something like the following:

Code (CSharp):

1. public SimpleMusicPlayer playerCom;
2. public Koreography koreo;
3.  
4. // At some point in the code:
5. int sampleTime = playerCom.GetSampleTimeForClip(koreo.SourceClipName);
6. int subdivisions = 2;
7. float beatTime = koreo.GetBeatTimeFromSampleTime(sampleTime, subdivisions);

Does this seem like it will get you what you need?

 

Awesome! Glad to hear things are working well! Your project sounds really neat and we'll hopefully get you moving quickly!

We highly recommend going with the latter. The overhead for Koreography itself is very low. The KoreographyTracks themselves can become heavy but that weight is entirely dependent upon how many KoreographyEvents you put in them. This also allows you to optimize your AudioClips in whatever way suits you best without having to affect others in your project!

A best practice for this setup would consist of the following:

• Music Setup
  • Use either the SimpleMusicPlayer or the MultiMusicPlayer. Nothing special here!
• Narrator Setup
  1. Somewhere in your Hierarchy, add an AudioSourceVisor component to a GameObject. If the GameObject does not have an AudioSource on it yet, adding the AudioSourceVisor will add it for you. The AudioSourceVisor is a very simple built-in script that watches the associated AudioSource for AudioClip playback and sends the timing information to Koreographer. This in conjunction with a custom script will get you what you need.
  2. A Custom NarratorController Script. The idea here is that the NarratorController is in charge of AudioClip playback and Koreography registration. The NarratorController would be told something like "PlayIntro()." Within this method you would load the AudioClip into the AudioSource mentioned in #1 above. You would load the associated Koreography into the Koreographer. You then call the Play() method on the AudioSource and the AudioVisor will do its work!
     • At the end of all of this you may want to unload the Koreography from the Koreographer singleton/instance. Leaving a large quantity of unused Koreography loaded in a Koreographer will affect performance. That said, by "large quantity" you'd probably need a something in the upper 10s to 100s in order to actually notice anything.
     • Note that you can also ignore the direct reference to the AudioClip! The Koreography has that for you already: just use the SourceClip property ;D

That should do it for you. It has the benefit of leaving your music setup alone while enabling you to control your SFX/Voice in whatever way works best for you! Please let us know if we can be of further help!

 

Hey team - I'm really loving Koreographer Pro so far, but I've hit a snag. I'm making a notecatch prototype, and I'm trying to use the MultiMusicPlayer to handle the music. Right now, it contains 8 Music Layers (Drums 1, Drums 2, Bass, Synth 1, etc.), each with a Koreography that includes onoff events with Note data imported from the music's corresponding MIDI files.

I'd like to do the typical thing where you generate visible notes, the player surfs across the track, and the track is only audible if the player is hitting the notes. I can do that just fine with a single SimpleMusicPlayer, but I don't know how to get the Layers out of a MultiMusicPlayer object in order to do the same for the 8 music track goal.

Is there a way to iterate through each Music Layer referenced in an MMP in order to get their Koreographies and thus read note data / change their volumes? I don't think hard-coding the track names for each level/song is the most efficient way; I'm sure there's something I'm overlooking.

For posterity, some sample code for 1-track approach:

Code (CSharp):

1. // Kudos to the SonicBloom team for the sample code this is built upon.
2. // other variables exist; these are the most relevant.
3.     List<KoreographyEvent> rhythmEvents;
4.     List<GameObject> rhythmObjects;
5.  
6. // Use this for initialization
7.     void Start () {
8.         rhythmEvents = rhythmKoreo.GetTrackByID(rhythmTrackID).GetAllEvents();
9.         rhythmObjects = new List<GameObject>();
10.         bufferInSamples = (int)(Koreographer.GetSampleRate() * 0.1f); // The 0.1 is in seconds, not beats in this case.
11.  
12.         Koreographer.Instance.RegisterForEvents("TrackTestKTrack", FireEventDebugLog);
13.  
14.         GenerateTrack (); // Make the note objects
15.     }
16.  
17.     void GenerateTrack ()
18.     {
19.         foreach(KoreographyEvent evt in rhythmEvents)
20.         {
21.             GameObject s = GameObject.CreatePrimitive(PrimitiveType.Sphere);
22.             s.transform.position = new Vector3 ((evt.GetIntValue () - 48) * 1.8f, 0F, evt.StartSample / (Koreographer.GetSampleRate () * 0.04f));
23.             rhythmObjects.Add (s);
24.         }
25.     }
26.  
27.     void FireEventDebugLog(KoreographyEvent koreoEvent)
28.     {
29.         Debug.Log("KoreographyEventFired: " + koreoEvent.GetIntValue().ToString());
30.     }
31.  
32.     // Update is called once per frame
33.     void Update () {
34.  
35.         int curTime = Koreographer.GetSampleTime();
36.  
37.         if (Mathf.Abs(curTime - rhythmEvents[curCheckIdx].StartSample) <= bufferInSamples)
38.         {
39.             // Hit!
40.        
41.             int value = rhythmEvents [curCheckIdx].GetIntValue ();
42.  
43.             // difficulty branch...
44.  
45.             if (value == 48)
46.             {
47.                 if (Input.GetButtonDown ("LeftNote") && curCheckIdxHit == false)
48.                 {
49.                     Debug.Log("EVENT " + curCheckIdx + " HIT ON THE LEFT!");
50.                     curCheckIdxHit = true;
51.                     rhythmObjects [curCheckIdx].SetActive (false);
52.                 }
53.             }
54.  
55.             if (value == 50)
56.             {
57.                 if (Input.GetButtonDown ("RightNote") && curCheckIdxHit == false)
58.                 {
59.                     Debug.Log("EVENT " + curCheckIdx + " HIT ON THE RIGHT!");
60.                     curCheckIdxHit = true;
61.                     rhythmObjects [curCheckIdx].SetActive (false);
62.                 }
63.             }
64.  
65.             if (curCheckIdxHit == true) // volume tweaking logic if good play
66.             {
67.                 GameObject koreo = GameObject.Find ("KOREOGRAPHER");
68.                 koreo.GetComponent<AudioSource> ().volume = 1.0f;
69.             }
70.         }
71.  
72.         if (curTime > rhythmEvents[curCheckIdx] .StartSample)
73.         {
74.             if (curCheckIdxHit == false) // volume tweaking logic if bad play
75.             {
76.                 GameObject koreo = GameObject.Find ("KOREOGRAPHER");
77.                 koreo.GetComponent<AudioSource> ().volume = 0.0f;
78.  
79.             }
80.  
81.             //Debug.Log("EVENT " + curCheckIdx + " ADVANCED!");
82.             curCheckIdx++;
83.             curCheckIdxHit = false;
84.        
85.             // Add bounds checking...
86.         }
87.  
88.         transform.position = new Vector3(transform.position.x,0,curTime / (Koreographer.GetSampleRate () * 0.04f));
89.     }
90.  

 

Second, parallel question - about parallel events. Namely, is there a way to preserve chords in the Koreographer MIDI Converter? I'm noticing that they're getting crunched down to one note, so instead of getting 40 AND 42, I'm getting just one at a time.

 

Nice! Sounds like a fun project! I'm happy to help you out here. Luckily, this is a pretty easy thing to set up and support!

There is not currently a way to directly access the MusicLayer objects of a MultiMusicPlayer that were configured with the Inspector. That said, what you are attempting is entirely possible and actually pretty simple to implement!

Before diving into a full explanation, I would point out that if all you needed to do was to adjust the Volume of a layer, you can do so by using the MultiMusicPlayer's SetVolumeForLayer methods. You could also create your own AudioSource components and provide each MusicLayer with a reference to its own AudioSource to use for playback. This would allow you full control over all features of the AudioSource (connections to the Audio Mixer, effects, etc.), including volume.

That said, let's take a look at what we would recommend for your specific goal. The key insight is that you do not need to set up your Music Layers in the MultiMusicPlayer itself! The MultiMusicPlayer component has a LoadSong method that takes a list of MusicLayer objects (literally: List<MusicLayer>). The idea here is that you have a MusicManager behaviour that has a public or [SerializedField] "musicLayers" property which you configure for each level. That MusicManager is then responsible for calling "LoadSong" on the MultiMusicPlayer component and passing the configured MusicLayers along.

And now that you have this MusicManager behaviour with knowledge of all of your MusicLayers, you can have it hand one MusicLayer each to an instance of the behaviour that handles the tracking logic. Abstracting the pseudocode you posted above to run on a MusicLayer should be pretty simple:

Code (CSharp):

1.     // For storing the reference to the MusicLayer we're tracking.
2.     MusicLayer myLayer;
3.  
4.     // Standard for tracking/matching objects.
5.     List<KoreographyEvent> rhythmEvents;
6.     List<GameObject> rhythmObjects;
7.  
8.     // Do initialization in this method, rather than Start().
9.     public void SetMusicLayer(MusicLayer layer)
10.     {
11.         myLayer = layer;
12.  
13.         rhythmEvents = myLayer.KoreoData.GetTrackByID(rhythmTrackID).GetAllEvents();
14.         rhythmObjects = new List<GameObject>();
15.         bufferInSamples = (int)(Koreographer.GetSampleRate() * 0.1f); // The 0.1 is in seconds, not beats in this case.
16.  
17.         GenerateTrack(); // Make the note objects
18.  
19.         // NOTE: Assume that this component is disabled in the Inspector.  This
20.         //  makes it so that we don't get NullReferenceExceptions in the Update()
21.         //  call every frame until we're ready.  Enabling the component means
22.         //  that Update will be called.
23.         enabled = true;
24.     }
25.     void Update()
26.     {
27.         int curTime = Koreographer.GetSampleTime();
28.         if (Mathf.Abs(curTime - rhythmEvents[curCheckIdx].StartSample) <= bufferInSamples)
29.         {
30.             // Hit!
31.    
32.             //-------------------------------
33.             // ...Unchanged stuff here...
34.             //-------------------------------
35.             if (curCheckIdxHit == true) // volume tweaking logic if good play
36.             {
37.                 myLayer.AudioSourceCom.volume = 1f;
38.             }
39.         }
40.         if (curTime > rhythmEvents[curCheckIdx] .StartSample)
41.         {
42.             if (curCheckIdxHit == false) // volume tweaking logic if bad play
43.             {
44.                 myLayer.AudioSourceCom.volume = 0f;
45.             }
46.             //Debug.Log("EVENT " + curCheckIdx + " ADVANCED!");
47.             curCheckIdx++;
48.             curCheckIdxHit = false;
49.    
50.             // Add bounds checking...
51.         }
52.         transform.position = new Vector3(transform.position.x,0,curTime / (Koreographer.GetSampleRate () * 0.04f));
53.     }

The important lines above are 1, 13, 23, 37, and 44. Each of those are described here:

• Line 1: Store a reference to the MusicLayer for this Koreography. There's a little more to a MusicLayer object than just the Koreography and AudioSource references but that's really the core of what you need. It should be noted that these are Objects, not structs. This means that the MultiMusicPlayer takes these in and copies the Object References into its own List. This means that your modifications to the MusicLayers themselves will affect those that the MultiMusicPlayer is aware of but adjusting the List you passed in will not affect the MultiMusicPlayer's list (it's an ordering thing only).
• Line 13: You can get the Koreography used by the MusicLayer by accessing the KoreoData property.
• Line 23: This is a special little trick that I threw in here to keep the Update() logic simple (no need to check for a Null list of events). Effectively you can create an instance of this component at Edit Time and link it to your MusicManager in the Inspector. If you do so, it would be best to disable the component by default (uncheck the enabled box at the top of it's GUI in the inspector). This means that Update won't get called. By setting the component to enabled once the track is generated, we have sidestepped any need to do basic error checking in the Update method (and created a better performing component as well - no needless Update calls!).
• Lines 37 and 44: When the MultiMusicPlayer is handed a list of MusicLayers it goes through an "Initialization" step. Part of this is creating an AudioSource for every MusicLayer that does not have one pre-configured. As mentioned above, the MusicLayer objects are passed by reference which means that once you've called LoadSong, all MusicLayers should have an AudioSource component set to them. You can access this AudioSource component directly by accessing the MusicLayer's AudioSourceCom property.

Using this structure you can better control the configuration of your MultiMusicPlayer and have more precise control over how each layer acts. Further, you can send the data to whatever structure or manager that you design for your specific implementation problem! (Of course, the adjustments made above aren't the whole picture for the "watcher" component but it should be enough to get you pointed in the right direction!)

 

Unfortunately this is not something currently supported. MIDI files do not by default store special "Chord" information, but rather every note that was hit to make the chord. You would need a special tool or a special MIDI instrument setup that would take a single "note" and interpret it as a chord.

The best way to do this right now is to adjust the MIDI data in your DAW or a MIDI Editor and then import with the MIDI Converter. Alternatively, you could also adjust the Koreography Events that result from the import process. We do have "MIDI Chord Detection" on our roadmap but we do not yet have a timeline for when this feature will be implemented (attempted?). I hope this answers your question!

 

Wow! You've given me plenty to chew on, and chew on it I shall. Before making these changes though, I've noticed that I've begun to get stuttering in the audio playback - any tips on reducing this?

I think it does - so to be clear, preserving simultaneous MIDI events is not a current feature, but it's a maybe-eventually one? Cool! That's fine too. Thanks for your help.

 

I take it you're not also experiencing choppy framerate? One quick thing to check would be to see if setting the AudioClips' Load Type to Streaming helps at all. I would also check out the Profiler and see if something is taking a lot of time. It could simply be that 8 streams is too many. Alternatively you may find that something else is going on!

Another quick test would be to see if reducing the number of layers helps at all. If not, you might be looking at something like artifacts resulting from resampling or some such.

In a manner of speaking, yes. Chords are special in that they are typically well-known: you can look for "closely packed groups" of note events, inspect the notes, and look up a chord that fits the group (e.g. C+E+G = Major C-chord). This of course doesn't work for every possible chord type, but it would probably get the common ones.

That said, I have reread your initial question and I think I understand what's going on: Koreographer currently does not allow two OneOffs to occur at the same location! This has more to do with not having the UI to show "overlap" than it does an actual internal issue. That said, if you really do need to import every note and you're seeing some simply not come through, try importing the notes as Spans instead! Spans can overlap without issue! And if you're only checking against a KoreographyEvent's StartSample then they're effectively the same as OneOffs.

That said, you could also adjust the MIDI files to offset the note events ever-so-slightly or collapse all of your Spans into OneOffs once they've been imported into Koreographer. The trick here would be to do so with Snap-to-Beat turned off. As long as OneOffs don't appear at the exact same sample position, they can be as close as you'd like (note that a difference of one sample in timing is 0.000022 seconds for a 44100Hz audio file).

Hopefully this helps!

 

Just a (hopefully) simple question. How expensive are calling events? I am targeting mobile platforms and I currently have 8 event tracks (different patterns) firing, and I will soon have a few more. I currently haven't noticed any performance issues by having this many tracks running but thought I should ask before I go too far down the hole of having too many event tracks. Do you have any recommendations when it comes to have multiple tracks?

 

The vast majority (read: pretty much all) of the weight of an event callback is the work that is done within the callback function itself. Koreographer is optimized internally to process the KoreographyTracks with a handful of if-checks per-Track each frame. Effectively, there is minimal overhead for each KoreographyTrack added to the system.

We don't have a specific number to recommend at this point, no. We've not heard any issues about this from our users and in our own test applications we've not seen bottlenecks arise from having too many KoreographyTracks. (One of our applications frequently has five or more Koreography packs loaded at once, each with multiple Koreography Tracks, using the MultiMusicPlayer. This is a desktop game but we'll be exporting it to WebGL.)

This probably wasn't the clean, crisp answer you were looking for but we simply don't have the data at present. In short, the amount of processing overhead that a KoreographyTrack impinges upon the system is extremely small. If you do start to find that the KoreographyTrack processing becomes a burden, please let us know!

 

Thanks very much Eric, that's all (and more ) I wanted to hear, thanks again for your timely response.

 

Any update on this? It's been a while...

 

@sound-and-vision Nothing more to say than that we're drawing close to releasing the next update. We have a few i's to dot and t's to cross still but we are drawing close to a release! Stay tuned!

 

I've read through the posts, but I'm still not totally clear if koreographer can be used at runtime to analyze a song.

Our use case, we want to allow the user to load custom music into the game, play that music, and have the level bounce and react accordingly.

So just to clarify:
1. Is that possible right now? Can the audio data be generated at runtime, on-request. Within a reasonable amount of time (<10 seconds)
2. It looks like the user needs to enter the BPM manually if they import a custom song. Is that still the case? Is it expected to change in this forthcoming update? Ideally users can just throw a bunch of music at the game, and dont have to care what the BPMs are.

Cheers,

 

Hi @shawnblais! Happy to answer your questions!

Unfortunately, no. This is not currently possible with Koreographer. The analysis functions in Koreographer are currently editor-only. It is possible to generate the metadata (Koreography) at runtime but you need to bring your own analysis methods for this. Doable, but we don't currently provide an out-of-the-box solution.

Yes. BPM must be specified by the user. Keep in mind that the tools make this relatively easy, especially if you have a MIDI file that matches your audio (exported from a DAW, for example). This will not change for the upcoming v1.4 release. It is something we are absolutely investigating but it's a ways off. This FAQ entry outlines some of the challenges we're facing as we investigate solutions.

There are several other assets available for Unity that allow for runtime analysis of playing audio to provide some measure of "musical feel". Results with these assets tend to vary widely (a lot of this is due to the same points made in the previously linked FAQ entry).

I hope this is helpful!