Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

SoundSynth - Free audio synthesis framework.

Discussion in 'Assets and Asset Store' started by KulestarUK, Aug 21, 2013.

  1. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Please note this is not quite finished yet and is coming soon - I'll be posting some samples and a webplayer as soon as possible :)

    Update: This project is temporarily frozen whilst we release a major game first. It may be a little while yet but I'll keep you guys posted.

    Hey guys!

    I'm Luke and I'm super proud to introduce you to SoundSynth, an audio synthesis platform which will be free to use anywhere, including in Unity!

    $soundSynth-banner.png


    Firstly, you're probably wondering what makes SoundSynth so special. It contains a multitude of features, so here goes an attempt to sum them up!

    - Performs live synthesis of instruments and vocals (it can sing; If you know of Vocaloid, this is similar).
    - Live text-to-speech. Speak out the players name or their stats for example and make your game lots more personal.
    - A custom markup known as SML, or Speech Markup Language. This is used in the above live text to speech to allow emotion into your dialogue*.
    - No external plugins. Built from scratch so that it works in the webplayer and always will do.
    - Supports any voice or instrument you throw at it. You can make it sing or speak with your own voice, too.
    - Localization. So long as the voice you use supports multiple dialects or languages, or you have voices from other languages, you can potentially localize your speech. Coupled with PowerUI's localization, this can have outstanding effects.
    - Tiny files. SoundSynth was built to perform audio synthesis for a browser based MMO, so we wanted to shave every last bit from our files, and it really shows here!
    - Because it's synthesized, lip-sync becomes amazingly straight forward. There's no need to pre analyse your audio or provide a seperate sync file; a simple callback method tells your game when a certain sound is starting.
    - Nitro (that's our javascript like scripting language) is used to control the synthesizer and to make custom sound effects.
    - Shares code with PowerUI. If you have both, you can make a small space saving.
    - Been in development for 18 months (on and off).


    SoundSynth works with SoundStudio but of course doesn't have to. SoundStudio is the program used to create the files, such as premade songs to play in the background, voices, instruments and more. We intend on distributing some instruments and voices for free so you won't have to pay anything to use this platform.

    Do note that SoundStudio itself will come at a small price of around $50 and has a free trial, which we feel is super well worth the functionality you get from using it!

    * We have plans for a WYSIWYH (what you see is what you hear) SML editor but this may or may not be included in the SoundStudio initial release.


    What about sound quality?

    We've produced an entirely new method of synthesis which we refer to as anchor-driven blended synthesis. Rather than simply tacking waveforms together, 'anchors' describe e.g. the pitch and volume at a particular point in time. These anchors are visually displayed in SoundStudio to form a whole new way of generating audio like so:

    $anchors.png

    -> Vertical height represents the key or pitch.
    -> The colour of the line represents volume.

    This enables fine tuning mainly pitch and volume control which is a vital component of singing and speech. But, waveforms are fixed at a particular pitch, so to correctly synthesize this audio waveforms from two pitches are blended together to produce the exact pitch required at a given point. Combined with nitro also having access to modifying anchors, wonderfully high quality and dynamic synthesis can be achieved.



    We hope you guys are as excited about SoundSynth as we are! We have been in discussions with people from all walks of life about it and fully intend on making it open source so it can go on to benefit, for example, people with speech impairment or speech loss. We're really excited about how far a platform originally designed to run our MMO's background audio has come, and we really hope it can make a difference.


    Please ask any questions at all here or feel free to nudge us if you're keen to give it a try!


    All the best,
    Luke Briggs
    Head developer at Kulestar, UK.
     
    Last edited: Sep 10, 2013
    JanusMirith likes this.
  2. chelnok

    chelnok

    Joined:
    Jul 2, 2012
    Posts:
    680
    Woa! Hope your synth sounds as amazing as your description :) Very interested. Any change to see some videos or something?
     
  3. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Thanks! :D We ran a test a while back of sound quality and some people couldn't tell the difference between the synthetic audio and the real thing which is something we were really pleased about; It requires quite a high quality voice to achieve that but I'd like to build up a few of those and distribute them for free. I'm planning on releasing an example webplayer sometime over the next few weeks so stay tuned for that :D
     
  4. chelnok

    chelnok

    Joined:
    Jul 2, 2012
    Posts:
    680
    Sure thing, thread already subscribed ..betatester available if needed :)
     
  5. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Awesome! Thanks for the support, it's really appreciated :D
     
  6. pixelsteam

    pixelsteam

    Joined:
    May 1, 2009
    Posts:
    924
    Yes very interested in hearing this...
     
  7. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    I'll be getting some examples out as soon as possible (some point in the next few weeks) so stay tuned for those :)

    SoundStudio is undergoing some pretty hefty maintenance at the moment - It's quite a graphical program and is being swapped around to render with OpenGL as it was showing some graphical lag on audio with lots of notes. Once that's all done you guys will be able to see (and hear!) what it's all about :)
     
  8. I am da bawss

    I am da bawss

    Joined:
    Jun 2, 2011
    Posts:
    2,574
    Sounds good, any sample as to assess the quality of the text-to-speech function?
     
  9. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    To save space, text to speech is based on languages being a set of rules alongside nitro (scripting language) which helps with the more complex rules. At the moment there are around 100 rules for general english which makes it ok at the moment but a little choppy on more exotic/ underused words. These rules are stored globally so over time it should improve (and we're actively adding rules at the moment) - an example of text to speech will be posted as soon as possible in a few weeks or so :)
     
  10. YoungDeveloper

    YoungDeveloper

    Joined:
    Jun 28, 2013
    Posts:
    63
    Looks promising, samples would be great :).
     
  11. nixter

    nixter

    Joined:
    Mar 17, 2012
    Posts:
    320
    Can't wait to see...errr, hear more!
     
  12. TylerO

    TylerO

    Joined:
    Aug 21, 2011
    Posts:
    35
    But will it blend?

    Looking forward to it Luke, just as I have since the beginning, surely it's been longer than 18 months? D:
     
  13. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hehe you're probably thinking of GalaTropis; that's been in development on and off for 4 and a half years now! It's quite a large game to say the least and still going :p
     
  14. grizzly

    grizzly

    Joined:
    Dec 5, 2012
    Posts:
    357
    Awesome stuff, Luke! Looking forward to the demo. :D
     
  15. TylerO

    TylerO

    Joined:
    Aug 21, 2011
    Posts:
    35
    Yeah, pretty sure I was thinking of that. :p

    Nao, finish it so that I may dominate in it? >:D
     
  16. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Thanks a lot grizzly! Me too, I really can't wait to see what you guys think of the real thing :D



    Lol if by dominate you mean loose your global power/ nation to me then sure :D Hopefully GalaTropis isn't that far out too - Although it was hit by a pretty significant bug that unfortunately originated from Unity (and actually resulted in a big rewrite of the games code to nitro mentioned above), it should certainly be seeing a release this year :)
     
  17. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey everyone!

    I just wanted to let you know what's currently occuring:

    - I've added a little note about sound quality to the first post :)

    - SoundStudio is being converted to run on OpenGL as it's quite a graphical program. This is quite the task so we're not setting any specific release dates yet but are hoping that it shouldn't be more than a few weeks.

    - At the same time, SoundStudio code is being removed from SoundSynth. At the moment the two are closely related and a preprocessor strips the studio code from SoundSynth to produce the standalone SoundSynth source. This worked great and made developing them significantly quicker until we were considering making SoundSynth an open source project, at which point we realised the two needed to be fully separated.

    Once thats all done I'll be releasing some up to date audio samples so you guys know what to expect :)
     
  18. blueivy

    blueivy

    Joined:
    Mar 4, 2013
    Posts:
    630
    update?
     
  19. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey blueivy! I'm really sorry for the delay - We're getting ready to start releasing a major game so that has caused this project to get frozen temporarily (and all of my time to go elsewhere!). This particular game is intended to use this framework too so it will be resumed once again as soon as we start getting some feedback about the game we're working on.
     
  20. Play_Edu

    Play_Edu

    Joined:
    Jun 10, 2012
    Posts:
    722
    Great job man.Can't wait Any demo or video available???
     
  21. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey! There's no demo just yet as we're focusing on getting a major game out on alpha at the moment but I'll see what I can do - it might be a little while yet though so stay tuned :)
     
  22. blueivy

    blueivy

    Joined:
    Mar 4, 2013
    Posts:
    630
    Update?
     
  23. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Heya!

    We're putting everything into releasing a game at the moment; once that's good to go we'll be right back onto this. Unfortunately we've had to prioritise, but as soon as SoundSynth is ready we'll let you guys know :)
     
  24. blueivy

    blueivy

    Joined:
    Mar 4, 2013
    Posts:
    630
    No problem! This is just something I'm craving for! :D
     
  25. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hehe I know the feeling! This is something I absolutely love working on as it's really showed me the beauty in what sound really is, but sometimes you've gotta do what you've gotta do :p We have a sound team lined up to use this so it will be front and center when the time comes :)
     
  26. ProjectOne

    ProjectOne

    Joined:
    Aug 9, 2010
    Posts:
    442
    Bookmarked
     
  27. IzzySoft

    IzzySoft

    Joined:
    Feb 11, 2013
    Posts:
    376
  28. MoonbladeStudios

    MoonbladeStudios

    Joined:
    Oct 14, 2013
    Posts:
    185
    any news about the project?
     
  29. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Heya!

    Yes actually it did start moving again recently! :D The studio which acts as the interface for creating content for the synth engine is quite graphically intensive, and even with a bunch of optimizations it was pretty laggy when multiple windows were open so it got rebuilt from the ground up - it's now hardware accelerated and back on track again so stay tuned for updates!
     
    MoonbladeStudios likes this.
  30. ZeoWorks

    ZeoWorks

    Joined:
    Dec 13, 2012
    Posts:
    76
    Any updates? :D Super excited!
     
  31. Freddy888

    Freddy888

    Joined:
    Sep 13, 2014
    Posts:
    165
    This looks interesting, any news on how it's going please ?
     
  32. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hiya Freddy!

    We started working with composers who showed a strong interest in working directly with SoundSynth but it's still frozen at the moment unfortunately; we've taken on a lot of commercial projects recently so it's hard to say when we'll have the time available to continue with it. We will absolutely be returning to it because everything that makes SoundSynth special is really important to us too, but please don't depend on it being available anytime soon :)

    Please do let me know if there's anything else I can help out with!
     
  33. Freddy888

    Freddy888

    Joined:
    Sep 13, 2014
    Posts:
    165
    Hey :)

    Thanks for the update and advice, I look forward to what you may come up with :)
     
  34. JanusMirith

    JanusMirith

    Joined:
    Jul 23, 2013
    Posts:
    4
    Thanks for the update, good to hear the issue is that you are having too much work! I look forward to seeing this system in action.
     
  35. drudiverse

    drudiverse

    Joined:
    May 16, 2013
    Posts:
    218
    Sorry, i have to say that this thread is a dissapointment.


    A super exciting announcement of a free synth, available for Unity!

    well.. 17 months later, this thread is abit late, and, well. dissapointing. can't you just put out your alpha project version, we would clear it up. if it was not near release, better not tell a million unity users to be excited about it.
     
  36. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey Druidiverse,

    No problem, I fully understand your point and I'm really sorry that it ended up being this way. Things are, as you might have expected, a little more complicated than that; for the majority of the time, this project has been frozen for a multitude of reasons:

    - A prime piece of portability didn't work properly on iOS, Xbox or PS3 - it was too awkward on those platforms. We had to almost completely rebuild the synth to get it to a point such that it did, as our aim is to reduce size where it matters most.
    - The studio outgrew the graphics API's it was built on top of as it ended up being much more graphical than originally planned. This was the big one. Opening a few instruments at a time made it practically stop; to fix this one, we had to almost completely rebuild the studio too to run it on OpenGL instead. This naturally came with a whole host of other complications. Things started getting pretty messy, so we started over.
    - In the meantime, we ended up taking on lots of other projects and grew in unexpected ways.
    - The technology itself; vocal synthesis is extremely complicated. Emotion is complicated, and multi-lingual synthesis is even worse :) We have of course had the core of this down for a while, but as we're approaching it from a new angle we had to start completely from scratch. Mapping out the English language in particular, with its thousands of regional dialects too, and then condensing each down into a minimal tree of vocal rules is, well, super time consuming. Our original intention was to crowd source this where everybody can help to improve the vocal rules, but of course there needs to be a good initial set to get things going.

    In short, you'd probably be surprised at just how close this thing was to being released when that message first went out. It just had some relatively minor issues which ended up being massive to resolve. As the first of those issues changed the file formats that we used, we decided against allowing multiple versions of files flying around (think of an mp3 that may or may not work on iOS - it just wasn't right).

    However, having said that, this project will eventually push forward again. We all believe in it's core values and can't wait for the day that you speak to a fridge and it's SoundSynth forming the response - a free open world of speech and audio synthesis. It'll just be later than we first hoped.
     
  37. drudiverse

    drudiverse

    Joined:
    May 16, 2013
    Posts:
    218
    Hi Kulestar, sorry for fretting about it, it is abit like an anticlimax! ... Scientists have just discovered the solution to home fusion batteries that can be used for hoverboards! oh no sorry there was a measurement error.

    I'm an audio programmer, just learning that unity has audio synthesis potential, so i am very keen to peek at some sound design engines in Unity.

    It's primarily a speech synth? yes that must be a difficult and grand project. There are comprehensive frequency charts of vowels from different accents and languages, i.e. the frequencies that the filters have to be at, all measured and listed in academic excel tables for different places. I found that it was nice to use 2-3 filters at same not on different octave to have more control of the tone of the sound.

    I thought that it was a synth primarily for sounds.

    I am just starting a DSP code and i am fairly good at understanding other's code and learning from it, I would have been happy to see if you found a solution for some kind of live audio streaming, using 10-20ms buffers of sound that are pre-processed and laid one after the other into sample accurate sound playback that can be controlled live??!?!

    Could unity could do that efficiently? Perhaps a dll would be more convenient, do you know?

    What part of the audio code could not run in ps3, xbox and iOS? a general unity impasse?

    Most text to speech projects don't do too many accents and emotion in the voice?! even ibm didnt quite get there for the emotion imho. the C64 speech synth sounds very fun though!? and a speak and spell voice is a super start for unity, leaps in front of what it can do so far? i could perhaps do a speak and spell program in a day, type in a phrase and it reads phrases? would be cool! If the gui halted the audio processing that sounds rough, sorry about that kulestar, and so you tried to code a shader for it? i.e. a new gui? that would be waaaay difficult. i am avid to know about it :p

    did you get some good bandpass filters with adjusteable curves? i'm quite interested in them.

    It sounds like too much of a heavy project for me to figure out, but you are welcome to send me some elements of it if you want, i would be interested to know how you coded it from A to Z, and i would probably do a spinoff synth and send it back :D

    mostly keen about the audio stream buffer and synth, do you use buffers for having a live audio synthesis?

    rather than opengl gui, you could maybe have one text box with ascii sliders that respond to mouse coordinates, i was thinking of using cubes for the synth panel. i dunno :p it sounds like a very heavy program. if i can give a hand then that's cool.

    Antony
     
    Last edited: Nov 15, 2014
  38. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey Antony!

    Yep you're totally right there - human vocals are complex in part because of the vast number of combinations of sounds and how incredibly well tuned our hearing is too! It only takes a tiny problem with the audio and it immediately sounds wrong.

    As for your questions about filters, SoundSynth actually approaches it from a completely different angle and as a result it doesn't strictly need filters at all. I'll try my best to describe what it does and what makes it different in clear language for anybody else who is interested - wish me luck! :p

    The standard speech synthesis approach involves getting somebody to recite phonemes - these are the building blocks of speech. They'll say every phoneme, for example, "AH" and every combination of phonemes too, like "AH-T" (at). Some languages like Chinese and Japanese are very syllable oriented, which in turn means they have many, many thousands of these combinations. For the best results, all of these recordings must be stored in a big uncompressed "wave table", and this means a single voice takes up a huge amount of space. Even then, the synthesis isn't exactly straight forward - it involves complex frequency analysis and filtering to merge the pairs together, and you end up with something that sounds like a robot horrifically mangling our beautiful languages. Better sounding ones target syllables (which take up even more space) and other approaches include pre-recorded phrases, like "Turn right here" which has almost no flexibility and isn't really synthesis at all!

    So, we decided to chuck phonemes out the window - they have size issues and still sound awful anyway. What if you go down below the phonemes?

    To the tones and beyond!
    We focused on tones for both vocals and instruments. These are essentially a single cycle in the audios waveform; of course, focusing on tones is nothing new - DJ synth decks have been doing this for years. So, what's so special about how SoundSynth uses tones?

    1. Code inclusion. Meet SoundScript, the HLSL of audio.
    We built a new scripting language which is specifically designed to deal with the blending of tones. It's also used for sound effects and also things like controlling events.

    2. Combined with frequency decoupling.
    The tones of an instrument or chunk of speech are almost completely different at high from low frequencies. We wanted to make sure that this sound could be accurately reconstructed at any frequency; the more information you give it, the better range it has. As tones are tiny, it doesn't take much space to provide a tone for each octave.

    3. Combined with notes being represented as a polygon. (one is shown in the first post; we call them notes made from "anchors").
    This provides direct and very detailed control over the frequency and volume which is vital to good quality speech.

    4. Alignment. Tone alignment is vital for producing "harmonies", so we essentially created a relatively simple active shifting system which tries to ensure optimal alignment all the time.

    So, once a tone has been captured, ideally a few of them across a good range, soundscript and sound effects "warble" the sound - essentially make it wobble around with slight variations in frequency, volume and shift - almost immediately giving it a realistic sound. Importantly the scripts themselves are very simplistic as they are tweaking the sound at the anchor level. Or, more technically, fluctuate the volume and frequency almost randomly but at fairly large intervals (the minimum gap between anchors).

    So at this point we gave it a run through with some basic vocal and instrument tones, keeping it fairly simple. We played back a synthetic "ah" (the dentist sound!) and compared to a recorded one and couldn't tell the difference at all. The waveforms looked identicle too. So, once we were pleased with the underlying tonal sounds, we started building on top of it.

    Up we go!
    We then started focusing on the blending between tones, and this is where language specific rules, dialect rules and sometimes speaker specific rules come in too (although these are very rare). These rules consider the transition that occurs in the mouth (or in an instrument) when you move from producing one tone to another. In essence, we focused on the combination of tones and describe the transition with code. It was then time to try something a little more complex - synthesizing a tonal phoneme [not everything an instrument or voice produces is tonal; more on this shortly!].

    Phonemes themselves are interesting for a multitude of reasons. For starters they consist of a series of sounds and transitions and the intensity of these components changes depending on where the phoneme is used. It's this detail which standard synthesizers miss out on. Plus phonemes can be entirely spoken during a transition between other phonemes; shortcuts that produce a similar but fundamentally different sound (e.g. "ha" - both sounds are produced in one downward motion; a normal a but with a much more breathy start). By focusing on tones the level down, we had full control over the phoneme construction. The blending was also mostly taken care of lower down too, however the phoneme pairs introduce additional rules which varies the individual components. Extra rules happen at a syllable level too. At this point we got it to produce a full word, "Hello", and got it to be sung too by nudging the anchors around and introducing a basic vibrato sound effect (which wobbles the frequency around and gets more pronounced towards the end of the sound). Singing is relatively straight forward when you have very simple visual frequency control. We even had a piano playing in the background, and it sounded completely convincing - everything was sounding great, until we started adding more instruments.

    Non-tonal sounds
    Aka plosives, or stops, these things do have a frequency component but it's almost never used because producing one leaves your mouth in a position where it can't repeat the sound again. You can produce an "aah" for as long as you want, but a "c" as in "cow" is a different story; it's a plosive. These are stored and reproduced as-is, because even when sung their frequency component is still almost never used.

    [Try this! Sing the word "pop" without adding fancy stuff to the o. You'll find it's practically impossible! Two plosives means there's only one thing you can add the fancy "singing" stuff to.]

    GDI isn't suited for rendering gradiented lines. Like, at all.
    At this point the amount of notes was getting high and things were beginning to show signs of problems ahead. Loads of optimisations and awkward patching later, the studio was still grinding to a halt when a few instruments were open simultaneously. We decided to dump the renderer all together and jump to OpenGL. It was downhill from here!

    It was also around this point that we found out something awkward - really awkward - SoundScript doesn't run on iOS! Apple just don't allow it - it would be code that gets streamed in, and they don't like it. Not even Google or Mozilla have been allowed to get around it either. As you can see from above, SoundScript is so fundamental to the whole process, and more specifically it's existance in practically everything like voice, language and instrument files. We needed to rework a lot of the synth to be able to work around this. We then ended up with a studio and a synth which were, basically, shattered.

    So after a bit of rebuilding and some tests, we started to see signs that the studio was coming back to life; this time with OpenGL and hardware accelerated. The whole thing was looking great..most of the time! We had traded one issue for many others, including having to drop support on a bunch of studio platforms, and of course had practically rebuilt the studio and synth from scratch as well as changing all the file formats essentially obliterating all of the content we had already created. To retain studio platform support, we decided to do a U turn on OpenGL and turn the studio into a HTML5 program instead. This way it could at least take away all of the awkward rendering issues and replace it with something that worked on just about any platform at all. This is roughly where we are today; it's basically waiting for the change from OpenGL to HTML5 to get finished - in itself its probably something that would take a few weeks to do, but in the meantime we've been taking on other projects that have blocked it up.

    I think that answers most of your questions! To more directly answer some of your other ones:

    Mainly a vocal thing?: Vocals have been our main focus yes - that's because if the synth is good enough to sing in some foreign language, it is probably powerful enough to be a violin, too!

    Bandpass filters: Tone control = these things are really easy to make with soundscript :)

    Live?: Yep absolutely - the fundamental idea is to get this thing saying things like the players name.

    Streamable?: Yep! The files are miniature. Single notes are even smaller; we've been basing a skill in an MMO on players creating music together using soundsynth so this is why it's still important to us that it gets finished.

    Is it efficient?: During tests it took around 5% of the CPU on a pretty basic PC when it's actively synthesizing using Unities current audio API (which has been improved in Unity 5), so it's not too bad at all; From what we could tell, it does less operations than loading an MP3, although this will need to be retested when we finish the changes.

    Buffers?: There is a buffer but it's one that Unity provides. They appear to create and destroy it literally every time too, so I'm hoping fixing things like this is what they mean by audio performance improvements in Unity 5! Other buffers like ones used for an echo sound effect can be created in soundscript.

    Where does emotion come from?: Great question - the tone set is simply recorded in each emotion, plus emotion often consists of emphasis in different places or words being stretched out (emphasis below). Tones are tiny, so this doesn't take up much space - by being so compact, it can easily gain extra detail like this. Another part of SoundSynth, "Speech markup language", SML, describes which emotion to use in a CSS like way:

    <speech style="speed:90%;teeth:clenched;word-emphasis:strong;volume:loud;">I'm <emp>angry</emp>!</speech>

    We played around with the idea of a "what you see is what you hear" editor which generates SML; this would be where e.g. red text = anger, and large text = it's louder, but of course something like that would be lots more work!

    Thanks for the offer to help! :) The synth and studio are basically a mess at the moment (as you can see above!) and the real bulk of the work is in collecting tones from as many instruments as we can as well as catching all of those speech rules - if you have any ideas on catching every speech rule in English syllables (e.g. "cat" starts with a "k", or the two t's in "tat" are two different phonics of the t phoneme) then that would be greatly appreciated. We believe there's a few thousand of them at least; the awkward thing is our brains must already know what they all are - we just can't exactly get them out!
     
    Last edited: Nov 15, 2014
    blueivy likes this.
  39. drudiverse

    drudiverse

    Joined:
    May 16, 2013
    Posts:
    218
    Wow that's a mega project. Thanks for the awsome explanations there. I will have to come back to this and think of the voice synth processing as it's super complex. at least it seems that unity does all the audio in 16 bit, which i just have read enables 96db dynamic range of the sound files.

    i will have to re read and return to this in free time, it's very broad! You certainly seem to have organised the project very much, you will prevail well with it!
     
  40. blueivy

    blueivy

    Joined:
    Mar 4, 2013
    Posts:
    630
    Is it possible that you could release a YouTube video of the synth saying a few phrases?
     
  41. drudiverse

    drudiverse

    Joined:
    May 16, 2013
    Posts:
    218
    Fascinating, that sounds very complex. It's a scientific study, managing all the tonal and timbral and frequency compnenents of speech. vowel morphing is fun, as you can take any synthesizer sound and apply a sound on it using FFT.

    one of teh most fascinating things i have found in audio once was sound morphing in between two sounds, for example a crow call and a cat miaow, taking all the frequencies of either sound and mixing each one together independantly, i dont know they did it but it is very fun. I have written a vowel filter which can make any sound into aa ee oo etc, i dont know how to make phrases with st and c and other consonants, if it's easy i will see if i can make one from unity, same as with the C64 they had a speech synth on it.

    good luck!!!
     
  42. blueivy

    blueivy

    Joined:
    Mar 4, 2013
    Posts:
    630
    This would be really useful tool and it seems a lot of research went into this. Is it still frozen?
     
  43. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey! Yes it is, however things are possibly showing signs of changing as one of the major internal projects which has been blocking it for a while now [PowerUI's "InfiniText"] very recently left active development (just yesterday). No promises yet though of course - We invested a lot in that project too so Soundsynth depends on how well it goes down. Soundscript did see a little bit of motion but it's still very much considered a frozen project overall. One other important thing is it's now seeing internal competition with a relatively newer project called the Blade engine, however Soundsynth is certainly climbing up the priorities at the moment :)
     
    blueivy likes this.
  44. jerotas

    jerotas

    Joined:
    Sep 4, 2011
    Posts:
    5,572
    Sounds cool!
     
  45. nosyrbllewe

    nosyrbllewe

    Joined:
    Oct 18, 2012
    Posts:
    182
    Hey, I was just wondering. Has there been any progress made recently? I am anxious to find out.
     
    blueivy likes this.
  46. KulestarUK

    KulestarUK

    Joined:
    Aug 16, 2013
    Posts:
    269
    Hey! I'd like to apologise (again!) about the silence on this project - it was originally intended to be a big flagship project, but as soon as it started hitting rendering issues (fairly complete history above, along with a description of how it works), it got bumped out of the way by then smaller ones - they've grown into something much bigger since. But yes, there has actually been some pretty awesome motion which will get carried forward over the next few days. It turns out that the new kid on the block, the Blade Engine, which looked like it was threatening Soundstudio/ Soundsynth when it first showed up, actually introduced some functionality and a mental model which is extremely useful to Soundstudio too. In short, Blade is a graphics system which uses lots of curves, and digital audio is nothing but curves - So a happy relationship was born :p

    Instrument tones as mentioned above are the largest component of the file sizes and Blade will potentially help compress those a lot, plus the notes (made from anchors seen above) will also fully support curves too resulting in more direct control over subtle changes (rather than previously depending on SFX to make those subtle changes - we ended up trying to reduce dependency on SFX as much as possible due to that original iOS problem). Assuming everything goes through, it would be awesome if we could at least have something to demo soon - that's not a promise though! Please don't rely on this project being available anytime soon as it is still low priority/ frozen compared to other ones :)
     
    JanusMirith and blueivy like this.
  47. JanusMirith

    JanusMirith

    Joined:
    Jul 23, 2013
    Posts:
    4
    Thanks for the update, I do really look forward to seeing this system in action.
     
  48. cxode

    cxode

    Joined:
    Jun 7, 2017
    Posts:
    268
    Is it safe to assume this project is dead forever?