Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Unicode font rendering broken with Devanagari fonts?

Discussion in 'Editor & General Support' started by chargrove, Jun 15, 2014.

  1. chargrove

    chargrove

    Joined:
    Jun 15, 2014
    Posts:
    1
    Hi there,

    I am very interested in rendering Unicode text involving Devanagari characters (used for languages like Hindi, Sanskrit, etc.).

    http://www.unicode.org/charts/PDF/U0900.pdf

    I have several TTF fonts that have very good support for Devanagari, and which render well in web browsers, Notepad etc.

    However, neither the stock Unity GUI font renderer, nor any of the several font-rendering-related assets that I've downloaded, has managed to render it correctly.

    The issue is with the extended glyphs that these fonts provide to support combinations of characters; these glyphs are beyond the range provided in the PDF linked above, and are used as part of the character->glyph mapping in the font. In the case of Devanagari fonts, these glyphs are critical; the characters in the PDF above are sufficient to define all of the characters needed for the language, but insufficient to render them in a legible manner.

    For example, the simple word "yajña" (in transliterated text) is represented in Devanagari as \u092F (ya), \u091c (ja), \u094d (virama), \u091e (ña). Using a decent Devanagari-enabled font (anything from Arial Unicode MS, to third-party ones like Siddhanta (http://svayambhava.blogspot.com/p/siddhanta-devanagariunicode-open-type.html) ), this renders correctly, because the last three characters (ja, virama, ña) actually resolve to a single glyph, which is a special character used for the character transliterated as "jña". So the correct display of this word would have one glyph for "ya" and one glyph for "jña".

    This isn't an isolated case; there are many such glyphs, and similar techniques are used to handle essential language features like vowel modifiers.

    Unfortunately, when I try to render the above Unicode character sequence in Unity, instead of getting two glyphs, I get three: one for "ya", one for "j" (i.e. ja + virama), and one for "ña". This is structurally correct (a Hindi reader could comprehend it in a pinch), but highly illegible.

    The deal breaker though is when one tries to use vowel modifiers, particularly the vowel modifier "i" (\u093F) which is supposed to be displayed before the consonant in question; in Unity these are showing up afterwards and visually connecting to the following letter, which is incorrect enough that reading becomes impossible.

    Devanagari is important to me, and it works well in a lot of applications. How can I also get it to work well in Unity? Any help would be most appreciated.
     
  2. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    I have no experience with Devanagari. TTFs have a table called GSUB, which performs glyph substitution. So, in a trivial example, it might say replace a followed by e with æ. I have seen this used in Farsi, which has substitution rules for glyphs which appear at the start and end of words. Unity, however, does process the GSUB table, so your strings are rendered as sequences of glyphs that match the characters in the string. Ashkan would had the Farsi problem worked out a solution whereby in c# script code the Farsi substitution rules got implemented. Essentially he has a function that you give a string to, and it returns a string with the correct sequence of characters.
     
  3. abusaadp

    abusaadp

    Joined:
    Aug 16, 2014
    Posts:
    2
    Hi,
    You can use my answer from this link. I am using it in my unity app in hindi and it is working properly.
     
  4. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Hi abusaadp

    We are having the same problems described by chargrove.

    Your script solves one of the problems, and you mention that one can program other substitutions.

    Unfortunately none of us has knowledge in Devanagari and identifying each problem is quite difficult.

    Do you have all the necessary substitutions programmed already? If so, we are willing to pay you for sharing.

    Thank you
     
  5. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Hi Graham

    I guess in your response you meant "Unity, however, does NOT process the GSUB table", correct?

    As a reference, this page explains all the substitutions in Devanagari:
    https://www.microsoft.com/typography/OpenTypeDev/devanagari/intro.htm

    But trying to implement most of them has been impossible. For example these two:

    Halant + Ra = Rakaar : The below-base form of "Ra"

    blwf.gif

    Ra + Halant = Reph : The above-base form of the letter "Ra"
    abwf.gif

    But we are unable to find the glyphs Rakaar or Reph within the font. They might be within the GSUB table but since Unity does not read or process such table they cannot be replaced by code.

    Do you know if Unity has plans to support the GSUB table and render correctly other writing systems such as Devanagari?

    In the meantime, what do you suggest to do?

    Thank you
     
  6. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Um, yes, I meant Unity doesn't support GSUB.

    As I said before, I have no experience with Devanagari. If the font has no entry for Rakaar, then I'd assume it's not used commonly. I haven't even see a Unicode glyph for Rakaar. GSUB is just a substitution table, it won't have glyphs in it.

    I have no knowledge of any development guys at Unity working or planning to work on font processing, so think GSUB support isn't on the cards.
     
  7. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Thank you Graham

    We have no experience with Devanagari but in order to localize our game into Hindi we need to make sure everything reads well.

    Rakaar or Reph are not stand alone characters. They are printed with the previous or next character all-together.

    When I use the same font within Word or even within Unitron, all words are rendered correctly. But not within Unity.

    Is there anyway we can get access to those hidden characters?
     
  8. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Well, where do you think the glyphs are hiding? If the font that you have has those glyph, then I'm sure we can work out how to get them. If the font doesn't have them, then working out how OS font rendering makes them will be needed.
     
  9. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Thank you Graham

    How can we figure out if the missing glyphs are within the font or rendered by the system?
     
  10. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Look to see what glyphs are contained in the font? (On my Mac I usually use FontForge, which is amazing.) What's puzzling is how an OS can know what the Reph looks like if it's not in the font.
     
  11. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Rakaar is also known as Vattu and is U+F30A.
    Reph is also known as Repha and is U+F306.

    Seems to be 22 versions of I-matra, U+F30C onwards. Wow.
     
  12. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Thank you Graham

    Pretty weird. When I open any font with FontForge none of the Devaganari characters are there, not even Chinese, Japanese or Korean which we know for sure are there. Obviously those you mention are not there either. But this must be a FontForge error.

    Now when trying to render any of those characters you mention with Unity, this is what it shows:

    PrivateUse.png
     
  13. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Well, I downloaded the font from the link that @chargrove mentioned. Poking about in FontForge helped me find the unicode values. Now, I think that these glyphs are not intended to be drawn as isolated characters, but rather consumed by the rules embedded into the font. (I've not come across private use glyphs before, but did notice FontForge mention private use.)
     
  14. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Thank you again Graham

    This Siddhanta font does include the missing glyphs. And you are correct, those glyphs are supposed to be rendered together with other glyphs as a single character.

    Is it possible to merge two glyphs in Unity?
     
  15. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Not sure, probably not. This thread is making me think it's best to recommend you to create a custom font that includes the devanagari glyphs you need. (I mean, work out what strings your game will render, and create copies of those strings as rendered on your machine, and break them into glyphs.)
     
  16. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    I think there is hope Graham.

    Just programmed the substitution Ra + Halant = Reph using the new font.

    Now, not only the Reph glyph is rendered, but it is rendered correctly merging it with the next glyph.

    Before:
    Before.png

    After:
    After.png
     
  17. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Wow! That's great news.
     
  18. roopmay

    roopmay

    Joined:
    Feb 20, 2015
    Posts:
    3
    We also have been facing an issue with rendering words with conjunct consonants. Examples of these are words like मस्त, कष्ट,पुन्हा. These words render fine in other editors with a font by the name Mangal. However when we use the same font inside Unity, these words render incorrectly.
    I have been following the messages on this thread. By reading what Bakno and Graham have written it appears that one needs to do these substitutions through code. However building a generic rule for substitutions and then implementing it will be a challenge. More so because Devnagri is used in multiple languages and each of the languages may have exceptions / idiosyncrasies.
    So unless such substitutions are available in the font itself it is practically impossible to write code that will do these substitutions.
    This problem will also surface for other Indian languages (like Gujarati, Bengali which are not written using Devnagri script) where there are conjunct words.
    From Graham's posts it appears that unless we do this ourselves Unity will not render these words.
    Since it is absolutely critical for us to render text in Devnagri, we will be left with no option but to move away from Unity if this is not supported.
    Isn't there a specific font in Unity that one can use? If not has someone figured out the algorithm that needs to be implemented to make this happen?
     
  19. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    The fonts contain a table called GSUB. It instructs a font renderer what sequences of unicode codes to replace. Unity doesn't implement the use of GSUB. My suggestion is really to implement this in script code. I have no idea how complex this actually is. I know with Farsi we were able to help the community come up with a solution.

    Alternatively, and again, I have no experience doing this, if you build is to a platform that supports Plugins, possibly you can perform the font rendering to a texture that Unity has access to. Note that Plugins are not supported on web player.
     
  20. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    This was our final solution...

    We created a text editor in Unity using this font...

    http://svayambhava.blogspot.com/p/siddhanta-devanagariunicode-open-type.html

    And asked the translator to write our translations using this editor.

    Then we use that same font within our Unity game when rendering Hindi translations.

    Actually, we merged that font with the one we regularly use, so we don't need to code the change of font/skin.
     
  21. roopmay

    roopmay

    Joined:
    Feb 20, 2015
    Posts:
    3
    Many thanks, Graham and Bakno. I will spend some more time understanding your replies.
     
  22. Rohini Deshmane

    Rohini Deshmane

    Joined:
    Jan 17, 2015
    Posts:
    1
    I am using Google Input Tools to type in Devanagari http://www.google.com/inputtools/windows/
    This works in all the applications including the mono editor but not in unity. Why do we need to code for Devenagari rendering in Unity when everywhere else it is easily available? Please see the screenshot attached.

    final.png
     
  23. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Well, as I have tried to explain, the font processing in Unity copes with the glyphs, but does not read the GSUB and GPOS tables that font renderers use to convert strings into rendering components.
     
  24. roopmay

    roopmay

    Joined:
    Feb 20, 2015
    Posts:
    3
    We used an editor to type Devnagri text with Siddhanta as the font. We then copied the text from that editor into assets with the font in Unity set to Siddhanta (precisely the same .ttf file placed in the assets folder). However Unity still does not render it.
    Bakno, is this what you did too? Or are we doing something different?
     
  25. Reloaded23

    Reloaded23

    Joined:
    Jan 4, 2014
    Posts:
    27
    We are also facing problem similar to this.

    Currently i am facing problem in displaying hindi words. I have tried using GUIText and NGUI’s UILabel but some characters are not displaying correctly. For eg.

    Sound in hindi is written as but GUi text display’s like this .

    Please correct me if i am doing something wrong. Any kind of help would be appreciated.

    I have posted the question in unity answers as well. You can check http://answers.unity3d.com/questions/953758/multi-language-problem.html
     
  26. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Sorry, I just found your post. What we did is a text editor programmed in Unity, only for devanagari. The translator used this editor to do his job, and sent us the saved text file, which has the correct glyph order for Unity. Do I make any sense?
     
  27. gamedevindia

    gamedevindia

    Joined:
    Jun 17, 2015
    Posts:
    4
    @bakno

    Can you please elaborate? I didn't get the "text editor programmed in Unity" part.
    What exactly should I do to get the Devnagari glyph rendered correctly in Unity3D?
     
  28. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    One image will explain it better...
     

    Attached Files:

  29. gamedevindia

    gamedevindia

    Joined:
    Jun 17, 2015
    Posts:
    4
    Ohh..!! Got it. So do you have this tool for sale somewhere?
     
  30. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    Attached is the Windows editor for you to download.

    We can sell the source-code for a reasonable price. Just send me a private message if you are interested.

    Hopefully, the source code is not necessary, and this tool can help others with the same problem.

    Attachment was bigger than max size allowed. Please use this link to download...

    http://www.baknogames.com/affiliate/DevanagariEditor.zip
     
    Last edited: Jun 29, 2015
  31. ashvinray.khalya

    ashvinray.khalya

    Joined:
    Aug 21, 2014
    Posts:
    1
    @bakno..
    HI. How you manage to write half letter in unity editor.. I am not able to do this.
    Please help me.
     
  32. bakno

    bakno

    Joined:
    Mar 18, 2007
    Posts:
    604
    The soft-keyboard shows all possible glyphs combinations.
     
  33. hsnk

    hsnk

    Joined:
    Jul 24, 2015
    Posts:
    8
    Hi Graham,

    You said
    I happen to know both Farsi and Hindi. Maybe we could use that solution to come up with a long term solution for Hindi?
     
  34. nareshkumaryakkala

    nareshkumaryakkala

    Joined:
    Jan 26, 2013
    Posts:
    3
  35. anirudha_vfx

    anirudha_vfx

    Joined:
    Oct 12, 2013
    Posts:
    11
    I was facing the same problem with Devnagri few months back. The half character glyph was not getting rendered in Unity. I'm was using XML files to read text from external source.
    To get the rendering right, we used the same font for typing in the editor and for rendering in Unity.
    Our setup is simple. Our translator was using MS Word with the "Devnagri-Chanakya" font set. Then the text was copied from editor to the XML. As XML parser have special character delimiters, we used the [CData] tags for the input text.
    We created a new GUI skin in Unity with the same font "Devnagri-Chanakya".
    The text field in Unity with our gui style renders all the glyphs right as they should be.

    If you are using google transliteration, copy the text from google to MS Word. Select the text and change its font to the same font(make sure its unicode) you are going to use in Unity. Than copy the text from MS word to Unity. It should work.
    We tested this on Unity 5.3.1.
     
  36. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    @nareshkumaryakkala , do you have any code that you can share?

    could really use some help in rendering hindi correctly.
     
  37. nareshkumaryakkala

    nareshkumaryakkala

    Joined:
    Jan 26, 2013
    Posts:
    3
    @vivalavida
    My project is not yet finished, Currently i have written rules for Telugu and Tamil.
    Hindi rules are little complex compared to Telugu, i have to work on that.But unfortunately i have moved to some other project and this is on hold. I will let you know when i finish it.
     
  38. obsidian1269

    obsidian1269

    Joined:
    May 12, 2011
    Posts:
    24
    Hello everyone! I too ran into this rather difficult issue recently and I wanted to provide you all with my simple solution. It took me a very long time to figure this out after writing many scripts to make up for the lack of GSUB and GPOS support within Unity. So for those of you still struggling with this issue, here is my solution.

    You should know right off the bat that this solution requires the use of a special (non-unicode) legacy font called "Chanakya" which is available to download here:

    http://indiatyping.com/index.php/download/199-chanakya-font

    Once you have this font installed, you will be able to display Hindi text based on a specially encoded version of the language that tells this font what characters to display and in what order. You are able to encode your current (unicode) Hindi text using a handy online converter found here:

    http://indiatyping.com/index.php/font-converter/unicode-to-chanakya-font-converter

    Once you've converted the text, simply copy it to your destination and you should be good to go! I personally pasted it into Excel, exported to XML, and after importing Chanakya into Unity, am able to load and display Hindi text properly.

    :)
     
  39. Srimasis

    Srimasis

    Joined:
    Feb 2, 2013
    Posts:
    7
  40. sourabhkatkar369

    sourabhkatkar369

    Joined:
    Mar 4, 2017
    Posts:
    1
    @obsidian1269
    Hey obsidian, thanks for easiest solution.Its really helpful...
     
  41. arvunity

    arvunity

    Joined:
    Aug 21, 2013
    Posts:
    1
    @obsidian1269
    Hi, thanks for the easiest solution. But problem is there with
    जंक्शन (half ka) using this method. Do you have any idea about this problem.
     
  42. obsidian1269

    obsidian1269

    Joined:
    May 12, 2011
    Posts:
    24
    I am not aware of any issues with my solution. I am also not able to read Hindi so I don't know what you mean by:
    जंक्शन (half ka)

    Can you explain the problem and show an example with a link to a screenshot? I'd like to know if my solution has issues because I will need to fix the problem if one exists. Thanks!
     
  43. JoystickLab

    JoystickLab

    Joined:
    Mar 18, 2016
    Posts:
    14
    Hello, any idea how can I write Bengali in Unity. Hindi and Bengali have a lot of similarities, so hope there might be a solution.
     
  44. abusaad

    abusaad

    Joined:
    May 14, 2013
    Posts:
    4
  45. vipinsharmathegeek

    vipinsharmathegeek

    Joined:
    Jul 13, 2019
    Posts:
    1
  46. jugnu

    jugnu

    Joined:
    Jul 17, 2012
    Posts:
    28
    Your Solution works well but it has issues on WebGl. But only on chotti ee ki matra and other Matra forms. Do you have a solution for this.
     
  47. karmatha

    karmatha

    Joined:
    Aug 25, 2012
    Posts:
    50
    So very nice to have all these workaround, but the real issue is, is Unity Technologies every going to pick up this GSUP issue? It seems like you are alienating a pretty large portion of your user base by not being able to properly support Devaganari and other font sets.
     
    Inbasagar likes this.
  48. kedoti7228

    kedoti7228

    Joined:
    Feb 6, 2022
    Posts:
    1
    as you know it's very frustrating to write long Hindi sentences. It takes too long and is hard to get your point across in a timely manner. Fortunately, these tools exist to help you write fast with near-native quality. https://foxmaya.com/tools/hindi-voice-typing