Berkelium plugin: rendering interactive websites

Discussion in 'Showcase' started by JeDi, Jul 6, 2010.

  1. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    You can use Berkelium_Window_executeJavascript to execute a javascript function. We still have to implement javascript calls with a return value to support communication in both directions.
     
  2. ccross20

    ccross20

    New Member

    Joined:
    Jul 12, 2010
    Messages:
    5
    Thanks, I'll give that a try.
     
  3. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    I updated the plugin in git, it uses dirty rectangle drawing now and should be a lot more efficient.

    I'm leaving on a holiday now, but I'll try to follow the discussion a bit. Implementing stuff will be more difficult.

    Greetz,
    JeDi
     
  4. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    What's the current difference between the main branch and reissgrant's fork right now besides an alpha key color?
     
  5. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    I merged everything except the alpha key color functionality.

    I implemented a couple of things a bit differently though:
    - I added support for both the backspace and return keys, and we now send the correct pressed/release event
    - Because there can be multiple draw calls per update (one for each dirty rectangle), I implemented a callback system that sends a signal to unity each time a rect of pixels needs to be set

    The callback system will also come in handy once we implement callbacks from JavaScript functions
     
  6. waltermana

    waltermana

    New Member

    Joined:
    Jun 8, 2010
    Messages:
    172
    i was wondering if you were ever planning on adding mac compatibility to yours??? the other plugin isnt really being supported anymore and leaves alot to be wanted
     
  7. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    Great, you guys are superstars.
     
  8. reissgrant

    reissgrant

    Member

    Joined:
    Aug 20, 2009
    Messages:
    727
    I'll modify my fork for now to eliminate confusion, as it was intended to be merged with JeDi's changes.

    *EDIT modified my fork to direct everyone to the master branch.
     
  9. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    I have a mac myself, but I am leaving on a holiday to America for almost a month. Berkelium is supported on mac, so in theory it is definitely possible. There is no easy precompiled bundle on mac like there is for windows though, so it would involve building Berkelium ourselves.

    What is wrong with htmlTexture by the way? I just had a very quick look at it, but it seemed to work OK by me.
     
  10. AliAzin

    AliAzin

    Member

    Joined:
    Jun 25, 2009
    Messages:
    49
    Awesome job.
    I've tested it, but I have a problem. when I play the unity editor on test project everything works correctly but when I want to stop it, unity crashes. also in build, when I want to do something like loading a new level, unity stops working. Does anyone have same problems?
     
  11. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    Hmm, wonder if it's a problem w/ the library issuing a callback when Unity isn't ready for it. Does the page you're viewing when it crashes have anything moving on it? Flash add, flashing text, anything that would cause a rect to be marked as dirty?
     
  12. AliAzin

    AliAzin

    Member

    Joined:
    Jun 25, 2009
    Messages:
    49
    No, I've simply load the unity docs from local. I have no problem when its running and I can go through all links, but as soon as I press the play button to stop the editor, it crashes.
     
  13. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    Try calling Berkelium_destroy() in OnApplicationQuit and see if that helps any.
     
  14. AliAzin

    AliAzin

    Member

    Joined:
    Jun 25, 2009
    Messages:
    49
    No, that doesn't help. I think the "Berkelium_destroy()" is the problem itself. because when I call it the same crash happens .

    I tested it on several machines but it always does the same.
     
  15. reissgrant

    reissgrant

    Member

    Joined:
    Aug 20, 2009
    Messages:
    727
    Yes, in my C# script I commented out this line and the crash stops.
    Code (csharp):
    1.  
    2.     void OnDisable() {
    3.         // Destroy the web window
    4.           //Berkelium_Window_destroy(m_Texture.GetInstanceID()); <-- coment out
    5.  
    I'll have to debug further but this will stop the crash for now.

    I think it's a matter of both C# and C++ trying to do deconstructing and cleanup simultaneously.
    Awesomium had the same issue.
     
  16. AliAzin

    AliAzin

    Member

    Joined:
    Jun 25, 2009
    Messages:
    49
    It doesn't work for me. but I commented out the
    Code (csharp):
    1. DLLimport ...
    2. ... Berkelium_destroy();
    and two other lines in OnDestroy() and OnApplicationQuit() functions, and the crash gone.
     
  17. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    I'm sorry for the brief report, but I'm in the airport on my iPod.

    The crash is caused by the new beta version of berkelium and will be resolved once we use a newer version.
     
  18. AliAzin

    AliAzin

    Member

    Joined:
    Jun 25, 2009
    Messages:
    49
    Thank you very much for your help,it has been very helpful.
     
  19. waltermana

    waltermana

    New Member

    Joined:
    Jun 8, 2010
    Messages:
    172
    the problem isnt it working - i have it working just fine inside my scene - the issue is that its not still be updated and maintainted - for instance if i want to go to cnn.com, the flash components dont really work right and there is no way to scroll down the page

    i guess my question would be then is there any plan to make it all packaged right for mac - i dream of having enough time to do it on my own :p
     
  20. Pulov

    Pulov

    Member

    Joined:
    Feb 20, 2010
    Messages:
    573
    Hi
    this looks amazing, at first did not saw any utility for my project but now that I've seen the transparency screenshot an idea has come to my mind.

    It could be used to build fancy menus with flash with a 3d background. But probably it would be "hard" to control unity from flash... dont know, I'm just an artist... If possible this would be an awesome step forwar in the accesibility to the UI building for artists.
    _ __ _ _ _
     
  21. Dranore

    Dranore

    New Member

    Joined:
    May 23, 2007
    Messages:
    57
    Is it safe to assume this will NOT work with the iPhone?
     
  22. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    It will not work w/ the iPhone. It calls an external (Windows only right now) dll.
     
  23. zhuangfengzi

    zhuangfengzi

    New Member

    Joined:
    Feb 7, 2010
    Messages:
    11
    WHAT'S HAPPENED?
    DllNotFoundException: E:/UnityBerkeliumPlugin/UnityBerkeliumPlugin/test-project/Assets/Plugins/UnityBerkeliumPlugin.dll
    UnityBerkelium.Start () (at Assets\Plugins\UnityBerkelium.cs:43)


    DllNotFoundException: E:/UnityBerkeliumPlugin/UnityBerkeliumPlugin/test-project/Assets/Plugins/UnityBerkeliumPlugin.dll
    UnityBerkelium.Update () (at Assets\Plugins\UnityBerkelium.cs:90)
     

    Attached Files:

  24. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Hi,

    sorry for the late reply, but I was in the States the last month, and in the National Parks the last couple of weeks (so no decent internet connection).

    About the error in the editor: did you copy the berkelium binaries to your editor's directory (in Program Files)? This is not a very clean solution, but for now the only way to get the plugin to work inside the editor (as far as I know).

    There is a new Berkelium version available at http://github.com/sirikata/berkelium/downloads
    It seems that there also is a mac and linux version available for download now, so we could make a windows+mac bundle for the plugin.

    I won't be in my office until the 23th of August, because I am going to GDC Europe and Gamescon, but if anyone else is interested in trying the new Berkelium version, be my guest ;)

    Greetz,
    JeDi
     
  25. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Hi,

    I updated UnityBerkeliumPlugin to the latest version of Berkelium. It now has support for calling Unity functions from javascript (still very simple).

    I added a simple test page to show the functionality.

    I also cleaned up the code a bit. You'll see when you have a look at the test project.

    As always, if you want this to work in the editor, don't forget to copy all files from berkelium/bin to your editor's directory (defaults to C:\Program Files\Unity\Editor).

    Greetz,
    JeDi
     
  26. niosop2

    niosop2

    Member

    Joined:
    Jul 23, 2009
    Messages:
    1,028
    Great job! Thanks again for your hard work.
     
  27. gokhanertem

    gokhanertem

    New Member

    Joined:
    Aug 31, 2009
    Messages:
    95
    this is very nice , thank you for this share
     
  28. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    It seems that the demo project always stalls on the second(!) run (within the editor) immediately after hitting the play button.

    The log just says:
    First reference to Berkelium initialized: initializing the library

    The first run works completely fine though.

    Win7, latest revision

    Thank you for this nice lib!

    xad

    PS: Scrolling (e.g. on www.google.de) results in some visual artifacts when build. Looks like the dirty rectangle is not computed properly (?).
     
  29. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Hi,

    Thanks for trying the new build.

    The scrolling artifacts are normal because I didn't implement scrolling yet (which is handled separately in Berkelium). When you do page scrolling everything should be working fine, because then the whole page is updated. I am working on the scrolling implementation right now, so I will keep you updated.

    I have the same problem with stalling the second time I press play. I am developing in vmware though, and I had this problem before while it worked fine when running Windows natively. It seems it is a real problem now, so I will investigate and try to fix that as soon as possible.

    Greetings,
    JeDi
     
  30. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Thank you for your fast reply and confirmation. These kind of bugs are hard to track.

    If you have some suggestions how to debug the plugin I´m willing to help out.

    xad
     
  31. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    The plugin hangs the second time, in the initialization function of Berkelium. I sent a message to the Sirikata Dev List about it.

    I am going to do a quick test and see if this also happens outside of Unity (so initializing Berkelium, creating a window, destroying the window, destroying Berkelium and then initialize it again).
     
  32. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
  33. Korry

    Korry

    New Member

    Joined:
    Sep 4, 2010
    Messages:
    1
    So i can get the plug in to work but i can't type. but i can type in your test app what do you do to get it to work?
     
  34. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Did you download the latest version from http://github.com/jdierckx/UnityBerkeliumPlugin (using the Download Source button)?

    If you want it to work in the editor, don't forget to copy all files from berkelium/bin to your editor directory (defaults to C:\Program Files\Unity\Editor in windows).
     
  35. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Hi,

    Currently Unity crashes or just does not execute the script when I call executeJavascript() with a string size >= 128.

    Since it ends up in a hard crash it seems to be a low level conversion issue between C#->C++->Berkelium. I remember that the guy from awesomium had a similar issue in the past due to inconsistent runtime settings. Can anybody confirm that?

    xad
     
  36. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Ok, short update to my last bug report. It seems to be less problematic. The executejavascript method in C++ allocates only wchar array of 100 letter which was exceeded in my case. Changing this value to 1024 fixed this issue. Since it crashes if the string is longer then the given string I think the current conversion should be made more robust...

    xad
     
  37. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Oh, I see. To be honest, the code to convert to a wstring was provided by someone else and I didn't really had a close look at it.

    I think we can easily change the code to use a heap array that allocates just enough memory for the javascript string. I don't know if there is a better way to do the string conversion...

    I will try this now, and commit the fix if it works (it probably will).

    Edit: I committed the fix.
     
  38. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Thank you very much, I will test the latest changes ASAP. :)

    One more thing I recognized:
    I execute a js command eg. every second to update my HUD (800x600 texture which shows some simple icons like a map). Using the profiler I could confirm that the RenderWebsite.update() command in these frames jumps from <1ms to 73ms for this frame which leads to a significant lag during animations (even on my high end machine I´m working here). I checked out a MT solution like from here but since the texture is written in C++ it seems to be necessary to deal with MT on this end instead (I got invalid Direct3D accesses here from C++ side...). Any idea how to approach this problem?

    xad
     
  39. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    The problem is that the SetPixels and Apply calls are very expensive, but it works in both DirectX and OpenGL.

    If we want rendering to be much more efficient, we could force OpenGL rendering and update the texture in C++. The code would be simpler too (including scrolling, which I can't get to work easily using the current method), but we would loose DirectX compatibility.

    For now, try to keep dirty rectangles as small as possible. Are you using flash? Because one of the problems there is that it always marks the entire flash rectangle as dirty.
     
  40. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Ah, thank you for the information. I see now what to look for.
    I´m using javascript code on the HTML side only to position some small icons on a radar, no flash involved. The position data etc. is translated to a JSON object and then applied in HTML via js. Most of the time there are around five icons and about four numbers located all over the interface. Shouldn´t be that bad and actually isn´t a problem in the webbrowser directly and with an earlier awesomium implementation...

    xad
     
  41. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Was the Awesomium implementation using SetPixels, or did it update the OpenGL texture directly?

    I would love to do some profiling of the different things involved in this implementation (using callbacks, using SetPixels, ...) once I can find some free time (or money) to do so.
     
  42. xadhoom

    xadhoom

    New Member

    Joined:
    Sep 2, 2010
    Messages:
    91
    Hmm, I cannot answer that from the top of my head.

    A very interesting read is the approach used by MyGUI for their Berkelium plugin (they also support Awesomium). They use a Buffer object to do the texture updating (and dirty rectangling stuff). This is of course completely done in C++.

    xad
     
  43. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Yes, MyGUI has integrated Berkelium, but they use the OGRE rendering engine, which has support for fast access to texture objects, using C++. They can also directly copy the buffer because OGRE supports the texture format directly.

    The problem with SetPixels is that we have to provide it with a color array in stead of a buffer, so we have to convert the buffer pixel by pixel. When using OpenGL directly, we can also use the buffer directly and that would bring a huge speed increase.

    Maybe Unity3 has better support for updating textures with different texture formats?
     
  44. reissgrant

    reissgrant

    Member

    Joined:
    Aug 20, 2009
    Messages:
    727
    This has been discussed many times, and the conclusion is, that until we can make directX calls, we are stuck with "setPixels()". Unless we use OpenGL but who wants to limit themselves to that?

    The reason we can't make directX calls is because "there is no way to get the video device pointer outside of Unity" as written by Aras. Requests for this pointer have gone unanswered. I think with UnityAR coming soon, SOMETHING must be done because AR will not run smoothly (or, as nearly smooth as it could) without these directX calls.

    As for going through each pixel, I strongly recommend including the lookup table I suggested (and added to the berkelium source, at one point) so that you don't have to calculate the float value each channel of each pixel, rather you read the value from a precalculated lookup table.

    As well, you can try to store the old buffer and memcmp it to the new "dirty" rectangle and determine if it is in fact dirty.
     
  45. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Yep, I will add the lookup table you implemented, I totally forgot about that :) Will that increase performance a lot though? I thought that would only still matter in mobile devices and such...

    As for the dirty rectangles, I did some quick checks and it seems that it only reports rectangles that are indeed dirty, apart from flash which always reports its rectangle as dirty.

    Thanks for the hints.
     
  46. Renzokuken

    Renzokuken

    New Member

    Joined:
    Jul 17, 2009
    Messages:
    26
    Interesting, This could revolutionize puzzle based gaming, or even gaming full stop, for example making a first person shooter and having a force field over a door with a key code entry, a website made specifically to the game with video logs could be implemented, say the player gets a log name to search for and from the log entry they get the code, this would allow more fluid gaming in both single and multiplayer gaming. Well done!
     
  47. JeDi

    JeDi

    New Member

    Joined:
    Jun 16, 2010
    Messages:
    52
    Turns out you have to explicitly focus the web window (I thought that was done automatically when sending click events).

    I added that functionality, and now text carets are correctly rendered when editing text fields.

    Haven't had the time to incorporate the lookup table. Maybe later today...
     
  48. rasmusbs

    rasmusbs

    New Member

    Joined:
    Sep 14, 2010
    Messages:
    12
    How is the work in progress on the scrolling feature?
    I'm currently working on a project where we really want to implement a browser feature for different infoboxes on various objects. Hopefully we can keep our data in plain html/flash content and use a browser for displaying inside unity.

    Great work you guys have done so far!
     
    Last edited: Oct 11, 2010
  49. deskchicken

    deskchicken

    New Member

    Joined:
    Oct 10, 2008
    Messages:
    9
    This is very cool. Is this still in development, and are there any plans to wrap it for OSX?
     
  50. chirhotec

    chirhotec

    New Member

    Joined:
    Mar 30, 2010
    Messages:
    39
    I wanted to get this working with Unity GUI, and fortunately, basic compatibility wasn't too hard.

    All I had to do was add a read-only accessor to the Texture2D in RenderWebsite. Since I wanted to attach this to an object with a renderer (my HUD is tied into my First Person Controller), I created an enum so designers could specifically set the desired render type (object renderer, gui texture, or generic render to texture). Displaying it with Unity GUI was pretty easy once I had access to the texture in the other scripts. The only caveat is that by default the texture is flipped vertically (it was doing the same thing for GUI Texture). In my GUI Code I used the GUIUtility.ScaleAroundPivot to fix this. (I put all this in the BerkeliumTestGui Script)

    Attached is your original demo, but now with three scenes, demonstrating each rendering usage. Just as a warning, I moved all the files around to better match my project structure (I have multiple plugins, and wanted to separate the scripts that are only needed for demo purposes).


    Now, getting input capture to work with the texture when in Unity GUI (or GUI Texture).... any ideas? It gets the keyboard input just fine, but not mouse clicks.
     

    Attached Files: