Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

htmlTexture plugin

Discussion in 'Made With Unity' started by bliprob, May 15, 2007.

  1. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Hi everyone, total Unity newbie here. This is my first post here.

    Recently we acquired a Unity Pro license and I began to experiment with porting some of our work from TGE to Unity. One of the requirements was to display an animated Flash swf file in a texture.

    So, I did that. Starting from the TexturePlugin sample, I have created a plugin that creates an offscreen webview, loads it from a URL, and copies the bitmap to an OpenGL texture every update. Animated SWFs play well, and since it's using webkit, it uses the real Flash plugin and supports Flash video and all actionscript (unlike GameSWF which supports a limited subset). And since it's just an HTML viewer, you can load HTML into the texture too.

    Here are a couple of pictures:



    You can see it's picking up my local google cookies. (Yes, the scrollbars can be hidden.) And...



    Since we're new to Unity, I have no idea if this would to interesting or useful to others. If there is interest, we would be happy to make a release for community.
     
  2. aaronsullivan

    aaronsullivan

    Joined:
    Nov 10, 2005
    Posts:
    986
    Very cool work. People have been requesting this in the past.

    Just to clarify. What kind of prerequisites will the user have for this functionality? Will it work for Windows?
     
    ankakusu and anuraganand like this.
  3. garyhaus

    garyhaus

    Joined:
    Dec 16, 2006
    Posts:
    601
    OMG yes!!! Please post this! It could be HUGELY useful! Thank you so much in advance!

    Gary
     
    ankakusu likes this.
  4. forestjohnson

    forestjohnson

    Joined:
    Oct 1, 2005
    Posts:
    1,370
    Wow thats really cool. people have been clamoring for this for a While.
     
  5. AaronC

    AaronC

    Joined:
    Mar 6, 2006
    Posts:
    3,552
    Thats stunning! Well done man, you nailed something really cool there.
    AC
     
  6. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    Shame it will be a pro only feature :(
    Maybe the big boys will integrate something of this spiffyness tho. We can only hope.
     
  7. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    aaronsullivan: It uses webkit, so it supports whatever Safari can display (HTML+CSS, Quicktime, Flash, Shockwave, etc., even the Unity web player, for the truly insane). It has a function that lets you call into JavaScript on the page, so you can do setVar/getVar for Flash, or play/pause QuickTime movies, although it's currently not returning a value because the whole string-marshalling-by-casting-to-an-int thing freaked me out.

    Today I've only done the Mac build (it's all we needed for this prototype) but it can be extended to Windows without too much trouble. I briefly looked at the webkit for windows project, and the build process looks hellish, so I'd probably just use an offscreen ActiveX IE control to render the content. The usual IE vs. Safari issues would apply.
     
  8. kaaJ

    kaaJ

    Joined:
    Mar 21, 2007
    Posts:
    150
    Looks great very usefull!

    This will be one of the first things I'll try if I ever can afford a pro license :-D

    .kaaJ
     
  9. shaun

    shaun

    Joined:
    Mar 23, 2007
    Posts:
    728
    This is really neat - Im guessing it only works on Mac for now? How about browser support?

    Are you hooking into some kind of API for Safari and sending that to OpenGL?

    Cheers
    Shaun
     
  10. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Here's another example. The texture displays a Flash SWF that uses your webcam and connects to a remote webconference via the Flash communication server.



    (In this case, both of the videos are the same, although obviously it could be any number of remote videos).

    By the way, that's the Avacaster web conferencing system shown in the texture.
     
    ankakusu likes this.
  11. DaveyJJ

    DaveyJJ

    Joined:
    Mar 24, 2005
    Posts:
    1,558
    Yikes!!! Awesome!!!

    Does this mean I can then pull in a simple HTML formatted table in Kevin's server side php code for high scores?????

    I couldn't echo my returns into a table!

    That would be awesome!!!!!!

    And wait a second, if you can then put a functioning Flash UI on a webpage and pull that onto a texture, you could do Flash-based UIs for the game ... or not?
     
  12. StarManta

    StarManta

    Joined:
    Oct 23, 2006
    Posts:
    8,773
    That's awesome work!

    Does it support Flash webcam sites, too? That'd be pretty swank.
     
  13. Omar Rojo

    Omar Rojo

    Joined:
    Jan 4, 2007
    Posts:
    494
    I think not, the HTML page is saved as a bitmap, and so, losing all functionality, you get all the visuals though. It's a screen-shot.

    .ORG
     
  14. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    StarManta: Sure, that's a live flash webcam site in the Avacast screenshot there.

    DaveyJJ: yes, you could show any HTML page, so your high scores page would work too.

    If you want to give it a try, I finally figured out how to make a working standalone build. Download it from http://robterrell.com/htmlTexture.app.zip. Click the cube to change the web site shown. There's only a few (my wordpress blog page, these forums, the Otee web page, a YouTube video, a DailyMotion video) in there right now. I was going to add a text entry thingy so you could go to any web page, but I can't seem to find one in the Unity manual. Any pointers?

    This is for demo purposes, only, folks, and it's not ready for general use, so please don't go digging the plugin out of the app bundle.
     
  15. drJones

    drJones

    Joined:
    Oct 19, 2005
    Posts:
    1,351
    um, WOW. that's awesome. something like this could open up so many interesting possibilities for novel gameplay interaction, really cool work ; )
     
  16. HiggyB

    HiggyB

    Unity Product Evangelist

    Joined:
    Dec 8, 2006
    Posts:
    6,183
    First off, hi Rob! Welcome to the Unity community. 8) :D I'd like to say kudos for this and ask what performance is like? You mention the rendering of a page with "animated content" (eg: Flash), what sort of frame rates do you get with your plugin (your test app ran great on my MBP)? Also, how large (file size) is your plugin?

    To everyone noting that this fits the feature needs folks have been asking for I must say that it doesn't quite fit the bill. Without a doubt this is a *fantastic* way to render web pages as an image, but this is does not equate to "support for HTML display". I say that because folks tend to want text-based HTML display, clickable links, etc (as opposed to rendering a page as an image). This is a fantastic step in the right direction but it's not quite all the way with respect to the requested feature.

    Edit: please note that I don't mean to take anything away from Rob's work, quite the opposite, he's done a great job quite quickly. It's just that I was trying to measure this against the long-requested feature of HTML rendering support. Sorry if my words above came off wrong to anyone (hope they didn't!). :oops:
     
  17. DaveyJJ

    DaveyJJ

    Joined:
    Mar 24, 2005
    Posts:
    1,558
    Yikes, that is neat. You've got to put that out to the community pronto. Can you provide two sample scenes ... 1 flat plane and one with the cube? I assume that the GUITexture can also be a simple plane rather than your city-fancy cube, right ? :wink:

    Oh, text entry found here ... it works beautifully ...

    http://www.unifycommunity.com/wiki/index.php?title=GUIBuilder

    I use that for a simple GUIText box that I click and enter a name into and then a submit button to post scores.
     
  18. DaveyJJ

    DaveyJJ

    Joined:
    Mar 24, 2005
    Posts:
    1,558
    True. But it's sure a cool demonstration of stuff that can be put to good use in the meantime, Tom. :D
     
  19. HiggyB

    HiggyB

    Unity Product Evangelist

    Joined:
    Dec 8, 2006
    Posts:
    6,183
    See my edit above... :oops:
     
  20. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Whoa. Tough crowd.

    I added a function to the plugin to send click events. (I know, next you'll want keypresses... one step at a time.) I figured I could cast a ray, find the texture pixel where the click happened, and pass that back to the plugin. If someone can help me figure out why this code returns (0,0) that is:

    Code (csharp):
    1.  
    2.         if (Input.GetMouseButtonUp(0)) {
    3.             RaycastHit hit;
    4.             if (Physics.Raycast (Camera.main.ScreenPointToRay(Input.mousePosition), out hit)) {
    5.                 int x = (int) hit.textureCoord.x * width;
    6.                 int y = (int) hit.textureCoord.y * height;
    7.                 htmlTexture_leftclick( x, y );
    8.             }
    9.  
    There is a box collider on the cube in the scene. Shouldn't that do it?
     
  21. dingosmoov

    dingosmoov

    Joined:
    Jul 12, 2005
    Posts:
    559
    Great work, much potential. Regarding your code I don't know. But I would guess you tried "GetMouseButtonDown" ?
     
  22. HiggyB

    HiggyB

    Unity Product Evangelist

    Joined:
    Dec 8, 2006
    Posts:
    6,183
    Sorry, I seem to be the only one that offered anything close to "tough crowd" type comments and I apologize for that (did so in my original post above). It seems you're already looking at how to offer even more functionality (eg: clicks) which is awesome, thanks for sharing.

    As to returning 0,0, where do your width and height variables come from? Are those non-zero/legit values?
     
  23. StarManta

    StarManta

    Joined:
    Oct 23, 2006
    Posts:
    8,773
    It looks like a casting problem to me - it's casting thit.textureCoord.x to an int (which makes it 0) before it multiplies by the width. Maybe try this:
    Code (csharp):
    1.  
    2.             int x = (int) (hit.textureCoord.x * width);
    3.  
    @dingo: The "proper" accepted place to take mouse clicks is in mouse up, not mouse down; that way it doesn't interfere with dragging, etc. Though I would expect that ultimately, bliprob will transmit both mouseup and mousedown as some sites (and especially Flash) rely on both to function properly.
     
  24. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    Could we get a universal binary of that please?
     
  25. bigkahuna

    bigkahuna

    Joined:
    Apr 30, 2006
    Posts:
    5,434
    -Very- cool! Welcome to the community!
     
  26. podperson

    podperson

    Joined:
    Jun 6, 2006
    Posts:
    1,371
    Awesome work... this is really something that Unity should do intrinsically (if only in the pro version) since you've demonstrated there's not much of a performance hit and it's a highly flexible option for building 2D UIs.
     
  27. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Ahhh... it appears that I can cast a ray at the cube and find the intersecting texture coordinates ONLY if I'm using a mesh collider. Which makes sense, when I think about it, because there's not any real relationship between a box collider and the mesh's faces; the box collider can have a totally different center and transform.

    I loaded a web page that shows the coordinates of mouse clicks, and now I get clicks:



    (For the cocoa programmers out there, I'm creating NSEvents for move, down, and up and feeding them into the event queue. The plugin has a function for mousedown, mouseup, and click (convenience function for both down-and-up). I haven't done the right button yet.)

    This is very fun to play with, but I've got to get some other work done, so I will spend a couple more hours on it today and make a build that lets you enter a URL to display. Let me know if there is anything else you'd like to see in a demo.

    Someone else ask to see a GUI texture, so I'll see if I can figure that out too.

    Lallander: Sorry, the next build will be universal.

    Tom: no need for apologies. I've now read the other wishlist threads and I totally understand where you're coming from. (By the way, I sent you an email last week, please let me know if you didn't get it.)
     
  28. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    Excellent news, I look forward to playing around with this.
     
  29. dingosmoov

    dingosmoov

    Joined:
    Jul 12, 2005
    Posts:
    559
    excellent, Excellent, EXCELLENT! Seriously...Wow.

    Can you post a webplayer?
     
  30. HiggyB

    HiggyB

    Unity Product Evangelist

    Joined:
    Dec 8, 2006
    Posts:
    6,183
    Plugins aren't supported in the web player.

    Edit for Rob: reply sent... :)
     
  31. Joe ByDesign

    Joe ByDesign

    Joined:
    Oct 13, 2005
    Posts:
    841
    Just tried the posted download app, VERY COOL! Would love to play around with this plugin.

    It's a brave new world! :)
     
  32. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    I put a new build of the demo app. You can grab it from here. The html page is displayed on a plane (easier to read!). There are "bookmark" spheres that you can click to see various sites. The cube is a "back" button.

    Obligatory screenshot:



    The latest build of the plugin supports scripted mousedown, mouseup, and mousemove actions. (There's a convenience function called "leftmouseclick()" that does all three in the proper order.)

    In the demo app, when you click, I cast a ray to the plane to find the intersection point, convert that to texture pixel coordinates, and pass a mouse click to the plugin. You can actually surf the web in Unity! I'm not hiding the scroll bars, and you can click them too. (In fact, I just read Casemon's post above inside the app, and decided to post a reply... but I don't have keystrokes implemented yet.)

    The plugin also implements back() and forward() functions, which are simply passed to the offscreen browser view for proper handling.

    Performance is about 5% lower because I turned on mouse-move events for the offscreen window. I'm thinking about making that optional. Flash animations (i.e. the YouTube and DailyMotion videos) still seem to play well.

    Please post any additional feature requests. If you've got Unity Pro and would like to test the plugin, send me a message here or in IRC.
     
  33. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Sorry, I'm such a noob. I lost my PMs from the forum. If you messaged me, I can't reply. Please send me an email instead: robterrell@gmail.com.

    Thanks!
     
  34. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    :cry:
     
  35. StarManta

    StarManta

    Joined:
    Oct 23, 2006
    Posts:
    8,773
    Wow, very cool! (Just navigated over to this post, though I couldn't type a reply in it.) Color me impressed.
     
  36. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    Lallander: crap. I forgot. Try the same link now for a new build. Now BOTH the plugin and the unity app are universal.

    edit: ppc is now tested -- and works fine.
     
  37. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    Excellent. That runs quite well.
     
  38. dingosmoov

    dingosmoov

    Joined:
    Jul 12, 2005
    Posts:
    559
    How can plugins be supported by the webplayer? Would the source of the plugin needed to be added to the Unity core code?

    Again Rob, amazing plugin, especially since it uses clicks etc!
     
  39. HiggyB

    HiggyB

    Unity Product Evangelist

    Joined:
    Dec 8, 2006
    Posts:
    6,183
    Yes, something like that. If we included a plugin's code into our source code then it becomes a built-in part of our player (Aegia's PhysX engine as an example, 3rd party source code licensed and rolled into our own source code).

    In the case of .NET DLLs we take the DLL and incorporate it directly into your *.unityweb file so it's downloaded and present on the client machine when they view your content. We could in theory enable similar behavior for C/C++ plugins but we have serious security concerns over doing that. The truth is that .NET DLLs are managed code and therefore don't have the same degree of capabilities as C/C++ plugins, thus they are "safer" to use. In the case of C/C++ plugins you could quite easily write dangerous code and use our player to propagate a virus or do other similarly dangerous/nefarious acts. So for now that door is closed for arbitrary 3rd party plugins in the Unity Web Player.
     
  40. Dranore

    Dranore

    Joined:
    May 23, 2007
    Posts:
    57
    Just wanted to say this is spiffy. If it can be extended to the fanciness of Doom3's flash interfaces, that'd be downright dope.
     
  41. diese440

    diese440

    Joined:
    May 25, 2007
    Posts:
    105
    Great, great !
    Just needs a Windows compatibility to be one of the best Unity plugIn, for the moment.

    Do you think is possible to extend the functionalities to support multi-textures playback ?

    Once again awesome work !
     
  42. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    For my project, I've used this plugin to display Flash interfaces and it works great.

    Sorry, I got busy with internal deadlines. Once I can clear the decks I will definitely make a Windows version of the plugin. Support for multiple textures has been done - I just need to clean it up before releasing it.

    Just curious... is anyone using this plugin for anything real?
     
  43. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
  44. DaveyJJ

    DaveyJJ

    Joined:
    Mar 24, 2005
    Posts:
    1,558
    Hang on ... Flash interfaces? Can you then enter data say on a Flash form on a web page?
     
  45. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    I haven't implemented the passing of keystrokes to the webview, but there's no reason why it can't be added. I just didn't need that.

    I have used the htmlTexture_sendJavascript() function to communicate with a Flash instance on a web page... so you can make a Flash GUI that is controlled by Unity code.
     
  46. bigkahuna

    bigkahuna

    Joined:
    Apr 30, 2006
    Posts:
    5,434
    How's this project coming along? Any new developments?
     
  47. nm8shun

    nm8shun

    Joined:
    Jul 14, 2007
    Posts:
    476
    Sorry to join the party so late on this one, but I'm really intrigued by this tool.

    So as I understand it, you can use a Flash page as a texture (and therefore a GUI). From what I can discern from the script it gets this page via http. Can it also do the same thing accessing a .swf or .html page stored locally?
     
  48. bliprob

    bliprob

    Joined:
    May 13, 2007
    Posts:
    901
    The plugin hasn't changed much in a while -- it now takes an ID parameter so you can use it for multiple textures (i.e. a different URL on each material as needed). I started adding support for keystrokes, but didn't finish because I didn't really need it, and got busy with other things (first the Wiimote plugin, and then another project came along).

    Yes, you can display a local SWF, it accepts an legal URL, i.e. a file:/// reference.

    I have gotten a few emails from people who can't figure out how to use it. If I get some free time I will write a more detailed tutorial on the wiki. Until then, if you need help figuring it out, it works EXACTLY like the Texture Plugin sample, which you can download from the Unity site. Look at the code for that and you'll get it.
     
  49. Lallander

    Lallander

    Joined:
    Apr 23, 2006
    Posts:
    429
    I must say that keystroke support would be fantastic but the project is pure awesome as is.
     
  50. nm8shun

    nm8shun

    Joined:
    Jul 14, 2007
    Posts:
    476
    I'm afraid I need a lot of guidance on this one. So if anyone can chime in I'd appreciate it.

    First, the .bundle that is downloadable on the wiki is only for deployment right? I mean, I as the game author don't actually make use of the plugin directly right?

    Instead, I should be creating the scripts listed on the wiki (http://www.unifycommunity.com/wiki/index.php?title=HtmlTexturePlugin) right?

    When I copy and paste the first sample script, I always get a parsing error and then am unable to apply it to the GUI Texture. Am I missing part of that initial script:


    Code (csharp):
    1.  
    2. public int width =512;
    3. public int height = 512;
    4. void Start() {
    5.     m_Texture = new Texture2D (width, height, TextureFormat.ARGB32, false);
    6.     m_Texture.Apply();
    7.     htmlTexture_start("http://google.com", width, height);
    8.     // put the texture on something
    9.     Transform.renderer.sharedMaterial.mainTexture = m_Texture;
    10. }
    11.  
    12. void Update() {
    13.     htmlTexture_update( m_Texture.GetInstanceID() );
    14. }
    15.  
    16. void OnApplicationQuit() {
    17.     htmlTexture_stop();
    18. }
    19.  
    I tried downloading and reading through the Plugin Texture, but I'm just too new at scripting to understand what I'm doing wrong. Any help from anyone would be appreciated. Thanks.