Display outside windows (like command-line application) inside scene?

Hi there,
I was wondering, after seeing a few of the excellent EditorVR features (like the windows you can place in the scene while editing), how to display an outside window like a Windows CMD window with a scrolling command-line application. I know this is not exactly EditorVR-specific but I have real trouble solving this question.

For context, I try to build a very simple scene, a room like a control room with walls full of scrolling and working command-line applications (built on python). There doesn't even need to be any kind of interaction, it should just be a cmd.exe running on the same pc.

Thank you for any pointers!

 

Hi @maxgent,

This can definitely be done with importing win32 specific functions into C# that allow you to capture a window. After you've done that you could craft a workspace similiar to ConsoleWorkspace that will allow you to render the results. Hooking back up input support to control the mouse pointer will require a little work, but is doable, too, also with win32 calls. I hope that points you in the right direction.

Cheers,
Amir

 

Hi @amirebrahimi_unity,
many thanks for your advice. I had seen such an approach but was put off by the complexity of implementing this. I will give it a new try as I've put some more work into the scripting environment. I'm coming from python so this is definitely hard for me

Thanks!

 

Hey there! I work on an app called V which might help. V is a universal overlay which injects itself into any Oculus VR application. We currently only support the Oculus SDK, and only display an embedded web browser, but cross-platform support for any kind of application is the end-goal. I've tested V on Oculus in EditorVR and it actually works great! You can view the docs, YouTube tutorials, or any kind of web content in a moveable 2D panel inside EVR or any other VR app.

Head over to http://www.hellov.io/ and give it a shot

P.S. there's no copy-paste support yet for URL's, so I've been using a TinyURL for the EVR docs tinyurl.com/evrdoc.

 

Hello @amirebrahimi_unity,

I have done some googling and managed to cobble together the following script from a few demos I found. With this script, the feature that @maxgent is looking for is partially implemented. That is, I can start a child app (in my case, another Unity app with an empty scene and a spinning cube) from my parent app and it renders inside my parent app. However, with what I have right now, there are some issues dealing with what seems to be layering/render ordering.

When the child app is loading, I see the Unity splash screen flickering on and off and when it's done, I only see the parent app's scene. However, if I alt-tab to another application and then back to the parent app, I see the child app rendering on top of the parent app's scene. If I move the mouse around and click and drag here and there, the child app's window within the parent app flickers occasionally.

So, my question is - how can I manage the render order of the child app's hwnd contents compared to my parent app's scene? Ultimately, what I want is for the parent scene stuff to be rendered on top of the child app's window (the parent app would act as a UI for the child app, for all intents and purposes with some sprites/UI elements here and there but a transparent view to the child app underneath).

Any help would be greatly appreciated!

Code (CSharp):

1. using UnityEngine;
2. using System.Runtime.InteropServices;
3. using System;
4. using System.Diagnostics;
5. using System.Collections;
6. using System.Threading;
7.  
8. public class ParentWindow : MonoBehaviour {
9.  
10.     [DllImport("User32.dll")]
11.     private static extern bool MoveWindow(IntPtr handle, int x, int y, int width, int height, bool redraw);
12.  
13.     private delegate int WindowEnumProc(IntPtr hwnd, IntPtr lparam);
14.     [DllImport("user32.dll")]
15.     private static extern bool EnumChildWindows(IntPtr hwnd, WindowEnumProc func, IntPtr lParam);
16.  
17.     [DllImport("user32.dll")]
18.     private static extern int SendMessage(IntPtr hWnd, int msg, IntPtr wParam, IntPtr lParam);
19.  
20.     [DllImport("user32.dll")]
21.     private static extern IntPtr GetActiveWindow();
22.  
23.     [DllImport("user32.dll")]
24.     private static extern IntPtr GetWindowLongPtr(IntPtr hWnd, int msg);
25.  
26.     private Process process;
27.     private IntPtr unityHWND = IntPtr.Zero;
28.  
29.     private const int WM_ACTIVATE = 0x0006;
30.     private const int GWLP_USERDATA = -21;
31.     private readonly IntPtr WA_ACTIVE = new IntPtr(1);
32.     private readonly IntPtr WA_INACTIVE = new IntPtr(0);
33.  
34.     private void Start () {
35.         try
36.         {
37.             process = new Process();
38.             process.StartInfo.FileName = "Child.exe";
39.             process.StartInfo.Arguments = "-parentHWND " + GetActiveWindow().ToInt32() + " " + Environment.CommandLine;
40.             process.StartInfo.UseShellExecute = true;
41.             process.StartInfo.CreateNoWindow = true;
42.  
43.             process.Start();
44.  
45.             process.WaitForInputIdle();
46.             // Doesn't work for some reason ?!
47.             //unityHWND = process.MainWindowHandle;
48.             EnumChildWindows(GetActiveWindow(), WindowEnum, IntPtr.Zero);
49.         }
50.         catch (Exception ex)
51.         {
52.             UnityEngine.Debug.Log(ex.Message + "\nCheck if path to Child.exe.");
53.         }
54.  
55.         StartCoroutine(Co_InitWindow());
56.     }
57.  
58.     private void OnDestroy()
59.     {
60.         try
61.         {
62.             process.CloseMainWindow();
63.  
64.             Thread.Sleep(1000);
65.             while (process.HasExited == false)
66.                 process.Kill();
67.         }
68.         catch (Exception)
69.         {
70.  
71.         }
72.     }
73.  
74.     private void ActivateUnityWindow()
75.     {
76.         ParentWindow.SendMessage(unityHWND, WM_ACTIVATE, WA_ACTIVE, IntPtr.Zero);
77.     }
78.  
79.     private void DeactivateUnityWindow()
80.     {
81.         ParentWindow.SendMessage(unityHWND, WM_ACTIVATE, WA_INACTIVE, IntPtr.Zero);
82.     }
83.  
84.     private int WindowEnum(IntPtr hwnd, IntPtr lparam)
85.     {
86.         unityHWND = hwnd;
87.         ActivateUnityWindow();
88.         return 0;
89.     }
90.  
91.     private IEnumerator Co_InitWindow()
92.     {
93.         /*
94.         while (true)
95.         {
96.             int userdata = GetWindowLongPtr(unityHWND, GWLP_USERDATA).ToInt32();
97.             if ((userdata & 2) == 2)
98.             {
99.                 // unity splash over
100.                 break;
101.             }
102.  
103.             yield return null;
104.         }
105.         */
106.  
107.         yield return new WaitForSeconds(5.0f);
108.  
109.         GetComponent<Camera>().enabled = false;
110.  
111.         MoveWindow(unityHWND, 0, 0, 1280, 720, true);
112.         ActivateUnityWindow();
113.     }
114. }
115.  

 

@MR_asilva -- interested to see what you've got as I'm not fully tracking what you're trying to do. Do you mind recording a video?

 

@amirebrahimi_unity,

What I'm trying to do is create what is essentially a steam or xbox platform/game loader where it loads child Unity apps and displays them (using the -parentHWND flag), but can also render its own UI on top of the child app.

The -parentHWND flag seems to be well-defined in how it should work for what it is meant to be used for (i.e. loading a Unity panel or window as a 3D rendering viewport inside of a non-Unity app). However, I can't seem to find any information about what is the expected behavior if you try to parent a Unity app to another Unity app by launching it from within the parent app with that flag as I am doing in the sample script above. What I would need to know is how do I access/manipulate the surface/texture/rendertarget/whatever that the child app is rendering to? Because in my test case, the parent and child Unity apps seem to be fighting over who gets to be shown at any given time and there's no way for me to tell them which should draw above the other.

Unfortunately, I do not have a quick project I can give you as I was getting nowhere with this solution and tried to solve it another way, but ended up finding another solution, which looked like it was going to work, but I am now running into issues with it as well (the technique used here: https://forum.unity3d.com/threads/s...dow-with-opaque-contents-lwa_colorkey.323057/). I will try to throw together another quick project to demonstrate.

Thanks in advance for any help you can provide.

 

Was not able to add this as an edit to my previous post, but here is a sample project. The Child.exe gets loaded by Test.exe, which was built from the Unity project enclosed. Also, I tried to add a sample Child.exe in the .zip for you to load, but it made the file too big to upload. So, here are the instructions that go with this:

1. Open the project in Unity and build it.
2. Create any old Unity project and build it (or use a Unity exe that you have laying around). If it's not named Child.exe and its data directory Child_Data, name them as such.
3. Put Child.exe and Child_Data in the same directory as the app you built in step 1.
4. Run the app built in step 1.

You will see that the two apps appear to be fighting for priority in the parent unity window. So, the question is, is there any way to control/manage this behavior?

 

I haven't looked at your project yet, but you could try and grab the native texture pointer using https://docs.unity3d.com/ScriptReference/Texture.GetNativeTexturePtr.html

You could hook one of the pre-render or post-render callbacks.

 

So, @MR_asilva, looking at your project further, I don't see this as VR or EditorVR-specific, so to get better coverage, I'd suggest trying other forums, too. I don't want to leave you empty handed, here are a few thoughts:

Try using a frame debugger to see order of operations. It's possible that final backbuffers are getting overwritten. I did notice after alt-tabbing that the screen would look coherent, but if I alt-tab again it was possible for me to get it back into a flicker-state.

Try making this adjustment, so that you can debug in the editor (which has the same problems):
process.StartInfo.FileName = Application.dataPath + "/../Build/Child.exe";

By doing that you can kick open a renderdoc session, which hopefully can capture that. Right click on the game view or scene view to launch renderdoc (you'll likely need to install renderdoc first).

 

Thank you for the reply. I didn't realize this was a VR-specific forum, sorry about that. I will post in a more applicable one.

RenderDoc didn't show anything conclusive - it seemed to just show what the parent app was drawing at all times. Also, running it in the editor produced a very strange artifact where the child seemed to keep going in some form after I had stopped the parent from playing in editor mode (#3 below) - part of the Unity UI was getting overwritten with black pixels - it was very strange. I've attached some images of what I'm seeing. Also, as you can see, while it was running, the game view was saying there was no camera when indeed there was, so I'm not sure how that was happening.

Update: new thread started here: https://forum.unity3d.com/threads/how-to-create-a-steam-like-launcher-shell.471657/