Exploring advanced VR modeling tools

Following the Do It Yourself Virtual Environments (DIYVE) workshop, where non-experts modeled their own virtual environments using SweetHome3D, the final step in the first case study of Sub Project B is to look at ‘Behaviour Modeling’ of virtual environments. By now we have seen that even non-experts are able to create realistic virtual use contexts. The next step, Behvaiour Modeling, is done by a more dedicated designer who is more experienced with programming and modeling (refered to as ‘prototyper’). Consequently we can try to use advanced tools to achieve higher levels of complexity, flexibility and interactivity in virtual environments.

Approach

Unlike the DIYVE workshop, where participants were guided through the tool and worked on an assignment during the workshop, the advanced modeling tool is introduced to the dedicated designer in a one-on-one meeting. It was decided to use Blender as a starting point for this part of the case study. Blender was also used for developing the VR demonstration projects, as well as the more advanced prototype shown during the Interactive Demonstration Session for the company. As such, it has prooven itself as a versatile tool for modeling and real-time virtual environments, it is an open piece of software, and well-known to the researcher.

The tool was introduced to the prototyper by starting with one of the virtual environments created during the DIYVE workshop. The virtual environment was imported to Blender, after which the tool was used to add ‘interactivity’. We decided to implement

  • A ‘first person shooter’ perspective (e.g. so you can walk around in the environment)
  • A working door, so the user can open and close a door, and
  • The ability to pick up virtual objects, move them and put them back

These exercises are considered feasible and relevant for behaviour modeling and are a good way to get to know the modeling tool. While working on these topics the prototyper noticed several positive and negative aspects of the tool, it’s user interface and the overall ‘way of working’. In general, it was concluded that Blender offers a lot of (and probably too much) functionality, but has quite a threshold for new users. Despite the experience of the prototyper in the field of programming, gaming and modelling, it was still difficult to grasp the modeling concepts that are used in Blender. Nevertheless, the above functionality could succesfully be added to the scene, with the exception of the third one.

Blender is used for behaviour modeling. Here one of the virtual environments created in the DIYVE workshop is imported.

Preliminary Conclusions

After the introductory session the prototyper worked with Blender individually over a period of several weeks (as a ‘friday afternoon’ activity). Spending more time on using the tool, the aforementioned conclusion was confirmed. The tool offers a lot of functionality, but fails to do this intuitively. Especially if such tools would be used once every other month or so, it is important to quickly (re)grasp the use of the tool rather than having to spend a week of training on it each time you use it.

There are several ways to address the challenges that emerged during this period of use. Firstly, the user interface of the tool could be adapted to the specific tasks of VR modeling (e.g. leave out unused functions, or hide them in deeper ‘UI layers’). The prototyper suggested to look at visual programming solutions; simple ‘if-then-else’ statements could be used to define the behaviour of a VR scene. Secondly, additional training could be provided to properly explain the prototyper the basics of the tool before diving into the complexity of VR modeling.

With plenty of solutions for any of these directions available, the real question is how do designers (or the company as a whole) treat the trade-off between investements in time and tools (e.g. training, specific UI’s) and the benefits of using VR they gain in return. In order to understand this trade-off, the company (and its designers) first need to experience the use of VR as prescribed during the case study. This experience should clarify the benefits of applying VR, and add insights about the use of tools (e.g. see where additional investements are required and/or justified).

Mixed Reality: Interactive Textures

Sub project B is currently investigating the possibilities of mixed reality. Mixed reality typically consists of a virtual and a real part. With augmented reality for instance, virtual information is added to real information (usually a video stream). It is also possible to mix two sources of virtual information.

Image a user interface designer who wants to see how his new GUI idea fits on a future product (e.g. the dashboard of a new car or the user interface of  industrial machinery). Here we can use mixed reality to let GUI designers insert their GUI prototype (which may be an interactive Flash prototype, a simple sketch or a Sketchify file) into a virtual environment, where a test-user can use the GUI on a virtual product.

Interactive Textures

A first implementation of this idea is shown in the short movie below. On the left, you can see a 3D virtual environment (a simple 3D object in Blender). Onthe right, a window pops up, showing an arbitrary traditional 2D application (in this case a webbrowser). This application is rendered in real-time on the cube in the virtual environment. When we browse in the webbrowser, the cube immediately shows the same output. The other way around also works; the browser can be controlled from the virtual environment. To imagine a more useful application, replace the webbrowser with a GUI prototype, and the 3d cube with a soda machine, a printer or a car dashboard!



Techniques used

The implementation presented here uses Virtual Network Computing, which is a standard and widely available technique for sharing software applications; you can even share an entire desktop over a network. This allows people to watch your desktop, but also to control mouse and keyboard input. The same trick is used here; the webbrowser is a ‘shared’ application that sends visual output and mouse/keyboard controls to the 3D environment. This way, the application can be rendered on the cube (or any 3d object), and mouse input can be sent from the 3d environment back to the 2d application.

Augmented Reality in Blender

Augmented Reality in Blender got some attention a while ago. Several people [here, here and here] reported successful use of the Python ARToolkit bindings, Blender and the Blender videotexture plugin. Because I needed to do some experiments with AR, I tried PyARTK but it failed to compile on any Linux platform (unlike ARToolkit itself, which works fine).

Not willing to spend several days on fixing dependencies I decided to just forget about PyARTK and connect Blender to ARToolkit directly. However, my lack of (sufficient) C skills called for a ‘creative’ solution here.

Tool chain

The above diagram illustrates the solution.

Tools used:

  • Vloopback 1.3 to split the video into two virtual streams
  • ARToolkit 2.72.1 to do the marker detection and tracking
  • Modified simpleTest to put matrix in buffer
  • Python script to read buffer and control BGE
  • VideoTexture plugin for Blender to show video stream

Firstly, I’m using vloopback to split the webcam stream into two virtual  streams, both accessible through /dev/video3 (in this case). One of these streams is used by a modified version of ARToolkit’s simpleTest. This tool analyses the video, finds a marker and outputs the resulting orientation matrix to a buffer in RAM (simply write the matrix to a tmpfs file). This buffer is read by a Python script in the Blender Game Engine, allowing it to align the camera and objects properly. The Game Engine uses the videoTexture plugin to show the webcam’s virtual stream in real-time.

Performance

The performance of this setup is better than I originally expected. I’ve tested it on two laptops (two dual cores, 2,5 Ghz and 1,6 Ghz) with NVidia graphic cards (G96M Quadro FX 770M, the other one is older). For now it’s Linux only (Tested on Ubuntu Karmic), mainly because of the vloopback trick. The other tools are pretty much platform independent I think. Compared to the standalone ‘simpleTest’ program there is some lag, but it doesn’t really affect the AR interaction.

Todo

  • I’m working on a multi-marker setup
  • Write some supporting python scripts for the BGE

VR in Blender

Initial experiments show that Blender is a feasible platform for certain VR applications. I’ve been using OpenCV to hook up a webcam and face tracking to Blender’s 3D viewports. The results are promising; an effective and cost efficient 3D display. The good thing about this is that Blender is very open, and allows for Python scripting. As I’m familiar with both, I suppose this is an interesting lead for initial VR prototyping.

Related to this, several people are working on Blender VR. This blog used OpenCV and a webcam to control the Blender Game Engine. This very old post on Blender  Nation mentions the use for Blender in a CAVE environment (status unknown). And of course Blender also supports the WiiMote.