Tilt Brush & Spatial UI Rapid Wireframing
Here at Holographic Interfaces we are working to design interaction models and user interfaces for a wide range of applications, each with its own challenges and peculiarities. Spatial User Interface (SUI) design tools are few and far between. There is no Sketch or Photoshop, vector graphic support, Bootstrap, nor a fully fleshed out Windows or iOS, so we must utilize what tools we do have to approximate. There are near-experimental tools in the form of Tilt Brush, Quill, and Medium, but that’s about it.
We’d like to present to you a methodology that we have tested in production and believe is a good first step in creating an efficient development pipeline. Some have done similar things at this point in time in the areas of game level development, but what we’re discussing here is the UI portion of a game or application, or even OS.
For this process we’ll be using Google’s Tilt Brush. The reason being that it combines 3D sketching with the ability to import transparent png files and 3D obj object files with their textures. This lets us create asset packs for various types of interfaces we might want to design. Want to create a spherical interface for hands on manipulation? Sketch it out, import some basic shape geometry to block out your SUI elements, and reposition until you have a pleasant, ergonomic layout. Get creative! There aren’t any hard and fast rules yet for SUIs!
Once you have a design you like, I recommend tracing the shapes of the 3d models you imported with a fine brush (I like to outline rectangles and then draw a corner-to-corner “X”). Export the scene to obj and bring it into Unity or Unreal Engine.
Note that TB doesn’t export any images or objects you imported to your scene, just the paint, so after importing your TB scene you’ll want to import the models and images you used originally and line them up according to your TB scene guides. You can then easily add interactivity to test with your VR headset, AR headset, Leap Motion, or whatever technology you like. It’s all incredibly easy, enjoyable, and quick.
We’ve found this to not only be much faster than designing in 2D and wrapping it around the user, but it lets you experiment a lot more, sense the ergonomics as you create, and define the spatial area the user will enjoy. This is not limited to buttons and content areas! You can use it to piece together scenes using imported furniture and figure models. You can use it to design physical interfaces if you like. Just create a 3d model content pack filled with switched, dials, and screens, an unadorned car model, and just piece it together until you’re happy, while sitting in the car!
We’re providing a basic starter asset pack to use. It has basic button shapes, a spherical and flat spacing grid model, some miscellaneous physical buttons and switches, and a couple other things we find useful for UI design. We’ve limited the number of models in the pack since the TB menu is not designed to accommodate a large number of files. We suggest keeping strict control over how many you let in there. Be creative, there are no rules yet! Show us what you create, and let us know if you have questions or suggestions for future asset packs!
Step 1: First you need to paint something in Tilt Brush. Maybe you did some thinking ahead of time, maybe you even sketched something on paper or in Photoshop. If you did sketch something, copy it to the Media Library/Images folder in the Tilt Brush folder of Documents. If you downloaded our asset pack, expand it to the Media Library. In TB, click that ellipsis in the lower right of the options panel, open your media library, then pull in your sketch. It’s great to be able to have a reference to go by.
Step 2: Use whatever brushes you like to paint your SUI. If you’re using our asset pack import the elements as needed from the Media Library panel to piece together different types of interface elements.
Step 3: It’s worth noting that TB doesn’t export your imported models, so to get sizes and placements I recommend painting over their edges with a fine brush. Only do this after you’re happy with the placement of everything, since the lines will make it harder to visualize the final product. I also recommend taking a video of the interface before painting over it, and saving the scene as well. This way the paint over is not saved, or saved as a separate scene.
Step 4: Back in your options menu go ahead and export the scene as an obj. Now you can import that model into Unity or Unreal Engine, along with whatever other 3D assets you used in your scene. Once in your 3D animation package align those imported models with the sketch lines. That’s it, you have the scene you created in TB inside of Unity or Unreal, ready to use in your game or simulation!