PERCEPTUAL COMPUTING: Perceptual 3D Editing

• Visit Intel® Developer Zone for Developing Windows* Apps for Intel Devices

1. Introduction

If you’re familiar with Perceptual Computing and some of its applications, you will no doubt be wondering to what degree the technology can be used to create and manipulate the 3D world. Ever since the first batch of 3D games, modelling and motion capture have been parts of our app-making toolkit, and a wide variety of software and hardware has sprung up to answer the call for more immersive and life-like crafting.

When looking at Perceptual Computing as the next natural technology to provide new solutions in this space, you might be tempted to think we can finally throw away our mice and keyboards and rely entirely on our real-world hands to create 3D content. When you start down this road, you begin to find it both a blessing and a curse, and this article will help you find your way with the help of one programmer’s map and a few sign posts.

Readers should have a basic understanding of Perceptual Computing as a concept and a familiarity with the hardware and software mechanisms required to control an application. No knowledge of programming or specific development platforms is required, only an interest in one possible evolutionary trajectory for 3D content creation.

2. Why Is This Important

It can be safety assumed that one of the benefits of increasingly more powerful devices will be the adoption of 3D as a preferred method of visual representation. The real-world is presented to us in glorious 3D, and remains our preferred dimension to interact and observe. It is fair to conclude that the demand for 3D content and the tools that create it will continue to increase, moving far beyond the modest needs of the games industry and become a global hunger.

The current methods for creating 3D content and scenes are sufficient for the present, but what happens when five billion users want to experience new 3D content on a daily basis? A good 3D artist is expensive and hard to find, and good 3D content takes a long time to create! What if there was another way to fulfill this need?

3. The Types of 3D Content

If you are familiar with 3D game creation, you will be aware of several types of 3D content that go into a successful title. The terrain and structures that make up the location, the characters that play their roles, and the objects that populate your world and make everything a little more believable. You also have 3D panels and ‘heads-up-displays’ to feed information to the player, and a variety of 3D special effects to tantalize the watcher. How would we accomplish the creation of these different types using no mouse, no keyboard, no controller, or sculpting hardware? What might this creative process look like with Perceptual Computing?

4. Editing Entire Worlds

The terrain in a scene is often stretched out over an extremely large area, and either requires a team of designers to construct or a procedural function to randomize the world. When no specific location detail is required, you could use Perceptual Computing Voice Recognition to create your desired scene in a matter of seconds.

Just imagine launching your new hands-free 3D editing tool by saying "New Scene."

You then decide you want some vegetation, so you bring them forth as though by magic with the words "Add Trees."

You want your scene to be set at midnight, so you say "Set Time To Midnight."

Finally to make your creation complete, you say "More Hills" and the tool instantly responds by adding a liberal sprinkling of hills into your scene.

The user has effectively created an entire forest world, lumpy and dark, in just a few seconds. You can perhaps see the possibilities for increased productivity here, but you also see that we have removed the need for any special 3D skills as well. Anyone can now create their own 3D landscapes; all they need is a voice and a few common phrases. If at any time they get confused, they can say "Help "and a full selection of command words displays.

5. Editing 3D in Detail

The world editing example is nothing remarkable, or exclusively the domain of Perceptual Computing, but suggestive of the types of interfaces that can be created when you think out of the box. The real challenge comes when you want to edit specific details, and this is where Perceptual Computing takes center stage.

Now imagine during the general world editing you wanted to create something specific, let’s say a particularly gnarled tree, the "Add Tree" command would be too generalized and random. So just as you would in real-life, you point at the screen and then say "Add Tree There."

Unfortunately the engine assumed you wanted the tree in context and selected the same tree as the previous additions. It is fortunate then that our revolutionary new tool understands various kinds of context, whether it be selection context or location context. By saying "Change Tree to Gnarled," the tree instantly transforms into a more appropriate visual.

As you increase the vocabulary of the tool, your user is able to add, change, and remove an increasing number of objects, whether they are specific objects or more general world properties. You can imagine the enormous fun you can have making things pop in and out of existence, or transforming your entire world with a single word.

For locomotion around your world, exactly the same interface is used but with additional commands. You could point to the top of a hill or distant mountain and say "Go There." Camera rotation could be tackled with a gestured phrase "Look At That," and when you want to save your position for later editing, use commands such as "Remember This Location" and "Return To Last Location."

6. The Trouble with 3D Editing

No article would be complete without an impartial analysis of the disadvantages to this type of interface, and the consequences for your application.

One clear advantage a mouse will have over a Perceptual coordinate is that the mouse pointer can set and hold a specific coordinate for seconds and minutes at a time without flinching. You could even go and make a cup of tea, and be very confident your pointer will be at the same coordinate when you return. A Perceptual coordinate however, perhaps provided by a pointing finger at the screen, will rarely be able to maintain a fixed coordinate for a fraction of a second, and the longer the user attempts to maintain a fixed point the more annoyed they will get.

A keyboard can instantly communicate one of 256 different states in the time it takes to look and press. To get the Perceptual Camera to identify one of 256 distinct and correct signals in the same amount of time would be ambitious at best.

Given these comparisons, it should be stated that even though you can increase productivity tenfold on the creation of entire worlds, the same level of production could decrease dramatically if you tried to draw some graffiti onto the side of a wall or building. If you could ever summon a laser to shoot out of your finger, or gain the power of eye lasers, you would quickly discover just how difficult it is to create even a single straight line.

The lesson here is that the underlying mechanism of the creative process should be considered entirely. We can draw a straight line with the mouse, touchpad, and pen because we’re practised at it. We are not practised at doing it with a finger in mid-air. The solution would be to pre-create the straight line ahead of time and have the finger simply apply the context so the software knows where to place the line. We don’t want to create a "finger pointer." We want to place a straight line on the wall, so we change the fundamental mechanism to suit our Perceptual approach, and then it works just fine.

7. Other types of 3D Editing

The same principals can be applied to the creation of structures, characters, creatures, inanimate objects, and pretty much anything else you can imagine for your 3D scene. A combination of context, pointing, and voice control can achieve an incredible range of creative outcomes.

Characters - Just as you design your avatars in popular gaming consoles or your favourite RPG, why not have the camera scan you to get a starting point for creating the character. Hair color, head size, facial features, and skin color can all be read instantly and converted into attributes in the character creation process. Quickly identifying which part of the body you want to work on and then rolling through a selection would be more like shopping that creating, and much more enjoyable.

Story Animation – Instead of hiring an expensive motion capture firm, why not record your own voice-over scripts in front of the Perceptual Camera. It would read not only your voice, but also track your upper body skeleton and imprint those motions onto the character you intend to apply the speech. Your characters will now sound and animate as real as the very best AAA productions!

Structures – The combination of a relatively small number of attributes can produce millions of building designs and all done in a few seconds. Take these two examples and the buildings created from two series of commands: "Five storeys. Set To Brick. Five Windows. [point] Add Door. [point] Remove Window" and "One storey. [point] Add Window. Go To Back. Add Three Doors. Set To Wood." Naturally the tool would have to construct the geometry and make smart decisions about the interconnectivity of elements but the types of element are not inexhaustible.

8. Tricks and Tips

Do’s

• Make it a habit to continually compare your Perceptual solution with the traditional methods. If it’s more difficult, or less satisfying, should it really be used?
• Try your new interface models on new users periodically. If your goal is a more accessible editing system, you should be looking for users without traditional creativity skills.
• Remember that when using voice recognition, individual accents and native languages will play a huge role in how your final software is received. Traditional software development will not have prepared you for the level of testing required in this area.
• Experiment with additional technologies that complement the notion of hands-free intelligent interfaces. Look at virtual reality, augmented reality, and additional sensors.

Don’ts

• Do not create interfaces that require the user to hold their arm forward for prolonged periods of time. It’s uncomfortable for the user, and very fatiguing long-term.
• Do not eliminate the keyboard, mouse, or controller from your consideration when developing new Perceptual solutions. You might find mouse- and voice-control is right for your project, or keyboard and "context pointing" in another.
• Do not assume project lengths can be determined when delving into this type of experimental development. You will be working with early technology in brand new territories so your deliverables should not be set in stone.

9. Final Thoughts

As technology enthusiasts, we wait for the days of Holodeck’s and chatting with your computer as a member of the family. It may surprise you to learn we’re already on that road and these magical destinations are a lot closer than you might think. Voice recognition is now usable for everyday applications; the computer can detect what we are looking at to gain context, and we have the processing power to produce software systems expert enough to fill in the gaps when required.

All we need is a few brave developers stubborn enough to reject yesterday’s solutions and become pioneers in search of new ones. Hopefully, I have painted an attractive picture of how creativity can be expressed without the need for traditional hardware, and highlighted the fact that the technology exists right now. This is not just a better tool or quicker process, but a wholesale transformation of how creativity can be expressed on the computer. By removing all barriers to entry, and eliminating the need for technical proficiencies, Perceptual Computing has the power to democratise creativity on a scale never before seen in our industry.

The application that inspired this article and the blog that tracked its seven week development can be found here: http://ultimatecoderchallenge.blogspot.co.uk/2013/02/lee-going-perceptual-part-one.html

Lee also chronicles his daily life as a coder, complete with screen shots and the occasional video here: http://fpscreloaded.blogspot.co.uk

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
Copyright © 2013 Intel Corporation. All rights reserved.
*Other names and brands may be claimed as the property of others.