HoloLens Development using Visual Studio 2015 and Unity HoloLens Technical Preview

[image credits]

Last week, I did a session at Dev Day 2016 on the topic “Developing for the HoloLens”, and this post is a tutorial that is complimentary to the session I did, demonstrating the steps of how a simple holographic application can be built and deployed in the HoloLens emulator, using the tools provided by Microsoft and Unity…

Table of Contents

1. What Are Mixed Reality Applications?
2. Software and Hardware Requirements for HoloLens Development
3. Developing 3D Holograms using Unity HoloLens Technical Preview
4. Building and Generating the VS2015 solution from Unity
5. Deploying the application in the HoloLens Emulator
6. Useful Links and resources

What Are Mixed Reality Applications?

In general, everyone would have quite a good idea of what Virtual Reality (VR) and Augmented Reality (AR) is all about, given the AR/VR devices that have proliferated the market in recent times, plus the large number of articles on AR and VR that are available on the technical new sites, blog articles, and vendor sites out on the internet. With the introduction of the HoloLens, Microsoft created one of the most ground breaking and disruptive innovations in display technology, which introduces a new term called “Mixed Reality” into the mix (no pun intended). Virtual, augmented, and mixed reality are easy ideas to grasp if you see what they try to offer in terms of how we see the world:

• Virtual Reality offers the user an immersive experience, where “presence” is the key factor. The external (real) world is shut off and the virtual world takes over the user and his senses. Interestingly, these type of applications and virtual spaces have been researched from some time back (including being heavily portrayed in early sci-fi movies and books) as evidenced by this early game called ‘Dactyl Nightmare’ from the early 90’s.
• Augmented Reality is all about overlaying the real world with graphics and animations which serve to be informative or otherwise entertaining. They could range from scientific application used out in the field, to mapping technology used by everyday people for driving or cycling. Examples range from map overlays, directional indicators and animations over roads and buildings when looked at through phones, and games like Pokemon Go.
• Mixed Reality is a concept brought into the mainstream recently, in a large sense by the introduction of the Microsoft HoloLens. This is a new way of looking at the world because unlike VR, you can still see the real world around you. And unlike AR, a user is given the feeling that any digital artifact (Hologram) that he is seeing, is part of the real world. This is mainly due to the fact that the Holograms seen through the HoloLens, become part of the real world around you, responding to the solid objects and surfaces in your room, responding to touch and sound, and behaving as if they are part of the real world.

[image credits]

With regards to the HoloLens device itself, I will not go into any detail on the specifications and what it is capable of… You can find a lot of its capabilities in the video at this link. Similar to the session that I did on Dev Day 2016 last week, we will take a developers' perspective and see what we need to know about setting up a development environment for programming the HoloLens. After that, we will dive into building and deploying a simple HoloLens application, using (free) tools provided by Microsoft and Unity.

Software And Hardware Requirements For HoloLens Development

The HoloLens development tools should be installed correctly exactly as instructed at this link.

The following tools provided by Microsoft and Unity need to be installed in your development system. Unity is a very popular cross platform game engine used to create games and interactive content for many platforms and devices. You can find many resources online for learning Unity. The Unity HoloLens Technical Preview is a build of Unity, that is implemented specifically to help create content for the HoloLens.

• Visual Studio 2015 Update 3 (Any edition including community)
• HoloLens Emulator
• Unity HoloLens Technical Preview (HTP)

When installing Visual Studio 2015 Update 3 (or if you have it already installed in your system), one important thing to note is that the below two nodes are selected in the feature selection during installation:

If the “Tools” and “Windows 10 SDK” are not selected under the Universal Windows App Development Tools node, make sure you select them. In case you have Visual Studio 2015 Update 3 already installed, you can modify the install from add/remove programs and add the above features to the installation.

One particular issue I had was that when I was trying to download the Unity HTP, there were some errors when I was trying to download from the Unity HTP site:

When I clicked the above shown link, sometimes it would download the setup file, and sometimes it would give an error saying the content was not available. In cases where the setup file was downloaded, when running the setup, I would get an error saying “no internet connection” midway through the install. These issues seem to be resolved when I checked again at the time of writing this article. But just in case anyone else is facing similar problems, just click the “Archive” tab, and download a previous beta (which is what I did), which should work without any problems:

Once the Unity HTP setup file is downloaded and run, another (very) important thing to note is to make sure that the “Windows Store .NET Scripting Backend” is selected in the installation feature selection, as shown in below image. If this is not selected, the installation of Unity HTP will not have the proper build configurations to generate the Visual Studio project required for deployment to the HoloLens.

Moving on to the hardware and platform requirements, make sure the development system you will be working on meets the following (minimum) criteria:

• 8GB of RAM (or more)
• GPU
  • Should support DirectX 11.0 or later
  • Should support WDDM 1.2 driver or later
• 64-bit CPU
  • CPU with 4 cores, or multiple CPUs with a total of 4 cores
• 64-bit Windows 10 OS
  • (Editions: Pro, Enterprise, or Education)
  • Home Edition NOT supported (due to next point)
• Enable “Hyper-V” in Windows (Not available in Windows Home Edition) from Control Panel > Programs and Features > Turn Windows features on or off
• BIOS Settings
  • Hardware Assisted Virtualization should be enabled
  • Second Level Address Translation (SLAT) should be enabled
  • Hardware based Data Execution Prevention (DEP) should be enabled

Basically, any decent machine today should meet the above criteria and requirements. The exception would be if you were running Windows Home edition, in which case you might need to consider investing in upgrading the operating system.

Developing 3D Holograms using Unity HoloLens Technical Preview

Ok, now that we have some idea about mixed reality applications, and the basic requirements for a development environment setup, lets fire up the Unity HTP and dive into creating 3D content for our HoloLens application (In each of the steps below, I will walk you through building the scene in Unity, and explain the reasons behind why we select certain settings and configurations).

When you fire up unity (from this point on, I will be referring to the Unity HTP as unity), you would be presented with the following window initially (when you run unity for the first time, it will ask you to create a free account if you don’t have one already):

Let's click on the new project button, which will take us to the screen shown below:

In the window shown above, enter a project name, select a project location where the project will be saved, and make sure to select the 3D radio button, and click “Create project”. You will be taken to the main workspace of unity, in the new project you have created:

Next, let's configure the main camera for our holographic application. Select the main camera in the left pane, and update its position to be at the origin in the world co-ordinate space by updating the X, Y, and Z values to Zero in the Inspector pane to the right. We will also select the clear flags value to be “Solid Color”:

The main camera is your window into the 3D world we are creating in unity. If we were designing a first person shooter, and not a HoloLens app, the main camera would be what you see through your monitor when you play the FPS game, so your monitor or screen will be displaying what the main camera can see. In the case of our HoloLens app, the HoloLens device will be worn on the head and our eyes will be the vantage point, therefore the main camera will display scenes (via the HoloLens) into our eyes as if our physical eyes are the window which will display what the main camera can see. This is why we will see 3D holograms merged into the actual physical world or room around us, as if they are part of the natural objects we see normally with our eyes.

Next (making sure the Clear flags setting is set to “Solid Color” as described in the last step), click on the background color to get the color selector window, and set the R, G, B, A values to zero:

Again, if this was a FPS game we were designing, we would have drawn a 3D world to be viewed through the main camera. But in the case of the HoloLens, the device can add light to the existing view you see of your real environment, but it cannot take light away from your eyes (i.e., it cannot display/render black). Anything the HoloLens renders as black will be transparent, which is why we set the color setting as shown above. This will render the whole scene transparent when the application runs and we will be able to see the real world through the HoloLens, plus any holograms we render in the scene.

Next, let's add a 3d object which will serve as our hologram (Holograms can be much more complicated, but for the purpose of this tutorial, our 3D cube will serve as a simple hologram). Click on the create drop down in the left pane (hierarchy window) and add a cube. Once added, select the cube in the left pane and set its position to be X = 0, Y = 0, and Z = 5 in order to position the cube 5 meters in front of our eyes (i.e., 5m in front of the main camera). All units of position are in meters, therefore it is very easy to translate the position values with the actual scene we will see via the HoloLens. When viewed through the HoloLens, the cube we just created will be 5m in front of us (placed in the real world), literally.

Now, let's enhance our 3D cube a bit, because it looks very plain and not very interesting. We will texture our cube and have it rotate perpetually, which is one step better than an inanimate white box. We will first texture our cube. Textures are basically image files that can be used to ‘skin’ our 3D objects. If you do a search online for ‘free textures for 3D meshes’, you would be able to download a lot of free textures. The unity store itself has a lot of free textures you can use. I will use a texture that I have which is an image file (WoodTexture.jpg) of wood like material. Just open the folder where you have your textures (i.e., image files) and drag the required texture to the assets pane:

Next (as shown in the below three images), right click inside the assets pane and create a material asset, and name it “CubeMaterial”. Once the material is created, select it in the asset pane, and then drag the texture we added earlier from within the asset pane into the “Albedo” selection property of the CubeMaterial in the inspector pane (you can play around with the metallic and smoothness settings, you will see a preview in the sphere shown in the bottom of the inspector pane). Once that is done, drag the CubeMaterial from within the assets pane and drop it on the cube in the hierarchy pane to the left. This will texture the cube we have in our scene window:

Now that we have a textured cube/hologram, let's animate it a bit by having it perpetually rotate around a particular axis. We will do this by adding a script which will define the behavior we would like to see. Right click within the assets pane and create a C# script, and name it “RotateCube”. Once created, right click the script and select open, which will open it in Visual Studio. Delete the Start() method in the script, and add a public instance variable of type float, with the value 20f. Inside the Update() method, add the following statement:

transform.Rotate(Vector3.up, speed * Time.deltaTime);

Don’t worry too much about the syntax, unity has a great scripting API and I recommend that you read through to get a better idea of what is possible. After the modifications to the code, the code should look similar to what is shown in the third screenshot below:

Next, save the modifications in Visual Studio and exit the IDE. Back in our unity work space, drag the script and drop it on the cube in the hierarchy pane. Once this is done, you can click on the “Game” tab on the work space window and click the play button to see the effect of the script on the cube. You should see a rotating, textured cube. Click the play button again to stop the animation.

Ok, we have just created a textured, rotating 3D hologram which we will build and deployed in the HoloLens emulator using Visual Studio. This will be explained in the next section. Save the scene we just created by going to File > Save Scene and give a name you prefer, and save the scene.

Building and Generating the VS2015 Solution from Unity

Now that our scene is ready, let's tweak some configuration settings required for building and generating the Visual Studio solution.

First, click on Edit > Project Settings > Quality, and in the inspection pane, set the option below the green windows store application icon to fast:

Next, go to File > Build Settings which will bring up the build settings window. Select the “Windows Store” option under platform, and click on the player settings button, which will present some configurations in the inspector pane. In these settings, select “Other Settings” and make sure the “Virtual Reality Supported” checkbox is checked, and that the windows holographic SDK is selected under the virtual reality SDK list (make sure you have selected the Windows Store platform in the build settings window, else you may not see the proper options in the inspector pane when you click player settings):

Finally, click on add open scenes button in the build settings window and add the scene we created. Select the SDK option to be “Universal 10” and the UWP Build Type option to be “D3D”. Make sure the unity C# Projects checkbox is checked, and click on Build. This will open a file dialog asking you where to save the generated Visual Studio solution. Create a new folder within the file dialog called VSapp and click on select folder, which will build the solution and save it in the VSapp folder. Once the build is done and the files are generated, the folder contents will be opened by the file dialog:

That is all there is to building and generating the Visual Studio solution based on our unity scene and holographic objects we created. In the next section which will also be the final step of this tutorial, we will open the generated solution in Visual Studio and deploy to the HoloLens emulator.

Deploying the Application in the HoloLens Emulator

If you’ve followed through the tutorial thus far, great! In this final step, let's actually deploy the HoloLens application in the emulator, using Visual Studio.

Open up Visual Studio 2015 in administrator mode (right click VS2015 > run as administrator), and open the solution file that was generated by unity (in the above case, we need to navigate into the VSapp folder we created and where the solution was saved). Once the solution is loaded and opened, right click the “Package.appxmanifest” file and click on view code. Modify the value of the “Name” attribute of the TargetDeviceFamily tag to be “Windows.Holographic”, and make sure the MinVersion and MaxVersionTested attribute values are similar to the second screenshot below:

Next, select the release mode, x86 configuration, and the HoloLens emulator from the device list as shown in below screenshot:

Now if we select Debug > Start without debugging from the menu, the Emulator should start up and our application should be deployed and automatically run in the emulator. Note that this procedure will take a bit of time, specially if the emulator is running for the first time:

You should be able to see the cube we created rotating in the emulator. You can navigate within the scene using the A, W, S, D keys, and look around by right-clicking the mouse and looking inside the emulator rendered scene. You should be able to walk around the spinning cube, look at if from different vantage points, and move away from it. The surrounding environment is all black, but in the real HoloLens device, this will be rendered transparent, and we will be able to see the real room or environment we are physically in, with the spinning cube hovering five meters in front of us.

Useful Links and Resources

The following list would help in any further R&D that you would be motivated to conduct on HoloLens development, as much as it was a good source of reference and information for me:

• Development Tools Installation: Link
• Unity HoloLens Technical Preview download: Link
• HoloLens 101: Link
• HoloLens Documentation: Link
• Spatial Mapping: Link

Conclusion…

I hope you enjoyed and found this tutorial useful and motivating. I will be covering more hololens development concepts in depth in future articles such as spatial mapping, emulator settings (specially the device portal option, which is very useful and helpful), input via gaze, gesture, and sound, and maybe some articles on unity (HTP) modelling and scripting as well. Any errata, comments, issues faced when following this tutorial, and general feedback, is most welcome in the comments section below.