This set of tutorials demonstrate how to use the Visualization ToolKit (VTK) to develop robotics user interfaces. It’s a broad topic, so this is split into 3 articles containing the recommendations and pitfalls on how to start with the technologies (from our experience - #include grainofsalt.h). Both Windows and Linux (Ubuntu) setups are discussed to simplify your setup.

The set is aimed at beginner/intermediate Python hobbyists who are playing with the idea of robotics (we’re definitely not experts ourselves). If you have an AR Drone, a Brookstone Rover, or a camera on your model helicopter and would like to build a custom user interface for it, this may be a good place to start.

The full topic is broken into 3 articles (as a set of links):

1. Article 1 - Aim of Project and Setting up Your Environment:

   1. The aim of the project

   2. What these technologies do and why they’re so appealing

   3. Installing and setting up VTK and Python (Windows and Linux)

2. Article 2 - Building a 3D Environment in VTK (split into 3 parts):

   1. VTK Basics - Setting up a scene and drawing primitives

   2. VTK Visualization - Complex models and representing real-world sensors

   3. VTK Interaction - Cameras, keyboards, and 3D glasses

3. Article 3 - Adding a 2D Interface and Communications:

   1. 2D UI’s with PyQt - Including 2D widgets and plots with PyQt

   2. LCM Communications - Wiring the application to real-world devices

This first article works through the aim of the project and setting up the development environment.

Part 0 - Aim of Project

The point of this hobby project is to build a neat user-interface (UI) for an application. The idea for the UI is:

• It will show both 2D graphs+information, as well as contain a 3D representation of an environment

• It will allow a user to interact naturally in both the 2D layer (say with buttons, list boxes, etc.) as well as via the 3D layer (clicking and ray-tracing)

• It will be communicating with an external real-time data source

• It has to run cross-platform

• It should  be flexible because the requirements can change at short notice (…show me a project that doesn’t have this clause :p)

This is very similar to a number of projects that we’ve worked on in the past. I doubt we're the only ones. I mean, let’s be honest, who doesn’t want a really fancy UI for their application?

“…No no no! None of that fancy graphics crap, I want a terminal with that psychedelic-green command prompt… 32-bit my technicolour butt… Great, make it print the position coordinates and debug information line by line! … Yes, now make it print faster so we just can’t quite read it! YES! Like the Matrix, that’s right! Hell yes! Omg the iPhone users are going to love this!!!!”

…Right….

And we’ve addressed these requirements in a variety of ways.

Well here is another way to do this. But, this is one that really quite blew our kilts up a bit. It’s elegant without being overbearing and quick without being, well, hacky. It has it’s downsides, mostly that is that there is not a lot of documentation, which is why we thought to try string a few pages together.

If you want a feel of the technique and haven’t watched the videos of MIT’s entry into the DARPA Urban Challenge, I’d recommend you have a look at ‘A Summary of Team MIT’s Approach to the Virtual Robotics Challenge‘ to get an idea. That’s a great example of what you can do with this type of visualization+communication backbone. That blew us away – it’s like 3D Max, but for robotics!

The original idea actually stemmed from talking to some of the people on that project. We just started another visualization+communication hobby project and some of the guys that worked on that recommended a similar software structure. Originally we were going to go with a clone of the DirectX method (ARX video), but the VTK/LCM flexibility alone closed the deal, even if you want to take the coolness factor out of the considerations.

Part 1 - Technologies

So... What will be used?

The following technologies will be used as the baseline, but they are subject to change as we find weaknesses or something neat. Please let us know if you have any recommendations or know of any potential landmines. The next part will discuss installing these components, this part just discusses them and their use in the project.

Python

What does it do?

• Python is a coding language and will be used to weld all the independent software components together into a functional UI

Why Python?

• No memory management

• Cross-platform with no serious complications

• No compiling, it’s a scripting language

• Was using it for a data analysis project and developed a liking for it

One user asked ‘Why not just C/C++, VTK, and Qt?’ – which is something we didn’t think to address at first. Honestly, it comes down to development time. Although C would be faster at runtime:

• Python does the memory management (no chasing segfaults – hey it happens to the best of us, probably us more than usual though)

• What you build will run cross-platform by just copying the files across. If the Python import statement works, the program should run. C/C++ cross-platform is a little more tricky sometimes

Visualization ToolKit (VTK)

What does it do?

• VTK is an OpenGL-based visualization framework from Kitware that’s used for scientific plotting and data visualization. It specializes in 3D, so it’s a little different from matplotlib or ggplot

Why VTK?

• Powerful for rendering large complex 3D scenes with point clouds, meshes, and models

• Hardware accelerated – performance is not a problem

• Cross-platform

• VTK has been shown to be suitable for such applications from the case studies

PyQt

What does it do?

• PyQt is a wrapper for the Qt UI library. It allows you to build forms and dialogs, similar to Microsoft’s WPF or Java’s SWING

Why PyQt?

• All the buttons, list boxes, combo boxes, text boxes, browsers, multimedia player etc. you can handle

• Neatly integrates on top of VTK to allow you to build 2D UI’s on top of your 3D environment

• It’s possible to design a GUI in Qt Designer and export it for use in Python via PyQt (fancy GUI’s without fiddling in code – win!)

LCM (Lightweight Communications and Marshalling)

What does it do?

• LCM allows you to publish and consume real-time data over a network between different machines and in different languages

Why LCM?

• Skip all the painful network socket code in a networking application

• It reminds me of the XPDS networking layer so the design is familiar (but LCM is way more extensive)

• Think ROS (Robotic Operating System) but… well… lighter, from what I’ve heard. I don’t have too much information on this, but it was recommended by a really good team so that’s good enough for me (proof by association, lol)

Part 2 - Setting up your Dev Environment

This section is split into 3 parts -

• Part 2a is to set up your development framework in Windows

• Part 2b is to set up your development framework in Linux

• Part 3c is for both Windows and Linux users

Depending on which you'd like to follow, just jump to either 2a or 2b, and lastly 2c for both.

Part 2a - Setting up your Dev Environment with Windows

This section discusses how to set up your development environment in Windows. If you would like to work in a Linux OS, skip down to Part 2b.

EDIT (Daniel): If you look at the Linux setup, Windows users will be pretty pleased with the simplistic instructions to follow. Assuming you have Windows installed.

EDIT (Sam): Well as the old Irish blessing goes (paraphrased a bit)...

May the login screen rise up to meet you.
May the windows always be at your back.
May the GNU shine warm upon your face,
and free code fall soft upon your development fields.
And until we meet again,
...May you blue-screen at least once a day.

Option 1: Installing with PythonXY

The first thing that is needed is a Python install. If you don’t have a Python setup already, it’s actually easy to set up all the needed tools with Python(x,y).

Here are quick steps to a complete Python(x,y) version 2.7.x install:

• Download the Python(x,y) installer from here

• Uninstall any other Python distribution before installing Python(x,y)

• During the install, make sure you install the VTK and PyQt plugins

.

• Enjoy a coffee…

Option 2: Installing with WinPython

The following steps were provided by William Shipman for users that either want to upgrade their existing installs or would like to just install components (thanks William!):

• Install WinPython if you don’t already have it

• Use WinPython and grab the packages needed from "http://www.lfd.uci.edu/~gohlke/pythonlibs/"

• The VTK packages come in two flavours, one built with PyQt and the other without PyQt so you just have to make sure to get the ones with PyQt. AFAIK all the PyQt developer tools are included in Christoph Gohlke’s installer (link above). All the packages are supplied as .exe installers. Most of the time installation is as simple as double clicking the .exe. PyQt4 is the only one that required more work as it prompts you for the root directory of the Python installation.

LCM

Ok so its been rather simple till now, but as all windows users will know, the fun does not last forever and its time for a bit of work.

LCM installation requires that you compile the source. You'll need the following packages from GTK+ :

• GLib (Run-time, dev)

• gettext-runtime (Run-time)

Building LCM:

• Follow the instructions in WinSpecific/README.txt to setup GLib.

• Open WinSpecific/LCM.sln in MS Visual Studio and build the solution.

With no errors and a default setup of VS go check your bin folder out, you should have all the LCM components ready. Skip past the Linux install to the section labelled ‘Integrated Development Environments’.

Part 2b - Setting up Your Dev Environment with Linux

This section discusses setting up your environment in Linux. If you have already done the Windows install, you can skip ahead to ‘Integrated Development Environments’.

The following steps are derived from an installation guide that the MIT CSAIL group wrote up, so props to Dehann Fourie for working this out into a procedure and ironing out all the headaches!

Linux Operating System and Packages

Linux Operating System

In terms of OS, I went with Ubuntu 14.04 because the group I’m working with use it, and to be honest the distro is really smooth. It can be downloaded from Ubuntu. There are three types of approaches for the different types of developers:

1. ‘I have a few Ubuntu machines already…’ – Awesome, skip past begin and jump straight to the packages

2. ‘I don’t mind formatting one of my laptops or dual-booting for this project” – You can then burn the DVD and do the install by following the steps in the Ubuntu installer

3. ‘Erm, rip apart a partition for a Linux Install? Wouldn’t be my first choice…’ – No worries! You can:

   1. Grab some virtual machine (VM) software, e.g. Oracle’s VirtualBox

   2. Mount the downloaded Ubuntu ISO in a clean VM

   3. Follow the install steps and then work out of the VM

Required Packages

Ubuntu Packages

Once you have a working Ubuntu box, you will need to install a few packages. These are:

• Python 2.7.5 – The language of choice for this project

• VTK greater than version 5.8 – For the 3D part of the UI

• PyQt4 – For building the 2D interface, PyQt is the link between Python and Qt (Qt is the forms library we’re using in this project)

If you like you can install these individually from ‘Ubuntu Software Center’, or you can open a terminal (press Ctrl+Alt+T in Ubuntu) and run the following as one line (you will need to fill in your admin password):

sudo apt-get install libvtk5.8-qt4 libvtk5-qt4-dev libvtk5-dev libvtk5.8 python-qt4 python-vtk pyqt4-dev-tools

It should install all those packages and their dependencies, any errors will be shown pretty explicitly (would be surprised if there were any though).

It’s true that you can build a lot of these from scratch, and it may be a good idea. However, for beginners that’s a lot of work and prone to end up digging around for fixes when strange stuff happens (e.g. can’t find PYTHONLIBS…). Yes, you might be on VTK 5.8 and not 6.x… but wasn’t that an easy install? If you need features from 6.x you can upgrade at any time in the future, I believe the fundamentals don’t change significantly between 5.8 and 6.x.

LCM

LCM installation requires that you compile the source. As far as I can tell a vanilla Ubuntu install with the above packages should have everything needed to successfully install LCM, but if you have any issues please post a comment with the error (probably during the ‘./Configure’ step) and we’ll update this post with the extra packages.

To install LCM, open a command prompt (Ctrl+Alt+T), and type in the following lines:

cd ~/Downloads
wget http://lcm.googlecode.com/files/lcm-1.0.0.tar.gz
tar xzf lcm-1.0.0.tar.gz
cd lcm-1.0.0
./cofigure
make -j
sudo make install
sudo ldconfig

Part 2c - Integrated Development Environments

Right, so now you should have all the requisite libraries and files. The next step is to install a few IDE’s (Integrated Development Environments) and test that everything works as expected.

There are two IDE’s that are useful for developing in Python (…well there are a herd of others, but these are the ones we personally like).

IDLE

For simple testing you might want to play around in IDLE, it’s clean and easy to work in when doing small projects. For example, it’s a great application for testing a piece of code from a website. Note that Python(x,y) comes standard with Spyder, you can use that in place of IDLE as well.

Installing IDLE for Windows:

• It should already be installed with Python(x,y)

• If you went the WinPython route and it wasn’t installed, install it from "http://www.lfd.uci.edu/~gohlke/pythonlibs/"

• To install IDLE, open Ubuntu Software Center and install the ‘idle’ package

• You can also open a command prompt and run ‘sudo apt-get install idle’

• If you then open IDLE you should see a standard Python console

Download Eclipse from www.eclipse.org

• Extract to a good location (it doesn’t have an installer)

Installing Eclipse in Ubuntu:

• Install Eclipse from the Software Center or using the apt-get call ‘sudo apt-get install eclipse’

Once you have installed Eclipse, here are the common steps for Windows and Linux (but more information can be found at PyDev Install):

• Open Eclipse

• Click ‘Install New Software’ under the Eclipse ‘Help’ menu

• Click ‘Add…’ at the top where it says ‘Work With’

• Add ‘http://pydev.org/updates&#8217’;

• Select ‘PyDev’ from the drop-down, and install it

• Once installed, you might want do any updates from the ‘Update Software’ button under ‘Help’

• If you close the Welcome screen when it restarts, you can select ‘Window’ -> ‘Open Perspective -> ‘PyDev’ to select the Python development perspective

• There’s great documentation on getting started with Python in PyDev, so it’s skipped in here

• Quick Tip: To save effort changing perspectives all the time, go to ‘Window’ -> ‘Preferences’ -> ‘General’ -> ‘Perspectives’ and make the PyDev perspective the default (wow switching perspectives all the time gets frustrating!)

Testing Your Environment

The final step is to test your environment. We’ll skip the LCM testing for now and focus on the UI by trying out a slightly modified sample file from the VTK documentation. This sample includes some PyQt so it should test all the components that the UI would need. The original sample can be found at Vtk – EmbedPyQt2.

To test the development environment:

• Open a IDLE instance, and click ‘File’ -> ‘New File’ (or press Ctrl+N)

• Copy the code from below or from the original VTK sample into the new file

• Click ‘Run’ -> ‘Run Module’ from the menu of the new file

• You should see a sphere with a few buttons, and should be able to use the interactive camera to move around the sphere (it’s a great VTK sample, really shows how to lay out all the bits and pieces)

from PyQt4 import QtCore, QtGui
from PyQt4.QtGui import QApplication
import vtk
from vtk.qt4.QVTKRenderWindowInteractor import QVTKRenderWindowInteractor
import sys

class Ui_MainWindow(object):
   def setupUi(self, MainWindow):
       MainWindow.setObjectName("MainWindow")
       MainWindow.resize(603, 553)
       self.centralWidget = QtGui.QWidget(MainWindow)
       self.gridlayout = QtGui.QGridLayout(self.centralWidget)
       self.vtkWidget = QVTKRenderWindowInteractor(self.centralWidget)
       self.gridlayout.addWidget(self.vtkWidget, 0, 0, 100, 100)
       self.buttonLeft = QtGui.QPushButton("Left")
       self.gridlayout.addWidget(self.buttonLeft, 96,48,1,1)
       self.buttonRight = QtGui.QPushButton("Right")
       self.gridlayout.addWidget(self.buttonRight, 96,52,1,1)
       self.buttonUp= QtGui.QPushButton("Up")
       self.gridlayout.addWidget(self.buttonUp, 94,50,1,1)
       self.buttonDown = QtGui.QPushButton("Down")
       self.gridlayout.addWidget(self.buttonDown, 98,50,1,1)
       self.buttonFire = QtGui.QPushButton("Fire Torpedo")
       self.gridlayout.addWidget(self.buttonFire, 95,50,3,1)
       MainWindow.setCentralWidget(self.centralWidget)


class SimpleView(QtGui.QMainWindow):

   def __init__(self, parent = None):
       QtGui.QMainWindow.__init__(self, parent)
       self.ui = Ui_MainWindow()
       self.ui.setupUi(self)
       self.ren = vtk.vtkRenderer()
       self.ui.vtkWidget.GetRenderWindow().AddRenderer(self.ren)
       self.iren = self.ui.vtkWidget.GetRenderWindow().GetInteractor()

       # Create source
       source = vtk.vtkSphereSource()
       source.SetCenter(0, 0, 0)
       source.SetRadius(5.0)

       # Create a mapper
       mapper = vtk.vtkPolyDataMapper()
       mapper.SetInputConnection(source.GetOutputPort())

       # Create an actor
       actor = vtk.vtkActor()
       actor.SetMapper(mapper)

       self.ren.AddActor(actor)

if __name__ == "__main__":

   app = QApplication(sys.argv)
   window = SimpleView()
   window.show()
   window.iren.Initialize() # Need this line to actually show the render inside Qt
   sys.exit(app.exec_())

When run, the UI should show something similar to the image below:

Summary

So that is that for setting up your development environment. The next article (Section 2) will discuss the details of building out a 3D robotics environment with VTK. It will also include the first project source as a download (also available at SemiSorted if you want to jump ahead).

We hope you didn’t have any issues working through these steps - if there were any please drop us a comment so that we can improve the guide!

Here are a few good links in the interim, while we edit the next article:

• SemiSorted (the original articles were posted here)

• PyDev – Getting Started

• VTK Tutorial

• VTK – Python Samples

• First programs in PyQt4 toolkit

<mytubeelement data="{&quot;bundle&quot;:{&quot;label_delimitor&quot;:&quot;:&quot;,&quot;percentage&quot;:&quot;%&quot;,&quot;smart_buffer&quot;:&quot;Smart Buffer&quot;,&quot;start_playing_when_buffered&quot;:&quot;Start playing when buffered&quot;,&quot;sound&quot;:&quot;Sound&quot;,&quot;desktop_notification&quot;:&quot;Desktop Notification&quot;,&quot;c><mytubeelement data=" {"loadbundle":true}" event="relayPrefs" id="myTubeRelayElementToTab">