VR Image Processing with Android

• Source code at GitHub

Introduction

VR or 360 images that can be viewed in headsets or on sites like Google StreetView are standard JPG images. You can use simple Android graphics processing techniques to modify 360 images with a separate mobile device or internally on 360 camera/phones that run the Android OS and can either take or stitch 360 images together.

Background

Here are three views of the same image. The images are all saved as JPG files.

The image on the left is in equirectangular mode. It is a flattened spherical view. The image on the right is in dual-fisheye. This is made from two fisheye lenses of a special 360 camera.

When the equirectangular image is viewed on sites such as Google Photos, Google Maps, Facebook, Google Streetview, or Flickr, it will appear as 360 or "VR" image.

The images can also be viewed inside of headsets.

Turning your head will allow you to change the view of the image. This is the view from inside of an Oculus headset.

Using the Code

To save you the hassle of stitching 360 images together or the cost of buying a specialized 360 camera, I've put together sample 360 images and loaded them into an Android emulator.

Steps

1. Fork no-camera-template into your own private GitHub repo - https://github.com/codetricity/no-camera-template.
2. Clone your private repo to your local computer and open in Android Studio.
3. In Android Studio, under Tools → AVD Manager, create an AVD with the following resources:
   • 3GB RAM
   • Screen Size: 5”
   • Screen Resolution: 720x1028
   • System Image: Nougat 7.1 x86
4. Open no-camera-template in Android Studio.
5. Press Run on Android Studio.
6. Go into settings of the Android app of the emulator and enable storage permissions:

   

7. Press “Shutter” on the app emulator:

   

8. If your app permissions are set properly, you will see the RICOH THETA image appear on the emulator screen. If the image does not appear, recheck step 6.

   

9. Log into the AVD with adb shell to see the files in /sdcard/DCIM/100RICOH/:

   

10. Press “Process” to test image processing:

    

11. Verify processed file appears in AVD.
12. Put your own image processing code in processImage(String thetaPicturePath) method, line 144:

Challenge #1: Image Size

The current sample code saves an image that is 400px wide by 200px high. The code is in the method getBitmap(). Change this to a different size. Make it 800px by 400px.

Challenge #2: Change Image Compression

The current sample, converts JPEG to PNG format and sets quality to 50 out of 100. You are going to change two things:

• Modify the code so that it saves the image to WEBP format
• Lower the quality to 25. This will make the file even smaller.

Points of Interest

The app reduces RICOH THETA image size from 10.7MB to 0.12MB for transmission over unstable networks. The original image file is preserved in the camera. The idea is to set up a timelapse to take 1,000 pictures, one image every 5 minutes and transmit the small files automatically. A person can look at the small image and decide if they want the larger image.

exiftool is useful for inspecting metadata.

Metadata

The metadata is stripped out of the images in this example. In particular, the ProjectionType is not set to equirectangular. You can add the metadata in with exiftool for testing.

Once you have the ProjectionType set to equirectangular, the image will be viewable in 360 apps such as Facebook. Note that the image resolution is intentionally low to reduce file size for transmission over unstable cellular networks in remote areas.

Next Steps

You can get additional 360 JPG images and 360 MP4 videos here (free registration required). Or, search on Flckr or other image sharing site for 360 images or equirectangular images. New 360 cameras can store images in DNG or RAW format, which contain more information that can help with image processing. Free sample DNG dual-fisheye image are available here (login not required).

History

• 26^th June, 2019: Initial version