Click here to Skip to main content
15,850,265 members
Articles / Web Development / Node.js
Tip/Trick

VRT - Realtime HTML5 Data Visualization Application

Rate me:
Please Sign up or sign in to vote.
5.00/5 (5 votes)
11 Nov 2015MIT4 min read 17.8K   7   5
A versatile visualization web browser application using standard web technologies.

 

Image 1

Introduction

Hello, my name is Odd Marthon Lende. I am employed as a Technical Support Engineer in Stavanger, Norway.

I started working on this project 3-4 years ago, it started out as a hobby project at work. The mission was to create a monitoring application. An application that receives data on a web service and reflects it back to browser clients.

This project has taken too much of my time and energy, it has been abandoned several times for many months. There have been periods (months, weeks) with a lot of new ideas and writing. To complete my work project, I have written a similar application using only Microsoft technology and a few 3rd party libraries in 3-4 months.

VRT

Is/should be a versatile visualization application in the browser using standard web technologies that enables the user/developer to create visualizations of data inside containers called widgets. These widgets have a standard interface that allows you to read data in a standard way and create your own visualization using the web technology of your choice (HTML, Canvas, WebGL, 3rd party libraries) inside this container. These programs can then be grouped together into a group called a dashboard. The widgets can then be positioned and scaled inside the body of the browser window. A standard interface for accessing and reading data into a common format (table/matrix), so it can easily be extended with different data sources. I have been working on reading JSON, CSV and also implemented a data source using the Web Audio API.

MC++
cd vrt             # Change the directory to where you cloned the repository
npm install        # Installs the required nodejs modules
node standalone.js # This starts the nodejs process

# If the process started successfully,
# you should be able to open the browser (only Google Chrome currently) to
# http://localhost:23485

VRT is still in early development and I will not spend any more time developing it if there is no interest for it. I've released the source on github and was hoping to get some developers/designers interested.

Image 2

The Struggle for Speed

D3

The first serious attempt was to create a simple table as a widget, I found a library called D3.

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.

Creating a table using HTML elements with not many columns and rows worked fine, try creating 500 000 - 1 000 000 elements and look at the browser memory / CPU and you'll know what I mean. D3 is a library that allows you to create SVG graphics, so for every data point there will be created a new DOM element or more in memory, it will bring the browser to a halt eventually.

D3 is a great library, but I was not very impressed with the speed and memory usage when loading a large amount of data.

I do not use it for any of the visualizations in the latest update, but I've used it in the UI a lot.

Canvas

I also experimented with canvas, the code has been deleted in the current revision, but it is still there in the repository. I tried optimizing, so a big difference here from D3 would be having the possiblity to not redraw all the data points when there is an update. To add a new line to the canvas when a new datapoint arrived, I would copy the image data and shift the whole image position horizontally and using the previous point from memory and then draw a line from the previous point to the new point. When pumping up the data thoughtput, it would start lagging a lot.

It's about being able to draw the data using as few programming steps as possible.

WebGL

This is the best solution to the problems. It provides the ability to harness the power of the graphics card and program with fewer JavaScript calls that will result in more speed.

Receive Data - Upload to graphics card - Render.

Image 3

To use WebGL, I added an extra layer, that is a chain of programs that runs on the graphics card. It is represented as a JavaScript object that is instantiated one or multiple times and connected together to create an rendering chain, it is an abstraction of a vertex/pixel shader and has some functionality to tween/transition parameters. This is how the blurring occurs when an UI dialog is opened/closed.

Please visit the github page for more information and fork or star it. If people want to develop it, I think the next step for me would be to start writing some documentation.

 

License

This article, along with any associated source code and files, is licensed under The MIT License


Written By
Norway Norway
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionWithout really knowing what you are doing at all Pin
Member 255500612-Nov-15 8:11
Member 255500612-Nov-15 8:11 
AnswerRe: Without really knowing what you are doing at all Pin
Romain FABBRI12-Nov-15 11:41
Romain FABBRI12-Nov-15 11:41 
AnswerRe: Without really knowing what you are doing at all Pin
Odd Marthon14-Nov-15 2:58
Odd Marthon14-Nov-15 2:58 
GeneralRe: Without really knowing what you are doing at all Pin
Member 800715518-Nov-15 3:28
Member 800715518-Nov-15 3:28 
Video ... guess more may review the github reference "if" you had made a point of informing us of the video and active demo. And I say "demo" in reference to the link "http://www.vrtapp.com", but sadly that is just a guess as it returns;
HTTP Error 503. The service is unavailable

May be a fine project ... may be the short attention span ... either way these two small point just caused another 52 codeproject posts from today to jump ahead Frown | :(
GeneralRe: Without really knowing what you are doing at all Pin
Odd Marthon18-Nov-15 5:22
Odd Marthon18-Nov-15 5:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.