Dart Ray Tracer Benchmark (Compared to C#)

As an exercise in Dart programming, this morning I've ported this Raytracer demo from C# to Dart.

You can get it from Github.

Two years ago, I used the same raytracer program to compare the performances of two C#-to-JavaScript compilers, JSIL and Saltarelle (see my blogpost).

Now I use the same program to compare C# and Dart.

Speed comparison is a marginal topic here (though I will be curious to see who is fastest in rendering the raytraced scene), I am more interested in the overall programming experience as this is my first attempt at Dart (apart the classic "hello world"). In brief, I have to decide if Dart is worth embracing, perhaps for a future replacement of C#, considering that Microsoft is not pushing it for client-side web development (they are investing in TypeScript instead).

Porting the raytracer Demo

So I download the 64 bit windows version of the Dart editor and decompress in my desktop. The editor doesn't need to be installed, it just runs from the directory. Good.

The editor is a stripped version of Eclipse, the same IDE is used for Java programming. For a long term Visual Studio user, it's a rather chaotic, inconsistent and ugly-looking IDE, and I have to force myself to get accustomed to it. My locale (Italian) apparently is not available (as any other locales), so I have to stick to English. Not a good first impression.

While editing the code, the IDE flickers a lot and lags in intellisense/syntax highlighting. Sometimes typing "." or "(" does not produce the intellisense dialog (don't know why!). And if the source code has too many errors, the whole syntax correction is unstable and unreliable, for instance for a while, it gave me a clueless "int is not a type", making me wonder if I had correctly imported the libraries.

Anyway at the end, I got used to the IDE and learned to cope with its defects.

For the raytracer demo, I started a new project, and copied the .html and .cs files from the old C# project. I then renamed to .dart and started to convert it all. The whole process, from installing Dart to seeing the raytraced image, took about three hours. I was afraid of getting stuck somewhere, but to my luck that didn't happen. Fear of the unknown, I suppose.

Knowing very little of Dart, I decided to resort to Google for everything, even for the most trivial problem. Dart documentation is scarce, but a trick I've learned is that if you "open declaration" of a method, you are taken directly to the source code and can see directly the API comments and/or peek around. Sometimes, this is quicker than Googling.

A page that helped me a lot is this comparison between languages, for those who already know another mainstream language. It bootstraps you in a very short time.

Back to the code. The first thing I do is to remove by find'n'replace all occurrences of "public" because in Dart, everything is public by default. Good, C# is too verbose in this regard. In my code, there are also some "private" variables that I rename by prefixing them with "_". That's how Dart marks privates. Don't like much this design, but I comprehend where it comes from.

Another easy replace is "extends" in place of ":" for class inheritance.

float data type is not available in Dart so I replace it with double, removing also the "f" from numeric constants (like 3.1415f).  It doesn't matter much because even in C#, floats are mapped to the JavaScript number data type.

Looking at the docs, I find that Double.MaxValue and Double.MinValue are different, in Dart they are: double.MAX_FINITE and double.MIN_POSITIVE.

Dart has no namespaces (it has libraries instead), so I completely remove them from the code.

import takes the place of using in Dart, with a little difference. You import files instead of namespaces. So for instance, if your projects has three .dart files, you have to import them where needed. In C#, once the files are added to the project, there is no need to declare them, unless they are on a different namespace (in that case, you use using). Dart approach is more dynamic, and clearly reminds of the JavaScript approach (and honestly, I think that makes sense in the context of a dynamic language).

Overall, I had to import the following libraries:

• "dart:html" to be able to work with the canvas (the raytraced image)
• "dart:js" for JavaScript interop (see later)
• "dart:math" for sqrt() (in place of Math.Sqrt).  Math.Floor() is done via double.floor()

To my surprise, Dart has no cast (e.g. (double)x), and in a certain sense, it's less tolerant than C#. For example, you can't pass "0" or "1" as a double parameter, but you have to be specific by writing "0.0" and "1.0". A bit annoying. Also, I had to specify the ~/ operator when dividing by an int number, which looks a bit weird.

Differently from C#, const(s) are necessarily static, while overloading operators are not static. It doesn't make a big difference, but now that I think of it, it makes more sense, because constants are defined regardless of the object instance, while operators do work on instances.

Two nice Dart features that helped to shorten the code, are constructor initialization (now also featured in C# 6.0) and property getter/setter with the  => syntax. For example:

int get ElapsedMilliseconds => getTime() - start_time;  

This syntax is not very intuitive and one needs to get used to it; but reading the Dart documentation it seems to be coherent with the approach of having functions as first-class citizens (something I really miss in C#).

The code shortens also for abstract classes, once a class is declared abstract, there is no need to specify "abstract" for methods, nor to write the "override" keyword when implementing the member.

Another nice feature is string interpolation, which is the equivalent of C#'s String.Format. DartEditor is able to perform in-string syntax highlight for string interpolation. Unexpected, very good.

Parameters by reference: apparently Dart doesn't have an equivalent of the "ref" C# keyword (I guess it's the same for "out"). But I discovered in my code the only "ref" usage was unnecessary, so I just dropped it. I like Dart's design here, arguments by reference make code unreadable. If more than a return parameter is needed, why not returning a structured object? (easy in true dynamic languages).

The code conversion became more complex when I had to do JavaScript interop. In this regard, the C# Saltarelle compiler is more practical because gives you full access to JavaScript by the mean of the[InlineCode] attribute. Here C# is totally different from Dart: the C# compiler doesn't pretend to live in a proprietary VM, but assumes from the start the code is running within the JavaScript VM, so all objects are accessible and tricked to be true C# objects (by the mean of [InlineCode] and other attributes like [Imported]). Working with inline JavaScript code is similar to going down at the assembler level, it can be more efficient but it's no longer C#. But that fits perfectly in the strange world of C# run by the browser.

Dart can't do the same because it's meant to run in its separate VM, and has to work the regardless whether it's running in a browser or not. So to be able to access JavaScript objects in Dart, you have to do interop.

Interop is really tedious to write, to a certain extent, it resembles working with Reflection in C#. But luckily for me, I had just one object to convert. It's called MersenneTwister, and it's used for the sole reason of providing the same random numbers across the two different implementations. Since the raytraced scene is generated randomly, I want exactly the same scene in both C# and Dart so I can compare their execution speeds. In a different context, I would have used Dart's native Random functions.

That's why I have a custom Random (and DartEditor will warn of the name conflict) and also a custom StopWatch, just to be sure time measurements are the same between the two implementations.

I was very happy to see that the interfaces for working with the canvas (CanvasElement, CanvasRenderingContext2D and ImageData) are almost identical to C#, so I had to do only minor changes.

Now it's finished, the code is completely converted.

I click run, Dartium quickly opens  and the familiar raytraced image is rendered on the screen. Cool, I didn't expect it to work at the first attempt!!!

Being to good to be true, I check for caching, fearing that the old C# was called in some way. But no, it's really Dart code running and doing ray-tracing! Hurray!

After that, it took me a while and lot of Googling to discover how to generate the needed JavaScript files. While developing, your program will run in Dartium within the DartVM. You can "Manage Launches" and run on generic browsers too, but if you want your JavaScript files generated for deploy, you have to click the tool option "Pub Build".

Ok. Done. All is ready. Let's see the benchmark results!

Results

I tested the program my old AMD Athlon II X4 640 CPU, running @3GHZ with 8GBRAM and Windows 7 X64. It's the same machine I used two years ago for the JSIL-vs-Saltarelle test.

Browsers:

• Firefox: 31.0
• Chrome: 36.0.19
• Chromium: 36.0.19
• Internet Explorer: 11.0.96

The program is set to draw the random scene pictured above, in a 640x480 canvas and report the average time per pixel spent in calculation (expressed in milliseconds).

Since there is a certain variability among different runs, I repeated the test several times taking the smallest number (this aint very scientific, I know). Such variability is evident in Chrome, where the first run is usually a lot faster. Refreshing the page never gives the same fast numbers. Maybe it's because of garbage collector kicking in, who knows!

Dart Results

When run in Chromium (aka Dartium), Dart can be executed in native mode because of the embedded DartVM. In all other cases, Dart code needs to be transpiled to JavaScript.

1. Dartium (DartVM) unchecked mode: 0.002,8 ms/pixel
2. Dartium (DartVM)   checked mode: 0.006,3 ms/pixel
3. Chrome   (JavaScript): 0.025,6 ms/pixel
4. Explorer (JavaScript): 0.108,3 ms/pixel (*)
5. Firefox  (JavaScript): 0.112,9 ms/pixel

(*) For some strange reason, Explorer didn't work from outside of the DartEditor, giving the exception "unable to bind to a null object".

C# Results

These numbers come from running the C# program (Chrome result is different from the two years ago test, due to browser updates).

• Firefox: 0.000,3 ms/pixel
• Chrome: 0.003,6 ms/pixel
• Explorer: 0.0126,1 ms/pixel

Remarks

First, we notice the native Dart VM fulfils the promise of being faster than compiled JavaScript. It's a 10x increase between unchecked mode and the fastest JavaScript browser. Not bad, but we expected that.

What we didn't expect is the poor performance of DartVM compared to C#. They are on the same order of magnitude, meaning that there is no big difference of using transpiled JavaScript or going native with Dart. If that holds true, the whole Dart purpose is in danger!

What are the reason of this? Hard to say. For sure, JavaScript runtimes have benefited from browsers war between vendors, while Dart is young and has no competitors. This is confirmed if you look at the numbers of the two years ago test: you will notice that, on the same machine, Chrome has halved its execution speed, in just two years!

Anyway what is totally disappointing for me is the even worse performance of the compiled JavaScript (dart2js). It's ten time slower than the C# counterpart! It is evidently slower even without doing any measurement, just by eye. Not only, Dart output is also bigger in terms of file size. The file dart_raytracerdemo.dart.js is 360 KB long (unminified), while the C# counterpart (mscorlib.js+raytracerdemo.js) is only 139 KB. The Dart team has a lot to improve here.

Conclusion

Overall, it was an interesting experience to port this program from C# to Dart. My conclusion is that Dart is a promising language, featuring all that I miss in C#, but still too young to go all-in with it. But it's also true that Microsoft failed to adopt C# as main web development language, so I would be happy if Dart takes its place one day.

CodeProject