The Importance of a Low Level Native Flat API in Windows

Innovation in software development is always important. New ideas for better software often requires “thinking outside the box” as they say. In the Windows world, software development can easily become stagnated if developers push too much for more higher level functionality at the expense of low level functionality.

WIN32 API

Having been a WIN32 API programmer for the last 15 years, while most programmers are using the much higher level .NET and WINRT runtimes, I see more and more why low level flat APIs are so important. Actually, the term low level is deceptive. Low level does mean more work compared to high level, but actually low level also means simplicity. The concepts of low level, once learned, are actually simpler in many ways, not being hindered by overly complex higher level concepts. It is the extra work that low level requires which often turns off programmers, since it is easy to desire a quick fix for a solution (meaning little or no work). But low level coding also provides more choices for customization. It encourages new ways of doing things, since it is only limited by one's imagination and skill.

Windows, while still based on its core WIN32 API, has lost much of its simplicity. The use of object oriented programming promotes the building of more and more complex code bases, which may seemingly help programmers by providing more higher level functionality, but it actually strays away from the beautiful simplicity of a low level flat API like the WIN32 API. So while one sees the benefits of the higher level code base, why should one even care about the low level native API ?

The High Level Must Be Built from the Low Level

What we may fail to appreciate is that the high level functionality must be built using the low level. The more one improves the low level functionality, the more options one has to develop the higher level engines software is built upon. I see this more and more when new hardware devices come to the market. For example, why does a hardware manufacturer first provide access to its devices via a high level class library, rather than first via a low level simple flat API say for C, rather than C++? Build the low level APIs first. Then one can more easily build better higher level libraries. If low level libraries are available first, some software developers may find ways to exceed the functionality first intended by the hardware manufacturer. What few may appreciate is that sometimes a low level, simple, flat API may provide better performance than a high level class library might. Even things like COM, namespace classes, etc. are built upon more simpler low level APIs. For example, if one were to see how a class object used in C++ could be emulated via just using C (with things like vector arrays), one would appreciate that the higher level is actually much more complex than one might think. True, modern compilers hide such complexity from us, but that does not mean it is no longer complex. There was a simple rule in programming. Less code means less work for the CPU. The problem though is that if the code is very high level, then the less code may hide the reality of what a compiler is generating and programmers lose control of performance. Less code no longer means less work for the CPU.

High level libraries are important for faster software development. But if an operating system like Windows is only improved at the higher level, then what happens is that innovation decreases. True, the average programmer may seem more productive, but the real innovators, the ones who tinker with the low level, have less and less to work with. Where are the new compiler builders ? While innovation increases when it comes to scripted languages, where are the new native code compilers coming from? In the “indie” (independent) world of programming languages, who is building the next generation of native code compilers?

A Lesson Learned from Building a GUI Framework

Having spent the last 15 years developing a GUI framework for use with a native code compiler, working only with the native WIN32 API, I have learned a thing or two about Windows, programming languages and native code programming. First, I have a deep appreciation for the WIN32 API and how much it offers. Sadly though, with the advent of .NET, little has been done to improve the low level WIN32 API. The API docs often make reference now to some APIs being labeled as legacy. Yet likely the functionality of such APIs is still maintained in some form or fashion in the Windows core DLLs since the higher level functionality may be based on it. In the old days, when there was a need to update an API, Microsoft at times would create a similar API call, but simply append an EX at the end of the function name. For example, CreateWindow became CreateWindowEx. This made sense when significant functionality needed to be added. But at least the core API continued to be improved. Today, the core APIs in Windows seem to have little or no improvement, but instead have been replaced by COM object calls instead. This programmer suggests that it may be better to provide both, low level API improvements and higher level companions. Sadly though, native APIs are an afterthought rather than the primary concentration today.

Watch the video of Herb Sutters talks “Why C++?” on Channel 9.

If high level functionality becomes more important than low level, then we may lose the real software innovators who create totally new ways to do things.

The richness of the WIN32 is, simply put, amazing! Microsoft put so many different ways of customizing things into the core WIN32 API, the choices are almost endless. Manufacturers of new hardware could learn a thing or two from this, by building low level libraries for their hardware which would appeal to C (not simply C++, but real C) programmers first and then build higher level wrappers over them for other languages. I had the opportunity to beta test a unique user input device which is now becoming popular and was shocked to find the interface libraries were only provided in C++ class libraries. The idea of providing the library in a simple DLL for use with C appeared to not have been considered and workarounds for languages like C had to be done via a C++ wrapper converting the library to a simple flat API usable by C. Personally, I would have thought that C would be the first choice, a simple flat API and then build a C++ class on top of that. It is the old “cart before the horse” syndrome.

In building my own GUI framework (on top of the WIN32), I found that building the framework in levels made a lot of sense. Often I would start adding functionality as a low level feature first and then build upon it. For example, I wanted to tap into OwnerDraw so I could customize controls. Rather than build a high level engine on top of OwnerDraw, I simply built a simpler low level ownerdraw engine which allowed me to have access to the core functionality of ownerdraw. It would take a lot more work for a user to draw something, but it provided more choices at a lower level. Then I built upon that engine a higher level (I called it Simplified OwnerDraw) where common tasks were built into higher level functions, so many common tasks could be done with just one line of code. But users of the framework though had a choice. They could use the higher level functions (ie. draw a 3D Button) if it sufficed, but if they needed more they could fall back to lower level drawing and draw things exactly as they desired. But I learned that the low level had to come first.

It Would Be Nice!

As a Windows programmer, I think it would be nice to see a resurgence in interest in improving and building upon the low level WIN32 API. By going back and improving the core APIs in Windows, even higher level layers like .NET could benefit. It could even be possible to see new innovations in the Windows world beyond what we think possible currently. Imagine a full blown version of Windows capable of running on Intel’s Edison device ! By fine tuning the WIN32 API and building upon it, it is possible. This is not simply wishful thinking or theory. When I was designing my own GUI framework (on top of the WIN32 API), I purposedly didn’t use the latest version of Windows or the latest computers. Unlike most programmers who write their software on the lastest, fastest computers and then test them on a legacy PC to see if it will work, I did the opposite. For example, when Windows XP was the current version of Windows I developed my GUI framework on a Windows 95 computer with a very slow 233 mhz CPU. While most XP computers were running CPUs in the gigahertz ranges (2 ghz average), I was developing on a 233 mhz CPU based computer and less RAM than typical for an XP computer. If my GUI framework ran on a less powerful machine, then imagine how well it would run on the latest PC. That is the power of the raw WIN32 API ! Minimal hardware and better performance.

It is not inconceivable to be able to write powerful Windows software which could run on an Intel Edison device (aka. Quark chip). Windows currently can’t, but it could if the operating system was slimmed down and the WIN32 API was tapped into.