Porting a Large Project from Windows to Linux

• Download master.zip - 8.4 MB
• Download from repository

Introduction

To port software to a new platform, an abstraction layer that limits the use of platform-specific headers to a small number of .cpps is highly recommended. Parallel versions of those .cpps can then be developed to support the new platform without having to change the rest of the software. Implementing those .cpps involves finding analogs for the platform-specific functions of whichever platform is already supported. However, the abstraction layer may also have to evolve to handle subtle differences between the platforms. And the new platform may require the use of a new compiler, which may also mean having to use a new build tool.

Background

The code discussed in this article is taken from the Robust Services Core (RSC). If this is the first time that you're reading an article about RSC, please take a few minutes to read this preface.

Platform Abstraction Layer

RSC was developed on Windows, but the goal was to eventually support Linux as well. A platform abstraction layer was therefore defined from the outset so that Windows-specific headers would be used by a limited set of files. Windows targets were placed in *.win.cpp files that totalled about 3.5K lines of code, less than 1.5% of the project when the port finally began. The abstraction layer consisted of the following headers, for which the port would have to develop Linux targets:

Header	Description
SysConsole.h	console functions
SysFile.h	directory functions
SysHeap.h	subclass of Heap for the default C++ heap
SysLock.h	low-overhead mutex
SysMemory.h	memory allocation and protection
SysMutex.h	mutex
SysSignals.h	POSIX signals
SysThread.h	threads
SysThreadStack.h	stack traces
SysTickTimer.h	high-resolution clock
SysTime.h	time-of-day clock
SysIpL2Addr.h	IP layer 2 address and functions
SysIpL3Addr.h	IP layer 3 address and functions
SysSocket.h	socket functions
SysTcpSocket.h	TCP socket functions
SysUdpSocket.h	UDP socket functions
RscLauncher.h	automatic rebooting on failure

Linux targets were placed in *.linux.cpp files, with the IP networking targets being developed first. This was fairly straightforward because Windows and Linux are very similar when it comes to socket types and functions. But they report errors differently, which led to some refactoring that improved RSC's error reporting for sockets, an area that needed work in any case.

The effort then turned to kernel functions. When RSC originally began development, C++11 was not yet available. By the time the port started, compilers were implementing C++20. But it was C++11 that had added most of the headers that could be used for the port. Instead of implementing Linux targets, some of the platform abstraction layer could be eliminated and replaced with C++11 headers that already handled platform differences. To that end, these C++ library headers were adopted:

Header	Description
<chrono>	high-resolution and time-of-day clocks
<condition_variable>	allows a thread to wait until it is notified or a timeout occurs
<filesystem>	directory functions (C++17)
<mutex>	mutexes
<ratio>	used by <chrono>
<thread>	threads

Adopting <chrono> also allowed RSC to remove Duration.h and TimePoint.h. Although both were platform independent, they supported time intervals and timestamps in a way that was incompatible with <chrono>.

Unlike <chrono>, <thread> eliminated very little code. It creates threads, but their stack sizes can't be specified. Its thread identifiers are opaque, and it features join, which is popular in toy examples but of limited use elsewhere. However, its sleep_for function proved useful. A major problem when creating a thread object is that the underlying thread often starts to run before its thread object has been fully constructed, because it has no idea that such an object is being created to wrap it. RSC has a feature-rich Thread class that includes a Pause function, but it can't be used until the Thread object has been constructed. After that, only Pause, never sleep_for, should be used. But sleep_for allows a thread to sleep for brief intervals, only proceeding once its Thread object has been fully constructed.

After adopting the new C++ library headers, the kernel portion of the platform adaptation layer looked like this:

Header	Description
SysConsole.h	retained
SysFile.h	replaced by FileSystem.h, which uses <filesystem>
SysHeap.h	retained
SysLock.h	replaced by Lock.h, which used <mutex> (this header was later deleted)
SysMemory.h	retained
SysMutex.h	replaced by Mutex.h, which uses <mutex>
SysSignals.h	retained
SysThread.h	retained but partially replaced by Gate.h, which uses <condition_variable> and <mutex>
SysThreadStack.h	renamed SysStackTrace.h, since C++23 will introduce <stacktrace>
SysTickTimer.h	replaced by <chrono> and the simple SteadyTime.h
SysTime.h	replaced by <chrono> and the simple SystemTime.h
RscLauncher.h	retained

Static Analysis Tool

The first issue was that RSC has a C++ static analysis tool that is regularly used to clean up the code. But instead of just registering complaints, it provides an editor that allows two-thirds of them to be fixed interactively. This has become so useful that the tool must support whatever C++ language features RSC uses. And that's a problem because the tool is, in large part, a C++ compiler, and supporting all of the language features handled by a fully compliant C++ compiler isn't practical.

Adopting <chrono>, however, forced the tool to evolve. The problem wasn't so much <chrono> as its use of <ratio>, which is a rat's nest of template metaprogramming that converts one ratio to another at compile time.

ratio uses integer literals as template arguments, something that the tool didn't support. The tool wanted a template argument to be a type. It represents a type with TypeSpec, an abstract class with derived classes DataSpec (almost all types) and FuncSpec (a function type). A template name is represented by TypeName, which had a vector of TypeSpec for the template arguments that follow. Literals, however, are on another branch of the class hierarchy, below CxxToken. A TemplateArg class was therefore created. It wraps either a TypeSpec or an Expression (which can be an integer literal), and almost all of its functions simply forward to whichever of these two classes it is wrapping.

Next, the tool had to know that one instantiation of std::ratio was convertible to another. It already supported type conversions, and conversion constructors and operators. But none of these apply to two different ratios, which are two distinct template instantiations. To solve this, StackArg::MatchWith was modified to strip the template arguments from a std::ratio instantiation to detect convertibility.

Finally, the Linux target for POSIX signal handling needed to use sigaction to register its signal handler. This provides the handler with additional information about a signal so that, to simplify debugging, it can map SIGSEGV to RSC's proprietary SIGWRITE when faulty code writes to a valid address that is write-protected. However, some special needs developer had also used sigaction to name a struct that is passed to the function sigaction. This meant that the tool had to support elaborated type specifiers: prefixing the keyword class, struct, union, or enum to a type to distinguish it from something else with the same name, such as a function. Supporting this was fairly easy: Parser::GetTypeSpec was modified to look for one of those keywords before the type name.

Evolving the tool also uncovered some bugs and bizarre behaviors that had to be rectified. In the end, about as much time was spent enhancing the tool as was spent actually porting to Linux.

Differences Between Windows and Linux

During the port, various subtle differences between Linux and Windows had to be resolved.

Text files. If a text file created on Windows is read on Linux, problems occur. Windows uses \r\n (CRLF) at the end of a line, whereas Linux simply uses \n (LF). The function FileSystem::GetLine, with the same signature as std::getline, was therefore added to remove the \r, which would otherwise appear at the end of every string returned by std::getline when Linux was reading a file created on Windows.

Directory separator. Linux uses / to separate the directories in a path, but Windows uses \. The constant PATH_SEPARATOR was therefore initialized using std::filesystem::path::preferred_separator, which knows the platform on which it is running.

Heaps. Windows lets you create additional heaps with the same capabilities as the default C++ heap. Linux, however, doesn't support additional heaps. RSC was using Windows' heap interface to create additional heaps, but it had also developed its own BuddyHeap to support write-protected memory, because a Windows heap crashes if write-protected. Porting to Linux therefore meant using BuddyHeap for all of the additional heaps. However, the size of a BuddyHeap is fixed and can only be expanded by a restart, which was unacceptable for one of the heaps. RSC therefore had to implement the expandable SlabHeap to complement BuddyHeap.

POSIX signals. The C++ standard defines SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV, and SIGTERM. Windows adds SIGBREAK, which is similar to SIGINT, but Linux adds SIGBUS, which is similar to SIGSEGV.

Gate.h. This provides an object on which a thread waits until it is either notified or a timeout occurs. It replaced Windows' CreateEvent, WaitForSingleObject, and SetEvent, whose platform-independent analogs are Gate::Gate (constructor), Gate::WaitFor, and Gate::Notify. Although Gate doesn't contain much code, getting it right involved a fair bit of investigation, as well as fixes for some subtle bugs. Remarkably, Gate needs three critical-region objects to implement its interface: a condition_variable, a mutex, and an atomic_bool.

Thread identifiers. When a thread exits, RSC does not remove its identifiers (both native and RSC) from its ThreadRegistry. It only marks them Deleted, which allows debug tools to still find the native identifier and map it to RSC's identifier when displaying trace records that were captured before the thread exited. This worked well on Windows, but not on Linux. When a thread exits, the POSIX threads implementation on Linux immediately reassigns its identifier to the next thread that it creates. Because that identifier was still in the ThreadRegistry, the new thread blocked, forever waiting for its Thread object to be marked Constructed. The code had to be changed to handle this scenario.

Thread priorities. Although POSIX threads support priorities, the Linux scheduler doesn't support them in a standard build. This was a concern because, although RSC runs most threads at the same priority, it has two threads that run at higher priorities to mimic sanity and clock interrupts. It turned out that both of those threads could run at the same priority as all others and still do their jobs. This is because both Windows and Linux perform context switching at a rate that can only be described as vigorous, and so the would-be high-priority threads are scheduled frequently.

Stack traces. Although it was easy to capture a thread's stack on Linux, function names looked like they had been affected by transmission line noise when displayed. To remain compatible with C, a C++ compiler mangles function names when it adds them to an object file that includes debug information. The Windows function that maps an address to a function name demangles the name, but the equivalent Linux function does not. A separate utility must be used to do it.

Socket unblocking. RSC supports restarts, which are a partial reboot of the system. During a restart, a thread exits unless its data needs to survive. However, this poses a problem for blocked threads. How will they unblock so that they can exit? There is no way to unblock a read from cin, so CinThread always survives. For an instance of UdpIoThread, which blocks on recvfrom, the approach on Windows was to delete the thread's socket, which immediately caused recvfrom to return with an error. On Linux, nothing happened: recvfrom continued to block the thread. Some people disparage Windows, but I find it to be a good operating system. In any case, a different solution now had to be found, which was to have a UdpIoThread send a message to its own socket to unblock its recvfrom.

Analogous Functions

If you're porting software from Windows to Linux or vice versa, you often have to find the equivalent of some function on the new platform. Searching on the current function's name and the new platform's name often returns a link to a site where your question has been answered. If that doesn't work, Windows functions are documented here, and Linux functions are documented here. At least Windows groups its functions by topic, unlike Linux. Looking at all of this, I became even more convinced that a large operating system should use an object-oriented language, which would help to organize its morass of functions.

The following tables list analogous Windows and Linux functions used in RSC's platform abstraction layer. First, the kernel functions:

Header	Windows	Linux
SysConsole.h	SetConsoleTitle	a bizarre escape sequence
SysHeap.h	HeapAlloc HeapSize HeapFree HeapValidate	malloc malloc_usable_size free mcheck (not thread safe)
SysMemory.h	VirtualAlloc VirtualFree VirtualLock VirtualProtect VirtualUnlock	mmap munmap mlock mprotect munlock
SysThread.h	SetPriorityClass _beginthreadex CloseHandle signal SetThreadPriority	setpriority pthread_create no analog when exiting a thread sigaction (preferable) pthread_setschedprio
SysStackTrace.h	RtlCaptureStackBackTrace SymFromAddr and SymGetLineFromAddr64	backtrace backtrace_symbols and __cxxabiv1::__cxa_demangle
RscLauncher.h	CreateProcessA WaitForSingleObject and GetExitCodeProcess	posix_spawnp waitpid

And the IP networking functions, which have the same, or very similar, names:

Header	Windows	Linux
SysIpL2Addr.h	gethostname	gethostname
SysIpL3Addr.h	getaddrinfo freeaddrinfo getnameinfo	getaddrinfo freeaddrinfo getnameinfo
SysSocket.h	socket bind getsockopt setsockopt ioctlsocket closesocket	socket bind getsockopt setsockopt ioctl close
SysTcpSocket.h	connect listen accept WSAPoll getsockname getpeername recv send shutdown	connect listen accept poll getsockname getpeername recv send shutdown
SysUdpSocket.h	recvfrom sendto	recvfrom sendto

Building with Multiple Compilers

When the port started, RSC was built using the .vcxproj files created through VS2022's Properties sheets. To support builds on multiple platforms, Visual Studio recommended adopting CMake, about which I knew nothing. It seemed to be a popular build system, was well documented, and there were lots of questions and answers about it on stackOverflow. However, all this information was fragmented, and the examples were usually simple. Nothing addressed how to migrate a large project from VS2022 builds to CMake builds.

Fortunately, a search turned up an open-source tool, CMakeConverter, which analyzes .vcxproj files to generate CMake files. Using this tool was straightforward, and it did a good job creating the files needed by CMake. But there was still some work to do. RSC had a properties file that most of its projects shared, and the tool didn't know how to handle this. But the files that it generated gave a good sense of how to structure a large project in CMake. And one of those files, GlobalSettingsInclude, was clearly intended to serve the same purpose as that shared properties file.

Once the overall CMake framework existed, it was possible to gradually modify its files after finding answers to things like how to set compiler and linker options. VS2022 had also evolved to be rather well integrated with CMake. No longer do you open VS2022 from a solution or project. You can delete those files and just open the folder containing the source code. VS2022 then analyzes its CMakeLists file to configure the solution. Clicking on Manage Configurations… opens a tab that allows you to add new build configurations (compiler plus platform combinations), which VS2022 saves in a JSON file.

It didn't take long to successfully perform a build. This was a surprise, because I loathe learning a tool that needs a lot of configuration, and the C++ build process is ridiculous to begin with. Another surprise was that x64 builds were much faster, because they now used Ninja instead of VS2022.

Builds could now use the gcc or clang compilers as well as MSVC. Both gcc and clang generated new compiler warnings. Some were resolved by changing the code, and others were suppressed by adding compiler options to CMake files. However, a problem occurred with clang. RSC has a set of tests to see if it survives when bad things happen. One test is to divide by zero. This test passed if compiled with MSVC or gcc, but clang entered an infinite loop that kept dividing by zero. After reading some Windows documentation, I added a call to _fpreset in RSC's structured exception handler for Windows, which fixed the problem.

To test on Linux, I settled on WSL2 and Ubuntu. When RSC is launched through the Debug menu, VS2022 creates a Linux Console Window tab. But VS2022 needs a bit of work in this area, because it doesn't forward all keyboard events to that window. I discovered that just hitting the enter key did not cause std::getline(cin, str) to return; enter had to be preceded by a space. Similarly, ctrl-C did nothing. To confirm that RSC wasn't the problem, those things had to be tested on an actual Ubuntu console.

Until recently, changing the build process for this port would have required far more effort. But VS2022 has evolved to the point where it supports Linux and other C++ compilers rather well. I originally thought that I would also end up with a degraded IDE experience, even if I managed to get everything to work. But the transition was far easier than anticipated, and I actually prefer using CMake.

History

• 7^th July, 2022: Expand the section about the static analysis tool
• 4^th July, 2022: Initial version