Multiple threads writing to a single file in C

I wrote a library and utility to do exactly this many years ago. It was for event message logging. I ended up using a big buffer in shared memory to hold the logged message data and then I had a separate process that monitored the shared memory and wrote it to a file periodically. We used a mutex to control access to the buffer.

This meant the normal sequence of events to log an event was acquire the mutex, copy the message text to the first available buffer slot, and release the mutex. It was pretty fast too. We had dozens of processes running so it was worked pretty heavily. The file writing process waited until the buffer was almost full and then wrote the buffer to a file and cleared it. It also wrote the buffer at midnight when the day rolled over. This is because we kept one file for all the events of one day.

The buffer of messages was around 1MB and writing it to the file was slow. The writer had to open the log file, seek to the end of it, write all the data, and then close the file, all while holding the mutex. That took a surprisingly long amount of time which held up the worker processes and threads. To speed things up, I had it copy the entire buffer to a locally allocated buffer and then write the data from there. This took a little longer BUT it meant the mutex was held (and workers blocked) for less than a millisecond, just for the duration of the memory copy. The sequence of events for writing the message buffer was then allocate memory for the copy, acquire the mutex, copy the buffer, reset the buffer's counters, release the mutex, open the file and seek to its end, write the data, close the file, release the allocated memory.

It is important to note my system used multiple processes so I used a mutex to control access to the buffer. I also wrote this for a multiple threaded environment and for it I used a critical section to control access because it is much more efficient than a mutex. This is because a mutex is a kernel-level object that can work between multiple processes. A critical section is not and can work only between multiple threads of a single process. Since your system is a single process, a critical section would be best. The logic and implementation is essentially the same for both designs except the file writer is in a separate thread instead of a separate process and a critical section is used instead of a mutex for access control. Also, the buffer can be in local heap memory instead of shared memory but that is optional.