I’ve been playing around with POSIX Shared Memory and I decided that I should write a bit on the different options when it comes to shared memory. So this is going to be a bit of a comparison between shared memory with POSIX Threads, POSIX Shared Memory objects but also a random collection of facts.

Also I guess I should begin stating my main source of information: The Linux Programming Interface by Michael Kerrisk. This book is worth every penny.

Let’s start with the motivation for shared memory. Shared memory is a form of low abstraction IPC which is really fast both in terms of latency and thoughtput. Obviously there are many options to communicate processes (sockets, pipes, message queues, files) besides shared memory and shared memory should not be the first option to consider. But it’s an option nevertheless.

Once you have settled for shared memory there are several alternatives.

• Using threads (like POSIX Threads)
• System V Shared Memory Segments
• POSIX Shared Memory objects
• Anonymous Memory map and/or memory-mapped files

I’m not going to talk about System V Shared Memory segments because it’s really similar to POSIX Shared Memory but the latters gives a more Unix-like API where the shared memory is represented / controlled though a file descriptor, and you can set permissions just like in files.

I tried to summarize the advantages / disadvantages in the following table:

Option	Advantages	Drawbacks
POSIX Threads shared memory	Easy All threads share the same virtuall address space	All memory is shared (no control on what is shared and what is not). More difficult to debug memory corruptions.
POSIX Shared memory (shm_open/mmap)	Fine control Shared memory survives a crash Forced to use cleaner data structures Can use file permission to restrict sharing Memory corruption is limited	More involved setup Processes have different memory mappings Cannot store pointers in the shared memory
Anonymous mmap and fork	Same as POSIX shm No need to deal with identifiers easier to setup than POSIX shared memory Processes have same memory mappings Only the mmaps are shared the rest is copy-on-write	Requires a parent/child relationship among processes

In the table I mention the virtual memory mapping issues, what is all that about? It’s about being able to store pointers on the shared memory and whenever that make sense. As I point out the memory can be mapped at different virtual addresses in different processes which means that you can’t use absolute pointer. I think this is better understood with a picture:

In the picture process 1 has stored the 0x70000100 address in the shared memory, that points to somewhere in the shared memory for process 1 but when process 2 reads that address it points to a completely different data in that process that it’s outside the shared memory. That is because the shared memory was mapped at 0x7000000 on process 1 but on 0xFF000000 on process 2.

This means that when using POSIX shared memory you will have to refrain from storing pointers and need to store offsets or indexes instead (ptrdiff_t, etc). That, in turns, means that you have to engineer your data structures for shared memory.

Here is list of random information about shared memory in no particular order:

• You need to understand volatile C keyword. C compiler assume that they program is the only one accessing the memory and therefore they may perform “optimizations” that may interfere with what you are actually trying to do. In particular the C compiler may generate code that:
  • caches the value of a memory access in a register or locally in the stack instead of accessing the “real” shared memory every time. So if you have a while (!shm->activated) loop and shm points to shared memory, the C compiler might cache the value and never access the shared memory again unless you specify use volatile
  • or the other way around. Even if you create a local variable that “copies” a value in shared memory the C compiler might optimize away that local variable and access the shared memory instead (under the false assumption that nobody else could have changed the memory there because it’s not shared).
• Instead of using volatile directly on the types you could create an ACCESS_ONCE macro in your code that change the type to volatile just for that variable access. The original ACCESS_ONCE() comes from the Linux Kernel. You can find more about it in this LWN.net article.
• Threads share everything. Memory and the virtual memory mappings for that memory. That is very conveniente but maybe that it’s too much free access. You lose control and protection. A thread can corrupt any memory of any other thread and therefore makes debugging a nightmare.
• In constrast process that share memory through shared memory object only share that specific memory and nothing else. providing more control and although a process can corrupt that shared memory, the corruption is “limited” to that shared memory only. That will make debugging easier.
• Virtual Memory
  • If you use threads the memory is mapped in the same virtual address in all threads, therefore pointer stored in that memory doesn’t need any translation. It’s very convenient since you can just store any data structure natively and you “don’t need to understand” what it’s going on.
  • POSIX shared memory is not mapped in the same addresses in all processes. So you can’t really store absolute pointers in the shared memory because they point to different data in different processes.
  • POSIX shared memory is more incovenient in that sense but then again it forces you to use “cleaner” data structures in the share memory which I think at the end it lead to better design.
• Altought there is MAP_FIXED option in mmap that can be used to map the shared memory to a specific address, there is no good portable way to find a suitable mapping for all processes involved
  • In 64-bit space you can assume that the “high” addresses (about 48-bit) are not used and just map there. But it’s not guaranteed
  • You would need to have some synchronization among all processes, to reach a consensus on what address to use for the mapping.
  • So it’s not practical.
• At the end you need some synchronization methods, like file locking, socket based, POSIX semaphores, etc.

Which datastructures are suitable for being put in POSIX shared memory:

• Ring buffers
• Arrays/Buffers/Pools

I will write a follow up post on how to write a lockfree ringbuffer stored on POSIX shared memory in my next post.