Inter-Process Communication - Part 1

Scope

The purpose of this article is to get the readers familiar with the different mechanisms that are available for communicating between two or more processes. This may also serve as a tutorial for the novice programmer. There might be several good tutorials on this subject, but here I will try to communicate my explorations of this subject. This article may not be technically perfect. Please send your suggestions and queries to .

Introduction

Inter-Process Communication, which in short is known as IPC, deals mainly with the techniques and mechanisms that facilitate communication between processes. Now, why do we need special separate mechanisms or techniques for communicating between processes? Why isn't it possible to have information shared between two processes without using such special mechanisms?

Let us start from something primitive. Imagine you have two glasses completely filled with water. One glass contains hot water and the other contains cold water. What can you do to make the temperature of water in both the glasses equal? The simplest answer will be to mix the water from both the glasses in a glass with much bigger capacity. Once water is mixed, the temperature becomes equal after some time. If one can remember, this will be framed as a problem with some numerical data in a High-School Physics examination. If we go by principles, then the phenomenon here is conduction. If we go by our topic of IPC, then we can say that since the two glasses were full, we had to use another glass with a larger capacity to mix the contents in order to balance their heat energy.

Have you ever wondered about the communication medium used in telephones? What about the blood transporting system in the human body which communicates blood to different parts of the body? What about my fingers which are typing this document? My brain is doing so many things at a time. How is it directing one of my fingers to hit one key and some other finger to hit another key? How is it synchronizing the typing work that is done by both my hands? How is it also directing me to type the letters of a word that are actually coming to my mind?

Don't worry. I am not going to give a class in Biology. But it would be good if one can imagine a few more situations where we are using inter-process communication, though not necessarily in the human body or in a computer program.

So, where are we now? We know that some medium or other is required for communication between different processes. Similarly, when it comes to computer programs, we need some mechanism or medium for communication. Primarily, processes can use the available memory to communicate with each other. But then, the memory is completely managed by the operating system. A process will be allotted some part of the available memory for execution. Then each process will have its own unique user space. In no way will the memory allotted for one process overlap with the memory allotted for another process. Imagine what would happen otherwise!

So, now the question - how do different processes with unique address space communicate with each other? The operating system's kernel, which has access to all the memory available, will act as the communication channel. Similar to our earlier example, where the glass with hot water is one process address space, the glass with cold water is another, and the glass with the larger capacity is the kernel address space, so that we pour both hot water and cold water into the glass with larger capacity.

What next? There are different IPC mechanisms which come into use based on the different requirements. In terms of our water glasses, we can determine the specifics of both pouring the water into the larger glass and how it will be used after beign poured.

Basic IPC

OK, enough of glasses and water. The IPC mechanisms can be classified into the following categories as given below:

pipes
fifos
shared memory
mapped memory
message queues
sockets

Pipes

Pipes were evolved in the most primitive forms of the Unix operating system. They provide unidirectional flow of communication between processes within the same system. In other words, they are half-duplex, that is, data flows in only one direction. A pipe is created by invoking the pipe system call, which creates a pair of file descriptors. These descriptors point to a pipe inode and the file descriptors are returned through the filedes argument. In the file descriptor pair, filedes[0] is used for reading whereas filedes[1] is used for writing.

Let me explain a scenario where we can use the pipe system call: consider a keyboard-reader program which simply exits after any alpha-numeric character is pressed on the keyboard. We will create two processes; one of them will read characters from the keyboard, and the other will continuously check for alpha-numeric characters. Let us see how the filedes returned by pipe can be of use in this scenario: (Text version: kbdread-pipe.c.txt)

/***** KEYBOARD HIT PROGRAM *****/
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <ctype.h>

int filedes[2];

void *read_char()
{
	char c;
	printf("Entering routine to read character.........\n");
	while(1) {
		/* Get a character in 'c' except '\n'. */
		c = getchar();
		if(c == '\n')
			c = getchar();
		write(filedes[1], &c, 1);
		if(isalnum(c)) {
			sleep(2);
			exit(1);
		}
	}
}

void *check_hit()
{
	char c;
	printf("Entering routine to check hit.........\n");
	while(1) {
		read(filedes[0], &c, 1);
		if(isalnum(c)) {
			printf("The key hit is %c\n", c);
			exit(1);
		} else {
			printf("key hit is %c\n", c);
		}
	}
}
		
int main()
{
	int i;
	pthread_t tid1, tid2;
	pipe(filedes);
	/* Create thread for reading characters. */
	i = pthread_create(&tid1, NULL, read_char, NULL);
	/* Create thread for checking hitting of any keyboard key. */
	i = pthread_create(&tid2, NULL, check_hit, NULL);
	if(i == 0) while(1);
	return 0;
}

Save and compile the program as cc filename.c -lpthread. Run the program and check the results. Try hitting a different key every time.

The read_char function simply reads a character other than '\n' from the keyboard and writes it to filedes[1]. We have the thread check_hit, which continuously checks for the character in filedes[0]. If the character in filedes[0] is an alpha-numeric character, then the character is printed and the program terminates.

One major feature of pipe is that the data flowing through the communication medium is transient, that is, data once read from the read descriptor cannot be read again. Also, if we write data continuously into the write descriptor, then we will be able to read the data only in the order in which the data was written. One can experiment with that by doing successive writes or reads to the respective descriptors.

So, what happens when the pipe system call is invoked? A good look at the manual entry for pipe suggests that it creates a pair of file descriptors. This suggests that the kernel implements pipe within the file system. However, pipe does not actually exist as such - so when the call is made, the kernel allocates free inodes and creates a pair of file descriptors as well as the corresponding entries in the file table which the kernel uses. Hence, the kernel enables the user to use the normal file operations like read, write, etc., which the user does through the file descriptors. The kernel makes sure that one of the descriptors is for reading and another one if for writing.

I am not going to go into the details of the pipe implementation on the kernel side. For further reading, one can refer the books mentioned at the end of this article.

FIFOs

FIFOs (first in, first out) are similar to the working of pipes. FIFOs also provide half-duplex flow of data just like pipes. The difference between fifos and pipes is that the former is identified in the file system with a name, while the latter is not. That is, fifos are named pipes. Fifos are identified by an access point which is a file within the file system, whereas pipes are identified by an access point which is simply an allotted inode. Another major difference between fifos and pipes is that fifos last throughout the life-cycle of the system, while pipes last only during the life-cycle of the process in which they were created. To make it more clear, fifos exist beyond the life of the process. Since they are identified by the file system, they remain in the hierarchy until explicitly removed using unlink, but pipes are inherited only by related processes, that is, processes which are descendants of a single process.

Let us see how a fifo can be used to detect a keypress, just as we did with pipes. The same program where we previously used a pipe can be modified and implemented using a fifo.

(Text version: write-fifo.c.txt)

/***** PROGRAM THAT READS ANY KEY HIT OF THE KEYBOARD*****/
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <ctype.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>

extern int errno;

void *read_char()
{
	char c;
	int fd;
	printf("Entering routine to read character.........\n");
	while(1) {
		c = getchar();
		fd = open("fifo", O_WRONLY);
		if(c == '\n')
			c = getchar();
		write(fd, &c, 1);
		if(isalnum(c)) {
			exit(1);
		}
		close(fd);
	}
}

int main()
{
	int i;
	pthread_t tid1;
	i = mkfifo("fifo", 0666);
	if(i < 0) {
		printf("Problems creating the fifo\n");
		if(errno == EEXIST) {
			printf("fifo already exists\n");
		}
		printf("errno is set as %d\n", errno);
	}
	i = pthread_create(&tid1, NULL, read_char, NULL);
	if(i == 0) while(1);
	return 0;
}

Compile this program using cc -o write_fifo filename.c. This program reads characters (keypresses), and writes them into the special file fifo. First the program creates a fifo with read-write permissions using the function mkfifo. See the manual page for the same. If the fifo exists, then mkfifo will return the corresponding error, which is set in errno. The thread read_char continuously tries to read characters from the keyboard.

Note that the fifo is opened with the O_WRONLY (write only) flag .

Once it reads a character other than '\n', it writes the same into the write end of the fifo. The program that detects it is given below: (text version detect_hit.c.txt):

/***** KEYBOARD HIT PROGRAM *****/
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>

extern int errno;

void *check_hit()
{
	char c;
	int fd;
	int i;
	printf("Entering routine to check hit.........\n");
	while(1) {
		fd = open("fifo", O_RDONLY);
		if(fd < 0) {
			printf("Error opening in fifo\n");
			printf("errno is %d\n", errno);
			continue;
		}
		i = read(fd, &c, 1);
		if(i < 0) {
			printf("Error reading fifo\n");
			printf("errno is %d\n", errno);
		}
		if(isalnum(c)) {
			printf("The key hit is %c\n", c);
			exit(1);
		} else {
			printf("key hit is %c\n", c);
		}
	}
}
		
int main()
{
	int i;
	i = mkfifo("fifo", 0666);
	if(i < 0) {
		printf("Problems creating the fifo\n");
		if(errno == EEXIST) {
			printf("fifo already exists\n");
		}
		printf("errno is set as %d\n", errno);
	}
	pthread_t tid2;
	i = pthread_create(&tid2, NULL, check_hit, NULL);
	if(i == 0) while(1);
	return 0;
}

Here, again, it first tries to create a fifo which is created if it does not exist. We then have the thread check_hit which tries to read characters from the fifo. If the read character is alphanumeric, the program terminates; otherwise the thread continues reading characters from the fifo.

Here, the fifo is opened with the flag O_RDONLY.

Compile this program with cc -o detect_hit filename.c. Now run the two executables in separate terminals, but in the same working directory. Irrespective of the order in which you run, look for the message fifo already exists on the console. The first program (either of the two) that you run will not give any error message for creation of the fifo. The second program that you run will definitely give you the error for creation of the fifo. In the terminal where you run write_fifo, give input to standard output from your keyboard. You will get the message regarding the key hit on the keyboard on the terminal running the executable detect_hit. Analyze the working of the two programs by hitting several keys.

I have used two different programs for exhibiting the usage of fifos. This can be done within a single program by forking the routines which are called in the two program as threads. But I did this to show that unlike pipes, fifos can be used for communication between unrelated processes.

Now try running the program again. You will get the message that the fifo already exists even when you first run either of the two programs. This shows that fifos are persistent as long as the system lives. That is, the fifos will have to be removed manually - otherwise they will be permanently recognized by the file system. This is unlike pipes which are inherited as long as the process that created the pipe is running. Once this process dies, the kernel also removes the identifiers (file descriptors) for the pipe from the the file tables.

The usage is rather simple and the main advantage is that there is no need for any synchronization mechanism for accesses to the fifo. There are certain disadvantages: they can only be used for communication between processes running on the same host machine. Let us explore other IPC mechanisms to see what have they in store.

Shared Memory

Shared Memory is one of the three kinds of System V IPC mechanism which enables different processes to communicate with each other as if these processes shared the virtual address space; hence, any process sharing the memory region can read or write to it. One can imagine some part of memory being set aside for use by different processes.

The System V IPC describes the use of the shared memory mechanism as consisting of four steps. Taken in order, they are:

Fetching an identifier for the shared memory area - shmget (shared memory get)
Using the identifier to get the shared memory address - shmat (shared memory attach),
Detaching the shared memory area after use - shmdt (shared memory detach) and
Finally using the address to control accesses, permissions, receive information and destroy the shared memory area - shmctl (shared memory control).

Let us examine the workings of the above system calls. Recall the keyboard hit program; we shall, once again, see another version of it, this time using the system calls associated with the shared memory mechanism.

The code given below creates a shared memory area and stores the information of any key hit on the keyboard. Let us see the code first: (text version: write-shm.c.txt)

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <errno.h>
#include <string.h>
#include <ctype.h>

extern int errno;

#define SIZE	1

char *read_key;
int shmid;

int shared_init()
{
	if((shmid = shmget(9999, SIZE, IPC_CREAT | 0666)) < 0) {
		printf("Error in shmget. errno is: %d\n", errno);
		return -1;
	}
	if((read_key = shmat(shmid, NULL, 0)) < 0) {
		printf("Error in shm attach. errno is: %d\n", errno);
		return -1;
	}
	return 0;
}

void read_char()
{
	char c;
	while(1) {
		c = getchar();
		if(c == '\n') {
			c = getchar();
		}
		strncpy(read_key, &c, SIZE);
		printf("read_key now is %s\n", read_key);
		if(isalnum(*read_key)) {
			shmdt(read_key);
			shmctl(shmid, IPC_RMID, NULL);
			exit(1);
		}
	}
}

int main()
{
	if(shared_init() < 0) {
		printf("Problems with shared memory\n");
		exit(1);
	}
	read_char();
	return 0;
}

Here we have a shared memory variable named read_key. The program first initializes the shared memory area read_key. This is done by generating a shared memory identifier shmid using the system call shmget. In the context of the program, the first parameter for shmget is 9999, which is the key. This key is used to allocate a shared memory segment. The second parameter, SIZE (defined as a macro with the value 1), suggests that the shared memory segment will hold only one of the type of the shared memory variable, that is, only 1 character. The IPC_CREAT flag (third parameter) suggests that a new shared memory segment has to be created, with read-write permissions (IPC_CREAT logically OR ed with 0666). This will return a valid shared memory segment identifier on successful allocation. The identifier will be stored in shmid. If shared memory segment allocation fails, then -1 is returned and the errno is set appropriately.

The key which is used to get a shared memory segment can be generated randomly using the built-in function ftok to get a unique key. Refer to the manual page for the usage.

Once the segment identifier is obtained, we have to attach the shared memory segment to some address. This is done with the shmat system call. This uses the segment identifier shmid as the first parameter. The second parameter is the address of the shared memory segment; when this is given as NULL (as in this program), the kernel will choose a suitable address. The third parameter is the flag specification which can be set if required or left as zero (see man page of shmdt for details). On success the shared memory segment is attached to read_key, otherwise -1 is returned along with the appropriate setting of the errno.

If either shmget or shmat fails, the process terminates. On success from both system calls, we proceed by invoking the read_char function, which reads keyboard inputs other than '\n' ("Enter" key) and copies them to read_key in the shared memory. If the keyboard input is an alphanumeric character, the program stops reading inputs from the keyboard and the process terminates.

We have another program running separately (it does not have to be in the same working directory) in the local system, which tries to read the data written in the shared memory area. The code is given below: (text version: read-shm.c.txt)

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <errno.h>
#include <string.h>
#include <ctype.h>

extern int errno;

#define SIZE	1

char *detect_key;
int shmid;

int shared_init()
{
	if((shmid = shmget(9999, SIZE, 0444)) < 0) {
		printf("Error in shmget. errno is: %d\n", errno);
		return -1;
	}
	if((detect_key = shmat(shmid, NULL, SHM_RDONLY)) < 0) {
		printf("Error in shm attach. errno is: %d\n", errno);
		return -1;
	}
//	detect_key = NULL;
	return 0;
}

void detect_hit()
{
	char c;
	c = *detect_key;
	while(1) {
		if(c != *detect_key) {
			if(isalnum(detect_key[0])) {
				printf("detect_key is %s\n", detect_key);
				shmdt(detect_key);
				shmctl(shmid, IPC_RMID, NULL);
				exit(1);
			} else {
				printf("detect_key is %s\n", detect_key);
			}
			c = *detect_key;
		}
	}
}

int main()
{
	if(shared_init() < 0) {
		printf("Problems with shared memory\n");
		exit(1);
	}
	detect_hit();
	return 0;
}

Here, again, we have a shared memory initialization routine, which in fact does not create a new shared memory segment, but rather tries to get access to the existing shared memory segment. Compared to the previous program, the absence of IPC_CREAT flag suggests that we do not have to create a new shared memory segment. Instead, we simply have to get the corresponding segment identifier which can be used to attach the existing shared memory segment to some address. The mode 0444 restricts access to the shared memory segment to 'read only'. If no shared memory segment with key 9999 exists, we will get an error, which will be returned in errno. Once we get a valid identifier, we attach the shared memory segment to an address. While attaching, we use the flag SHM_RDONLY which specifies that the shared memory segment will be available only for reading.

Next, we have the function detect_hit, which checks whether the pressed key was an alphanumeric character. The first program obviously has to run first; otherwise, the second program will show errors during the shared memory initialization, since it would be trying to get the identifier for a non-existent shared memory segment.

The example shown here doesn't require any synchronization of access to the shared memory segment. That is because only one program writes into the shared memory and only one program reads from the shared memory area. But again, there is a problem here. What if the detection program (second one) is started long after some user has started hitting the keys (running the first program)? We will not be able to track the previously hit keys. The solution for this is left as an exercise to the readers.

The entry in /proc/sysvipc/shm gives a list of shared mermory in use. Readers can compare the entries before running, during running and after running the programs. Try to interpret the entry in /proc/sysvipc/shm.

Once the two programs identify an alphanumeric character, they will terminate. As part of that process, the shared memory area is detached by using the system call shmdt. In fact, upon exiting the detaching is done automatically. But the shared memory segment is not destroyed. This has to be done by invoking the system call shmctl, which takes the identifier for the shared memory area as an argument, as well as the command IPC_RMID, which marks the shared memory segment as destroyed. This has to be done, otherwise the shared memory segment will persist in memory or in the swap space.

At this point, observation of the entries in /proc/sysvipc/shm can be very useful. If the shared memory segment is not destroyed, the entries will reflect this. Try this by running the program without shmctl.

This is the fastest IPC mechanism in the System V IPC services. However, the System V shared memory mechanism does not have any kind of scheme to ensure that one sees consistent data in the shared memory region. That is, a process can read a shared memory area at the same time another process is writing to it. The programmer can then come across inconsistent data while executing the programs. This suggests that accesses to the shared memory region have to be mutually exclusive; this is achieved via the use of the semaphore mechanism. We can make the semaphore access the memory region to lock it and then release the semaphore when done.

The shared memory mechanism can be used when the processes access the shared memory areas at different times. One may wonder why we can't make the first process store the data in some file and make another process read the data from the file. But, reading data from a file involves things like:

execution of system calls like open, read and close.
accessing the secondary storage device (generally hard disk) which involves I/O operations.

These things are not significant if we have a small amount of data to be read. But when we have large amounts of data stored in a file, then the load of the two activities mentioned above increases significantly and there is a considerable amount of reduction in the performance of the "reading program". This, again, has a solution, which is the use of memory mapping - something I'll discuss at another time.

Conclusion

We have seen the use of the primary IPC mechanisms, and also one of the System V IPC mechanisms. We have seen some simple uses for pipes, fifos, and the shared memory mechanism. But one may come across some very complex programs where these mechanisms will have to be used in a very strict and precise manner. Otherwise, the program along with the programmer, will be dumped to /dev/null. There are still more things to be learned, not only for you but also for me. I shall come up with more in the next part, in which we will explore semaphores, message queues, memory mapping and sockets, and probably try to solve a few practical problems.

[BIO] I completed my B. Tech in Computer Science & Engineering from a small town called Trichur, in Kerala, God's Own Country in India. Presently I am working in Naturesoft Pvt. Ltd, Chennai, India as a Programmer. I spend my free time reading books on Linux and exploring the same. Also, I have a good appetite for Physics. My motive in life is to go forward, onward and upward.

Copyright © 2004, Hiran Ramankutty. Released under the Open Publication license unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 104 of Linux Gazette, July 2004

<-- prev | next -->