Fork, pipes, sockets, and inter-process communication
Mohamed Moataz El Zein
Fork()
So, what is fork()? I briefly introduced it in the previous article, but let's get into more details now. So fork() creates an entirely new process by literally copying said process and creating a new one. The child process is essentially an exact copy of the parent process (with some exceptions). The entire virtual address space of the parent (the process that called fork()) is replicated to the child, including threads, file descriptors, and more. As I mentioned in the last article, they both share the same file descriptors, and those file descriptors therefore point to the same open file description as the parent (they point to the same file struct in the kernel).
On success, fork() returns the PID. If we are currently the child process, it returns 0, else, it returns the parent's PID. Let's check out a quick example.
#include <stdio.h>
#include <unistd.h>
// gcc simple_fork.c -o simple_fork && ./simple_fork
int main()
{
char *some_random_variable = "hello";
pid_t p = fork();
if (p < 0)
{
perror("fork failed");
return -1;
}
if (p == 0) //this is the newly created child
{
some_random_variable = "child!";
printf("I am a completely new child process: %s\n", some_random_variable);
}
else //this is the parent
{
some_random_variable = "parent!";
printf("I am the original parent: %s\n", some_random_variable);
}
}
//Output:
//I am the original parent: parent!
//I am a completely new child process: child!
So, depending on the returned value of fork(), you can know whether you are the child or the parent. When dealing with threads and reading/writing to a variable like some_random_variable above, you will want to use synchronization primitives or locks to avoid unexpected behaviour (corruption, data races, etc..). But, since this is fork(), this is absolutely not the case, since, as you may recall, fork() creates a new process, & each process has its own address space, and so those variables are not shared, and there are no race conditions. They are in completely different physical memory locations.
Threads in fork()
I mentioned above that when using fork(), the parent process is almost perfectly replicated. But what if there were any running threads at the time of calling fork()? In that case, while the entire memory space is indeed duplicated, one would expect that the same threads that have been (& possibly still are) running on the parent would still be running on the child process, but no! Only the thread that called fork() would still be running on the child process. Here is what the POSIX specification says:
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.
Now let's show a quick example to drive this point home.
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>
// gcc threaded_fork.c -o threaded_fork && ./threaded_fork
void *dummy_thread()
{
sleep(2);
printf("This is some dumb thread\n");
return NULL;
}
int main()
{
pthread_t th[10];
int i;
for (i = 0; i < 2; i++) //create 2 threads at the parent process
{
pthread_create(&th[i], NULL, &dummy_thread, NULL);
}
pid_t p = fork();
if (p < 0)
{
perror("fork failed");
return -1;
}
if (p == 0)
{
printf("I am a completely new child process\n");
// wait for all the threads to finish
// (hint: this will fail, since those threads aren't
// actually running on the new child process)
for (i = 0; i < 2; i++)
{
if (pthread_join(th[i], NULL) != 0)
{
perror("Child: Failed to join thread");
}
}
}
else
{
printf("I am the original parent\n");
// wait for all the threads to finish
// those will succeed normally
for (i = 0; i < 2; i++)
{
if (pthread_join(th[i], NULL) != 0)
{
perror("Parent: Failed to join thread");
}
}
}
}
//Output:
//I am the original parent
//I am a completely new child process
//Child: Failed to join thread: Invalid argument
//Child: Failed to join thread: Invalid argument
//This is some dumb thread
//This is some dumb thread
As we can see above, we call pthread_create() to create 2 threads in the parent process, then we try to join (wait) for the 2 threads to complete in both the child and the parent. And as seen in the above output, the child process fails to join the threads since there actually aren't any running threads in the new child process (except the main thread), while the parent process joins the running threads successfully.
A need for communication (IPC)
Now that we've gotten fork() out of the way and can create multiple processes from within our process, the need may arise for those processes to communicate! So, how do we do that? A simple option would be to communicate via files. Since we know how to read() and write() to a file, why don't we do just that? Especially given that both parent and child share the same file descriptors, and thus the same open file description entry! This is valid, so let's try it.
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
// gcc ipc_using_files.c -o ipc_using_files && ./ipc_using_files
int main()
{
int file_descriptor = open("ipc_comm.txt", O_RDWR | O_CREAT | O_TRUNC, 0644);
pid_t p = fork();
if (p < 0)
{
perror("fork failed");
return -1;
}
if (p == 0)
{
// Child
char buffer[100];
lseek(file_descriptor, 0, SEEK_SET);
// The above line of code is extremely important
// If you can recall, we mentioned that both parent and child share
// the same file descriptors, and therefore, the same open file description.
// This means that assuming the parent has already written to the file,
// the field loff_t f_pos in the struct *file of the open file description
// has been incremented, and thus, we have to set this f_pos back to 0
// to be able to read from the start of the file!
// If the above line is removed, then read won't be able to read anything, since
// it will start reading from the position starting after the last
int len = read(file_descriptor, buffer, 100);
buffer[len] = '\0';
printf("This is the message received from the parent: %s\n", buffer);
}
else
{
// Parent
char *msg = "Message for the child from the parent\n";
write(file_descriptor, msg, strlen(msg));
}
}
//Output: This is the message received from the parent: Message for the child from the parent
The above code has quite a few issues. Particularly, it assumes that the parent will run first, write to the file, and then the child process will read whatever the parent has written. This ordering is of course not guaranteed, so we can just introduce a sleep in the child. There are quite a few other issues with the above code, but we won't delve into them. For now, this serves as proof that we can have IPC using files. The issue is that this isn't the most ideal way to communicate. It isn't the most secure, and it is quite expensive because we want non-persistent communication, so there is no need to hit the disk (Some would point out that we will likely only hit the buffer cache, and you’d be correct!). A perfect scenario would be something like an in-memory queue between the processes. Enter pipes.
Pipe()
A pipe is just that, an in-memory queue, where a process can read/write to a queue, and another one can do the same. It is unidirectional. If a process tries to write to a full queue, it should be put to sleep, and if a process attempts to read from an empty queue, it should be put to sleep as well, and wake up when data is ready to be consumed. This is all handled by the kernel, and so is secure and is not the programmer's responsibility, yay!
The best thing about pipes is that they use the exact same interface as files, so you get a file descriptor for a pipe, and you can write() to it and read() from it as well. Pipe() expects a 2-element array to fill. It fills the first element with the file descriptor to read from, and the second element with the file descriptor to write to. Let's check a simple pipe usage
#include <unistd.h>
#include <stdio.h>
#include <string.h>
// gcc single_process_pipe.c -o single_process_pipe && ./single_process_pipe
int main()
{
char *msg = "Message in a pipe.\n";
char buf[BUFSIZ];
int pipe_fd[2]; // array of 2 elements,
// that contains the 2 pipe file descriptors
if (pipe(pipe_fd) == -1)
{
perror("Pipe failed: ");
return -1;
}
ssize_t writelen = write(pipe_fd[1], msg, strlen(msg) + 1);
printf("Sent: %s [%ld, %ld]\n", msg, strlen(msg) + 1, writelen);
ssize_t readlen = read(pipe_fd[0], buf, BUFSIZ);
printf("Rcvd: %s [%ld]\n", buf, readlen);
close(pipe_fd[0]);
close(pipe_fd[1]);
}
So, we write to fd[1] and read what we wrote from fd[0]. Pretty neat, but not super useful. Let's kick it up a notch and use a pipe between 2 processes using fork()!.
#include <unistd.h>
#include <stdio.h>
#include <string.h>
// gcc multi_process_pipe.c -o multi_process_pipe &&./multi_process_pipe
int main()
{
char *msg = "Message in a pipe.\n";
char buf[BUFSIZ];
int pipe_fd[2];
if (pipe(pipe_fd) == -1)
{
perror("Pipe failed: ");
return -1;
}
pid_t p = fork();
if (p < 0)
{
perror("Fork failed: ");
return -1;
}
if (p == 0)
{
// child
// read from parent from pipe_fd[0]
close(pipe_fd[1]); // un-needed pipe that parent is writing to
ssize_t readlen = read(pipe_fd[0], buf, BUFSIZ);
printf("Child rcvd: %s [%ld]\n", buf, readlen);
}
else
{
// parent
// write to child from pipe_fd[1]
close(pipe_fd[0]); // un-need pipe that child is reading from
ssize_t writelen = write(pipe_fd[1], msg, strlen(msg) + 1);
printf("Parent sent: %s [%ld, %ld]\n", msg, strlen(msg) + 1, writelen);
}
}
Communication between processes on different hosts using sockets
So now, how do we deal with communication between different machines? Pipes work fine when a parent forks a child, but in this case, we would have to utilize the network, and here come sockets. Sockets don't have to be between different machines, they can be used for processes on the same machine as well. So, what is a socket? A socket is essentially a data structure that includes 2 queues. It is accessed through a file descriptor as well, and we can therefore write() to it & read() from it.
/**
* struct socket - general BSD socket
* @state: socket state (%SS_CONNECTED, etc)
* @type: socket type (%SOCK_STREAM, etc)
* @flags: socket flags (%SOCK_NOSPACE, etc)
* @ops: protocol specific socket operations
* @file: File back pointer for gc
* @sk: internal networking protocol agnostic socket representation
* @wq: wait queue for several uses
*/
struct socket {
socket_state state;
kmemcheck_bitfield_begin(type);
short type;
kmemcheck_bitfield_end(type);
unsigned long flags;
struct socket_wq __rcu *wq;
struct file *file;
struct sock *sk;
const struct proto_ops *ops;
};
Above is the socket struct in linux, and this field struct sock *sk is where all the meat and potatoes are. It is unfortunately, huge, so I won't be including it in this article, but here is the link for the above socket struct, and the sock struct as well to check them out if you'd like. Let’s start by looking at network sockets. Each socket is uniquely identified as a 5-part tuple (source IP address, source port, destination IP address, destination port, transport protocol). I won't get into too much detail concerning sockets, since this is a huge topic on its own, so let's just look into a simple example of client and server processes using sockets to communicate over the network.
A simple echo server
#define MAX_IN 100
#define MAX_OUT 100
#define MAX_QUEUE 100
// setup the addrinfo struct used by the socket
struct addrinfo *setup_address(char *host, char *port)
{
struct addrinfo *server;
struct addrinfo hints;
memset(&hints, 0, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
int rv = getaddrinfo(host, port, &hints, &server);
if (rv != 0)
{
printf("getaddrinfo failed: %s\n", gai_strerror(rv));
return NULL;
}
return server;
}
// gcc echo_server.c -o echo_server &&./echo_server
int main()
{
char send_buffer[MAX_OUT];
char rcv_buffer[MAX_IN];
int n;
char *host = "localhost";
char *port = "8080";
struct addrinfo *server = setup_address(host, port);
// create the socket, and hold on to its file descriptor
int server_sock = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
printf("Server socket created: %d\n", server_sock);
// bind the fd socket to the address we set up
n = bind(server_sock, server->ai_addr, server->ai_addrlen);
if (n != 0)
{
perror("Bind failed: ");
return -1;
}
// listen on this socket
n = listen(server_sock, MAX_QUEUE);
if (n != 0)
{
perror("Listen failed: ");
return -1;
}
printf("Server started listening on %s:%s\n", host, port);
char *msg = "Server says Hello!";
while (1)
{
// Accept a new client connection, obtaining a new socket fd to communicate through
int conn_socket = accept(server_sock, NULL, NULL);
printf("New connection received %d\n", conn_socket);
while (1)
{
// keep reading and writing to this socket
memset(rcv_buffer, 0, sizeof(rcv_buffer)); // clear the rcv buffer
n = read(conn_socket, rcv_buffer, MAX_IN);
if (n <= 0)
{
// read failed, socket likely closed
// continue and try to accept a new connection
close(conn_socket);
break;
}
printf("Server received from client: %s\n", rcv_buffer);
n = write(conn_socket, msg, strlen(msg) + 1); // send data to the socket
sleep(1);
}
}
}
A simple echo client
#define MAX_IN 100
#define MAX_OUT 100
// get the addrinfo struct for the required host
struct addrinfo *lookup_host(char *host, char *port)
{
struct addrinfo *server;
struct addrinfo hints;
memset(&hints, 0, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
int rv = getaddrinfo(host, port, &hints, &server);
if (rv != 0)
{
printf("getaddrinfo failed: %s\n", gai_strerror(rv));
return NULL;
}
return server;
}
// gcc echo_client.c -o echo_client &&./echo_client
int main()
{
char send_buffer[MAX_OUT];
char rcv_buffer[MAX_IN];
int n;
struct addrinfo *server = lookup_host("localhost", "8080");
// create the socket that will be used to conntect to the server
int sockfd = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
// use the socket to connect
n = connect(sockfd, server->ai_addr, server->ai_addrlen);
if (n != 0)
{
perror("Connect failed: ");
return -1;
}
char *msg = "Client says Hello!";
while (1)
{
write(sockfd, msg, strlen(msg) + 1); // send data to the socket
memset(rcv_buffer, 0, sizeof(rcv_buffer)); // clear the rcv buffer
n = read(sockfd, rcv_buffer, MAX_IN);
if (n <= 0)
{
perror("Read failed: ");
return -1;
}
printf("Client received from server: %s\n", rcv_buffer);
}
close(sockfd);
}
The above code examples show a simple server/client setup, where a server creates a socket, binds it to a certain host & port, and then accepts a connection to it. After accepting a new connection, it then spawns a new socket through which it can communicate with the client. There are 2 peculiarities in the above code. Let’s dive into them.
The server can’t handle more than 1 connection at a time. When it accepts a new connection, it just keeps on reading and writing to it and doesn’t accept any new connections until this specific one has been terminated by the client. How do we fix that? Easy enough, we fork() every time we receive a new connection, and let the new child process handle it! We’ll see a code example below
The second peculiarity is a more sinister one, and it’s why would accept() spawn a new socket to communicate through? Can’t we just use the original server_sock and write to it? The answer to that question is that sockets can have multiple states (socket_state in the socket struct above). Let’s discuss two of them.
LISTENING state: This is the state that a socket is in when listening for a connection. You get a listening socket when you create a socket, bind an address to it, and listen on it. Those sockets aren’t meant for communication, and thus aren’t specifically used for writing.
ESTABLISHED state: This is the state of a socket that returns from the accept() syscall. This type of socket can indeed be used for communication.
So, how do we get the ESTABLISHED socket from the accept() syscall? It’s ridiculously simple, really. When a connection comes in for a listening socket, idenitifed by the 5-part tuple (source IP address, source port, destination IP address, destination port, transport protocol), accept() creates a second clone socket that inherits everything from the main socket, except for the file descriptor (a new one is returned) and the state (it’s now ESTABLISHED). There are quite a few other states that I won’t get into, but that’s the gist of it. Now, onto showing you an example of how to use fork() with sockets to handle multiple connections in parallel.
#define MAX_IN 100
#define MAX_OUT 100
#define MAX_QUEUE 100
// setup the addrinfo struct used by the socket
struct addrinfo *setup_address(char *host, char *port)
{
struct addrinfo *server;
struct addrinfo hints;
memset(&hints, 0, sizeof(hints));
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
int rv = getaddrinfo(host, port, &hints, &server);
if (rv != 0)
{
printf("getaddrinfo failed: %s\n", gai_strerror(rv));
return NULL;
}
return server;
}
// gcc echo_server.c -o echo_server &&./echo_server
int main()
{
char send_buffer[MAX_OUT];
char rcv_buffer[MAX_IN];
int n;
char *host = "localhost";
char *port = "8080";
struct addrinfo *server = setup_address(host, port);
// create the socket, and hold on its file descriptor
int server_sock = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
printf("Server socket created: %d\n", server_sock);
// bind the fd socket to the address we set up
n = bind(server_sock, server->ai_addr, server->ai_addrlen);
if (n != 0)
{
perror("Bind failed: ");
return -1;
}
// listen on this socket
n = listen(server_sock, MAX_QUEUE);
if (n != 0)
{
perror("Listen failed: ");
return -1;
}
printf("Server started listening on %s:%s\n", host, port);
char *msg = "Server says Hello!";
while (1)
{
// Accept a new client connection, obtaining a new socket fd to communicate through
int conn_socket = accept(server_sock, NULL, NULL);
printf("New connection received %d\n", conn_socket);
pid_t p = fork();
if (p == 0)
{
// we're the child process, so let's start using the conn_socket
close(server_sock); // we're the child, we don't need the server_sock
while (1)
{
// keep reading and writing to this socket
memset(rcv_buffer, 0, sizeof(rcv_buffer)); // clear the rcv buffer
n = read(conn_socket, rcv_buffer, MAX_IN);
if (n <= 0)
{
// read failed, socket likely closed
// continue and try to accept a new connection
close(conn_socket);
break;
}
printf("Server received from client: %s\n", rcv_buffer);
n = write(conn_socket, msg, strlen(msg) + 1); // send data to the socket
sleep(1);
}
close(conn_socket); // done with the conn
exit(0); // now the child process can die
}
else
{
// else, we're the parent, so we do nothing so we can accept
// yet more connections!
close(conn_socket); // parent can close the conn_socket, because it's not using it
}
}
}
As you can see with the code above, we fork() for every connection, i.e, we create a new process. Now for now, this works, but in practice, a better solution would just be to use threads to avoid the many performance penalties & allocations of extra processes, but that’s a topic for another day.
References:
Berkley CS162 Lecture 5 https://www.youtube.com/watch?v=MKcSCaP0ZrQ
Linux Man Pages
http://arthurchiao.art/blog/tcp-listen-a-tale-of-two-queues/
And a lot of other stuff :)
Subscribe to my newsletter
Read articles from Mohamed Moataz El Zein directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by