Split a string by a delimiter

Tokenizing a string in C typically involves breaking a string into smaller pieces or tokens based on a specified delimiter. The standard library provides a function called strtok that can be used for this purpose. Here's a simple example with comments explaining each step:

#include <stdio.h>
#include <string.h>

int main() {
    // Input string to tokenize
    char inputString[] = "Hello,World,Tokenization,Example";

    // Delimiter used for tokenization (comma in this case)
    const char delimiter[] = ",";

    // Token pointer to store the current token during iteration
    char *token;

    // Use strtok to get the first token
    token = strtok(inputString, delimiter);

    // Continue getting tokens until strtok returns NULL
    while (token != NULL) {
        // Print the current token
        printf("Token: %s\n", token);

        // Use strtok with NULL to continue tokenization
        token = strtok(NULL, delimiter);
    }

    return 0;
}

In the provided C code, a string is tokenized using the strtok function, which breaks the input string, "Hello,World,Tokenization,Example," into smaller pieces based on the specified delimiter (,). The program iterates through the resulting tokens, printing each one on a new line. The tokenization process begins by calling strtok with the input string and delimiter, and subsequent calls use NULL as the first argument to continue tokenization. The token pointer is updated during each iteration, and the loop continues until there are no more tokens. This example demonstrates a basic tokenization scenario, but it's important to note that strtok modifies the original string, and complex cases may require additional handling and error checks.

Subscribe to my newsletter

Read articles from Jyotiprakash Mishra directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Jyotiprakash Mishra

I am Jyotiprakash, a deeply driven computer systems engineer, software developer, teacher, and philosopher. With a decade of professional experience, I have contributed to various cutting-edge software products in network security, mobile apps, and healthcare software at renowned companies like Oracle, Yahoo, and Epic. My academic journey has taken me to prestigious institutions such as the University of Wisconsin-Madison and BITS Pilani in India, where I consistently ranked among the top of my class. At my core, I am a computer enthusiast with a profound interest in understanding the intricacies of computer programming. My skills are not limited to application programming in Java; I have also delved deeply into computer hardware, learning about various architectures, low-level assembly programming, Linux kernel implementation, and writing device drivers. The contributions of Linus Torvalds, Ken Thompson, and Dennis Ritchie—who revolutionized the computer industry—inspire me. I believe that real contributions to computer science are made by mastering all levels of abstraction and understanding systems inside out. In addition to my professional pursuits, I am passionate about teaching and sharing knowledge. I have spent two years as a teaching assistant at UW Madison, where I taught complex concepts in operating systems, computer graphics, and data structures to both graduate and undergraduate students. Currently, I am an assistant professor at KIIT, Bhubaneswar, where I continue to teach computer science to undergraduate and graduate students. I am also working on writing a few free books on systems programming, as I believe in freely sharing knowledge to empower others.