Converting MNIST Data to CSV or Individual Files using .NET

In the .NET ecosystem, handling the conversion of the MNIST dataset from its original binary format directly within a C# application is not as straightforward as in Python due to the lack of ready-to-use libraries for this task. However, you'll see that it is not as bad as one might think.

Reading MNIST Binary Files in C

To read the MNIST binary files in C#, you need to manually parse the binary data.

1. MNIST File Format: MNIST binary files for images have a specific format:

   • Magic number (4 bytes)

   • Number of images (4 bytes)

   • Rows (4 bytes)

   • Columns (4 bytes)

   • Pixel data (1 byte per pixel)

2. Read and Process the Files: We will be using the BinaryReader class in .NET to read these binary files.

   Example C# Code to Convert MNIST to CSV

   Here is the code for reading the MNIST image files and converting them to a CSV file in C#:

    static void ConvertToCsv(string imagePath, string labelPath, string outputCsvPath)
        {
            using (var imageStream = new FileStream(imagePath, FileMode.Open, FileAccess.Read))
            using (var labelStream = new FileStream(labelPath, FileMode.Open, FileAccess.Read))
            using (var imageReader = new BinaryReader(imageStream))
            using (var labelReader = new BinaryReader(labelStream))
            using (var csvWriter = new StreamWriter(outputCsvPath, false, Encoding.UTF8))
            {
                // Read and validate magic numbers and number of items
                var imageMagicNumber = ReadInt32BigEndian(imageReader);
                var numberOfImages = ReadInt32BigEndian(imageReader);
                var rows = ReadInt32BigEndian(imageReader);
                var cols = ReadInt32BigEndian(imageReader);
   
                var labelMagicNumber = ReadInt32BigEndian(labelReader);
                var numberOfLabels = ReadInt32BigEndian(labelReader);
   
                if (imageMagicNumber != 2051 || labelMagicNumber != 2049)
                    throw new Exception("Invalid MNIST file format.");
   
                if (numberOfImages != numberOfLabels)
                    throw new Exception("The number of images does not match the number of labels.");
   
                for (var i = 0; i < numberOfImages; i++)
                {
                    var label = labelReader.ReadByte();
                    var pixels = imageReader.ReadBytes(rows * cols);
   
                    var csvLine = new StringBuilder(label.ToString());
                    foreach (var pixel in pixels)
                    {
                        csvLine.Append($",{pixel}");
                    }
                    csvWriter.WriteLine(csvLine.ToString());
   
                    if (i % 1000 == 0)
                        Console.WriteLine($"Processed {i} images to CSV.");
                }
            }
            Console.WriteLine("CSV conversion complete.");
        }
   

   The code above writes out the CSV file in chunks and optionally flushes the data to the file every 1,000 images. This helps memory usage, especially when processing large datasets.

   If you want to convert to individual images instead, here is the way to do it:

    static void ConvertToImages(string imagePath, string labelPath, string outputImageDir)
        {
            using (var imageStream = new FileStream(imagePath, FileMode.Open, FileAccess.Read))
            using (var labelStream = new FileStream(labelPath, FileMode.Open, FileAccess.Read))
            using (var imageReader = new BinaryReader(imageStream))
            using (var labelReader = new BinaryReader(labelStream))
            {
                // Read and validate magic numbers and number of items
                var imageMagicNumber = ReadInt32BigEndian(imageReader);
                var numberOfImages = ReadInt32BigEndian(imageReader);
                var rows = ReadInt32BigEndian(imageReader);
                var cols = ReadInt32BigEndian(imageReader);
   
                var labelMagicNumber = ReadInt32BigEndian(labelReader);
                var numberOfLabels = ReadInt32BigEndian(labelReader);
   
                if (imageMagicNumber != 2051 || labelMagicNumber != 2049)
                    throw new Exception("Invalid MNIST file format.");
   
                if (numberOfImages != numberOfLabels)
                    throw new Exception("The number of images does not match the number of labels.");
   
                Directory.CreateDirectory(outputImageDir);
   
                for (var i = 0; i < numberOfImages; i++)
                {
                    var label = labelReader.ReadByte();
                    var pixels = imageReader.ReadBytes(rows * cols);
   
                    using (var img = new Bitmap(cols, rows))
                    {
                        for (var r = 0; r < rows; r++)
                        {
                            for (var c = 0; c < cols; c++)
                            {
                                var pixelIndex = r * cols + c;
                                var colorValue = pixels[pixelIndex];
                                img.SetPixel(c, r, Color.FromArgb(colorValue, colorValue, colorValue));
                            }
                        }
                        img.Save(Path.Combine(outputImageDir, $"label_{label}_image_{i}.png"));
                    }
   
                    if (i % 1000 == 0)
                        Console.WriteLine($"Processed {i} images to PNG.");
                }
            }
            Console.WriteLine("Image conversion complete.");
        }
   

   Both of the methods above make use of a method called ReadInt32BigEndian which handles the conversion from big-endian (as stored in MNIST files) to little-endian, which is the default byte order used by .NET. This is done by reversing the bytes read by BinaryReader, which reads data in little-endian format by default.

     static int ReadInt32BigEndian(BinaryReader reader)
        {
            var data = reader.ReadBytes(4);
            Array.Reverse(data);
            return BitConverter.ToInt32(data, 0);
        }
   

   The .NET Console Project can be found here: https://github.com/tjgokcen/MNISTConverter-NET