Pneumonia detection with CNN

Problem

Pneumonia is one of the leading causes of pediatric morbidity and mortality worldwide, which is why early and accurate detection of pneumonia remains critical in clinical settings, particularly for children. Envision an automated system that quickly diagnoses pneumonia from chest X-rays, thereby reducing patient wait times and easing the workload of doctors.

Can we develop a tool that achieves this while minimizing human error?

How can we ensure that this model is both robust and reliable despite challenges like image variability?

Methodology

To attain the answers to the questions above, the following steps were involved:

• Data Collection

• Data Preprocessing

• Model Building

• Model Training and Evaluation

Data Collection

The dataset was sourced from Kaggle, originally contributed by the Guangzhou Women and Children's Medical Center in China. It comprised 5,863 pediatric chest X-ray images, divided into training, validation, and test sets. These were further categorized into “Normal” and “Pneumonia” classes.

To ensure accuracy, two expert physicians reviewed each image, and a third expert validated the labels to eliminate misclassification. Low-quality and unreadable scans were also discarded during preprocessing.

The x-ray images of patients with pneumonia were blurry, while the normal x-ray images were clear. This is expected, as pneumonia casts an unusual white or hazy shadow on the normally dark lungs.

During exploratory analysis, I discovered a class imbalance—the number of pneumonia images significantly outnumbered normal ones, increasing the risk of overfitting. This imbalance was visualized in the training data, highlighting the need for data augmentation.

Data Preprocessing

Before feeding the images to the model, I transformed their raw pixel values from the 0–255 scale into a smooth 0–1 range by dividing by 255. This simple yet powerful step not only standardizes the data but also supercharges the CNN's learning process, leading to faster convergence during training.

#Normalizing the X data
x_train = np.array(x_train) / 255
x_val = np.array(x_val) / 255
x_test = np.array(x_test) / 255

Data augmentation was used to artificially expand the training dataset by applying random modifications to the images. Given the class imbalance in the dataset, where pneumonia images outnumber normal ones, augmentation played a critical role in preventing overfitting.

The data augmentation processes involved:

• Randomly flipping images vertically

• Randomly rotating some training images by 50 degrees

• Randomly zooming some training images by 30%

• Randomly shifting images horizontally by 15% of the width

• Randomly shifting images vertically by 20% of the height

Randomly flipping images horizontally

image_gen = ImageDataGenerator(
        rotation_range = 50,  # randomly rotate images in the range provided
        zoom_range = 0.3, # Randomly zoom image
        width_shift_range=0.15,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)
        horizontal_flip = True,  # randomly flip images
        vertical_flip=True)  # randomly flip images

image_gen.fit(x_train)

Model Building

With the data appropriately prepared, the next step involves designing and building the Convolutional Neural Network (CNN) model. The architecture of the network is designed using Keras' Sequential API, which allows for a straightforward stacking of layers.

model = Sequential() #API
model.add(Conv2D(32 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu' , input_shape = (150,150,1)))
model.add(BatchNormalization())
model.add(MaxPool2D((2,2) , strides = 2 , padding = 'same'))
model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.1))
model.add(BatchNormalization())
model.add(MaxPool2D((2,2) , strides = 2 , padding = 'same'))
model.add(Conv2D(64 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(BatchNormalization())
model.add(MaxPool2D((2,2) , strides = 2 , padding = 'same'))
model.add(Conv2D(128 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(MaxPool2D((2,2) , strides = 2 , padding = 'same'))
model.add(Conv2D(256 , (3,3) , strides = 1 , padding = 'same' , activation = 'relu'))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(MaxPool2D((2,2) , strides = 2 , padding = 'same'))
model.add(Flatten())
model.add(Dense(units = 128 , activation = 'relu'))
model.add(Dropout(0.2))
model.add(Dense(units = 1 , activation = 'sigmoid'))
model.compile(optimizer = "rmsprop" , loss = 'binary_crossentropy' , metrics = ['accuracy'])
model.summary()

In this model, the convolutional layers act as powerful feature extractors that scan each input image to capture essential details. As I deepen the network by adding more filters (32, 64, 128, 256), it becomes increasingly adept at identifying complex patterns that indicate the presence of pneumonia. The max pooling layers then condense these rich feature maps, which speeds up the training process and ensures the model remains robust to small image shifts. After pooling, the flatten layer converts these compact maps into a one-dimensional vector, making it easier for the subsequent dense layers to analyze the data. To prevent overfitting, I implemented a Dropout layer that randomly deactivates 50% of neurons during training, enhancing the model's ability to generalize. Finally, the sigmoid activation function in the output layer transforms the processed data into a clear probability, effectively signaling whether an X-ray image shows signs of pneumonia.

Model Training and Evaluation

After successfully building the model, the next step was to train it using the preprocessed data.

During training, the model iteratively learns the optimal weights by comparing its predictions against the true labels. The training process is monitored by evaluating the accuracy of both the training and validation sets over several epochs

Here are the results:

The significant difference in the validation and training accuracy scores at the 6th epoch shows signs of overfitting. We can also see that the model performed best at the 9th epoch. This is because, at the 9th epoch, the validation and training accuracy scores are almost the same, which is a sign of good performance in the model.

The predicted values from the model were converted to 0s and 1s and then compared to the target values, resulting in an accuracy score of 83%. Here’s a breakdown of how the model performed in each class:

The confusion matrix above reveals that the model successfully identified 387 X-ray scans showing pneumonia, with just 3 cases misclassified. On the other side, it correctly recognized 131 normal X-ray scans, though it mistakenly flagged 103 normal scans as pneumonia. The accompanying classification report offers further insight into the model's performance, with a precision of 0.79, a remarkable recall of 0.99, and an overall F1 score of 0.87, underscoring the model's robust ability to detect pneumonia while balancing false positives.

Conclusion

This project demonstrates that it is indeed possible to develop an automated tool that not only achieves rapid and accurate pneumonia detection from chest X-ray images but also significantly minimizes human error. Through careful preprocessing, intelligent data augmentation, and the design of a robust CNN architecture, the model was able to learn complex patterns and generalize well despite inherent challenges like image variability. By incorporating techniques such as normalization to standardize data and dropout to prevent overfitting, I ensured that the model remained reliable and resilient in the face of diverse imaging conditions.

Thank you for reading.

I hope you learnt something.