"Unmasking Outliers: Detecting and Removing Anomalies in Your Data"

Meemansha PriyadarshiniMeemansha Priyadarshini
3 min read

Table of contents

Outliers are the data points which differ significantly from the complete dataset. They may occur due to error in data entry , variability in the dataset and measurement errors. Availability of outliers in the dataset can cause various problems like misinterpretation of the accurate data analysis , create misleading relationship between variables which can decrease the modelling performance and generalization.

There are various methods to find out outliers in your dataset.

  1. By visualization of the data

    You can use boxplots and scatterplots to find out the outliers which are far away from the cluster of data points

    A box plot displaying data distribution. The box represents the interquartile range, with the median marked inside. There are several outliers above the main data distribution.

  2. By Z-square and IQR method( Inter Quartile range)

    These two methods are statistical methods of finding out the outliers.

    In Z-square method : A data point with a Z-score greater than 3 or less than -3 is often considered an outlier.

    z-score = (data - mean) / standard deviation

As you can see the x-axis describes the z-score , if the value is greater than 3 or less than -3, we consider it as outlier

Remove outliers from the dataset using z-score.

A Python script showing a function for removing outliers from a DataFrame using the Z-score method. The script imports pandas and numpy libraries, calculates z-scores, and filters the DataFrame based on a threshold. The original DataFrame and the cleaned DataFrame are printed.

In IQR method: Points outside Q1−1.5×IQR or Q3+1.5×IQR are considered outliers.

FormuIa: QR = Q3 - Q1

  • To find the IQR, we first need to find the median (middle value) of the lower and upper half of the data. This is the second quartile (Q2).

  • The first quartile (Q1) is the value below which 25% of the data falls.

  • The third quartile (Q3) is the value above which 75% of the data falls.

A box plot diagram illustrating the Interquartile Range (IQR). Key elements include the box that spans from the first quartile (Q1) to the third quartile (Q3), with the median inside. Whiskers extend from the box to the "minimum" (Q1 - 1.5IQR) and "maximum" (Q3 + 1.5IQR). Outliers are indicated beyond the whiskers.

Remove outliers from the dataset using IQR.

A screenshot of a code snippet in Python. The code defines a function, , which removes outliers from a DataFrame using the Interquartile Range (IQR) method. The function calculates the first (Q1) and third (Q3) quartiles, computes the IQR, and removes rows that have values beyond 1.5 times the IQR from Q1 and Q3. It prints the original DataFrame, removes outliers, and prints the cleaned DataFrame.

Conclusion

Detecting and removing outliers is a crucial step in data preprocessing to ensure the accuracy and reliability of your analysis. Outliers can distort statistical analyses and lead to misleading conclusions. By employing visualization techniques like boxplots and scatterplots, and statistical methods such as the Z-score and Interquartile Range (IQR), you can effectively identify and handle these anomalies. Removing outliers helps in improving the performance and generalization of your models, leading to more robust and trustworthy results. Always remember to carefully consider the context and potential reasons for outliers before deciding to remove them, as they might carry significant information about your data.

0
Subscribe to my newsletter

Read articles from Meemansha Priyadarshini directly inside your inbox. Subscribe to the newsletter, and don't miss out.

outliersstatisticsMachine Learningdata analysis

Written by

Meemansha Priyadarshini
Meemansha Priyadarshini

I am a certified TensorFlow Developer and enjoy writing blogs to share my knowledge and assist others.