Week 7 : 1. Logistic Regression

Despite its name, Logistic regression is actually a Classification algorithm, commonly used for binary classification. Usually, true or false , 0 or 1.

Logistic regression works by estimating the input belongs to one class, it takes input applies weights and calculates a probability score between 0 and 1. When the probability scores crosses a threshold (often 0.5) it classifies the observation into one category, otherwise into other category.

Working :

In logistic regression, we try to predict the chance that something belongs to a certain group (like whether someone is diabetic or not). Unlike linear regression, which gives any number as an answer, logistic regression gives a probability between 0 and 1.

To do this, it uses a special curve called the sigmoid function. This curve looks like an “S” shape — as the input gets very large, the probability gets close to 1, and as the input gets very small, the probability gets close to 0.

The model uses a cutoff point (usually 0.5) to decide the final prediction. If the predicted probability is less than 0.5, we say the answer is false (e.g., not diabetic). If it’s 0.5 or more, we say the answer is true (e.g., diabetic).

Once we have trained the logistic regression model and obtained the optimal parameters θ, we can use the model to make predictions on new data.

• We compute the linear combination y of the features and model parameters.

• We then pass y through the sigmoid function to obtain the predicted probability.

• If y is less than 0.5, we predict the sample belongs to the negative class (0); otherwise, we predict it belongs to the positive class (1).

Cost function :

To train logistic regression model, we need a cost function to calculate the value of error produced by the model. Unlike Linear regression which uses Mean Squared Error (MSE), logistic regression uses log-loss function (Binary cross entropy) cost function.

log-loss function :

it can be written as :

Performance metric :

In machine learning its pretty important to know how well the model is working. To calculate the performance of a model we use different techniques. For classification algorithms we use confusion metrix. From the confusion matrix, we can calculate important scores like accuracy, precision, recall, specificity, and F1-score to better understand the model’s strengths and weaknesses.

Confusion Matrix :

A confusion matrix is a tabular representation of the performance of a classification model that categorizes predictions into four categories:

• True Positives (TP): Instances where the model correctly predicts the positive class.

• True Negatives (TN): Instances where the model correctly predicts the negative class.

• False Positives (FP): Instances where the model incorrectly predicts the positive class (Type I error).

• False Negatives (FN): Instances where the model incorrectly predicts the negative class (Type II error).

With the help of these four values, we can calculate True Positive Rate (TPR), False Negative Rate (FPR), True Negative Rate (TNR), and False Negative Rate (FNR).

Even if data is imbalanced, we can figure out that our model is working well or not. For that, the values of TPR and TNR should be high, and FPR and FNR should be as low as possible.

Accuracy : Accuracy measures the overall correctness of the model's predictions and is calculated as the ratio of correct predictions (TP + TN) to the total number of predictions (TP + TN + FP + FN).

Specificity : Specificity measures the ability of the model to correctly identify negative instances out of all actual negative instances. It is calculated as the ratio of true negatives (TN) to the total number of actual negatives (TN + FP).

Precision : Precision quantifies the ability of the model to correctly identify positive instances out of all instances predicted as positive. It is calculated as the ratio of true positives (TP) to the total number of predicted positives (TP + FP).

Recall : Recall measures the ability of the model to correctly identify positive instances out of all actual positive instances. It is calculated as the ratio of true positives (TP) to the total number of actual positives (TP + FN).

F1- score : The F1-score is the harmonic mean of precision and recall and provides a balanced measure of a model's performance. It is calculated as :