Classification Algorithms | Fakhruddin Khambaty's Learning Hub

Part 1: Logistic Regression

Despite its name, Logistic Regression is a classification algorithm, not regression! It predicts the probability of belonging to a class.

📈 The S-Curve Magic

Logistic Regression uses the "sigmoid" function to squeeze any number into a probability between 0 and 1.

Think of it as a volume knob that only goes from 0% to 100% - it can't go beyond!

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create and train model
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X_train, y_train)

# Predict
y_pred = log_reg.predict(X_test)

# Get probabilities
y_proba = log_reg.predict_proba(X_test)
print("Probability of class 0 and 1:", y_proba[0])

# Evaluate
print("Accuracy:", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))

💡 When to Use Logistic Regression

Binary classification (Yes/No, Spam/Not Spam)
You need probability outputs
You want interpretable coefficients
Features have linear relationship with log-odds

Part 2: Naive Bayes

Naive Bayes uses probability and Bayes' theorem to classify. It's "naive" because it assumes features are independent (which they usually aren't!).

🎲 The Doctor's Diagnosis

A doctor sees symptoms: fever, cough, fatigue. Based on past patients:

P(Flu | Fever) = 80%
P(Flu | Cough) = 70%
P(Flu | Fatigue) = 60%

Naive Bayes combines these probabilities to make a diagnosis!

from sklearn.naive_bayes import GaussianNB, MultinomialNB

# Gaussian Naive Bayes (for continuous features)
gnb = GaussianNB()
gnb.fit(X_train, y_train)
print("Gaussian NB Accuracy:", gnb.score(X_test, y_test))

# Multinomial Naive Bayes (for count data, like text)
# Great for text classification!
from sklearn.feature_extraction.text import CountVectorizer

texts = ["I love this movie", "Terrible film", "Great acting", "Awful plot"]
labels = [1, 0, 1, 0]

vectorizer = CountVectorizer()
X_text = vectorizer.fit_transform(texts)

mnb = MultinomialNB()
mnb.fit(X_text, labels)

# Predict new text
new_text = vectorizer.transform(["I enjoyed this"])
print("Prediction:", mnb.predict(new_text))  # Likely: [1] (positive)

✅ Naive Bayes Pros & Cons

Pros: Fast, works well with small data, great for text classification

Cons: "Naive" independence assumption rarely holds, may produce poor probability estimates

Part 3: Support Vector Machines (SVM)

SVM finds the optimal boundary (hyperplane) that separates classes with maximum margin.

🏗️ Building a Fence

Imagine separating cats from dogs with a fence. SVM builds the fence:

As far as possible from both groups
The "support vectors" are the animals closest to the fence
If classes can't be separated linearly, SVM can transform the space (kernel trick)!

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

# SVM requires scaling!
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Linear SVM
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear.fit(X_train_scaled, y_train)
print("Linear SVM:", svm_linear.score(X_test_scaled, y_test))

# RBF Kernel SVM (for non-linear data)
svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svm_rbf.fit(X_train_scaled, y_train)
print("RBF SVM:", svm_rbf.score(X_test_scaled, y_test))

# Get probabilities (slower)
svm_proba = SVC(kernel='rbf', probability=True)
svm_proba.fit(X_train_scaled, y_train)
probabilities = svm_proba.predict_proba(X_test_scaled)

SVM Kernels

Kernel	Use Case	Description
linear	Linearly separable data	Straight line/plane boundary
rbf (default)	Most common, non-linear	Creates curved boundaries
poly	Polynomial relationships	Higher-degree curves

⚠️ SVM Tips

Always scale your data! SVM is very sensitive to feature scales
C parameter: Higher C = Less regularization = May overfit
gamma: Higher gamma = More complex boundaries = May overfit

Part 4: K-Nearest Neighbors (KNN)

KNN is the simplest algorithm: To classify a new point, look at its K nearest neighbors and vote!

For a full deep dive (how it works step-by-step, distance metrics, choosing K, scaling, and Python code), see the dedicated k-Nearest Neighbors (kNN) lesson.

🏘️ The Neighborhood Analogy

"Tell me who your friends are, and I'll tell you who you are."

If most of your 5 nearest neighbors like pizza, you probably like pizza too!

from sklearn.neighbors import KNeighborsClassifier

# KNN also needs scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create KNN with K=5
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)

print("KNN Accuracy:", knn.score(X_test_scaled, y_test))

# Finding optimal K
from sklearn.model_selection import cross_val_score

k_range = range(1, 20)
scores = []

for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    cv_scores = cross_val_score(knn, X_train_scaled, y_train, cv=5)
    scores.append(cv_scores.mean())

# Plot to find best K
plt.plot(k_range, scores)
plt.xlabel('K')
plt.ylabel('CV Accuracy')
plt.title('Finding Optimal K')
plt.show()

🎯 Choosing K

K too small (K=1): Overfitting - too sensitive to noise
K too large: Underfitting - loses local patterns
Odd K: Avoids ties in binary classification
Typical range: 3-10, use cross-validation to find best

Part 5: Which Algorithm to Choose?

Algorithm	Best For	Pros	Cons
Logistic Regression	Binary classification, need probabilities	Fast, interpretable, probabilistic	Assumes linear relationship
Naive Bayes	Text classification, small data	Very fast, works with small data	Independence assumption
SVM	High-dimensional data, clear margins	Works well in high dimensions	Slow on large data, needs scaling
KNN	Simple problems, no assumptions	No training, intuitive	Slow prediction, curse of dimensionality

# Quick comparison of all algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score

models = {
    'Logistic Regression': LogisticRegression(),
    'Naive Bayes': GaussianNB(),
    'SVM': SVC(),
    'KNN': KNeighborsClassifier()
}

print("Model Comparison (5-fold CV):\n")
for name, model in models.items():
    scores = cross_val_score(model, X_scaled, y, cv=5)
    print(f"{name:20s}: {scores.mean():.3f} ± {scores.std():.3f}")

Part 6: Classification Evaluation Metrics

Metric	Formula	When to Use
Accuracy	(TP + TN) / Total	Balanced classes
Precision	TP / (TP + FP)	Cost of false positives is high
Recall	TP / (TP + FN)	Cost of false negatives is high
F1 Score	2 × (Prec × Rec) / (Prec + Rec)	Balance of precision and recall
AUC-ROC	Area under ROC curve	Comparing models, imbalanced data

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, 
    f1_score, roc_auc_score, confusion_matrix, classification_report
)

# All metrics at once
print(classification_report(y_test, y_pred))

# Individual metrics
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))
print("F1:", f1_score(y_test, y_pred))

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:\n", cm)
# [[TN, FP],
#  [FN, TP]]

Summary

🎯 Quick Decision Guide

Text classification? → Naive Bayes or Logistic Regression
Need probabilities? → Logistic Regression
High-dimensional data? → SVM
Small dataset, simple problem? → KNN
Best overall for tabular data? → Try XGBoost/Random Forest first!

⚡ Pro Tips

Always scale data for SVM and KNN
Use cross-validation to compare models
Don't rely only on accuracy - check precision, recall, F1
For imbalanced data, use F1 score or AUC-ROC

🚫 Common Mistakes in Classification

Using accuracy alone when one class is rare — A model that always predicts the majority class can have high accuracy; use F1, precision, recall, or AUC-ROC.
Not scaling for KNN or SVM — Both use distances; features on different scales will dominate; always scale (e.g. StandardScaler) before fitting.
Ignoring the business cost of errors — Missing a disease (low recall) vs false alarm (low precision) have different costs; choose the metric that matches the problem.

💭 Short reflection

In one sentence: when would you care more about recall than precision? Give a real-world example (e.g. medical test, spam filter).

✅ CORE (Must know)

Classification: predict a category (binary or multiclass).
Logistic Regression: outputs probability; decision boundary; use for interpretability and probabilities.
KNN: assign class by majority of k nearest neighbors; scale features; sensitive to k.
SVM: finds max-margin hyperplane; kernel for non-linear boundaries; scale features.
Metrics: accuracy, precision, recall, F1; confusion matrix (TN, FP, FN, TP).
Imbalanced data: accuracy misleading; use F1, precision-recall curve, or AUC-ROC.

📚 NON-CORE (Good to know)

ROC curve and AUC interpretation.
Multiclass: one-vs-one vs one-vs-rest.
Probability calibration (e.g. Platt scaling).
When to prefer which algorithm (text vs tabular, size of data).

🎯 Classification Algorithms