NLP - Natural Language Processing | Fakhruddin Khambaty's Learning Hub

Part 1: What is NLP?

NLP (Natural Language Processing) is a field of AI that helps computers understand, interpret, and generate human language.

📌 In One Sentence

NLP = turning text into numbers (e.g. word counts, TF-IDF, or dense vectors) so a computer can classify, search, or generate language. We preprocess (lowercase, tokenize, remove stopwords), then vectorize, then train a model—so every step is "text → numbers → prediction or output."

🤖 The Translation Problem

Computers speak in 0s and 1s. Humans speak in words with context, sarcasm, and nuance.

NLP bridges this gap! It converts human language into numbers that machines can process.

Real-World NLP Applications

Application	Example
Chatbots & Virtual Assistants	Siri, Alexa, ChatGPT
Sentiment Analysis	Analyzing product reviews, social media
Machine Translation	Google Translate
Text Summarization	News article summaries
Spam Detection	Email spam filters
Named Entity Recognition	Extracting names, dates, locations

Part 2: Text Preprocessing

Raw text is messy! Before analysis, we need to clean and standardize it.

Original

"The QUICK brown Fox!!!"

→

Lowercase

"the quick brown fox!!!"

→

Remove Punctuation

"the quick brown fox"

→

Tokenize

["the", "quick", "brown", "fox"]

Common Preprocessing Steps

import re
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer

# Download required data
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

text = "The QUICK brown foxes are jumping over the lazy dogs!!!"

# Step 1: Lowercase
text = text.lower()
print("Lowercase:", text)

# Step 2: Remove punctuation and special characters
text = re.sub(r'[^a-zA-Z\s]', '', text)
print("No punctuation:", text)

# Step 3: Tokenize (split into words)
tokens = word_tokenize(text)
print("Tokens:", tokens)

# Step 4: Remove stopwords (common words like "the", "is", "are")
stop_words = set(stopwords.words('english'))
tokens = [word for word in tokens if word not in stop_words]
print("Without stopwords:", tokens)

# Step 5: Lemmatization (reduce to base form)
lemmatizer = WordNetLemmatizer()
tokens = [lemmatizer.lemmatize(word) for word in tokens]
print("Lemmatized:", tokens)

# Output:
# Lowercase: the quick brown foxes are jumping over the lazy dogs!!!
# No punctuation: the quick brown foxes are jumping over the lazy dogs
# Tokens: ['the', 'quick', 'brown', 'foxes', 'are', 'jumping', 'over', 'the', 'lazy', 'dogs']
# Without stopwords: ['quick', 'brown', 'foxes', 'jumping', 'lazy', 'dogs']
# Lemmatized: ['quick', 'brown', 'fox', 'jumping', 'lazy', 'dog']

Stemming vs Lemmatization

Method	How it Works	Example	Pros/Cons
Stemming	Chops off word endings	"running" → "runn"	Fast, but crude
Lemmatization	Uses dictionary to find base form	"running" → "run"	Accurate, but slower

Part 3: Converting Text to Numbers

ML models need numbers, not words! We need to convert text into numerical vectors.

Method 1: Bag of Words (CountVectorizer)

📦 The Bag of Words Concept

Imagine throwing all words from a document into a bag. We count how many times each word appears.

"The cat sat on the mat" → {the: 2, cat: 1, sat: 1, on: 1, mat: 1}

from sklearn.feature_extraction.text import CountVectorizer

documents = [
    "I love machine learning",
    "Machine learning is awesome",
    "I love deep learning too"
]

# Create vectorizer
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)

# View the vocabulary
print("Vocabulary:", vectorizer.get_feature_names_out())

# View the vectors
print("Vectors:\n", X.toarray())

# Output:
# Vocabulary: ['awesome' 'deep' 'is' 'learning' 'love' 'machine' 'too']
# Vectors:
# [[0 0 0 1 1 1 0]    # "I love machine learning"
#  [1 0 1 1 0 1 0]    # "Machine learning is awesome"
#  [0 1 0 1 1 0 1]]   # "I love deep learning too"

Method 2: TF-IDF (Better!)

⚖️ TF-IDF: Term Frequency × Inverse Document Frequency

TF: How often a word appears in THIS document

IDF: How rare the word is across ALL documents

Words appearing in every document (like "the") get LOW scores. Unique, important words get HIGH scores!

from sklearn.feature_extraction.text import TfidfVectorizer

# TF-IDF Vectorizer
tfidf = TfidfVectorizer()
X_tfidf = tfidf.fit_transform(documents)

print("TF-IDF Vectors:\n", X_tfidf.toarray().round(2))

# Notice: "learning" (appears in all docs) has lower scores
# "awesome", "deep" (unique) have higher scores

Part 4: Sentiment Analysis Example

Let's build a simple sentiment classifier to predict if reviews are positive or negative!

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, classification_report

# Sample data
reviews = [
    "This movie was amazing! I loved it",
    "Terrible film, waste of time",
    "Great acting and wonderful story",
    "Boring and predictable, would not recommend",
    "Best movie I've seen this year",
    "Awful experience, horrible plot",
    "Fantastic! A must watch",
    "Disappointing and dull"
]
labels = [1, 0, 1, 0, 1, 0, 1, 0]  # 1=positive, 0=negative

# Vectorize
tfidf = TfidfVectorizer(stop_words='english')
X = tfidf.fit_transform(reviews)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.25)

# Train Naive Bayes classifier
model = MultinomialNB()
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))

# Test with new reviews
new_reviews = ["This was absolutely wonderful!", "I hated every minute"]
new_vectors = tfidf.transform(new_reviews)
print("Predictions:", model.predict(new_vectors))  # [1, 0]

Part 5: Named Entity Recognition (NER)

NER extracts important entities like names, places, dates, and organizations from text.

import spacy

# Load English model
nlp = spacy.load('en_core_web_sm')

text = "Apple Inc. was founded by Steve Jobs in California on April 1, 1976. The company is worth $2 trillion."

# Process text
doc = nlp(text)

# Extract entities
print("Named Entities:")
for ent in doc.ents:
    print(f"  {ent.text} → {ent.label_}")

# Output:
# Named Entities:
#   Apple Inc. → ORG (Organization)
#   Steve Jobs → PERSON
#   California → GPE (Geopolitical Entity/Location)
#   April 1, 1976 → DATE
#   $2 trillion → MONEY

📋 Common Entity Types

PERSON: People names
ORG: Companies, organizations
GPE: Countries, cities, states
DATE: Dates and times
MONEY: Monetary values
PRODUCT: Product names

Part 6: Word Embeddings (Word2Vec)

Word embeddings capture meaning of words as dense vectors. Similar words have similar vectors!

🎯 The Magic of Word Embeddings

Word2Vec learns that:

King - Man + Woman ≈ Queen

It captures semantic relationships between words!

from gensim.models import Word2Vec
from gensim.models import KeyedVectors

# Sample sentences (typically you'd use MUCH more data)
sentences = [
    ['king', 'queen', 'royal', 'palace'],
    ['man', 'woman', 'child', 'family'],
    ['python', 'programming', 'code', 'computer'],
    ['machine', 'learning', 'data', 'algorithm']
]

# Train Word2Vec model
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)

# Get vector for a word
king_vector = model.wv['king']
print("Vector for 'king' (first 10 dims):", king_vector[:10])

# Find similar words
similar = model.wv.most_similar('python', topn=3)
print("Similar to 'python':", similar)

# Using pre-trained embeddings (much better!)
# Download: https://nlp.stanford.edu/projects/glove/
# glove = KeyedVectors.load_word2vec_format('glove.6B.100d.txt')

💡 Pre-trained Embeddings

For real projects, use pre-trained embeddings trained on billions of words:

GloVe: Stanford's pre-trained vectors
FastText: Facebook's embeddings (handles unknown words)
BERT: Context-aware embeddings from Google

🚫 Common Mistakes in NLP

Different preprocessing at train vs test — Use the same tokenizer, stopwords, and vectorizer (fit on train, transform on test).
Relying only on Bag of Words for sentiment — Use TF-IDF or embeddings to capture more meaning when order matters.
Not handling empty text — After removing stopwords some documents can be empty; handle them or the vectorizer may fail.

💭 Short reflection

In one sentence: why does Bag of Words ignore word order and why might that be a problem for meaning (e.g. “not good” vs “good”)?

✅ CORE (Must know)

Preprocessing: lowercase, tokenize, remove stopwords, lemmatize/stem.
Bag of Words: count matrix; simple, ignores order.
TF-IDF: weight by importance; penalizes very common terms.
Word embeddings: dense vectors (e.g. Word2Vec, GloVe, FastText); capture similarity.
Pipeline: preprocess → vectorize → train classifier (e.g. Naive Bayes, SVM).

📚 NON-CORE (Good to know)

NER, sentiment analysis, transformers (BERT) for context.

Summary

Concept	Key Points
Text Preprocessing	Lowercase, remove punctuation, tokenize, remove stopwords, lemmatize
Bag of Words	Count word occurrences - simple but ignores order
TF-IDF	Weights words by importance - penalizes common words
Named Entity Recognition	Extract names, places, dates, organizations from text
Word Embeddings	Dense vectors capturing word meaning and relationships

🎯 NLP Pipeline

Collect text data
Preprocess (clean, tokenize, remove stopwords)
Vectorize (TF-IDF or embeddings)
Train model (Naive Bayes, SVM, or neural network)
Evaluate and iterate

Knowledge check

Why fit a TF-IDF vectorizer inside a pipeline on the training documents only?

Previous: Deep Learning Next: Classification Algorithms

💬 NLP - Natural Language Processing