Data Analysis Techniques — The Fun Part Where Data Finally Talks

You’ve collected the data (remember Data Collection Methods?), and you know the big idea of Data Science. Now it’s time to interrogate the data like a sympathetic detective who also loves charts.

---

Why this chapter matters (quick and spicy)

You can collect mountains of data and even build slick NLP embeddings (shoutout to our previous dive into Natural Language Processing), but if you don't analyze the data well, your models will be politely wrong and your stakeholders will be quietly furious. Data analysis is the bridge between raw collection and actual intelligence — it's where patterns emerge, bias is uncovered, and ideas turn into robust features.

Think of it as: Data Collection = grocery shopping. Data Analysis = deciding whether to eat kale or make guacamole. Both matter, but one determines whether people actually enjoy dinner.

---

Roadmap: What we’ll cover

1. Exploratory Data Analysis (EDA)
2. Data cleaning & handling missing/erroneous values
3. Feature engineering & transformations
4. Dimensionality reduction & visualization
5. Clustering & unsupervised patterns
6. Statistical tests, correlation vs causation
7. Quick NLP-specific analysis tips (because we just studied NLP)

---

1) Exploratory Data Analysis (EDA) — flirt before you commit

EDA is the ritual of asking the data a hundred small questions before you try to make it propose marriage.

• Start with summary statistics: mean, median, std, quartiles.
• Visualize: histograms, boxplots, scatterplots, pairplots.
• Look for weirdness: skew, multimodality, gaps, and outliers.

Example checklist:

• Are distributions normal-ish? (No? Fine.)
• Any columns with many missing values?
• Are there suspicious duplicates?

Code-snippet (Python/pandas seaborn):

import seaborn as sns
import pandas as pd
sns.pairplot(df.sample(500), hue='label')

---

2) Data cleaning — the boring vaccine that prevents disaster

Common tasks:

• Impute missing values: mean/median for numeric, mode or "missing" token for categorical, or model-based imputation.
• Handle outliers: clip, transform, or keep and model robustly (e.g., use median).
• Convert types: datetimes, categories, numeric strings.
• De-duplicate and sanity-check ranges (e.g., negative ages?).

Pro-tip: Document everything. Future you and colleagues will worship you like a documentation saint.

---

3) Feature engineering — where creativity meets math

Feature engineering often beats fancy algorithms. It's the secret ingredient.

• Create interactions (age * income), bins, and aggregated statistics (rolling means for time series).
• Date features: hour, day-of-week, month, is_holiday.
• Text features: TF-IDF, n-grams, sentiment scores, named entity counts.
• Encoding categorical variables: one-hot, target encoding, embeddings.

Quick table: When to use what encoding

Categorical size	Suggested encoding
Small (<10 values)	One-hot
Medium (10–100)	Target encoding / frequency encoding
Large (>100)	Embeddings or hashing

---

4) Dimensionality reduction & visualization — make high-dim feel human

When your feature space is that 900-dimensional horror movie, reduce it.

• PCA: linear, great for variance capture and preprocessing.
• t-SNE: non-linear, excellent for visual clusters but fiddly and non-deterministic.
• UMAP: similar to t-SNE but faster and preserves global structure better.

Use-cases:

• PCA for feature compression before a model.
• t-SNE/UMAP for visualization and cluster discovery.

Code hint:

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
coords = pca.fit_transform(feature_matrix)

---

5) Clustering & unsupervised pattern discovery

When you don’t have labels, you still have curiosity.

• K-Means: fast, assumes spherical clusters.
• DBSCAN: density-based, finds arbitrary shapes, good for noisy data.
• Hierarchical: tree of clusters, interpretable dendrograms.

Question to ask: "Do clusters align with a business metric?" If not, they're pretty murals but not actionable.

---

6) Stats, correlation, and that Myth of Causation

• Correlation != causation. Repeat: correlation != causation. (Put a sticky note on your monitor.)
• Use A/B tests, randomized experiments, or causal inference tools (propensity scores, instrumental variables) for causal claims.
• Basic tests: t-test, chi-square, ANOVA for group differences.

Mini-checklist before claiming causality:

1. Is there a plausible mechanism?
2. Are confounders controlled?
3. Is the effect robust across slices?

---

7) Quick NLP-specific analysis tips (bridging from our NLP session)

• Inspect token distributions and vocabulary size — heavy-tailed distributions are normal.
• Use TF-IDF to find discriminative words; use embeddings (BERT, Word2Vec) for semantics.
• Topic modeling (LDA) for exploratory themes, but validate with human reading.
• Pay attention to class imbalance in labels (common in text classification).

Pro-tip: Examine model inputs that cause wrong predictions — often you’ll spot annotation or data-collection issues.

---

Contrasting perspectives: More analysis vs. more model complexity

• Some say: "Feature engineering is dead, just use huge models." Reality: big models help, but good features + sane analysis still win in most applied settings.
• Others push end-to-end learning. That's powerful but brittle when data is limited or biased.

The practical answer: do both — solid analysis first, then choose model complexity based on evidence.

---

Closing — Takeaways (and a mild pep talk)

• EDA & cleaning are non-negotiable. They save time and credibility.
• Feature engineering is often higher ROI than algorithm shopping.
• Dimensionality reduction and clustering help you see structure, but translate findings into business questions.
• Statistics guard you against wild claims; use causal tools when making causal statements.
• For NLP, bias-check your vocabulary and look under the hood of embeddings.

Final thought: Data analysis is half science, half detective work, and a sprinkle of theater. Be curious, be skeptical, and be scrupulous.

Go forth: run an EDA, make some plots, and then make someone say, "Huh — that actually changes what we do."