Statistics for ML — FAANG-Level Lab

Goal: Confidence intervals, hypothesis testing, and interpretation for ML engineering.

Outcome: You can quantify uncertainty and avoid p-value traps.

Section 1 — Estimators (Mean/Variance)

Task 1.1: Unbiased sample variance

Implement sample mean and unbiased sample variance (ddof=1) without calling np.var(..., ddof=1).

• ●mean = sum(x)/n
• ●unbiased var = sum((x-mean)^2)/(n-1)

Explain: Why divide by (n-1) instead of n?

Section 2 — Confidence Interval for Mean (Normal approx)

Task 2.1: 95% CI for mean

Compute a 95% CI for mean using normal approximation: CI = mean ± z * s/sqrt(n), where z≈1.96.

• ●Use unbiased sample std

FAANG gotcha: CI is about the mean, not individual outcomes.

Task 2.2: Coverage simulation

Simulate repeated sampling from Normal(mu=0, sigma=1). Estimate how often 95% CI contains true mean.

• ●Run many trials
• ●Count coverage

Explain: Why isn't coverage exactly 0.95 in finite simulation?

Section 3 — Hypothesis Testing (Two-sample test intuition)

Task 3.1: Permutation test for A/B (no scipy)

Given samples A and B, test whether mean(B) - mean(A) is significant via permutation.

• ●Combine samples
• ●Shuffle and split
• ●Compute diff distribution
• ●p-value = fraction of diffs >= observed (two-sided if needed)

FAANG gotcha: p-value is not P(H0 true).

Section 4 — Multiple Comparisons (Gotcha)

Task 4.1: Bonferroni correction

If you run m tests at alpha=0.05, Bonferroni uses alpha/m per test.

Compute adjusted alpha for m=20 and explain why this matters in feature slicing / metric dashboards.