Note 13-15 for 334.

Author: localhost433

notes/courses/MATH-UA-334/13-hypothesis-testing.md

@@ -0,0 +1,135 @@
+---
+title: Hypothesis Testing
+date: 2026-03-04
+---
+
+## 1. Introduction to Hypothesis Testing
+
+Hypothesis testing is a formal statistical procedure used to make decisions about the underlying properties of a population based on a sample of observations. The objective is to evaluate whether there is sufficient statistical evidence to reject a default baseline assumption in favor of an alternative claim.
+
+We formalize our problem using two competing hypotheses:
+
+- **Null hypothesis ($H\_0$):** This represents the default status quo, a statement of "no effect," "no discovery," or "no difference."
+- **Alternative hypothesis ($H\_1$):** This represents the active claim we wish to prove, representing a "discovery" or a significant deviation from the baseline.
+
+---
+
+## 2. Statistical Formulation
+
+Suppose we observe data $X = (X\_1, \dots, X\_n)$ drawn from a probability distribution $f\_\theta$, where $\theta \in \Theta$ is an unknown parameter. The parameter space $\Theta$ is partitioned into two disjoint subsets, $\Theta\_0$ and $\Theta\_1$.
+
+The formal hypothesis testing problem is stated as:
+$$
+    H\_0: \theta \in \Theta\_0 \quad \text{vs.} \quad H\_1: \theta \in \Theta\_1
+$$
+By definition, we require $\Theta\_0 \cap \Theta\_1 = \emptyset$.
+
+> Since it's a partition of the parameter space $\Theta$.
+
+**Definition (Statistical Test):**
+A test is a formal decision rule, defined as a function $T$ that maps the observed data $X$ to the set of hypotheses $\{H\_0, H\_1\}$. Based on the observed values, the test explicitly instructs us to either "Accept $H\_0$" or "Reject $H\_0$" (which implies accepting $H\_1$).
+
+### 2.1 Types of Hypotheses
+
+Hypotheses are broadly categorized based on the specific number of parameter values they contain.
+
+- **Simple Hypothesis:** The hypothesis precisely specifies exactly one single value for the parameter. For example, $H\_0: \theta = \theta\_0$. Thus, $|\Theta\_0| = 1$.
+- **Composite Hypothesis:** The hypothesis specifies a range or multiple possible values for the parameter. For example, $H\_1: \theta > \theta\_0$ or $H\_1: \theta \neq \theta\_0$. Thus, $|\Theta\_1| > 1$.
+
+### 2.2 Examples of Testing Scenarios
+
+**Example 1: Coin Tossing (Simple vs. Simple)**
+Suppose I toss a coin with bias $p$ exactly $4$ times, and $X$ of the tosses turn out to be heads. Suppose we have some prior knowledge that the bias is either $0.5$ or $0.7$.
+We formulate the hypothesis testing problem as:
+$$
+    H\_0: p = 0.5 \quad \text{vs.} \quad H\_1: p = 0.7
+$$
+A test $T$ would take the observed number of heads $X \in \{0, 1, 2, 3, 4\}$ and map it to a decision in $\{H\_0, H\_1\}$.
+
+**Example 2: Normal Mean Testing**
+Suppose we have a sample $X\_1, \dots, X\_n \sim \mathcal{N}(\mu, \sigma^2)$ with a known variance $\sigma^2$ but an unknown mean $\mu$.
+
+- **Simple vs. Simple:** $H\_0: \mu = \mu\_0$ vs. $H\_1: \mu = \mu\_1$.
+- **Two-sided test:** $H\_0: \mu = \mu\_0$ vs. $H\_1: \mu \neq \mu\_0$. (Simple vs. Composite)
+- **One-sided test:** $H\_0: \mu = \mu\_0$ vs. $H\_1: \mu > \mu\_0$. (Simple vs. Composite)
+
+---
+
+## 3. Evaluating a Statistical Test
+
+Whenever we make a decision using a statistical test, we risk making one of two distinct types of errors:
+
+1. **$\alpha$ - Type I Error (False Positive):** We incorrectly reject the null hypothesis $H\_0$ when it is actually true.
+    - The probability of committing a Type I Error is called the **Significance Level**, denoted by $\alpha$.
+    - $\alpha = \prob(\text{Output } H\_1 \mid H\_0 \text{ is true})$.
+2. **$\beta$ - Type II Error (False Negative):** We incorrectly accept the null hypothesis $H\_0$ when the alternative $H\_1$ is actually true.
+    - The probability of a Type II error is denoted by $\beta$.
+    - The **Power** of the test is defined as $1 - \beta$, which is the probability of correctly rejecting $H\_0$ when $H\_1$ is true.
+    - $1 - \beta = \prob(\text{Output } H\_1 \mid H\_1 \text{ is true})$.
+
+In rigorous statistical practice, it is mathematically impossible to simultaneously minimize both $\alpha$ and $\beta$ for a fixed sample size $n$. The standard frequentist paradigm dictates that we fix the significance level $\alpha$ at a pre-determined, strictly controlled threshold (such as $0.05$ or $0.01$) and then actively seek the specific test that maximizes the statistical power $1 - \beta$.
+
+> $\alpha$ and $\beta$ move in opposite directions. $\alpha$ and Power move in the same direction.
+
+---
+
+## 4. The Likelihood Ratio Test (Simple vs. Simple)
+
+When both $H\_0$ and $H\_1$ are simple hypotheses (e.g., $H\_0: \theta = \theta\_0$ and $H\_1: \theta = \theta\_1$), the **Neyman-Pearson Lemma** provides the absolute optimal test that maximizes power for a given significance level $\alpha$. This optimal test is the **Likelihood Ratio (LR) Test**.
+
+### 4.1 The Decision Rule
+
+The Likelihood Ratio is defined as the ratio of the likelihood of the data under the alternative hypothesis to the likelihood of the data under the null hypothesis:
+$$
+    \text{LR}(X) = \frac{f\_{\theta\_1}(X\_1, \dots, X\_n)}{f\_{\theta\_0}(X\_1, \dots, X\_n)}
+$$
+The formal decision rule for the Likelihood Ratio Test states that we should reject $H\_0$ if the likelihood ratio strictly exceeds a specific critical threshold $c$:
+$$
+    \text{Reject } H\_0 \iff \text{LR}(X) > c
+$$
+The critical value $c$ is meticulously chosen to ensure that the probability of a Type I error exactly equals our desired significance level $\alpha$, that:
+$$
+    \prob(\text{LR}(X) > c \mid \theta = \theta\_0) = \alpha
+$$
+
+### 4.2 Example: Normal Mean Testing
+
+Suppose $X\_1, \dots, X\_n \sim \mathcal{N}(\mu, \sigma^2)$. We want to find the exact LR test for $H\_0: \mu = \mu\_0$ versus $H\_1: \mu = \mu\_1$, assuming $\mu\_1 > \mu\_0$.
+
+**Step 1: Construct the Likelihood Ratio**
+$$
+    \text{LR}(X) = \frac{\exp\left(-\frac{1}{2\sigma^2} \sum\_{i=1}^n (X\_i - \mu\_1)^2\right)}{\exp\left(-\frac{1}{2\sigma^2} \sum\_{i=1}^n (X\_i - \mu\_0)^2\right)}
+$$
+By expanding the squares inside the exponential and simplifying, we get:
+$$
+    \text{LR}(X) = \exp\left( \frac{n(\mu\_1 - \mu\_0)}{\sigma^2} \overline{X}\_n - \frac{n(\mu\_1^2 - \mu\_0^2)}{2\sigma^2} \right)
+$$
+
+**Step 2: Simplify the Rejection Region**
+We reject $H\_0$ when $\text{LR}(X) > c$. Taking the natural logarithm of both sides:
+$$
+    \begin{align*}
+        \frac{n(\mu\_1 - \mu\_0)}{\sigma^2} \overline{X}\_n - \frac{n(\mu\_1^2 - \mu\_0^2)}{2\sigma^2} &> \ln c \\\\
+        \overline{X}\_n &> \frac{\sigma^2}{n(\mu\_1 - \mu\_0)} \ln c + \frac{\mu\_1 + \mu\_0}{2} = \tau
+    \end{align*}
+$$
+Because $\mu\_1 > \mu\_0$, the inequality direction is strictly preserved. The test mathematically reduces to: **Reject $H\_0$ if $\overline{X}\_n > \tau$.**
+
+**Step 3: Determine the Critical Threshold**
+We want $\prob(\overline{X}\_n > \tau \mid \mu = \mu\_0) = \alpha$.
+Under $H\_0$, the sample mean follows $\overline{X}\_n \sim \mathcal{N}(\mu\_0, \sigma^2/n)$.
+Standardizing this variable gives:
+$$
+    \prob\left( \frac{\overline{X}\_n - \mu\_0}{\sigma/\sqrt{n}} > \frac{\tau - \mu\_0}{\sigma/\sqrt{n}} \right) = \alpha
+$$
+Because the standardized variable is a standard Normal $Z$, we set $\frac{\tau - \mu\_0}{\sigma/\sqrt{n}} = z\_\alpha$, where $z\_\alpha$ is the upper $\alpha$-quantile of the standard normal distribution. This yields the final threshold:
+$$
+    \tau = \mu\_0 + z\_\alpha \frac{\sigma}{\sqrt{n}}
+$$
+
+---
+
+## References
+
+1. Rice, J. A. (2007). *Mathematical Statistics and Data Analysis* (3rd ed.). Thomson Brooks/Cole.
+2. Han, Y. (2026). Lecture 13: Simple Hypothesis Testing.

notes/courses/MATH-UA-334/14-p-value-conf-sets.md

@@ -0,0 +1,135 @@
+---
+title: p-Values and Confidence Sets
+date: 2026-03-09
+---
+
+## 1. Moving Beyond Simple Hypothesis Testing
+
+In the previous lecture, we established that the Likelihood Ratio (LR) test is perfectly optimal for differentiating between two simple hypotheses. However, this foundational paradigm is highly restrictive for practical applications.
+
+1. **Composite Hypotheses:** The LR test cannot be directly applied when the alternative hypothesis $H\_1$ is composite (e.g., $H\_1: \theta > \theta\_0$).
+2. **Difficult Power Calculations:** Controlling the Type II error probability ($\beta$) is mathematically difficult when dealing with composite alternative spaces, because $\beta$ must be calculated for every single parameter configuration within $H\_1$:
+    $$
+        \beta = \max\_{\theta \in \Theta\_1} \prob(\text{Accept } H\_0 \mid \theta)
+    $$
+
+Therefore, in broad practical usage, most statistical tests are exclusively designed to strictly control the Type I error (the significance level $\alpha$), without explicitly optimizing for $\beta$. We accomplish this by relying on test statistics with known null distributions.
+
+> In addition, under a practical scenario, we fix $\alpha$ because false positives are considered worse than false negatives.
+
+---
+
+## 2. Test Statistics and Critical Regions
+
+The standard strategy for composite hypothesis testing is to construct a specific measurable function of the data and the parameter, $h(X, \theta)$, such that under the null hypothesis $H\_0$, the sampling distribution of $h$ is completely known and free of unknown parameters.
+
+For a test of $H\_0: \theta = \theta\_0$, we calculate the test statistic evaluated at the null parameter: $T(X) = h(X, \theta\_0)$.
+
+We then establish a **rejection region** based strictly on critical values. For a one-sided test, we reject $H\_0$ if $T(X) > c\_\alpha$. The threshold $c\_\alpha$ is systematically chosen such that:
+$$
+    \prob(T(X) > c\_\alpha \mid H\_0) = \alpha
+$$
+If the test statistic falls outside this narrowly constructed region, we conclude that the observed data is fundamentally incompatible with the null hypothesis, and we reject $H\_0$.
+
+---
+
+## 3. The $p$-Value
+
+When simply reporting "Reject" or "Accept", a significant amount of statistical context is lost. A test statistic that barely crosses the threshold is treated identically to one that massively exceeds it. The **$p$-value** addresses this limitation by reporting the continuous strength of the evidence against the null hypothesis.
+
+**Definition (P-Value):**
+The $p$-value is the probability, calculated precisely under the assumption that the null hypothesis $H\_0$ is true, of observing a test statistic at least as extreme as the one that was actually observed in the sample data.
+
+If $T\_\text{obs}$ is the realized, observed value of our test statistic $T(X)$, the $p$-value for a right-sided test is:
+$$
+    \text{$p$-value} = \prob(T(X) \ge T\_\text{obs} \mid H\_0)
+$$
+
+### 3.1 Properties of the $p$-Value
+
+1. **Decision Rule:** A $p$-value perfectly acts as an alternative decision mechanism. We reject $H\_0$ if and only if the calculated $\text{$p$-value} \le \alpha$.
+2. **Uniform Distribution Under the Null:** A fascinating mathematical property is that if the null hypothesis is completely true, and the test statistic is continuous, the $p$-value itself acts as a random variable that is uniformly distributed on the interval $[0, 1]$.
+    $$
+        \text{$p$-value} \sim \text{Unif}[0, 1] \quad \text{under } H\_0
+    $$
+
+---
+
+## 4. Confidence Sets and Duality
+
+Hypothesis testing aims to determine if a specific, isolated parameter value $\theta\_0$ is plausible. A **Confidence Set** essentially extends this logic by finding *all* possible parameter values that are plausible given the observed data.
+
+**Definition (Confidence Set):**
+A $(1 - \alpha)$-confidence set (or interval) $CI(X)$ is a data-dependent interval constructed such that the true parameter $\theta\_0$ is contained within the set with a probability of at least $1 - \alpha$ prior to sampling.
+$$
+    \prob(\theta\_0 \in CI(X) \mid \theta = \theta\_0) \ge 1 - \alpha \quad \text{for every } \theta\_0
+$$
+
+### 4.1 The Duality Principle
+
+There is a profound mathematical duality between hypothesis testing and confidence intervals. A confidence interval simply consists of all the null hypothesis values that would *not* be rejected by a level-$\alpha$ hypothesis test.
+
+Let $A(\theta\_0)$ be the acceptance region of a level-$\alpha$ test for $H\_0: \theta = \theta\_0$.
+$$
+    \prob(X \in A(\theta\_0) \mid \theta = \theta\_0) \ge 1 - \alpha
+$$
+The corresponding confidence interval is constructed by simply pivoting this probability statement to isolate the parameter:
+$$
+    CI(X) = \{ \theta\_0 : X \in A(\theta\_0) \}
+$$
+
+### 4.2 Example 1: Normal Mean with Unknown Variance
+
+Suppose $X\_1, \dots, X\_n \sim \mathcal{N}(\mu, \sigma^2)$, with both parameters fully unknown. We wish to test $H\_0: \mu = \mu\_0$ versus $H\_1: \mu \neq \mu\_0$.
+
+Because the true variance $\sigma^2$ is unknown, we use the sample variance $S\_n^2$. Our standard test statistic leverages the Student's t-distribution:
+$$
+    T(X) = \frac{\overline{X}\_n - \mu\_0}{S\_n / \sqrt{n}} \sim t\_{n-1} \quad \text{under } H\_0
+$$
+
+The symmetric acceptance region for a level-$\alpha$ test is:
+$$
+    A(\mu\_0) = \{ X : -t\_{n-1, \alpha/2} \le \frac{\overline{X}\_n - \mu\_0}{S\_n / \sqrt{n}} \le t\_{n-1, \alpha/2} \}
+$$
+
+To find the corresponding confidence interval, we mathematically pivot the inequality inside the acceptance region to isolate $\mu\_0$ in the center:
+$$
+    \begin{align*}
+        -t\_{n-1, \alpha/2} &\le \frac{\overline{X}\_n - \mu\_0}{S\_n / \sqrt{n}} \le t\_{n-1, \alpha/2} \\\\
+        -t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}} &\le \overline{X}\_n - \mu\_0 \le t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}} \\\\
+        -\overline{X}\_n - t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}} &\le -\mu\_0 \le -\overline{X}\_n + t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}} \\\\
+        \overline{X}\_n - t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}} &\le \mu\_0 \le \overline{X}\_n + t\_{n-1, \alpha/2} \frac{S\_n}{\sqrt{n}}
+    \end{align*}
+$$
+Thus, the exact $(1-\alpha)$ confidence interval for $\mu$ is directly derived from the hypothesis test's acceptance criteria.
+
+### 4.3 Example 2: Normal Variance Testing
+
+Suppose we instead wish to test the variance of our normal sample, setting $H\_0: \sigma = \sigma\_0$ versus a two-sided alternative $H\_1: \sigma \neq \sigma\_0$.
+
+A natural test statistic is derived using the sample variance $S\_n^2$:
+$$
+    T(X) = \frac{(n-1)S\_n^2}{\sigma\_0^2} = \frac{\sum\_{i=1}^n (X\_i - \overline{X}\_n)^2}{\sigma\_0^2} \sim \chi\_{n-1}^2 \quad \text{under } H\_0
+$$
+
+The acceptance region for a level-$\alpha$ test involves the critical values of the Chi-Square distribution:
+$$
+    A(\sigma\_0) = \{ X : c\_{1 - \alpha/2} \le \frac{\sum\_{i=1}^n (X\_i - \overline{X}\_n)^2}{\sigma\_0^2} \le c\_{\alpha/2} \}
+$$
+
+By pivoting this acceptance region, we extract a confidence interval for the unknown variance $\sigma^2$:
+$$
+    \begin{align*}
+        c\_{1 - \alpha/2} &\le \frac{(n-1)S\_n^2}{\sigma\_0^2} \le c\_{\alpha/2} \\\\
+        \frac{1}{c\_{\alpha/2}} &\le \frac{\sigma\_0^2}{(n-1)S\_n^2} \le \frac{1}{c\_{1 - \alpha/2}} \\\\
+        \frac{(n-1)S\_n^2}{c\_{\alpha/2}} &\le \sigma\_0^2 \le \frac{(n-1)S\_n^2}{c\_{1 - \alpha/2}}
+    \end{align*}
+$$
+This precisely constructs the $(1-\alpha)$ confidence interval for $\sigma^2$.
+
+---
+
+## References
+
+1. Rice, J. A. (2007). *Mathematical Statistics and Data Analysis* (3rd ed.). Thomson Brooks/Cole.
+2. Han, Y. (2026). Lecture 14: P-value, confidence set.