as promised, more

Author: maxbiostat

assignments/2021_A1.pdf

@@ -0,0 +1,148 @@
+\documentclass[a4paper,10pt, notitlepage]{report}
+\usepackage{geometry}
+\geometry{verbose,tmargin=30mm,bmargin=25mm,lmargin=25mm,rmargin=25mm}
+\usepackage[utf8]{inputenc}
+\usepackage[sectionbib]{natbib}
+\usepackage{amssymb}
+\usepackage{amsmath}
+\usepackage{enumitem}
+\usepackage{xcolor}
+\usepackage{cancel}
+\usepackage{mathtools}
+\usepackage{caption}
+\usepackage{subcaption}
+\usepackage{float}
+\PassOptionsToPackage{hyphens}{url}\usepackage{hyperref}
+\hypersetup{colorlinks=true,citecolor=blue}
+
+
+\newtheorem{thm}{Theorem}
+\newtheorem{lemma}[thm]{Lemma}
+\newtheorem{proposition}[thm]{Proposition}
+\newtheorem{remark}[thm]{Remark}
+\newtheorem{defn}[thm]{Definition}
+
+%%%%%%%%%%%%%%%%%%%% Notation stuff
+\newcommand{\pr}{\operatorname{Pr}} %% probability
+\newcommand{\vr}{\operatorname{Var}} %% variance
+\newcommand{\rs}{X_1, X_2, \ldots, X_n} %%  random sample
+\newcommand{\irs}{X_1, X_2, \ldots} %% infinite random sample
+\newcommand{\rsd}{x_1, x_2, \ldots, x_n} %%  random sample, realised
+\newcommand{\bX}{\boldsymbol{X}} %%  random sample, contracted form (bold)
+\newcommand{\bx}{\boldsymbol{x}} %%  random sample, realised, contracted form (bold)
+\newcommand{\bT}{\boldsymbol{T}} %%  Statistic, vector form (bold)
+\newcommand{\bt}{\boldsymbol{t}} %%  Statistic, realised, vector form (bold)
+\newcommand{\emv}{\hat{\theta}}
+\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
+\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
+
+% Title Page
+\title{Exam 2 (A2)}
+\author{Class: Bayesian Statistics \\ Instructor: Luiz Max Carvalho}
+\date{02/06/2021}
+
+\begin{document}
+\maketitle
+
+\textbf{Turn in date: until 16/06/2021 at 23:59h Brasilia Time.}
+
+\begin{center}
+\fbox{\fbox{\parbox{1.0\textwidth}{\textsf{
+    \begin{itemize}
+    \item Please read through the whole exam before starting to answer;
+    \item State and prove all non-trivial mathematical results necessary to substantiate your arguments;
+    \item Do not forget to add appropriate scholarly references~\textit{at the end} of the document;
+    \item Mathematical expressions also receive punctuation;
+    \item You can write your answer to a question as a point-by-point response or in ``essay'' form, your call;
+    \item Please hand in a single, \textbf{typeset} ( \LaTeX) PDF file as your final main document.
+ Code appendices are welcome,~\textit{in addition} to the main PDF document.
+    \item You may consult any sources, provided you cite \textbf{ALL} of your sources (books, papers, blog posts, videos);
+    \item You may use symbolic algebra programs such as Sympy or Wolfram Alpha to help you get through the hairier calculations, provided you cite the tools you have used.
+    \item The exam is worth 100 %$\min\left\{\text{your\:score}, 100\right\}$
+    marks.
+    \end{itemize}}
+}}}
+\end{center}
+% \newpage
+% \section*{Hints}
+% \begin{itemize}
+%  \item a
+%  \item b
+% \end{itemize}
+% 
+\newpage
+
+\section*{Background}
+
+This exam covers applications, namely estimation, prior sensitivity and prediction.
+You will need a working knowledge of basic computing tools, and knowledge of MCMC is highly valuable.
+Chapter 6 in \cite{Robert2007} gives an overview of computational techniques for Bayesian statistics.
+
+\section*{Inferring population sizes -- theory}
+
+Consider the model
+\begin{equation*}
+ x_i \sim \operatorname{Binomial}(N, \theta),
+\end{equation*}
+with \textbf{both} $N$ and $\theta$ unknown and suppose one observes $\boldsymbol{x} = \{x_1, x_2, \ldots, x_K\}$.
+Here, we will write $\xi = (N, \theta)$.
+
+\begin{enumerate}[label=\alph*)]
+ \item (10 marks) Formulate a hierarchical prior ($\pi_1$) for $N$, i.e., elicit $F$ such that $N \mid \alpha \sim F(\alpha)$ and $\alpha  \sim \Pi_A$.
+ Justify your choice; 
+ \item (5 marks) Using the prior from the previous item, write out the full joint posterior kernel for all unknown quantities in the model, $p_1(\xi \mid \boldsymbol{x})$. \textit{Hint:} do not forget to include the appropriate indicator functions!;
+ \item (5 marks) Is your model identifiable?
+ \item (5 marks) Exhibit the marginal posterior density for $N$, $p_1(N \mid \boldsymbol{x})$;
+ \item (5 marks) Return to point (a) above and consider an alternative, uninformative prior structure for $\xi$, $\pi_2$.
+ Then, derive $p_2(N \mid \boldsymbol{x})$;
+ \item (10 marks) Formulate a third prior structure on $\xi$, $\pi_3$, that allows for the closed-form marginalisation over the hyperparameters $\alpha$ -- see (a) -- and write out $p_3(N \mid \boldsymbol{x})$;
+ \item (10 marks) Show whether each of the marginal posteriors considered is proper.
+ Then, derive the posterior predictive distribution, $g_i(\tilde{x} \mid \boldsymbol{x})$, for each of the posteriors considered ($i = 1, 2, 3$).
+ \item (5 marks) Consider the loss function
+ \begin{equation}
+ \label{eq:relative_loss}
+  L(\delta(\boldsymbol{x}), N) = \left(\frac{\delta(\boldsymbol{x})-N}{N} \right)^2.
+ \end{equation}
+ Derive the Bayes estimator under this loss.
+\end{enumerate}
+
+\section*{Inferring population sizes -- practice}
+Consider the problem of inferring the population sizes of major herbivores~\citep{Carroll1985}.
+In the first case, one is interested in estimating the number of impala (\textit{Aepyceros melampus}) herds in the Kruger National Park, in northeastern South Africa.
+In an initial survey collected the following numbers of herds: $\boldsymbol{x}_{\text{impala}} = \{15, 20, 21, 23, 26\}$.
+Another scientific question is the number of individual waterbuck (\textit{Kobus ellipsiprymnus}) in the same park.
+The observed numbers of waterbuck in separate sightings were $\boldsymbol{x}_{\text{waterbuck}} = \{53, 57, 66, 67, 72\}$ and may be regarded (for simplicity) as independent and identically distributed.
+
+\begin{figure}[H]
+     \centering
+     \begin{subfigure}[b]{0.45\textwidth}
+         \centering
+         \includegraphics[scale=0.75]{figures/impala.jpeg}
+         \caption{Impala}
+     \end{subfigure}
+     \begin{subfigure}[b]{0.45\textwidth}
+         \centering
+         \includegraphics[scale=0.75]{figures/waterbuck.jpeg}
+         \caption{Waterbuck}
+     \end{subfigure}
+        \caption{Two antelope species whose population sizes we want to estimate.}
+        \label{fig:antelopes}
+\end{figure}
+
+
+\begin{enumerate}[label=\alph*)]
+\setcounter{enumi}{8}
+ \item (20 marks) For each data set, sketch the marginal posterior distributions $p_1(N \mid \boldsymbol{x})$, $p_2(N \mid \boldsymbol{x})$ and $p_3(N \mid \boldsymbol{x})$.
+ Moreover, under each posterior,  provide (i) the Bayes estimator under quadratic loss and under the loss in (\ref{eq:relative_loss}) and (ii) a 95\% credibility interval for $N$.
+ Discuss the differences and similarities between these distributions and estimates: do the prior modelling choices substantially impact the final inferences? If so, how?
+ \item (25 marks) Let $\bar{x} = K^{-1}\sum_{k =1}^K x_k$ and $s^2 = K^{-1}\sum_{k =1}^K (x_k-\bar{x})^2$.
+ For this problem, a sample is said to be \textit{stable} if $\bar{x}/s^2 \geq (\sqrt{2} + 1)/\sqrt{2}$ and \textit{unstable} otherwise.
+ Devise a simple method of moments estimator (MME) for $N$.
+ Then, using a Monte Carlo simulation, compare the MME to the three Bayes estimators under quadratic loss  in terms of relative mean squared error. 
+ How do the Bayes estimators compare to MME in terms of the statibility of the generated samples? 
+ \textit{Hint}: You may want to follow the simulation setup of~\cite{Carroll1985}. 
+\end{enumerate}
+
+\bibliographystyle{apalike}
+\bibliography{a2}
+\end{document}          

assignments/2021_A1_solutions.pdf

@@ -0,0 +1,134 @@
+\documentclass[a4paper,10pt, notitlepage]{report}
+\usepackage[utf8]{inputenc}
+\usepackage{natbib}
+\usepackage{amssymb}
+\usepackage{amsmath}
+\usepackage{enumitem}
+\usepackage{dsfont}
+\usepackage{xcolor}
+\usepackage{url}
+\usepackage{cancel}
+\usepackage{mathtools}
+\usepackage{newclude}
+\usepackage{booktabs}
+\usepackage[normalem]{ulem}
+
+%%%%%%%%%%%%%%%%%%%% Notation stuff
+\newcommand{\pr}{\operatorname{Pr}} %% probability
+\newcommand{\vr}{\operatorname{Var}} %% variance
+\newcommand{\rs}{X_1, X_2, \ldots, X_n} %%  random sample
+\newcommand{\irs}{X_1, X_2, \ldots} %% infinite random sample
+\newcommand{\rsd}{x_1, x_2, \ldots, x_n} %%  random sample, realised
+\newcommand{\bX}{\boldsymbol{X}} %%  random sample, contracted form (bold)
+\newcommand{\bx}{\boldsymbol{x}} %%  random sample, realised, contracted form (bold)
+\newcommand{\bT}{\boldsymbol{T}} %%  Statistic, vector form (bold)
+\newcommand{\bt}{\boldsymbol{t}} %%  Statistic, realised, vector form (bold)
+\newcommand{\emv}{\hat{\theta}}
+\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
+\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
+\DeclareMathOperator*{\argmax}{arg\,max}
+\DeclareMathOperator*{\argmin}{arg\,min}
+
+\DeclareRobustCommand{\bbone}{\text{\usefont{U}{bbold}{m}{n}1}}
+\DeclareMathOperator{\EX}{\mathbb{E}} %% Expected Value
+
+%%%%
+\newif\ifanswers
+\answerstrue % comment out to hide answers
+
+% Title Page
+\title{Fist exam (A1)}
+\author{Class: Bayesian Statistics \\ Instructor: Luiz Max Carvalho \\ TA: Isaque Pim}
+\date{22 May 2024}
+
+\begin{document}
+\maketitle
+
+\begin{center}
+\fbox{\fbox{\parbox{1.0\textwidth}{\textsf{
+    \begin{itemize}
+        \item You have 4 (four) hours to complete the exam;
+        \item Please read through the whole exam before you start giving your answers;
+        \item Answer all questions briefly;
+        \item Clealy mark your final answer with a square, circle or preferred geometric figure;
+        \item The exam is worth $\min\left\{\text{your\:score}, 100\right\}$ marks.
+        \item You can bring \textbf{\underline{one} ``cheat sheet''} A4 both sides, which must be turned in together with your answers.
+    \end{itemize}}
+}}}
+\end{center}
+
+\newpage
+
+\section*{1. I like 'em short.}
+
+For a prior distribution $\pi$, a set $C_x$ is said to be an
+$\alpha$-credible set if $$P^\pi (\theta \in C_x |x) \geq 1-\alpha.$$
+This region is called an HPD $\alpha$-credible region (for highest posterior density) if it can be written in the form:
+\begin{equation*}
+    \{\theta; \pi(\theta|x) > k_{\alpha}\} \subset C_x^\pi \subset \{\theta; \pi(\theta|x) \geq k_{\alpha}\},
+\end{equation*}
+where $k_{\alpha}$ is the largest bound such that
+$P^\pi (\theta \in C_x^\alpha |x) \geq 1-\alpha$.
+This construction is motivated by the fact that they minimise the volume among $\alpha$-credible regions.
+A special and important case are \textit{HPD intervals}, when $C_x$ is an interval $(a, b)$. 
+
+\begin{enumerate}[label=\alph*)]
+ \item (20 marks) Show that if the posterior density  (i) is unimodal and (ii) never uniform for all intervals of ($1 - \alpha$) probability mass of $\Omega$, then the HPD region is an interval and it is unique.
+ 
+ \textbf{Hint:} formulate a minimisation problem on two variables $a$ and $b$ with a probability restriction and solve for the Lagrangian.
+
+ \item (20 marks) We can also use decision-theoretical criteria to pick between credible intervals.
+ A first idea is to balance between the volume of the region and coverage guarantees through the loss function $$L(\theta, C) = \operatorname{vol}(C) + \mathds{1}_{C^c}(\theta).$$
+ Explain why the above loss is problematic.
+ \item * (20 bonus marks) Define the new loss function $$L^*(\theta, C) = g\left(\operatorname{vol}(C)\right) + \mathds{1}_{C^c}(\theta),$$
+ where $g$ is increasing and $0 \leq g(t) \leq 1$ for all $t$. Show that the Bayes estimator $C^\pi_x$ for $L^*$ is a HPD region. 
+\end{enumerate}
+\ifanswers
+\nocite{*}
+\include*{sol1}
+\fi
+
+\section*{2. Savage!} 
+
+We will now study the case of point hypothesis testing as a case of two nested models.
+Let $\theta_0 \in \Omega_0 \subset \Omega$.
+We want to compare model $M_0: \theta = \theta_0$ to $M_1: \theta \in \Omega$.
+That is, under model $M_1$, $\theta$ can vary freely.
+Assume further that the models are \textit{properly nested}, that is, 
+$$P(x | \theta, M_0) = P(x | \theta = \theta_0, M_1).$$
+
+\begin{enumerate}[label=\alph*)] 
+ \item (25 marks)  Given observed data $x$, show that the Bayes Factor $\operatorname{BF_{01}}$ can be written as
+    \begin{equation*}
+        \operatorname{BF_{01}} = \frac{p(\theta_0| x, M_1)}{p(\theta_0|M_1)},
+    \end{equation*}
+    where the numerator is the posterior under $M_1$ and the denominator the prior probability under $M_1$.
+  \item (25 marks) Apply the result from part (a) to the problem of testing whether a coin is fair.
+    Specifically, we want to compare $H_0: \theta = 0.5$ against $H_1: \theta \neq 0.5$, where theta is the probability of the coin landing heads.
+    Given $n=24$ trials and $x = 3$ heads and employing a uniform prior on $\theta$, calculate the Bayes factor $\operatorname{BF_{01}}$.
+    Based on the Bayes factor, would you prefer $H_0$ over $H_1$? How strong should the prior be for a change in this preference?
+\end{enumerate}
+\textbf{Note}: The ratio above is called the \textit{Savage-Dickey} ratio. It provides a straightforward way to compute Bayes factors, which can be more intuitive and less computationally intensive than other methods.
+\ifanswers
+\include*{sol2}
+\fi
+
+\section*{3. Hey, you're biased! }
+
+Let $\bX = (\rs)$ be a random sample from an $\operatorname{Exponential}(\theta)$ distribution with $\theta > 0$ and common density $f(x \mid \theta) = \theta^{-1}\exp(-x/\theta)\mathbb{I}(x > 0)$ w.r.t. the Lebesgue measure on $\mathbb{R}$.
+
+\begin{enumerate}[label=\alph*)]
+ \item (10 marks) Find a conjugate prior for $\theta$;
+ \item (20 marks)  Exhibit the Bayes estimator under quadratic loss for $\theta$, $\delta_B(\bX)$;
+ \item (10 marks) Show that the bias $\delta_B(\bX)$ is $O(n^{-1})$.
+ \item $\ast$ (10 bonus marks) Show how to obtain the uniformly minimum variance unbiased estimator (UMVUE) from $\delta_B(\bX)$ by taking limits of the hyperparameters.
+ \end{enumerate}
+
+\ifanswers
+\include*{sol3}
+\fi
+
+\bibliographystyle{apalike}
+\bibliography{refs}
+
+\end{document}          

assignments/2021_A2.pdf

@@ -0,0 +1,22 @@
+@article{Dickey1971,
+  title={The weighted likelihood ratio, linear hypotheses on normal location parameters},
+  author={Dickey, James M},
+  journal={The Annals of Mathematical Statistics},
+  pages={204--223},
+  year={1971},
+  publisher={JSTOR}
+}
+
+@book{Shao2003,
+  title={Mathematical Statistics},
+  author={Shao, Jun},
+  year={2003},
+  publisher={Springer Science \& Business Media}
+}
+@book{Robert2007,
+  title={The Bayesian choice: from decision-theoretic foundations to computational implementation},
+  author={Robert, Christian P and others},
+  volume={2},
+  year={2007},
+  publisher={Springer}
+}