Relative Efficiency and Consistency

Learning Outcomes

  • Cramér-Rao Inequality
  • Relative Efficiency
  • Consistency

Cramér-Rao Inequality

Let \(f(x_1, \ldots, x_n;\theta)\) be the joint PMF of PDF of \(X_1, \ldots,X_n\) and \(T=t(X_1,\ldots,X_n)\) be an unbiased estimator of \(\theta\). Then

\[ Var(T) \ge \frac{1}{nI(\theta)} \] If \(Var(T)=\frac{1}{nI(\theta)}\), then \(T\) is considered an efficient estimator of \(\theta\).

Example

Let \(X_1,\ldots,X_n\overset{iid}{\sim}Pois(\lambda)\), show that \(\bar X\) is an efficient estimator of \(\lambda\).

Example

Let \(X_1,\ldots,X_n\overset{iid}{\sim}N(\mu,\sigma^2)\), show that \(\bar X\) is an efficient estimator of \(\mu\).

Relative Efficiency

Given 2 unbiased estimators \(\hat\theta_1\) and \(\hat\theta_2\) of a parameter \(\theta\) , with variances \(V(\hat\theta_1)\) and \(V(\hat\theta_2)\), respectively, then the efficiency of \(\hat\theta_1\) relative to \(\hat\theta_2\) is defined as

\[ releff(\hat\theta_1,\hat\theta_2)=\frac{\hat\theta_1}{\hat\theta_2} \]

Example

Let \(X_1,\ldots,X_n\) be a random sample from a population with mean \(\mu\) and variance \(\sigma^2\).

  • \(\hat\mu_1=(X_1+X_2)/2\)
  • \(\hat\mu_2=X_1/4+\frac{\sum^{n-1}_{i=2}X_i}{2(n-2)}+X_n/4\)
  • \(\hat\mu_3=\bar X\)

Find the relative efficiency of \(\hat\mu_3\) with respect to \(\hat\mu_1\) and \(\hat\mu_2\).

Consistency

Let \(X_1,\ldots,X_n\) be a random sample from a distribution with parameter \(\theta\). The estimator \(\hat \theta\) is a consistent estimator of the \(\theta\) if

  1. \(E\{(\hat\theta-\theta)^2\}\rightarrow0\) as \(n\rightarrow \infty\)
  2. \(P(|\hat\theta-\theta|\ge \epsilon)\rightarrow0\) as \(n\rightarrow \infty\) for every \(\epsilon>0\)

Weak Law of Large Numbers

Let \(X_1,\ldots,X_n\) be iid random variables with \(E(X_i)=\mu\) and \(Var(X_i)=\sigma^2<\infty\). Let \(\bar X_n=\frac{1}{n}\sum^n_{i=1}X_i\), for every, \(\epsilon>0\),

\[ \lim_{n\rightarrow\infty} P(|\bar X-\mu|<\epsilon) = 1 \]

that is, \(\bar X_n\) converges in probability to \(\mu\).

Strong Law of Large Numbers

Let \(X_1,\ldots,X_n\) be iid random variables with \(E(X_i)=\mu\) and \(Var(X_i)=\sigma^2<\infty\). Let \(\bar X_n=\frac{1}{n}\sum^n_{i=1}X_i\), for every, \(\epsilon>0\),

\[ P(\lim_{n\rightarrow\infty} |\bar X-\mu|<\epsilon) = 1 \]

that is, \(\bar X_n\) converges almost surely to \(\mu\).