Fisher信息量学习笔记

本文记录了Fisher 信息量的两种等价计算方式.

说明

本文积分以$\int$表示$\int_{-\infty}^{+\infty}$ (主要原因是懒,请谅解.)

Fisher 信息量

定义

设总体概率密度函数$p(x;\theta),\theta \in \mathbb{\Theta}$满足下列条件:

1. 参数空间$\Theta$是直线上的一个开区间;
2. 支撑$S=\{x|p(x;\theta)\}$与$\theta$无关;
3. 导数$\dfrac{\partial }{\partial \theta}p(x;\theta)$对一切$\theta \in \Theta$都存在;
4. 对$p(x;\theta)$,积分与微分可以交换顺序,即:

$$\dfrac{\partial}{\partial \theta}\int p(x;\theta)\,\mathrm{d}x=\int \dfrac{\partial}{\partial \theta}p(x;\theta)\,\mathrm{d}x$$

5. 期望$E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]^2$存在.

则称

$$I(\theta)=E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]^2$$

为总体分布的Fisher 信息量.

另一种表示法

Fisher信息量$I(\theta)$也可以表示为

$$I(\theta)=-E\left[\dfrac{\partial ^2}{\partial \theta ^2}\ln p(x;\theta)\right]$$

有时候这种计算方法更加方便

证明

首先,有:

$$\begin{equation} E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]=0 \end{equation}$$

这是因为:

$$\begin{aligned} &E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]\\ =&\int \dfrac{\partial \ln p(x;\theta)}{\partial \theta} p(x;\theta)\, \mathrm{d}x \\ =&\int \dfrac{1}{p(x;\theta)}\dfrac{\partial p(x;\theta)}{\partial \theta} p(x;\theta)\, \mathrm{d}x\\ =&\int \dfrac{\partial p(x;\theta)}{\partial \theta} \, \mathrm{d}x\\ =&\dfrac{\partial }{\partial \theta}\int p(x;\theta)\, \mathrm{d}x\\ =&0 \end{aligned}$$

然后,有

$$\begin{equation} \dfrac{\partial }{\partial \theta}E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]=0 \end{equation}$$

又因为:

$$\dfrac{\partial \ln p(x;\theta)}{\partial \theta}=\frac{1}{p(x;\theta)}\frac{\partial p(x;\theta)}{\partial \theta}$$

所以

$$\begin{equation} \frac{\partial p(x;\theta)}{\partial \theta}=p(x;\theta) \dfrac{\partial \ln p(x;\theta)}{\partial \theta} \end{equation}$$

故,有:

$$\begin{aligned} &\dfrac{\partial }{\partial \theta}E\left[\dfrac{\partial}{\partial \theta}\ln p(x;\theta)\right]\\ =& \dfrac{\partial }{\partial \theta}\int \dfrac{\partial \ln p(x;\theta)}{\partial \theta} p(x;\theta) \, \mathrm{d}x\\ =& \int \dfrac{\partial }{\partial \theta}\left(\dfrac{\partial \ln p(x;\theta)}{\partial \theta} p(x;\theta)\right) \, \mathrm{d}x\\ =& \int \left(\dfrac{\partial p(x;\theta) }{\partial \theta}\dfrac{\partial \ln p(x;\theta)}{\partial \theta}+ p(x;\theta)\dfrac{\partial^2 \ln p(x;\theta)}{\partial \theta^2}\right) \, \mathrm{d}x\\ =& \int p(x;\theta) \left(\dfrac{\partial \ln p(x;\theta)}{\partial \theta}\right)^2\, \mathrm{d}x + \int p(x;\theta) \dfrac{\partial^2 \ln p(x;\theta)}{\partial \theta^2}\,\mathrm{d}x\\ =&E\left[\left(\dfrac{\partial \ln p(x;\theta)}{\partial \theta}\right)^2\right]+E\left[\dfrac{\partial^2 \ln p(x;\theta)}{\partial \theta^2}\right]\\ =&0 \end{aligned}$$

所以

$$E\left[\left(\dfrac{\partial \ln p(x;\theta)}{\partial \theta}\right)^2\right]=-E\left[\dfrac{\partial^2 \ln p(x;\theta)}{\partial \theta^2}\right]$$

即:

$$I(\theta)=-E\left[\dfrac{\partial^2 \ln p(x;\theta)}{\partial \theta^2}\right]$$