The Csiszár-Kullback inequality

Today we give an inequality, which is very useful in proving the convergence of a solution to PDEs tends to the stationary solution as time tends to infinity.

Lemma. Let \Omega\in\mathbb R^d be a domain and let f, g\in L^1(\Omega) satisfy f\geq 0, g>0 and \int_\Omega fdx = \int_\Omega gdx = 1. Furthermore, let \Phi\in C^1(\mathbb R) satisfy

\Phi(s) \geq \Phi(1) + \Phi'(1)(s-1) + \gamma^2(s-1)^21_{\{s<1\}}

for all s\in\mathbb R and some \gamma>0, where 1_{\{s<1\}} is the characteristic function on A\subset \mathbb R. Finally, let

H_\Phi[f] = \int_\Omega \Phi\left(\dfrac{f}{g}\right)gdx.


\|f - g\|^2_{L^1(\Omega)} \leq \dfrac{4}{\gamma^2}(H_\Phi[f] - H_\Phi[g])

Proof. First, we try to estimate H_\Phi[f] - H_\Phi[g].

H_\Phi[f] - H_\Phi[g] = \int_{\Omega}(\Phi(f/g) - \Phi(1))gdx

\geq \int_{\Omega}(\Phi'(1)(f/g-1) + \gamma^2(f/g-1)^21_{\{f/g<1\}})gdx

= \Phi'(1)\int_{\Omega}(f - g)dx + \gamma^2\int_{f<g}(f/g-1)^2gdx

= \gamma^2\int_{f<g}(f/g-1)^2gdx (since \int_\Omega fdx = \int_\Omega gdx =1)

Now, it is sufficient to prove that

\gamma^2\int_{f<g}(f/g-1)^2gdx \geq \dfrac{\gamma^2}{4}\|f-g\|^2_{L^1(\Omega)}

or equivalently

4\int_{f<g}(f/g-1)^2gdx \geq \|f-g\|^2_{L^1(\Omega)}

We have

\|f - g\|_{L^1(\Omega)} = \int_{\Omega}|f-g|dx

= \int_{f<g}(g-f)dx + \int_{f\geq g}(f-g)dx

= \int_{f<g}(g-f)dx + \int_{f\geq g}fdx - \int_{f\geq g}gdx

= \int_{f<g}(g-f)dx +(\int_{\Omega}fdx - \int_{f<g}fdx) - (\int_{\Omega}gdx - \int_{f<g}gdx)

= 2\int_{f<g}(g-f)dx (since \int_\Omega fdx = \int_\Omega gdx = 1).

We now have to show that

\int_{f<g}(f/g-1)^2gdx \geq (\int_{f<g}(g-f)dx)^2.

Using Hölder’s inequality and \int_\Omega gdx = 1, we have

\int_{f<g}(g-f)dx = \int_{f<g}(1-f/g)gdx

\leq \int_{f<g}|(1-f/g)\sqrt{g}||\sqrt{g}|dx

\leq \left(\int_{f<g}|(1-f/g)^2g|dx\right)^{1/2}\left(\int_\Omega gdx\right)^{1/2}

= \left(\int_{f<g}|(1-f/g)^2g|dx\right)^{1/2}


(\int_{f<g}(g-f)dx)^2 \leq \int_{f<g}|(1-f/g)^2g|dx.

This completes the proof.


About baotangquoc

Lecturer School of Applied Mathematics and Informatics Hanoi University of Science and Technology No 1, Dai Co Viet Street, Hanoi
This entry was posted in Entropy. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s