Image colorization by using graph bi-Laplacian

Abstract

Image colorization aims to recover the whole color image based on a known grayscale image (luminance or brightness) and some known color pixel values. In this paper, we generalize the graph Laplacian to its second-order variant called graph bi-Laplacian, and then propose an image colorization method by using graph bi-Laplacian. The eigenvalue analysis of graph bi-Laplacian matrix and its corresponding normalized bi-Laplacian matrix is given to show their properties. We apply graph bi-Laplacian approach to image colorization by formulating it as an optimization problem and solving the resulting linear system efficiently. Numerical results show that the proposed method can perform quite well for image colorization problem, and its performance in terms of efficiency and colorization quality for test images can be better than that by the state-of-the-art colorization methods when the randomly given color pixels ratio attains some level.

Appendices

Appendix 1. Proof of Proposition

Part (1)::

Denote the right-hand side term of (7) as

$$ E(u)=\sum\limits_{i}\left( \sum\limits_{j\in \mathcal{N}_{i}}w_{ij}(u_{j}-u_{i})\right)^{2}. $$

It is obvious that E(u) is a quadratic form. In E(u), the terms involving u_i are

$$ E_{1}=\left( \sum\limits_{j\in \mathcal{N}_{i}}w_{ij}(u_{i}-u_{j})\right)^{2} +\sum\limits_{j\in \mathcal{N}_{i}}\left( \sum\limits_{k\in \mathcal{N}_{j}}w_{jk}(u_{j}-u_{k})\right)^{2}. $$

Note that in the second term k can be equal to i since $i\in \mathcal {N}_{j}$. Expanding the first terms of E₁, we get

$$ \begin{array}{@{}rcl@{}} E_{11}&=&\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il}(u_{i}-u_{j})(u_{i}-u_{l})\\ &=&\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il}{u_{i}^{2}} -{\sum}_{j\in \mathcal{N}_{i}}{\sum}_{l\in \mathcal{N}_{i}}w_{ij}w_{il}(u_{i}u_{l}+u_{i}u_{j})+\cdots\\ &=&\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il}{u_{i}^{2}}- 2\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il}u_{i}u_{j}+\cdots \end{array} $$

where the terms which are not related with u_i are omitted. In the second term of E₁, u_i appears only when k = i or l = i; hence, we have

$$ \begin{array}{@{}rcl@{}} E_{12}&=&\sum\limits_{j\in \mathcal{N}_{i}}\left( \sum\limits_{k\in \mathcal{N}_{j}, k\neq i}w_{jk}(u_{j}-u_{k})+w_{ij}u_{j}-w_{ij}u_{i}\right)^{2}\\ &=&\sum\limits_{j\in \mathcal{N}_{i}}w_{ij}^{2}{u_{i}^{2}}-2w_{ij}^{2}u_{i}u_{j} -2\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j},k\neq i}w_{ij}w_{jk}u_{i}u_{j}\\ &&+ 2\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j},k\neq i}w_{ij}w_{jk}u_{i}u_{k}+\cdots\\ &=&\sum\limits_{j\in \mathcal{N}_{i}}w_{ij}^{2}{u_{i}^{2}}-\!2\!\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j}}w_{ij}w_{jk}u_{i}u_{j} + 2\!\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j},k\neq i}w_{ij}w_{jk}u_{i}u_{k}+\cdots \end{array} $$

where the terms not involving u_i are omitted. By adding E₁₁ and E₁₂ together, we get that the terms contain u_i are

$$ \begin{array}{@{}rcl@{}} &&\left( \sum\limits_{j\in \mathcal{N}_{i}}w_{ij}^{2}+\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il}\right){u_{i}^{2}}\\ &&- 2\!\left( \sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il} + \sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j}}w_{ij}w_{jk}\right)\!u_{i}u_{j} +\! 2\!\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j},k\neq i}w_{ij}w_{jk}u_{i}u_{k}. \end{array} $$

By using the definition of graph bi-Laplacian, we obtain that E(u) is the following quadratic form

$$E(u)=u^{T}{{\Delta}_{w}^{2}} u.$$

Part (2)::

According to the formulas (3)–(5), the main diagonal elements of ${{\Delta }_{w}^{2}}$ is positive since W is non-negative and connected. the summation of the i th row of the matrix ${{\Delta }_{w}^{2}}$ is

$$ \begin{array}{@{}rcl@{}}\sum\limits_{j = 1}^{n}({{\Delta}_{w}^{2}})_{ij} &=&\sum\limits_{j\in \mathcal{N}_{i}}w_{ij}^{2}+\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{l\in \mathcal{N}_{i}}w_{ij}w_{il} -\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j}}w_{ij}w_{jk}\\ &&-\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{i}}w_{ij}w_{ik} +\sum\limits_{j\in \mathcal{N}_{i}}\sum\limits_{k\in \mathcal{N}_{j}, k\neq i}w_{ij}w_{jk}.\hspace{2cm} \end{array} $$

The first, the third, and the fifth terms sum to zero; meanwhile, the second and the fourth terms sum to zero. Hence, it follows directly that the row sum is zero.

Part (3)::

Assume $j\in \mathcal {N}_{i}$. Since G is an undirected graph and W is symmetric, we have $i\in \mathcal {N}_{j}, w_{ij}=w_{ji}$. By definition (4), we get

$$({{\Delta}_{w}^{2}})_{ji}= \sum\limits_{k\in \mathcal{N}_{i}}w_{ji}w_{ik}+ \sum\limits_{k\in \mathcal{N}_{j}}w_{ji}w_{jk}=({{\Delta}_{w}^{2}})_{ij}, j\in \mathcal{N}_{i}. $$

The symmetry of $({{\Delta }_{w}^{2}})_{ik},k\in \mathcal {N}_{j}, j\in \mathcal {N}_{i}, k\neq i$ is obvious from definition (5). Hence, ${{\Delta }_{w}^{2}}$ is symmetric. The positive semi-definiteness is a direct consequence of Part (1), which shows that $u^{T}{{\Delta }_{w}^{2}} u\ge 0$ for all $u\in \mathbb {R}^{n}$.

Part (4)::

From equation (7), it is obvious that ${{\Delta }_{w}^{2}}\mathbf {1}= 0.$ Hence, 0 is an eigenvalue of ${{\Delta }_{w}^{2}}$. Since ${{\Delta }_{w}^{2}}$ is positive semi-definite, its eigenvalues are non-negative. So 0 is the smallest eigenvalue of ${{\Delta }_{w}^{2}}$.

Part (5)::

It is a direct consequence of Parts (1), (3), and (4).

Part (6)::

Assume u is an eigenvector with eigenvalue 0. Then, by equation (??), we know that

$$\sum\limits_{j\in \mathcal{N}_{i}}w_{ij}(u_{j}-u_{i})= 0, \quad \forall i. $$

Or equivalently,

$$\sum\limits_{j\in \mathcal{N}_{i}}\tilde{w}_{ij}u_{j} = u_{i}, \quad \forall i, $$

where $\tilde {w}_{ij} = \frac {w_{ij}}{{\sum }_{j\in \mathcal {N}_{i}}w_{ij}}$. Since w_ij ≥ 0, we have $\tilde {w}_{ij}\ge 0$ and ${\sum }_{j\in \mathcal {N}_{i}}\tilde {w}_{ij}= 1$. If u attains maximum u₀ at some vertex i, then we can easily derive that $u_{j}=u_{0}, \forall j\in \mathcal {N}_{i}$. Since the graph is connected, we have u_k = u₀ at each vertex k. With similar argument, if we assume u attains minimum u₀ at some vertex i, we get that u_k = u₀ at each vertex k. In other words, the eigenvector of ${{\Delta }_{w}^{2}}$ corresponding to eigenvalue 0 must be the constant one vector. Hence, the conclusion follows.

Using the symmetric property of ${{\Delta }_{w}^{2}}$, we can give another proof of Part (1). Using the notations as above, the derivative about E(u) with respect to u_i is

$$ \begin{array}{@{}rcl@{}} &&\frac{\partial E}{\partial u_{i}} = 2\left( \sum\limits_{l\in\mathcal{N}_{i}}w_{il}(u_{l}-u_{i})\right) \left( -\sum\limits_{j\in \mathcal{N}_{i}} w_{ij}\right)+ 2\sum\limits_{j\in\mathcal{N}_{i}}\left( \sum\limits_{k\in \mathcal{N}_{j}}w_{jk}(u_{k}-u_{j})\right)w_{ji}.\\ \end{array} $$

It is easy to check that

$$\frac{\partial E}{\partial u}= 2{{\Delta}_{w}^{2}}u. $$

On the other hand, since Δ_w is symmetric by definition, we have

$$ \frac{\partial}{\partial u}\left( u^{T}{{\Delta}_{w}^{2}}u\right) = {{\Delta}_{w}^{T}}u+{\Delta}_{w}u = 2{{\Delta}_{w}^{2}}u. $$

Hence, the equality (7) holds.

Appendix 2. Proof of Proposition 2

Part (1)::

It can be proved similarly as Part (1) in Proposition 2.

Part (2)::

From part (2) of Proposition (2), the row sum of ${{\Delta }_{w}^{2}}$ equals 0. That is,

$$\sum\limits_{j = 1}^{n}({{\Delta}_{w}^{2}})_{ij}=\overline{D}_{ii}-\sum\limits_{j = 1}^{n}{\overline{W}_{ij}}= 0.$$

Then

$$\sum\limits_{j = 1}^{n}(\overline{{{\Delta}_{w}^{2}}})_{ij}= 1-\frac{{\sum}_{j = 1}^{n}{\overline{W}_{ij}}}{\overline{D}_{ii}}= 0.$$

Part (3)::

Assume λ is an eigenvalue of $\overline {{{\Delta }_{w}^{2}}}$ with eigenvector v, i.e.,

$$\overline{{{\Delta}_{w}^{2}}}v = \lambda v\Leftrightarrow\overline{D}^{-1}{{{\Delta}_{w}^{2}}}v = \lambda v.$$

Multiplying this eigenvalue equation with $\overline {D}^{1/2}$ from the left yields

$$\overline{D}^{-1/2}{{{\Delta}_{w}^{2}}}v = \lambda \overline{D}^{1/2}v\Leftrightarrow\overline{D}^{-1/2}{{{\Delta}_{w}^{2}}}\overline{D}^{-1/2}\overline{D}^{1/2}v = \lambda \overline{D}^{1/2}v.$$

Letting $\omega =\overline {D}^{1/2}v$, we get $\overline {\overline {{{\Delta }_{w}^{2}}}}\omega =\lambda \omega$ and vice versa. Part (2) implies that the eigenvalues of $\overline {\overline {{{\Delta }_{w}^{2}}}}$ and $\overline {{{\Delta }_{w}^{2}}}$ are the same.

Part (4)::

By Part (1), it is obvious that $\overline {{{\Delta }_{w}^{2}}}\mathbf {1}= 0$.

Part (5)::

The statement about $\overline {\overline {{{\Delta }_{w}^{2}}}}$ can be derived from (1) and then the statement about $\overline {{{\Delta }_{w}^{2}}}$ follows from Part (3).

Part (6)::

Assume u is an eigenvector with eigenvalue 0. Then, by equation (8), we know that

$$\sum\limits_{j\in \mathcal{N}_{i}}w_{ij}\left( \frac{u_{i}}{\sqrt{\overline{D}_{ii}}} -\frac{u_{j}}{\sqrt{\overline{D}_{jj}}}\right)= 0, \quad \forall i.$$

Since the graph is connected, with a similar argument as the proof of Part (6) in Proposition 1, we get the following equality hold at all vertices k

$$\frac{u_{k}}{\sqrt{\overline{D}_{kk}}}=\text{const}, \quad \forall k.$$

It means the eigenvector of $\overline {\overline {{{\Delta }_{w}^{2}}}}$ with eigenvalue 0 must be $\overline {D}^{1/2}\mathbf {1}$. Hence, the statement of $\overline {\overline {{{\Delta }_{w}^{2}}}}$ holds. The statement for $\overline {{{\Delta }_{w}^{2}}}= 0$ holds by Part (2)

Appendix 3. Proof of Lemma 1

Assume x is a nonzero vector such that A_KKx = 0. That is, $D_{K}A{D_{k}^{T}}x = 0$. Then, we have $x^{T}D_{K}A{D_{k}^{T}}x = 0$. Denote $y={D_{K}^{T}}x$. For the symmetric matrix A, we construct the polynomial

$$ \begin{array}{@{}rcl@{}} p(t)&=&(ty+Ay)^{T}A(ty+Ay)\\ &=&t^{2}y^{T}Ay+ 2ty^{T}A^{2}y+y^{T}A^{3}y\\ &=&2t\|Ay\|^{2}_{2}+y^{T}A^{3}y. \end{array} $$

The positive semi-definite property of A ensures that p(t) ≥ 0 for all real t. However, if ∥Ay∥₂ ≠ 0 then for sufficiently large negative values of t we would have p(t) < 0. We conclude that ∥Ay∥₂ = 0, so Ay = 0, i.e., $A{D_{k}^{T}}x = 0$. By proposition 1, the eigenvector of A corresponding to eigenvalue 0 is constant one vector. Then, we have ${D_{K}^{T}}x =\mathbf {1}$. It can not hold since ${D_{K}^{T}}x$ must contain zeros due to the fact that D_K is a subsampling matrix.

Appendix 4. Proof of Theorem 1

Note that $\tilde {A}$ is a block upper triangular with diagonal blocks A_KK and I. Hence, $\det {(\tilde {A})}=\det {(A_{KK})}\det {(I)}=\det {(A_{KK})}\neq 0$, which implies the uniqueness of solution.