FormNet: Formatted Learning for Image Restoration

Jianbo Jiao; Wei Chih Tu; Ding Liu; Shengfeng He; Rynson W.H. Lau; Thomas S. Huang

doi:10.1109/TIP.2020.2990603

FormNet: Formatted Learning for Image Restoration

Jianbo Jiao, Wei Chih Tu, Ding Liu, Shengfeng He, Rynson W.H. Lau^*, Thomas S. Huang

^*Corresponding author for this work

Computer Science

Research output: Contribution to journal › Article › peer-review

13 Citations (Scopus)

Abstract

In this paper, we propose a deep CNN to tackle the image restoration problem by learning formatted information. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a residual formatting layer and an adversarial block to format the information to structured one, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method performs favorably against existing approaches quantitatively and qualitatively.

Original language	English
Article number	9084384
Pages (from-to)	6302-6314
Number of pages	13
Journal	IEEE Transactions on Image Processing
Volume	29
DOIs	https://doi.org/10.1109/TIP.2020.2990603
Publication status	Published - 2020

Bibliographical note

Funding Information:
Manuscript received October 20, 2018; revised December 9, 2019; accepted April 13, 2020. Date of publication May 6, 2020; date of current version May 12, 2020. This work was supported in part by the EPSRC Project See-bibyte under Grant EP/M013774/1, in part by the Hong Kong Ph.D. Fellowship Scheme from the Research Grants Council of Hong Kong, in part by the National Natural Science Foundation of China under Grant 61972129, Grant 61972162, and Grant 61702194, in part by the Guangzhou Key Industrial Technology Research Fund under Grant 201802010036, in part by the CCF-Tencent Open Research Fund under Grant CCF-Tencent RAGR20190112, and in part by the NVIDIA Corporation with the Donation of GPU Card. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jana Ehmann. (Corresponding authors: Shengfeng He; Rynson W. H. Lau.) Jianbo Jiao is with the Department of Engineering Science, University of Oxford, Oxford OX1 2JD, U.K. (e-mail: jianbo@robots.ox.ac.uk).

Publisher Copyright:
© 1992-2012 IEEE.

Keywords

CNN
format
GAN
Image restoration
residual

ASJC Scopus subject areas

Software
Computer Graphics and Computer-Aided Design

Access to Document

10.1109/TIP.2020.2990603

Cite this

@article{ed94fc464b134162aefe95d9b915d6af,

title = "FormNet: Formatted Learning for Image Restoration",

abstract = "In this paper, we propose a deep CNN to tackle the image restoration problem by learning formatted information. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a residual formatting layer and an adversarial block to format the information to structured one, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method performs favorably against existing approaches quantitatively and qualitatively.",

keywords = "CNN, format, GAN, Image restoration, residual",

author = "Jianbo Jiao and Tu, {Wei Chih} and Ding Liu and Shengfeng He and Lau, {Rynson W.H.} and Huang, {Thomas S.}",

note = "Funding Information: Manuscript received October 20, 2018; revised December 9, 2019; accepted April 13, 2020. Date of publication May 6, 2020; date of current version May 12, 2020. This work was supported in part by the EPSRC Project See-bibyte under Grant EP/M013774/1, in part by the Hong Kong Ph.D. Fellowship Scheme from the Research Grants Council of Hong Kong, in part by the National Natural Science Foundation of China under Grant 61972129, Grant 61972162, and Grant 61702194, in part by the Guangzhou Key Industrial Technology Research Fund under Grant 201802010036, in part by the CCF-Tencent Open Research Fund under Grant CCF-Tencent RAGR20190112, and in part by the NVIDIA Corporation with the Donation of GPU Card. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jana Ehmann. (Corresponding authors: Shengfeng He; Rynson W. H. Lau.) Jianbo Jiao is with the Department of Engineering Science, University of Oxford, Oxford OX1 2JD, U.K. (e-mail: jianbo@robots.ox.ac.uk). Publisher Copyright: {\textcopyright} 1992-2012 IEEE.",

year = "2020",

doi = "10.1109/TIP.2020.2990603",

language = "English",

volume = "29",

pages = "6302--6314",

journal = "IEEE Transactions on Image Processing",

issn = "1057-7149",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

}

TY - JOUR

T1 - FormNet

T2 - Formatted Learning for Image Restoration

AU - Jiao, Jianbo

AU - Tu, Wei Chih

AU - Liu, Ding

AU - He, Shengfeng

AU - Lau, Rynson W.H.

AU - Huang, Thomas S.

N1 - Funding Information: Manuscript received October 20, 2018; revised December 9, 2019; accepted April 13, 2020. Date of publication May 6, 2020; date of current version May 12, 2020. This work was supported in part by the EPSRC Project See-bibyte under Grant EP/M013774/1, in part by the Hong Kong Ph.D. Fellowship Scheme from the Research Grants Council of Hong Kong, in part by the National Natural Science Foundation of China under Grant 61972129, Grant 61972162, and Grant 61702194, in part by the Guangzhou Key Industrial Technology Research Fund under Grant 201802010036, in part by the CCF-Tencent Open Research Fund under Grant CCF-Tencent RAGR20190112, and in part by the NVIDIA Corporation with the Donation of GPU Card. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jana Ehmann. (Corresponding authors: Shengfeng He; Rynson W. H. Lau.) Jianbo Jiao is with the Department of Engineering Science, University of Oxford, Oxford OX1 2JD, U.K. (e-mail: jianbo@robots.ox.ac.uk). Publisher Copyright: © 1992-2012 IEEE.

PY - 2020

Y1 - 2020

N2 - In this paper, we propose a deep CNN to tackle the image restoration problem by learning formatted information. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a residual formatting layer and an adversarial block to format the information to structured one, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method performs favorably against existing approaches quantitatively and qualitatively.

AB - In this paper, we propose a deep CNN to tackle the image restoration problem by learning formatted information. Previous deep learning based methods directly learn the mapping from corrupted images to clean images, and may suffer from the gradient exploding/vanishing problems of deep neural networks. We propose to address the image restoration problem by learning the structured details and recovering the latent clean image together, from the shared information between the corrupted image and the latent image. In addition, instead of learning the pure difference (corruption), we propose to add a residual formatting layer and an adversarial block to format the information to structured one, which allows the network to converge faster and boosts the performance. Furthermore, we propose a cross-level loss net to ensure both pixel-level accuracy and semantic-level visual quality. Evaluations on public datasets show that the proposed method performs favorably against existing approaches quantitatively and qualitatively.

KW - CNN

KW - format

KW - GAN

KW - Image restoration

KW - residual

UR - http://www.scopus.com/inward/record.url?scp=85087513215&partnerID=8YFLogxK

U2 - 10.1109/TIP.2020.2990603

DO - 10.1109/TIP.2020.2990603

M3 - Article

AN - SCOPUS:85087513215

SN - 1057-7149

VL - 29

SP - 6302

EP - 6314

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

M1 - 9084384

ER -

FormNet: Formatted Learning for Image Restoration

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this