Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss

Jianbo Jiao; Ying Cao; Yibing Song; Rynson Lau

doi:10.1007/978-3-030-01267-0_4

Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss

Jianbo Jiao^*, Ying Cao, Yibing Song, Rynson Lau

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

129 Citations (Scopus)

Abstract

Monocular depth estimation benefits greatly from learning based techniques. By studying the training data, we observe that the per-pixel depth values in existing datasets typically exhibit a long-tailed distribution. However, most previous approaches treat all the regions in the training data equally regardless of the imbalanced depth distribution, which restricts the model performance particularly on distant depth regions. In this paper, we investigate the long tail property and delve deeper into the distant depth regions (i.e. the tail part) to propose an attention-driven loss for the network supervision. In addition, to better leverage the semantic information for monocular depth estimation, we propose a synergy network to automatically learn the information sharing strategies between the two tasks. With the proposed attention-driven loss and synergy network, the depth estimation and semantic labeling tasks can be mutually improved. Experiments on the challenging indoor dataset show that the proposed approach achieves state-of-the-art performance on both monocular depth estimation and semantic labeling tasks.

Original language	English
Title of host publication	Computer Vision – ECCV 2018
Subtitle of host publication	15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV
Editors	Yair Weiss, Vittorio Ferrari, Cristian Sminchisescu, Martial Hebert
Publisher	Springer
Pages	55-71
Number of pages	17
Edition	1
ISBN (Electronic)	9783030012670
ISBN (Print)	9783030012663
DOIs	https://doi.org/10.1007/978-3-030-01267-0_4
Publication status	Published - 7 Oct 2018
Event	15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany Duration: 8 Sept 2018 → 14 Sept 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11219
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	15th European Conference on Computer Vision, ECCV 2018
Country/Territory	Germany
City	Munich
Period	8/09/18 → 14/09/18

Bibliographical note

Funding Information:
Acknowledgments. This work is partially supported by the Hong Kong PhD Fellowship Scheme (HKPFS) from the RGC of Hong Kong.

Publisher Copyright:
© Springer Nature Switzerland AG 2018.

Keywords

Attention loss
Monocular depth
Semantic labeling

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-030-01267-0_4

Cite this

Jiao, J., Cao, Y., Song, Y., & Lau, R. (2018). Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss. In Y. Weiss, V. Ferrari, C. Sminchisescu, & M. Hebert (Eds.), Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV (1 ed., pp. 55-71). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11219). Springer. https://doi.org/10.1007/978-3-030-01267-0_4

Jiao, Jianbo ; Cao, Ying ; Song, Yibing et al. / Look deeper into depth : monocular depth estimation with semantic booster and attention-driven loss. Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV. editor / Yair Weiss ; Vittorio Ferrari ; Cristian Sminchisescu ; Martial Hebert. 1. ed. Springer, 2018. pp. 55-71 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{01e57ca8c6924db1875101013230eb30,

title = "Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss",

abstract = "Monocular depth estimation benefits greatly from learning based techniques. By studying the training data, we observe that the per-pixel depth values in existing datasets typically exhibit a long-tailed distribution. However, most previous approaches treat all the regions in the training data equally regardless of the imbalanced depth distribution, which restricts the model performance particularly on distant depth regions. In this paper, we investigate the long tail property and delve deeper into the distant depth regions (i.e. the tail part) to propose an attention-driven loss for the network supervision. In addition, to better leverage the semantic information for monocular depth estimation, we propose a synergy network to automatically learn the information sharing strategies between the two tasks. With the proposed attention-driven loss and synergy network, the depth estimation and semantic labeling tasks can be mutually improved. Experiments on the challenging indoor dataset show that the proposed approach achieves state-of-the-art performance on both monocular depth estimation and semantic labeling tasks.",

keywords = "Attention loss, Monocular depth, Semantic labeling",

author = "Jianbo Jiao and Ying Cao and Yibing Song and Rynson Lau",

note = "Funding Information: Acknowledgments. This work is partially supported by the Hong Kong PhD Fellowship Scheme (HKPFS) from the RGC of Hong Kong. Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2018.; 15th European Conference on Computer Vision, ECCV 2018 ; Conference date: 08-09-2018 Through 14-09-2018",

year = "2018",

month = oct,

day = "7",

doi = "10.1007/978-3-030-01267-0_4",

language = "English",

isbn = "9783030012663",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "55--71",

editor = "Yair Weiss and Vittorio Ferrari and Cristian Sminchisescu and Martial Hebert",

booktitle = "Computer Vision – ECCV 2018",

edition = "1",

}

Jiao, J, Cao, Y, Song, Y & Lau, R 2018, Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss. in Y Weiss, V Ferrari, C Sminchisescu & M Hebert (eds), Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV. 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11219, Springer, pp. 55-71, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 8/09/18. https://doi.org/10.1007/978-3-030-01267-0_4

Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss. / Jiao, Jianbo; Cao, Ying; Song, Yibing et al.
Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV. ed. / Yair Weiss; Vittorio Ferrari; Cristian Sminchisescu; Martial Hebert. 1. ed. Springer, 2018. p. 55-71 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11219).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Look deeper into depth

T2 - 15th European Conference on Computer Vision, ECCV 2018

AU - Jiao, Jianbo

AU - Cao, Ying

AU - Song, Yibing

AU - Lau, Rynson

N1 - Funding Information: Acknowledgments. This work is partially supported by the Hong Kong PhD Fellowship Scheme (HKPFS) from the RGC of Hong Kong. Publisher Copyright: © Springer Nature Switzerland AG 2018.

PY - 2018/10/7

Y1 - 2018/10/7

N2 - Monocular depth estimation benefits greatly from learning based techniques. By studying the training data, we observe that the per-pixel depth values in existing datasets typically exhibit a long-tailed distribution. However, most previous approaches treat all the regions in the training data equally regardless of the imbalanced depth distribution, which restricts the model performance particularly on distant depth regions. In this paper, we investigate the long tail property and delve deeper into the distant depth regions (i.e. the tail part) to propose an attention-driven loss for the network supervision. In addition, to better leverage the semantic information for monocular depth estimation, we propose a synergy network to automatically learn the information sharing strategies between the two tasks. With the proposed attention-driven loss and synergy network, the depth estimation and semantic labeling tasks can be mutually improved. Experiments on the challenging indoor dataset show that the proposed approach achieves state-of-the-art performance on both monocular depth estimation and semantic labeling tasks.

AB - Monocular depth estimation benefits greatly from learning based techniques. By studying the training data, we observe that the per-pixel depth values in existing datasets typically exhibit a long-tailed distribution. However, most previous approaches treat all the regions in the training data equally regardless of the imbalanced depth distribution, which restricts the model performance particularly on distant depth regions. In this paper, we investigate the long tail property and delve deeper into the distant depth regions (i.e. the tail part) to propose an attention-driven loss for the network supervision. In addition, to better leverage the semantic information for monocular depth estimation, we propose a synergy network to automatically learn the information sharing strategies between the two tasks. With the proposed attention-driven loss and synergy network, the depth estimation and semantic labeling tasks can be mutually improved. Experiments on the challenging indoor dataset show that the proposed approach achieves state-of-the-art performance on both monocular depth estimation and semantic labeling tasks.

KW - Attention loss

KW - Monocular depth

KW - Semantic labeling

UR - http://www.scopus.com/inward/record.url?scp=85055419750&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01267-0_4

DO - 10.1007/978-3-030-01267-0_4

M3 - Conference contribution

SN - 9783030012663

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 55

EP - 71

BT - Computer Vision – ECCV 2018

A2 - Weiss, Yair

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

A2 - Hebert, Martial

PB - Springer

Y2 - 8 September 2018 through 14 September 2018

ER -

Jiao J, Cao Y, Song Y, Lau R. Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss. In Weiss Y, Ferrari V, Sminchisescu C, Hebert M, editors, Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV. 1 ed. Springer. 2018. p. 55-71. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2018 Oct 6. doi: 10.1007/978-3-030-01267-0_4

Look deeper into depth: monocular depth estimation with semantic booster and attention-driven loss

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this