LDRNet: Enabling Real-Time Document Localization on Mobile Devices

Han Wu; Holland Qian; Huaming Wu; Aad van Moorsel

doi:10.1007/978-3-031-23618-1_42

LDRNet: Enabling Real-Time Document Localization on Mobile Devices

Han Wu
, Holland Qian
, Huaming Wu^*
, Aad van Moorsel

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Modern online services often require mobile devices to convert paper-based information into its digital counterpart, e.g., passport, ownership documents, etc. This process relies on Document Localization (DL) technology to detect the outline of a document within a photograph. In recent years, increased demand for real-time DL in live video has emerged, especially in financial services. However, existing machine-learning approaches to DL cannot be easily applied due to the large size of the underlying models and the associated long inference time. In this paper, we propose a lightweight DL model, LDRNet, to localize documents in real-time video captured on mobile devices. On the basis of a lightweight backbone neural network, we design three prediction branches for LDRNet: (1) corner points prediction; (2) line borders prediction and (3) document classification. To improve the accuracy, we design novel supplementary targets, the equal-division points, and use a new loss function named Line Loss. We compare the performance of LDRNet with other popular approaches on localization for general documents in a number of datasets. The experimental results show that LDRNet takes significantly less inference time, while still achieving comparable accuracy.

Original language	English
Title of host publication	Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Subtitle of host publication	International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I
Editors	Irena Koprinska, Paolo Mignone, Riccardo Guidotti, Szymon Jaroszewicz, Holger Fröning, Francesco Gullo, Pedro M. Ferreira, Damian Roqueiro, Gaia Ceddia, Slawomir Nowaczyk, João Gama, Rita Ribeiro, Ricard Gavaldà, Elio Masciari, Zbigniew Ras, Ettore Ritacco, Francesca Naretto, Andreas Theissler, Przemyslaw Biecek, Wouter Verbeke, Gregor Schiele, Franz Pernkopf, Michaela Blott, Ilaria Bordino, Ivan Luciano Danesi, Giovanni Ponti, Lorenzo Severini, Annalisa Appice, Giuseppina Andresini, Ibéria Medeiros, Guilherme Graça, Lee Cooper, Naghmeh Ghazaleh, Jonas Richiardi, Diego Saldana, Konstantinos Sechidis, Arif Canakoglu, Sara Pido, Pietro Pinoli, Albert Bifet, Sepideh Pashami
Publisher	Springer
Pages	618-629
Number of pages	12
Edition	1
ISBN (Electronic)	9783031236181
ISBN (Print)	9783031236174
DOIs	https://doi.org/10.1007/978-3-031-23618-1_42
Publication status	Published - 31 Jan 2023
Event	Workshops on SoGood, NFMCP, XKDD, UMOD, ITEM, MIDAS, MLCS, MLBEM, PharML, DALS, IoT-PdM 2022, held in conjunction with the 21st Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2022 - Grenoble, France Duration: 19 Sept 2022 → 23 Sept 2022

Publication series

Name	Communications in Computer and Information Science
Publisher	Springer
Volume	1752
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	Workshops on SoGood, NFMCP, XKDD, UMOD, ITEM, MIDAS, MLCS, MLBEM, PharML, DALS, IoT-PdM 2022, held in conjunction with the 21st Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2022
Country/Territory	France
City	Grenoble
Period	19/09/22 → 23/09/22

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

Document localization
Mobile devices
Real time

ASJC Scopus subject areas

General Computer Science
General Mathematics

Access to Document

10.1007/978-3-031-23618-1_42

Cite this

Wu, H., Qian, H., Wu, H., & van Moorsel, A. (2023). LDRNet: Enabling Real-Time Document Localization on Mobile Devices. In I. Koprinska, P. Mignone, R. Guidotti, S. Jaroszewicz, H. Fröning, F. Gullo, P. M. Ferreira, D. Roqueiro, G. Ceddia, S. Nowaczyk, J. Gama, R. Ribeiro, R. Gavaldà, E. Masciari, Z. Ras, E. Ritacco, F. Naretto, A. Theissler, P. Biecek, W. Verbeke, G. Schiele, F. Pernkopf, M. Blott, I. Bordino, I. L. Danesi, G. Ponti, L. Severini, A. Appice, G. Andresini, I. Medeiros, G. Graça, L. Cooper, N. Ghazaleh, J. Richiardi, D. Saldana, K. Sechidis, A. Canakoglu, S. Pido, P. Pinoli, A. Bifet, ... S. Pashami (Eds.), Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I (1 ed., pp. 618-629). (Communications in Computer and Information Science; Vol. 1752). Springer. https://doi.org/10.1007/978-3-031-23618-1_42

Wu, Han ; Qian, Holland ; Wu, Huaming et al. / LDRNet : Enabling Real-Time Document Localization on Mobile Devices. Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I. editor / Irena Koprinska ; Paolo Mignone ; Riccardo Guidotti ; Szymon Jaroszewicz ; Holger Fröning ; Francesco Gullo ; Pedro M. Ferreira ; Damian Roqueiro ; Gaia Ceddia ; Slawomir Nowaczyk ; João Gama ; Rita Ribeiro ; Ricard Gavaldà ; Elio Masciari ; Zbigniew Ras ; Ettore Ritacco ; Francesca Naretto ; Andreas Theissler ; Przemyslaw Biecek ; Wouter Verbeke ; Gregor Schiele ; Franz Pernkopf ; Michaela Blott ; Ilaria Bordino ; Ivan Luciano Danesi ; Giovanni Ponti ; Lorenzo Severini ; Annalisa Appice ; Giuseppina Andresini ; Ibéria Medeiros ; Guilherme Graça ; Lee Cooper ; Naghmeh Ghazaleh ; Jonas Richiardi ; Diego Saldana ; Konstantinos Sechidis ; Arif Canakoglu ; Sara Pido ; Pietro Pinoli ; Albert Bifet ; Sepideh Pashami. 1. ed. Springer, 2023. pp. 618-629 (Communications in Computer and Information Science).

@inproceedings{004ace738cbd436a9ec26065f3d1ce10,

title = "LDRNet: Enabling Real-Time Document Localization on Mobile Devices",

abstract = "Modern online services often require mobile devices to convert paper-based information into its digital counterpart, e.g., passport, ownership documents, etc. This process relies on Document Localization (DL) technology to detect the outline of a document within a photograph. In recent years, increased demand for real-time DL in live video has emerged, especially in financial services. However, existing machine-learning approaches to DL cannot be easily applied due to the large size of the underlying models and the associated long inference time. In this paper, we propose a lightweight DL model, LDRNet, to localize documents in real-time video captured on mobile devices. On the basis of a lightweight backbone neural network, we design three prediction branches for LDRNet: (1) corner points prediction; (2) line borders prediction and (3) document classification. To improve the accuracy, we design novel supplementary targets, the equal-division points, and use a new loss function named Line Loss. We compare the performance of LDRNet with other popular approaches on localization for general documents in a number of datasets. The experimental results show that LDRNet takes significantly less inference time, while still achieving comparable accuracy.",

keywords = "Document localization, Mobile devices, Real time",

author = "Han Wu and Holland Qian and Huaming Wu and \{van Moorsel\}, Aad",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Workshops on SoGood, NFMCP, XKDD, UMOD, ITEM, MIDAS, MLCS, MLBEM, PharML, DALS, IoT-PdM 2022, held in conjunction with the 21st Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2022 ; Conference date: 19-09-2022 Through 23-09-2022",

year = "2023",

month = jan,

day = "31",

doi = "10.1007/978-3-031-23618-1\_42",

language = "English",

isbn = "9783031236174",

series = "Communications in Computer and Information Science",

publisher = "Springer",

pages = "618--629",

editor = "Irena Koprinska and Paolo Mignone and Riccardo Guidotti and Szymon Jaroszewicz and Holger Fr{\"o}ning and Francesco Gullo and Ferreira, \{Pedro M.\} and Damian Roqueiro and Gaia Ceddia and Slawomir Nowaczyk and Jo{\~a}o Gama and Rita Ribeiro and Ricard Gavald{\`a} and Elio Masciari and Zbigniew Ras and Ettore Ritacco and Francesca Naretto and Andreas Theissler and Przemyslaw Biecek and Wouter Verbeke and Gregor Schiele and Franz Pernkopf and Michaela Blott and Ilaria Bordino and Danesi, \{Ivan Luciano\} and Giovanni Ponti and Lorenzo Severini and Annalisa Appice and Giuseppina Andresini and Ib{\'e}ria Medeiros and Guilherme Gra{\c c}a and Lee Cooper and Naghmeh Ghazaleh and Jonas Richiardi and Diego Saldana and Konstantinos Sechidis and Arif Canakoglu and Sara Pido and Pietro Pinoli and Albert Bifet and Sepideh Pashami",

booktitle = "Machine Learning and Principles and Practice of Knowledge Discovery in Databases",

edition = "1",

}

Wu, H, Qian, H, Wu, H & van Moorsel, A 2023, LDRNet: Enabling Real-Time Document Localization on Mobile Devices. in I Koprinska, P Mignone, R Guidotti, S Jaroszewicz, H Fröning, F Gullo, PM Ferreira, D Roqueiro, G Ceddia, S Nowaczyk, J Gama, R Ribeiro, R Gavaldà, E Masciari, Z Ras, E Ritacco, F Naretto, A Theissler, P Biecek, W Verbeke, G Schiele, F Pernkopf, M Blott, I Bordino, IL Danesi, G Ponti, L Severini, A Appice, G Andresini, I Medeiros, G Graça, L Cooper, N Ghazaleh, J Richiardi, D Saldana, K Sechidis, A Canakoglu, S Pido, P Pinoli, A Bifet & S Pashami (eds), Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I. 1 edn, Communications in Computer and Information Science, vol. 1752, Springer, pp. 618-629, Workshops on SoGood, NFMCP, XKDD, UMOD, ITEM, MIDAS, MLCS, MLBEM, PharML, DALS, IoT-PdM 2022, held in conjunction with the 21st Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2022, Grenoble, France, 19/09/22. https://doi.org/10.1007/978-3-031-23618-1_42

LDRNet: Enabling Real-Time Document Localization on Mobile Devices. / Wu, Han; Qian, Holland; Wu, Huaming et al.
Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I. ed. / Irena Koprinska; Paolo Mignone; Riccardo Guidotti; Szymon Jaroszewicz; Holger Fröning; Francesco Gullo; Pedro M. Ferreira; Damian Roqueiro; Gaia Ceddia; Slawomir Nowaczyk; João Gama; Rita Ribeiro; Ricard Gavaldà; Elio Masciari; Zbigniew Ras; Ettore Ritacco; Francesca Naretto; Andreas Theissler; Przemyslaw Biecek; Wouter Verbeke; Gregor Schiele; Franz Pernkopf; Michaela Blott; Ilaria Bordino; Ivan Luciano Danesi; Giovanni Ponti; Lorenzo Severini; Annalisa Appice; Giuseppina Andresini; Ibéria Medeiros; Guilherme Graça; Lee Cooper; Naghmeh Ghazaleh; Jonas Richiardi; Diego Saldana; Konstantinos Sechidis; Arif Canakoglu; Sara Pido; Pietro Pinoli; Albert Bifet; Sepideh Pashami. 1. ed. Springer, 2023. p. 618-629 (Communications in Computer and Information Science; Vol. 1752).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - LDRNet

T2 - Workshops on SoGood, NFMCP, XKDD, UMOD, ITEM, MIDAS, MLCS, MLBEM, PharML, DALS, IoT-PdM 2022, held in conjunction with the 21st Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2022

AU - Wu, Han

AU - Qian, Holland

AU - Wu, Huaming

AU - van Moorsel, Aad

PY - 2023/1/31

Y1 - 2023/1/31

N2 - Modern online services often require mobile devices to convert paper-based information into its digital counterpart, e.g., passport, ownership documents, etc. This process relies on Document Localization (DL) technology to detect the outline of a document within a photograph. In recent years, increased demand for real-time DL in live video has emerged, especially in financial services. However, existing machine-learning approaches to DL cannot be easily applied due to the large size of the underlying models and the associated long inference time. In this paper, we propose a lightweight DL model, LDRNet, to localize documents in real-time video captured on mobile devices. On the basis of a lightweight backbone neural network, we design three prediction branches for LDRNet: (1) corner points prediction; (2) line borders prediction and (3) document classification. To improve the accuracy, we design novel supplementary targets, the equal-division points, and use a new loss function named Line Loss. We compare the performance of LDRNet with other popular approaches on localization for general documents in a number of datasets. The experimental results show that LDRNet takes significantly less inference time, while still achieving comparable accuracy.

AB - Modern online services often require mobile devices to convert paper-based information into its digital counterpart, e.g., passport, ownership documents, etc. This process relies on Document Localization (DL) technology to detect the outline of a document within a photograph. In recent years, increased demand for real-time DL in live video has emerged, especially in financial services. However, existing machine-learning approaches to DL cannot be easily applied due to the large size of the underlying models and the associated long inference time. In this paper, we propose a lightweight DL model, LDRNet, to localize documents in real-time video captured on mobile devices. On the basis of a lightweight backbone neural network, we design three prediction branches for LDRNet: (1) corner points prediction; (2) line borders prediction and (3) document classification. To improve the accuracy, we design novel supplementary targets, the equal-division points, and use a new loss function named Line Loss. We compare the performance of LDRNet with other popular approaches on localization for general documents in a number of datasets. The experimental results show that LDRNet takes significantly less inference time, while still achieving comparable accuracy.

KW - Document localization

KW - Mobile devices

KW - Real time

UR - https://www.scopus.com/pages/publications/85149864827

U2 - 10.1007/978-3-031-23618-1_42

DO - 10.1007/978-3-031-23618-1_42

M3 - Conference contribution

AN - SCOPUS:85149864827

SN - 9783031236174

T3 - Communications in Computer and Information Science

SP - 618

EP - 629

BT - Machine Learning and Principles and Practice of Knowledge Discovery in Databases

A2 - Koprinska, Irena

A2 - Mignone, Paolo

A2 - Guidotti, Riccardo

A2 - Jaroszewicz, Szymon

A2 - Fröning, Holger

A2 - Gullo, Francesco

A2 - Ferreira, Pedro M.

A2 - Roqueiro, Damian

A2 - Ceddia, Gaia

A2 - Nowaczyk, Slawomir

A2 - Gama, João

A2 - Ribeiro, Rita

A2 - Gavaldà, Ricard

A2 - Masciari, Elio

A2 - Ras, Zbigniew

A2 - Ritacco, Ettore

A2 - Naretto, Francesca

A2 - Theissler, Andreas

A2 - Biecek, Przemyslaw

A2 - Verbeke, Wouter

A2 - Schiele, Gregor

A2 - Pernkopf, Franz

A2 - Blott, Michaela

A2 - Bordino, Ilaria

A2 - Danesi, Ivan Luciano

A2 - Ponti, Giovanni

A2 - Severini, Lorenzo

A2 - Appice, Annalisa

A2 - Andresini, Giuseppina

A2 - Medeiros, Ibéria

A2 - Graça, Guilherme

A2 - Cooper, Lee

A2 - Ghazaleh, Naghmeh

A2 - Richiardi, Jonas

A2 - Saldana, Diego

A2 - Sechidis, Konstantinos

A2 - Canakoglu, Arif

A2 - Pido, Sara

A2 - Pinoli, Pietro

A2 - Bifet, Albert

A2 - Pashami, Sepideh

PB - Springer

Y2 - 19 September 2022 through 23 September 2022

ER -

Wu H, Qian H, Wu H, van Moorsel A. LDRNet: Enabling Real-Time Document Localization on Mobile Devices. In Koprinska I, Mignone P, Guidotti R, Jaroszewicz S, Fröning H, Gullo F, Ferreira PM, Roqueiro D, Ceddia G, Nowaczyk S, Gama J, Ribeiro R, Gavaldà R, Masciari E, Ras Z, Ritacco E, Naretto F, Theissler A, Biecek P, Verbeke W, Schiele G, Pernkopf F, Blott M, Bordino I, Danesi IL, Ponti G, Severini L, Appice A, Andresini G, Medeiros I, Graça G, Cooper L, Ghazaleh N, Richiardi J, Saldana D, Sechidis K, Canakoglu A, Pido S, Pinoli P, Bifet A, Pashami S, editors, Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I. 1 ed. Springer. 2023. p. 618-629. (Communications in Computer and Information Science). Epub 2023 Jan 30. doi: 10.1007/978-3-031-23618-1_42