Quantifying the use of domain randomization

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Standard

Quantifying the use of domain randomization. / Ani, Mohammad; Basevi, Hector; Leonardis, Ales.

25th International Conference on Pattern Recognition (ICPR 2020). 2020.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Harvard

Ani, M, Basevi, H & Leonardis, A 2020, Quantifying the use of domain randomization. in 25th International Conference on Pattern Recognition (ICPR 2020). 25th International Conference on Pattern Recognition, Milan, Italy, 10/01/21.

APA

Ani, M., Basevi, H., & Leonardis, A. (Accepted/In press). Quantifying the use of domain randomization. In 25th International Conference on Pattern Recognition (ICPR 2020)

Vancouver

Ani M, Basevi H, Leonardis A. Quantifying the use of domain randomization. In 25th International Conference on Pattern Recognition (ICPR 2020). 2020

Author

Ani, Mohammad ; Basevi, Hector ; Leonardis, Ales. / Quantifying the use of domain randomization. 25th International Conference on Pattern Recognition (ICPR 2020). 2020.

Bibtex

@inproceedings{c289cf4310c747598e208a909b306db4,
title = "Quantifying the use of domain randomization",
abstract = "Synthetic image generation provides the ability to efficiently produce large quantities of labeled data, which addresses both the data volume requirements of state-of-the-art vision systems and the expense of manually labeling data. However, systems trained on synthetic data typically under-perform systems trained on realistic data due to mismatch between the synthetic and realistic data distributions. Domain Randomization (DR) is a method of broadening a synthetic data distribution to encompass a realistic data distribution and provides better performance when the exact characteristics of the realistic data distribution are not known or cannot be simulated. However, there is no consensus in the literature on the best method of performing DR. We propose a novel method of ranking DR methods by directly measuring the difference between realistic and DR data distributions. This avoids the need to measure task-specific performance and the associated expense of training and evaluation. We compare different methods for measuring distribution differences, including the Wasserstein and Fr{\'e}chet Inception distances. We also examine the effect of performing this evaluation directly on images and features generated by an image classification backbone. Finally, we show that the ranking generated by our method is reflected in actual task performance.",
author = "Mohammad Ani and Hector Basevi and Ales Leonardis",
year = "2020",
month = dec,
day = "10",
language = "English",
booktitle = "25th International Conference on Pattern Recognition (ICPR 2020)",
note = "25th International Conference on Pattern Recognition, ICPR 2020 ; Conference date: 10-01-2021 Through 15-01-2021",
url = "https://www.micc.unifi.it/icpr2020/",

}

RIS

TY - GEN

T1 - Quantifying the use of domain randomization

AU - Ani, Mohammad

AU - Basevi, Hector

AU - Leonardis, Ales

PY - 2020/12/10

Y1 - 2020/12/10

N2 - Synthetic image generation provides the ability to efficiently produce large quantities of labeled data, which addresses both the data volume requirements of state-of-the-art vision systems and the expense of manually labeling data. However, systems trained on synthetic data typically under-perform systems trained on realistic data due to mismatch between the synthetic and realistic data distributions. Domain Randomization (DR) is a method of broadening a synthetic data distribution to encompass a realistic data distribution and provides better performance when the exact characteristics of the realistic data distribution are not known or cannot be simulated. However, there is no consensus in the literature on the best method of performing DR. We propose a novel method of ranking DR methods by directly measuring the difference between realistic and DR data distributions. This avoids the need to measure task-specific performance and the associated expense of training and evaluation. We compare different methods for measuring distribution differences, including the Wasserstein and Fréchet Inception distances. We also examine the effect of performing this evaluation directly on images and features generated by an image classification backbone. Finally, we show that the ranking generated by our method is reflected in actual task performance.

AB - Synthetic image generation provides the ability to efficiently produce large quantities of labeled data, which addresses both the data volume requirements of state-of-the-art vision systems and the expense of manually labeling data. However, systems trained on synthetic data typically under-perform systems trained on realistic data due to mismatch between the synthetic and realistic data distributions. Domain Randomization (DR) is a method of broadening a synthetic data distribution to encompass a realistic data distribution and provides better performance when the exact characteristics of the realistic data distribution are not known or cannot be simulated. However, there is no consensus in the literature on the best method of performing DR. We propose a novel method of ranking DR methods by directly measuring the difference between realistic and DR data distributions. This avoids the need to measure task-specific performance and the associated expense of training and evaluation. We compare different methods for measuring distribution differences, including the Wasserstein and Fréchet Inception distances. We also examine the effect of performing this evaluation directly on images and features generated by an image classification backbone. Finally, we show that the ranking generated by our method is reflected in actual task performance.

M3 - Conference contribution

BT - 25th International Conference on Pattern Recognition (ICPR 2020)

T2 - 25th International Conference on Pattern Recognition

Y2 - 10 January 2021 through 15 January 2021

ER -