Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems

Chengwen Du; Tao Chen

doi:10.1145/3674805.3686673

Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems

Chengwen Du, Tao Chen^*

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

40 Downloads (Pure)

Abstract

Background: Fairness testing for deep learning systems has been becoming increasingly important. However, much work assumes perfect context and conditions from the other parts: well-tuned hyperparameters for accuracy; rectified bias in data, and mitigated bias in the labeling. Yet, these are often difficult to achieve in practice due to their resource-/labour-intensive nature.

Aims: In this paper, we aim to understand how varying contexts affect fairness testing outcomes.

Method: We conduct an extensive empirical study, which covers cases, to investigate how contexts can change the fairness testing result at the model level against the existing assumptions. We also study why the outcomes were observed from the lens of correlation/fitness landscape analysis.

Results: Our results show that different context types and settings generally lead to a significant impact on the testing, which is mainly caused by the shifts of the fitness landscape under varying contexts.

Conclusions: Our findings provide key insights for practitioners to evaluate the test generators and hint at future research directions.

Original language	English
Title of host publication	ESEM '24
Subtitle of host publication	Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
Editors	Xavier Franch, Maya Daneva, Silverio Martínez-Fernández, Luigi Quaranta
Publisher	Association for Computing Machinery (ACM)
Pages	107-118
Number of pages	12
ISBN (Print)	9798400710476
DOIs	https://doi.org/10.1145/3674805.3686673
Publication status	Published - 24 Oct 2024
Event	18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement - Barcelona, Spain Duration: 20 Oct 2024 → 25 Oct 2024

Exhibition

Exhibition	18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
Abbreviated title	ESEM 2024
Country/Territory	Spain
City	Barcelona
Period	20/10/24 → 25/10/24

Keywords

Fairness Testing
DNN Testing
Software Engineering for AI

Access to Document

10.1145/3674805.3686673Licence: Creative Commons: Attribution-ShareAlike (CC BY-SA)

DuC2024ContextsFinal published version, 3.42 MBLicence: Creative Commons: Attribution-ShareAlike (CC BY-SA)

Cite this

Du, C., & Chen, T. (2024). Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems. In X. Franch, M. Daneva, S. Martínez-Fernández, & L. Quaranta (Eds.), ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 107-118). Association for Computing Machinery (ACM). https://doi.org/10.1145/3674805.3686673

Du, Chengwen ; Chen, Tao. / Contexts Matter : An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems. ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. editor / Xavier Franch ; Maya Daneva ; Silverio Martínez-Fernández ; Luigi Quaranta. Association for Computing Machinery (ACM), 2024. pp. 107-118

@inproceedings{285aa067adf34332b406a3e0de8db3be,

title = "Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems",

abstract = "Background: Fairness testing for deep learning systems has been becoming increasingly important. However, much work assumes perfect context and conditions from the other parts: well-tuned hyperparameters for accuracy; rectified bias in data, and mitigated bias in the labeling. Yet, these are often difficult to achieve in practice due to their resource-/labour-intensive nature.Aims: In this paper, we aim to understand how varying contexts affect fairness testing outcomes.Method: We conduct an extensive empirical study, which covers cases, to investigate how contexts can change the fairness testing result at the model level against the existing assumptions. We also study why the outcomes were observed from the lens of correlation/fitness landscape analysis.Results: Our results show that different context types and settings generally lead to a significant impact on the testing, which is mainly caused by the shifts of the fitness landscape under varying contexts.Conclusions: Our findings provide key insights for practitioners to evaluate the test generators and hint at future research directions.",

keywords = "Fairness Testing, DNN Testing, Software Engineering for AI",

author = "Chengwen Du and Tao Chen",

year = "2024",

month = oct,

day = "24",

doi = "10.1145/3674805.3686673",

language = "English",

isbn = "9798400710476",

pages = "107--118",

editor = "Xavier Franch and Maya Daneva and Silverio Mart{\'i}nez-Fern{\'a}ndez and Luigi Quaranta",

booktitle = "ESEM '24",

publisher = "Association for Computing Machinery (ACM)",

address = "United States",

note = "18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024 ; Conference date: 20-10-2024 Through 25-10-2024",

}

Du, C & Chen, T 2024, Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems. in X Franch, M Daneva, S Martínez-Fernández & L Quaranta (eds), ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery (ACM), pp. 107-118, 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Barcelona, Spain, 20/10/24. https://doi.org/10.1145/3674805.3686673

Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems. / Du, Chengwen; Chen, Tao.
ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ed. / Xavier Franch; Maya Daneva; Silverio Martínez-Fernández; Luigi Quaranta. Association for Computing Machinery (ACM), 2024. p. 107-118.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Contexts Matter

T2 - 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

AU - Du, Chengwen

AU - Chen, Tao

PY - 2024/10/24

Y1 - 2024/10/24

N2 - Background: Fairness testing for deep learning systems has been becoming increasingly important. However, much work assumes perfect context and conditions from the other parts: well-tuned hyperparameters for accuracy; rectified bias in data, and mitigated bias in the labeling. Yet, these are often difficult to achieve in practice due to their resource-/labour-intensive nature.Aims: In this paper, we aim to understand how varying contexts affect fairness testing outcomes.Method: We conduct an extensive empirical study, which covers cases, to investigate how contexts can change the fairness testing result at the model level against the existing assumptions. We also study why the outcomes were observed from the lens of correlation/fitness landscape analysis.Results: Our results show that different context types and settings generally lead to a significant impact on the testing, which is mainly caused by the shifts of the fitness landscape under varying contexts.Conclusions: Our findings provide key insights for practitioners to evaluate the test generators and hint at future research directions.

AB - Background: Fairness testing for deep learning systems has been becoming increasingly important. However, much work assumes perfect context and conditions from the other parts: well-tuned hyperparameters for accuracy; rectified bias in data, and mitigated bias in the labeling. Yet, these are often difficult to achieve in practice due to their resource-/labour-intensive nature.Aims: In this paper, we aim to understand how varying contexts affect fairness testing outcomes.Method: We conduct an extensive empirical study, which covers cases, to investigate how contexts can change the fairness testing result at the model level against the existing assumptions. We also study why the outcomes were observed from the lens of correlation/fitness landscape analysis.Results: Our results show that different context types and settings generally lead to a significant impact on the testing, which is mainly caused by the shifts of the fitness landscape under varying contexts.Conclusions: Our findings provide key insights for practitioners to evaluate the test generators and hint at future research directions.

KW - Fairness Testing

KW - DNN Testing

KW - Software Engineering for AI

U2 - 10.1145/3674805.3686673

DO - 10.1145/3674805.3686673

M3 - Conference contribution

SN - 9798400710476

SP - 107

EP - 118

BT - ESEM '24

A2 - Franch, Xavier

A2 - Daneva, Maya

A2 - Martínez-Fernández, Silverio

A2 - Quaranta, Luigi

PB - Association for Computing Machinery (ACM)

Y2 - 20 October 2024 through 25 October 2024

ER -

Du C, Chen T. Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems. In Franch X, Daneva M, Martínez-Fernández S, Quaranta L, editors, ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery (ACM). 2024. p. 107-118 doi: 10.1145/3674805.3686673

Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systems

Abstract

Exhibition

Keywords

Access to Document

Fingerprint

Cite this