Retaining Semantics in Image to Music Conversion

Zeyu Xiong; Pei Chun Lin; Amin Farjudian

doi:10.1109/ISM55400.2022.00051

Retaining Semantics in Image to Music Conversion

Zeyu Xiong^*, Pei Chun Lin, Amin Farjudian

^*Corresponding author for this work

Mathematics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We propose a method for generating music from a given image through three stages of translation, from image to caption, caption to lyrics, and lyrics to instrumental music, which forms the content to be combined with a given style. We train our proposed model, which we call BGT (BLIP-GPT2-TeleMelody), on two open-source datasets, one containing over 200,000 labeled images, and another containing more than 175,000 MIDI music files. In contrast with pixel level translation, the BGT model retains the semantics of the input image. We verify our claim through a user study in which participants were asked to match input images with generated music without access to the intermediate caption and lyrics. The results show that, while the matching rate among participants with low music expertise is essentially random, the rate among those with composition experience is significantly high, which strongly indicates that some semantic content of the input image is retained in the generated music.

Original language	English
Title of host publication	2022 IEEE International Symposium on Multimedia (ISM)
Publisher	IEEE
Pages	228-235
Number of pages	8
ISBN (Electronic)	9781665471725
ISBN (Print)	9781665471732
DOIs	https://doi.org/10.1109/ISM55400.2022.00051
Publication status	Published - 23 Jan 2023
Event	24th IEEE International Symposium on Multimedia, ISM 2022 - Virtual, Online, Italy Duration: 5 Dec 2022 → 7 Dec 2022

Publication series

Name	IEEE International Symposium on Multimedia
Publisher	IEEE
ISSN (Electronic)	2766-0001

Conference

Conference	24th IEEE International Symposium on Multimedia, ISM 2022
Country/Territory	Italy
City	Virtual, Online
Period	5/12/22 → 7/12/22

Bibliographical note

Funding Information:
This research was supported by the Ministry of Education, R.O.C., under the grant TEEP@AsiaPlus, the Ministry of Science and Technology, R.O.C., under the grant No. MOST 109-2221-E-035-063-MY2, and by Feng Chia University under the 2022 Project Research Grant.

Publisher Copyright:
© 2022 IEEE.

Keywords

machine learning
media composition
media semantics

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Signal Processing
Media Technology

Access to Document

10.1109/ISM55400.2022.00051

Cite this

@inproceedings{511d49d9bca0494ebe18af62d3321c6d,

title = "Retaining Semantics in Image to Music Conversion",

abstract = "We propose a method for generating music from a given image through three stages of translation, from image to caption, caption to lyrics, and lyrics to instrumental music, which forms the content to be combined with a given style. We train our proposed model, which we call BGT (BLIP-GPT2-TeleMelody), on two open-source datasets, one containing over 200,000 labeled images, and another containing more than 175,000 MIDI music files. In contrast with pixel level translation, the BGT model retains the semantics of the input image. We verify our claim through a user study in which participants were asked to match input images with generated music without access to the intermediate caption and lyrics. The results show that, while the matching rate among participants with low music expertise is essentially random, the rate among those with composition experience is significantly high, which strongly indicates that some semantic content of the input image is retained in the generated music.",

keywords = "machine learning, media composition, media semantics",

author = "Zeyu Xiong and Lin, {Pei Chun} and Amin Farjudian",

note = "Funding Information: This research was supported by the Ministry of Education, R.O.C., under the grant TEEP@AsiaPlus, the Ministry of Science and Technology, R.O.C., under the grant No. MOST 109-2221-E-035-063-MY2, and by Feng Chia University under the 2022 Project Research Grant. Publisher Copyright: {\textcopyright} 2022 IEEE.; 24th IEEE International Symposium on Multimedia, ISM 2022 ; Conference date: 05-12-2022 Through 07-12-2022",

year = "2023",

month = jan,

day = "23",

doi = "10.1109/ISM55400.2022.00051",

language = "English",

isbn = "9781665471732",

series = "IEEE International Symposium on Multimedia",

publisher = "IEEE",

pages = "228--235",

booktitle = "2022 IEEE International Symposium on Multimedia (ISM)",

}

TY - GEN

T1 - Retaining Semantics in Image to Music Conversion

AU - Xiong, Zeyu

AU - Lin, Pei Chun

AU - Farjudian, Amin

N1 - Funding Information: This research was supported by the Ministry of Education, R.O.C., under the grant TEEP@AsiaPlus, the Ministry of Science and Technology, R.O.C., under the grant No. MOST 109-2221-E-035-063-MY2, and by Feng Chia University under the 2022 Project Research Grant. Publisher Copyright: © 2022 IEEE.

PY - 2023/1/23

Y1 - 2023/1/23

N2 - We propose a method for generating music from a given image through three stages of translation, from image to caption, caption to lyrics, and lyrics to instrumental music, which forms the content to be combined with a given style. We train our proposed model, which we call BGT (BLIP-GPT2-TeleMelody), on two open-source datasets, one containing over 200,000 labeled images, and another containing more than 175,000 MIDI music files. In contrast with pixel level translation, the BGT model retains the semantics of the input image. We verify our claim through a user study in which participants were asked to match input images with generated music without access to the intermediate caption and lyrics. The results show that, while the matching rate among participants with low music expertise is essentially random, the rate among those with composition experience is significantly high, which strongly indicates that some semantic content of the input image is retained in the generated music.

AB - We propose a method for generating music from a given image through three stages of translation, from image to caption, caption to lyrics, and lyrics to instrumental music, which forms the content to be combined with a given style. We train our proposed model, which we call BGT (BLIP-GPT2-TeleMelody), on two open-source datasets, one containing over 200,000 labeled images, and another containing more than 175,000 MIDI music files. In contrast with pixel level translation, the BGT model retains the semantics of the input image. We verify our claim through a user study in which participants were asked to match input images with generated music without access to the intermediate caption and lyrics. The results show that, while the matching rate among participants with low music expertise is essentially random, the rate among those with composition experience is significantly high, which strongly indicates that some semantic content of the input image is retained in the generated music.

KW - machine learning

KW - media composition

KW - media semantics

UR - http://www.scopus.com/inward/record.url?scp=85147542391&partnerID=8YFLogxK

U2 - 10.1109/ISM55400.2022.00051

DO - 10.1109/ISM55400.2022.00051

M3 - Conference contribution

AN - SCOPUS:85147542391

SN - 9781665471732

T3 - IEEE International Symposium on Multimedia

SP - 228

EP - 235

BT - 2022 IEEE International Symposium on Multimedia (ISM)

PB - IEEE

T2 - 24th IEEE International Symposium on Multimedia, ISM 2022

Y2 - 5 December 2022 through 7 December 2022

ER -

Retaining Semantics in Image to Music Conversion

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this