Demo page for Improve bilingual TTS using language and phonology embedding with embedding strength modulator

Improve bilingual TTS using language and phonology embedding with embedding strength modulator

Abstract: In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology with embedding masks. An embedding strength modulator is specially designed to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.

1. Comparing the speech naturalness and speaker similarity of different bilingual systems:

English:

1.1. Drag the bars to adjust the volume level.
BASE	EM	ESM

1.2. It is mostly detected during childhood and student years itself.
BASE	EM	ESM

1.3. In other words, my employer is always telling me to do things.
BASE	EM	ESM

1.4. No food was available at the Press Club.
BASE	EM	ESM

1.5. Therefore, one cartridge case of this type was not recovered.
BASE	EM	ESM

Code-switching:

1.6. 最良心的新款SUV！
BASE	EM	ESM

1.7. Something Just Like This 1分钟试听版送给你。
BASE	EM	ESM

1.8. 好的，小爱想陪你一起听，绿钻会员歌曲：See You Again。
BASE	EM	ESM

Mandarin:

1.9. 下午啦，快醒醒，要开始工作啦。
BASE	EM	ESM

1.10. 好的，小爱帮你找回记忆中的旋律。
BASE	EM	ESM

1.11. 抱歉，查询有失败，请稍后再试吧。
BASE	EM	ESM

2. ESM component analysis:

2.1. Children this age fight to get into the bathroom.
Base combination	Reference combination	Dynamic phonology embedding	Static phonology embedding	Dynamic language embedding	Static language embedding
		(Articulation)	(Intonation)	(Speaking rate)	(Pause duration)

2.2. Next, the pores plug up and trap the oil inside.
Base combination	Reference combination	Dynamic phonology embedding	Static phonology embedding	Dynamic language embedding	Static language embedding
		(Articulation)	(Intonation)	(Speaking rate)	(Pause duration)

3. Control phonology for enhancing expressiveness:

3.1. Genes of intracellular calcium metabolism and blood pressure control in primary hypertension.
Base combination	Reference combination	Double Adjustment

3.2. The American people have learned from the depression.
Base combination	Reference combination	Double Adjustment

4. Control phonology for smooth transition:

4.1. 好呀，我们一起来听I fall in love too easily。
Base combination	Smooth transition	Reference combination

4.2. 好的，为你播放抖音神曲twinkle twinkle little star。
Base combination	Smooth transition	Reference combination