Audio-visual speech recognition with a hybrid ctc/attention architecture


Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, Maja Pantic


Audias-UAM, Universidad Autonoma de Madrid, Madrid, Spain
CRIM, Montreal (Quebec), Canada
Speechlab, Shanghai Jiao Tong University, China
Omilia – Conversational Intelligence, Athens, Greece
Brno University of Technology, Speech@FIT and IT4I Center of Excellence, Brno, Czechia
Phonexia, Czechia

Publication Date

Desember 18, 2018

Recent works in speech recognition rely either on connectionist temporal classification (CTC) or sequence-to-sequence models for character-level recognition. CTC assumes conditional independence of individual characters, whereas attention-based models can provide nonsequential alignments. Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption. In this paper, we use the recently proposed hybrid CTC/attention architecture for audio-visual recognition of speech in-the-wild. To the best of our knowledge, this is the first time that such a hybrid architecture architecture is used for audio-visual recognition of speech. We use the LRS2 database and show that the proposed audio-visual model leads to an 1.3% absolute decrease in word error rate over the audio-only model and achieves the new state-of-the-art performance on LRS2 database (7% word error rate). We also observe that the audio-visual model significantly outperforms the audio-based model (up to 32.9% absolute improvement in word error rate) for several different types of noise as the signal-to-noise ratio decreases.