Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognition

September 21, 2023 · Entered Twilight · 🏛 IEEE International Conference on Acoustics, Speech, and Signal Processing

Repo contents: .gitignore, .gitmodules, CODE_OF_CONDUCT.md, CONTRIBUTING.md, Fairseq-README.md, LICENSE, README.md, docs, egs, entry.sh, entry_loop.sh, examples, fairseq, fairseq_cli, hubconf.py, pyproject.toml, scripts, setup.py, tests, train.py

Authors Chen Xu, Xiaoqian Liu, Erfeng He, Yuhao Zhang, Qianqian Dong, Tong Xiao, Jingbo Zhu, Dapeng Man, Wu Yang arXiv ID 2309.12234 Category cs.CL: Computation & Language Cross-listed eess.AS Citations 1 Venue IEEE International Conference on Acoustics, Speech, and Signal Processing Repository https://github.com/xuchennlp/S2T ⭐ 12 Last Checked 2 months ago

Abstract

In this study, we present synchronous bilingual Connectionist Temporal Classification (CTC), an innovative framework that leverages dual CTC to bridge the gaps of both modality and language in the speech translation (ST) task. Utilizing transcript and translation as concurrent objectives for CTC, our model bridges the gap between audio and text as well as between source and target languages. Building upon the recent advances in CTC application, we develop an enhanced variant, BiL-CTC+, that establishes new state-of-the-art performances on the MuST-C ST benchmarks under resource-constrained scenarios. Intriguingly, our method also yields significant improvements in speech recognition performance, revealing the effect of cross-lingual learning on transcription and demonstrating its broad applicability. The source code is available at https://github.com/xuchennlp/S2T.