EMNLP 2026 Findings 2026
Haolong Zheng, Siyin Wang, Xulin Fan, Zengrui Jin, Mark Hasegawa-Johnson
An RL post-training method that teaches auditory LLMs to better use few-shot demonstrations for low-resource speech and audio tasks.
Audio LLMs In-Context Learning
30-sec summary
FSA-GRPO combines an ASR accuracy reward with a semantic-alignment reward that encourages auditory LLMs to use relevant in-context examples. Trained only on high-resource English adult speech, it improves few-shot adaptation for children's ASR, multilingual ASR, speech translation, and audio understanding while largely preserving zero-shot performance.
Why it matters
It offers a practical way to adapt auditory LLMs when labeled in-domain data are scarce, private, or unavailable. The method outperforms direct tuning on related out-of-domain data in low-resource child ASR and transfers across both tasks and model families.
@inproceedings{fsa_grpo,
title={FSA-GRPO: Teaching Auditory LLMs to Use Few-shot Demonstrations},
author={Zheng, Haolong and Wang, Siyin and Fan, Xulin and Jin, Zengrui and Hasegawa-Johnson, Mark},
booktitle={Findings of EMNLP},
year={2026}
, address={Budapest}}
Interspeech 2026 2026
Kaimeng Jia, Minzhu Tu, Zengrui Jin, Siyin Wang, Chao Zhang
A study showing how human perceptual supervision from speech quality assessment can improve dysarthric speech severity assessment.
Speech Quality
30-sec summary
This paper investigates automatic dysarthric speech severity assessment on open-vocabulary utterances, leveraging human-annotated perceptual ratings from speech synthesis quality assessment as out-of-domain supervision.
Why it matters
It shows that perceptual annotations aligned with human listening judgments can transfer across domains, improving intelligibility and naturalness prediction for dysarthric speech with self-supervised speech encoders.
@inproceedings{sacred,
title={Augmenting Dysarthric Speech Severity Assessment with MOS Supervision},
author={Jia, Kaimeng and Tu, Minzhu and Jin, Zengrui and Wang, Siyin and Zhang, Chao},
booktitle={Proc. Interspeech},
year={2026},
address={Sydney}
}
Nature Machine Intelligence 2026 Featured
Siyin Wang*, Zengrui Jin*, Changli Tang, Qiujia Li, Bo Li, Chen Chen, Yuchen Hu, Wenyi Yu, Yixuan Li, Jimin Zhuang, Yudong Yang, Mingqiu Wang, Michael Han, Yifan Ding, Junwen Bai, Tom Ouyang, Shuo-yiin Chang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Guangzhi Sun, Zhehuai Chen, Ji Wu, Bowen Zhou, Yuxuan Wang, Tara Sainath, Yonghui Wu, Chao Zhang
A comprehensive review of LLMs for audio modalities, corperated with Google DeepMind and Nvidia.
Audio LLMs
30-sec summary
A survey of how LLMs are reshaping computer audition across four pillars: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. It frames audio as a rich source of semantic, emotional, and contextual cues for more natural machine intelligence.
Why it matters
It gives visitors a field-level map behind my research agenda: moving from task-specific audio models toward audio-native systems that can perceive, reason, generate, and interact through sound.
@article{auditory_intelligence,
title={Towards general auditory intelligence: Large multimodal models for machine listening and speaking},
author={Wang, Siyin and Jin, Zengrui and Tang, Changli and Li, Qiujia and Li, Bo and Chen, Chen and Hu, Yuchen and Yu, Wenyi and Li, Yixuan and Zhuang, Jimin and Yang, Yudong and Wang, Mingqiu and Han, Michael and Ding, Yifan and Bai, Junwen and Ouyang, Tom and Chang, Shuo-yiin and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Lu, Lu and Sun, Guangzhi and Chen, Zhehuai and Wu, Ji and Zhou, Bowen and Wang, Yuxuan and Sainath, Tara and Wu, Yonghui and Zhang, Chao},
journal={Nature Machine Intelligence},
year={2026}
}
ICLR 2026 Poster 2026 Featured
Siyin Wang*, Wenyi Yu*, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yuxuan Wang, Chao Zhang
The first end-to-end model that unifies vision, speech, text and action in a streaming full-duplex framework.
Multimodal Interaction Full-duplex
30-sec summary
ELLSA is an end-to-end full-duplex model that simultaneously perceives and generates across vision, text, speech, and action. Its SA-MoE architecture routes modalities to specialized experts while sharing a unified attention backbone.
Why it matters
It makes full-duplex multimodal interaction concrete: the same system can support dialogue/action turn-taking, speaking while acting, context-grounded visual QA, defective-instruction rejection, and action barge-ins.
@inproceedings{ellsa,
title={End-to-end listen, look, speak and act},
author={Wang, Siyin and Yu, Wenyi and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Lu, Lu and Wang, Yuxuan and Zhang, Chao},
booktitle={Proc. ICLR},
year={2026},
address={Rio de Janeiro}
}
ICML 2026 Poster 2026
Yudong Yang, Xuezhen Zhang, Zhifeng Han, Siyin Wang, Jimin Zhuang, Zengrui Jin, Jing Shao, Guangzhi Sun, Chao Zhang
A study of compositional speech-audio attacks and mitigation strategies for multimodal LLMs.
Security
30-sec summary
This work introduces SACRED-Bench, a red-teaming benchmark for audio-based attacks that combine speech overlap, multi-speaker dialogue, speech-audio mixtures, and diverse spoken instruction formats.
Why it matters
It shows that audio is not just another input modality; it is a new safety surface. SALMONN-Guard jointly inspects speech, audio, and text, reducing attack success in the benchmark and motivating audio-aware defenses.
@inproceedings{sacred,
title={Speech-audio compositional attacks on multimodal LLMs and their mitigation with SALMONN-Guard},
author={Yang, Yudong and Zhang, Xuezhen and Han, Zhifeng and Wang, Siyin and Zhuang, Jimin and Jin, Zengrui and Shao, Jing and Sun, Guangzhi and Zhang, Chao},
booktitle={Proc. ICML},
year={2026},
address={Seoul}
}
NeurIPS 2025 Poster 2025 Featured
Wenyi Yu*, Siyin Wang*, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang
A codec-free full-duplex speech LLM for simultaneous speech understanding and generation in natural conversation.
Full-duplex
30-sec summary
SALMONN-omni is a standalone, codec-free full-duplex speech LLM. Instead of injecting audio codec tokens, it uses a dynamic thinking mechanism inside the LLM backbone to decide when to listen, speak, and transition between states.
Why it matters
It tackles the core friction in spoken AI interaction: modular full-duplex systems accumulate errors, while codec-injection systems can degrade on speech. SALMONN-omni moves toward a single speech-native model for turn-taking, backchanneling, echo cancellation, and context-dependent barge-in.
@inproceedings{salmonn_omni,
title={SALMONN-omni: A standalone speech LLM without codec injection for full-duplex conversation},
author={Yu, Wenyi and Wang, Siyin and Yang, Xiaoyu and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Sun, Guangzhi and Lu, Lu and Wang, Yuxuan and Zhang, Chao},
booktitle={Proc. NeurIPS},
year={2025},
address={San Diego}
}
ACL 2025 Poster 2025 Featured
Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yu Tsao, Junichi Yamagishi, Yuxuan Wang, Chao Zhang
A dataset and benchmark for speech quality assessment with natural language reasoning and descriptive feedback.
Speech Quality
30-sec summary
QualiSpeech reframes speech quality assessment from scalar scoring to detailed natural-language evaluation. It annotates 11 low-level quality aspects and includes comments with reasoning about noise, distortion, timing, naturalness, effort, and overall quality.
Why it matters
It turns speech quality evaluation into something models can explain, not only predict. The benchmark tests whether auditory LLMs can identify degradation types and temporal characteristics in a human-readable way.
@inproceedings{qualispeech,
title={QualiSpeech: A speech quality assessment dataset with natural language reasoning and descriptions},
author={Wang, Siyin and Yu, Wenyi and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Lu, Lu and Tsao, Yu and Yamagishi, Junichi and Wang, Yuxuan and Zhang, Chao},
booktitle={Proc. ACL},
year={2025},
address={Vienna}
}
ICLR 2025 Poster 2025
Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang, Eng Siong Chng
A study showing how audio LLMs can produce descriptive and human-aligned speech quality evaluations.
Speech Quality
30-sec summary
This paper builds a natural language-based speech evaluation corpus from authentic human ratings, covering MOS, multi-dimensional quality analysis, degradation causes, and descriptive A/B comparisons.
Why it matters
It shows audio LLMs can become more human-aligned quality evaluators: the proposed ALLD approach improves MOS prediction and A/B testing while generating explanations that go beyond a single score.
@inproceedings{alld_sqa,
title={Audio large language models can be descriptive speech quality evaluators},
author={Chen, Chen and Hu, Yuchen and Wang, Siyin and Wang, Helin and Chen, Zhehuai and Zhang, Chao and Yang, Chao-Han Huck and Chng, Eng Siong},
booktitle={Proc. ICLR},
year={2025},
address={Singapore}
}
ICASSP 2025 Oral 2025 Featured
Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Chao Zhang
Using auditory LLMs to evaluate speech quality across MOS, speaker similarity, A/B testing, and natural language descriptions.
Speech Quality
30-sec summary
This work adapts auditory LLMs to multiple speech quality assessment tasks, including MOS prediction, speaker similarity, A/B testing, and natural-language description generation.
Why it matters
It is an early bridge between specialized speech quality models and general auditory LLMs, showing that one audio-language model can cover several evaluation formats that are usually handled separately.
@inproceedings{auditory_sqa,
title={Enabling auditory large language models for automatic speech quality evaluation},
author={Wang, Siyin and Yu, Wenyi and Yang, Yudong and Tang, Changli and Li, Yixuan and Zhuang, Jimin and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Sun, Guangzhi and Lu, Lu and Zhang, Chao},
booktitle={Proc. ICASSP},
year={2025},
address={Hyderabad}
}
ISCSLP 2024 Best Student Paper Candidate 2024
Siyin Wang, Chao Zhang
A method for speaker diarization with unlimited number of speakers using dynamic linear.
Speaker Diarization
30-sec summary
This paper studies speaker diarization when the number of speakers is not fixed in advance, using a dynamic linear approach to avoid a hard limit on speaker classes.
Why it matters
It captures an earlier thread in my speech research: making multi-speaker processing more flexible for realistic conversations, where systems must answer who spoke when without knowing the speaker count beforehand.
@inproceedings{dynamic_linear_diarization,
title={Speaker diarization for unlimited number of speakers using dynamic linear},
author={Wang, Siyin and Zhang, Chao},
booktitle={Proc. ISCSLP},
pages={368--372},
year={2024},
address={Beijing}
}
EMNLP 2024 Poster 2024 Featured
Siyin Wang, Huck Yang, Ji Wu, Chao Zhang
A Bayesian method for selecting in-context examples across speech, text, and visual tasks.
In-Context Learning
30-sec summary
ByCS selects in-context examples by applying Bayes' theorem in reverse: instead of only estimating output from input and examples, it uses inverse inference conditioned on the test input to judge which examples are likely to help.
Why it matters
It makes example selection less heuristic and more portable. The same idea is evaluated across speech, text, and image tasks, connecting in-context learning with multimodal robustness.
@inproceedings{bycs,
title={Bayesian example selection improves in-context learning for speech, text, and visual modalities},
author={Wang, Siyin and Yang, Chao-Han Huck and Wu, Ji and Zhang, Chao},
booktitle={Proc. EMNLP},
year={2024},
address={Miami}
}
ICASSP 2024 Oral 2024 Featured
Siyin Wang, Huck Yang, Ji Wu, Chao Zhang
The first work of speech-based in-context learning, exploring the performance of Whisper in dialect ASR.
In-Context Learning
30-sec summary
This paper tests whether Whisper can adapt from labelled speech examples placed in context. The proposed SICL method performs test-time adaptation without gradient updates and improves dialect ASR with only a few examples.
Why it matters
It opens a speech-side counterpart to text in-context learning: spoken examples can steer an ASR model toward dialect, speaker, and lexical patterns without retraining.
@inproceedings{sicl,
title={Can Whisper perform speech-based in-context learning?},
author={Wang, Siyin and Yang, Chao-Han Huck and Wu, Ji and Zhang, Chao},
booktitle={Proc. ICASSP},
year={2024},
address={Seoul}
}
No publications match the current filters.
