C³Benchmark

Challenges in Complex Conversations

Spoken Dialogue Model Evaluation

Paper

Dataset

Code

Comprehensive evaluation of Spoken Dialogue Models (SDMs) across Chinese and English dialogue data, measuring performance in phonological ambiguity, semantic understanding, omission, coreference, and multi-turn interactions.

Models Evaluated

Spoken Dialogue Models

Languages

Chinese & English

Rank	Model	Phonological	Semantic	Ambiguity	Omission	Coreference	Multi-turn	Context-dep.	Overall
🥇	GPT-4o-Audio-Prev.🔗	53.45	70.59	62.02	16.18	91.11	47.06	51.45	55.68
🥈	Qwen2.5-Omni🔗	48.28	32.35	40.31	15.20	68.15	95.59	59.64	51.91
🥉	Kimi-Audio🔗	46.55	29.41	37.98	10.29	87.41	—	48.85	43.42
#4	GLM-4-Voice🔗	27.59	15.69	21.64	6.37	68.98	58.82	44.73	35.49
#5	Step-Audio🔗	29.31	21.57	25.44	10.78	57.31	41.18	36.43	32.03
#6	VITA-Audio🔗	31.03	18.63	24.83	7.84	74.81	60.29	47.65	38.52
#7	LLaMA-Omni🔗	15.52	12.75	14.13	5.88	56.94	55.88	39.57	29.39
#8	MooER-Omni🔗	18.97	46.08	32.52	4.90	36.02	41.18	27.37	29.43
#9	Freeze-Omni🔗	8.62	11.76	10.19	6.86	47.22	44.12	32.73	23.72
#10	Moshi🔗	10.34	9.80	10.07	2.94	24.63	—	13.79	11.93

Rank	Model	Phonological	Semantic	Ambiguity	Omission	Coreference	Multi-turn	Context-dep.	Overall
🥇	Qwen2.5-Omni🔗	27.03	6.78	16.90	27.86	55.83	82.89	55.53	40.08
🥈	GPT-4o-Audio-Prev.🔗	29.73	5.93	17.83	44.29	54.17	13.16	37.20	29.45
🥉	MooER-Omni🔗	20.27	2.12	11.19	32.14	32.50	63.16	42.60	30.04
#4	Kimi-Audio🔗	20.27	4.24	12.25	29.29	40.00	—	34.64	23.45
#5	Step-Audio🔗	22.97	5.08	14.03	17.86	50.83	7.89	25.53	20.93
#6	VITA-Audio🔗	8.11	3.39	5.75	6.43	33.33	63.16	34.31	22.88
#7	GLM-4-Voice🔗	18.92	2.54	10.73	5.71	16.67	10.53	10.97	10.87
#8	Freeze-Omni🔗	16.22	1.69	8.96	4.29	10.83	11.84	8.99	8.97

Ambiguity

Overall performance on phonological and semantic ambiguity resolution tasks.

Phonological Ambiguity

Understanding of sound-based ambiguities and phonological variations in spoken language.

Semantic Ambiguity

Disambiguation of word meanings and semantic interpretations in different contexts.

Context-dependency

Understanding statements that require contextual information from dialogue history.

Omission

Ability to handle missing information and implicit context in dialogue.

Coreference

Resolution of pronouns and references to previously mentioned entities.

Multi-turn Interaction

Coherence and context maintenance across multiple conversation turns.

Examples

Phonological Ambiguity

The sentence below is hard to understand:
You're going to the party?

Could you tell me what it means?
Answer: Rising intonation indicates a question: "Are you going to the party?"
The sentence below with pause is hard to understand:
He saw the man / with glasses.

Could you tell me what it means?
Answer: The pause after "man" indicates that "with glasses" is an additional description, meaning "He saw the man, and he was wearing glasses."
Here is a sentence:
He saw the man with glasses.

The intended meaning is:
He saw the man, and he was wearing glasses.

Please read it out with the correct pause.
Answer: He saw the man / with glasses.

Semantic Ambiguity

The sentence below contains ambiguity:
Mr. Smith loves music more than his wife.

Please tell me how to understand it.
Answer: The phrase "more than his wife" is ambiguous.
1. Mr. Smith loves music more than he loves his wife.
2. Mr. Smith loves music more than his wife does.
The sentence below contains ambiguity:
Mike failed to make friends with John because he is too selfish.

Please tell me how to understand it.
Answer: The pronoun "he" is ambiguous.
1. Mike failed to make friends with John because Mike is too selfish.
2. Mike failed to make friends with John because John is too selfish.
The sentence below contains ambiguity:
She can't bear children so she never talks about them.

Please tell me how to understand it.
Answer: Ambiguity in "bear":
1. Unable to give birth to children
2. Unable to tolerate children.

Omission

Please listen carefully to the following sentence, after the sentence ends, I will ask you some questions:
Your advice made me happy but Tom angry.

Some content has been omitted in the sentence, please fill in the omitted parts and provide the complete sentence.
Answer: Your advice made me happy but your advice made Tom angry.
Please listen carefully to the following sentence, after the sentence ends, I will ask you some questions:
Your advice made me happy but Tom angry.

Is there any omission of content in the sentence?
Answer: Yes, there is.
Please listen carefully to the following sentence, after the sentence ends, I will ask you some questions:
Once a worker, Pang Long now becomes a famous singer.

Some content has been omitted in the sentence, please fill in the omitted parts and provide the complete sentence.
Answer: Once he was a worker, Pang Long now becomes a famous singer.

Coreference

Please listen carefully to the following sentence, after the sentence ends, I will ask you some questions:
The trophy doesn't fit into the brown suitcase because it is too large.

Are there any instances of pronouns being used to refer to nouns or noun phrases in the sentences above?
Answer: Yes.
The city councilmen refused the demonstrators a permit because they feared violence.

What does the word 'they' refer to: The city councilmen or The demonstrators?
Answer: The city councilmen.
Joan made sure to thank Susan for all the help she had recieved.

What does the word 'she' refer to: JoanorSusan?
Answer: Joan.

Multi-turn Interaction

Do you know anything about health food?
Can you give me some examples?
Really? Why we must have warm and fresh water?
Anything else?
Do you know anything about health food? If this question has been raised before, please directly provide the previous answer。

Phonological Ambiguity

下面有一个断句可能不正确的句子：
这个苹果不大好吃。

这个句子想表达的意思是：
这个苹果不好吃。

请用正确的断句读出这句话。
Answer: 这个苹果 / 不大好吃。
可以请你帮我一个忙吗？我不太理解下面这段话，请你帮我给出合理的解释：
没有凳子时应该站着吃饺子；有醋时应该蘸着吃饺子。
Answer: 站立的站和蘸醋的蘸发音相同，造成混淆。
可以请你帮我一个忙吗？我不太理解下面这段话，请你帮我给出合理的解释：
第一个苹果不大，好吃。第二个苹果，不大好吃。
Answer: 断句不同语义不同，第一个苹果比较小，但味道好，第二个苹果味道不好。

Semantic Ambiguity

可以请你帮我一个忙吗？我不太理解下面这段话，请你帮我给出合理的解释：
大人说：“这是小白兔的书吧。”
孩子说：“不是小白兔的书，这是我的书。”
Answer: 小白兔的书可以是有关小白兔的书，也可以是属于小白兔的书，大人和孩子理解不同所以会造成歧义。
可以请你帮我一个忙吗？我不太理解下面这段话，请你帮我给出合理的解释：
甲说：“这是什么意思。”
乙说：“没什么，意思意思。”
甲说：“这就没意思了。”
乙说：“小意思小意思。”
甲说：“你这人真有意思。”
乙说：“其实也没别的意思。”
甲说：“那我就不好意思了。”
乙说：“其实是我不好意思。”
Answer: “意思”一词多义，在对话不同位置分别表示
“含义”、“送礼表意”、“无趣”、“简单”、“有趣”、“意图”、“难为情”
等多种意思，容易混淆。
可以请你帮我一个忙吗？我不太理解下面这段话，请你帮我给出合理的解释：
重庆的夏天能穿多少穿多少，重庆的冬天能穿多少穿多少。
Answer: 冬天多穿，夏天少穿。

Omission

我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，我出差来到苏州，我想抽空去旅游，能帮我推荐游乐场吗？我想去。”
乙说：“你想找个什么价位的呢？”
甲说：“消费水平最好在中等的。”

上面这段对话的最后一句中存在省略现象，请补全被省略的内容，给出省略内容被补全后的完整语句。
Answer: 消费水平最好在中等的游乐场。
我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，我和家人一起来苏州旅游，能帮我在吴江地区找一个好玩的景点吗？”
乙说：“好的，您希望找个贵一点的还是便宜一点的景点呢？”
甲说：“偏贵一点的吧。”

上面这段对话的最后一句中存在省略现象，请补全被省略的内容，给出省略内容被补全后的完整语句。
Answer: 偏贵点的景点吧。
我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，我和家人一起来苏州旅游，能帮我在吴江地区找一个好玩的景点吗？”
乙说：“好的，您希望找个贵一点的还是便宜一点的景点呢？”
甲说：“偏贵一点的吧。”

请问对话的最后一句中是否存在省略现象？
Answer: 是。

Coreference

我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“我们一家来苏州旅游，想在姑苏区找一个便宜一点的景点。”
乙说：“符合您要求的景点可多了，不知道该给你推荐哪个，你对景点类型有要求吗？”
甲说：“我对这个类型没有什么要求，只要便宜的。”

上面这段对话的最后一句中存在代词指代具体内容的现象，请使用代词指代的内容替换代词，给出代词被替换后的完整语句。
Answer: 我对景点类型没有什么要求，只要便宜的景点。
我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，下周五我想去厦门。请问有上午从深圳出发的高铁吗？”
乙说：“您可以乘坐高铁一六零二，深圳北到厦门北。”
甲说：“这个车几点出发呢？

上面这段对话的最后一句中存在代词指代具体内容的现象，请使用代词指代的内容替换代词，给出代词被替换后的完整语句。
Answer: 高铁一六零二几点出发呢？
我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，下周五我想去厦门。请问有上午从深圳出发的高铁吗？”
乙说：“您可以乘坐高铁一六零二，深圳北到厦门北。”
甲说：“这个车几点出发呢？

请问对话的最后一句中是否存在使用代词指代具体内容的现象？
Answer: 是。

Multi-turn Interaction (Only Input)

你好，帮我推荐一个价位在四百至五百元之间的酒店，评分要在四点五分以上的，谢谢。
雅悦酒店北京西直门店的周边有啥好玩的地方吗？
我和妈妈去吃饭，想找一家有酸辣蕨根粉的餐馆，最好人均消费在一百到一百五十元之间的，评分在四分以上的，有推荐的吗？
青年餐厅的地址在哪呢？周边有啥好吃的餐馆？
好的，我约了同学出去玩，帮我推荐一个没有门票费的景点吧。
童话树屋适合游玩多长时间呢？
好的，谢谢。
你好，帮我推荐一个价位在四百至五百元之间的酒店，评分要在四点五分以上的，谢谢。

如果之前出现过该问题，则直接给出之前的答案。

Responses

Below are model response examples for benchmark test cases. Each block shows the input example and outputs from different SDMs, including audio and text.
Green indicates correct, red indicates incorrect.

Phonological Ambiguity

Input Example

下面有一个断句可能不正确的句子：
这个苹果不大好吃。
这个句子想表达的意思是：
这个苹果不好吃。
请用正确的断句读出这句话。

GLM-4-Voice Correct

这个苹果/不大好吃。

A Spoken Dialogue Model Incorrect

这个苹果不大/好吃。

Semantic Ambiguity

Input Example

The sentence below contains ambiguity:
Mr. Smith loves music more than his wife.
Please tell me how to understand it.

GPT-4o-Audio-Preview Correct

The sentence "Mr. Smith loves music more than his wife" is ambiguous because it can be interpreted in two different ways:
1. Mr. Smith loves music more than he loves his wife. In this interpretation, music is of greater importance or brings more joy to Mr. Smith than his relationship with his wife.
2. Mr. Smith loves music more than his wife loves music. In this interpretation, Mr. Smith's passion or interest in music surpasses that of his wife.
To clarify the intended meaning, additional context or a rephrased sentence would be helpful.

A Spoken Dialogue Model Incorrect

This sentence contains ambiguity because it's unclear whether Mr. Smith loves music more than his wife loves music or more than his wife loves something else.

Omission

Input Example

我有一道考察语义理解的题目如下，请听完后回答我的问题：
甲说：“你好，我和家人一起来苏州旅游，能帮我在吴江地区找一个好玩的景点吗？”
甲说：“好的，您希望找个贵一点的还是便宜一点的景点呢？”
甲说：“偏贵一点的吧。”
请问对话的最后一句中是否存在省略现象？

GPT-4o-Audio-Preview Correct

是的，对话中的最后一句话存在省略现象。完整的表述应该是 “便贵一点的景点吧”。省略的是“的景点”这部分。

A Spoken Dialogue Model Incorrect

好的，我们来分析一下。甲说他和家人来苏州旅游，想在吴江找一个好玩的景点。乙问甲是希望找个贵一点的还是便宜一点的景点。甲说他更喜欢贵一点的。从这里看，甲的意思是希望景点质量好，但预算可以稍微高一些。乙没有表现出任何反对意见，所以甲的请求没有被拒绝。因此，对话的最后一句中没有省略现象。

Coreference

Input Example

Joan made sure to thank Susan for all the help she had recieved.
What does the word "she" refer to: Joan or Susan?

GPT-4o-Audio-Preview Correct

In the sentence, the word "she" refers to Joan, indicating that Joan is the one who made sure to thank Susan for all the help she had received.

A Spoken Dialogue Model Incorrect

In this context, the word "she" refers to "SUSAN". The sentence "John thanked Susan for all the help she had received" indicates that Susan provided the assistance that John is acknowledging.

Multi-turn Interaction

Input Example First Input

你好，帮我推荐一个价位在四百至五百元之间的酒店，评分要在四点五分以上的，谢谢。

Middle Input: ... Last Input

你好，帮我推荐一个价位在四百至五百元之间的酒店，评分要在四点五分以上的，谢谢。
如果之前出现过该问题，则直接给出之前的答案。

GPT-4o-Audio-Preview Correct

First Response

在这个价格范围内，我为您推荐几家评分在4.5分以上的酒店：
1. 汉庭酒店（或类似酒店品牌）：这类经济型酒店通常具有较高的性价比，在用户评价中也往往有不错的口碑。
2. 怡莱酒店：作为汉庭酒店集团旗下的品牌，怡莱酒店通常提供高品质的服务，且价格适中。
3. 7天连锁酒店：该连锁酒店在全国范围内有众多分店，也有很多口碑不错的店面。
以上都是一些比较常见且受欢迎的品牌连锁酒店，您可以根据地理位置和具体需求进行选择。

Last Response

当然，之前推荐的是一些经济型连锁酒店品牌，这些酒店通常价格符合您的预算，而且评分也往往在4.5分以上。可以考虑以下这些品牌：
1. 汉庭酒店：遍布全国，性价比高，服务也很不错。
2. 怡莱酒店：汉庭酒店集团旗下品牌，提供高品质服务，价格适中。
3. 7天连锁酒店：全国有很多分店，评价也较好。
这些都是经济型连锁酒店，您可以在预订网站上查看具体的酒店位置和评价，选择符合您需求的酒店。

A Spoken Dialogue Model Incorrect

First Response

好的，我找到了几家符合您要求的酒店。比如，成都锦江之星武侯祠店，评分4.8，价格在450元左右。还有成都望江宾馆，评分4.7，价格在480元左右。最后是成都武侯祠快捷酒店，评分4.6，价格在440元左右。您可以选择自己喜欢的。

Last Response

你好，根据你的要求，我推荐如家快捷酒店。它在中国很多城市都有分店，评分都在4.5分以上，价格在400到500元之间。你可以通过酒店的官网或者在线旅游平台预订。希望这个推荐能帮到你！

SDM Results Submission

Want to submit your SDM results to C3Bench?

Please follow the instructions at: C3Bench Submission Guide

The link provides detailed steps for preparing and submitting your Spoken Dialogue Model results to the leaderboard.

Citation

If you use C3Bench in your research, please cite us:

@inproceedings{c3bench,
          title={C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations},
          author={Ma, Chengqian and Tao, Wei and Guo, Yiwen},
          booktitle={arXiv Preprint},
          url={https://arxiv.org/abs/2507.22968},
          year={2025},
        }
}

C3Benchmark

Ambiguity

Phonological Ambiguity

Semantic Ambiguity

Context-dependency

Omission

Coreference

Multi-turn Interaction

Phonological Ambiguity

Semantic Ambiguity

Omission

Coreference

Multi-turn Interaction

Phonological Ambiguity

Semantic Ambiguity

Omission

Coreference

Multi-turn Interaction (Only Input)

C³Benchmark