Sophon-Realtime v1: Open voice AI that listens, laughs and lets you interrupt

What it's about

Adventists AI has released Sophon-Realtime v1, a family of open voice models for real-time conversation. What sets them apart is that they're full-duplex: they can listen and speak at the same time, the way people do on the phone. The AI listens the whole time, jumps in as soon as you stop talking and goes quiet the moment you interrupt. Classic voice assistants, by contrast, usually work like a walkie-talkie: you talk, then they talk.

The developer also says the model hears how you say something (hesitation, emphasis, sarcasm, laughter, sighs) and answers with tone of its own. It laughs, sighs or clears its throat when the conversation calls for it. Code and weights are released under the Apache 2.0 license, so you can use them commercially too. The project page is on Hugging Face and the code is on GitHub. If you don't have the hardware, the developer says you can try it in a ModelScope Studio.

Three variants, one brain

All three variants use the same 32B model. It listens, decides when it's its turn and writes the reply. The only difference is how that reply gets turned into sound:

The "tags" are small markers in the reply text, for things like laughing, sighing, breathing, coughing or a stressed syllable. The speech model performs these markers instead of reading them out. The developer's advice: pick Tag if you want the voice to act, and Flash if speed and lots of concurrent users matter more. Both share the same interfaces, so you can switch later.

How the conversation works

The model listens nonstop. At every pause, it uses a single token (a unit of text) to decide whether you're done, still talking, just dropping in an "mhm" (back-channeling) or thinking. From your very first pause, it drafts a reply and plays it as soon as it decides it's its turn. If you keep talking, it throws the draft away before you hear it. That's where the short response time comes from.

Text and voice run together, with the text half a step ahead. So every reply comes with a transcript you can check, and that tools or larger background models can process further. The conversation history stores the actual audio from both sides. If you interrupt the AI, it only remembers the part of its reply you actually heard.

The tone "listening" is handled by DuplexJev, another open model from the developer that makes speech decisions. The E2E variant takes a different route: it writes replies directly in vector language, meaning the strings of numbers a speech model builds internally while listening. A decoder turns those into sound, so listening and speaking share a single representation. This variant is explicitly a research preview only.

Benchmarks, according to the developer

All the numbers below come from Adventists AI. Some were measured in-house, and there are no independent tests yet.

• VoiceBench (conversation quality, judged by gpt-4o-mini): Sophon scores 82.8, on par with GPT-4o-mini-Audio. GPT-4o-Audio scores 86.8. Other open full-duplex models trail well behind, including MiniCPM-o 4.5 at 76.1, Freeze-Omni at 55.2 and Moshi at 29.5.
• Big Bench Audio (1,000 spoken reasoning tasks): 81.8 percent, compared with 90 percent for GPT-Live-1. Open competitors such as Raon SpeechChat reach 58 percent, Moshi 4 percent. Worth knowing: the developer measured Sophon's score itself (transcription, judged by gpt-4.1), while the comparison numbers come from the Artificial Analysis leaderboard.
• Full-Duplex-Bench v3 (tool calls with real, halting speech, 100 tasks): Pass@1, the share of tasks solved on the first try, is 48 percent. That puts it ahead of Grok (43 percent) and Ultravox v0.7 (41 percent), but behind GPT-Realtime (60 percent) and Gemini Live 3.1 (54 percent). The response rate is 100 percent. One caveat: the developer says Sophon's figure is the "best of several runs", which can tilt the comparison in its own favor.
• Paralinguistics, meaning understanding emotion, irony, emphasis and intonation: DuplexJev scores 48.6, ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6).

Live mode and customization

The repo's live/ folder contains a gateway that speaks the protocol of OpenAI's GPT-Live. Existing GPT-Live apps only need to change the address and key. When the AI has to look something up or run an action, it says something brief like "One moment, let me check", hands the task to your backend and then tells you the result in its own words. A larger model can also work in the background: the developer says pairing it with DeepSeek V4 Pro raises Big Bench Audio from 81.2 to about 83.

You can customize persona, tone, domain knowledge and tool use with LoRA. This trains only a small add-on module and leaves the base model unchanged. It runs with the ms-swift framework (version 4.6 or later). The audio encoder, voice and turn-taking stay untouched, and the sound tags are preserved. According to the developer, a single 80 GB GPU is enough.

What you need to try it

Sophon-Realtime isn't a model for your gaming PC. For Tag and Flash, the developer recommends a GPU with 80 GB of VRAM (video memory), such as a data center card in the A100 or H100 class. The E2E variant needs two of those GPUs. The Tag weights are around 42 GB. So expect to need well over 42 GB of VRAM and at least as much free disk space, ideally more for the environment. According to the guide, Linux, CUDA (so Nvidia cards) and Python 3.12 are supported. The developer doesn't mention smaller quantized versions or GGUF files for tools like llama.cpp or Ollama. Macs and CPU-only setups aren't supported.

Installation is just pip install sophon-realtime, followed by sophon-realtime serve. The first launch downloads the weights, and then the server runs on port 8000. Using the scripts in the repo, the first launch takes 15 to 20 minutes and later launches 5 to 8 minutes. The model is served via vLLM-Omni with two interfaces: a WebSocket connection for true duplex conversations and a /v1/chat/completions endpoint (audio in, text and audio out). If you don't have suitable hardware, you can try it online in the ModelScope Studio or rent a cloud GPU.

Our take

Open voice models have lagged well behind the closed offerings from OpenAI and Google so far, especially in real conversations with interruptions. If the numbers hold up, Sophon-Realtime closes a good chunk of that gap: it's clearly ahead of well-known open projects like Moshi and on par with GPT-4o-mini-Audio, but still behind GPT-4o-Audio, GPT-Live-1 and GPT-Realtime. It's most interesting for companies that want to run voice assistants themselves, for customer service or companion apps, for example, without sending conversation data to US providers. The Apache license and GPT-Live compatibility fit that use case well.

The caveats: every benchmark comes from the developer. Some were measured in-house or are the "best of several runs". Independent tests are still pending. The nonverbal vocalization scores are only given for Chinese and English, so it's unclear how well other languages such as German work. And for hobbyists, the bar stays high: without an 80 GB GPU, the model won't run locally.