WeMM-Video: Tencent's open models watch and listen to video at the same time

Tencent has released four open models that analyze a video's picture and sound together. They describe scenes, transcribe speech and answer questions about what's going on. Tencent's own tests say even the small 4B version beats a rival seven times its size. Nobody has independently checked that yet.

Model releases Try it yourself Nov 20, 2026 8 min read Source: Reddit r/LocalLLaMA
Illustrative image, AI-generated

What's going on

Tencent has released WeMM-Video, a family of open AI models built to understand video, picture and sound together. The WeChat Vision team developed it, according to explainx.ai together with Tsinghua University. There are four sizes: 2B, 4B and 9B as dense models and 35B-A3B as a mixture-of-experts model. In that design the network is split into many specialists ("experts"), and only about 3 billion of the roughly 35 billion parameters do any work for each token.

The weights live in a Hugging Face collection and in the GitHub repository Tencent/WeMM-Video. The repo also has Docker setups, quickstart guides and the pipeline Tencent used to produce its test results. The technical report went up on arXiv on November 19, 2026 (2611.26409). The license is Apache 2.0. Tencent's license file says bundled third-party components keep their own terms. The model card doesn't say which components those are.

The launch was strikingly quiet. According to OrcaRouter's blog, all 18 repositories (four sizes in five formats each) were uploaded on November 17 within about five minutes, apparently by a script. Tencent published no blog post, no press release and no launch page. When OrcaRouter ran its analysis, the base checkpoints had zero downloads.

What the models can do

WeMM-Video is an omni model: a single model that handles several kinds of input. Give it a video with sound and it can describe events, transcribe speech and answer questions about what you see and hear. It also takes silent video, audio files on their own, images and text. Output is always text, in Chinese or English.

That sets it apart from most open video models, which only look at the frames and ignore the audio. An omni model can tell you who's speaking, what was said, or whether the music fits the scene. The report calls existing open models "brittle" on real-world video. They miss details, pin sounds on the wrong event and make up things that appear in neither the picture nor the audio. Tencent lists live shopping, short-drama analysis and interactive video as use cases.

Every model has two modes. Instruct mode answers directly and quickly. Thinking mode writes out a reasoning trace first, step-by-step reasoning you can follow. That costs more time and tokens, but according to the report it does much better on most reasoning tasks. You switch modes with the enable_thinking parameter in the chat template.

How it's built and trained

Each model has three parts. A vision encoder turns video frames into tokens, the small chunks a language model can process. An audio encoder does the same for sound. The language model processes those tokens together with the text. Image and audio tokens are interleaved in time order and tagged with text timestamps, so the model can match a spoken sentence to the frame where it's said. According to explainx.ai, the architecture is modeled on video-SALMONN 2.

The foundation comes from Alibaba. The dense models build on Qwen3.5-2B, -4B and -9B, the MoE model on Qwen3.6-35B-A3B. The audio side is a Whisper-style encoder initialized from Qwen2.5-Omni. It reads 16 kHz audio in 30-second chunks. A Q-Former, a kind of compressor, squeezes each 0.1-second window into one token, which works out to 10 audio tokens per second. The language model itself is a hybrid: only every fourth layer uses full attention, and the other three use a leaner linear variant.

Training used about 1.2 trillion multimodal tokens. Joint audio-video data makes up the biggest share at 45.4 percent, and the most common task is captioning at 37.7 percent. The data covers sports, esports, movies, short dramas, interviews, shows and live shopping, among other things. Pretraining ran in three stages. First only the audio connector was trained, then almost the whole model except the vision encoder, and finally the context was extended with long videos and long audio.

Post-training followed in three steps:

  • Multi-teacher OPD (on-policy distillation): stronger "teacher" models pass on text and image skills, while models from the family itself keep the audio skills intact.
  • Anti-repetition MrDPO: the base models tended to repeat themselves endlessly. Tencent trained that out with deliberately collected examples of such loops, using LoRA adapters (small add-on modules).
  • Specialized reinforcement learning: learning through rewards, with rules for verifiable tasks and scoring rubrics for open-ended ones.

Benchmarks: strong, but only Tencent's numbers

Tencent says it tested on more than 120 benchmarks. In thinking mode, the flagship 35B-A3B reportedly beats Qwen3.5-Omni-Flash and Seed2.0-Mini-0428 on 16 of 17 general audio-video tests. The gap to Gemini-3.0-Flash narrows, but the report doesn't claim a win there. The 9B model reportedly beats the larger Qwen3-Omni-30B-A3B-Instruct.

OrcaRouter pulled together the report's numbers for the 4B model (thinking mode):

Benchmark WeMM-Video-4B Qwen3-Omni-30B-A3B MiniCPM-o-4.5-9B
Video-MME (with audio) 83.9 77.3 69.2
Video-MME v2 (with audio) 48.1 36.3 28.1
Daily-Omni 82.5 70.7 80.2
WorldSense 60.3 54.0 55.7
LVBench (silent) 65.3 50.2 50.9
LongVideoBench (silent) 74.8 63.5 66.0
Video-Holmes 58.8 – 64.3
FutureOmni 50.9 55.1 56.1

The standout is the 15-point lead on LVBench, a long-video test where small models usually struggle. On pure audio, though, the 4B model loses to the proprietary Qwen3.5-Omni-Flash: 75.2 to 80.4 on MMAU and 67.3 to 74.0 on MMAR. On images and text it's roughly level, for example 78.4 vs. 76.9 on MMMU and 96.5 vs. 95.3 on DocVQA. It trails on OCRBench (85.7 to 89.1) and GPQA (73.2 to 76.4).

The catch: no third party has verified these numbers yet. The report says competitor scores come from official figures "where available, otherwise from our own evaluations." On some tasks, AI models such as GPT-4o graded the answers. The mode also changes between tables: video tests ran in thinking mode, audio tests in instruct mode. Neither Artificial Analysis nor LMArena lists WeMM so far.

The fine print: names and context

OrcaRouter points out two places where the label doesn't match the contents. First, the number in the name only refers to the language model. Add the vision and audio parts and the models get bigger: 3.28 billion (2B), 5.61 billion (4B), 10.49 billion (9B) and 36.18 billion parameters (35B-A3B). The audio part of the 4B model alone has 1.07 billion parameters. In BF16, the full 16-bit format, the 4B file takes up 10.45 GiB.

Second, the config allows a theoretical 262,144 tokens of context, meaning how much text, image and audio the model can keep in view at once. But every launch command Tencent published sets it to 8,192 tokens. Since audio alone eats 10 tokens per second, that's enough for just over 13 minutes of pure audio, and far less once frames are added. The vLLM recipe also takes only 16 frames per video instead of the possible 128. The repository doesn't say under which settings the long-video tests ran.

What you need to try it

The official guide requires Python 3.12, an NVIDIA GPU with a driver for CUDA 12.9, and FFmpeg. Versions are pinned to torch 2.10.0, torchvision 0.25.0, torchcodec 0.10.0 and transformers 5.2.0. Loading through Transformers needs trust_remote_code=True, which runs code from the model repository. That's why the README recommends reading the code first and pinning a specific commit. For serving, there are Docker images with patched vLLM 0.18.0 and SGLang 0.5.16. They expose an OpenAI-compatible API on port 8000. Each request can include up to four images, two videos and two audio files.

Formats on offer are BF16, FP8, AWQ W4A16, GGUF Q4_K_M and NVFP4. The 4-bit formats are shrunk through quantization: the model's numbers are stored at lower precision, which saves a lot of memory. As a rough estimate, the 2B model in 4-bit needs about 4 GB of VRAM, the 4B around 6 GB, the 9B about 8 GB and the 35B-A3B around 24 GB, plus headroom for longer videos. In BF16, the 4B model alone takes more than 10 GiB. There's a GGUF variant with its own runtime. The sources don't make clear whether it runs usefully on a Mac or CPU only. The official path is built for NVIDIA.

The bigger picture

WeMM-Video fills a real gap. Open models that understand a video's picture and sound together are rare, and rarer still in sizes that fit on consumer GPUs. If Tencent's numbers hold up, the 4B model beats Qwen3-Omni-30B-A3B, seven times its size, at video understanding, while staying close to its base model on images, documents and text. A single model could then handle screenshots, PDFs and video.

Still, be careful. Every benchmark comes from Tencent itself, Tencent picked the baselines, and the top scores come from the more expensive thinking mode. The model names understate the real memory needs. The practical 8,192-token limit seriously restricts work with long videos. And technically, the project depends on Alibaba's Qwen models. Only independent testing will show whether the claims hold up in practice, and so far there isn't any.

Sources

#tencent#video understanding#omni-modal#open-weights#multimodal
Read the original source

Try it yourself: WeMM-Video-4B

Explained step by step, no prior knowledge needed – on your own computer or free in your browser.

What can I do with it?

WeMM-Video watches a video and listens to its audio at the same time. It can describe what happens, transcribe speech and answer questions like "Who says what in which scene?" or "Does the music fit the mood?" It also accepts silent video, audio files on their own, images, documents and text, and officially replies only in Chinese or English. Tencent's own tests say even the 4B model beats the much larger Qwen3-Omni-30B-A3B at video understanding, especially on long videos. On pure audio it trails proprietary models like Qwen3.5-Omni-Flash. The biggest practical limit: the bundled launch commands allow only 8,192 tokens of context and take just 16 frames per video, so long clips with sound fill memory fast.

Who is it worth it for?

It's worth it if you have an NVIDIA GPU and want to automatically describe, search or analyze videos, for example as a content creator, researcher or developer of video tools. If you just want to chat, or you're on a Mac or AMD machine, other models will serve you better, because the official setup is built around NVIDIA and CUDA.

Ideas: What you can do with it

Tag your vacation videos

Run short phone videos through one by one and get a description of each with location, number of people and what's happening. That gives you a searchable archive of your own video collection.

Analyze talks and interviews

Feed in a recording with sound and ask for a transcript plus key points. Because the model also sees the picture, it can combine slide content with what's being said.

Draft subtitles for clips

For short social media videos, have it output the speech with approximate timestamps. Use the result as a rough draft for subtitles that you clean up afterwards.

Accessible video descriptions

Have the model describe what happens on screen while nobody is talking. You can turn that into draft audio descriptions for blind or visually impaired viewers.

Turn a tutorial into written instructions

Feed in a craft or cooking video and ask for written step-by-step instructions, including tips that are only spoken and never shown on screen.

Read documents and screenshots

Tencent says the model is also strong on documents (DocVQA 96.5), so you can snap a photo of an invoice or take a screenshot and ask specific questions about it.

Size5.6 B parameters
Download11 GB (about 15 min at 100 Mbit/s)
Licenseapache-2.0
TypeImage + text (understands images)

What your computer needs

Graphics card (NVIDIA) at least 14 GB VRAMCalculated, not a vendor figure: according to the Model Card, the BF16 weights are 10.45 GiB (about 11.2 GB). 11.2 GB × 1.2 + 1 GB comes to roughly 14.4 GB. Tencent only tested on an H20 data center card. Videos, images and long answers need extra memory, so a card with 16 GB or more makes sense. For smaller cards there are FP8, AWQ and GGUF (4-bit) variants, plus the smaller WeMM-Video-2B.
Mac (Apple Silicon M1–M5) from 24 GB unified memoryThe GPU and CPU share the memory – the model can use about 70% of it
Storage approx. 13 GB freeModel download plus programs

2B (3.28B parameters): for cards with about 4 GB of VRAM in 4-bit, good for trying it out. 4B (5.61B): a solid compromise for cards with about 6–8 GB in 4-bit (AWQ or GGUF Q4_K_M). In BF16 it needs more than 10.45 GiB plus context, so a 16 GB card. 9B (10.49B): about 8–12 GB in 4-bit, more like 24 GB in BF16. 35B-A3B (36.18B, 3B active): about 24 GB of VRAM in 4-bit, and according to Tencent the strongest model. FP8 and NVFP4 mainly pay off on newer NVIDIA cards with hardware support for those formats. The memory figures are rough estimates, not vendor specs.

  • Officially required: an NVIDIA GPU with a driver for CUDA 12.9, Python 3.12 and FFmpeg.
  • The names understate the size: "4B" actually has 5.61 billion parameters, 1.07 billion of them for the audio part. In BF16 that's 10.45 GiB before context.
  • Audio uses 10 tokens per second. With the default 8,192-token context you fit just over 13 minutes of pure audio, and far less with video frames.
  • More frames per video and more context need noticeably more VRAM. The vLLM default of 16 frames is deliberately frugal.
  • There's a GGUF version (Q4_K_M) with its own runtime. The sources don't show whether it runs at a usable speed on a Mac or CPU.
How do I find out what my computer has?
  • Windows: Press Ctrl + Shift + Esc → Task Manager → Performance tab → click GPU on the left. At the bottom you'll see Dedicated GPU memory (that's your VRAM). You'll find system memory under Memory.
  • Mac: Apple menu at the top left → About This Mac. You'll see your chip (e.g. M2) and memory (e.g. 16 GB). On Apple chips, graphics and processor share this memory.
  • Linux: Type nvidia-smi in the terminal (shows your graphics card and its memory) and free -h for system memory.

Guide

This model runs with its developers' own program (Tencent/WeMM-Video on GitHub). The commands below come from the Model Card and the developers' instructions.

Windows 10 / 11

Windows 10 / 11: not officially tested. The source only covers a Linux setup with an NVIDIA GPU and doesn't mention Windows. TorchCodec (used for video) in particular can cause trouble on Windows. The safer bet is WSL2, a Linux environment inside Windows, with the Linux instructions.

  1. Check the prerequisites. Install Python 3.12, Git and FFmpeg. FFmpeg has to be on your system PATH so the ffmpeg command works. You also need an NVIDIA GPU with a current driver that supports CUDA 12.9. Check everything in PowerShell.
    py -3.12 --version
    git --version
    ffmpeg -version
    nvidia-smi
  2. Download the code. Grabs the WeMM-Video code with its sample files and switches into the folder.
    git clone https://github.com/Tencent/WeMM-Video.git
    cd WeMM-Video
  3. Create a virtual environment. Creates an isolated Python folder and activates it. If you get an error about scripts, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once first. Afterwards, (.venv) appears at the start of the line.
    py -3.12 -m venv .venv
    .venv\Scripts\Activate.ps1
  4. Install PyTorch. Downloads PyTorch for CUDA 12.9. That's several gigabytes.
    python -m pip install --index-url https://download.pytorch.org/whl/cu129 torch==2.10.0 torchvision==0.25.0 torchcodec==0.10.0
  5. Install the remaining packages. Installs Transformers and the helper packages for images and audio.
    python -m pip install transformers==5.2.0 accelerate pillow librosa soundfile scipy requests
  6. Download the model. Downloads the roughly 11 GB of weights from Hugging Face.
    hf download tencent/WeMM-Video-4B
  7. Analyze your first video. Analyzes the bundled Apollo 11 clip, sound included. The description appears in the window. If video loading fails (TorchCodec), switch to WSL2.
    python examples/transformers_video.py --model tencent/WeMM-Video-4B --video examples/media/apollo-11-first-steps.mp4 --prompt "Describe what happens and summarize what you hear."
  8. Result. Your terminal shows an English description of the Apollo 11 sample video: what you see and what's being said.

macOS

macOS: not officially tested. The official guide assumes an NVIDIA GPU, which Macs don't have. According to the Model Card, Tencent only ran a text test on the CPU. On a Mac this is an experiment, and it will be slow. The GGUF variant with its own runtime (native/README.md in the repo) is meant for CPUs and is probably the better choice.

  1. Check the prerequisites. You need the Xcode command line tools (which include Git), Python 3.12 and FFmpeg, for example via Homebrew. Plan on at least 16 GB of RAM, since the whole model sits in memory.
    xcode-select --install
    python3.12 --version
    ffmpeg -version
  2. Download the code. Grabs the WeMM-Video code with its sample files and switches into the folder.
    git clone https://github.com/Tencent/WeMM-Video.git
    cd WeMM-Video
  3. Create a virtual environment. Creates an isolated Python folder. Afterwards, (.venv) appears at the start of the line.
    python3.12 -m venv .venv
    source .venv/bin/activate
  4. Install the packages. On a Mac, leave out the CUDA index URL so pip fetches the regular PyTorch build. Then install Transformers and the helper packages.
    python -m pip install torch==2.10.0 torchvision==0.25.0 torchcodec==0.10.0
    python -m pip install transformers==5.2.0 accelerate pillow librosa soundfile scipy requests
  5. Download the model. Downloads the roughly 11 GB of weights from Hugging Face.
    hf download tencent/WeMM-Video-4B
  6. Test with text first. Save this code as demo.py in the project folder and run it with python demo.py. It asks a simple arithmetic question from the Model Card. If you get a time as the answer after a while, the model is working.
    import torch
    from transformers import AutoModelForImageTextToText, AutoProcessor
    m = "tencent/WeMM-Video-4B"
    proc = AutoProcessor.from_pretrained(m, trust_remote_code=True)
    model = AutoModelForImageTextToText.from_pretrained(m, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto").eval()
    msg = [{"role": "user", "content": [{"type": "text", "text": "A 20-minute meeting starts at 09:45. When does it end?"}]}]
    inp = proc.apply_chat_template(msg, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, enable_thinking=False).to(device=model.device, dtype=model.dtype)
    with torch.inference_mode():
        out = model.generate(**inp, max_new_tokens=128, do_sample=False)
    print(proc.tokenizer.decode(out[0, inp.input_ids.shape[1]:], skip_special_tokens=True))
  7. Analyze a video. If the text test works, try the bundled Apollo 11 clip. On a Mac this can take a very long time.
    python examples/transformers_video.py --model tencent/WeMM-Video-4B --video examples/media/apollo-11-first-steps.mp4 --prompt "Describe what happens and summarize what you hear."
  8. Result. Your terminal shows an English description of the Apollo 11 sample video: what you see and what's being said.

Linux

Linux: supported. This is the official target system: Python 3.12, an NVIDIA GPU with a driver for CUDA 12.9, and FFmpeg. Without a strong GPU, there's the GGUF variant with its own CPU runtime (see native/README.md in the repo).

  1. Check the prerequisites. You need Python 3.12, Git, FFmpeg (a program that reads video and audio) and a current NVIDIA driver that supports CUDA 12.9. CUDA is NVIDIA's compute interface. nvidia-smi shows your GPU and the supported CUDA version. Expect a download of about 12 GB for the model.
    python3.12 --version
    git --version
    ffmpeg -version
    nvidia-smi
  2. Download the code. This pulls the WeMM-Video code and its sample files onto your machine and switches into the folder. Run all further commands from this folder.
    git clone https://github.com/Tencent/WeMM-Video.git
    cd WeMM-Video
  3. Create a virtual environment. A virtual environment (venv) is an isolated folder for Python packages, so nothing interferes with the rest of your system. If it worked, (.venv) appears at the start of your prompt.
    python3.12 -m venv .venv
    source .venv/bin/activate
  4. Install PyTorch. PyTorch is the compute library that runs the model. This command downloads the CUDA 12.9 build. That's several gigabytes, so be patient.
    python -m pip install --index-url https://download.pytorch.org/whl/cu129 \
      torch==2.10.0 torchvision==0.25.0 torchcodec==0.10.0
  5. Install the remaining packages. Now add Transformers (which loads the model) and helper packages for images and audio. At the end you should see "Successfully installed".
    python -m pip install transformers==5.2.0 accelerate pillow librosa soundfile scipy requests
  6. Download the model. This pre-downloads the roughly 11 GB of weights into your Hugging Face cache. Skip it and the script downloads them on first run.
    hf download tencent/WeMM-Video-4B
  7. Analyze your first video. The sample script watches the bundled Apollo 11 clip, sound included. The answer shows up in the terminal after a while. The model loads its own code from Hugging Face (trust_remote_code), which is intended for this model.
    python examples/transformers_video.py --model tencent/WeMM-Video-4B \
      --video examples/media/apollo-11-first-steps.mp4 \
      --prompt "Describe what happens and summarize what you hear."
  8. Result. Your terminal shows an English description of the Apollo 11 sample video: what you see and what's being said.

What to do after your first result

  • Pass your own MP4 file with sound via --video and ask your own question with --prompt, in English or Chinese.
  • In the Python example from the Model Card you can also pass images (type image), audio-only recordings (type audio) or videos without sound (type video). Use max_frames and fps to control how many frames the model takes from the video.
  • Set enable_thinking=True and the model reasons step by step before answering (thinking mode).
  • For advanced users: with the patched vLLM or SGLang (or the Docker images from the repo), the model runs as a server with an OpenAI-compatible API. You can test it with python examples/chat.py video-audio --model tencent/WeMM-Video-4B.

Try it

Describe in three sentences what happens in this video. (silent video)
Transcribe the speech in this audio file, then briefly summarize what it's about.
Who speaks when in this video, and what exactly does each person say? Give approximate timestamps. (video with sound, use_audio enabled)
Does the background music fit the mood of the scene? Back up your answer with what you see and hear.
Watch this cooking video and turn it into a recipe with an ingredient list and steps, including tips that are only spoken.

Tips for better results

  • Use thinking mode (enable_thinking) for questions that need reasoning, such as sequences of events or causes. Instruct mode is enough for quick descriptions and transcripts.
  • Ask in English or Chinese, since those are the only officially supported languages. You can ask for a summary in another language afterwards.
  • Short clips under a minute give the best results with the default settings. Split long videos into sections.
  • Ask specific questions instead of general ones: "What does the woman on the left say after the door opens?" makes better use of the model's timestamp skills than "What's happening?"

If something goes wrong

Stick to the exact versions: torch 2.10.0, torchvision 0.25.0, torchcodec 0.10.0 and transformers 5.2.0. Older transformers versions don't know the model.

Loading requires trust_remote_code=True. As the README recommends, review the code first and pin a specific commit with the revision parameter.

Stock vLLM or SGLang won't do: Tencent ships patched versions (vLLM 0.18.0, SGLang 0.5.16) as Dockerfiles. The easiest route is to run those Docker images.

If the audio gets ignored, you're usually missing use_audio: true in the request. Without it, the model only analyzes the frames.

Without FFmpeg, videos and audio can't be loaded. Install it first.

With long videos, the request fails or gets cut off because context is capped at 8,192 tokens. Trim the clip or carefully raise max-model-len, but that costs VRAM.

“out of memory,” a crash, or your computer freezes

The model is too big for your VRAM (video memory). Pick a smaller variant from the table above and close other programs, especially games and browsers with lots of tabs.

It runs, but extremely slowly

Your CPU is probably doing the work instead of your graphics card. In Ollama, the PROCESSOR column of ollama ps shows how much runs on the GPU. If it says "100% CPU", update your graphics driver or pick a smaller variant that fits entirely into VRAM (video memory).

“ollama” or “python” is not found

Close the terminal window and open it again so it recognizes the newly installed program. For Python on Windows: reinstall it and check Add python.exe to PATH.

Windows: "Running scripts is disabled on this system"

Run the script with exactly the command above: powershell -ExecutionPolicy Bypass -File try_windows.ps1. This allows execution for this one time only.

Error message when loading the model (version, "trust_remote_code", unknown architecture)

Brand-new models often need the latest software version. With Ollama: update Ollama. With Python: pip install -U transformers in the activated environment. The Model Card usually lists the exact version.

Tokenstrom generates the scripts and notebook automatically. Please take a quick look before running them – they're plain, readable text. Mind the model's license: apache-2.0.

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.