Openjutsu: Swap video characters locally with ComfyUI and MiniMax H3

What it's about

Openjutsu is an open-source project by D3voz (GitHub: D3voz/openjutsu) that recreates "Genjutsu", the character swap feature of the commercial service Higgsfield, so it runs locally. You upload a video and a photo of a character, click "Generate", and the main person in the video is replaced by your character. Camera work, cuts, motion and soundtrack stay intact. The project was built for Comfy's "Open Source Higgsfield" challenge, and the author says it was built mostly with Claude. The code is MIT-licensed. The models it uses have their own licenses.

Everything runs on your own GPU with open weights. MiniMax H3 generates the video. SAM 3.1 (a segmentation model that marks people in an image and tracks them across the video) detects the people. Qwen3-VL 8B (a model that understands images and video frames) describes images and scenes and writes prompts. The app is a small Next.js frontend on top of a ComfyUI workflow (ComfyUI is a node-based interface for AI image and video models). It uses the Comfy SDK and is modeled on the reference app "comfy-reshoot".

The modes

• Simple motion transfer: One character replaces the main person, meaning the one who fills most of the frame.
• Advanced motion transfer: 1–4 characters, each with an optional role ("the drummer"). Each tracked person is assigned a role per shot. People without a character stay unchanged. With two characters, "One pass per person" gives each its own run, which takes about twice as long. A Montage tab handles edited clips where people change without a hard cut (pans, selfie handoffs).
• Object swap: Change tracked items (hair, jacket, car), remove them ("Remove"), change the background, or place new objects with "Add". Only the affected area is regenerated.
• From picture: FLUX prepares the character in the pose and framing of the video frame before the video is rendered. There is also an optional "Relight" with nine lighting moods or a custom lighting description. The author himself calls it very beta.
• Post-processing: "Fix faces" redraws faces, because H3 renders small faces poorly. "Fix colours" softens the harsh H3 look with detail contrast and tone mapping.

How it works

For each clip, SAM 3.1 tracks all people and keeps the largest match. The edit mask gets 10% headroom above the head and grows by 40 pixels, so characters with different builds have room. The motion template is the source clip, in which the person appears as a sharp grey "mannequin" in an exact silhouette. This keeps pose, arms and fingers readable. From that, H3 generates the new person using the photo as a reference.

In advanced mode, Qwen3-VL looks at each SAM track per shot and names the subject. Another node resolves duplicate assignments by colour. "Fix faces" crops the person's face across the whole video to 512×512, reworks it in blocks of about 6.5 seconds and pastes it back. You can choose "Match the photo" (closest likeness, but invented facial expressions) or "Follow the mouth" (keeps the video's expressions, with slightly less likeness).

Known limits and tips

• Mirrors and reflections don't work yet: The reflection still shows the original person, and H3 picks up their appearance.
• Name roles by function, not by clothing. Clothing words "leak" into the result.
• "Check mask" checks only the mask, in about one minute per 3 seconds of video: cyan is the newly generated area, orange is the tracked person.
• "No lip-sync" helps with songs the people in the video aren't singing along to. The original track stays in the final video.
• People visible for fewer than 12 frames or in under 2% of the frame stay unchanged, because H3 would render them as grey figures.
• A larger motion reference size (768 or 1376 px instead of 512) can improve accuracy slightly, but costs about a third more or 2.4 times the compute time, respectively.
• Without a speed LoRA (an add-on model that speeds up generation), use 20 or more steps. The default is the DMAD LoRA with 6 steps.

What you need to try it

According to the README, you need an NVIDIA GPU. It was tested with an RTX 4060 Ti (16 GB of VRAM) and 64 GB of system RAM, because the H3 INT8 models are streamed from RAM. You also need ComfyUI 0.41.2 or newer, Node.js 22.6 or newer and Python. Many custom nodes (including SolarRenderer, MiniMaxH3 Context Loop, KJNodes, VideoHelperSuite and Impact Pack) and several model files are also required. ComfyUI Manager can find some by name, and Openjutsu reports missing packages and files before it starts. Plan for plenty of disk space for the models (the source doesn't give exact sizes). On Windows, start_openjutsu.bat launches everything. Manually, it's the proxy, npm ci, npm run build and npm start, then http://127.0.0.1:3000. On Windows, ComfyUI must be started with AIOHTTP_NOSENDFILE=1. Runtime: a 6-second 480p clip takes about 6–13 minutes. 736p is the default, and 1080p is about 2.5 times slower and needs more memory. A 17-second video at 768p for "Fix faces" took about 28 minutes. The source says nothing about Mac, CPU or Colab.

Our take

Openjutsu shows how far local video AI has come: a service that otherwise exists only as a paid offering can be rebuilt with open models on a 16 GB card. The project isn't a new model release but a convenience layer over existing workflows. The installation effort (many custom nodes, many models) remains high. Waiting times are long, and problems like reflections, small faces and H3's harsh look are only partly handled by extra steps. License notes: "Fix faces" uses InsightFace buffalo_l (non-commercial research only) and Ultralytics YOLO weights (AGPL-3.0). The source doesn't back up a comparison with Higgsfield itself, and quality claims come from the author. According to the README, cloud deployment through the Comfy Developer Platform hasn't been tested yet. Because people can be swapped convincingly in videos, responsible use is an issue. The README doesn't address it.