NVIDIA Nemotron 3.5 brings open models for embedding, reranking and document parsing

Windows 10 / 11

Windows 10 / 11: not officially tested. The source only lists Linux commands and requires an NVIDIA graphics card (CUDA) plus FlashAttention 2. This is untested on Windows and often difficult. WSL2 (Linux on Windows) with the Linux steps is the safer route.

1. Check the prerequisites. You need an NVIDIA graphics card with about 21 GB of VRAM (video memory) and a current NVIDIA driver. Install Python 3.12 and Git. You also need about 20 GB of free disk space.

   python --version
   git --version
   nvidia-smi

2. Create a folder. Create a new, empty folder for the project and change into it.

   mkdir nemotron-embed
   cd nemotron-embed

3. Create an environment. A virtual environment keeps the packages separate from the rest of your PC. If PowerShell blocks scripts, allow them for this session with the first command.

   Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
   python -m venv .venv
   .venv\Scripts\Activate.ps1

4. Install the packages. This downloads PyTorch, Transformers and the FlashAttention kernels. It takes a few minutes. If PyTorch gets installed without CUDA, you need the CUDA build of PyTorch.

   pip install torch "transformers==5.20.0" accelerate pillow
   pip install kernels
   pip install "sentence-transformers[image]==6.1.0"

5. Save the example script. Save this code as demo.py in the nemotron-embed folder.

   from sentence_transformers import SentenceTransformer
   
   model = SentenceTransformer(
       "nvidia/Nemotron-3.5-Embed-8B-BF16",
       device="cuda",
       model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"},
   )
   
   query = "What is NVIDIA NeMo Framework?"
   image_url = "https://docs.nvidia.com/nemo-framework/user-guide/24.07/_images/nemo-llm-mm-stack.png"
   nemo_text = "NeMo Framework offers comprehensive functionalities for developing both LLMs and Multimodal Models."
   biology_text = "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences."
   names = ["nemo_image_text", "nemo_text", "nemo_image", "biology_text"]
   documents = [
       {"image": image_url, "text": nemo_text},
       nemo_text,
       {"image": image_url},
       biology_text,
   ]
   
   q = model.encode_query([query])
   d = model.encode_document(documents, batch_size=1)
   scores = model.similarity(q, d)
   
   print(f"query_embedding.shape={q.shape}")
   print(f"document_embeddings.shape={d.shape}")
   for name, s in zip(names, scores[0]):
       print(f"{name}_similarity={s.item():.6f}")

6. Run it and load the model. On first run, the program downloads the model automatically (about 16.8 GB). This can take a while.

   python demo.py

7. Result. The terminal shows the shape of the vectors (4096 values) and four similarity scores for the example documents.

macOS

macOS: not supported. The source's example fails with an error without an NVIDIA graphics card (CUDA). Macs have no CUDA support. Use a Linux PC with an NVIDIA card, a cloud machine (e.g. Colab with a GPU), or the smaller Nemotron-3.5-Embed-1B instead.

Linux

Linux: supported. The source lists Linux with an NVIDIA graphics card (CUDA), Python 3.12 and FlashAttention 2. AMD cards aren't mentioned in the source. Without a CUDA card, the example won't run.

1. Check the prerequisites. You need Python 3.12, an NVIDIA graphics card with about 21 GB of VRAM (video memory) and a current NVIDIA driver. You also need about 20 GB of free disk space.

   python3.12 --version
   nvidia-smi

2. Create a folder. Create a new, empty folder for the project and change into it.

   mkdir nemotron-embed
   cd nemotron-embed

3. Create an environment. A virtual environment keeps the packages separate from the rest of your system.

   python3.12 -m venv .venv
   source .venv/bin/activate

4. Install the packages. This downloads PyTorch, Transformers, prebuilt FlashAttention kernels and Sentence Transformers. It takes a few minutes.

   pip install torch "transformers==5.20.0" accelerate pillow
   pip install kernels
   pip install "sentence-transformers[image]==6.1.0"

5. Save the example script. Save this code as demo.py in the nemotron-embed folder. It compares a question against four documents (text, image, both).

   from sentence_transformers import SentenceTransformer
   
   model = SentenceTransformer(
       "nvidia/Nemotron-3.5-Embed-8B-BF16",
       device="cuda",
       model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"},
   )
   
   query = "What is NVIDIA NeMo Framework?"
   image_url = "https://docs.nvidia.com/nemo-framework/user-guide/24.07/_images/nemo-llm-mm-stack.png"
   nemo_text = "NeMo Framework offers comprehensive functionalities for developing both LLMs and Multimodal Models."
   biology_text = "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences."
   names = ["nemo_image_text", "nemo_text", "nemo_image", "biology_text"]
   documents = [
       {"image": image_url, "text": nemo_text},
       nemo_text,
       {"image": image_url},
       biology_text,
   ]
   
   q = model.encode_query([query])
   d = model.encode_document(documents, batch_size=1)
   scores = model.similarity(q, d)
   
   print(f"query_embedding.shape={q.shape}")
   print(f"document_embeddings.shape={d.shape}")
   for name, s in zip(names, scores[0]):
       print(f"{name}_similarity={s.item():.6f}")

6. Run it and load the model. On first run, the program downloads the model automatically (about 16.8 GB). Then it computes the scores.

   python demo.py

7. Check the output. You should see values like nemo_text_similarity around 0.59 and biology_text_similarity around 0.25. The higher the value, the better the document matches the question. Small deviations are normal.
8. Result. The terminal shows the shape of the vectors (4096 values) and four similarity scores for the example documents.