“Encrypted while the AI computes”: What confidential AI really protects

What this is about

On November 9, Google unveiled two new variants of its eighth TPU generation. TPUs are Google's in-house AI accelerators. One of them, the TPU 8tx, is designed to keep sensitive data and model weights “protected and isolated, even while they're actively in use”, according to Google. That's the promise of confidential computing: data isn't only encrypted on disk and in transit across the network, but also at the moment it's being processed. Apple, Google, Meta, Cohere and Anthropic use similar language to pitch their “private” AI clouds.

This article looks at what exactly gets encrypted, what doesn't, and who you still have to trust in the end. Up front: confidential AI isn't a seamless vault. It's a chain of individual protections, and with the TPU 8tx, several links in that chain haven't even been described yet.

The principle: a sealed room with a certificate of authenticity

Three building blocks work together:

1. Trusted Execution Environment (TEE): a sealed-off area inside the processor. On server CPUs, the technologies are called AMD SEV-SNP and Intel TDX. They encrypt an entire virtual machine (VM) with a key that only the processor knows. The cloud provider and its hypervisor, the software that manages the VMs, only see scrambled data in memory.
2. Attestation: the chip issues a signed report stating which firmware and code are currently running, much like a tamper-proof seal. A verifier compares this report against known reference values.
3. Key release: only if the check passes does a key server hand over the keys for user data or model weights. That's exactly how Anthropic describes its research design for confidential inference from June 2025. In it, the only code allowed to run is code that several engineers have reviewed and an internal build server has signed.

AI adds another complication: the heavy lifting is done by the accelerator, not the CPU. Data travels over the PCIe bus, the connection between CPU and card. It sits in HBM, the stacked memory right next to the chip. And large models are spread across many accelerators via chip-to-chip links such as Nvidia's NVLink. Each of these stops needs its own protection.

The Nvidia details come from its threat model. It states explicitly: “GPU memory encryption is not the protection mechanism.” So HBM is protected only by access rules. Anyone who opens the chip package and probes inside is outside the scope of the protection, according to Nvidia. Meta's WhatsApp whitepaper also points out that HBM is unencrypted.

What Google actually said about the TPU 8tx

Google's blog post only mentions “hardware-level confidential compute” without naming a specific technology. The 8tx uses the same silicon as the TPU 8t training chip, paired with hosts based on 5th-generation AMD EPYC. Google promises half the latency of the TPU 8t and a “near-zero” performance hit, both vendor claims. AMD's press release reveals more: the system uses SEV in the EPYC CPUs. Prompts, model weights and intermediate results therefore stay “encrypted in CPU memory and in transit between the host and accelerators”. The system also verifies an accelerator's identity and security configuration before granting it direct access to protected memory. Google executive Mark Lohmeyer told Network World:

“Then the entire system is attested, so we know it is what it claims to be.”

Plenty of questions remain open. The analysis site Beri has compiled them: Is the HBM on the TPU encrypted? Are the TPU-to-TPU links that distributed large models run over protected? Which SEV generation is used, and does it include SEV-SNP with its integrity protection? Does the TPU produce its own attestation, and can you verify it without trusting a Google service? Who holds the keys to the weights? Google's documentation also contradicts itself. The Confidential VM docs state clearly: “You can't attach TPUs to Confidential VM instances.” The TPU machine table, however, marks the TPU v6e as confidential-computing capable, even though Lohmeyer describes the 8tx as the “first” with hardware-backed confidentiality. Our guess, which Google hasn't confirmed: the label could refer to the “Titanium Intelligence Enclave”, which according to InfoQ Google uses for its Private AI Compute service starting with the Trillium TPU generation (v6e). There's no price, no list of regions and no firm date. Amin Vahdat only told Bloomberg “next year”.

What the protection costs in performance

For Nvidia, there are now hard numbers. In an arXiv paper, Amean Asad and Ansgar Grunseid tested a server with eight B200 GPUs and Intel TDX, each time with and without protection on the same machine. The main cost factors:

• PCIe: copies between CPU and GPU go through encrypted intermediate buffers, known as bounce buffers. Transfer speed to the GPU drops from 51.6 to 3.5 GB/s. That mainly slows down model loading.
• Every GPU command costs about 12 microseconds extra (15.7 instead of 3.45 µs). That slows down text generation when the software sends many small commands one at a time.
• NVLink encryption: when GPUs compute together (all-reduce), throughput drops from 185 to 156 GB/s.

With unmodified default software such as the SGLang inference engine, a single-GPU test lost 34 to 39 percent of its throughput. With full CUDA graphs, which bundle many GPU commands into one package ahead of time, plus software tweaks, the loss was under 1 percent. A large MoE model on eight GPUs came in at about 3 percent. Training across eight GPUs costs 11 to 13 percent, and as much as 24 percent with FP8. There are further limitations: GPU profiling tools are disabled, GPUDirect RDMA for fast server-to-server connections is blocked, and each B200 reserves about 700 MB of memory for the protection. The authors stress that they didn't evaluate the security itself.

The gaps: soldering irons, forged seals, misconfigurations

The biggest weakness is physical. In TEE.fail (October 2025, Georgia Tech, Purdue, Synkhronix), researchers clipped a so-called interposer, an eavesdropping device costing under $1,000, between DDR5 memory modules and the motherboard. Because TDX and SEV-SNP encrypt deterministically, the same plaintext always produces the same ciphertext, and that lets attackers draw conclusions. According to the project page, the researchers used this to steal Intel's attestation key and forge TDX attestations. From an SEV-SNP VM on Zen 4 and Zen 5 EPYC, they extracted signing keys even with “Ciphertext Hiding” enabled. Zen 5 is the architecture of the 5th EPYC generation, the same one inside the TPU 8tx hosts. Nvidia's GPU protection falls too: because its attestation reports aren't bound to a specific CPU VM, an attacker can fake that an AI workload is running protected while it's actually computing in plaintext, the researchers say. The attack requires physical access and root privileges. Before that, the Battering RAM (KU Leuven/Birmingham, interposer for under $50) and WireTap attacks had already cracked DDR4 systems. The vendors' response: Intel and AMD declare physical attacks out of scope, and according to bulletin SB-3040, AMD plans no countermeasures. According to the researchers, Nvidia recommends only trusting servers in physically secured environments.

Then there are implementation mistakes. In its audit of WhatsApp's Private Processing, Trail of Bits found 28 issues, eight of them severe. Attestation reports could be reused indefinitely, and the firmware patch level was merely claimed, not verified. Meta has fully fixed 16 of the issues and partially fixed four. Developer Juraj Bednar ran his own checks on several providers and concluded that attestation covers “parts of the pipeline, not the whole thing”. At one provider, encryption ends at the router instead of the compute node.

Apple, Google, Meta, Cohere, Anthropic: who verifies whom?

Apple goes the furthest in its blog post on the expansion. The company relies on two roots of trust from different vendors and counts everything from firmware to app as part of the trusted computing base. Apple explicitly doesn't rely on confidential computing alone. For Google, NCC Group notes, according to The Register, that the data is largely safe from everyone except Google itself: the risk is limited “unless Google as a whole organization decides to do so”. Kaveh Razavi of ETH Zurich calls the TPU platform “more opaque” than SEV-SNP. In its security model, Cohere openly lists the residual risks: side channels, physical attacks, metadata such as the timing and size of requests, and the fact that the operator can stop the environment at any time.

What companies and GDPR officers should ask

In its position paper from October 2025, the French agency ANSSI concludes that confidential computing is “not secure enough” against a hostile administrator. On its own, it also isn't enough for the SecNumCloud requirements in section 19.6. According to Borncity, Germany's BSI added confidential computing to its C5:2026 cloud catalog for the first time in April 2026. The report doesn't say how binding that is. Our take on the US CLOUD Act: the law requires providers to hand over data they control. The technology can prevent a provider from ever having plaintext in the first place. But whoever controls the reference values, software updates and key servers could, in theory, push modified software. That's exactly what NCC Group's remark is getting at. The legal question hasn't been settled.

Concrete questions to ask providers:

1. Which stops are covered: CPU memory, PCIe, HBM, chip-to-chip links?
2. Does the accelerator provide its own attestation, and is it bound to the CPU VM and the session?
3. Can I or an independent third party verify the attestation without trusting one of the provider's services?
4. Who runs the key server, and who decides which software counts as “good”?
5. Are there reproducible images, a transparency log and public audits with open findings?
6. Where are the servers, who has physical access, and how does the provider deal with attacks like TEE.fail?
7. What performance hit and loss of features apply to my configuration?

The bottom line

Confidential AI shifts trust, it doesn't eliminate it. Instead of trusting an admin, you trust the chipmakers, their firmware, the attestation service and the physical security of the data center. Against curious employees, compromised hypervisors and other customers on the same machine, the technology raises the bar noticeably. Against an operator who approves the software itself, and against anyone with access to the hardware, it offers only limited protection. Our take: with encrypted NVLink and a documented threat model, Nvidia is more transparent with Rubin. That said, Nvidia doesn't mention HBM encryption either, and the rack-level details are so far just vendor claims. For the TPU 8tx, only the CPU side is documented so far. Google still has to show that its solution is more than a well-shielded host CPU with an encrypted line to the TPU.