ThinkingBox: Database checks become the reward for training AI agents

What this is about

Microsoft and Hugging Face have published a second blog post on ThinkingBox, a benchmark for AI agents handling business workflows backed by databases. The first post covered how agents claim they're done while the database says otherwise. Now the same check is used as a reward for reinforcement learning (RL, learning by being rewarded for good outcomes). The ThinkingBox-Training code is on GitHub.

The authors report that for Qwen3.8-27B, the average success rate rose from 51.70% to 60.78%, and the number of tasks solved in all 20 attempts rose from 38 to 107. That took just 50 GRPO steps (GRPO is an RL method that compares multiple answers to the same task).

The idea: the grader becomes the teacher

ThinkingBox-Bench contains 507 tasks. Each one starts a real backend, gives the agent tools (via MCP) and a simulated user with private context. At the end, automated tests check the final state of the database, not the individual tool calls. That leaves the agent free to find new routes to a solution.

Since such a test usually returns 1 or 0, it works directly as a reward. Most rewards come from deterministic tests. A minority also use an LLM judge (GPT-5 Chat) for narrow rubric questions. Importantly, training used different tasks from the benchmark.

Which tasks are suited for training

With GRPO, a learning signal only appears if some of the attempts at a task (8 per task here) succeed and some fail. If all fail or all succeed, no trajectory gets an advantage. Each attempt is also expensive: a backend, a tool session and a model playing the simulated user.

So the authors estimate difficulty up front, relative to the base model ("offline policy-relative difficulty"). Each task is tried n times, and the success probability is estimated with a Jeffreys beta distribution. Tasks fall into bands:

The final pool contained 187 tasks. Selection happens once up front and stays fixed. The authors chose not to re-check during training.

Training setup

Each attempt is a complete episode with a fresh backend and an isolated MCP session. Each update used 18 groups of 8 rollouts, or 144 episodes, on 24 H100 GPUs (3 nodes of 8). The setup used token-mean GRPO with full-parameter optimization, AdamW with a learning rate of 1e-6 and no weight decay, for 50 steps. Groups with no variance were replaced DAPO-style. System errors such as backend timeouts were not counted as failures and were refilled instead. Training and evaluation must interpret model output identically (code in trainer/core/tb_interpret.py).

Results

pass@1 and 20/20 rose in all five domains (retail, auto insurance, travel, neobank, consulting). pass@20 was mixed. The biggest drop was in auto insurance (−7.0 points), and the biggest 20/20 gain was in retail (+24.5 points). The authors read this as "sharpening": the model didn't get broader, it got more repeatable, and some rare solution paths disappeared.

One example: in a hotel downgrade that requires manager approval, the base model changed the booking without approval in 15 of 20 attempts. The trained model left the booking and payment untouched in all 20.

Comparison (pass@1 / tasks with 20/20): Claude Opus 5.5 67.16% / 241, GPT-5.4 65.36% / 128, GPT-5.6 Sol 61.91% / 82, Qwen3.8-27B RLFT 60.78% / 107, GPT-6 Astra 58.31% / 231, Kimi-K3 57.37% / 68. Average success and repeatability produce different rankings.

Cost

The authors calculate cost per reliable task as the estimated cost of the 507×20 campaign divided by the number of tasks with 20/20. The campaign got cheaper, from $925.80 to $855.58 (−7.6%). Cost per successful attempt fell from $0.177 to $0.139, and cost per reliable task fell from $24.36 to $8.00. That is close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45) and Claude Opus 5.5 ($7.80). These figures are a comparative index, not real invoices, and they exclude training and probing costs.

Build it yourself

The microsoft/thinkingbox-training repository includes data selection, rollouts, rewards and Verl integration (Verl 0.9.0). It was tested on Linux/WSL with Python 3.12 and uv. You need NVIDIA GPUs with matching CUDA/NCCL, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and the judge. The reference configuration uses 3×8 H100s. The public example tasks are not the private 187-task pool, so you have to supply your own data, kept separate from the benchmark. Evaluation ran on vLLM with eight A100s.

Takeaways

The work shows that a simple, executable end-state test works well as a reward and clearly improves repeatability for a 27B model. There are caveats. The training pool is private. Training and test share a framework, simulated user, grading style and some tools, which limits how well results transfer. Coverage dipped slightly. All figures come from the authors, and the tasks are synthetic reconstructions. Their practical advice: check the difficulty distribution before training, align parsers between training and evaluation, and report both coverage and repeatability.