Apple's health AI study measures trust, not whether people actually get healthier
Apple is letting up to 200,000 iPhone users in the US ask health questions to OpenEvidence, an AI built for doctors. We set the study design alongside the Oxford study in Nature Medicine and OpenEvidence's contradictory test results. The upshot: the study can show whether people trust the AI. Whether it leads them to make better health decisions is something it can barely prove in its current form.

What's going on
On November 15, 2026, Apple launched the "Apple Health AI Usage Study." Up to 200,000 adults in the US will ask health questions in a dedicated study app. The answers come from OpenEvidence, an AI service used so far mainly by physicians. You can take part if you're at least 21 and have an iPhone 15 Pro or newer running iOS 26. According to Apple, the study aims to show what builds trust in health AI, how good the answers are, whether people understand them and how they act on them in daily life.
Most coverage simply retold the press release. We put three findings side by side: a large Oxford study on conversations between laypeople and AI, the contradictory independent tests of OpenEvidence, and what Apple leaves out about its study design. Our thesis: the study measures trust and satisfaction. Whether people make healthier decisions with the AI is something it can barely show in its currently known form.
What Apple measures and what's missing
According to Apple Newsroom, participants ask everyday questions, get sources from medical journals with every answer, and rate the answers in short in-app surveys. If they choose, they can share data from Apple Health with the app, such as resting heart rate, sleep, activity or lab results. According to Martin CID, participation lasts up to twelve months. That includes an intake survey, monthly check-ins and additional questionnaires around doctor visits.
Apple has put real effort into privacy, at least by its own account. The iPhone first strips identifying details from each question. It then passes through a cloud run by contractor UST Global, where it's separated from the participant ID. OpenEvidence deletes questions and answers within 24 hours. The study data is not supposed to feed AI training. There's no independent audit of these promises.
The guardrails, by contrast, are thin. The app is supposed to detect concerning questions and then point users to doctors or support services. Nobody is monitoring live. According to CNET, as cited by Martin CID, the study team won't reach out even if someone asks about self-harm.
What Apple doesn't mention matters more:
- no control group, meaning no comparison group that, say, googles as usual or gets no help at all
- no predefined endpoints, meaning no target measures against which success or failure is judged
- no ethics approval: Apple doesn't mention review by an ethics committee (an IRB in the US). We searched the US trial registry ClinicalTrials.gov for "Apple Health AI" and found no entry.
- no date for results and no commitment to publish them in a peer-reviewed journal
Finding 1: Health AI fails at conversation, not knowledge
In February 2026, a team led by Andrew Bean and Adam Mahdi (Oxford Internet Institute) published a randomized, preregistered study in Nature Medicine. Preregistered means the research question and analysis were made public in advance. 1,298 adults from the UK worked through case scenarios designed by doctors. Some used GPT-4o, Llama 3 or Command R+; the control group used their usual resources, mostly a search engine.
| Situation | Correct condition identified | Correct course of action chosen |
|---|---|---|
| Model alone, with the complete case | 94.9 % | 56.3 % |
| Laypeople in conversation with the model | under 34.5 % | under 44.2 % |
| Control group without AI | no worse than with AI | no worse than with AI |
So the knowledge was in the model, but it got lost in the conversation. Participants left out important details, and the models mixed good advice with bad. According to The Register, two users sent almost identical descriptions of a subarachnoid hemorrhage, a life-threatening brain bleed. One was told to lie down in a dark room. The other was correctly sent to the emergency room. Senior author Mahdi draws this lesson:
"We cannot rely on standardized tests alone to determine whether these systems are safe for public use."
He calls for testing with many different real users, similar to clinical trials for new drugs. This is exactly where Apple's approach is right in principle: the study puts real laypeople in front of real answers.
Finding 2: The tests of OpenEvidence contradict each other
OpenEvidence uses RAG (retrieval-augmented generation). The AI first looks things up in a medical library and then writes an answer with sources. According to an analysis published in JAMA, monthly users grew from 355 in January 2024 to more than 703,000 in June 2026. How good the tool actually is remains disputed:
| Study | Type | Result |
|---|---|---|
| Jagarapu et al., medRxiv, Dec. 2025 | Preprint, not peer-reviewed; 100 hard exam questions (MedXpertQA, 10 answer options each) | Quick search 28–34 %, "Deep Consult" 38–41 % correct; same answer on repeat only 77 and 72 % of the time |
| NYU Langone (Oermann), Nature Medicine, June 2026 | 500 MedQA questions, 500 HealthBench tasks, 100 real physician questions, blinded evaluation | Gemini 3.1 Pro 97.4 %, GPT-5.2 94.2 %, OpenEvidence 89.6 % on MedQA; frontier models win all three parts |
| Feng et al., arXiv, June 2026 | Preprint; 149 physicians rate 620 real OpenEvidence queries plus 187 HealthBench questions | OpenEvidence beats GPT-5.5, Gemini 3.1 Pro and Claude Opus 4.8 in all five categories |
| Artsi et al., npj Digital Medicine, Aug. 2026 | Review of 11 studies | strong on guideline questions, inconsistent on complex cases, fewer fabricated sources than general-purpose chatbots |
| Lotan et al., npj Health Systems, Sept. 2026 | 150 queries, 5 specialties, 4,979 sources checked | no fabricated sources, 3 attribution errors; whether the sources actually support the claims was not checked |
Important context: OpenEvidence demanded that the NYU study be retracted, accusing it of conflicts of interest and methodological flaws. Author Eric Oermann says he stands by the results. The counter-study by Feng et al. relies on queries from OpenEvidence's own platform. According to the blog iatroX, both sides have potential interests at stake. On top of that, different model versions were tested and the evaluation criteria differ: the preprint weights source quality and verifiability heavily, which is naturally where a source-focused tool shines.
Our finding: every one of these studies tests questions from doctors or exam questions. None tests how OpenEvidence performs in conversation with laypeople. Yet according to the Oxford study, that's exactly where the bottleneck is. A tool built for professionals who speak medical jargon is now reaching people who may not know which symptoms they need to mention.
Finding 3: What a controlled study would look like
It can be done differently, and OpenEvidence itself is doing it. The PRECISE trial is registered on ClinicalTrials.gov under NCT07037940. Internal medicine residents and attending physicians work through anonymized cases, randomly assigned to use either OpenEvidence or ChatGPT (GPT-4). Blinded reviewers rate the quality of the clinical reasoning using a validated scoring rubric. According to iatroX, no results are available yet.
| Oxford (Nature Medicine) | PRECISE (OpenEvidence) | Apple Health AI Study | |
|---|---|---|---|
| Participants | 1,298 laypeople (UK) | Physicians | up to 200,000 laypeople (USA) |
| Randomization | yes | yes | not stated |
| Control group | yes (usual resources) | yes (ChatGPT) | not stated |
| Primary outcome | correct diagnosis and correct course of action | quality of reasoning, blinded evaluation | trust, comprehension, satisfaction, self-reported behavior |
| Registration | preregistered | NCT07037940 | not found |
PRECISE, incidentally, doesn't measure whether patients end up better off either. But at least it lets you attribute an effect to a cause. With Apple, it remains unclear whether any change in behavior is due to the AI, to who signed up, or simply to taking part.
Why now, and what it means for Germany
The pressure is enormous. According to OpenAI, more than 40 million people worldwide ask ChatGPT health questions every day. Our math: Apple's entire study amounts to half a percent of a single day's users. Google offers a Gemini coach in its Google Health app that draws on medical records, and since November 13 it has been linking Gemini with Google Wallet in the US. Just how risky such services are is shown by another Nature Medicine study on ChatGPT Health: in real emergencies, it underestimated urgency in 52 percent of cases.
For readers in Germany, all of this remains theoretical for now. The study only runs in the US. And according to its terms of use, OpenEvidence excludes the EU, the EEA and the UK, including users on a VPN or with a foreign account. According to providers CliniAtlas and Vera Health, both competitors, OpenEvidence pulled out on April 27, 2026. The company reportedly cited regulatory uncertainty, above all from the EU AI Act. We couldn't find a statement from OpenEvidence itself. Whether the EU Medical Device Regulation (MDR) also played a role is unclear. According to a Bitkom survey from November 2025, 45 percent of people who search for health topics online had already used AI chatbots for it.
Analysis
Apple's study has genuine strengths. It tests with real laypeople, it's large, and it protects participants' data, as far as can be judged from the outside. That fulfills part of what the Oxford researchers are calling for. But the known metrics are soft factors: trust, comprehension, satisfaction and self-reported behavior. Without a comparison group, predefined endpoints and public registration, you can hardly conclude from this whether the AI helps people make better decisions.
Our take: trust is precisely the tricky part. The Oxford study shows that convincing-sounding answers can be wrong. A high trust score would then be a warning sign, not a success. Whether Apple can tell the two apart depends on details it hasn't published yet. That includes a question raised by its own privacy design: if conversations are deleted after 24 hours and questions are only sorted into categories on the device, can you still analyze where a conversation went wrong? That was the decisive finding in Oxford. Until Apple publishes a study protocol, this is mostly a large field test for a potential product, not an answer to whether health AI is safe for laypeople.
Sources
- Apple Newsroom: Apple launches research study to explore the use of AI for health apple.com
- Martin CID: Apple wants 200,000 iPhone owners to grade AI answers about their health martincid.com
- Chief Healthcare Executive: Apple plans to study how consumers use AI for health needs chiefhealthcareexecutive.com
- iMore: Apple's new health AI app is a study, not your doctor imore.com
- Mashable: Apple needs your help to make health AI better mashable.com
- Oxford Internet Institute: Pressemitteilung zur Nature-Medicine-Studie oii.ox.ac.uk
- Nature Asia: Pressemitteilung Bean et al. natureasia.com
- The Register: AI chatbots are no better at medical advice than a search engine theregister.com
- AAFP Community Blog: AI Health Tools for the General Public Fall Short aafp.org
- medRxiv: Jagarapu et al., Accuracy and repeatability of OpenEvidence medrxiv.org
- arXiv 2512.01191: Vishwanath et al. (NYU) arxiv.org
- arXiv 2606.28960: Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries arxiv.org
- iatroX: Why two 2026 studies reached opposite conclusions iatrox.com
- iatroX: OpenEvidence PRECISE trial iatrox.com
- npj Digital Medicine: Artsi et al., Systematic review of OpenEvidence nature.com
- npj Health Systems: Lotan et al., Reference quality of OpenEvidence nature.com
- Clinical AI Report: Is OpenEvidence available in the UK? clinicalaireport.com
- CliniAtlas: OpenEvidence left Europe cliniatlas.com
- Axios: 40 million people turn to ChatGPT for health care axios.com
- Google Blog: Fitbit personal health coach updates 2026 blog.google
- BetaNews: Gemini can now connect to Google Wallet betanews.com
- Pharmazeutische Zeitung: Wer ist schuld, wenn der Chatbot falsch liegt? pharmazeutische-zeitung.de
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.