Issue #011 — A quiet in-window week, so three catch-up audits — a 105-study speech meta-analysis, a 42-study passive-sensing scoping review, and an LLM mental-health safety benchmark — widen the measurement-discipline thesis across three modalities at once.
Weekly Intelligence · Week 11 · 17 July 2026 · Issue #011
After last week's brief in-window pair, the strict 7-day window went quiet again (10–17 July), so this issue runs on the catch-up track — three 2025–2026 audits that, read together, extend the newsletter's measurement-discipline thesis across three modalities at once: speech, wearable passive sensing, and large-language-model safety. Each reports strong headline numbers and the same load-bearing caveat underneath them.
Executive Summary
For the first time since Issue #010's two in-window digital-phenotyping papers, no new wearable, speech, multimodal, facial-CV, NLP, or digital-phenotyping primary detection result — and no new regulation or industry development — cleared the strict 7-day window (10–17 July). The strongest in-window candidate that surfaced, a Psychiatry Research general-population digital-phenotyping study (Sameh et al.), is a primary detection model published 25 May and so falls outside both the window and the backfill track's high-leverage-type gate; it is logged as deferred rather than padded in. Rather than stretch the issue, this week leans on the catch-up shelf, which happens to hold three results that line up into a single argument about measurement discipline — and, unusually, they make it from three different modalities. First, a JMIR Mental Health systematic review and meta-analysis (Maran et al., 22 Oct 2025) of 105 automatic-speech-analysis depression studies, whose pooled accuracy spans a wide 0.66–0.81 with I² heterogeneity of 94–99% and 47.6% of studies at high risk of bias — and whose verdict is that speech analysis "should be considered a complementary method," not a standalone diagnostic. Second, a JMIR scoping review (Shen et al., 14 Aug 2025) of 42 passive-sensing studies in clinically diagnosed populations, which quantifies the field's small-sample problem directly — a median of 60.5 participants, external validation in just 1 of 42 studies (2%), and anonymization addressed in only 6 of 42 (14%). Third, MHSafeEval (arXiv, 20 Apr 2026, preprint), a role-aware safety benchmark showing that the multi-turn, cumulative failure modes of LLM mental-health counselors are "systematically missed by existing static benchmarks." The through-line the newsletter has tracked since Issue #008 — impressive aggregate accuracy, unproven generalization — now has a fourth face: the very instruments used to measure safety and performance are themselves too coarse to trust. The honest read: the in-window frontier was static again, but the catch-up track keeps sharpening the same point from new angles.
Key Metrics
| Metric | Value | Source |
|---|---|---|
| Speech-analysis depression meta: studies / pooled accuracy range / high-bias share | 105 / 0.66–0.81 / 47.6% | Maran et al. · JMIR Ment Health · 22 Oct 2025 |
| Passive-sensing scoping review: studies / median participants / external-validation rate | 42 / 60.5 / 1 of 42 (2%) | Shen et al. · JMIR · 14 Aug 2025 |
| LLM safety benchmark: failure modes missed by static tests | role-dependent, cumulative, multi-turn | MHSafeEval · arXiv · 20 Apr 2026 |
Speech & Vocal Biomarkers
A 105-study meta-analysis: speech depression detection is "complementary," not diagnostic
Patricia Laura Maran, María Dolores Braquehais, Alexandra Vlaic, and colleagues published a systematic review and meta-analysis in JMIR Mental Health synthesizing 105 studies of automatic speech analysis (ASA) for detecting depression, drawn from eight databases across January 2013 to April 2025 and pooled with a three-level meta-analysis. The headline numbers look strong at the top of their range — pooled highest accuracy 0.81 (95% CI 0.79–0.83), sensitivity 0.84, specificity 0.83 — but the review's contribution is the spread and the caveats beneath it, not the ceiling. The pooled lowest estimates fall to accuracy 0.66, sensitivity 0.63, and specificity 0.60, and the heterogeneity is extreme: I² between 94% and 99%, driven by divergent populations, feature sets, and dataset characteristics, with Teager-Energy-Operator features and deep neural networks the best-performing configurations. Nearly half the corpus — 47.6% — carried a high risk of bias in at least one domain, most often insufficient preprocessing documentation and inadequate sample sizes. The authors' conclusion is deliberately deflationary: ASA "should be considered a complementary method" rather than a standalone diagnostic, and the field needs more high-quality, peer-reviewed work before clinical use. For this newsletter the meta-analysis is the speech-modality complement to the audits logged in Issues #008–#010: where Lee et al. (Issue #009) found no single wearable biomarker is sufficient and Alam et al. (Issue #010) found digital-phenotyping implementation too heterogeneous to reproduce, this shows the speech literature carries the same signature — a real signal, a believable top-line number, and a variance structure wide enough that the pooled figure is a summary of disagreement rather than a stable estimate. It also retro-frames the tidy 23-feature parsimony story from Lin et al. (Issue #010): parsimony is exactly the discipline a 94–99% I² field is missing.
Source: Maran PL, Braquehais MD, Vlaic A, et al. · JMIR Mental Health · 22 Oct 2025 · 10.2196/67802
📅 Catch-up — published 22 October 2025, outside the weekly window
Wearable Biosensors & Digital Phenotyping
A 42-study scoping review puts a number on the small-sample problem: median n = 60.5
ShiYing Shen, Wenhao Qi, Jianwen Zeng, and colleagues published a scoping review in the Journal of Medical Internet Research of 42 peer-reviewed studies using passive sensing from wearables or smartphones with machine learning to monitor clinically diagnosed mental disorders, searched across seven databases from January 2015 to February 2025. The studies were mostly cohort designs (23/42, 55%), concentrated on depression (55%) and anxiety (21%), and leaned on wrist-worn devices (76%) collecting heart rate (67%), movement index (60%), and step count (40%). What makes the review worth logging is that it quantifies the constraints this newsletter has argued qualitatively: the median sample was just 60.5 participants (IQR 54–99), 76% of studies used a single device, 45% monitored for under seven days, external validation appeared in only 1 of 42 studies (2%), and just 14% addressed anonymization. Despite those limits, top-line accuracy again looks excellent — a CNN-LSTM model reached 92.16% for anxiety detection — which is precisely the pattern the audit track keeps surfacing: high in-sample numbers atop thin, un-validated, privacy-light foundations. The authors' prescription echoes the field's emerging consensus — standardized protocols, larger longitudinal studies (≥3 months), explainable models, multimodal fusion, and real data-privacy frameworks. Read against the Alam systematic review from Issue #010, this is the sharper, more numeric cut of the same finding: Alam named the heterogeneity, Shen et al. put the median sample size and the 2% external-validation rate on the table. The 92% anxiety accuracy and the 2% external validation belong in the same sentence — the first is why the field is excited and the second is why the excitement is not yet earned.
Source: Shen S, Qi W, Zeng J, et al. · Journal of Medical Internet Research · 14 Aug 2025 · 10.2196/77066
📅 Catch-up — published 14 August 2025, outside the weekly window
AI/ML & Safety Benchmarks
MHSafeEval: static safety benchmarks miss the multi-turn failures that actually harm
A team introduced MHSafeEval, a role-aware framework for evaluating mental-health safety in large language models, built around a taxonomy (R-MHSafe) that characterizes clinically significant harm by the interactional role an AI counselor slips into — perpetrator, instigator, facilitator, or enabler — crossed with clinically grounded harm categories. Rather than score isolated single-turn prompts, the framework formulates safety assessment as trajectory-level discovery: it iteratively generates, evaluates, and refines client–counselor interactions through naturalistic adversarial multi-turn attacks, retains high-harm exchanges in a structured Harm Archive, and uses an LLM clinical safety judge to give graded severity feedback while steering toward under-explored failure regions. The load-bearing result is that this surfaces "substantial role-dependent and cumulative safety failures that are systematically missed by existing static benchmarks," and materially improves failure-mode coverage. For this newsletter the paper is the safety-layer counterpart to the Ishikawa & Duke benchmark audit from Issue #009: where that work showed detection leaderboards are unstable across reseeding and external transfer, this shows safety leaderboards are incomplete along a different axis — harm that only emerges across a conversation, not in any single turn, and that depends on the role the model drifts into. It sharpens the WHA79 "warm handoff, not a disclaimer" standard tracked since Issue #007 and the RAG suicide-risk stratifier from Issue #008 into a measurement question: a system that passes a static safety screen can still fail cumulatively in deployment, so a static pass is weak evidence of a safe counselor. It is a preprint, and its harm judgments themselves lean on an LLM judge — a dependency worth watching — but it lands squarely on the through-line that the field's yardsticks, for performance and now for safety, are less firm than the headline pass-rates imply.
Source: MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models · arXiv preprint · 20 Apr 2026 · arXiv:2604.17730
⚠️ Preprint — not yet peer reviewed 📅 Catch-up — published 20 April 2026, outside the weekly window
Forward Outlook
- Near-term: Three catch-up figures now travel as a set — the ASA meta-analysis's 94–99% I², the passive-sensing review's 2% external-validation rate, and MHSafeEval's "static benchmarks miss cumulative harm." Each is a citable line a reviewer can attach to a single-cohort claim, whether the modality is voice, wrist sensor, or chatbot. The next result worth flagging is the first detector or counselor to report stable, externally-validated, multi-turn-tested performance rather than a headline in-sample number — the artifact the field keeps asking for and rarely producing.
- Mid-term: The Maran "complementary, not standalone" verdict and the Shen "median n = 60.5, ≥3-month studies needed" prescription point the same way as the Issue #009–#010 arc: passive and vocal signals earn their keep as adjuncts routed into human care, not as autonomous screeners. If that framing holds across speech, wearables, and LLM counseling at once, the deployment shape converges on triage-grade, human-in-the-loop use — the same destination the governance track (WHA79, Issue #007) has been pressing from the policy side.
- Long-term: With a fourth consecutive theme — external validation (Issue #008), single-marker insufficiency and unstable benchmarks (Issue #009), implementation heterogeneity (Issue #010), and now measurement coarseness across modalities and safety (Issue #011) — the field's binding constraint looks less like model capacity and more like the trustworthiness of its own instruments. Accuracy and fluency will keep rising; whether the meta-analyses, scoping reviews, and safety benchmarks measuring them are disciplined enough to believe is the open question, and no new architecture closes it — only bigger samples, external validation, standardized reporting, and multi-turn safety evaluation do.
Sources used: 3 (0 in-window · 3 catch-up) · Week 11 · Next issue: 24 July 2026