Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale

Meta reports that its Catching JiTTests system, which uses LLMs to generate executable tests that fail on risky code diffs before they land, is now part of its production infrastructure rather than a research prototype. The system combines intent-aware and "dodgy diff" test-generation workflows with an LLM-as-Judge ensemble and a rule-based assessor, and uses a human-on-the-loop model where engineers review outcomes at an aggregate level. Meta says 589 engineers (64% of outreach) engaged with the system, which showed a 60% true-positive rate and helped catch 74 serious privacy bugs at the diff stage in the first half of 2026, lowering privacy SEVs by 20%.