The 9 October arXiv preprint QTestBench tests the tests written by large language models. A test file merely running is not treated as sufficient: the framework also measures whether generated tests accept correct solutions and catch faulty ones, using independent benchmark checkers as the reference. The paper’s first page lists the authors’ affiliations with NYU Abu Dhabi’s eBRAIN Lab and its Center for Quantum and Topological Systems. The work is presented as a preprint, not as a peer-reviewed publication.

The authors evaluate five models on eight tasks from PennyLane QHack and a fixed set of ten Qiskit HumanEval tasks. They report that the best test generator reached a test-quality score of 0.71 on QHack and 0.34 on Qiskit HumanEval; these scores apply to the paper’s tasks and scoring protocol, rather than representing general quantum-programming success rates. The study also uses checker-verified synthetic changes for three Qiskit tasks where naturally occurring faulty solutions were absent. Its contribution is an evaluation method showing why executable generated tests alone do not establish that those tests are trustworthy.