COSMIC
COSMIC v1 evaluation
Every COSMIC figure on the main page, in full: one frozen model, scored in a single pass.
How to read these numbers
Detection is reported at a fixed, low false-positive threshold, not as accuracy or AUC. The damaging failure is calling someone’s own recording AI-generated, so the rate at the deployed threshold is the number that matters.
Every threshold is reported with the rate it actually realised, and results are per generator and per human corpus, never pooled.
Detection by generator
Share of each generator’s recordings detected at the fixed threshold. Intervals (95%) resample whole artists and source recordings together. Generators with fewer than 100 recordings are not reported.
For a generator held out of training, mean detection across five held-out families was 75.3%: the figure to expect for a generator released after training.
Other detectors on the same recordings
COSMIC, four open-source detectors and one commercial detector scored the same 4,120 AI recordings (35 generator versions) and 6,000 human recordings in October 2026. The open-source detectors each read the same decoded audio and ran to their authors’ recipe; the commercial detector, SH Labs (version 4.3, its latest), was sent the same files.
The comparison is at a matched false-positive rate: each threshold is set to flag 1% of the human recordings. That is a comparison, not an operating point, so the table also shows each detector at its own published threshold.
Every detector after ordinary processing
False positives
Realised false-positive rate at the same threshold, on 60,474 human recordings in three groups. Per artist is the mean of per-artist rates; per track is false positives divided by tracks. The highest per-artist cell, 1.47%, is the one quoted.
The fourth row is a fresh population: 3,237 tracks from 817 creators absent from development data, scored once. It reads 0.95× the development rate for the same kind of music.
The last row is 3,163 clips under 60 seconds from 1,395 unseen creators. It realised about 2%, and 4–5% on clips under 20 seconds, which COSMIC now declines to score. It is never combined with the longer recordings.
No 0.1% operating point is published. No human population available produces enough false positives at that depth for the figure to mean anything.
Re-encoding and pitch or speed changes
The same recordings before and after each step, paired within track: 14,919 AI and 13,881 human recordings.
Production edits
Change in detection rate, in percentage points, for the same AI recordings after each edit. These 15 are ordinary mixing, mastering and distribution. The wider 70-manipulation evaluation is used to harden COSMIC and is not published.
Localized scan
The optional localized scan located 98.1% of AI passages lasting 45 seconds or longer in our evaluation. This measures passage location, rather than whole-track detection, and is not a promise about every recording.
The held-out check
A held-out set, never previously scored, was evaluated once after every figure above was fixed, against pass/fail criteria written in advance: 14,419 AI tracks from 42 generators and 36,982 human recordings. It has not been reopened.
Detection generalised to the held-out set. Every generator that could be compared reproduced its development result; the four largest populations, with 2,400 to 3,000 tracks each, were within 0.4 percentage points.
The false-positive rate was about 1.5× higher.
| Human collection | Development | Held-out |
|---|---|---|
| Collection 2, 1% threshold | 0.83% | 1.28% |
| Collection 2, stricter threshold | 0.50% | 0.73% |
| Collection 3, 1% threshold | 0.27% | 0.40% |
| Collection 3, stricter threshold | 0.13% | 0.21% |
Development figures use only the subset comparable with the held-out set, so they differ from the figures above.
For scale, splitting the human recordings into five blocks of whole artists gives 0.62× to 1.52× the average false-positive rate. The held-out 1.48–1.57× sits near the top of that range; the fresh creator pool did not inflate (0.95×).
The held-out set does not include Collection 1, so it could not test the 1.47% figure directly. The fresh pool is the same kind of music as Collection 1, so it is not a like-for-like retest of Collection 2 either. A fresh pool like Collection 2 has not yet been collected.
External sources
Industry figures on the main page come from these publications, not from COSMIC’s evaluation.