A new model arrives with a launch post, a chart of rising scores and a claim to be "state of the art". Within a day, the chart is everywhere. Your question is narrower: is it worth switching, testing, or ignoring?
The launch post can't answer that alone. The developer wrote it, about its own model, using results it chose to show. That doesn't make it wrong, but it makes it one source among several. If you need a refresher on what the model underneath is doing, start with our large language model explainer.
1. Read the scores for their settings
Every number in a launch table comes from a particular setup. Our benchmarks explainer covers what the common tests measure and why scores drift. For a launch post, three questions do most of the work.
What was the setup? Look for the prompt format, the number of attempts counted, any tools or agent scaffolding, and the reasoning or "effort" setting. A score at the highest effort level may not match the default you will actually call, and higher effort usually costs more time and tokens. Our guide to how reasoning models work explains that trade.
Are comparisons like for like? A chart that sets the new model at maximum effort beside a rival at default settings misleads even if every number is accurate. Check the footnotes.
What is missing? The developer chooses what to report. A benchmark that led last year's launch and has vanished from this one is information too.
2. Separate vendor-run results from independent ones
The most useful evidence often arrives after launch day, from someone who didn't build the model.
In April 2026, NIST's Center for AI Standards and Innovation (CAISI) evaluated DeepSeek V4 Pro and reported that it "scores better on DeepSeek's self-reported evaluations than on CAISI evaluations."[1] On DeepSeek's chosen benchmarks, the model looked roughly level with leading US models. On CAISI's suite, which included held-out and non-public tests, it performed closer to a US model released about eight months earlier.[1]
The report also shows how careful an independent evaluator must be. CAISI says it used the developer's recommended settings, reproduced one of DeepSeek's own results to rule out configuration errors, and chose its benchmark suite in advance.[1] Even so, it flags a gap with other evaluators:
"CAISI scores on SWE-Bench Verified tend to be lower than those of other evaluators, likely due to system prompt, scaffolding, and token budget differences."
— NIST CAISI, CAISI Evaluation of DeepSeek V4 Pro (May 2026).[1]
The lesson is not that developers' numbers are false. Setup choices move results enough that only figures with disclosed, matched settings can be compared.
Government institutes also test some models before release. When the UK and US AI safety institutes jointly evaluated OpenAI's o1 in 2024, they adapted their agent setup around tool-calling problems and stated that "this report makes no claims about the performance of other versions of o1."[2] Every evaluation describes one version, under one setup.
3. Open the system card, and check which model it describes
A system card or model card is the developer's technical document for a release. The idea traces to a 2018 paper proposing "model cards" that report evaluation results across conditions and state a model's intended uses.[3] Frontier developers now publish far longer versions; Anthropic says its cards document "the capabilities, safety evaluations, and responsible deployment decisions" for its models.[4]
Read the card for what the launch post skips: known limitations, safety and misuse testing, any external testing, and, above all, which version was tested. OpenAI's o1 card scoped its evaluations to specific checkpoints and later noted that an improved o1, launched after publication, was not covered.[5] It also warned:
"…exact performance numbers for the model used in production may vary slightly depending on system updates, final parameters, system prompt, and other factors."
— OpenAI, OpenAI o1 System Card (December 2024).[5]
Cards vary in what they disclose. The 2025 Foundation Model Transparency Index found average developer transparency fell from 58 to 40 out of 100 in a year, with companies most opaque about training data, compute and post-deployment impact.[6] A thin card is not proof of a problem, but it leaves more for you to test.
4. Check price, speed, limits and lifecycle
A developer's models page is usually where the terms live. Anthropic's, for example, lists each model's input and output price, comparative latency, context window, maximum output, training data cutoff and earliest retirement date.[7] Check:
- Price per input and output token, plus any cached-input or batch rates. Hidden reasoning tokens can add to the output bill.
- Latency. Time to first token, the wait from sending a prompt to receiving the first token, matters most for chat and voice.[8] Measure it on your own requests.
- Context window and maximum output. A large window shows what fits, not what the model reliably uses; see our context windows guide.
- Lifecycle. OpenAI commits to at least six months' notice before retiring generally available models, but says preview models "may be retired with much shorter notice, such as 2 weeks."[9]
5. Confirm the licence and availability
Licence. For open-weight releases, read the licence itself, not the word "open" in the headline. On Hugging Face, the licence sits in the model card's metadata, with custom licences marked "other" and linked.[10] Our guide to open vs. closed AI models explains what the differences mean. For hosted models, the commercial terms and usage policy play that role.
Availability. Note which API, cloud platforms, regions and consumer plans actually offer the model. Announced is not the same as available to you.
6. Run your own small test
The checks above narrow the field. Only a test on your own work decides it.
Keep it small and fair: 20 to 50 real cases with sensitive details removed, expected outcomes written down first, the same prompt and settings for each model, and the model identifier and date recorded. Include awkward cases and read every failure. Our benchmarks explainer sets out the method. Test at the settings you would actually pay for.
AI Chronicle
A worked example (hypothetical): a bookkeeping firm reads a launch post
This scenario is invented to illustrate the checklist. No model was tested and no results are reported.
A small bookkeeping firm uses a model to pull supplier, date, total and tax from scanned invoices. A new model launches with large claimed gains on document understanding and a lower output price.
The footnotes show the scores were run at the highest reasoning setting; the firm uses defaults. No independent evaluation exists yet. The system card covers a pre-release checkpoint and says nothing about scanned documents. Once reasoning tokens are counted, the output saving shrinks. And the model is labelled "preview".
None of this rules the model out; it means the chart can't decide. The firm runs 40 past invoices, including faded scans, foreign currencies and handwritten totals, through both models at default settings, scoring each field against the known value. Because it is a preview, they would wait for a stable release before switching, whatever the result.
Questions readers commonly ask
Are developers' own benchmark results unreliable?
They are a starting point. Their weakness is selection and setup, since the developer picks both. Treat them as claims until independent results or your own agree.
Where do independent evaluations come from?
Government institutes such as the UK AI Security Institute and NIST's CAISI publish some.[2][1] Academic groups and public leaderboards are others. Check which version and settings each tested.
What is the difference between a model card and a system card?
The terms overlap. "Model card" comes from the 2018 proposal for short evaluation documents;[3] frontier developers often say "system card" for longer reports covering capabilities, safety testing and deployment decisions.[4]
Should I switch as soon as a better model launches?
Only after it passes your own test at the settings and price you would use, and after checking whether it is a preview.[9]
The next step
You now have six checks for any launch: settings, who ran it, the system card, the terms, the licence, and your own test. See them applied to three assistants in our Claude vs. ChatGPT vs. Gemini comparison.
Sources and evidence note
This analysis relies on government evaluation reports, developer system cards and documentation, an academic transparency index and the original model cards paper. Results quoted from evaluations describe the model versions and setups tested at the time; they are not current rankings. Prices, limits and retirement dates are not stated here because they change; the article points to where they are published. The bookkeeping scenario is illustrative. No hands-on model test was run for this article. All sources accessed 16 October 2026.
Sources
- NIST Center for AI Standards and Innovation (1 May 2026, updated 2 May 2026), CAISI Evaluation of DeepSeek V4 Pro
Used for the self-reported versus CAISI comparison, roughly eight-month comparison, developer-recommended settings, reproduction check, pre-committed suite and the 24-word SWE-bench quotation.
- UK AI Security Institute (then AI Safety Institute) and US AI Safety Institute (18 December 2024), Pre-Deployment evaluation of OpenAI's o1 model
Used for the joint pre-release testing, tool-calling caveat and the 13-word version caveat.
- Mitchell and colleagues (2018, rev. 2019), Model Cards for Model Reporting
Used for the origin and purpose of model cards: evaluation across conditions and intended-use disclosure.
- Anthropic, Model system cards
Used as an example of a developer's system card index and for the quoted description of what the cards document.
- OpenAI (5 December 2024), OpenAI o1 System Card
Used for checkpoint scoping, the footnote on the later updated o1, and the 23-word quotation on production performance.
- Wan and colleagues (December 2025), The 2025 Foundation Model Transparency Index
Used for the fall in average score from 58 to 40 and areas of greatest opacity.
- Anthropic, Claude Platform documentation, Models overview
Used as an example of the fields a models page lists (price, latency, context, maximum output, cutoff, retirement); no values reproduced.
- Anthropic, Claude Platform documentation, Reducing latency
Used for the definition of time to first token.
- OpenAI API documentation, Deprecations
Used for minimum notice periods (at least six months for generally available models) and the quoted preview-model wording.
- Hugging Face Hub documentation, Model Cards
Used for licence metadata on model cards and the "other" custom-licence convention.
This article was researched and drafted with the help of AI tools, then edited and fact-checked by the AI Chronicle editorial team. Illustrations are original artwork created for AI Chronicle with AI image generation.