Radical transparency
Every metric on AI Model Rankings carries a source and a confidence rating. Every change is logged. Every dispute we've resolved — accepted or rejected — is published below. No competitor publishes their own mistakes; we do.
Accuracy scoreboard
These are small numbers on a young site — we publish them anyway. Growth here is the point.
Tracked data points983Every published metric value, each with a source.
Verified (2+ sources)85%Share of data points cross-checked against a second source.
Re-checked in last 30 days87%Share of data points our source agents have re-verified this month.
Corrections accepted137Times we were wrong and fixed it — all-time.
Median time to resolution27.5 daysHow long a flagged item sits before we accept or reject it.
Open reviews1876 rejected all-time (source didn't hold up).
Freshness report
Our source agents re-check every tracked price daily (06:00 UTC), run discovery/pricing/content agents daily (05:00 UTC), sweep every outbound link weekly on Mondays (07:00 UTC), and run self-audit loops weekly on Tuesdays (08:00 UTC). This is the actual distribution of when published data was last verified — not a promise, a measurement.
Verified within 7 days0 · 0%
Verified within 30 days857 · 87%
Verified within 90 days126 · 13%
Stale or never verified0 · 0%
Dispute ledger
Every correction we've resolved, newest first — whether we accepted it and fixed the data, or reviewed it and left the value as-is.
| Resolved | What was challenged | Type | Outcome | Time to resolve |
|---|---|---|---|---|
| Jul 12, 2026 | Track: Multimodal Output ConsistencyEssential for applications requiring integrated outputs (e.g., image and text), where visual and textual elements must align perfectly in meaning and style. Usually overlooked: Requires complex cross-modal evaluation frameworks and human judgment to assess coherence between different output modalities, which are costly and time-consuming. Source: A custom benchmark involving inputs with text and images, requiring outputs that integrate both seamlessly, scored by human evaluators for overall coherence and faithfulness to the input. | metric | Reviewed — no change | 16.6 days |
| Jul 12, 2026 | Track: Local Deployment AvailabilityEnables enhanced data privacy, reduced latency for specific use cases, and deployment in disconnected or on-premise environments. Usually overlooked: Most commercial models are API-first, and their local versions (if they exist) are often not directly comparable or publicly documented, requiring deeper investigation. Source: Vendor documentation, direct inquiry with sales/support, or community reports for open-source variants and self-hostable options. | metric | Reviewed — no change | 16.6 days |
| Jul 12, 2026 | Track: Information Density ScoreDirectly impacts actual cost-efficiency and user experience by minimizing irrelevant or redundant output, going beyond simple per-token pricing. Usually overlooked: Quantifying 'meaningful' information objectively is inherently subjective and difficult to automate at scale without complex semantic analysis. Source: Provide a set of tasks where brevity and conciseness are valued (e.g., summarization of long articles, fact extraction). Evaluate outputs for redundancy and conciseness via human scoring or comparison against a reference summary/extraction. | metric | Reviewed — no change | 16.6 days |
| Jul 12, 2026 | Track: Demographic Bias IndexCrucial for ethical deployment and regulatory compliance, particularly in applications that could disproportionately affect protected demographic characteristics. Usually overlooked: Developing comprehensive, culturally sensitive bias evaluation datasets and robust scoring methodologies is a significant and ongoing research challenge. Source: Utilize a standard fairness benchmark like ToxiGen or a custom suite of prompts designed to probe for gender, racial, and age biases, evaluated by human annotators or a secondary classification model for undesirable stereotypes. | metric | Reviewed — no change | 16.6 days |
| Jul 12, 2026 | Track: P99 Generation LatencyReveals the maximum wait time users might experience during interactions, which is critical for interactive applications and real-time systems. Usually overlooked: Most benchmarks focus on average or median latency, as P99 requires extensive testing under varying load conditions to accurately capture tail-end performance. Source: Measure latency for 10,000 requests of varying lengths and complexities, then calculate the 99th percentile of response times, excluding network outliers. | metric | Reviewed — no change | 16.6 days |
| Jul 12, 2026 | Track: Fine-tuning Accessibility IndexThe ability to fine-tune a model for specific domains or tasks can drastically improve performance and reduce prompt engineering effort, offering long-term cost savings and specialization. Usually overlooked: It's not a direct inference cost or performance metric and varies greatly by vendor, often requiring a deep dive into platform features and pricing models. Source: A composite score based on: (1) Official fine-tuning API availability, (2) Cost per hour/1M tokens for fine-tuning, (3) Supported methods (LoRA, full finetune), (4) Availability of open-source/community tools for fine-tuning that model. | metric | Reviewed — no change | 17.6 days |
| Jul 12, 2026 | Track: Knowledge RecencyFor applications requiring up-to-date information (e.g., news analysis, market trends), the cutoff date of a model's training data determines its inherent ability to provide relevant and accurate responses without external tools. Usually overlooked: It's often not directly disclosed or is vague ('up to early 2023'), and testing requires tracking real-world events that occurred after potential cutoffs. Source: Ask the model about specific, well-documented public events (e.g., major sports results, political changes, scientific discoveries) that occurred at known dates, progressively moving forward in time to determine the latest event it reliably 'knows'. Confirm with official vendor documentation if available. | metric | Reviewed — no change | 17.6 days |
| Jul 12, 2026 | Track: Guardrail Robustness ScoreFor consumer-facing or sensitive applications, strong guardrails prevent misuse, reputational damage, and compliance issues. Usually overlooked: Testing guardrails can be adversarial and requires specific ethical hacking expertise, and vendors are often reluctant to publicize failure points. Source: Use an automated red-teaming framework (e.g., open-source adversarial prompt generators) to test for common jailbreaks and unsafe content generation, assigning a score based on success rate. | metric | Reviewed — no change | 17.6 days |
| Jul 12, 2026 | Track: Advanced Instruction AdherenceFor automated workflows and integrations, strict adherence to complex formatting, length constraints, or negative constraints (e.g., do not use emojis) is crucial to prevent downstream errors. Usually overlooked: It's hard to quantify beyond a simple 'did it follow the instruction?' and requires specific, nuanced test cases that break typical correctness metrics. Source: Design a suite of 20 prompts, each with 3-5 distinct, complex, and potentially conflicting formatting or negative constraints. Score each model on how many constraints it perfectly follows across all prompts. | metric | Reviewed — no change | 17.6 days |
| Jul 12, 2026 | Track: P99 Inference LatencyFor real-time applications or user interfaces, consistent low latency is more important than average latency, as high outliers can significantly degrade user experience or cause system timeouts. Usually overlooked: Most comparisons focus on average or P50 latency, which doesn't capture the tail-end performance issues that impact critical user journeys. Source: Run 1000 identical simple inference requests (e.g., summarization of 50 tokens) and record the P99 latency from the model provider's API logs or a controlled testing environment. | metric | Reviewed — no change | 17.6 days |
| Jul 12, 2026 | Track: Deep Context UtilizationDetermines the practical effectiveness of large context windows, ensuring that critical information is leveraged regardless of its position within the input. Usually overlooked: Benchmarks often test context window *capacity* (can it hold the tokens?) but not how effectively information placed at various depths is recalled or used for reasoning. Source: Use a 'needle in a haystack' test where crucial, unique information is embedded at varying depths within a long context, and measure the model's ability to extract or correctly use it to answer a question. | metric | Reviewed — no change | 19.6 days |
| Jul 12, 2026 | Track: End-to-End API LatencyDirectly impacts user experience and system responsiveness in interactive applications, where overall request completion time is critical for fluidity. Usually overlooked: Most comparisons focus on Time-To-First-Token, which ignores the total time to receive a complete response, especially for larger or streaming outputs. Source: Perform 100 API calls with a representative prompt and expected output length (e.g., 200 tokens) during peak hours, measuring total wall-clock time from request initiation to full response reception, and report the P95. | metric | Reviewed — no change | 19.6 days |
| Jul 12, 2026 | Track: Visual Reasoning AccuracyCrucial for applications processing visual data, such as content moderation, medical imaging analysis, or creating descriptive alt-text, where understanding image context is key. Usually overlooked: Many benchmarks focus on text-only capabilities, and multimodal evaluations are complex, requiring specialized datasets and robust evaluation methods. Source: Evaluate against established VQA (Visual Question Answering) benchmarks (e.g., VQAv2, OK-VQA) or a custom set of complex visual reasoning prompts with known correct answers, and calculate the accuracy. | metric | Reviewed — no change | 19.6 days |
| Jul 12, 2026 | Track: Harmful Content Bypass RateEssential for public-facing applications where brand safety and ethical use are paramount, preventing misuse and reputational damage from generating unsafe content. Usually overlooked: Testing for safety often involves internal audits or red-teaming, and results are rarely made public due to security concerns or commercial sensitivity. Source: Utilize public red-teaming datasets (e.g., AdvBench, various adversarial prompt collections) and measure the percentage of successful harmful outputs generated by the model. | metric | Reviewed — no change | 19.6 days |
| Jul 12, 2026 | Track: Output DeterminismCritical for applications requiring reproducible results, such as automated testing or consistent data generation, where variability introduces unreliability. Usually overlooked: Most evaluations focus on quality or average performance over consistency, and robustly measuring determinism across diverse inputs is challenging. Source: Run 100 identical prompts with temperature 0 (or equivalent lowest variability setting), measure the percentage of identical outputs, averaged across 10 distinct, short, well-defined prompts. | metric | Reviewed — no change | 19.6 days |
| Jul 12, 2026 | Track: Tool Use RobustnessIn real-world agentic systems, tools often fail or return unexpected results; a robust model can recover or adapt, preventing workflow breakdowns. Usually overlooked: Most tool-calling benchmarks test ideal scenarios with perfect tool definitions and outputs, not the complexities of real-world tool integration. Source: Design 50 scenarios where tools might fail (e.g., API returns error, unexpected data format), or where multiple tools are applicable but one is more efficient. Test the model's ability to handle these gracefully (retry, select optimal tool, provide helpful error messages) and have human evaluators score performance. | metric | Reviewed — no change | 21.6 days |
| Jul 12, 2026 | Track: Multimodal CoherenceFor applications relying on combined inputs (image + text), the ability to deeply integrate information across modalities is crucial for accurate and relevant responses. Usually overlooked: Benchmarks often test modalities in isolation or with simple joint tasks, missing complex interactions and cross-modal reasoning. Source: Present 50 complex multimodal prompts (e.g., 'Analyze this image and describe the cultural significance of the depicted event, referencing the provided text about its history.') and have human evaluators score the response on how well it integrates and reasons across both image and text inputs. | metric | Reviewed — no change | 21.6 days |
| Jul 12, 2026 | Track: Complex Instruction AdherenceModels used for agentic workflows or complex document processing need to precisely follow intricate rules, not just general intent. Usually overlooked: Standard benchmarks often test simple instruction following; evaluating nuanced adherence requires handcrafted prompts with multiple, potentially conflicting, constraints. Source: Design 100 prompts with 3-5 distinct positive and negative constraints each (e.g., 'Summarize this article in 3 sentences, do not mention dates, and include a call to action'). Have human evaluators score adherence to each constraint, then calculate an average adherence score. | metric | Reviewed — no change | 21.6 days |
| Jul 12, 2026 | Track: Guardrail Evasion RateA low evasion rate ensures the model is less likely to generate harmful or inappropriate content, which is critical for public-facing applications. Usually overlooked: Testing guardrail robustness is complex and requires specialized adversarial prompting techniques not typically included in general benchmarks. Source: Use an automated red-teaming tool (e.g., AdvBench, open-source jailbreak datasets) or a team of human red-teamers to generate 500 adversarial prompts targeting specific safety policies and measure the percentage of successful evasions. | metric | Reviewed — no change | 21.6 days |
| Jul 12, 2026 | Track: Response Consistency (Deterministic Tasks)In production systems, predictable and consistent outputs are crucial for automation and trust, even if the model is slightly less 'creative' on a single run. Usually overlooked: Most benchmarks focus on a single best-shot attempt or average performance, not the variability of identical queries. Source: Run the same 100 deterministic queries (e.g., specific data extraction, function generation from spec) 10 times each and measure the percentage of identical outputs per query, then average. | metric | Reviewed — no change | 21.6 days |
| Jul 12, 2026 | Track: Long-Context Retrieval AccuracyA large context window is only useful if the model can reliably find and utilize specific information buried within it, which is critical for RAG and document processing. Usually overlooked: Most benchmarks only test general comprehension over shorter contexts, and the context window size itself doesn't guarantee effective information retrieval. Source: Implement the "Needle-in-a-Haystack" evaluation by embedding a unique, specific fact at varying depths within a 100K+ token document and measuring the model's accuracy in retrieving it. | metric | Reviewed — no change | 23.6 days |
| Jul 12, 2026 | Track: Prompt Injection ResistanceCrucial for deploying models securely in production, protecting against malicious actors bypassing safety features or system instructions. Usually overlooked: This is an adversarial and constantly evolving challenge, making standardized, evergreen benchmarks difficult and often proprietary. Source: Employ a specialized red-teaming team to conduct structured prompt injection attempts and score the model's resistance based on the complexity and success rate of bypasses. | metric | Reviewed — no change | 23.6 days |
| Jul 12, 2026 | Track: Instruction Following ConsistencyReliable automation and structured output generation depend heavily on a model's ability to consistently adhere to complex instructions. Usually overlooked: Quantifying this requires custom, multi-faceted test suites beyond typical academic benchmarks, making it hard to compare broadly. Source: Develop a proprietary benchmark with 100+ prompts containing varied negative constraints and structured output requirements, then measure the percentage of successful adherence. | metric | Reviewed — no change | 23.6 days |
| Jul 12, 2026 | Track: Fine-tuning Bias TransferCrucial for enterprises considering fine-tuning models on proprietary datasets, as it indicates the risk of inheriting or amplifying biases from the training data. Usually overlooked: Requires extensive experimentation with diverse fine-tuning datasets and subsequent bias evaluation, which is a costly and resource-intensive process. Source: Fine-tune the model on a small, specifically constructed dataset known to contain a particular bias, then evaluate the fine-tuned model on an established bias benchmark (e.g., Winogender Schema) to measure the change in bias metrics. | metric | Reviewed — no change | 24.6 days |
| Jul 12, 2026 | Track: Security Vulnerability RateIndicates the likelihood of the model generating code or instructions that introduce common security vulnerabilities, directly impacting the security posture of AI-assisted development. Usually overlooked: Requires specialized security analysis tools and expertise; most coding benchmarks focus primarily on functional correctness rather than security implications. Source: Custom benchmark of coding tasks (e.g., API endpoint generation, data validation logic) fed into static application security testing (SAST) tools like Semgrep or Bandit, counting reported vulnerabilities. | metric | Reviewed — no change | 24.6 days |
| Jul 12, 2026 | Track: Contextual Factuality ScoreDirectly impacts trust and safety for Retrieval Augmented Generation (RAG) applications or domain-specific query answering, where information must be traceable and accurate to provided sources. Usually overlooked: Generic factuality benchmarks exist, but measuring specific, context-bounded hallucination is harder and often requires domain expertise for accurate evaluation. Source: RAG-based evaluation benchmarks where models are given specific documents and prompted to answer questions *only* from those documents, with answers verified against the source material by human experts. | metric | Reviewed — no change | 24.6 days |
| Jul 12, 2026 | Track: Output Format AdherenceCrucial for automated processing and seamless integration into downstream systems, ensuring consistent structural integrity (e.g., complex JSON schema) and preventing parsing errors. Usually overlooked: Most benchmarks focus on content correctness, not strict structural integrity, often assuming simple JSON output is sufficient. Source: Automated validation against a predefined, complex JSON schema (e.g., using 'jsonschema' library) for a diverse set of prompts, measuring the ratio of schema-valid outputs. | metric | Reviewed — no change | 24.6 days |
| Jul 12, 2026 | Track: Domain-Specific Hallucination RateBeyond general knowledge, a model's tendency to hallucinate or misstate facts within a specific, niche domain (e.g., medical, legal) is crucial for trust and avoiding critical errors in specialized applications. Usually overlooked: General factuality benchmarks exist, but domain-specific factuality requires curated expert data and evaluation, which is expensive and time-consuming. Source: Create a curated, domain-specific dataset of factual questions/statements and expected answers. Evaluate model responses for factual accuracy and identify hallucinations, reporting as a percentage of incorrect or fabricated statements. | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: P99 LatencyWhile average latency is often cited, the consistency of response times (especially tail latency) is critical for real-time interactive applications where predictable performance prevents frustrating user experiences. Usually overlooked: Most comparisons focus on Time-To-First-Token or average latency, which hide the occasional severe delays that impact user perception and system reliability. Source: Conduct stress tests by sending a high volume of identical requests over a prolonged period (e.g., 1 hour, 1000 RPM) and record the 99th percentile latency for full response generation. | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: Fine-tuning Data Efficiency (Threshold)For businesses needing custom models, the amount of data required to achieve a target performance level post-fine-tuning directly impacts development costs, time, and ongoing maintenance. Usually overlooked: Benchmarks typically test pre-trained performance; fine-tuning behavior is harder to standardize and compare as it depends on target task and data quality. Source: Fine-tune models on a common target task dataset using varying data subset sizes (e.g., 100, 1000, 10000 examples) and report the smallest dataset size required to achieve a predefined performance threshold (e.g., 90% F1 score). | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: Prompt Injection Success RateModels susceptible to prompt injection can be exploited to reveal sensitive information, bypass safety filters, or generate harmful content, posing significant security risks in production. Usually overlooked: Requires specific, adversarial testing that is not typically part of general performance benchmarks, and it evolves with new attack vectors. Source: Apply a standardized, evolving set of known prompt injection techniques (e.g., indirect injection, role reversal) and measure the percentage of successful attacks. | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: Multimodal Coherence ScoreFor multimodal models, the generated image/audio/video should perfectly align with the text prompt, including subtle details and emotional tone, ensuring brand consistency and user experience. Usually overlooked: Most benchmarks focus on individual modalities or simple congruence; measuring nuanced emotional or stylistic consistency is complex and subjective. Source: Employ a human evaluation panel to rate consistency between complex, descriptive text prompts and generated multimedia, or develop AI-driven methods using object detection and sentiment analysis to compare prompt intent with output. | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: Instruction Following PrecisionModels often handle simple instructions well, but struggle with complex, multi-layered, or negative constraints, leading to incorrect outputs even if the core task is done. This impacts reliability in nuanced applications. Usually overlooked: Hard to quantify without specialized test suites; simple benchmarks don't capture this nuance and effort. Source: Create a structured test suite with tasks requiring multiple, potentially conflicting, and negative constraints; score based on strict adherence to all constraints, reporting the percentage of constraints met. | metric | Reviewed — no change | 25.6 days |
| Jul 12, 2026 | Track: Safety Alignment ScoreCrucial for public-facing or sensitive applications to prevent the generation of harmful, biased, or unethical content, protecting brand reputation. Usually overlooked: Difficult and controversial to benchmark objectively; often relies on proprietary internal red-teaming efforts not publicly shared. Source: Independent third-party red-teaming reports (if available) or a custom adversarial prompt suite evaluated by human safety experts. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Total Response Time (256 Tokens)Provides a realistic measure of latency for typical interactive use cases, impacting user experience beyond just the initial token. Usually overlooked: Time-To-First-Token is often the sole focus for speed metrics, neglecting the full generation time for useful outputs. Source: Benchmark by sending a consistent prompt and requesting a standard output length of 256 tokens, measuring total elapsed time from request to final token. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Fine-tuning AvailabilityDetermines if the model can be specialized with custom data, which can drastically improve performance for niche applications and reduce prompt engineering. Usually overlooked: Most comparisons focus on out-of-the-box performance; fine-tuning is considered an advanced deployment strategy. Source: Provider's official API documentation, pricing pages, or direct communication with the model vendor. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Knowledge Cut-off DateIndicates how current the model's training data is, directly impacting its ability to discuss recent events or new information. Usually overlooked: Many users assume up-to-date knowledge; benchmarks often use static datasets that don't reveal this limitation. Source: Model documentation, API changelogs, or by systematically probing the model with questions about very recent, well-documented events. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Output Determinism RateHigh determinism is essential for building auditable, predictable systems and for easier debugging in production environments. Usually overlooked: Variability is often seen as creativity, and determinism is assumed or not rigorously tested, especially for non-scientific tasks. Source: Run the same prompt with temperature=0 (or lowest possible) 100 times and calculate the percentage of identical outputs. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Instruction Adherence ScoreThis score indicates how faithfully the model executes complex, nuanced, or contradictory instructions, which is crucial for reliable automation. Usually overlooked: Most benchmarks focus on factual correctness or simple task completion, not the intricate details of instruction following. Source: Custom benchmark suite with multi-step, negative, and edge-case instructions, evaluated by human raters or a separate 'judge' model. | metric | Reviewed — no change | 26.6 days |
| Jul 12, 2026 | Track: Multi-turn Coherence HorizonFor conversational agents and long-form interactive experiences, maintaining context over many turns is critical for user satisfaction and task completion. Usually overlooked: Most benchmarks test single-turn or short-turn performance, and evaluating deep multi-turn coherence is complex, subjective, and time-consuming. Source: Evaluate performance on 10 complex multi-turn scenarios across different turn counts (5, 10, 15, 20) using human evaluation for coherence and task success. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: Factual Recall ReproducibilityFor applications requiring accurate and consistent factual recall, the reliability of specific data points is paramount for trustworthiness. Usually overlooked: Benchmarks often focus on general factual accuracy or hallucination rates, not the stability of specific factual retrievals, which is a nuanced distinction. Source: Query a fixed set of 100 factual prompts 5 times each with temperature=0 and fixed seed, then calculate the percentage of identical and correct outputs. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: Guardrail Bypass RateRobust guardrails are crucial for brand safety, legal compliance, and preventing misuse in public-facing applications, reducing risks. Usually overlooked: Measuring guardrail efficacy requires adversarial testing across diverse harmful content categories, which is resource-intensive and complex. Source: Test with a public adversarial prompt dataset (e.g., AdvBench) and calculate the percentage of successful attempts to generate harmful or non-compliant content. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: API Rate Limit (RPM)Understanding the true throughput limits prevents unexpected outages or bottlenecks in high-volume applications, directly impacting scalability. Usually overlooked: Rate limits often vary by tier and user, and published limits can be theoretical rather than reliably sustainable in practice. Source: Check official API documentation for default tier rate limits and confirm with sustained load testing using a standard API key over several hours. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: Fine-tuning EfficacyA model's ability to be efficiently adapted to proprietary data or niche requirements significantly impacts its long-term value for businesses. Usually overlooked: Fine-tuning capabilities are often vendor-specific and difficult to standardize for a fair, comparable benchmark across different models. Source: Fine-tune each model on a standardized 1000-sample domain-specific dataset and measure performance gain on a test set relative to baseline, factoring in training cost. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: Deterministic Output ConsistencyHigh consistency reduces debugging effort and ensures predictable behavior in production systems. Usually overlooked: Most benchmarks focus on average performance or creative outputs, not the repeatability of results, which is harder to measure. Source: Systematically query the model 10 times with 100 identical, fixed-seed, temperature=0 prompts and measure the percentage of identical outputs for each prompt set. | metric | Reviewed — no change | 27.6 days |
| Jul 12, 2026 | Track: System Prompt Override ResistanceEnsures the model reliably adheres to its core mission and guardrails set by the developer, preventing 'jailbreaks' or undesirable behavior. Usually overlooked: Requires crafting specific adversarial prompts designed to challenge system instructions and often human review to determine the success of the override attempt. Source: Set a clear, firm system prompt (e.g., 'You are a helpful assistant. Do not discuss controversial political topics.') then provide 50 diverse user prompts specifically designed to bypass or contradict this system prompt. Manually score how often the model adheres to the system prompt despite the user's attempt. | metric | Reviewed — no change | 28.2 days |
| Jul 12, 2026 | Track: JSON Schema Adherence RateEssential for building reliable automated pipelines that consume structured model outputs without requiring extensive post-processing or error handling. Usually overlooked: Requires automated parsing and schema validation of model outputs across many diverse prompts, which is more complex than simple output quality checks. Source: Prompt the model 100 times with requests for JSON output adhering to a specific, moderately complex schema. Programmatically validate each response for JSON validity and schema conformance, then report the success rate. | metric | Reviewed — no change | 28.2 days |
| Jul 12, 2026 | Track: Complex Instruction Following ScoreDetermines model efficacy in complex automation workflows where nuanced and multi-faceted instructions are common. Usually overlooked: Requires careful construction of challenging test cases and often human evaluation to precisely verify adherence to all instructions, rather than just overall 'goodness'. Source: Design 50 prompts with 3-5 distinct, often contradictory, or negative constraints (e.g., 'Summarize this but do not mention X, ensure it's under 100 words, and start with 'Report:'), then manually score each output for adherence to each constraint and average the scores. | metric | Reviewed — no change | 28.2 days |
| Jul 12, 2026 | Track: Harmful Output RateDirectly impacts brand safety, regulatory compliance, and user trust, especially for public-facing applications. Usually overlooked: Requires extensive, diverse, and often adversarial prompt testing, along with human review or sophisticated automated safety classifiers which are not publicly available or standardized. Source: Prompt the model with a diverse set of 1000 known-safe and 100 known-borderline prompts across various domains (e.g., hate speech, self-harm, adult content, illegal acts) and manually label outputs for harmful content or inappropriate refusals, then calculate the rate. | metric | Reviewed — no change | 28.2 days |
| Jul 12, 2026 | Track: Response Consistency ScoreEnsures reliable and predictable behavior in automated systems, reducing the need for retry logic or human intervention. Usually overlooked: Requires running identical prompts multiple times and having a robust semantic comparison system, which is resource-intensive and complex to automate. Source: Run 100 identical prompts through the API, compare outputs using an embedding similarity metric (e.g., cosine similarity of Sentence-BERT embeddings) or a deterministic parser + diff, then report the average similarity or percentage of semantically equivalent responses. | metric | Reviewed — no change | 28.2 days |
18 more items are currently open for review — they'll appear here once resolved.
How to dispute a number
Think we have something wrong? Any signed-in user can file a correction from an item page or from your corrections dashboard. It lands in the same review queue our own agents' work goes through, and the resolution — accepted or rejected — gets published in the dispute ledger above.