1. Capture a minimal failure fixture
Save the expected dimensions, product count and sanitized structured response before rerunning a failed comparison. Do not log API keys, raw customer supplements or full request headers. Preserve a request ID for joining operational events, and keep private payloads in access-controlled storage when genuinely necessary. A fixture should be small enough to explain the first incorrect value. Repeating a paid request until a plausible answer appears is not a diagnosis; it can hide the defect and waste the successful work from other providers.
2. Validate the structure before rendering
Validate shape, numeric type, bounds and winner range separately. Python booleans are integer subclasses, so a generic isinstance(value, int) check can accidentally accept true as a score. The fixture validator below rejects it explicitly. Real adapters should also check the number of reasons and the presence of a usable vote explanation. Keep a missing score distinct from a malformed response: a deliberate unknown may be a valid domain state, while a wrong matrix shape is a protocol failure.
import math
def validate_scores(rows, dimensions, products):
if not isinstance(rows,list) or len(rows) != dimensions:
raise ValueError('dimension_count')
for row in rows:
if not isinstance(row,list) or len(row) != products:
raise ValueError('product_count')
for value in row:
if value is None:
continue
if type(value) not in (int,float) or not math.isfinite(value) or not 0<=value<=100:
raise ValueError('invalid_score')
return rows
assert validate_scores([[90,None]],1,2) == [[90,None]]
for bad in [[[True,80]], [[101,80]], [[float('nan'),80]], [[90]]]:
try:
validate_scores(bad,1,2)
raise AssertionError('invalid fixture accepted')
except ValueError:
pass
print('malformed fixtures rejected without a model call')
3. Classify the first failing layer
Use stable categories such as identity_confirmation, product_not_found, provider_auth, provider_rate_limit, timeout, database and invalid_output. These categories are safer and easier to aggregate than unrestricted exception messages. Trace the value through identity resolution, sourced facts, rubric selection, adapter validation, aggregation and rendering. If a valid null becomes zero only in the chart, changing the prompt is the wrong fix. If the edition is wrong before scoring starts, every later calculation may be internally consistent and still answer the wrong question.
4. Turn the diagnosis into a regression case
After isolating the cause, replay the fixture with all outbound model calls blocked. Add a focused assertion for the behavior that failed, then test one successful path to avoid breaking valid results. Review whether stored reports were affected; a fixed renderer and a corrected historical dataset are different changes. Keep the incident explanation tied to observable evidence. For disagreements that the fixture cannot resolve, add a review notice rather than automatically changing scores or forcing models to agree.