1. Split specifications from volatile offers
Store stable product specifications separately from retailer offers and availability. A launch price is not a live quote, and a new collection date does not prove that a price is current. For each field, keep the edition and evidence conditions needed to interpret it. Review cadence should depend on the field and known changes, not an assumption that every row has the same lifetime. A small source-backed library can start with an editorial queue before investing in a complex crawler or paid search service.
2. Detect content changes deterministically
Build the fingerprint from a clearly defined evidence payload. Exclude administrative fields such as fetched_at so unchanged evidence stays reusable. Include source identity or confidence when a change in evidentiary basis should trigger re-evaluation. The example focuses on scorer input fields and demonstrates the no-change case explicitly. Its review date is a reminder to revisit a source, not a freshness guarantee. An actual collector must also verify that a successful HTTP response contains the expected product page rather than a redirect or an access-denied document.
import hashlib, json
from datetime import date, timedelta
def fingerprint(record):
fields = {k:record[k] for k in ('edition','facts','sources')}
raw = json.dumps(fields,sort_keys=True,separators=(',',':'))
return hashlib.sha256(raw.encode()).hexdigest()
def refresh(old, incoming, today):
changed = fingerprint(old) != fingerprint(incoming)
return {**incoming, 'fact_version':fingerprint(incoming),
'review_after':str(today+timedelta(days=30)), 'needs_evaluation':changed}
old = {'edition':'US / standard','facts':{'weight_g':200},'sources':['https://example.com/specs'],'fetched_at':'old'}
same = {**old,'fetched_at':'new'}
assert not refresh(old,same,date(2026,11,1))['needs_evaluation']
changed = {**same,'facts':{'weight_g':195}}
assert refresh(old,changed,date(2026,11,1))['needs_evaluation']
print('unchanged fetch reuses evaluations; changed evidence requests review')
3. Prioritize a queue without inventing popularity
A useful queue combines overdue review dates, repeated sourced lookup misses and confirmed specification conflicts. Keep a source outage in a separate status so it does not become evidence that a product does not exist. Deduplicate formatting variants only after identity confirmation. Search frequency can guide editorial effort, but a development test is not market demand. Separate production observations from prefetch and maintenance requests before using counts to label products popular or to schedule costly analysis.
4. Refresh facts before paying for reports
A daily job can inspect source dates, cache metrics and stored report samples without calling an AI model. New collection should be limited to justified products and must preserve attribution and unknowns. If evidence changes, invalidate dependent evaluations and warm only combinations with a reason to be used. Record the number of source checks and actual model calls separately. A queue that reliably handles ten justified changes is more useful than thousands of newly fetched records whose identity and compatibility have never been checked.