1. Save each valid provider result independently
A comparison with several providers should not be one all-or-nothing cache write. Validate and save a successful provider evaluation as soon as it is available. If another provider times out, a retry can reuse the completed evaluations. Give each unit a deterministic key that includes product facts, model and rubric. A complete panel report can still require every configured seat, but its failure should not erase successful intermediate work. A partial panel, when shown, must state which providers are missing.
2. Collect successes and errors separately
The example simulates three providers, with one temporary failure. It stores successful steps and retries only the failed provider on the second run. A thread pool limits fan-out; it does not establish cross-instance ownership or cancel an external request automatically. Configure actual socket/provider deadlines inside the worker. Calling Future.result with a timeout only limits how long that caller waits; it is not a substitute for a network timeout or a provider cancellation mechanism.
from concurrent.futures import ThreadPoolExecutor, as_completed
cache, calls = {}, []
def collect(providers, worker):
errors = {}
todo = [p for p in providers if p not in cache]
with ThreadPoolExecutor(max_workers=3) as pool:
futures = {pool.submit(worker,p):p for p in todo}
for future in as_completed(futures):
provider = futures[future]
try:
result = future.result()
if not isinstance(result,dict) or 'score' not in result:
raise ValueError('invalid result')
cache[provider] = result
except (TimeoutError, ValueError) as error:
errors[provider] = type(error).__name__
return errors
def first(p):
calls.append(p)
if p == 'B': raise TimeoutError('temporary')
return {'score':80}
assert collect(['A','B','C'], first) == {'B':'TimeoutError'}
calls.clear()
def retry(p):
calls.append(p)
return {'score':81}
assert collect(['A','B','C'], retry) == {}
assert calls == ['B']
print('only the unfinished provider was retried')
3. Add a durable work lease
An in-process mutex disappears on restart and does not coordinate multiple Cloud Run instances. Store a lease with a work key, random owner and expiry. Claim or renew ownership transactionally, and release only when the owner matches. Set expiry longer than the bounded work duration or implement renewal with failure handling. When a lease expires, recheck the result cache before starting again. Two workers can still overlap around failure boundaries, so external operations that support idempotency should use a stable key as well.
4. Retry only recoverable failures
Separate an invalid credential or unsupported model from a temporary timeout or rate limit. Use a finite retry count and jittered backoff, and respect a provider retry hint when available. Every attempted model request should consume an application budget even if it fails. Show real provider status to the user instead of an invented percentage. The goal is to resume a decision with minimal duplicated work; an endless spinner that repeatedly launches the same calls is neither reliability nor a good experience.