An LLM API should be treated as a variable external dependency. Responses can be slow, rate-limited, malformed or unavailable even when the rest of the application is healthy.
Set a request deadline. A retry that completes after the user or background job no longer needs the answer creates cost without value.
Validate structured output before it reaches application logic. Keep the original response for debugging with sensitive fields removed.
Monitor latency, error categories, token use and cost. Alerts should identify a degrading dependency before users report a complete outage.
Classify failures before retrying
Retry transient network failures and explicit capacity responses with exponential backoff and jitter. Do not retry invalid authentication, unsupported parameters or policy rejections indefinitely.Set a request deadline. A retry that completes after the user or background job no longer needs the answer creates cost without value.
Make operations idempotent
A repeated request must not duplicate an external action. Separate text generation from sending email, publishing content, creating an order or changing a record. Store state so a worker can resume safely after interruption.Validate structured output before it reaches application logic. Keep the original response for debugging with sensitive fields removed.
Design useful fallbacks
A fallback can use another model, a smaller context, cached output or a clear request to try again. Test whether quality and safety requirements still hold rather than assuming every provider is interchangeable.Monitor latency, error categories, token use and cost. Alerts should identify a degrading dependency before users report a complete outage.