A bug caused by nginx serving item from stale cache

Assuming you are aware of nginx proxy pass, nginx caches the response from upstream including the upstream response headers.
Our api was fronted by an nginx service which proxied the requests to the backend api service.

We enabled cors for a website which fetches data from our api.
What we noticed was that some of the requests were blocked by the browser as the api response was not returning the access-control-allow-origin with the website domain whitelisted.But the same exact api was returning the header with the domain randomly i.e 1 out of 2 requests(for example) failed.

We found that the api had nginx caching enabled and some servers had the old response (with headers) cached even though the latest changes were deployed to the backend api service.So the request was not hitting the backend service which had the website domain whitelisted and instead was served from cache.Few servers, had the cache miss and cache expiry which triggered a fresh request to the backend service.

There are multiple ways to avoid bugs of these kind such as where do you handle cors(at api gateway or in service itself), purging cache(could cause upstream surge) etc.