This is because you have forgotten to return a `Vary: Origin` header in the response. If you don’t do this, caches will presume the response is the same regardless of the Origin header in the request and so you will get the bug you describe.
Of course note that some common CDNs just ignore the `Vary` header. cough Cloudflare cough. Also IIRC not refusing to cache anything with a `Vary` header but caching it anyways and serving it no matter what headers the client sends.