There was an earlier post this year that showed SOTA models failed on counting the legs of 5-legged zebras and other counterfactual images.

https://news.ycombinator.com/item?id=44169413