That's why you use media queries to set display: none only for non-screen readers.

Of course, screen readers might not respect that, in which your solution would work.