You might not want rotation invariance in your algorithm. Our own perceptual system is not rotation-invariant, most obviously for faces in this illusion: https://en.wikipedia.org/wiki/Thatcher_effect