People have a pretty good sense of whether you're staring at them or something just beside or behind them. Not really from the angle of your eyes but the way you react (or not) when the other person looks back.

I wouldn't worry about that so much. And I worry about a lot of social things :)