The Fermi Paradox is about the (intelligently-created) radio silence, not the lack of little green tourists. We'd eventually notice an advanced civilization in a period that intersects our time/distance coordinates (i.e., if Alpha Centaurians had radio 4-1/2 years ago, we'd probably hear it).

But otherwise, yeah, we're imprisoned here by 'c'.