Exactly. You know inside that you have done nothing wrong, but nobody else knows that. They can only look at the available evidence and if they're talking to you it's because the evidence is spotty.

So, now you're gambling with whether or not the spotty evidence is spotty enough to make you look innocent or guilty. Therefore, everyone has something to hide because the whole truth will not always be evident and some partial truths can make you look guilty (ex. your cell phone location was very close to a murder scene. You didn't kill anyone, you walked by the spot a few minutes before it happened).