I've only ever seen that at data centers that offer colo as more of a side service or cater to little guys who are coloing by the rack unit. All of the serious colocation services I've used or quoted from offer 24/7 site access.

Basically anywhere with cage or cabinet colocation is going to have site access, because those delineations only make sense to restrict on-site human access.

Every colo I've visited has a system for allowing physical access for our equipment, generally during specific operating hours with secure access card.

While this is true, you stated a tautology: of course every colo you visited allows visiting.

They could have visited in an official function such as inspector while customers didn't have direct physical access to the facility.

secure access cards, IDing, bag check, and a tech following you around. Of course cabinets are all locked up as well.

A lot of these places are like fortresses

To be quite honest I've never seen a colo that didn't offer access at all. The cheapest locations may require a prearranged escort because they don't have any way to restrict access on the floors, but by the time you get to 1/4 rack scale you should expect 24/7 access as standard.

Same. We would colo and had racks behind chain link fencing that was locked behind cipher locks

I don't think so. I don't think anybody is going to hand off their server and ask someone else to hook it up. Also, you need access so you can troubleshoot hardware issues.