Its often possible to lock down the public IP entirely to not accept connections except what's initiated from the inside (like the cloudflare tunnel or otherwise reaching out).
Something like a Cloudflare+tunnel on one side, tailscale or something to get into it on the other.
Folks other than me have written decent tutorials that have been helpful.
