With passwords you are handing out what is needed to log in. With the ssh public key scheme, you only prove that you have the private key, but don't hand it out to the server.

I don't feel safe carrying a private key around with me on my mobile. It could be copied without my knowledge and brute forced in privacy. Whereas with password authentication, I can impose limits on the number of tries and also on the time between the tries. Am I wrong in thinking this?

This is why I have a completely separate private key that I generated for my phone, using a different password that's unrelated to the passwords I use for all my other machines' private keys, and I use password protection on the phone itself to thwart casual intrusion. If I lose my phone, it's trivial to remove the phone's public key from my servers' authorized_keys files, at which point the phone can no longer be used to log in.

Encrypt your private key!

This is very easy to do, and quite important for devices that can be stolen - including your desktop computer.

It's easy enough to setup ssh-agent on your desktop so you don't have to keep entering the password.

If you have a copy of the private key and the public key, you can bruteforce the passphrase. So use a big passphrase instead of a password.

passwords are replayable tokens. one-time passwords are not. people who actually are careful with machine security use ssh keys + sudo/opie.