Holy crap, yes. This was incredibly alarming. SSH keys are far and away more secure than username/password. I would be far more comfortable if it were possible to use GitHub with nothing but my SSH key. The ability to create a passcode-secured key is vital to me, because the keys and the passcodes have to be stolen to be useful at all.
And if you use ssh-agent, I'm pretty sure it doesn't even send your private key; SSH creates a challenge that your key is used to compute, so even someone eavesdropping with a keylogger on the server won't be able to sniff your password.
Not sending your private key inherent property of public-key authentication, and is used in all instances of SSH key-based authentication. ssh-agent is merely convenient "keyring" type application so you do not have to continually re-enter the passwords for the on disk encrypted private key files. (You are encrypting your ssh keys with a password right?)
And if you use ssh-agent, I'm pretty sure it doesn't even send your private key; SSH creates a challenge that your key is used to compute, so even someone eavesdropping with a keylogger on the server won't be able to sniff your password.