In VMware ESXi 5, SSH is enabled through the “Configuration / Security Profile / Services” section. When you connect using the root user, you have to specify the root password. If you want to use SSH keys, the only thing you have to do is copy them, over ssh, to a dedicated directory:
# cd ~/.ssh # scp id_rsa.pub authorized_keys root@esxi:/etc/ssh/keys-root/
You can now connect to the ESXi using your SSH keys (and the `ssh-agent`). So simple!
Source: SSH Keys & Lockdown Mode Caveat in ESXi 5