Any administrator that accesses his server from his home computer (or any not so safe computer for the matter) is easily vulnerable to giving access to the server to hackers. If the admin uses ssh an attacker could easily hijack the ssh command (or the graphical ssh menu entry) and present the admin with identical looking hacked version of ssh that remembers the server password and sends it to verybadguys.com.
Using /usr/bin/ssh might help if the source account in not root and root is really, really hard to get by an attacker (for root the hacker can simply replace your original ssh) but there probably are methods to hijack that even in a limited account (graphical ssh program with plugins? some kind of plugin like functionality for the console? are you sure you opened the right console?(do you really trust that menu entry?)). Even if the hacker does not know how to hijack your ssh he can use snapshots of your desktop to make an idea of how the server configuration looks like and use that information later when doing an attack (if he sees a config file password or cryptology key its obviously bad but sometimes even subtler information is decisive) .
This applies to many other remote administration taks (ftp included). If you do ftp from firefox a malicious firefox addon can simply remember your password.