Invasion Protection

I recently had problems of encroachment on my server. I use a dedicated core2quad, 8gb of RAM and 100MB of uplink in Debian (not Colinux).
What happened? Well the server was at 100% processor usage, using 98% of 100MB of uplink and in 5 days it transferred 1.2 TB.
Sought information with people who work with information security and they were unanimous in saying that the server had been invaded.
I would like to know how to protect my server of this type of attack.
I use the 1.3.6 server files out of Ragezone.