Well the memory wont be an issue now, 6gig is more than enough, but the processors may cause a prob, but they are 4x so... I have hired a third party source to monitor servers daily too so.. hopefully that keeps things safe.
It could also be the software, what OS and webserver are you running? Youtube, Imageshack, isohunt, all the major sites like ragezone run lighttpd instead of apache..