Hi there,
It really depends on what you want to change about the game experience bugfixes extra content different areas?
Simple changes would consist of sql scripts or just text edits while changes like models, textures, animation would require additional programs like maya mpq editors
And ofc it is possible to edit the core to with c++ knowlegde.
But basicly you would want to first learn how to compile your own server.
After that just experiment by trial and error make a backup change something see if it works and if not try to look why it doesn't work.
There are alot of choices on the way of developing your own server revision, I myself prefer ManGoS but that is just personal taste.
It is best to read abit about both major releases ManGoS,Ascent and try them out before committing yourself to compiling.
There are more than these 2 both subversions as well as totally independant just try and take what you like.
However if you think ManGoS is more suited to your taste than I suggest reading this excellent guide by Reaper-X
You must be registered to see links
Try not to just read step by step but rather look why those steps are required.
Guides are mostly rid of the background information which is rather important in case your going to maintain your server.
My experience with Ascent doesn't go further than running a server you can read some of the Ascent forum
WoW Ascent - WoW Ascent Tutorials at RaGEZONE.com
Hope this helped you abit on the way.