The question is, why would you want to do that in C#. A few years ago, I tried this approach for reading/writing packets and found marshaling not faster. It adds some unneeded complexity to the code and is sometimes not possible with complex mu packet structures. IMHO, it's the wrong "optimization" at the wrong place - there is simply no need to do such things.
And I don't agree that C# isn't suitable for a server application because of the missing memory management capabilities - I think the opposite is the case:
- The garbage collector is pretty efficient, the game client doesn't notice when it runs for a millisecond. It's not like it will noticeably lag on client side when it runs on the server.
- Memory leaks are less likely to occur, therefore the server might run longer without a restart.
- Caring about memory is error-prone and requires discipline. I've seen all kind of mistakes in my day-to-day job, even from good developers.
Making a game client in C# is a completely different story, of course. Frame-times must be constantly short to get no lags. A GC pause could cause lag there.