[Discussion] My two cents about IGCN source code.

[bramdebouvere]

I don't agree with everyone bashing on C# here. I must agree that there are limits to memory management. But if your code doesn't look like a total trainwreck you should actually never have too much memory issues on a server like this (unless you have a leak somewhere). I've coded servers in C# that received more than Gbps in traffic and never used more than 25MB of memory.
Of course, if you start using third party crap and some crappy frameworks provided by Microsoft, you might have a bad time.

nevS, nice project you have going there!

[ptr0x]

Yea. Considering pro's and con's its a even situation. either way people gona run into issues.
My friend and i tried to make server using IGCN premium files. i wasnt that dissappointed of the files condition but their team it self.
Some basic things that works well on free repacks did not work on theirs. yet they had things that you couldnt find in any public free repacks.
The support from them was slow as hell, and some minor bugs seemed not to be worthy of their time. Advertising new features that should be considered equal to scamming.
its a difficult situation where people like new trends ( latest seasons/features ) and this is why i think igcn files are so common amongst current servers.

Altough the post does not concern the files IGCN sell nowadays, I agree with you here.

I was a customer of IGCN too, before paying them everything is great. I was told that the files were very customizable through Lua scripts, a lot of options could be changed through XML, there was the possibility to pay for custom requests, etc. So I paid for their so called "premium service". I felt nothing but scammed. Their customer support is by far the worst customer support I have had the unhapiness to deal with. The S13 files are very buggy. And, well, their so called "Lua scripting" is done like if my 8 years old cousin programmed it. They provide no documentation regarding the API exposed to Lua scripts and the things you can actually tweak using their existent scripts is virtually 0. Their "lua scripting" is probably what most shocked me. It's not more than a fancy way to set values to variables.

- - - Updated - - -

I don't agree with everyone bashing on C# here. I must agree that there are limits to memory management. But if your code doesn't look like a total trainwreck you should actually never have too much memory issues on a server like this (unless you have a leak somewhere). I've coded servers in C# that received more than Gbps in traffic and never used more than 25MB of memory.
Of course, if you start using third party crap and some crappy frameworks provided by Microsoft, you might have a bad time.

nevS, nice project you have going there!

I don't think anyone is bashing on C#. Also I think you missed the point of what we were talking, but I will speak only for me here.

I did not meant that you can not do such things using pure C#. I just pointed out that there are far better languages to use in cases like a server in a client-server application. I take this for a fact: C++ is way better to do such an application than C# for tons of reasons - limitations regarding memory management is just 1 of those reasons.

However I think you missed what was told. By stating C# have memory management limitations (remember that I was always comparing its capacity to work with memory to C++ capacity to do the same) I was implying different problems than the ones you seem to got from what I told. I was not talking that the memory management limitations of C# imply in the incapacity of handling a large number of data bytes. These limitations results in a lot more problems in aspects as code readability and code maintainability. Imagine things like taking an address in memory and treating it in the code as a data structure in C# results in a code that is difficult to read and to maintain. Besides that, C# will execute codes regarding memory management a lot slower than any (ok, almost any) unmanaged language - this is just not a big problem nowadays due to the growing hardware power in average PCs nowadays.

Yet, C# is the language I like the most. I can create the cleanest code with a very high maintanability and readability using this language (overall, these aspects tend to be very much higher in C# codes than C++ codes). This is just not the truth in this particular case we are discussing.

[bramdebouvere]

Imagine things like taking an address in memory and treating it in the code as a data structure in C# results in a code that is difficult to read and to maintain.

That's all good and well. But I don't see why you would use structs and pointers in the first place. Classes are very optimized in .net. I don't see the need of using structs and pointers everywhere if you're not running on embedded systems / have legacy libraries that you need to call / need to interface with specific hardware / need exotic tricks to make more optimized code. I personally think that if you are using a lot of code like that to create a game server, something is terribly wrong (i'm talking if you're using c# specifically, not c++). I can totally see why one would do it in unmanaged languages, but in a managed language like c# it just seems like unnecessary complication to me, with little to no benefit in performance. And you are very correct to say that it would probably result in a lot of issues down the line. So the only thing I can really say to that is "just don't do that"..

But I realise that we're going offtopic, so maybe it's better if we keep this conversation going elsewhere.

[ptr0x]

That's all good and well. But I don't see why you would use structs and pointers in the first place. Classes are very optimized in .net. I don't see the need of using structs and pointers everywhere if you're not running on embedded systems / have legacy libraries that you need to call / need to interface with specific hardware / need exotic tricks to make more optimized code.

Data structures does not ends up in structs. Classes are as data structures as structs are. They are both very similar to each other - the main differences around classes and structs revolves around how each of them are managed in memory. So when I say "data structures" I'm referrring to any data structure in the code, both classes and structs. Well, while you should use classes in the majority of cases. You can only do pointer arithmetics with structs. There are a lot of cases where one should use structs over classes besides the cases you pointed out.

I personally think that if you are using a lot of code like that to create a game server, something is terribly wrong (i'm talking if you're using c# specifically, not c++). I can totally see why one would do it in managed languages, but in an unmanaged language like c# it just seems like unnecessary complication to me, with little to no benefit in performance. And you are very correct to say that it would probably result in a lot of issues down the line. So the only thing I can really say to that is "just don't do that"..

I can't disagree more here. All these features are what makes C# so great. The possibility to manage memory in a managed application, even with some limitations, are awesome. There are a lot of cases where you should use a lot of "code like that" to create a game server. There are A LOT of benefit in performance if you do things using pointer arithmetics instead of relying entirely in the CLR.Net; almost every code will execute very much faster if you code it the right way inside an unsafe block and make heavy use of pointers. That to not even mention particular cases where you can save hundreds of LoC using that approach instead of sticking 100% with managed code.

TL;DL If I was doing such a project in C# I would certainly not stick myself with managed C#. I would use C++/CLR where it fits and certainly would use a lot of unmanaged code (like pointers and structs) in the code where it is better applied.

But I realise that we're going offtopic, so maybe it's better if we keep this conversation going elsewhere.

Ye, lets get back to the topic ^^

[ptr0x]

Just a little adendum to the c# vs c++ discussion:

I somehow saw a post I wrote in codereview.stackexchange showing a simple wrapper to marshal raw data into marshalable c# classes. Link: https://codereview.stackexchange.com...r-a-tcp-server

[nevS]

Just a little adendum to the c# vs c++ discussion:

I somehow saw a post I wrote in codereview.stackexchange showing a simple wrapper to marshal raw data into marshalable c# classes. Link: https://codereview.stackexchange.com...r-a-tcp-server

The question is, why would you want to do that in C#. A few years ago, I tried this approach for reading/writing packets and found marshaling not faster. It adds some unneeded complexity to the code and is sometimes not possible with complex mu packet structures. IMHO, it's the wrong "optimization" at the wrong place - there is simply no need to do such things.

And I don't agree that C# isn't suitable for a server application because of the missing memory management capabilities - I think the opposite is the case:

• The garbage collector is pretty efficient, the game client doesn't notice when it runs for a millisecond. It's not like it will noticeably lag on client side when it runs on the server.
• Memory leaks are less likely to occur, therefore the server might run longer without a restart.
• Caring about memory is error-prone and requires discipline. I've seen all kind of mistakes in my day-to-day job, even from good developers.

Making a game client in C# is a completely different story, of course. Frame-times must be constantly short to get no lags. A GC pause could cause lag there.

[ptr0x]

The question is, why would you want to do that in C#. A few years ago, I tried this approach for reading/writing packets and found marshaling not faster. It adds some unneeded complexity to the code and is sometimes not possible with complex mu packet structures. IMHO, it's the wrong "optimization" at the wrong place - there is simply no need to do such things.

And I don't agree that C# isn't suitable for a server application because of the missing memory management capabilities - I think the opposite is the case:

• The garbage collector is pretty efficient, the game client doesn't notice when it runs for a millisecond. It's not like it will noticeably lag on client side when it runs on the server.
• Memory leaks are less likely to occur, therefore the server might run longer without a restart.
• Caring about memory is error-prone and requires discipline. I've seen all kind of mistakes in my day-to-day job, even from good developers.

Making a game client in C# is a completely different story, of course. Frame-times must be constantly short to get no lags. A GC pause could cause lag there.

I think you are missing the points here..

The discussion does not revolves around while c# memory management limitations execute slower than c++. The problem is not how much CPU cycles the C# server will have to execute to accomplish it (or at least it shouldn't be it).

The problem is more about aspects of code quality (like maintainability).

So, to answer your first question: "why would you want to do that in C#". I believe you are talking about why one would go with the Marshal-way to convert raw memory into code objects. The answer is simple: it is the very best alternative to do it in C# so far (when we have ref returns and the possibility to direct cast a pointer into a blittable object it can change). To do this task in C# you have 3 options: A - you can go with the "ReadByte, ReadWord, ReadDWord" method when you will load each offset of a memory chunk separately into the correct variable; B - you can do it with Marshal or C - you can do it with blittable struct cast and pointers.

The first option (A) is very, very, VERY bad for tons of reasons ranging from execution time to many aspects of code quality. Imagine you have to initialize a class of hundreds of variables. Using this method you will write a function to initialize this object from a memory stream once and pray for the gods you don't need ever more to touch this function. If for some reason you need to modify this code you will spend a LOT of time trying to check if the memory offsets are right with your new modifications. This is just NOT the right solution here.

The option C gives you the best performance but it is not recommended because the data objects must be blittable leaving you in bad hands when you have to declare an array for example. For some odd reason C# won't let you declare a blittable array of any element in a struct (even if the elements of the array are blittable).

So finally we have the option B which is using the Marshal class and its attributes. Yes, the code will be bigger with the attribute decorations (MarshalAs) and stuff but it is the better known way to do it in C#. You can create a code that runs 100% in managed mode. The execution is somewhat okay considering the amount of tasks the cpu have to execute to accomplish this but it is not a big problem considering the hardware power we have nowadays.

I don't know which option you are using to do this in your project but I strongly disagree when you say that "marshaling is not faster and adds some unneeded complexity to the code". Well, when you say that there are complex mu packet structures that can not be represented using Marshal I have no clue what you are talking about. The marshal class is made to fit every possible data type in memory. This is the method used in every pinvoke (platform invoke) code. Could you show an example of such a complex structure that can not be represented in a classe using Marshal?

[nevS]

Well, I guess we have different approaches in mind when working with packets. I probably use something similar to your option A, but I don't parse packets into structs or classes. Imho, defining all this message structs just adds a lot of unneeded boilerplate and sometimes complexity. I work directly on a byte array (and soon pooled Memory/Span<byte>) with some convenience extension methods. So basically, I extract the data I need into local variables and pass them to my game action functions. When sending packets I also use extension methods to set the bytes.

Imagine you have to initialize a class of hundreds of variables. Using this method you will write a function to initialize this object from a memory stream once and pray for the gods you don't need ever more to touch this function.

Packets coming from clients are usually very small (under 5 fields), so this is not an issue here. For outgoing packets it's similar - most are relatively small. The big ones are a bit of an issue, but it's a trade-off I made here. The original code doesn't make things much better in this cases :D

If for some reason you need to modify this code you will spend a LOT of time trying to check if the memory offsets are right with your new modifications.

Sure, but the same applies for structs and marshalling. You'll have to correctly define them as well and hope marshalling works as expected. The original game server/client also uses different byte order (little/big-endian) for same types of fields, sometimes structs are packed, sometimes not - big PITA. I wonder how marshalling behaves when it runs under different CPU architectures (possible with .net core at linux), e.g. ARM.

I don't know which option you are using to do this in your project but I strongly disagree when you say that "marshaling is not faster and adds some unneeded complexity to the code".

I wrote a small test which compares my way of reading packets to using struct marshalling. My way is about 100 times faster in this test (if I leave the validation away it's more like 800 times). I know this is more like a theoretical test and that's no bottleneck in the real world, but still. You can find the test code here: https://gist.github.com/sven-n/d5a25...3b6e3a1bcd04e0

Well, when you say that there are complex mu packet structures that can not be represented using Marshal I have no clue what you are talking about. The marshal class is made to fit every possible data type in memory. This is the method used in every pinvoke (platform invoke) code. Could you show an example of such a complex structure that can not be represented in a classe using Marshal?

I encountered problems when I tried to use arrays of structs. Some packets also have some dynamically sized blocks, e.g. the effects in player meet packets. Simplified example:
[Header]
[Player Count], for each:
[[Fixed block of Player Data] [Effects counter] [Dynamic block of effects, one byte each]]

I don't know if I were too stupid some years ago, but I didn't manage to use Marshal in a simple way. My first problem was, that only the first player got parsed/serialized. Even if I could get that to work, the next problem was, that I couldn't define a variable length of the array of effects. Maybe my approach was flawed and it required more manual work, like multiple marshal calls for one packet.

The problem is more about aspects of code quality (like maintainability).

I hope I got your point now... So, you're telling me that maintaining C++ code is easier than a C# and code quality in this specific aspect will be better in C++? Well, you can write horrible code in any language - you saw this IGCN sources, right?
My experience so far is, that packet handling is one of the easier and smaller aspects during development of a server. The other benefits of C# are way more valuable for me.
And my current approach might not be the final one... I'm thinking about defining packet structures as data which feeds some dynamic code generator - would be just as fast as writing my manual code but highly flexible.

[ptr0x]

Well, I guess we have different approaches in mind when working with packets. I probably use something similar to your option A, but I don't parse packets into structs or classes. Imho, defining all this message structs just adds a lot of unneeded boilerplate and sometimes complexity. I work directly on a byte array (and soon pooled Memory/Span<byte>) with some convenience extension methods. So basically, I extract the data I need into local variables and pass them to my game action functions. When sending packets I also use extension methods to set the bytes.

Well, what you call "uneeded boilerplate" I call better readability and maintainability of code. Of course it sums up some extra lines of code, but so what? A well done code documentation sums up an enormous amount of LoC. Will you stop documenting your code because it adds "uneeded boilerplate"? I think you are facing this issue the wrong way...

Packets coming from clients are usually very small (under 5 fields), so this is not an issue here. For outgoing packets it's similar - most are relatively small. The big ones are a bit of an issue, but it's a trade-off I made here. The original code doesn't make things much better in this cases :D

As I'm still learning the game's protocol I can't say much here besides what I already told. You are facing readability and maintainability as if you shouldn't care about it. Today you probably won't face much issues regarding code quality. Tomorrow, however, when you get more people working on your project or when you need to maintain a code wrote 5 years ago you will miss these days when you COULD do it with code quality in mind but you didn't, because it adds "unneeded boilerplate".

Sure, but the same applies for structs and marshalling. You'll have to correctly define them as well and hope marshalling works as expected. The original game server/client also uses different byte order (little/big-endian) for same types of fields, sometimes structs are packed, sometimes not - big PITA. I wonder how marshalling behaves when it runs under different CPU architectures (possible with .net core at linux), e.g. ARM.

No, the same does not applies for marshaling. The amount of complexity added using the "stream reader" option is huge. The added complexity using the "marshal" option is minimal. I have no idea why you think the two scenarious would be the same to maintain. About the endianess you have ways to deal with it. However I have doubts about if this is still the case. As far as I know the client were compiled to IA-32 architecture, which is little endian. You have to deal with it in the case you are compiling for another architecture platform (but it that is the case you will have a LOT more issues to think about than this - this will be a piece of cake near the concerns you will have).

I wrote a small test which compares my way of reading packets to using struct marshalling. My way is about 100 times faster in this test (if I leave the validation away it's more like 800 times). I know this is more like a theoretical test and that's no bottleneck in the real world, but still. You can find the test code here: https://gist.github.com/sven-n/d5a25...3b6e3a1bcd04e0

Yup, it is a fact that a generic (C#'s generic) function using Marshal will execute slower than a stream reader. However I never say the opposite. In contrary, I said the Marshal option will execute slower but it should not be an issue if we consider the hardware power servers have nowadays. To develop a big project like this you will have to either go for parallelism or a multi threaded application. The amount of cpu cycles needed to execute the Marshal solution should be insignificant in the overall execution of the process. Of course I'm considering you are developing this project to deploy in a server that fit its requirements. If execution time is gold for your project, go for blittable structs, pointers and C++/CLI, not stream reader.

I encountered problems when I tried to use arrays of structs. Some packets also have some dynamically sized blocks, e.g. the effects in player meet packets. Simplified example:
[Header]
[Player Count], for each:
[[Fixed block of Player Data] [Effects counter] [Dynamic block of effects, one byte each]]

I don't know if I were too stupid some years ago, but I didn't manage to use Marshal in a simple way. My first problem was, that only the first player got parsed/serialized. Even if I could get that to work, the next problem was, that I couldn't define a variable length of the array of effects. Maybe my approach was flawed and it required more manual work, like multiple marshal calls for one packet.

You probably did something wront when you tried to do it. To declare an array of structs it's as simple as adding the UnmanagedType.ByValArray in the field declaration.
About the dynamic length array in the packets: yes, you can't declare that using marshal directly. But you can't either in any other possible situation. You can declare an IntPtr and then use Marshal functions to alloc and copy the actual length of elements of the array. This is exactly the same "boilerplate" you have to do if you code this kind of structure in c/c++. You can't let the compiler handle a dynamic length array because dynamic length arrays do not have a fixed length known at compile-time.

I hope I got your point now... So, you're telling me that maintaining C++ code is easier than a C# and code quality in this specific aspect will be better in C++? Well, you can write horrible code in any language - you saw this IGCN sources, right?
My experience so far is, that packet handling is one of the easier and smaller aspects during development of a server. The other benefits of C# are way more valuable for me.
And my current approach might not be the final one... I'm thinking about defining packet structures as data which feeds some dynamic code generator - would be just as fast as writing my manual code but highly flexible.

Yep, I'm saying considering you are developing considering code quality, in this SPECIFIC case C++ will produce a code that is easier to read and maintain than C#'s.
Of course each language have its strengths. I'm not saying C# is useless or impossible or even just bad to develop this application. I'm just saying you have better options, like C++.
Of course² we have to consider C++ is very much more complex and complete language than C#. It takes a LOT more time to master C++ (if that is even possible) than it is needed to master C#. When I say C++ is more suitable to do this than C# I'm considering you have the knowledge to do that in C++ (you will use C++'s garbage collector system, you will use STL and boost, you will use smart pointers, you know code patterns and when to use each in C++, etc).

To conclude I really suggest you take a day or two to read a bit more about code quality and the importance behind it in a long-term software development.

Best regards.