[C++] Serializing packets

[AcarX]

Hello people. I'm working on an emulator and I need some advice on how to serialize packets. The way I've been storing packet data until now is:

NetworkMessage packet = {    // Header and stuff
                             (BYTE)(characterID & 0xFF), 
                             (BYTE)((characterID >> 8) & 0xFF), 
                             (BYTE)((characterID >> 16) & 0xFF), 
                             (BYTE)((characterID >> 24) & 0xFF), 
                             // Rest of packet 
                        };

NetworkMessage is a simple class that wraps std::vector<BYTE> with some useful functions and operators. characterID is a 32 bit unsigned integer. Is there a better way of doing this in terms of cleaner code and better performance?

 

[ispyder]

If you want to write the complete serialization protocol yourself, use:
- Boost
- Cereal

If you want to write only the description of your objects and generate the serialization protocol automatically, use:
- Avro
- Thrift
- Google Protocol Buffers
- Google Flat Buffers
- Cap'n proto (written by the author of Google Protocol Buffers v2)

Also, check ZeroMQ for a high-performance asynchronous messaging library (I believe WoW TrinityCore uses it...)

 

[AcarX]

Thanks for the suggestions. Google Protocol Buffers looks interesting I'll check it out.

 

[ispyder]

Keep in mind that Apache Thrift also offers an entire RPC framework.

I think Avro also has RPC for Java, Python and Ruby.

PS: I find Avro to be ugly as hell. I find the syntax of Google Protocol Buffers and Thrift better.

 

[AcarX]

I got a few questions though. I stumbled upon this comparison:

You must be registered to see links

What'd be the upsides/downsides of using GoogleProtocolBuffers/FlatBuffers/raw structs?

 

[ispyder]

Check the following two articles:

You must be registered to see links

You must be registered to see links

 

[AcarX]

Thanks for the articles. From what I've read protocol buffers are used by Google, well tested and have a nice documentation whereas flat buffers are better performance wise but not as widely used as protocol buffers. So if I were to choose between protocol buffers and flat buffers which one would be better?

 

[ispyder]

Before choosing something for a big project you should try all of them by writing some simple examples to see with which you're more comfortable.

Also, it's better to choose something that:
- is more widely used and has better documentation (if it takes you hours searching google to get some sample codes then it's a big NO-NO)
- has an easier syntax and it's easier to understand because if it's a team project it has to be easy to learn by new people

If you choose something with fewer people using it and with a poorer documentation, you're kinda on your own because no one will be able to help you.

Personally, I'd choose between Thrift (nice syntax + has an RPC framework) and Protocol-Buffers (nicer syntax).

Performance-wise, they're both fast if used correctly:
- Thrift -> Facebook, later Apache
- Protocol-Buffers -> Google

It all depends on the code you write.

PS:
- Avro - it's syntax is ugly as hell: the C++ part of it is ugly. the JSON part is... I can't even express how ugly I find it... )
- Cap'n proto - also ugly as hell. I find it uglier than Avro
- Flat-Buffers - I don't like its syntax; it's nicer than Avro's and Cap'n proto's, but I still find it very ugly

 

[AcarX]

Thanks for all the information. I'll go with protocol buffers then.

 

[InCube]

Why don't you just use AES 256 public/private keys ?

 

[AcarX]

I couldn't quite catch that. I need a way to serialize packet structures so I cannot see how's it related with encryption.

 

[InCube]

I couldn't quite catch that. I need a way to serialize packet structures so I cannot see how's it related with encryption.

#pragma pack(push, 1) // exact fit - no padding
struct MyStruct
{
char b;
int a;
int array[2];
};
#pragma pack(pop)

and memcpy to BYTE*

 

[AcarX]

That looks.. Pretty simple and fast. I've tried using structs without the pragma before but it'd have 0xCC bytes due to padding. Thanks for the heads up it's a life saver.

 

[s-p-n]

If fast is what you're looking for, target the bottlenecks- with networking it's the network itself. It's not so much how you serialize/unserialize the data- that can be very, very fast. But what's the point in preparing the packets very fast, if you're using TCP and your programs are sitting there waiting for a response from eachother because of redundant checking by the protocol to assure the packets are completely received and in perfect order?

So think of your algorithm in general. If you can design a system that uses UDP and beats TCP speeds 99% of the time- for example, sending redundant packets and designing your systems in such a way that losing a packet, or receiving them out of order is okay, you can design a networking application that is very fast. It is much harder to do, but it's possible to use UDP without going completely insane. You just need to be very good at abstracting a low level system into high level operations, while maintaining the value that said low level system offers.

For example, if you need ordered packets, just give each one a number and fire them all out as soon as possible. TCP does this (be it, not necessarily ASAP) but it's complex.. Sometimes TCP will send something from A to B, and then a confirmation back to A that A must receive before sending more packets to B. It's better if A and B just keep throwing packets at each other without waiting on confirmation. In most use cases, the fact that you receive packets at all can assume a connection- so if you stop receiving packets in UDP for some period of time and under the condition you actually expect to receive packets, you can assume this connection is lost. In reality, there's no such thing as a connection between IPs. But networking is hard enough, so for brevity I won't bother bloating this post with how IP actually works.

 

[AcarX]

Thanks for the excellent answer. Although as I stated at the first post I'm working on an emulator and I do not have any option but to use the built in protocol which is TCP. I'll definitely consider using UDP if I ever get the chance to work on my own game though.

 

[ispyder]

I'll definitely consider using UDP if I ever get the chance to work on my own game though.

Also, read the articles on this blog:

You must be registered to see links

 

[s-p-n]

Thanks for the excellent answer. Although as I stated at the first post I'm working on an emulator and I do not have any option but to use the built in protocol which is TCP. I'll definitely consider using UDP if I ever get the chance to work on my own game though.

I'm forced to use TCP on my own game, so I understand

And Gaffer on Games is an excellent wealth of information btw.

 

[jonnybravo]

You could always use a Condom hahaha to serialize.

But in C++ you can do

create a class that holds the packet ID's then create a std::map and pattern to return it.

Or with a class that add's the packet ID then a class that registers the packetID useing a template that registers all packet Id's into one location.

Or you can create a class with ENUM packet ID's use a template to search thru those to grab the one you want then create a struct which then points to creating the object and then send it on its way.

 

[AcarX]

Well creating a struct for each type of packet felt like quite a hassle so right now I just have a class called 'PacketManager' which simply takes required data + a message container and fills it with packet data. While not optimal, it's at least better than having packets randomly in handler functions.

 

[Szaka]

Example of obiective handling received packets would look like this: Packet is an abstract class, it has virtual methods read() and execute(), and all you do is implement a factory pattern which would return derived instance from Packet.