Welcome!

Join our community of MMO enthusiasts and game developers! By registering, you'll gain access to discussions on the latest developments in MMO server files and collaborate with like-minded individuals. Join us today and unlock the potential of MMO server development!

Join Today!

deletexxxy

Initiate Mage
Joined
Aug 6, 2014
Messages
56
Reaction score
42
This is actually incorrect and many, if not all sources do this (Even Extalia did when I visited them back in day). In GMS the drops happened as soon the mob died, not after the dead animation. Feel free to confirm in old GMS videos ;)

You can especially see this when Bigfoot gets killed. In pretty much all servers the drops are delayed due to long butt animation, but in GMS they instant dropped.

Surely this is not right.

I'm assuming this video is around v62, since it's from 2008 and about pirates:


Look at the drops from every monster (orange mush, green mush, red snail, slime) - the items drop after the death animation.
Then here's a side-by-side comparison:
no delay:


delay:


Which one do you think most closely resembles the video?

I'm not privy to how nexon handles Bigfoot's drop delay, but in general, there is definitely a delay.
Maybe there is a misunderstanding as to what 'death animation delay' really is.
 
Moderator
Staff member
Moderator
Joined
Jul 30, 2012
Messages
1,102
Reaction score
430
Surely this is not right.

I'm assuming this video is around v62, since it's from 2008 and about pirates:


Look at the drops from every monster (orange mush, green mush, red snail, slime) - the items drop after the death animation.
Then here's a side-by-side comparison:
no delay:


delay:


Which one do you think most closely resembles the video?

I'm not privy to how nexon handles Bigfoot's drop delay, but in general, there is definitely a delay.
Maybe there is a misunderstanding as to what 'death animation delay' really is.

From what I understand (but this is while ago), Nexon handles all the drop speed by mob death and skill usage. Example, if you use genesis, there's a delay of when the drops drop until the holy lights blast on the mobs (even though server sided the mob is dead as soon you pressed the genesis button afaik). To be honest I am not 100% sure how Nexon handles it, but long ago (2 years+) I did look at a lot of videos and concluded its surely not by dead animation only (dropping when dead animation is over), but who knows.

I could be wrong obviously.

The way I have done it (while not the most clean way) is giving each skill an exclusive drop delay, but none of them is after mob dead animation, but none of them is also instant due to the delays of weapons, skills, etc.

Example, an arrow shooting would be really looking ugly if the drop is instant, the mob would drop the items way too early, and not when its dead yet on screen. Same story for Genesis and similar skills like mentioned earlier. They have like a few second delay, but i made it drop as soon the holy lights hit the mob, like (AFAIK) GMS did according to videos I saw back then.

Anyway, this is a really minor detail that I/we have spend way more time on then I would like to admit those years ago, just for 'authentic' value :p
 
Last edited:
Initiate Mage
Joined
Aug 6, 2014
Messages
56
Reaction score
42
From what I understand (but this is while ago), Nexon handles all the drop speed by mob death and skill usage. Example, if you use genesis, there's a delay of when the drops drop until the holy lights blast on the mobs (even though server sided the mob is dead as soon you pressed the genesis button afaik). To be honest I am not 100% sure how Nexon handles it, but long ago (2 years+) I did look at a lot of videos and concluded its surely not by dead animation only (dropping when dead animation is over), but who knows.

I could be wrong obviously.

The way I have done it (while not the most clean way) is giving each skill an exclusive drop delay, but none of them is after mob dead animation, but none of them is also instant due to the delays of weapons, skills, etc.

Example, an arrow shooting would be really looking ugly if the drop is instant, the mob would drop the items way too early, and not when its dead yet on screen. Same story for Genesis and similar skills like mentioned earlier. They have like a few second delay, but i made it drop as soon the holy lights hit the mob, like (AFAIK) GMS did according to videos I saw back then.

Anyway, this is a really minor detail that I/we have spend way more time on then I would like to admit those years ago, just for 'authentic' value :p

Ohh, I see what you mean. I thought you were saying there was no delay, but after seeing Genesis it makes sense now. You're right, it's not just the death animation duration!
The comment in my first post was because I didn't see any sort of delay in the code.
 
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Since no one is actually talking about the code, I have some questions. Forgive me if some of them are naive; I haven't worked in C++ much.

1. Why is there no scripting engine? The point of a scripting engine is to decouple the actual scripts (language, framework, API) from the core code, and avoid a recompilation. What benefits did you see in removing it? As a follow up, you only have around 30 npc scripts; were there more when a scripting engine existed for Titan? If so how would you add the vast number of scripts that were present then or are present in Odin sources? Do you have any plans to write a language converter or transpiler?

2. I took a brief look into the code and didn't see many smart pointers - is there any reason why it's not consistent? You mentioned that you focused on performance and memory efficiency. Have you done any load testing and/or used tools like valgrind to expose memory leaks through high coverage scenarios? How can you support your claims? A consistent theme across MapleStory development is a lack of load testing tools to objectively analyze performance. Most people equate performance to memory consumption on startup, which is a highly flawed metric, and it seems like you've done the same without offering more detail.

3. I noticed the architecture/structure is actually very similar to post-RMI-removal Odin, but there are some peculiarities in where data is held. Why do you load all guilds and player names? Why are fame timestamps kept in the World server? How come online players are kept in the world, instead of in each channel? How do you efficiently send a message to only players in a certain channel?

4. I have not seen any synchronization in key parts of the code, like in packet sending and throughout Map. Am I mistaken here? If not, have you actually tested the implications of async writes per-client? I haven't worked with asio in a while, but do you have a write queue with a strand to handle this? What is the highest number of players you've had in the same map? If that number is 1-2, I expect to see a multitude of concurrency problems and unexpected behavior as that number scales.

5. I noticed in your drop expiry code you have a single task that runs every 8 (or 180) seconds that clears drops and respawns monsters. Unless I'm misunderstanding something, this does not have expected behavior, where each individual drop lasts for 180 seconds. Consider the case where the expiry task ran, and 5 seconds later, an item was dropped. In the next run, it would be there for 175 seconds and would not be removed, which means it actually stays on the map for 355 seconds instead of 180. Since the same Mob objects are used throughout the life of a Map, this will have race conditions because the code does not seem to be threadsafe, and still wouldn't simulate variable numbers of spawns from spawn points like in GMS.

6. I haven't dug into your wz loading code, but what benefits are there to load directly from the .wz files rather than another format (sql, xml)? Does the wz format have faster than O(N) lookups? What about the nx format, does that have faster than O(N) lookups? Have you explored it as a potential solution?

7. It is very unclear to me why all the packet handling code is in Player, even for login packets, even though the Player class seems to represent a logged-in Character object (with stats, guilds, etc). Is there any reason why you removed the "Client" abstraction and have this kind of implementation?

8. The PacketCreator usage seems a little disappointing; surely there is a better way to send packets than doing "PacketCreator packet26; ... PacketCreator packet37; ..." ?

Edit:
9. It also seems that there is no spawn delay for items dropped from monsters. Is this correct? The delay for dropped items need to match the duration of the monster's death animation. Is this anywhere in the source?

Thanks for your time!

Hi, thank you for the questions, which I gladly answer.

First of all I want to state that I have not taken any java/odinms source/repack as a basis for design choices and such.
Originally, I started with TitanMS2, but over the years I completly re-wrote it myself.

1. There were multiple considerations for having no scripting engine.

1.1.: It creates a dependency on another library.

1.2.: It requires creating matching script functions for the C++ functions where needed.

1.3.: Ofcourse, native is more efficient in therms of memory and speed.

It's true that there aren't very much npcs, mostly basic like job advancers and some more.
As I explained, MS features still need some work including this.

Now ofcourse this is something that can and needs to be re-valueted from time to time and different people will come to different conclusions.
Now that work focuses more on MS-Features, this will need to be re-checked again.

2. As, especially C++11 brought new features including smart pointers, auto, constexpr, and so on, I considered where and how to use those new features. There were a total of thousands of revisions, including alot of tests with smart pointers. I have deliberately created a new version control basis without old history's, just saying this addiotionally here. Anyways, I have decided to use smart pointers for some code/classes where it in my oppinion is more needed, first, for example, the Item related code. It's true that it isn't consistent yet. Though this isn't much about memory overhead in my oppinion. unique_ptr generally doesn't have overhead that really needs to be mentioned, and shared_ptr rather for reference counting. I am not using valgrind or other such tools (yet), but I do check for memory leaks and smart pointers ofcourse are, to my experience, unless there is real missusage like a dead reference to some pointer, immune to memory leaks.

I use performance testing tools, mostly "Very Sleepy", though. That tool is a profiler which shows the amount of cpu time and real time that was spent in functions and details to it, like the percentage of the total cpu time usage of function x, which helps alot.

Startup is also very fast, as I said. And that includes, as I also said, the loading of all .wz data that the server uses. Now that isn't definite proof for good performance generally, but gives some hints that that is the case, same goes for memory usage, which is very low after the startup is done.

3. I cannot comment on odinms and such structures, because I have never extensively checked that out.
Now the fame, player names and guild stuff goes back to some idea I had: Is it possible to reduce mysql querys while not loosing functionality?
With that special handling, I wanted to check that out. I think it's an interesting and working thing. It goes like this:
On server startup, guild data is loaded. Any guild data is only manipulated in the application memory. At server shutdown, the guild data is saved to mysql. Fame data is generally only held in the memory while the application runs. Now that's not fully GMS like, as then at some point the fame dates are lost, but as I said this were just some ideas that might be subject to change at some time again. And for player names: They are also loaded at startup and then worked with in memory. For some features like buddylist's and other stuff names are needed and this avoids some mysql querys. Now ofcourse this also can be questioned and I am willing to re-consider this at anytime. Actually, right now there is no way to send packets to players in a specific channel. Actually the reason is because I never ran into a case where that was needed, as far as I remember. But if you can name me ideas where that might be needed, I will consider adding this.

4. As currently it is not in fact multithreaded in therms of multiple threads being actually used (apart of 1 thread for acceptor and 1 for all handling stuff and 1 for asio timers internally by asio), no locks or whatever are needed.
My highest players online was 45 for a short time (up to 1 hour). There were no issues with lags or high cpu usage or such.
I have not used strand yet, but I am positive that the current code can handle very much players concurrently.
The asynchronous pattern has much benefits in my oppinion, even though it's full potential is not yet used due to lack of multiple worker threads currently. But it's true that intensive testing is needed for asynchronous writes and such.

5. As I said, as currently there aren't multiple worker threads, thread safety is not needed currently and not an issue.
As for the timing, you are corrent, it isn't fully GMS-like nor fully correct rigth now. I already planned to change that so drops are correctly handled and monster spawns also need to be checked to be made really GMS-like and correct. But it works well currently (especially monster spawns).

6. First I want to go into the performance related question. As I stated, there are a huge number of wz loading operations on server startup, with my current code it's very fast, therefore I accepted this as an optimal solution. I used vana's MCDB for some time and before that wz xml's, but I think direct .wz loading is the best solution for multiple reasons. Though, I currently cannot state what Big O is as I have yet to calculate and properly find it out.

Pro's of .wz loading include:

* no need for additional conversation/steps/files in between, usage of the official files with it's content
* with my code, loading it is very fast
* this allows an, in my oppinion, rather easy version change: sql (for example mcdb) is stuck to specific coloumns and such, therefore, this needs changes often, and just like wz xmls it needs to be extracted first, which again needs a tool on it's own

These are the primary reasons for it.

7. The Player class indeed combines very much code and it could have been done different, but I think the way it is done is acceptable, even though I take skeptical views on this seriously.

8. Actually I re-coded packetcreator not long ago while in private, I think this way is the best, even though I agree the variable naming looks a bit weird.
You create a PacketCreator object, which itself allocates some memory. Then you write packet data on that memory, if needed, it allocates more with realloc. Then it's send to one or more clients, including copying the packet bytes to a new buffer which is allocated for an async write operation to the client. By RAII, once the scope is left, PacketCreator object is destroyed and it's destructor deallocates the packet memory. I think this is quite a good solution, but I welcome suggestions from anyone, including you.

9. I will need to further check how it is in GMS first, before commenting this point.

I hope I could answer the questions well enough, thanks for your interest and you can ofcourse answer to this and react to my answers point by point.

Greetings

Buya
 
Initiate Mage
Joined
Aug 6, 2014
Messages
56
Reaction score
42
Thanks for taking the time to reply! I have some more comments and counterpoints:

1. There were multiple considerations for having no scripting engine.

1.1.: It creates a dependency on another library.

1.2.: It requires creating matching script functions for the C++ functions where needed.

1.3.: Ofcourse, native is more efficient in therms of memory and speed.

1.1: Why does this matter? Is this really a dealbreaker? You have no issue using libraries already (asio, poco), and aren't libraries meant to be used? I'd understand if the scripting engine library is very large and you wanted to keep size down when distributing a binary, but that's irrelevant here. I guess the worst case is longer compile time? But if that's really a consideration, think of all the compiling it would save when you want to create a new NPC script, or a quest script, or portal script? Or 15 in the same day? What about when changing scripts or adding functionality? What if there's an exploit/bug in one of the scripts that you need to fix immediately? Are you going to recompile and restart the server for all of those cases?

1.2: Well yeah, that's kind of the point! If you expose core functionality through a new API, then SomeRandomGuy54 can come in and start coding scripts without having any knowledge of where to add handling for it, or which functions are in which parts of the core code! The Java section had several guides to NPC scripting, help threads, etc. which required very little knowledge of how the server code really worked. It's a great way to get people to contribute without the daunting task of knowing the server code.

1.3: This is a nonfactor. We're talking about milliseconds here. I would take a 10x slowdown for the benefits I've listed above any day. Consider that the player doesn't care if it took the NPC window 1ms or 50ms to show. The difference in player experience is negligible.

Please use a scripting engine!

2. As, especially C++11 brought new features including smart pointers, auto, constexpr, and so on, I considered where and how to use those new features. There were a total of thousands of revisions, including alot of tests with smart pointers. I have deliberately created a new version control basis without old history's, just saying this addiotionally here. Anyways, I have decided to use smart pointers for some code/classes where it in my oppinion is more needed, first, for example, the Item related code. It's true that it isn't consistent yet. Though this isn't much about memory overhead in my oppinion. unique_ptr generally doesn't have overhead that really needs to be mentioned, and shared_ptr rather for reference counting. I am not using valgrind or other such tools (yet), but I do check for memory leaks and smart pointers ofcourse are, to my experience, unless there is real missusage like a dead reference to some pointer, immune to memory leaks.

I think you misunderstood what I was saying; I agree with you that smart pointers are what should be used. I just noticed that you use raw pointers in various parts of the code as well, and I was wondering why it wasn't consistently all smart pointers.

I use performance testing tools, mostly "Very Sleepy", though. That tool is a profiler which shows the amount of cpu time and real time that was spent in functions and details to it, like the percentage of the total cpu time usage of function x, which helps alot.

Startup is also very fast, as I said. And that includes, as I also said, the loading of all .wz data that the server uses. Now that isn't definite proof for good performance generally, but gives some hints that that is the case, same goes for memory usage, which is very low after the startup is done.

Yeah this is my problem with current server "performance analytics." Why this is all inherently flawed comes down to code coverage. Startup performance and memory usage have no relevance to server performance. How can it? The startup code is maybe 2-5% of all code paths!
A profiler will not help unless most code paths are being used. Otherwise, you won't accurately know what's a bottleneck! Without adequate code coverage (achieved through a great testing framework, or through sheer ccu) you cannot effectively gauge server performance and behavior. This is why there are so many random problems with Java sources, like monsters stop spawning and other strange behavior. There just isn't any functional/load testing.

3. I cannot comment on odinms and such structures, because I have never extensively checked that out.
Now the fame, player names and guild stuff goes back to some idea I had: Is it possible to reduce mysql querys while not loosing functionality?
With that special handling, I wanted to check that out. I think it's an interesting and working thing. It goes like this:
On server startup, guild data is loaded. Any guild data is only manipulated in the application memory. At server shutdown, the guild data is saved to mysql. Fame data is generally only held in the memory while the application runs. Now that's not fully GMS like, as then at some point the fame dates are lost, but as I said this were just some ideas that might be subject to change at some time again. And for player names: They are also loaded at startup and then worked with in memory. For some features like buddylist's and other stuff names are needed and this avoids some mysql querys. Now ofcourse this also can be questioned and I am willing to re-consider this at anytime. Actually, right now there is no way to send packets to players in a specific channel. Actually the reason is because I never ran into a case where that was needed, as far as I remember. But if you can name me ideas where that might be needed, I will consider adding this.

Maintaining things in memory without having write-through is highly flawed - I'm sure you understand this. What if there is a segfault or crash and your server does not gracefully exit? There is no way to guarantee that your shutdown function will be run. Please don't say that your server won't crash.
Also consider that MySQL itself is highly performant - far more performant (and scalable) than your server. It can handle the requests you send to it, and you need to use it for its purpose - to save data. There is too much risk and limited benefit to manipulating everything in memory without writing through.

4. As currently it is not in fact multithreaded in therms of multiple threads being actually used (apart of 1 thread for acceptor and 1 for all handling stuff and 1 for asio timers internally by asio), no locks or whatever are needed.

This is good, but I'm not sure how this would actually work well with larger numbers of players. If there are 100 players online and some bad packet causes 10 players in a map to d/c, processing for the other 90 players will be frozen while those 10 players are saved to the db, correct? I'm not sure if that's a good idea. If you end up responsibly saving data, this will also be magnified.
If you design your server to actually be scalable (not in 1 process) and fault tolerant (minimal state) then you can use several I/O threads and 1 processing thread per channel and get around these problems.

6. First I want to go into the performance related question. As I stated, there are a huge number of wz loading operations on server startup, with my current code it's very fast, therefore I accepted this as an optimal solution.
Yeah this is fine, I was just more curious on what made you settle for .wz specifically when there were alternatives.
Do also note that you may not need to load so much wz data. For example, you load all MobData and MapData but those probably shouldn't be preloaded. There are many monsters and maps which aren't encountered by players. Until they're necessary, they will just sit there taking up space in your heap. You will see the negative impacts of this as you update your server version and more properties and nodes are added to the wz files. You also only keep the bare minimum properties. When more features are added, you will need to load more properties. This is also why I was asking about lookup speed - there may come a time when it doesn't make sense to preload everything.


7. The Player class indeed combines very much code and it could have been done different, but I think the way it is done is acceptable, even though I take skeptical views on this seriously.
You need to have a Client object above Player which wraps the session and player object - this should handle the packet handling code. Honestly, look at Odin's object oriented design, it's good. Don't look at modern sources, but look at old Odin code like v62/OdinTeh. I hope the point of your project isn't to be different, but to create the best design. Naturally, having the best design will require investigation of how it has been implemented in the past.
For example, your map objects aren't behind an interface, but they should be.
There really should be no reason for a Player object which represents a character to have login packet handling code in it - there is no character during login!

8. Actually I re-coded packetcreator not long ago while in private, I think this way is the best, even though I agree the variable naming looks a bit weird.
You create a PacketCreator object, which itself allocates some memory. Then you write packet data on that memory, if needed, it allocates more with realloc. Then it's send to one or more clients, including copying the packet bytes to a new buffer which is allocated for an async write operation to the client. By RAII, once the scope is left, PacketCreator object is destroyed and it's destructor deallocates the packet memory. I think this is quite a good solution, but I welcome suggestions from anyone, including you.

Buya

This is where my limited C++ experience may not help, but what is wrong with making the packet functions into classes that extend PacketCreator? That way you can shrink these 3 lines:
PHP:
PacketCreator packet37;
packet37.SpawnMonster(mob, mob_constants::kSpawnTypesOldSpawn);
player->send_packet(&packet37);
into one:
PHP:
player->send_packet(&SpawnMonsterPacket(mob, mob_constants::kSpawnTypesOldSpawn));
And you get to keep the RAII benefits, right?

Edit:
I also want to say that I'm very impressed by your commitment to documentation and interaction. Good work!
 
Last edited:
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Thanks for taking the time to reply! I have some more comments and counterpoints:



1.1: Why does this matter? Is this really a dealbreaker? You have no issue using libraries already (asio, poco), and aren't libraries meant to be used? I'd understand if the scripting engine library is very large and you wanted to keep size down when distributing a binary, but that's irrelevant here. I guess the worst case is longer compile time? But if that's really a consideration, think of all the compiling it would save when you want to create a new NPC script, or a quest script, or portal script? Or 15 in the same day? What about when changing scripts or adding functionality? What if there's an exploit/bug in one of the scripts that you need to fix immediately? Are you going to recompile and restart the server for all of those cases?

1.2: Well yeah, that's kind of the point! If you expose core functionality through a new API, then SomeRandomGuy54 can come in and start coding scripts without having any knowledge of where to add handling for it, or which functions are in which parts of the core code! The Java section had several guides to NPC scripting, help threads, etc. which required very little knowledge of how the server code really worked. It's a great way to get people to contribute without the daunting task of knowing the server code.

1.3: This is a nonfactor. We're talking about milliseconds here. I would take a 10x slowdown for the benefits I've listed above any day. Consider that the player doesn't care if it took the NPC window 1ms or 50ms to show. The difference in player experience is negligible.

Please use a scripting engine!



I think you misunderstood what I was saying; I agree with you that smart pointers are what should be used. I just noticed that you use raw pointers in various parts of the code as well, and I was wondering why it wasn't consistently all smart pointers.



Yeah this is my problem with current server "performance analytics." Why this is all inherently flawed comes down to code coverage. Startup performance and memory usage have no relevance to server performance. How can it? The startup code is maybe 2-5% of all code paths!
A profiler will not help unless most code paths are being used. Otherwise, you won't accurately know what's a bottleneck! Without adequate code coverage (achieved through a great testing framework, or through sheer ccu) you cannot effectively gauge server performance and behavior. This is why there are so many random problems with Java sources, like monsters stop spawning and other strange behavior. There just isn't any functional/load testing.



Maintaining things in memory without having write-through is highly flawed - I'm sure you understand this. What if there is a segfault or crash and your server does not gracefully exit? There is no way to guarantee that your shutdown function will be run. Please don't say that your server won't crash.
Also consider that MySQL itself is highly performant - far more performant (and scalable) than your server. It can handle the requests you send to it, and you need to use it for its purpose - to save data. There is too much risk and limited benefit to manipulating everything in memory without writing through.



This is good, but I'm not sure how this would actually work well with larger numbers of players. If there are 100 players online and some bad packet causes 10 players in a map to d/c, processing for the other 90 players will be frozen while those 10 players are saved to the db, correct? I'm not sure if that's a good idea. If you end up responsibly saving data, this will also be magnified.
If you design your server to actually be scalable (not in 1 process) and fault tolerant (minimal state) then you can use several I/O threads and 1 processing thread per channel and get around these problems.


Yeah this is fine, I was just more curious on what made you settle for .wz specifically when there were alternatives.
Do also note that you may not need to load so much wz data. For example, you load all MobData and MapData but those probably shouldn't be preloaded. There are many monsters and maps which aren't encountered by players. Until they're necessary, they will just sit there taking up space in your heap. You will see the negative impacts of this as you update your server version and more properties and nodes are added to the wz files. You also only keep the bare minimum properties. When more features are added, you will need to load more properties. This is also why I was asking about lookup speed - there may come a time when it doesn't make sense to preload everything.



You need to have a Client object above Player which wraps the session and player object - this should handle the packet handling code. Honestly, look at Odin's object oriented design, it's good. Don't look at modern sources, but look at old Odin code like v62/OdinTeh. I hope the point of your project isn't to be different, but to create the best design. Naturally, having the best design will require investigation of how it has been implemented in the past.
For example, your map objects aren't behind an interface, but they should be.
There really should be no reason for a Player object which represents a character to have login packet handling code in it - there is no character during login!



This is where my limited C++ experience may not help, but what is wrong with making the packet functions into classes that extend PacketCreator? That way you can shrink these 3 lines:
PHP:
PacketCreator packet37;
packet37.SpawnMonster(mob, mob_constants::kSpawnTypesOldSpawn);
player->send_packet(&packet37);
into one:
PHP:
player->send_packet(&SpawnMonsterPacket(mob, mob_constants::kSpawnTypesOldSpawn));
And you get to keep the RAII benefits, right?

Edit:
I also want to say that I'm very impressed by your commitment to documentation and interaction. Good work!

Hi,

I will consider adding a scripting engine, even though I think it atleast isn't a high priority.

The reasons for non-consistent use of smart pointers are that I wasn't sure if I should use them everywhere for some time and it would take some work to change it. Also it would have limited benefit only. So that might be done in some time but is on a low priority right now.

To the performance and testing thing: It is right that some proper (automated) testing would be beneficial and I will try to work something out for that.

when it comes to the stuff where I applied that most-in-memory-handling like guild and fame data and the player names (though for player names it isn't the same as ofcourse when a character is created, it and it's name is saved to mysql just as most other data, too).
You are right about the risk of a crash or a non-standard shutdown, where guild data especially might be at risk. as the fame data is just to save some dates, that's of less importance. So yeah I accept the necessity of changes there, especially when it comes to guilds. I will start work on that soon. Even though I am not sure yet how it should be done: either save all guild changes, or do it as timed autosaving? Ofcourse you may tell me what you would choose.

You are right that multiple worker threads or even multiple processes need to be considered.
Though this also has to do with the way it is built up (login, world, channels), as I said, in my source it currently is not split into multiple servers.
Also I am thinking about using asynchronous querys, Poco offers that more or less. Currently it is blocking.

Now to your suggestion about Client/Player: I will check out some java (odinms based) repacks/sources to get some impressions of that.
Currently Session is on top and holds the Player object. Session handles the packet/sending and receiving, Player handles the specific packet reading and handling currently. I will definitely check if and how it could be done better.

As to .wz, I am sure that this is the right and best choice. Actually as far as I remember I found my .wz loading to be faster and better than the previous sql loading I had, that's also one reason for it. Your note about the pre-loading has some truth and ofcourse some later loading can be considered. In fact, at first I loaded map data dynamically when it was created, but after discovering that even though the whole data was loaded, the ram usage was low and the startup fast, I decided to make it pre-load everything. If I decide to do some work on an up-to-date GMS branch, I will defenitely check how the ram usage is at high versions when everything is pre-loaded. My standpoint is, that if ram usage is rather low and the loading is fast, I prefer full pre-loading of the .wz data for numerous reasons. The main reason why I prefer .wz loading is, that it's the original files/data. I have read from the .nx format and it's an interesting try, but I am not convinced (yet).

Regarding the packetcreator, your thought is not really right. Non-class functions like you thought of are the way it's done in C, as classes don't exist there. In fact as far as I know the odinms based repacks also have a packetcreator class or whatever where all packet functions are.
My oppinion on that is that it's good the way I made it because it clearly shows the steps it does. I don't think it's a good idea to try to make it into one line. Though I could imagine re-using the packetcreator object so it doesn't need to be created multiple times within the same function.

Thanks for your input and your apprecreation of my effort and work.

You are welcome to further reply, this is an interesting conversation.

Greetings

Buya
 
Initiate Mage
Joined
Aug 6, 2014
Messages
56
Reaction score
42
To the performance and testing thing: It is right that some proper (automated) testing would be beneficial and I will try to work something out for that.
+1! You'd be a first in the Maplestory community :D

when it comes to the stuff where I applied that most-in-memory-handling like guild and fame data and the player names (though for player names it isn't the same as ofcourse when a character is created, it and it's name is saved to mysql just as most other data, too).
You are right about the risk of a crash or a non-standard shutdown, where guild data especially might be at risk. as the fame data is just to save some dates, that's of less importance. So yeah I accept the necessity of changes there, especially when it comes to guilds. I will start work on that soon. Even though I am not sure yet how it should be done: either save all guild changes, or do it as timed autosaving? Ofcourse you may tell me what you would choose.
In this case, the best way to do this is simply to save every time data is changed. You can hold it in memory and process reads with the in-memory cache, but if something changes (like guild capacity, emblem, notice) then just save the change. Think about how often guild writes would happen (very infrequently); there is no benefit to saving all guilds on a timer. This is also the best way to guarantee data preservation.

You are right that multiple worker threads or even multiple processes need to be considered.
Though this also has to do with the way it is built up (login, world, channels), as I said, in my source it currently is not split into multiple servers.
Also I am thinking about using asynchronous querys, Poco offers that more or less. Currently it is blocking.
Asynchronous queries could work, but be careful about which ones you make async. Sometimes, you actually need to wait for the transaction to complete before moving on.
This is why you could have the main thread running the game loop and a processing threadpool which you can queue these tasks on, then they do the action and put the result onto the channel's events queue for the next tick. If I was going to design a MapleStory server, this is probably what I would do for the core processing.

Now to your suggestion about Client/Player: I will check out some java (odinms based) repacks/sources to get some impressions of that.
Currently Session is on top and holds the Player object. Session handles the packet/sending and receiving, Player handles the specific packet reading and handling currently. I will definitely check if and how it could be done better.
Yay! +1

As to .wz, I am sure that this is the right and best choice. Actually as far as I remember I found my .wz loading to be faster and better than the previous sql loading I had, that's also one reason for it. Your note about the pre-loading has some truth and ofcourse some later loading can be considered. In fact, at first I loaded map data dynamically when it was created, but after discovering that even though the whole data was loaded, the ram usage was low and the startup fast, I decided to make it pre-load everything. If I decide to do some work on an up-to-date GMS branch, I will defenitely check how the ram usage is at high versions when everything is pre-loaded. My standpoint is, that if ram usage is rather low and the loading is fast, I prefer full pre-loading of the .wz data for numerous reasons. The main reason why I prefer .wz loading is, that it's the original files/data. I have read from the .nx format and it's an interesting try, but I am not convinced (yet).
+1! I agree, preloading is usually the better choice, when applicable.


Regarding the packetcreator, your thought is not really right. Non-class functions like you thought of are the way it's done in C, as classes don't exist there. In fact as far as I know the odinms based repacks also have a packetcreator class or whatever where all packet functions are.
My oppinion on that is that it's good the way I made it because it clearly shows the steps it does. I don't think it's a good idea to try to make it into one line. Though I could imagine re-using the packetcreator object so it doesn't need to be created multiple times within the same function.
I think you may have misunderstood, I am not talking about "non-class functions". I am talking about classes which inherit from PacketCreator. Here is a sample implementation to make this more clear:
Instead of
PHP:
void PacketCreator::EndControlMob(int mob_object_id)
{    
    write<short>(send_headers::kSPAWN_MONSTER_CONTROL);
    write<signed char>(0);
    write<int>(mob_object_id);
}

...

// packet
PacketCreator packet1;
packet1.EndControlMob(mob->get_object_id());
control->send_packet(&packet1);

You could have:
PHP:
class EndControlMobPacket : public PacketCreator 
{
public:
    EndControlMobPacket(int mob_object_id) : PacketCreator()
    {    
        write<short>(send_headers::kSPAWN_MONSTER_CONTROL);
        write<signed char>(0);
        write<int>(mob_object_id);
    }
}

...

// packet
control->send_packet(&EndControlMobPacket(mob->get_object_id());

This has:
1. Simpler usage; now it is 1 line and still explicit what is getting constructed
2. No/negligible performance impact, and also still automatically frees the buffer when it goes out of scope.

There are probably other/better ways to solve it, but I'm not a big fan of 3 lines to send a packet. It gets messy in long methods where you have multiple packets to send, and then have to start naming the PacketCreators like packet26, packet1, packet37, etc.

Also, the whole reason for your implementation is because of automatically freeing the byte buffer - what if you just use a vector and keep it on the stack?

Thanks for your input and your apprecreation of my effort and work.

You are welcome to further reply, this is an interesting conversation.

Greetings

Buya
No problem man, you are certainly the most engaged person I've seen in a long time!
 
Last edited:
Joined
Oct 14, 2008
Messages
960
Reaction score
197
+1! You'd be a first in the Maplestory community :D


In this case, the best way to do this is simply to save every time data is changed. You can hold it in memory and process reads with the in-memory cache, but if something changes (like guild capacity, emblem, notice) then just save the change. Think about how often guild writes would happen (very infrequently); there is no benefit to saving all guilds on a timer. This is also the best way to guarantee data preservation.


Asynchronous queries could work, but be careful about which ones you make async. Sometimes, you actually need to wait for the transaction to complete before moving on.
This is why you could have the main thread running the game loop and a processing threadpool which you can queue these tasks on, then they do the action and put the result onto the channel's events queue for the next tick. If I was going to design a MapleStory server, this is probably what I would do for the core processing.


Yay! +1


+1! I agree, preloading is usually the better choice, when applicable.



I think you may have misunderstood, I am not talking about "non-class functions". I am talking about classes which inherit from PacketCreator. Here is a sample implementation to make this more clear:
Instead of
PHP:
void PacketCreator::EndControlMob(int mob_object_id)
{    
    write<short>(send_headers::kSPAWN_MONSTER_CONTROL);
    write<signed char>(0);
    write<int>(mob_object_id);
}

...

// packet
PacketCreator packet1;
packet1.EndControlMob(mob->get_object_id());
control->send_packet(&packet1);

You could have:
PHP:
class EndControlMobPacket : public PacketCreator 
{
public:
    EndControlMobPacket(int mob_object_id) : PacketCreator()
    {    
        write<short>(send_headers::kSPAWN_MONSTER_CONTROL);
        write<signed char>(0);
        write<int>(mob_object_id);
    }
}

...

// packet
control->send_packet(&EndControlMobPacket(mob->get_object_id());

This has:
1. Simpler usage; now it is 1 line and still explicit what is getting constructed
2. No/negligible performance impact, and also still automatically frees the buffer when it goes out of scope.

There are probably other/better ways to solve it, but I'm not a big fan of 3 lines to send a packet. It gets messy in long methods where you have multiple packets to send, and then have to start naming the PacketCreators like packet26, packet1, packet37, etc.

Also, the whole reason for your implementation is because of automatically freeing the byte buffer - what if you just use a vector and keep it on the stack?


No problem man, you are certainly the most engaged person I've seen in a long time!

Hi,

you are right when it comes to guild saving, actually there are not that often changes to it, so yeah, I will implement guild saving for all changes when they happen instead of at standard server shutdown. As for fame data and player names I will think further of that.

About the packetcreator, I think your idea isn't bad but what you proposed isn't practicable, as there shouldn't be classes instead of functions everywhere. Maybe the solution is simpler than it seems:
The packetcreator object can be locally scoped by using it like this:
PHP:
{
// packet
PacketCreator packet;
packet.EndControlMob(mob->get_object_id());
control->send_packet(&packet);
}
In that case, there wouldn't be a need to use numbers for the variable, which is indeed a bit weird and not so good.
So I will consider adding local scope brackets and stop using the numbering for the variable name.

I am not sure what you mean with using vectoring in this context. unsigned char arrays can be on the stack, but only when the size is rather little, as the stack has a limited size only. There needs to be care, else a stack overflow can/will occur at runtime. This would only work for packets can not only have a little size. Apart from huge packets, alot of packets have loops or whatever where it's not always predictable how huge it is going to be. But there optimization potential: If multithreading is not used, actually, it would be possible to have one pre-allocated huge packetcreator buffer that is always used, because the packetcreator buffer is copied to a buffer for the async_write function for each client, after the packet is created. so the packetcreator buffer has a little scope only right now. so there could be a buffer that is allocated on statup, having max packet size (around 65 kilobytes) which is always written on with the packet data. though for multithreading this approach will not work, atleast not without locks/atomics.

I will have to further study and test asynchronous querys and possible design changes to using multiple worker threads and whatever.

Thanks for your input and your appreciation.

Greetings

Buya
 
Joined
Oct 14, 2008
Messages
960
Reaction score
197
missed u bro

Hi,

good to see you are still around, I knew you would see this so I waited ^^
Also missed you ofcourse.

@all:

Everyone is welcome to contribute by suggestions, bug reports (here, by PM or by github) or by pull requests over git.

I am positive this project will bring this community and the developments forward.
It is of high importance to have a very good basis regardless of which version is being developed.

Greetings

Buya
 
Last edited:
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Hi,

I got some news:

The guild data saving has been re-coded and works now perfectly.
It is now always saved to mysql upon guild creation and guild data changes, as it should be,
thanks Shmoconut for your comments, they made me do this fast.

also, I advise everyone who has already checked out/cloned/forked my github to do a fresh checkout/clone (delete the existing and get it again), because of internal changes. the github project stayed the same, though.
 
Experienced Elementalist
Joined
Mar 29, 2007
Messages
248
Reaction score
2
Thank you so much for your work !!
waiting for more updates and fixes ;)
Question -
I know that there are many things to fix but
do you know how to fix the animation of drops and mesos rates bug (i put rates to "1" and still mesos from a snail is 100 mesos)?
;)
 
Last edited:
Initiate Mage
Joined
Sep 21, 2015
Messages
4
Reaction score
0
you
Hi everyone,

I want to share the v0.62 localhost I use with you.
It has multiple features, including multiclient.

link:
or

noXXXXXXX
 
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Thank you so much for your work !!
waiting for more updates and fixes ;)
Question -
I know that there are many things to fix but
do you know how to fix the animation of drops and mesos rates bug (i put rates to "1" and still mesos from a snail is 100 mesos)?
;)

I cannot say when that will be fixed, in my oppinion it's not important considering that there is still work needed on MS-features. It will be fixed in the right time.

I now continue my work on skills, some work is needed for pirate skills and a few others.

Dear@Buya, can you show set tutorial?

I will post a tutorial thread with detailed information of how to set it up in some time.
 
Experienced Elementalist
Joined
Mar 29, 2007
Messages
248
Reaction score
2
I cannot say when that will be fixed, in my oppinion it's not important considering that there is still work needed on MS-features. It will be fixed in the right time.

I now continue my work on skills, some work is needed for pirate skills and a few others.



I will post a tutorial thread with detailed information of how to set it up in some time.
OK, i'll keep waiting :)
 
Initiate Mage
Joined
Nov 26, 2013
Messages
16
Reaction score
0
I cannot say when that will be fixed, in my oppinion it's not important considering that there is still work needed on MS-features. It will be fixed in the right time.

I now continue my work on skills, some work is needed for pirate skills and a few others.



I will post a tutorial thread with detailed information of how to set it up in some time.
i want to see a tutorial too!
 
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Hi everyone,

sorry but the tutorial will take longer.
The source includes a short guide to setting it up already, though.

I am currently implementing BBS feature for guilds.

More improvements to MS-features are planned after that is done: guild alliances and almost full skills (door and some pirate skills will be implemented, for example).

After those things, monster carnival pq is planned, other pqs might be worked on too in some time.

Greetings

Buya
 
Joined
Oct 14, 2008
Messages
960
Reaction score
197
Hi everyone,

there is progress:

I am looking trough the client pseudo-code to correct packet structures, multiple commits regarding login and cashshop packets have been made.

This includes packet information, that as far as I know no one else had in this right way.

Also, the implementation of multiple more MS-features are planned (guild bbs under work, planned for example: guild alliances, some skills that don't work yet like door and battleship, proper cashshop inventory) and considered (rings, marriage, PQ's including monster carnival pq, MTS = Maple Trading System, minigames), it will take some time ofcourse but I am satisfied with the progress.

I will at some time publish a tutorial, sorry that it takes so long.
Also I will post some pictures and include them in main thread to show that this is real.

Also, I must say sadly that I couldn't get the v0.83 localhost to launch because I am on Windows 8.1. v0.62 works after multiple tries always. Therefore currently there are no plans to create a v0.83 branch. If I can get it to work sometime, I will gladly start a v0.83 branch. Either way I would delay that because it's more important to get more MS features to v0.62 first, which would then be an optimal basis.

Greetings

Buya
 
Last edited:
Back
Top