[Development] MuServer on Node.js

[pafa7a]

Hi, ragezoners

Today I decided to play a bit with MuOnline sources and I've saw that there aren't many solutions for running a MuServer on Linux/Mac OS. Everything is coupled with Windows, which is not great, because of the prices for hosting a Windows server. Not to mention the speed, reliability, security, etc.

Well, I decided to start this initiative with a purely educational purpose. I am far from thinking that I will bring it to production ready. I'm not even sure if I won't give up after a while. I'm working on this one only when I have free time and when I want to

So far what have I done:

We've got a Node.js ConnectServer who starts 1xTCP & 1xUDP servers. The TCP server listens on port 44405 and it's purpose is to communicate with the Client. The part with the Client communication is 100% done. The UDP server is for communicating with GS & JS. The progress there is on 50% - the part with GS is done, JS is still in progress (need to check what's the purpose of the data comming from JS and if I can use it. Otherwise, I can ignore it. Will see)

Since it is javascript, I thought it would be better if instead of ServerList.dat we have ServerList.json, respectively in a valid JSON format.

I've implemented an option to reload the settings in ServerList.json via the terminal - you don't need to restart the server for the changes to take effect. I also implemented something that ordinary ConnectServers don't have - on reload, to send a packet to all connected clients to update their information as well - for example, if I add a new server in ServerList.json, I can type "reload" and see my changes directly in the client (without having to restart ConnectServer or Client).

After some time I will probably release a Github repo and make it open-source. I will most likely restructure this post as well so that it is more organized and not so much focused on ConnectServer, but rather on the whole picture.

Right now I have the following list in my head of things to put on Node.js: ConnectServer, JoinServer, DataServer, GameServer.

Demo video of ConnectServer.js:

Please share an opinion what you think about this project, would you take part in open-source and what do you think in general xD

Spoiler: Update 11.03.2023

Github repo:

You must be registered to see links

Spoiler: Update 11.04.2023

- Updated root README.md
- Created a Discord server for the project -

You must be registered to see links

- The libs "mu-packet-manager" and "mu-db" are now publicly available in npmjs.com
- JoinServer TCP handlers are moved into separate files
- Added handler for login packets & returning response to GS
- Created GitHub project to tackle all future development & contributions:

You must be registered to see links

- Registered domains "

You must be registered to see links

" and "

You must be registered to see links

"

Upcoming plans:
- Complete user login logic in JoinServer, so it's using the DB to verify the credentials & returns the proper response
- Refactor the ConnectServer TCP & UDP servers - move all handlers into separate files

 

[ijk]

Well done, looking forward to see your project on Github.

I'm trying to rewrite server with modern cpp and make it work on Linux/MacOs/Windows

 

[nevS]

Well, based on my experience with my project (see my sig), this is an very ambitious project
However, do what you think is best. If I had listened to other people when I started, I wouldn't be where I am now...

 

[marecznyjo]

Hi, ragezoners

Today I decided to play a bit with MuOnline sources and I've saw that there aren't many solutions for running a MuServer on Linux/Mac OS. Everything is coupled with Windows, which is not great, because of the prices for hosting a Windows server. Not to mention the speed, reliability, security, etc.

Well, I decided to start this initiative with a purely educational purpose. I am far from thinking that I will bring it to production ready. I'm not even sure if I won't give up after a while. I'm working on this one only when I have free time and when I want to

So far what have I done:

We've got a Node.js ConnectServer who starts 1xTCP & 1xUDP servers. The TCP server listens on port 44405 and it's purpose is to communicate with the Client. The part with the Client communication is 100% done. The UDP server is for communicating with GS & JS. The progress there is on 50% - the part with GS is done, JS is still in progress (need to check what's the purpose of the data comming from JS and if I can use it. Otherwise, I can ignore it. Will see)

Since it is javascript, I thought it would be better if instead of ServerList.dat we have ServerList.json, respectively in a valid JSON format.

I've implemented an option to reload the settings in ServerList.json via the terminal - you don't need to restart the server for the changes to take effect. I also implemented something that ordinary ConnectServers don't have - on reload, to send a packet to all connected clients to update their information as well - for example, if I add a new server in ServerList.json, I can type "reload" and see my changes directly in the client (without having to restart ConnectServer or Client).

After some time I will probably release a Github repo and make it open-source. I will most likely restructure this post as well so that it is more organized and not so much focused on ConnectServer, but rather on the whole picture.

Right now I have the following list in my head of things to put on Node.js: ConnectServer, JoinServer, DataServer, GameServer.

Demo video of ConnectServer.js:

You must be registered to see links

Please share an opinion what you think about this project, would you take part in open-source and what do you think in general xD

What I can recommend immediately is to begin converting it from Javascript to Typescript. With such complicated project you will either have to write zillions of unit tests or you will run into thousands of issues due to millions of small typo mistakes that can happen. Static typing will help you tremendously and keep your code base maintainable.

It's especially important if you want to go open source with it. I don't believe you'll have the time to prepare a very thorough documentation with extensive testing - Typescript will at least make the code easier to work with for someone joining your project later.

Beside that, as nevS said - it's an ambitious project and I wish you all the best!

 

[wolfulus]

I did this 8 years ago, perhaps these might help you:

You must be registered to see links

(define structures for parsing and serialization)

You must be registered to see links

helpers for reading/writing into a binary buffer

You must be registered to see links

old native enc/dec module to work with packet encryption

This last one won't even build anymore because this is 8 years old and it used GYP.

You can try upgrading it to a newer node version if you want, you just need to grab the source using `npm install mupacket --ignore-scripts` and look at `node_modules/mupacket` folder because I got rid of those projects on my GitHub page.

Also, if I could give you a tip, just don't get stuck with old server poop. I saw that you're trying to mimic the original server architecture, and this is a complete BS. You just need to speak the game's protocol, what's behind it doesn't matter.

For example, Redis can be your best friend here. For example your joinserver can literally be a redis instance (you just need an atomic get/set of account status), your global messages can travel through redis, your connectserver can just subscribe to a "gameservers" topic and let gameservers publish their own ip/port with a TTL and let redis itself expire them whenever gameserver goes down. ExDB data can also live inside redis, and database tasks just need to be ordered. Guess what? Add them to a message queue and process them in order.

Also, those "dataserver" "connectserver" "joinserver" "exdb" just exists to solve problems that were hard to solve at the time they were developed.

- ConnectServer exists just to provide a list of "live" servers and propagate status information for those connected.
- DataServer exists to solve ordered database access (for example logout and login again, you'll never see your character rollback levels because server did not save it yet)
- JoinServer exists to make it impossible to log two accounts at the same time (acts as a global distributed k/v lock). Also has some management BS in it and distributes global messages across game servers.
- ExDB just holds relational guild info in memory for quick access.

Anyways, good luck!

 

[Zidax]

Very interesting project you have there mate. Can't wait to try the actual game.

 

[pafa7a]

Update:
Github repo:

You must be registered to see links

Added a test coverage for the ConnectServer, reorganized the file structure a bit (everything was in a single index.js file before )

 

[rodrigoreisb]

Hey bro that's awesome. Can you upload the client for testing?

 

[pafa7a]

Hey bro that's awesome. Can you upload the client for testing?

You must be registered to see links

This is the main that I'm using while building the node.js project. The only changes in the client that I've did was to change the IP and the port

 

[rodrigoreisb]

I took a look at your project but it's not clear for me how to run it, with testing purposes. Would be great if you add some text instructions on how to do run the server for test (even without having to set a database).

 

[pafa7a]

I took a look at your project but it's not clear for me how to run it, with testing purposes. Would be great if you add some text instructions on how to do run the server for test (even without having to set a database).

Thanks for the suggestion. I've updated the README file and added a guide how to use the project + "dockerized" the project, so there are two ways of setup

 

[rodrigoreisb]

Thanks for the suggestion. I've updated the README file and added a guide how to use the project + "dockerized" the project, so there are two ways of setup

Thanks a lot bro. I will check it out as soon as I have some free time. Keep up the good job.

 

[pafa7a]

Update 11.04.2023:
- Updated root README.md
- Created a Discord server for the project -

You must be registered to see links

- The libs "mu-packet-manager" and "mu-db" are now publicly available in npmjs.com
- JoinServer TCP handlers are moved into separate files
- Added handler for login packets & returning response to GS
- Created GitHub project to tackle all future development & contributions:

You must be registered to see links

- Registered domains "

You must be registered to see links

" and "

You must be registered to see links

"

Upcoming plans:
- Complete user login logic in JoinServer, so it's using the DB to verify the credentials & returns the proper response
- Refactor the ConnectServer TCP & UDP servers - move all handlers into separate files

 

[exe19890522]

nice job,i wish you will success on mu.Although I am not studying Mu, I also want to study other games and use Nodejs to create an emulator

 

[zipper20032]

Interesting project, but as far as I know from my experience with Node, it's having an awful computational power because MU it's not a JavaScript based game or a web game.
I think it's good to replace the DataServer, ConnectServer and JoinServer with a Node app because these ones are being used at some certain specific moments defined, similar to a web API being called, and these don't require high computation. But for the GameServer, there's a total different story, and you probably saw a GameServer source code. Node JS is having many limitations, and the main loop is async, so non-blocking IO is applied here (e.g. a character will receive the position correctly and the other characters around your character aren't cause the performance was bad during the constant position update and other million things, and being async means it doesn't care in what order the things are arriving or are processed without having a processing index order or timestamp on send).
Anyway, would be interesting to see it, so good luck.

 

[pafa7a]

Interesting project, but as far as I know from my experience with Node, it's having an awful computational power because MU it's not a JavaScript based game or a web game.
I think it's good to replace the DataServer, ConnectServer and JoinServer with a Node app because these ones are being used at some certain specific moments defined, similar to a web API being called, and these don't require high computation. But for the GameServer, there's a total different story, and you probably saw a GameServer source code. Node JS is having many limitations, and the main loop is async, so non-blocking IO is applied here (e.g. a character will receive the position correctly and the other characters around your character aren't cause the performance was bad during the constant position update and other million things, and being async means it doesn't care in what order the things are arriving or are processed without having a processing index order or timestamp on send).
Anyway, would be interesting to see it, so good luck.

I also can't wait to see what will happen when I start implement the GameServer critical part. If node.js can't handle it most probably will end up by splitting some part to a different language (for example Rust) that can be a separate process started from Node and doing it's thing. Will see One thing I know for sure - I don't want to use C/C++/C#, because I want to do something different & new

 

[zipper20032]

Sure! That's good! Yes, I've read about Rust and it's pretty awesome for game servers.

 

[pafa7a]

progress update:
Account authentication handler in the JoinServer - done.
We have a new server called "WebServer" that contains 'mu-web-admin" directory which is a Next.js project. So far I've implemented the protected pages & routes logic, implemented the login & auth mechanism, security mechanism. Next time I will focus on styling the login page, implement a logout & style the core of the dashboard
also the web is added as part of the docker-compose setup and it builds & runs the production version of the app.

The idea of this mu-web-admin will be to have everything that you need as an administrator on one place. For example - stop & start the servers, configure the servers, watch logs in real-time, create/edit/delete accounts, modify warehouse, etc. All admin tools that you can think of in just one place accessible from your browser

 

[Sam3000]

I did this 8 years ago, perhaps these might help you:

You must be registered to see links

(define structures for parsing and serialization)

You must be registered to see links

helpers for reading/writing into a binary buffer

You must be registered to see links

old native enc/dec module to work with packet encryption

This last one won't even build anymore because this is 8 years old and it used GYP.

You can try upgrading it to a newer node version if you want, you just need to grab the source using `npm install mupacket --ignore-scripts` and look at `node_modules/mupacket` folder because I got rid of those projects on my GitHub page.

Also, if I could give you a tip, just don't get stuck with old server poop. I saw that you're trying to mimic the original server architecture, and this is a complete BS. You just need to speak the game's protocol, what's behind it doesn't matter.

For example, Redis can be your best friend here. For example your joinserver can literally be a redis instance (you just need an atomic get/set of account status), your global messages can travel through redis, your connectserver can just subscribe to a "gameservers" topic and let gameservers publish their own ip/port with a TTL and let redis itself expire them whenever gameserver goes down. ExDB data can also live inside redis, and database tasks just need to be ordered. Guess what? Add them to a message queue and process them in order.

Also, those "dataserver" "connectserver" "joinserver" "exdb" just exists to solve problems that were hard to solve at the time they were developed.

- ConnectServer exists just to provide a list of "live" servers and propagate status information for those connected.
- DataServer exists to solve ordered database access (for example logout and login again, you'll never see your character rollback levels because server did not save it yet)
- JoinServer exists to make it impossible to log two accounts at the same time (acts as a global distributed k/v lock). Also has some management BS in it and distributes global messages across game servers.
- ExDB just holds relational guild info in memory for quick access.

Anyways, good luck!

Hey, mupacket unavailable, can you share? Want to look at it

 

[pafa7a]

Hey, mupacket unavailable, can you share? Want to look at it

His repo is unavailable, but you can explore the code directly from npmjs:

You must be registered to see links