[C++/JavaScript] Rapid Emulator Development

Has anyone ever thought about this? After a long and insightful discussion with a friend of mine and a lot of reflection, I've decided the way to go while developing emulators involves using any system setup so that code can be added on the fly without damaging the running state of the server.

It's even more powerful when that system involves an execution environment set forth by a language like C or C++ which is exposed to a scripting language capable of metaprogramming or dynamic modification of objects (Lua, Javascript, etc). Some dialects of Lisp are suited for this style of development as well.

My reflections on the idea prefer to settle into a C++ base and JS as the scripting engine. The idea here is to implement several classes (exposed as objects in JS) that provide common needed functionality for emulation, but leave the huge majority of the implementation of the emulator for the scripting language.

We need a class for Database interaction (primarily an interface like php's mysql interface that makes it simple to make queries and retrieve the results), a handful of classes for networking (creating a listening server, byte buffers, etc). There are some others which arise with need, potentially including encryption/decryption (though this can usually be easily implemented in JS). Finally, the JS needs to be able to dynamically load a file into the running context ('eval' doesn't work here unless you provide a file i/o api to JS so that it can read in the javascript source).

Once this is done, there are several issues involved here. Unless a powerful task pool of some sort is created for JS requests, you need to serialize all input to the JS engine. The easiest way when i/o is multithreaded is to simply put a mutex around the code that executes the JS handler callbacks. Once acquired, a callback in JS (implemented by some system.js file that should never change once written) will execute a local function implemented in JS with parameters of the connection ID (client id, unique per client.. this is required for serialization, optionally a client object could be passed, it makes no difference), along with a byte buffer for the data in the packet, and optionally the opcode (it could be extracted with the byte buffer either way).

In this function, a local property which contains an array is addressed (such as this.opcode_handlers). This array contains an associate map from opcode to handler function. If this map does not contain the opcode, some default action is fired. If it does, that function is executed.

Once this is done, we're set up for RED. We can construct files that implement a single packet or a group of packets that make up some feature in the emulator, and an index javascript file which loads the base-group of these files that we've created and tested at emulator startup. Each of these JS files will automatically grab the handler table and insert their own handler functions on load and will modify the prototype of any system objects as necessary to add the functionality (such as a client prototype to add a guild name, etc). Finally, the script must implement an uninstall function (say the file is test.js, it needs to implement test_uninstall). This will systematically undo the prototype changes and handler callbacks.

Now, once the emulator starts up, we can have our client connect to it and add code to handle packets in a separate JS file. Once we finish some code, that script can be uninstalled and re-loaded from disk. Then we can get to a point where we're in-game running around, testing features, then add an entire feature in-game with 20-50 lines of code out in JS, save the file, then in-game toss a command (like /loadhandler for GM commands), which will load that file. Now those features will instantly be available to all connected clients without dropping connection or requiring any kind of restart.

The end result is a development environment for implementing MMO features quickly without need to constantly restart the server and/or client. In fact, in the case of JS, a GM command could accept JS code and execute it in the proper context passed through in-game chat and permit the admin to replace individual function objects and properties without loading any external files to quickly test/fix functionality, which can then be added into the actual file, and once all features are tested and debugged, the script file can be added to the index so that when the emulator is restarted, it is automatically loaded and a part of the running emu.

Has anyone here thought about this or tried it? I've primarily chosen C++/JS due to their similarity and because v8 is quite powerful (speed is a nice bonus) and rewriting JS code to C++ after all functionality is implemented is incredibly easy. Also, v8 embedding and exposing is quite pleasant through the system they've designed.

I do feel that an approach like this could bring an incredible speed to developing emulators. Now a person can sit there with a packet log (yes, logging packets is another thread entirely, and a good packet analyzer can cut down dramatically on development time, but being able to dynamically send packets to the client through JS to test what fields do what is also incredibly powerful) and run through the features of the game, implementing them one by one without trying to run around changing lots of code to add new features (updating client object with new properties, adding functions, etc). This ends up cluttering up the code, making it difficult to maintain and extend. With the prototyping in JS, all functionality is modularized and easy to locate/maintain, and when all is said and done, the prototypes can be merged together to produce the finalized code; porting from JS to C++ is quite easy. In addition, some walking code can walk over active object prototypes, handlers, etc and generate some C++ code that resembles roughly the final object classes needed to implement all of the available functionality, and with a little bit of file API with regexes, you could even write an export system that incrementally converts the running JS emulator into a basic C++ emulator (not fully, but a large majority of the work for you).



Reading more and more of the v8 source code makes this feel like a better choice with every passing page of code :). A simple psuedo code formulation of the idea is below, I'm working on a little better personal documentation into v8 since what Google provides is terribly lacking. It's really a powerful engine and very well integrated into C++ embedding.

So the first file would be the system emulator file, the fixed server that shouldn't really ever change once it's completed (basically the emulator driver):

Code:

---

// system.js

function MyEmu(){
  this.next_unique_id = 1;
  this.game_callbacks = new Array();
  this.login_callbacks = new Array();
  this.client_list = new Array();

  this.onLoginConnect = function(from_ip){
    // Code here, send whatever packets, construct a new client object, etc...
    // Return the indicated unique ID (integer)
    return this.next_unique_id++;
  }

  this.onLoginData = function(conn_id,buffer){
    // Do whatever...
   
    // 1. Validate conn_id
    // 2. Check opcode and dispatch a handler
    var opcode = buffer.readShort();
    var handler = this.login_callbacks[opcode];

    if(handler == null || handler == undefined){
      // Call default...
    }else{
      handler(conn_id,buffer);
    }

    // Do whatever else...
  }

  this.onGameConnect = function(from_ip){
    // Do whatever...
  }

  this.onGameData = function(conn_id,buffer){
    // Do whatever...
  }
 
  // We first need to establish our listening servers
  this.login_server = new ListenServer();
 
  this.login_server.setListenMask('127.0.0.1/24');
  this.login_server.setListenPort(15779);
  this.login_server.setConnectionCallback(this.onLoginConnect);
  this.login_server.setDataCallback(this.onLoginData);

  // Game server
  this.game_server = new ListenServer(this.login_server);
  this.game_server.setConnectionCallback(this.onGameConnect);
  this.game_server.setDataCallback(this.onGameData);
}


// Include our addons
file_include('addons.js');

// Instantiate our emulator object and start the listen servers
var emu = new MyEmu();
emu.login_server.start();
emu.game_server.start();

---

An example of addons.js would be:

Code:

---

// addons.js
file_include('addons/login_server.js');
file_include('addons/game_init.js');
file_include('addons/game_server_list.js');
file_include('addons/game_server_char.js');
file_include('addons/game_server_chatsystem.js');
// ....

---

I'm working on a base system using v8 that should do all of this, I'll post the very base framework in source-code when I finish so you guys can give it a shot.

Wow I would really like to see this going further.

I love it. Wish you luck in that.

So after working on this for a while with v8, I got it working with exposing some basics. It does what I want, I'm satisfied with performance (it's actually very fast), and it does exactly what I had hoped. I'll lay out the basic system facilities, but I won't post the code, primarily because in the business of having an edge, you don't have one if you give everyone your tools :). That and it took 2000+ lines of code and many hours of me banging my head against the table trying to get v8 to do what I wanted (and still the memory management stuff is iffy, I need to force v8 to do GC cycles sometimes and I still need to implement factories and sinks for objects as GC cycles happen far too infrequently to accommodate large allocated objects, this is primarily because to me, "large" means the data I allocate is large, but to v8, the only data it sees is a small pointer in a v8::External, so it takes hundreds of thousands of these to trigger a GC pass).

I literally just took the shell example included with v8, moved all of the functions they defined into a Shell object then exposed that to the global object as "Shell", then in my JS for convenience I did:

PHP Code:

---

print = function(x){Shell.print(x+"\n");} 


---

And my Shell.print is modified to not put a newline at the end.

It's not by any means "complete" as there will be features coming even in the near future I need to add (such as logging support and perhaps a file i/o), however it is a great jumpstart on the project. You can use any JS engine to expose facilities like these with similar criteria and the outcome should be the same.

Further, you could also take a look at CommonJS. I talked with the guys a bit during some of the issues I was having with the GC of v8 and they all seem pretty smart. They're trying to get together a standard for an application-level API for javascript so more programming can be done in it (not just web-scripting). I may submit a few proposals there or do some of the coding to get things moving along.

The specifications I have are as follows:

{Global Scope}
{Network Object}
- ByteBuffer ctor function
- ClientSocket ctor function
- ServerSocket ctor function

{Shell Object}
- print function (prints every argument passed separated by a space to stdout)
- read function (reads a file into a string and returns that string)
- load function (reads a file in, compiles and executes it)
- quit function (quits the system)
- version function (returns the JS engine version string)


{ByteBuffer Class}
- readByte(n) function (reads n individual bytes from the buffer, returns as an array of Integers, if n = undefined, returns only a single Integer)
- readWord(n) function
- readDWord(n) function
- readQWord(n) function (reads n QWords from the buffer, returns an array of Objects, with 'lowpart' and 'highpart', meaning the low and high dwords as javascript cannot intrinsically handle 64bit integers, this can also be implemented via an unsafe reinterpret_cast to fill a number value with the 64bit, however this is difficult to convert once in javascript so I opted for an object return like this)
- readFloat(n) function
- readDouble(n) function
- writeByte(b1,b2,...) function (writes b1, b2, .... to the buffer as individual bytes, returns 'this' for call chaining)
- writeWord(...) function
- writeDWord(...) function
- writeQWord(...) function
- writeFloat(...) function
- writeDouble(...) function
- rewind function (rewinds the internal buffer pointer to 0)
- forward function (forwards the internal buffer pointer to the furthest written position in the buffer)
- seek(pos) function (moves the internal pointer to the indicated position so long as it falls between where rewind and forward would get you)
- tell function (returns the current position of the pointer)
- size function (returns the total size of the buffer being used)
- cursize function (returns buf.forward().tell(), without moving the ptr)
- reset function (resets the buffers internal and max pointer back to 0)
- expand(n) function (increases the max pointer by n)
- resize(n) function (resizes the buffer to n bytes)

I'll update this post with the ClientSocket and ServerSocket classes later.

There are some oddities between the two, and both support async operations, and as I came to find out the hard way, you should never call your 'stop' function in an async callback, it deadlocks the entire app. That was a fun one to debug, and it really made me feel like an idiot. I wrap calls to JS callbacks now with setting a bool in the object to true (in_async to be precise), then in 'stop' i check if that flag is set, if so I throw a javascript exception to prevent its usage.

If I ever had time to do something like that I'll do it for sure, thanks for that more than interesting post, once again. :)

I'm in support just because you mentioned Lisp.

I know this is grave digging , but i fell in love with the ideal. Would you mind explaining it a bit further, so that i can make a emulator base, based off of this.