[Dev] Patcher

[Yekun]

Hello Ragezoners!

I've been seen that some MapleStory servers doesn't have a Patcher to do the updates and they send the whole WZ file to their players download.

Why whole file update isn't a good option for MapleStory

Spoiler

Let's think a bit:

If you wanna change/add a string on String.wz? Just edit with a WZ editor (HaRepacker for example) and send to your players. It's 5 MB or something about that. Not too much, anyone can download without problems.

But if you wanna add a mob or a map? Or a mob AND a map? This becomes a huge update. In v83 for example, the Map.wz is about 620MB and the Mob.wz is about 470MB. Download all of this for just a few monsters/maps?

That's why I'm making a good Patcher for any server that works fine and is easy to use/configure. And will be open source.

Why do we don't have many patchers?

Spoiler

Suppose that you want add a lot of Mobs and Maps ou your server.

You can get a existing patcher, generate the update file and put on your host (or wherever it need to be), and your players will receive the update as well, fine, no problems.

After 1 month, you want add more Mobs and Maps. Then, you do the same thing, generate the update file and the players will receive, no problems.

After this update, new players wants to play on your server but something happens!:

They get the Mobs/Maps only from the current update, they don't get the mobs/maps from the others updates because the patcher only does one update file. That's a big problem!

But you have a patcher that store the old updates and concats with the new update, so, all the updates (old and new) are 1 file too, but the new players can receive the updates. Amazing!

But...if you create many updates, with many monsters, maps, items, whatever, this files becomes huge, now, it's a 40MB update file. The players must download 40MB every time they wanna play just for updates check

But it's not problem anymore with this Patcher!

Guys....I SHOW YOU....

THE........ *looking for a cool name*

YEKUN PATCHER!

*WOOOOOOOOOOOOO* *YEEEEEEEEAAAAH* *WOHOOOOOOO* *AMAZING NAME*
*SO CREATIVE* *BEST NAME OF MY LIFE* *POWERFUL CREATIVITY* *FANTASTIC* *NICE NAME, UP* *53X*

Just kidding, It's only MapleStory Patcher.

Features

Spoiler

Well, the ideia for this patcher is to create updates without be large to read and patch, that's why it has:

Intelligent update finder
The patcher doesn't need to analyze the whole update file every time, only when a update is needed.

Intelligent patching system
The patch system is smart enought to know that, if a parent property doesn't exists it must and will be created with the right property type.

Organized update files
The update files are good organized supporting comments and ignoring extra spaces.

Lexer-based
The analyze system have a custom lexer that will identify the updates and possible errors.

Structures

Spoiler

Let's see the structure of update files

update.yekun
This is the main update file. This file holds the information about all updates

@'String.wz' = '408A9B95E3EC2CEC846619AC9':'FirstStringUpdate.yekun'
@'String.wz' = '54605FEDC560DBE63369013FB':'AnotherStringUpdate.yekun'

@'Mob.wz' = '98GHE89EG8935634GL0AEY5LK':'Mob_update0.yekun'

Confusing? Let me explain

@ - token for identifying a update for a file
String.wz - the file name
408A9B95E3EC2CEC846619AC9 - the file hash
FirstStringUpdate.yekun - the file that have information about the update

So, let understand

When the player opens the patcher, it'll check if there's a update available comparing the hash (don't worry about it, I have a trick). If the hash isn't the same, then, it need a update. Then the patcher will read the information about the update (FirstStringUpdate.yekun, for example). Let's see:

FirstStringUpdate.yekun
This is the update file. It holds all the information of update

#'Mob.img'
{
    +"!100100/$name" = "Amazing Snail"; //change the snail name
}

# - identifier for a update section
Mob.img - the section to be updated
+ - means "add if not exists/change the value"
!,$ = identifiers for property types (! = sub property, $ = string property)

So, the first update is just for changing a mob name.You can see that the file is well organized and supports comments. So you can commend you update file for informations. (why? I don't know, but it's amazing).

If you want to remove a wz property? Like, you want remove the snail mob, for example.Just change the '+' token to '-' (minus)

#'Mob.img'
{
    -"100100/name"; //beginners gonna hate
    //don't need to specity the type when deleting
}

Don't worry about tokens yet, when the patcher got finished, I'll support and a file with all informations about structures, tokens, tutorials very explained will be available for download.

But you can realize that is easy to add/change/remove properties.

About images, you'll be able to manipulate image properties (like item images) converting the image to base64 string. About the songs, I'm working on a solution for it.

Screenshots

Spoiler

Okay, time to see some images. Don't worry about the design, I'm not designer, just googled the maplestory bg and logo and organized. (but I did the button/progressbar/dialog)

First concept:

Spoiler

*all the possible errors will be included on a separated file too to help you*

here's a gif showing how the dialog appears to the user:

I have a old version of this patcher working but with a bit different structure and a bad code (just did for let the concept working)

Spoiler

I'll update this thread when some new will be added.

Doubts? Questions? Suggestions? Leave a comment

---

Thanks for:

Snow - MapleLib
haha01haha - HaRepacker for testing & MapleLib understanding

This patcher is being developed in C#

 

[ngnam87]

i'm very interested in this. I want my server get a lots patch of wz file(like MapleCrystal did), but i cannot discovery the big code in HarePacker source
hope this awsome soon come to the community

 

[greenelfx]

Wow, great job on this!

 

[Utsukushi]

This is awesome

 

[sunnyboy]

Looks pretty cool
DeadCode some1 beat u to it

 

[maplefreak26]

Nice! I've been wanting to make something like this but never learned how to. Other patchers that did the same were horribly outdated.

 

[Shmoconut]

Hi dude,

I actually wrote a patcher a couple years ago, but never ended up using it.
There may be some things you have overlooked, or some things about your design that could be improved (seeing hash comparison of the files kind of set off red flags) so I'll elaborate on how my old system worked.
Like yours, mine also used MapleLib, but I built mine in to the redirector. Essentially, the redirector is built with a specific version, e.g. 1. When the redirector is launched, it contacts the website's wzversion.html file (or something like that), which shows the current patch version. If the versions are different, it downloads patch zip archives for all versions necessary.

For example, if the redirector has a version of 1 and the current patch version is 4, it will download wzpatch2.zip, wzpatch3.zip, and wzpatch4.zip. It will unzip each (in memory) and apply patches to the player's wz files.

The patch zip files have an info.patch file, which contains the names of the changed wz files, along with the names of the image files changed.
Unfortunately, I don't have real examples of what it would look like, but it's something like this:

Mob.wz
100100.img 1232423.img a few other imgs here
Map.wz
910000000.img a few other imgs here
another wz file name here
some more img data here

Then the rest of the files in the archive would be the actual updated img files, like 100100.img and 910000000.img. The redirector would parse the info.patch file of each downloaded patch zip archive and iteratively apply each patch using MapleLib.

Finally, the redirector would update its version and save it in a dedicated folder in the user's app home directory (along with other necessary, unrelated data). After the first launch, the redirector loads its version before comparing with the head patch version.

There are some pretty nice advantages from doing it this way. First, the version check is in constant time because it's a very simple Rest-esque call. Next, the download size is kept at a minimum, because the only downloaded data are the changed image files. To get smaller than this, you would have to find a way to dump only the relevant data inside of the image (which would require modifying maplelib directly). It's compressed in zip so it's made a bit smaller, and you don't have to do any special handling for pictures (.png, etc) because they are all ultimately built into the image file.

I don't know how easy you plan to make the process of actually creating a patch, but it seems like you are requiring the user to manually enter all patch information, as demonstrated in your FirstStringUpdate.yekun file. This would probably get quite painful for larger patches, and with the system I described, it would be trivial to make a program that automatically packages the zip archive with basic user supplied information (the changed wz files and their corresponding image files). I planned to do that eventually, but never got around to it because I never really used the patching system.

Oh well, I've touched on pretty much everything important, so maybe you can refine your ideas and design from this, or improve upon what I've described!

 

[Bertie Bott]

Looks awesome. My immediate question is, will this be cross OS compatible? On MACs, the same file/folder structure exists within .app files. One of the reasons I have always avoided using patchers with my server was because only the Windows users could take advantage of it. Would be great if MAC users can take part in the fun too.

 

[sunnyboy]

Looks awesome. My immediate question is, will this be cross OS compatible? On MACs, the same file/folder structure exists within .app files. One of the reasons I have always avoided using patchers with my server was because only the Windows users could take advantage of it. Would be great if MAC users can take part in the fun too.

Would be better if a mac was for games, but it isn't made for it so if anything no issue if it isn't

 

[Bertie Bott]

No actually it is an issue and something I would like to request if there currently is not support for it. Thanks Yekun.

 

[Hilia]

I think it will not be that hard to make it compatibel with mono (assuming this is in C#, since HarePacker and MapleLib is mentioned)

 

[Yekun]

Hi dude,

I actually wrote a patcher a couple years ago, but never ended up using it.
There may be some things you have overlooked, or some things about your design that could be improved (seeing hash comparison of the files kind of set off red flags) so I'll elaborate on how my old system worked.
Like yours, mine also used MapleLib, but I built mine in to the redirector. Essentially, the redirector is built with a specific version, e.g. 1. When the redirector is launched, it contacts the website's wzversion.html file (or something like that), which shows the current patch version. If the versions are different, it downloads patch zip archives for all versions necessary.

For example, if the redirector has a version of 1 and the current patch version is 4, it will download wzpatch2.zip, wzpatch3.zip, and wzpatch4.zip. It will unzip each (in memory) and apply patches to the player's wz files.

The patch zip files have an info.patch file, which contains the names of the changed wz files, along with the names of the image files changed.
Unfortunately, I don't have real examples of what it would look like, but it's something like this:

Mob.wz
100100.img 1232423.img a few other imgs here
Map.wz
910000000.img a few other imgs here
another wz file name here
some more img data here

Then the rest of the files in the archive would be the actual updated img files, like 100100.img and 910000000.img. The redirector would parse the info.patch file of each downloaded patch zip archive and iteratively apply each patch using MapleLib.

Finally, the redirector would update its version and save it in a dedicated folder in the user's app home directory (along with other necessary, unrelated data). After the first launch, the redirector loads its version before comparing with the head patch version.

There are some pretty nice advantages from doing it this way. First, the version check is in constant time because it's a very simple Rest-esque call. Next, the download size is kept at a minimum, because the only downloaded data are the changed image files. To get smaller than this, you would have to find a way to dump only the relevant data inside of the image (which would require modifying maplelib directly). It's compressed in zip so it's made a bit smaller, and you don't have to do any special handling for pictures (.png, etc) because they are all ultimately built into the image file.

I don't know how easy you plan to make the process of actually creating a patch, but it seems like you are requiring the user to manually enter all patch information, as demonstrated in your FirstStringUpdate.yekun file. This would probably get quite painful for larger patches, and with the system I described, it would be trivial to make a program that automatically packages the zip archive with basic user supplied information (the changed wz files and their corresponding image files). I planned to do that eventually, but never got around to it because I never really used the patching system.

Oh well, I've touched on pretty much everything important, so maybe you can refine your ideas and design from this, or improve upon what I've described!

I really liked your idea, It solves a little issue that was in my mind and is more fast to download and check. I think I'll implement it (and of course, you'll receive the credits too), thanks for this amazing suggestion

@Bertie Bott
As @Hilia said, we can translate it to Mono to work on MACs.

I forgot something: This patcher is currently being developed in C#.

 

[Bertie Bott]

@Yekun, sounds good. If you need a MAC user to test anything for you, please don't hesitate to contact me. Would be more than happy to help.

 

[Snopboy]

Great, good luck with your development!
If you want a better, more appealing design, PM me and I'll design the patcher

 

[Snopboy]

I'm working on the design, I'd appreciate it if you'd check it out, comment there any suggestions or criticism.
Thank you!
http://forum.ragezone.com/f119/maplestory-server-launcher-patcher-updater-1045284/

 

[Rakeda]

jesus christ, im in love with this idea, I tried making one a while back but gave up at checksums, any eta when you will release the source/compiled project?

 

[Darter]

I think it will not be that hard to make it compatibel with mono (assuming this is in C#, since HarePacker and MapleLib is mentioned)

lol why on earth would you want to have a cross platform patcher thats designed specifically for a game that runs on windows......?

 

[Fraysa]

No actually it is an issue and something I would like to request if there currently is not support for it. Thanks Yekun.

Why would you need a patcher for OS X if MapleStory isn't compatible with OS X?

 

[Zygon]

Try using a binary diff like

You must be registered to see links

and create a wrapper for it. Trying to make a patcher program with MapleLib/WzLib seems silly to me when a general purpose binary diff program will be faster to both generate a patch and apply it. I made a simple C# wrapper program using xdelta back in like v10x-v11x days and it worked pretty well.

If you're really insistent about continuing, then I wrote a comparison algorithm a while back that I used to see what changed in the wz files and create a psuedo-patch:

(Note: Was a heavily WIP piece of software in like 2012 -> I haven't worked on it since)

Main.java

package net;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileReader;
import java.util.LinkedList;
import java.util.Properties;
import wz.WzDirectory;
import wz.WzFile;
import wz.WzImage;
import wz.WzVersion;
import wz.io.WzInputStream;
import wz.tool.WzKeyGenerator;
import wz.tool.WzTool;

/**
 *
 * @author Brent
 */
public final class Main {

    private static String OLD_FILE;
    private static String NEW_FILE;
    private static short FILE_VERSION;
    private static String OUTPUT;
    private static FileInputStream ofin, nfin;

    public static void main(String[] args) {
        Properties p = new Properties();
        try {
            FileReader fr = new FileReader("config.ini");
            p.load(fr);
            fr.close();
        } catch (Exception e) {
            System.out.println("[ERROR] Unable to load configurations, check that they exist.");
            e.printStackTrace();
            return;
        }
        OLD_FILE = p.getProperty("OLD_FILE", "");
        NEW_FILE = p.getProperty("NEW_FILE", "");
        FILE_VERSION = Short.parseShort(p.getProperty("FILE_VERSION", "-1"));
        OUTPUT = p.getProperty("OUTPUT", "");
        if (OLD_FILE.isEmpty() || NEW_FILE.isEmpty() || OUTPUT.isEmpty() || FILE_VERSION == -1) {
            System.out.println("[ERROR] Missing configurations for patch creator.");
            return;
        }
        File oldF = new File(OLD_FILE);
        File newF = new File(NEW_FILE);
        File out = new File(OUTPUT);
        if (!oldF.exists() || !newF.exists()) {
            System.out.println("[ERROR] Missing files needed to compare.");
            return;
        }
        if (out.exists()) {
            out.delete();
        }
        try {
            out.createNewFile();
        } catch (Exception e) {
            System.out.println("[ERROR] Unable to create output.");
            e.printStackTrace();
            return;
        }
        // preparing is over
        
        // begin compare
        WzFile old = new WzFile(OLD_FILE, FILE_VERSION);
        WzFile nWz = new WzFile(NEW_FILE, FILE_VERSION);
        try {
            ofin = new FileInputStream(oldF);
            nfin = new FileInputStream(newF);
            byte[] key = WzKeyGenerator.generateKey(
                    WzTool.getIvByVersion(
                    WzVersion.GMS));
            // XXX add version changeable in config later
            WzInputStream oin = new WzInputStream(ofin, key);
            WzInputStream nin = new WzInputStream(nfin, key);
            old.parse(oin);
            nWz.parse(nin);
        } catch (Exception e) {
            System.out.println("[ERROR] Unable to load WZ Files.");
            e.printStackTrace();
            return;
        }
        LinkedList<WzChange> c = compareFiles(old, nWz);
        // done comparing, create patch file
        
        Writer w;
        try {
            
        } catch (Exception e) {
            System.out.println("[ERROR] Unable to create patch.");
            e.printStackTrace();
            return;
        }
    }
    
    private static byte[] getContentHeader() {
        int size = 0;
        byte[] data = new byte[0xFFFF];
        
        return data;
    }
    
    private static byte[] getRawData(long offset, int size) throws Exception {
        nfin.getChannel().position(offset);
        byte[] ret = new byte[size];
        nfin.read(ret);
        return ret;
    }
    
    private static LinkedList<WzChange> compareFiles(WzFile o, WzFile n) {
        LinkedList<WzChange> c = new LinkedList<WzChange>();
        WzDirectory or = o.getRootDirectory();
        WzDirectory nr = n.getRootDirectory();
        compareDirectoryContents(or, nr, c);
        return c;
    }
    
    private static void compareDirectoryContents(WzDirectory i, WzDirectory cmp, LinkedList<WzChange> c) {
        for (WzDirectory o : i.getDirectories()) {
            boolean found = false;
            for (WzDirectory n : cmp.getDirectories()) {
                if (o.getName().equals(n.getName())) {
                    compareImageContents(o, n, c);                   
                    found = true;
                    break;
                }
            }
            if (!found) {
                System.out.printf("[INFO] Unable to find %s in new files. Processed as deletion.%n", o.getName());
                WzChange ch = new WzChange();
                ch.type = 1;
                ch.path = o.getParent().getFullPath();
                ch.name = o.getName();
                c.add(ch);
            }
        }
        for (WzDirectory o : cmp.getDirectories()) {
            boolean found = false;
            for (WzDirectory n : i.getDirectories()) {            
                found = true;
            }
            if (!found) {
                System.out.printf("[INFO] Unable to find %s in old files. Processed as new directory.%n", o.getName());
                // XXX handle new directory entry, this could be messy :S
            }
        }
    }
    
    private static void compareImageContents(WzDirectory i, WzDirectory cmp, LinkedList<WzChange> c) {
        for (WzImage img : i.getImages()) {
            boolean found = false;
            for (WzImage nimg : cmp.getImages()) {
                if (img.getBlockSize() != nimg.getBlockSize()) {
                    System.out.printf("[INFO] Found a change in block size with old %s%n", img.getName());
                    WzChange ch = new WzChange();
                    ch.type = 0;
                    ch.path = img.getParent().getFullPath();
                    ch.name = img.getName();
                    ch.checkSum = nimg.getChecksum();
                    ch.blockSize = nimg.getBlockSize();
                    try {
                        ch.data = getRawData(nimg.getOffset(), nimg.getBlockSize());
                    } catch (Exception e) {
                        System.out.println("[ERROR] Error occured while loading new data.");
                        e.printStackTrace();
                        continue;
                    }
                    c.add(ch);
                }
            }           
            if (!found) {
                System.out.printf("[INFO] Unable to find %s in new files. Processed as deletion.%n", img.getName());
                WzChange ch = new WzChange();
                ch.type = 1;
                ch.path = img.getParent().getFullPath();
                ch.name = img.getName();
                c.add(ch);
            }
        }
    }
}

WzChange.java:

package net;

public final class WzChange {

    public int type;
    public String path;
    public String name;
    public byte[] data;
    public int checkSum;
    public int blockSize;

    public void serialize(Writer out) {
        out.write(type);
        if (type == 0) {
            out.writeMapleString(path);
            out.writeInteger(checkSum);
            out.writeInteger(blockSize);
            out.writeInteger(data.length);
            out.write(data);
        } else {
            out.writeMapleString(path);
            out.writeMapleString(name);
        }
    }
}

 

[Hilia]

lol why on earth would you want to have a cross platform patcher thats designed specifically for a game that runs on windows......?

Because there is a v62 client with mac support xD