Help decide the future of PK2s!

[pushedx]

I've been working with GFXFileManager lately. This DLL contains the logic for working with the PK2 archives. By writing your own GFXFileManager that maintains compatibility with the client, you have control over how files are processed for the client. This means you do not need the PK2 format anymore.

Right now, I am able to load files from the hard drive with no PK2s at all. Here are two videos showing this.
ISRO:

You must be registered to see links

VSRO:

You must be registered to see links

It should be noted, that currently I have only implemented the 'reading' aspect of the DLL. I'll do 'writing' later on, as it should not pose any problems, as my PK2 tools already use the writing functions from GFXFileManager. I'll be adding write support soon, and after a little more testing, I think the core API will be 'done', so I'm comfortable at this point starting a brainstorming thread for uses.

Anyways, I think this opens the door for a number of possibilities with customization, but I'd like to hear what the community thinks. For example, here's a list of some things that could be done:

• Shared client files - Consider this: users download the base client files for the Silkroad version they want to play on. Let's say, for example, VSRO 1.188. Now, rather than distribute these files again, private servers would only distribute a custom configuration file, along with all files they modify with a special prefix to them.
  
  To give an example, let's say I had a server and was giving out the changed files. I'd give out the configuration file and 'edx_divisioninfo.txt', 'edx_gateport.txt', 'edx_sv.t', and so on. The purpose of this prefix is to separate everyone's files from one another, so there are no conflicts. The purpose of the configuration file is to tell the GFXFileManager dll which set of files to load.
  
  Now let's say the player wanted to try another private server. Let's pretend this server used a prefix of rz. The player would download the rz configuration file and 'rz_divisioninfo.txt', 'rz_gateport.txt', 'rz_sv.t', and so on. Now the player can access another private server using the same shared files, but the GFXFileManager would load the 'rz' files instead of the 'edx'.
  
  
• Texture mod packs - I'm not sure if this is even a big deal anymore, but the system would be similar to the shared client files, except it is meant for textures and sounds and allows people to easily swap out "themes". It would come with a manager to allow people to see potential conflicts as well as other little features.
  
  
• A new PK2 format - I'm not a huge fan of this, because all you would be doing is delaying the inevitable. Eventually someone will reverse the format and make tools for it, just like every other file format there is out there. While it's understandable to want to try and prevent people from certain cheats, the amount of time and effort vs. reward is hard to justify a new PK2 format.
  
  However, what would be more realistic is to implement specific file based security for important files. For example, for any server_dep files, compress, encrypt, digitally sign, then add more verification logic to GFXFileManager. Since you would have source to GFXFileManager, you could better protect it with packers that make use of code macros.
  
  I think this is more realistic and practical compared to writing a new solid file archive to replace PK2s. Using this approach, it would fit in well with the two previous ideas while still giving people a little more flexibility with modifying "cosmetic" files.

I'm sure there's more creative things that could be done, so that is what this thread is for. I'd like to hear from the community about possible ideas or desires with making something that would benefit everyone and be worth the time doing.

 

[NoNStop]

I agree with the first idea,the shared client files,it would be awesome for players who are on slow connections,however,it will be bad for some private server owners since they don't wanna their work to go public.
Iam thinking of smth,what about creating a new format and apply the shared client idea with them,so every server has a single file contains the edits,only edits,so there would be smth like "Original.w/e" and "(ServerName).w/e" which contains the files which are edited by the owner of the server.
I think I can't explain it well due to my low English skills but I think you got it,didn't you?

 

[sladlejrhfpq]

its just awesome, even for testing, it would be much easier, exporting and importing to pk2 files takes too much time

 

[Jangan]

like the first idea!

 

[egsro]

first idea rocks
=============
offtopic i would like from you to show how to disable & enable the enc. files in the vsro's clients

 

[MeGaMaX]

awesome you rock i like the idea so much that's will be more easy for every one to protect his / her files , and its will be more easy to edit the client.

thanks pushedx every time you made something make us feel we are not alone and we have some perfect ppls

you really did much for this community thank you

greetings andrew

 

[Caipi]

The first one is really awesome, although your tools are working pretty well it still needs some time to extract and reimport stuff - updates could be done more easily with an extracted file folder for testing purpose.
And since I know how someone can suffer from a low speed internet I'm totally for the 1st option, thanks for all the work you've done so far!

 

[Statiic]

its my dream

 

[Naty48]

well , im updating myself with Sro forums sometimes and until now i found only but useless things *for me at lest
this one is pretty awesome .

but from some one like you i would expect of using better edit tools (for example Notepad++ and not some Windows Notepad..)
well , anyway since i have nothing to do with that anyway so i wish you good luck and this is really sounds awesome .

good job mate .

 

[ZiCO]

its my dream

It's my pushedx xD
w/e he does is awesome

 

[jMerliN]

Do what I suggested originally: just explode it into a folder structure. The compression isn't really necessary, or at most just toss a gzip on the files. If you were to read /blah/lulz.txt via the API, just load %SILKROAD%/{pk2name}/blah/lulz.txt.

The idea is quite simple. What I'd prefer more is a public effort to completely publish the network protocol, so a decent sandbox can be built.

I'd suggest a more abstract approach: make the modified API call into a plugin stack so people can extend it to do other neat stuff. Something I think would be interesting is:

Make reads issue a GET to an HTTP server with caching so people distributing their own client could have updates happen automatically (99.99% of traffic will be conditional gets followed by a 304 not modified). The idea being you simply set up your webserver to serve these files and instead of downloading some massive client, the person needs only the basic executable code and a launcher, instead of emulating the update server etc (it's also quite a bit more flexible than an update server as it's fairly efficient, and one can make it even more efficient with a bin diffing capability). This also means it'd be pretty easy to host on a LAN for friends to play. You can even namespace it and let people look at multiple servers, so

You must be registered to see links

goes to /data/epicsro.com/* (local cache). PUT can be made to replace the files and require an Auth header so a very simple updating tool can be made via a website (choose the files, click upload, voila) for remote servers.

Couple the above with a public effort to completely reverse the network protocol and sandboxes on the latest version can be widespread so that an architecture like that will be useful (since every server would run the same version).

 

[pushedx]

Thanks everyone for the replies. I'll be starting a release thread soon in the appropriate section. This thread will just be for general discussion of the project, so keep any ideas coming!

Here is my final WIP video if anyone is interested:

You must be registered to see links

(I accidently recorded with music playing, so if you don't like it mute )

I think I can't explain it well due to my low English skills but I think you got it,didn't you?

I understand what you are saying.

The main point of the format is to share the "common" files between all private servers based on the same file set. If any server had a custom version for a file, then that file wouldn't be part of the "shared" setup.

For example, you download and install the base VSRO 1.188 client files. For any file that a private server does not modify, they simply don't redistribute that file. This means at least 90% of the files used in Silkroad won't actually be redistributed again and again, since they are always the same and never modified.

This isn't easily doable with archive setups since you'd have to modify the archive each time. In terms of the files that are distributed, it just depends on the system used. A "difference" file could be used to reduce the size of the downloads. The client would then duplicate the original file and apply the changes to the new file, leaving the old intact. Alternatively, they could just distribute the entire file. I've not really worked on a final system for that yet, so there's options if enough people were interested.

When it comes to the actual unique private server files though, they really would need to accept the fact that people will find a way to get access to the file no matter what, so they just distribute an unprotected file. A lot of people try to fight this, but they are wasting their time (in my opinion).

Consider all the private servers that have "custom files" and look at how many have had their blowfish key found, any modifications in their file format released, and so on. Are there even any long running servers that this hasn't happened on yet? What does it even gain them? In theory, exclusive content is needed to make a server stand out, but that only works when the files aren't public and you have very little competition. At this stage, exclusive content isn't going to be enough with how many servers there are.

The PK2 format has long been compromised, as pretty much any game file format has. If anyone really has something they want to keep private and protected, the only solution is to not distribute it in the first place!

If someone really, really wants to try and keep their files protected, then the only way is to use this system and make a private implementation of it that implements compression, encryption, digital file signing, as well as run-time verification. That is a lot of work, and really hard to get right. All that for what? DDJ files people can steal through Direct3D detours? Txt files people can hook the client to rip? As soon as someone does reverse it and make tools, then you are back at square 1, which is why I'm not a huge fan of it in practice in this context.

Client side protection just isn't viable with Silkroad unless you had source code to the client. Without it, you need custom file handling, an obfuscated client with a lot of VM'ed code, and a run-time protection that doesn't actually suck (not cheap). All this is possible with enough time and resources, but it would take forever and not really make the server more desirable to play on than others.

I think ultimately, that is what matters the most. Content is still important, but people care about more than just that. The rates of a server, the management, the community, events, the "donations" or perception if it's "pay 2 win". These are all the main things that make and break a server. Focusing on content in this setup is just impractical because if you can do it, someone else can too.

Do what I suggested originally: just explode it into a folder structure. The compression isn't really necessary, or at most just toss a gzip on the files. If you were to read /blah/lulz.txt via the API, just load %SILKROAD%/{pk2name}/blah/lulz.txt.

Yea, that's how it works right now (I'm just using Win32 file i/o). I feel that compression is good for exchange (keep bandwidth usage low), but not really needed for on disk files with hard drives as large as they are nowadays. What I had in mind was lzma though, as that's what ISRO updated to a few years back because it offered better compression over zlib, at the expense of slightly more processing power on decompression.

I'd suggest a more abstract approach: make the modified API call into a plugin stack so people can extend it to do other neat stuff.

Definitely, a plugin system is what I'd like to aim for to accomplish the custom security and features needed for each server. However, I'm starting out with the base project without any extras, so people are free to experiment as the want. It's quite a big change, and still experimental, so I don't see a lot of people rushing to try it out yet.

In that regards, by releasing everything as I'm about to, I'm conceding the fact most people would rather have private implementations for their servers rather than use a common shared system. Until people realize how DRM does not work, and if the client has your files, you cannot protect them, then I don't think that will ever change. So, I won't stand in their way, which is better for everyone since I'm not particularly committed to Silkroad. This project was just random because I needed something to do because of how bad Diablo 3 went.

Something I think would be interesting is:

I still like the idea of changing up how updates are done. Silkroad.exe is (still) horrible and the download server isn't too great overall. With the free standing file system, people should hopefully be able to start looking into better alternatives for handling updates and content delivery. However, it's understandable why people just use what works as it is.

I always liked the web based approach. What you can actually accomplish now with this project is stream files to the client as they are needed. Similarly to how a lot of games are streaming content to reduce download sizes, the same system could be implemented with Silkroad. This way, if you want to try out a new server, you don't need to download 2GB worth of files. You only might need a small portion to get into game and test out the server. The most obvious issue with this though is that it'd be hard to do 'smoothly', but I think it could possibly work out.

As for the network protocols, I'd not hold my breath on that. After you do so much dev with a game like Silkroad, and understand how it works and is setup, it'd probably just be better to invest the time and resources into your own game instead. That way, you don't have to worry about any legal issues and you have full control over everything.

At this point, we all aren't getting any younger. Considering how much there is to Silkroad packet wise, I know for sure I'm never going to sit down and try to reverse the entire protocol. That is, unless it was a well paying hourly job. Other than that, it's just not worth it to me at this stage, which is why I've not attempted to setup anything for it.

but from some one like you i would expect of using better edit tools (for example Notepad++ and not some Windows Notepad..)

For regular development, I use the right click context menu to choose Notepad++, Hex Workshop, Excel, or whatever else I need. Out of security habits, I typically never left click to open files, as the extra step gives a brief second to think about the action before doing it. When I'm making a video, I try to keep my typical developer habits out, so that's why a lot of things might seem 'awkward'. Opening in notepad, only having windows on screen I use (I have dual monitors), going to File->Save, hiding desktop icons, are all intentional simplifications in the video that carry over from making tutorials. Rest assured though, I do make use of the good stuff normally though.

 

[asd234]

Thanks everyone for the replies. I'll be starting a release thread soon in the appropriate section. This thread will just be for general discussion of the project, so keep any ideas coming!

Here is my final WIP video if anyone is interested:

You must be registered to see links

(I accidently recorded with music playing, so if you don't like it mute )



I understand what you are saying.

The main point of the format is to share the "common" files between all private servers based on the same file set. If any server had a custom version for a file, then that file wouldn't be part of the "shared" setup.

For example, you download and install the base VSRO 1.188 client files. For any file that a private server does not modify, they simply don't redistribute that file. This means at least 90% of the files used in Silkroad won't actually be redistributed again and again, since they are always the same and never modified.

This isn't easily doable with archive setups since you'd have to modify the archive each time. In terms of the files that are distributed, it just depends on the system used. A "difference" file could be used to reduce the size of the downloads. The client would then duplicate the original file and apply the changes to the new file, leaving the old intact. Alternatively, they could just distribute the entire file. I've not really worked on a final system for that yet, so there's options if enough people were interested.

When it comes to the actual unique private server files though, they really would need to accept the fact that people will find a way to get access to the file no matter what, so they just distribute an unprotected file. A lot of people try to fight this, but they are wasting their time (in my opinion).

Consider all the private servers that have "custom files" and look at how many have had their blowfish key found, any modifications in their file format released, and so on. Are there even any long running servers that this hasn't happened on yet? What does it even gain them? In theory, exclusive content is needed to make a server stand out, but that only works when the files aren't public and you have very little competition. At this stage, exclusive content isn't going to be enough with how many servers there are.

The PK2 format has long been compromised, as pretty much any game file format has. If anyone really has something they want to keep private and protected, the only solution is to not distribute it in the first place!

If someone really, really wants to try and keep their files protected, then the only way is to use this system and make a private implementation of it that implements compression, encryption, digital file signing, as well as run-time verification. That is a lot of work, and really hard to get right. All that for what? DDJ files people can steal through Direct3D detours? Txt files people can hook the client to rip? As soon as someone does reverse it and make tools, then you are back at square 1, which is why I'm not a huge fan of it in practice in this context.

Client side protection just isn't viable with Silkroad unless you had source code to the client. Without it, you need custom file handling, an obfuscated client with a lot of VM'ed code, and a run-time protection that doesn't actually suck (not cheap). All this is possible with enough time and resources, but it would take forever and not really make the server more desirable to play on than others.

I think ultimately, that is what matters the most. Content is still important, but people care about more than just that. The rates of a server, the management, the community, events, the "donations" or perception if it's "pay 2 win". These are all the main things that make and break a server. Focusing on content in this setup is just impractical because if you can do it, someone else can too.



Yea, that's how it works right now (I'm just using Win32 file i/o). I feel that compression is good for exchange (keep bandwidth usage low), but not really needed for on disk files with hard drives as large as they are nowadays. What I had in mind was lzma though, as that's what ISRO updated to a few years back because it offered better compression over zlib, at the expense of slightly more processing power on decompression.



Definitely, a plugin system is what I'd like to aim for to accomplish the custom security and features needed for each server. However, I'm starting out with the base project without any extras, so people are free to experiment as the want. It's quite a big change, and still experimental, so I don't see a lot of people rushing to try it out yet.

In that regards, by releasing everything as I'm about to, I'm conceding the fact most people would rather have private implementations for their servers rather than use a common shared system. Until people realize how DRM does not work, and if the client has your files, you cannot protect them, then I don't think that will ever change. So, I won't stand in their way, which is better for everyone since I'm not particularly committed to Silkroad. This project was just random because I needed something to do because of how bad Diablo 3 went.



I still like the idea of changing up how updates are done. Silkroad.exe is (still) horrible and the download server isn't too great overall. With the free standing file system, people should hopefully be able to start looking into better alternatives for handling updates and content delivery. However, it's understandable why people just use what works as it is.

I always liked the web based approach. What you can actually accomplish now with this project is stream files to the client as they are needed. Similarly to how a lot of games are streaming content to reduce download sizes, the same system could be implemented with Silkroad. This way, if you want to try out a new server, you don't need to download 2GB worth of files. You only might need a small portion to get into game and test out the server. The most obvious issue with this though is that it'd be hard to do 'smoothly', but I think it could possibly work out.

As for the network protocols, I'd not hold my breath on that. After you do so much dev with a game like Silkroad, and understand how it works and is setup, it'd probably just be better to invest the time and resources into your own game instead. That way, you don't have to worry about any legal issues and you have full control over everything.

At this point, we all aren't getting any younger. Considering how much there is to Silkroad packet wise, I know for sure I'm never going to sit down and try to reverse the entire protocol. That is, unless it was a well paying hourly job. Other than that, it's just not worth it to me at this stage, which is why I've not attempted to setup anything for it.



For regular development, I use the right click context menu to choose Notepad++, Hex Workshop, Excel, or whatever else I need. Out of security habits, I typically never left click to open files, as the extra step gives a brief second to think about the action before doing it. When I'm making a video, I try to keep my typical developer habits out, so that's why a lot of things might seem 'awkward'. Opening in notepad, only having windows on screen I use (I have dual monitors), going to File->Save, hiding desktop icons, are all intentional simplifications in the video that carry over from making tutorials. Rest assured though, I do make use of the good stuff normally though.

holy cow dude! u write alot!

i like the idea.. cuz when i started doing server stuff i had 2 do updates with SMC cuz i was ruining pk2's by not putting stuff in properly.. its good help for newbi's but not every one will agree with it

Do what I suggested originally: just explode it into a folder structure. The compression isn't really necessary, or at most just toss a gzip on the files. If you were to read /blah/lulz.txt via the API, just load %SILKROAD%/{pk2name}/blah/lulz.txt.

The idea is quite simple. What I'd prefer more is a public effort to completely publish the network protocol, so a decent sandbox can be built.

I'd suggest a more abstract approach: make the modified API call into a plugin stack so people can extend it to do other neat stuff. Something I think would be interesting is:

Make reads issue a GET to an HTTP server with caching so people distributing their own client could have updates happen automatically (99.99% of traffic will be conditional gets followed by a 302 not modified). The idea being you simply set up your webserver to serve these files and instead of downloading some massive client, the person needs only the basic executable code and a launcher, instead of emulating the update server etc (it's also quite a bit more flexible than an update server as it's fairly efficient, and one can make it even more efficient with a bin diffing capability). This also means it'd be pretty easy to host on a LAN for friends to play. You can even namespace it and let people look at multiple servers, so

You must be registered to see links

goes to /data/epicsro.com/* (local cache). PUT can be made to replace the files and require an Auth header so a very simple updating tool can be made via a website (choose the files, click upload, voila) for remote servers.

Couple the above with a public effort to completely reverse the network protocol and sandboxes on the latest version can be widespread so that an architecture like that will be useful (since every server would run the same version).

i dont get it.. are u trying say that its possible to make online client?

 

[jMerliN]

i dont get it.. are u trying say that its possible to make online client?

No. I'm saying downloading files and performing basic modifications to them is what HTTP was invented to do. Think about it, when you go to

You must be registered to see links

, you perform a GET against /, the webserver figures out an index page and runs its plugins and finally sends the output to you (be it streaming a file from the filesystem or the output of some CGI/FastCGI module, like PHP). The HTML that sends to you has references to javascript files, images, etc. That stuff doesn't exist on your computer, so your browser automatically starts fetching it for you. Once it downloads those files, if the server gives cache-control headers, the browser will store the resources in a cache. Depending on cache control options, it will always look at cache until it expires or it will issue a conditional GET to the server on each future request. That looks like a GET with a "If-*" header that asks the server "is this file modified on the server?" If the server replies with a 304 not modified, the client uses its cached version. The 302 message can be thousands of times smaller than re-downloading the file each time, and if an unconditional cache is set, a conditional get won't even be issued so bandwidth is 100% conserved.

When you run SRO, you aren't accessing all the files in the game. You're reading a few basic files that are needed at all times, and whatever textures/map files that are needed to render the environment around you. The idea is that the fact that you can progressively download content as you need it is a "streaming client" (like what WoW and modern MMO games can do, by delaying downloads until you need the content). So when you launch the game, the client can issue a request against

You must be registered to see links

which would return a list of needed files and the client would go ahead and cache those, and future load requests from the PK2 API would simply issue a request against that webserver to get the resources it needs.

HTTP has semantics for updating (which means you can create a webpage where server operators can update files remotely, if you don't feel like doing it over SFTP, or directly on the server). It also has semantics for sending compressed content.

As pushedx brought up, it can be a bit difficult. You need a lot of bandwidth laying around to handle 10 people trying to stream a 50MB file at the same time quickly enough to get it before the client crashes. Also, if the GET request takes a significant time and the client blocks on that load request, you'll see a pretty big delay before the file finally gets to you, but in the future it will be instantaneous. It works best if your files are segmented into "area" and it's predictable which files you need to fetch at any given time, which isn't necessarily how this PK2 interface would work. We could potentially augment this by checking the player's location and having some REST based service that would give a list of files needed for that area, but it's a pretty difficult problem to get it to work well and be smooth so it doesn't bother the users. The good thing, though, is that updates are immediate and they require no re-loading of the client, and no awkward and oversized patch phase. That might be worth it, in the end, and being able to jump in-game with like 200MB of files instead of 2GB of files is pretty cool.

 

[egsro]

i wont read all of these !

 

[asd234]

i still didint get it.. but thanks for going into details..

 

[LegendaryDev]

Amazing project this makes the life easier in editing/updating

 

[asd234]

Amazing project this makes the life easier in editing/updating

yea it does! but u still need 2 know where things are.. cuz if u put that come's from media into data u might f up ur client.. xD

but yea.. jMerlin and Pushedx and Jangan should work together and make awsome client that will be shared for public.. even though alot of people dont deserve it but its still smth that shudnt be kept private..

also.. quick question..

lets say u will make jMerlin's idea work.. dont you need 2 open extra ports? and woudnt it make server easier target for hackers?

 

[Mykha]

Great news !. It will be much easier also faster to edit now. Thank you

 

[pushedx]

lets say u will make jMerlin's idea work.. dont you need 2 open extra ports? and woudnt it make server easier target for hackers?

Nope, the files are coming from a web server, which would be part of the "cloud". You wouldn't be opening any ports on your servers and the servers that host these files (sh)would be DDoS protected. You could just use a dedicated web server, but you'd need to have a lot of bandwidth, at least a 1gbit connection, and DDoS protection, as if the server gets taken down, then everyone's game play is ruined since they can't download files. That's why the cloud approach might be best, but it wouldn't be cheap. It would be an interesting idea though, I might make a little example of it sometime later.