SMD file format

[Beyond2000!]

What is the file format for the PT smd file ?

I mean, what are it

 

[bobsobol]

The format of PT SMD files is undocumented binary data. I have already hunted high and low for information about this, and the best advise I have seen is to disassemble PTs executables and reverse engineer the way in which it interprets it's SMD files. This challenge seems to have either surpassed everyone else who has attempted it, or they have not been willing to share the fruits of their labour.

I might suggest that the Beta release of PT2 seems to use the same (or very similar) SMD format for some of their models.

Between the executables for the PT client, the PT server and the PT 2 client, you have 3 sources of executable code to examine. As far as I'm aware, there is no published file structure, or source code capable of making sense of these files.

It is clear that the creators of PT and PT 2 have created their binary SMD files from ASE files however. So if you can think how you would represent an ASE, and an ASCII SMD file in a binary manner, that appears to be what they have done.

It is also worth noting that they have obscured BMP and TGA files by simply corrupting, or XORing their headers. So there may be some level of obfuscation involved as well.

As I don't want to scare you off the project before you even start, I must say that I, and many others here will be wishing you the very best of luck, and I encourage anyone who has any information on this perplexing file to come forward and offer up what they know.

 

[Beyond2000!]

Many tks

i

 

[Beyond2000!]

Ok, things are going fine, so far

Found the maximum amount of Materials used. PT Smd files allows a maximum of 4000 materials. If it have more then this, it will return the message "MATERIAL Counter Error (ReadAse)"

I was able to identify inside the structure where it stores the LastFrame and the materials count.

The main problem is that PT handles a enormous amount of memory to be allocated. No wonder this game is excessivelly memory eating.

Juts to you guys have an idea of the tremendous amount of memory this poop uses. In one of the routines it needs (2130000*4) bytes to handle 01 single material inside the smd

And this without using virtualloc api. It loads directly from the game. Once i suceeded to decode this bastard, i

 

[bobsobol]

Afraid I can't help with the token, but you may have noticed their is a TGA de-obfuscator around here somewhere. (check RFT)

I've never found source for it, and attempts to understand it's workings have failed me. It seems to do more than the BMP patcher by Oscuro (IMS), but I think that what it does is more than needs to be. It reads the obscured TGA into a Picture control and writes that out as a 32-bit TGA. I personally much prefer the command line interface of the BMP patcher, as this can be scripted much easier.

Of course, I'm sure you know that game.exe will read either obfuscated, or plain TGAs and BMPs. So from where you are, I might suggest you simply use the tools that are available to make your textures "normal" and concentrate on the binary SMD.

I only mention this because I fear you could easily get side-tracked, when it sounds like you are doing a fantastic job as you are, and tools to read and translate the BMPs and TGAs already exist. (all be it in undocumented, closed source form)

 

[Beyond2000!]

Hi Bob

i

 

[DarkKnightH20]

Afraid I can't help with the token, but you may have noticed their is a TGA de-obfuscator around here somewhere. (check RFT)

I've never found source for it, and attempts to understand it's workings have failed me. It seems to do more than the BMP patcher by Oscuro (IMS), but I think that what it does is more than needs to be. It reads the obscured TGA into a Picture control and writes that out as a 32-bit TGA. I personally much prefer the command line interface of the BMP patcher, as this can be scripted much easier.

Of course, I'm sure you know that game.exe will read either obfuscated, or plain TGAs and BMPs. So from where you are, I might suggest you simply use the tools that are available to make your textures "normal" and concentrate on the binary SMD.

I only mention this because I fear you could easily get side-tracked, when it sounds like you are doing a fantastic job as you are, and tools to read and translate the BMPs and TGAs already exist. (all be it in undocumented, closed source form)

Obscura made the BMP header patcher :

As for the source of the TGA utility, here it is:

You must be registered to see links

(VB6, utility by Telatoro)

Unfortunately, I never messed around with SMD files back in my primary PT days. I wish you luck. If I encounter anything useful, I'll be sure to post it here.

 

[Beyond2000!]

Many tks

the smd decoding is goinf fine... So far, the structure is like this:

[SMD.Data:
SMD.Data.Signature: B$ 0 #018
SMD.Data.Member01: D$ 0
SMD.Data.Member02: D$ 0 #0C
SMD.Data.Member05: D$ 0
SMD.Data.Member06: D$ 0
; since he read and write the 1st 556 bytes we simply fill the rest of the structure
SMD.Data.Member07: B$ 0 #01D8]


I

 

[Beyond2000!]

Many tks

the smd decoding is going fine... So far, the structure is like this:

[SMD.Data:
 SMD.Data.Signature: B$ 0 #018
 SMD.Data.Member01: D$ 0 ; materials count ?
 SMD.Data.Member02: D$ 0 #0C
 SMD.Data.Member05: D$ 0
 SMD.Data.Member06: D$ 0
 ; since he read and write the 1st 556 bytes we simply fill the rest of the structure
 SMD.Data.Member07: B$ 0 #01D8]

; folowed by a array of structure. Seems to be a array of material structure
; below seems to be an structure array, based on the amount of materials found. So, each material is a structure. So we have XXX materials followed in sequence below this.
; Each material  structure seems to have 40 Bytes long

I

 

[Beyond2000!]

Updated info

; SMD structure
;;
    The 1st thing the routines do is allocate 556 bytes to read the inputed SMD data, anc opy it to a buffer.
    So, it reads from SMD_Data.Signature to SMD.Data.Member07
;;
[SMD.Data:
 SMD.Data.Signature: B$ 0 #018
 SMD.Data.Member01: D$ 0 ; material count ?
 SMD.Data.Member02: D$ 0 #0C
 SMD.Data.Member05: D$ 0
 SMD.Data.Member06: B$ 0 #080; A buffer containing 128 bytes long
 ; since he read and write the 1st 556 bytes we simply fill the rest of the structure
 SMD.Data.Member07: B$ 0 #015C]


; below is a array of a structure (Material structure) whose amiunt is defined in SMD.Data.Member01
; This structure seems to have 40 Bytes long

[SMD.Material:
 SMD.Material.Data01: B$ 0 #40]

 

[Beyond2000!]

Updated info

; SMD structure
;;
    The 1st thing the routines do is allocate 556 bytes to read the inputed SMD data, anc opy it to a buffer.
    So, it reads from SMD_Data.Signature to SMD.Data.Member07

The signature type is of 2 types:
SMD Model data Ver 0.62
SMD Stage data Ver 0.72

The one i

 

[Beyond2000!]

Here comes a question...

What a hell is doing a disassembler inside PT ????????????????

also, this game uses zlib library zlib-1.1.4 from Jean-loup Gailly
and uses SSLeay-0.9.0b security library

 

[SoyEdu]

keep going man, pt is an old game but lol, this game is old and inside of it are alot secrets and isnt ez to extrat the true . Gl.

 

[Beyond2000!]

Great work

how did you got the "Refuge of ancients.max" ?? I can

 

[SoyEdu]

oh u saw it, well, i gave up trying to make a smd converter so i decided to make my own map, i started the map from scratch and moddel it in max so i made the ase file and i wont release it now, i will but later, i still have many things to do but i will release the smd so will be just to add in game not to modify, but what i can see is that u r on ur way to reveal the smd format so keep doing

 

[Beyond2000!]

Ok, but if you will release in smd format, how you will make PT load it ? I mean, the current format PT loads is Binary Smd and not Ascii SMD. Unless you load it from a .ase file, then you can load your map.

Hopefully, i

 

[Beyond2000!]

Also uses Lua 5.0 library

 

[bobsobol]

Yes, Most Game.exe files floating around have SSLea linked in, and all the current ones (post-AOR anyway) use LUA for the attack particle effects. Why they are static-linked, I'm not sure, but WoW Static links LUA for the UI overlay and such, so that's far from the stranges thing PT does.

The "bits of" disassembler I find in many game.exes are not used, but I gather that this has occurred at some point in PE-dumping... it happens like that. I've not noticed it in official KPT downloads, but I've not really looked either.

If they are still their, I would suggest that a PE dumper is (for what reason, only Yedang would know) used in the Game.exe build process to create the executable.

I have for, some time, suspected that the original source code for part (if not all) of the client and server is lost.

The rummer mill would have us believe that PT was created by a bunch of Collage / Uni students in Korea for them to play. It ran like many LAN games, with one player hosting a game that the others could log into. Pre-AOR (Age of Renaissance) betas I remember playing would have been not far from that. The game was sold and commercialised, a login system bolted on, and a non-3D client adapted to be a dedicated server. However, the commercial running and developement of the game has changed hands many times since then. I remember seeing the Betas go from Triglow Pictures to something else (beginning with P?) to Yedang Online.

Evidence within the way the game works supports this theory, but I have never found factual information that would prove it.

If it is even partly true, I would say that it's entirely possible that what the commercial organisations bought was not complete in source form, and some parts still need to be disassembled / dumped from executables; even by them! :scared: Either that, or they are at least static linked binary objects that have been generated by PE dumping.

I've been working on making a disassembly of the game.exe re-assemble in my spare time... (working slowly) and it's clear that there is a lot in the PE that is never used. You could null large areas of the file and see no adverse side effects. Particularly the assembly source code. If, and when I can get the listing to assemble and link, I have a profiler that can tell exactly which bits are not used, which are used, but don't need to be initialised etc.

However, my assembly days ended with DOS 5 and the 286 24-bit addressing. That was far enough from my Z80 beginnings for me to say "sod this lets learn C." I'm only really looking at assembling again (rather than messing with Olly or W32Dassem) because I've discovered that I can re-assemble small 32-bit PEs to 64-bit with GoAsm with very little modification. For small Windows Services, this reduces a 64-bit server the WOW3264 overhead. BUT... 3Meg of DirectX game is NOT a small PE. :?: I don't know if I will ever get it to work, but it's fun to try.

You, however, seem to be well on the way to making a complete breakthrough in the understanding of SMDs. I am very excited for you.

 

[Beyond2000!]

Many tks Bob

Regarding the disassemblement process.. Well.. i

 

[bobsobol]

hmm... sorry to hear that. My source listings currently stand a 24.5 Meg, including the Import and Export Tables that, I'm not sure need to be kept.

I've broken it down into Code, Data, RData. But much of the files are filled with DB 00h lines one after another which are just a complete waste of space and make the whole thing a complete PITA to edit. So I'm going through using some judicious S&R work to bum this down to something manageable.

Other than that, it won't assemble because the sections are not defined as .CODE or .DATA and will probably balk when I try, because of the Code sections which have fallen through as DBs... though most are streams of Nill Operations where security code (NProtect or XTrap) have been hacked out with a Hex editor I imagine. But some seem to be messed up jump tables.

If you are interested in what I have, you are welcome to a copy, just tell me how you'd like me to send it, as I'd rather not post it as an attachment since it's so messy. It comes from the 1871 No-XTrap game.exe, which has seemed to be the most stable for me. But if it's unlikely to be of assistance, by all means wait till I've cleaned it up some more.