New PWI compressed elements.data

[Nigel728]

Today there was a new expansion on PWI. Yesterday elements.data was 98mb, today its 19mb. Looking at the raw data, some parts are still readable so uncompressed, rest must be compressed somehow. Have looked at it but hard to figure compression algorithms just from raw data. If anyone has any ideas....

 

[Bola]

probably they are following the Swordsman online which has mppc compression on elements.data.

 

[Nigel728]

Yay you are right, I am still muddling through the specifics of it (like the fact that nothing tells you the uncompressed length of each record to expect, only how many bytes of compressed data there are). But at least I've got a workable start on it...

 

[juninhobks]

probably they are following the Swordsman online which has mppc compression on elements.data.

Yes, you're right!
All structure has been changed, and a new encryption has been generated from v500+ version..
And not only did elements.data change, gshop.data also changed its format, and +1 was added to gshop3.data as of v586+

 

[Nigel728]

Been looking at this again so will describe what I've found, in case its useful to anyone else. New compressed elements.data works like -

4 bytes of version (352 as of today)
4 bytes of offset1 (same stupid "offsets" random binary data as uncompressed elements.data had before lists 1,20,100)
4 bytes denoting length of some large header that idk what its for
n bytes of the large header
27 bytes of offset2
89 bytes of offset3
Then the data for the actual lists begins, so "equipment add-on" list is first as usual.

Each list is like:
4 bytes of number of items in the list
For each item in the list
- 4 bytes of the item ID
- 2 bytes of the size of compressed data for this list entry
4 bytes of the total compressed size (= sum of all the 2 byte compressed sizes across the whole list)
For each item in the list
- MPPC compressed data
- Since compression generates a bitstream (rather than bytestream) each item is padded as necessary to end on a byte boundary

That's as much as I have figured out, but I believe you can read the entire file like that.

Edit: TalkProc isn't where you'd expect. NpcEssence list is immediately followed by FaceTextureEssence list. idk where TalkProc is. I've only dug as far as FashionEssence because that's what I was interested in.