Author Topic: Lemmings DAT Anticompressor  (Read 22116 times)

0 Members and 1 Guest are viewing this topic.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Lemmings DAT Anticompressor
« on: August 27, 2005, 07:23:00 AM »
download link - major bugs, expect a new version within the next week...

This program allows you to "compress" files into .DAT files, however, in reality, the compressed files will be bigger than the originals.  Anyway, ccexplore could release a program that actually does some compression, because I'm not planning to, since I wasted way too much time debugging this anticompressor.
I can add an append feature if anyone wants it, since at the moment it will only overwrite DAT files.

This program is bound to have bugs since I can't possibly compress every possible file combination... ever.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #1 on: August 27, 2005, 07:46:15 AM »
I must say, "Anticompressor" is such an apt name for this.  ;)

Anyway, one way to gauge the reliability of your compression algorithm is as follows:  after you compressed stuff, decompress it back and check that it matches the original decompressed data byte by byte.  I have that verification build-in with every program I've released that uses compression (well, only myvgaspec so far).

This way, if there's a bug, at least you catch it before it ruins the user's files.

Then test your compression algorithm through the set of DAT files that comes with the game, making sure they all pass.  This is the extent of testing I've done on mine.  I won't say it achieves good coverage by any means, but it does at least give me a reasonable degree of confidence.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #2 on: August 28, 2005, 06:24:05 AM »
Ok... anticompression apparently doesn't work... :'(
So... I rewrote my compression code so that it actually does some amount of compression (2x size of Psygnosis files) and neither LemEdit nor CustLemm will load the recompressed data...  =8O  I'd guess they, being DOS apps require the files to be a compressed somewhat well...

I've tested my compressor, and it seem to have 100% accuracy (though I should add the error-checking mentioned above) but the files still won't load, so I shall try to optimise my code...

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #3 on: August 28, 2005, 07:40:29 AM »
Quote from: Mindless  link=1125127380/0#2 date=1125210245
Ok... anticompression apparently doesn't work... :'(
So... I rewrote my compression code so that it actually does some amount of compression (2x size of Psygnosis files) and neither LemEdit nor CustLemm will load the recompressed data... &#A0;=8O &#A0;I'd guess they, being DOS apps require the files to be a compressed somewhat well...

That's a bit surprising.  LemEdit doesn't produce particularly well-compressed files either.  And a level is only about the size of 2k, which is peanuts even by DOS standards.

Send me what you have (uncompressed + what your program produces).  Not that I'd necessarily be able to fix anything, but I do want to see it for myself.

Incidentally, possible stupid but common mistakes (I committed mistake #2 on my first attempt):

1) you did remember to include the header, right?
2) the compressed_size field in the header includes the 10 bytes in the header itself (that is, if the pure data compresses down to 1000 bytes, the header should read 1010 for the compressed size, not 1000)
3) remember the header's compressed and decompressed sizes are stored in big-endian.  So a size of, say, 0x0400 (1024 bytes in decimal) should have its bytes be stored as 0x04, 0x00, not 0x00, 0x04.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #4 on: August 28, 2005, 08:32:27 AM »
Fixed the problem, it was in my lazily coded raw chunk encoder... see what too many one byte raw chunks get you... :P Not sure why LemEdit and CustLemm can't handle repetitive raw one byte chunks as long as it follows the file format... Oh, well...
Hopefully I can work the bugs out of the far reference chunks routine...

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #5 on: August 28, 2005, 08:43:27 AM »
Quote from: Mindless  link=1125127380/0#4 date=1125217947
Not sure why LemEdit and CustLemm can't handle repetitive raw one byte chunks as long as it follows the file format...

It should.  At least CustLemm should.  If it were me I would be more inclined to look over the code you wrote more carefully.  (But if you truly believe that LemEdit/CustLemm is the problem, do send me the problematic DAT file your program produced, if possible, so I can take a more detailed look at how exactly CustLemm is choking on it.)

Incidentally, remember that, although closely modeled after the decompression routine in the game itself, ldecomp can serve as yet another test.  If ldecomp wouldn't accept it either, then you really should review your code.  Good luck!  ;)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #6 on: August 28, 2005, 08:47:14 AM »
Quote from: ccexplore (not logged in)  link=1125127380/0#5 date=1125218607
Incidentally, remember that, although closely modeled after the decompression routine in the game itself, ldecomp can serve as yet another test.

Then again, I don't think I ever updated ldecomp to perform detailed error checking.  So to really see that it works, make sure that not only does ldecomp not crash, but also that the result it produces match the original uncompressed data.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #7 on: August 28, 2005, 05:30:39 PM »
Here's by far the best example, since it doesn't crash CustLemm: a recompressed VGAGR0.DAT using lots of one byte raw chunks.  Instead of crashing, there are graphical errors.
http://it.travisbsd.org/lemmings/_misc/VGAGR0.recompressed.zip
And when decompressed by ldecomp, the data is identical to the decompressed data of the original.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #8 on: August 28, 2005, 08:44:31 PM »
Well, it's kind'a working, but when it is, it passes your method's compression ratio.
Here's the example included with myvgaspec and a recompressesed version. My recompressed version is half the size of yours.  Now, if I can only get it to give me constant results.
http://it.travisbsd.org/lemmings/_misc/examplevgaspec.recompressed.zip

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #9 on: August 29, 2005, 12:40:19 AM »
Quote from: Mindless  link=1125127380/0#7 date=1125250239
Here's by far the best example, since it doesn't crash CustLemm: a recompressed VGAGR0.DAT using lots of one byte raw chunks. &#A0;Instead of crashing, there are graphical errors.

Hmm, that is a bit odd I must say.  I guess I could check it again later against a version (not yet done) of ldecomp that has more stringent checks added in, but the fact that it produces apparently correct data seem to indicate it should be doing the right thing.

Does this happen only with VGAGR# files, or have you gotten similar problems with levelpak files or VGASPEC# files?

I can imagine problems with VGAGR# and VGASPEC# files since they are somewhat large and the typical limits a DOS program can run into is either 32k or 64k.  But what your program produces is only around 20-25k.  Of course, it's perfectly possible that the game might have set aside smaller amounts of memory for holding compressed data.  (Doesn't help either that the amount can potentially be different for different kinds of files.)

I found that the ONML's vgagr2.dat does have a data section whose compressed size exceeds the 20k, namely the first section (22379 bytes).  This is actually smaller than your example file's first section which was only 20611 bytes.  On the other hand, it's definitely the case that in all the official vgagr# files from Lemmings and ONML, their second data sections were all under 20k.  Hmm...do you remember if the second data section is for terrain or object graphics?

==================

This is rather not good news for me, since it means I probably should look into improving my own algorithm as well in terms of the compression ratio, if Lemmings is indeed so unforgiving.  Good thing that at least so far no one has run into similar problems yet with myvgaspec.

Thanks for alerting me to this.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #10 on: August 29, 2005, 01:07:46 AM »
You should only run into that problem if your compressor compresses in MANY one byte chunks, so basicly your compressor should never cause a problem.

---

Terrain is first in the vgagr# files, then objects.

---

On another note, I've decompressed and recompressed some of the Lemmings DATs, namely "main.dat" and "vgagr0.dat" and my compression method produces files that are a often few kB smaller than the Psynosis compression! :D

MAIN.DAT - 56,472 bytes
recompressed - 53,445 bytes

LEVEL000.DAT - 3,722 bytes
recompressed - 3,717 bytes

VGAGR0.DAT - 24,464 bytes
recompressed - 22,928 bytes

VGASPEC0.DAT - 28,655 bytes
recompressed - 28,146 bytes

ADLIB.DAT - 12,988 bytes
recompressed - 12,790 bytes

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #11 on: August 29, 2005, 01:25:33 AM »
Quote from: Mindless  link=1125127380/0#10 date=1125277666
You should only run into that problem if your compressor compresses in MANY one byte chunks, so basicly your compressor should never cause a problem.

I'm not 100% sure.  Keep in mind that all along, I simply test my compression algorithm by doing a comparison with the original decompressed data and the re-decompressed (is that a word?  ;P) data.  I haven't really tested it out in LemEdit/Custlemm like you did, with the exception of the example I made for myvgaspec.

From what I know about how the game decompresses, there should be no reason why it would choke on too many one-byte chunks.  So whether the problem is because the data is too large or because there are too many one-byte chunks, there's definitely something that is worth investigating.

Quote
On another note, I've decompressed and recompressed some of the Lemmings DATs, namely "main.dat" and "vgagr0.dat" and my compression method produces files that are a few kB smaller than the Psynosis compression! :D

Good job!  B)

Of course, the computers back then are a lot slower and has less RAM, so it's quite possible that Psynosis has to stick with a less efficient algorithm in order to ship the game on time.  ;P

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #12 on: August 29, 2005, 02:16:22 AM »
Alright, I think I've worked out all the bugs... it probably needs more error checking, but I'm gonna release it anyway...
A working version of my ex-anticompressor: http://it.travisbsd.org/lemmings/files/lemmingsdatcompressor/lemmingsdatcompressor_1_1_1.zip

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #13 on: August 30, 2005, 12:10:16 PM »
I've been rummaging thru main.dat and here's what's what.
Code: [Select]
main.dat section 0:
 lemming animations

main.dat section 1:
 impending doom (nuke) counter
 terrain destruction masks (nuke, mine, bash, maybe others)

main.dat section 2:
 skill numbers

main.dat section 3:
 brown background
 lemmings holding signs
 music/fx sign

main.dat section 4:
 purple text
 blinking lemming eyes
 scroller (lemmings and reel)
 difficulty selector sign

main.dat section 5:
 unknown

main.dat section 6:
 skill panel graphics
 green numbers and letters


And here's an empty main.dat (all 0x00's).  Invisible Lemmings!
http://it.travisbsd.org/lemmings/_misc/empty_main.zip

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #14 on: August 30, 2005, 06:44:06 PM »
Cool!  B)  The final missing piece of information finally falls into place.

Did you figure this out by examining the data in the actual file, or did you ask Mike for help?

If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.  I presume the format should be either some version of a planar bitmap.  Most likely a 3-bit one, possibly with an additional mask.

Is section #5 (decompressed) large?  If so it's most likely the intro screen (main menu) as a single bitmap.

Thanks!  B)

P.S.  Oh, and don't forget to do it for the Xmas Lemmings, which have different graphics in particular the lemmings animations.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #15 on: August 30, 2005, 10:42:30 PM »
Quote from: ccexplore (not logged in)  link=1125127380/0#14 date=1125427446
Cool!  B)  The final missing piece of information finally falls into place.

Did you figure this out by examining the data in the actual file, or did you ask Mike for help?

I'm just messing with the bits in the file, I haven't seen Mike around for a long time... :'(

Quote from: ccexplore (not logged in)  link=1125127380/0#14 date=1125427446
If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.  I presume the format should be either some version of a planar bitmap.  Most likely a 3-bit one, possibly with an additional mask.

Well, at the moment I'm working on the Lemmings animation data which is 2bpp planar bitmaps with black being transparent (no mask).

Quote from: ccexplore (not logged in)  link=1125127380/0#14 date=1125427446
Is section #5 (decompressed) large?  If so it's most likely the intro screen (main menu) as a single bitmap.

Thanks!  B)

P.S.  Oh, and don't forget to do it for the Xmas Lemmings, which have different graphics in particular the lemmings animations.

Hmm? I'm numbering the data sections from 0 to 6, so section 5 decompressed is 758 bytes.

The intro screen is composed of a brown background with seperate planar bitmaps for the Lemmings holding signs and the Lemmings logo.

On a similar note, it's interesting that the shadow behind the Lemmings logo and the Lemmings holding signs is dynamic, therefore if you change the logo graphic or the sign graphics, the shadow on the intro screen is created at runtime to match the graphic.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #16 on: August 30, 2005, 11:07:49 PM »
Quote from: Mindless  link=1125127380/15#15 date=1125441750
Well, at the moment I'm working on the Lemmings animation data which is 2bpp planar bitmaps with black being transparent (no mask).

Apparently this only applies to the first two animations, which are the walking animations.  They are 2bbp planar, black as transparent, 16x10 pixels, 8 frames each.

Some of the other animations (mining, basing, exploding?) require more planes. (possibly different dimensions, 8x10?)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #17 on: August 30, 2005, 11:39:25 PM »
Quote from: Mindless  link=1125127380/15#15 date=1125441750
Hmm? I'm numbering the data sections from 0 to 6, so section 5 decompressed is 758 bytes.

I was just venturing a wild guess for the section you listed as "unknown".  Well, 758 bytes is clearly not the intro graphics.

What happens when you modify the data of that section?  Where the game messes up/crashes/changes should give us a clue about that section

Quote
On a similar note, it's interesting that the shadow behind the Lemmings logo and the Lemmings holding signs is dynamic,

Wow, didn't think the programmers/Mike would pull something that fancy (well okay, it's not that hard, but still).  Cool.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #18 on: August 30, 2005, 11:44:35 PM »
Quote from: ccexplore (not logged in)  link=1125127380/0#14 date=1125427446
If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.

Alternatively, since we can calculate, you just need to give for each section the sequence of bitmaps, and their dimensions, how many bpp, and whether there's a mask.  (This is assuming there are no unused or non-graphics bytes between one bitmap and the next.)

Thanks again, actually I'll probably look into this too as soon as I have a chance.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #19 on: August 30, 2005, 11:58:38 PM »
Quote from: Mindless  link=1125127380/15#16 date=1125443269
Some of the other animations (mining, basing, exploding?) require more planes. (possibly different dimensions, 8x10?)

I would expect them to be larger, not smaller......

One thing I've planned in mind (but will probably take some time to implement, if ever) currently is an app that helps explore this sea of uncharted bits.  I envision it to be like this:  There are 5 columns:  one that interprets the bits as just monochrome bitmaps (in effect, displaying the planes individually), next column interprets them as 2 bpp, the next as 3 bpp, another as 4 bpp, and yet another as 5 bpp (4 bpp+mask, in essense).  You can set the program to start interpreting from a particular offset, and also to set the width.  The program will then go thru a certain number of scanlines (say, 300) assuming that you have a sequence of bitmaps of the specified width.  If you guess right, then you'll get something legible, otherwise you get garbage and should try another width.

This should hopefully help with figuring out how exactly the bitmap are laid out.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #20 on: August 31, 2005, 12:41:09 AM »
That would be a useful program!

Anyway, as to my comment about the smaller animations, that was because I was missing a frame.
Code: [Select]
main.dat section 0:
 lemming animations -  black = transparent
   [0x0000] walking (r)       8 frames, 16x10 px, 2bpp planar, 40 Bytes/frame
   [0x0140] jumping (r)       1 frame,  16x10 px, 2bbp planar, 40 B/f
   [0x0168] walking (l)       8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x02A8] jumping (l)       1 frame,  16x10 px, 2bpp planar, 40 B/f
   [0x02D0] digging          16 frames, 16x14 px, 3bpp planar, 84 B/f
   [0x0810] climbing (r)      8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0990] climbing (l)      8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0B10] drowning         16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x0D90] post-climb (r)    8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0F10] post-climb (l)    8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x1090] brick layer (r)  16 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x1570] brick layer (l)  16 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x1A50] bash (r)         32 frames, 16x10 px, 3bpp planar, 60 B/f
   [0x21D0] bash (l)         32 frames, 16x10 px, 3bpp planar, 60 B/f
   [0x2950] mine (r)         24 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x30A0] mine (l)         24 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x37F0] fall (r)          4 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x3890] fall (l)          4 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x3930] pre-umbrella (r)  4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3AB0] umbrella (r)      4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3C30] pre-umbrella (l)  4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3DB0] umbrella (l)      4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3F30] splat(ter?)      16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x41B0] exit(ter?)        8 frames, 16x13 px, 2bpp planar, 52 B/f
   [0x4350] burn(er?)        14 frames, 16x14 px, 4bpp planar, 112 B/f
   [0x4970] blocker          16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4BF0] shrug(ger?) (r)   8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4D30] shrug(ger?) (l)   8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4E70] exploder         16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x50F0] explosion         1 frame,  32x32 px, 3bpp planar, 384 B/f

The jumping animation refers to the action that a Lemming does when it reaches a height of terrian that it can scale w/o being a climber.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #21 on: September 01, 2005, 02:30:53 AM »
I completeded disecting section 0, and I also wrote a tool something like you mentioned above, if anyone wants it, I'll put it up.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #22 on: September 01, 2005, 03:37:03 AM »
Quote from: Mindless  link=1125127380/15#21 date=1125541853
I completeded disecting section 0, and I also wrote a tool something like you mentioned above, if anyone wants it, I'll put it up.

Cool!  I don't know if I'll ever do anything with it, but it's cool that you've completed the entire section 0.  And yeah, put up the tool.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #23 on: September 01, 2005, 05:04:32 AM »
Code: [Select]
main.dat section 1:
 terrain destruction masks
   [0x0000] bash (r)          4 frames, 16x10 px, 1bpp planar, 20 B/f
   [0x0050] bash (l)          4 frames, 16x10 px, 1bpp planar, 20 B/f
   [0x00A0] mine (r)          2 frames, 16x13 px, 1bpp planar, 26 B/f
   [0x00D4] mine (l)          2 frames, 16x13 px, 1bpp planar, 26 B/f
   [0x0108] nuke              1 frames, 16x22 px, 1bpp planar, 44 B/f
 nuke counter
   [0x0134] "9"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x013C] "8"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0144] "7"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x014C] "6"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0154] "5"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x015C] "4"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0164] "3"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x016C] "2"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0174] "1"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x017C] "0"               1 frame,  8x8 px,   1bpp planar, 8 B/f


Tool: http://it.travisbsd.org/lemmings/_misc/gxplr.zip
Limitations: Many. Loads palette from ground0o.dat.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #24 on: September 01, 2005, 05:53:53 AM »
Quote from: Mindless  link=1125127380/15#23 date=1125551072
Code: [Select]
main.dat section 1:
 &#A0;terrain destruction masks
 &#A0; &#A0;[0x0000] bash (r) &#A0; &#A0; &#A0; &#A0; &#A0;4 frames, 16x10 px, 1bpp planar, 20 B/f
 &#A0; &#A0;[0x0050] bash (l) &#A0; &#A0; &#A0; &#A0; &#A0;4 frames, 16x10 px, 1bpp planar, 20 B/f
 &#A0; &#A0;[0x00A0] mine (r) &#A0; &#A0; &#A0; &#A0; &#A0;2 frames, 16x13 px, 1bpp planar, 26 B/f
 &#A0; &#A0;[0x00D4] mine (l) &#A0; &#A0; &#A0; &#A0; &#A0;2 frames, 16x13 px, 1bpp planar, 26 B/f
 &#A0; &#A0;[0x0108] nuke &#A0; &#A0; &#A0; &#A0; &#A0; &#A0; &#A0;1 frames, 16x22 px, 1bpp planar, 44 B/f

<drools...>Ooh ooh ooh, I had been wanting this for such a long time.  This is heaven (well, it'll come in handy someday anyway in some unexpected way, I'm sure), thanks!

:party: <- (well, for me, anyhow ;P)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #25 on: September 01, 2005, 05:56:41 AM »
By the way, you can probably drop the word "planar" from future listings.  Because of the way the VGA/EGA hardware works, it simply doesn't make sense programming-wise to use anything else if you know you'll work with a specific graphics mode.  So all bitmaps should be planar.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #26 on: September 01, 2005, 10:38:01 AM »
Well, here's my attempt at section #2.  Unfortunately, most of it is still indecipherable junk to me, and I also found things that doesn't seem to be used in the game.  Anyhow:

Code: [Select]
main.dat section 2:
 skill number digits
    [0x1900] right "0", 1 frame, 8x8, 1bpp
    [0x1908]  left "0", 1 frame, 8x8, 1bpp
    [0x1910] right "1", 1 frame, 8x8, 1bpp
    [0x1918]  left "1", 1 frame, 8x8, 1bpp
    [0x1920] right "2", 1 frame, 8x8, 1bpp
    [0x1928]  left "2", 1 frame, 8x8, 1bpp
    [0x1930] right "3", 1 frame, 8x8, 1bpp
    [0x1938]  left "3", 1 frame, 8x8, 1bpp
    [0x1940] right "4", 1 frame, 8x8, 1bpp
    [0x1948]  left "4", 1 frame, 8x8, 1bpp
    [0x1950] right "5", 1 frame, 8x8, 1bpp
    [0x1958]  left "5", 1 frame, 8x8, 1bpp
    [0x1960] right "6", 1 frame, 8x8, 1bpp
    [0x1968]  left "6", 1 frame, 8x8, 1bpp
    [0x1970] right "7", 1 frame, 8x8, 1bpp
    [0x1978]  left "7", 1 frame, 8x8, 1bpp
    [0x1980] right "8", 1 frame, 8x8, 1bpp
    [0x1988]  left "8", 1 frame, 8x8, 1bpp
    [0x1990] right "9", 1 frame, 8x8, 1bpp
    [0x1998]  left "9", 1 frame, 8x8, 1bpp

Each digit is actually only 4 pixels wide, so they made two versions of each digit, one with the digit on the left half of the 8x8 box, the other with the digit on the right half of the 8x8 box.  This makes it convenient to construct any 2-digit number within an 8x8 box, like the numbers in the skill toolbar.


Code: [Select]
 mystery 8x16 characters
    [0x19A0] "%", 1 frame, 8x16, 3bpp
    ...
(characters included, in order)
% 0 1 2 3 4 5 6 7 8 9 . A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

These 8x16 characters takes up all the space up to the end of section #2.  The intended palette is clearly not the one from groundXo.dat.  I'm not able to figure out where they are actually used; maybe they aren't.  Note: they are not the green letters you see when playing a level.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #27 on: September 01, 2005, 11:51:05 AM »
Only partially figured out parts of sections #3 and #4 so far:

Code: [Select]
main.dat section 3:
 Lemmmings holding signs
   [0x9488]  F1 sign, 1 frame, 120x61, 4 bpp
   [0xA2D4]  F2 sign, 1 frame, 120x61, 4 bpp
   [0xB120]  F3 sign*, 1 frame, 120x61, 4 bpp
   [0xBF6C]  Difficulty Selection sign**, 1 frame, 120x61, 4 bpp
   [0xCDB8]  Exit to DOS sign, 1 frame, 120x61, 4 bpp
   [0xDC04]  F4 sign, 1 frame, 120x61, 4 bpp
*with the music/fx set to blank
**with the difficulty set to FUN

This is the end of section 3, contains the "Lemmings holding signs" graphics you see on the main menu screen.

Code: [Select]
main.dat section 4:
 purple text
    [0x69B0] "!", 1 frame, 16x16, 3bpp
    ...
 (contains the ASCII characters, in order, from '!' to "~")

This is end of section 4, contains all the purple text characters you see throughout the game.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #28 on: September 01, 2005, 12:20:28 PM »
I seem to do best with the font graphics in all sections so far:
Code: [Select]
main.dat section 6:
 Green text characters:
 [0x1900]  "%", 1 frame, 8x16, 3bpp
 ...

% 0 1 2 3 4 5 6 7 8 9 - A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Again, located at end of section.  Notice parallel with end of section 2 (note correction:  my earlier description of end of section 2 listed the character '.' after '9'.  It should be a '-' instead just like here, the '-' is the minute-second separator in the display of time remaining).  These characters are the ones actually used in the game when playing a level.

Offline Shvegait

  • Posts: 772
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #29 on: September 01, 2005, 02:04:35 PM »
I think I know what the other set of characters is for. Could they be the green text when playing the game in IBM PS2 or "High Performance" mode? (These characters are different from the ones when playing with PC Compatible or AMSTRAD modes).

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #30 on: September 01, 2005, 08:50:11 PM »
Quote from: Shvegait  link=1125127380/15#29 date=1125583475
I think I know what the other set of characters is for. Could they be the green text when playing the game in IBM PS2 or "High Performance" mode? (These characters are different from the ones when playing with PC Compatible or AMSTRAD modes).

That sounds quite plausible.  So you got the PS2/High Perf mode to work?  Is it DOSBox or just plain Windows DOS?

Offline Shvegait

  • Posts: 772
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #31 on: September 01, 2005, 08:54:06 PM »
DOSBox. I don't think they actually "work", but they at least load up to the level screen, which is enough to see that the letters are in a different font. If I recall correctly, one of the modes doesn't have mouse (clicking) support, one of them will freeze between levels, in one of them the level won't even start (but the graphics will load)... Weird stuff happens for sure.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #32 on: September 01, 2005, 10:15:19 PM »
Code: [Select]
main.dat section 3:
 main screen
   [0x0000] brown background             1 frame, 320x104, 2bpp
   [0x2080] Lemmings logo                1 frame, 632x94,  4bpp

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #33 on: September 01, 2005, 11:23:20 PM »
Code: [Select]
main.dat section 4:
 blinking eyes
   [0x0000] blink1         8 frames, 32x12, 4bpp, 192 B/f
   [0x0600] blink2         8 frames, 32x12, 4bpp, 192 B/f
   [0x0C00] blink3         8 frames, 32x12, 4bpp, 192 B/f
   [0x1200] blink4         8 frames, 32x12, 4bpp, 192 B/f
   [0x1800] blink5         8 frames, 32x12, 4bpp, 192 B/f
   [0x1E00] blink6         8 frames, 32x12, 4bpp, 192 B/f
   [0x2400] blink6         8 frames, 32x12, 4bpp, 192 B/f
 scroller (lemmings and reel)
   [0x2A00]  left Lemming 16 frames, 48x16, 4bpp, 384 B/f
   [0x4200] right Lemming 16 frames, 48x16, 4bpp, 384 B/f
   [0x5A00] reel           1 frame,  16x16, 4bpp, 128 B/f
 difficulty selector sign
   [0x5A80] mayhem sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x5E4C] taxing sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x6218] tricky sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x65E4] fun sign       1 frame,  72x27, 4bpp, 972 B/f

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #34 on: September 02, 2005, 02:30:58 AM »
Code: [Select]
main.dat section 6:
 skill panel graphics
   [0x0000] skill panel  1 frame, 320x40, 4bpp, 6400 B/f

That about raps it up, except for section 5, which still completely perplexes me.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #35 on: September 02, 2005, 09:43:19 AM »
Actually, I think we're still missing the graphics for the music options (you know, the F3 one where you select between no music [blank], music [music note], or just sound effects ["FX"]).

And technically, there are still bytes at the start of section 2 where we can't find a purpose for, although that does mean we could care less about it for now.

Maybe section 5 is only used in one of the "High Perf" or whatever machine options.  How thoroughly have you tested that section?  I assume you've identified the general contents of each section by zeroing it out and see what parts of the game are affected.  Have you try this:  zero out section 5, pick some level, and then nuke it?  Just a total random (and probably wrong) hunch, see carefully if anything changes.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #36 on: September 02, 2005, 11:39:49 AM »
Oh, oops, I forgot about the beginning of section 2...

Section 5 doesn't seem to have any effect on the VGA game, unless it effects the special graphics levels. I've fillled it with both 0x00's and 0x0F's and nothing seems to change, I played the first few levels without noticing anything wrong, and even with the whole main.dat fillled with 0x00's the explosion pixels are still there.

The thing I've noticed is that section 5 is the only one that has a decompressed size which is not divisible by 8, leading me to believe that it is not graphical data.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #37 on: September 02, 2005, 12:19:17 PM »
Code: [Select]
main.dat section 2:
 skill panel (high performance)
   [0x0000] skill panel       1 frame,  320x40, 4bpp

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #38 on: September 02, 2005, 12:27:06 PM »
Quote from: ccexplore (not logged in)  link=1125127380/30#35 date=1125654199
Actually, I think we're still missing the graphics for the music options (you know, the F3 one where you select between no music [blank], music [music note], or just sound effects ["FX"]).

Never mind, that was my fault, I wasn't paying attention to the offsets (or actually, it's really gxplr's fault for the way it handles the +/-).  It turns out that I actually only read the section up to 0xEA50 or so, but gxplr wraps back to 0 at that point because it's still trying to read a 120x61x4bpp bitmap, which in the process causes it to go off the end of section and so it chooses to wrap around to 0.  So I didn't even realize I haven't actually reach the end of section yet.

Anyway, it turns out the music and FX icons are after the main signs, so here we go, the corrected listing for the end of section 3:

Code: [Select]
main.dat section 3:
 Lemmmings holding signs
   [0x9488]  F1 sign, 1 frame, 120x61, 4 bpp
   [0xA2D4]  F2 sign, 1 frame, 120x61, 4 bpp
   [0xB120]  F3 sign*, 1 frame, 120x61, 4 bpp
   [0xBF6C]  Difficulty Selection sign**, 1 frame, 120x61, 4 bpp
   [0xCDB8]  Exit to DOS sign, 1 frame, 120x61, 4 bpp
   [0xDC04]  F4 sign, 1 frame, 120x61, 4 bpp
   [0xEA50]  music note icon for F3 sign, 1 frame, 64x31, 4 bpp
   [0xEE30]  "FX" icon for F3 sign, 1 frame, 64x31, 4 bpp
*with the music/fx set to blank
**with the difficulty set to FUN


This time, I used a calculator and so we have indeed reach the real end of section 3.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #39 on: September 02, 2005, 12:38:42 PM »
Quote from: Mindless  link=1125127380/30#36 date=1125661189
The thing I've noticed is that section 5 is the only one that has a decompressed size which is not divisible by 8, leading me to believe that it is not graphical data.

Certainly possible.  What I had in mind when I asked about nuking is that instead of storing the explosion's scattered pixels/particles as bitmaps (which hardly makes sense), they could just store a table of (x,y)s and/or velocity vectors to represent the explosion pixels and how they move over time.  But you said you tested explosions, so I guess that isn't it.

Perhaps one way to find out if it is nongraphical data would be to compare the data with the corresponding section in cgamain.dat I guess.  If it's completely unrelated to graphics, then that section should be identical in both files.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #40 on: September 02, 2005, 12:49:47 PM »
Quote from: ccexplore (not logged in)  link=1125127380/30#39 date=1125664722
Perhaps one way to find out if it is nongraphical data would be to compare the data with the corresponding section in cgamain.dat I guess. &#A0;If it's completely unrelated to graphics, then that section should be identical in both files.

Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

Could it have something to do with the copy protection thing?  How does that work in a real, non-cracked version anyway?

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #41 on: September 02, 2005, 03:11:49 PM »
Quote from: ccexplore (not logged in)  link=1125127380/30#38 date=1125664026
...(or actually, it's really gxplr's fault for the way it handles the +/-). &#A0;It turns out that I actually only read the section up to 0xEA50 or so, but gxplr wraps back to 0 at that point because it's still trying to read a 120x61x4bpp bitmap, which in the process causes it to go off the end of section and so it chooses to wrap around to 0. &#A0;So I didn't even realize I haven't actually reach the end of section yet.

I should have listed that in the limitations. It's to prevent crashes, as you've probably guessed. &#A0;;)

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #42 on: September 03, 2005, 05:23:41 AM »
Quote from: ccexplore (not logged in)  link=1125127380/30#40 date=1125665387
Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

Could it have something to do with the copy protection thing?  How does that work in a real, non-cracked version anyway?

http://groups.google.com/groups?q=%22russell.dat%22+lemmings&qt_s=Search

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #43 on: September 03, 2005, 11:41:21 PM »
I've collected the information on main.dat into one document:

http://www.geocities.com/guestlevels/lemmings/maindat.txt

There are some slightly missing info that I still need to fill in, but everything else is there, including all the tables we posted here for the offsets to the various graphics and animations.

I'm too lazy to manually insert line breaks, so to best view this document, you should save it (right click on the link and choose "Save Target") to your machine and then open it with Notepad or Word.  Make sure to turn word-wrap on in Notepad.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #44 on: September 04, 2005, 03:48:17 AM »
Intro Screen Palette
Code: [Select]
00 [0x000000] r:0   g:0   b:0   black
01 [0x201008] r:128 g:64  b:32  browns
02 [0x180C08] r:96  g:48  b:32  "
03 [0x0C0004] r:48  g:0   b:16  "

04 [0x08021F] r:32  g:8   b:124 purples
05 [0x100B24] r:64  g:44  b:144 "
06 [0x1A1629] r:104 g:88  b:164 "
07 [0x26232F] r:152 g:140 b:188 "

08 [0x001400] r:0   g:80  b:0   greens
09 [0x001804] r:0   g:96  b:16  "
10 [0x001C08] r:0   g:112 b:32  "
11 [0x002010] r:0   g:128 b:64  "

12 [0x343434] r:208 g:208 b:208 white
13 [0x2C2C00] r:176 g:176 b:0   yellow
14 [0x10142C] r:64  g:80  b:176 blue
15 [0x382024] r:224 g:128 b:144 pink


Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #45 on: September 04, 2005, 04:40:02 AM »
Great!  maindat.txt is now updated with the palette information, so it should have everything someone would need for extracting all the bitmaps out of main.dat.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #46 on: September 04, 2005, 05:08:34 AM »
Quote from: Shvegait  link=1125127380/30#31 date=1125608046
DOSBox. I don't think they actually "work", but they at least load up to the level screen, which is enough to see that the letters are in a different font. If I recall correctly, one of the modes doesn't have mouse (clicking) support, one of them will freeze between levels, in one of them the level won't even start (but the graphics will load)... Weird stuff happens for sure.

Actually, I just tried the "PS/2" mode and it seems to work fine in DOSBox, maybe it's an XP/Win98 difference?

I also ran both the "High Performance" and the "PS/2" mode in plain DOS Windows and see the flashing behavior when you're in a level.  This seems to prove my theory that the game tries to do the palette switching trick, so that separate color palettes can be used in separate areas of the screen.  This also explain why they can afford a different green text font and skills panel for those modes.

Of course, the timing is no longer correct for modern PCs, and it just plain confuses DOSBox.

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #47 on: September 04, 2005, 05:09:38 AM »
Quote from: ccexplore (not logged in)  link=1125127380/45#46 date=1125810514
Actually, I just tried the "PS/2" mode and it seems to work fine in DOSBox

Note that "work fine" means doesn't crash.  The colors are indeed a little messed up, but that's to be expected since palette switching tricks are not emulated correctly by DOSBox.

Offline Mindless

  • Posts: 722
    • View Profile
    • Lemmings Level Database
Re: Lemmings DAT Anticompressor
« Reply #48 on: April 24, 2023, 12:21:26 AM »
Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

I finally solved the mystery.  It contains the PC speaker sounds that play if you don't have enough memory to load ADLIB.DAT (but ADLIB.DAT is loaded before MAIN.DAT, so the game never works in that case).  Interestingly, I can't find any normal way to get the PC speaker sounds to play.  They don't play normally, even if you don't have an AdLib-capable sound card.  I was only able to get them to play by modifying the game to skip AdLib initialization.  (Of course, maybe this is just a bug in the version of the game that I'm using.  Edit: It was.)

Edit 2:  There are 3 different versions of Lemmings for DOS with the VGALEMMI.EXE files dated 1991-05-10, 1991-05-24, and 1991-07-19.  I was using the version dated 1991-05-24, which has a bug that makes PC speaker sound effects impossible.  Incidentally, it also has the bug where the music starts over at track 1 every time you fail a level.

Edit 3: An easy way to tell which version you have is based on the message when you exit the game.  If it says "Have a nice DOS!" then you have the buggy version.
« Last Edit: July 15, 2023, 08:51:09 PM by Mindless »

Offline jkapp76

  • Posts: 388
    • View Profile
Re: Lemmings DAT Anticompressor
« Reply #49 on: April 24, 2023, 02:58:51 AM »
I'd love to hear PC-speaker music!
...Jeremy Kapp