Lemmings DAT Anticompressor

[Mindless]

download link - major bugs, expect a new version within the next week...

This program allows you to "compress" files into .DAT files, however, in reality, the compressed files will be bigger than the originals.  Anyway, ccexplore could release a program that actually does some compression, because I'm not planning to, since I wasted way too much time debugging this anticompressor.
I can add an append feature if anyone wants it, since at the moment it will only overwrite DAT files.

This program is bound to have bugs since I can't possibly compress every possible file combination... ever.

[ccexplore]

I must say, "Anticompressor" is such an apt name for this.  

Anyway, one way to gauge the reliability of your compression algorithm is as follows:  after you compressed stuff, decompress it back and check that it matches the original decompressed data byte by byte.  I have that verification build-in with every program I've released that uses compression (well, only myvgaspec so far).

This way, if there's a bug, at least you catch it before it ruins the user's files.

Then test your compression algorithm through the set of DAT files that comes with the game, making sure they all pass.  This is the extent of testing I've done on mine.  I won't say it achieves good coverage by any means, but it does at least give me a reasonable degree of confidence.

[Mindless]

Ok... anticompression apparently doesn't work...
So... I rewrote my compression code so that it actually does some amount of compression (2x size of Psygnosis files) and neither LemEdit nor CustLemm will load the recompressed data...  =8O  I'd guess they, being DOS apps require the files to be a compressed somewhat well...

I've tested my compressor, and it seem to have 100% accuracy (though I should add the error-checking mentioned above) but the files still won't load, so I shall try to optimise my code...

[ccexplore]

Ok... anticompression apparently doesn't work...
So... I rewrote my compression code so that it actually does some amount of compression (2x size of Psygnosis files) and neither LemEdit nor CustLemm will load the recompressed data... &#A0;=8O &#A0;I'd guess they, being DOS apps require the files to be a compressed somewhat well...

That's a bit surprising.  LemEdit doesn't produce particularly well-compressed files either.  And a level is only about the size of 2k, which is peanuts even by DOS standards.

Send me what you have (uncompressed + what your program produces).  Not that I'd necessarily be able to fix anything, but I do want to see it for myself.

Incidentally, possible stupid but common mistakes (I committed mistake #2 on my first attempt):

1) you did remember to include the header, right?
2) the compressed_size field in the header includes the 10 bytes in the header itself (that is, if the pure data compresses down to 1000 bytes, the header should read 1010 for the compressed size, not 1000)
3) remember the header's compressed and decompressed sizes are stored in big-endian.  So a size of, say, 0x0400 (1024 bytes in decimal) should have its bytes be stored as 0x04, 0x00, not 0x00, 0x04.

[Mindless]

Fixed the problem, it was in my lazily coded raw chunk encoder... see what too many one byte raw chunks get you... Not sure why LemEdit and CustLemm can't handle repetitive raw one byte chunks as long as it follows the file format... Oh, well...
Hopefully I can work the bugs out of the far reference chunks routine...

[ccexplore]

Not sure why LemEdit and CustLemm can't handle repetitive raw one byte chunks as long as it follows the file format...

It should.  At least CustLemm should.  If it were me I would be more inclined to look over the code you wrote more carefully.  (But if you truly believe that LemEdit/CustLemm is the problem, do send me the problematic DAT file your program produced, if possible, so I can take a more detailed look at how exactly CustLemm is choking on it.)

Incidentally, remember that, although closely modeled after the decompression routine in the game itself, ldecomp can serve as yet another test.  If ldecomp wouldn't accept it either, then you really should review your code.  Good luck!  

[ccexplore]

Incidentally, remember that, although closely modeled after the decompression routine in the game itself, ldecomp can serve as yet another test.

Then again, I don't think I ever updated ldecomp to perform detailed error checking.  So to really see that it works, make sure that not only does ldecomp not crash, but also that the result it produces match the original uncompressed data.

[Mindless]

Here's by far the best example, since it doesn't crash CustLemm: a recompressed VGAGR0.DAT using lots of one byte raw chunks.  Instead of crashing, there are graphical errors.
http://it.travisbsd.org/lemmings/_misc/VGAGR0.recompressed.zip
And when decompressed by ldecomp, the data is identical to the decompressed data of the original.

[Mindless]

Well, it's kind'a working, but when it is, it passes your method's compression ratio.
Here's the example included with myvgaspec and a recompressesed version. My recompressed version is half the size of yours.  Now, if I can only get it to give me constant results.
http://it.travisbsd.org/lemmings/_misc/examplevgaspec.recompressed.zip

[ccexplore]

Here's by far the best example, since it doesn't crash CustLemm: a recompressed VGAGR0.DAT using lots of one byte raw chunks. &#A0;Instead of crashing, there are graphical errors.

Hmm, that is a bit odd I must say.  I guess I could check it again later against a version (not yet done) of ldecomp that has more stringent checks added in, but the fact that it produces apparently correct data seem to indicate it should be doing the right thing.

Does this happen only with VGAGR# files, or have you gotten similar problems with levelpak files or VGASPEC# files?

I can imagine problems with VGAGR# and VGASPEC# files since they are somewhat large and the typical limits a DOS program can run into is either 32k or 64k.  But what your program produces is only around 20-25k.  Of course, it's perfectly possible that the game might have set aside smaller amounts of memory for holding compressed data.  (Doesn't help either that the amount can potentially be different for different kinds of files.)

I found that the ONML's vgagr2.dat does have a data section whose compressed size exceeds the 20k, namely the first section (22379 bytes).  This is actually smaller than your example file's first section which was only 20611 bytes.  On the other hand, it's definitely the case that in all the official vgagr# files from Lemmings and ONML, their second data sections were all under 20k.  Hmm...do you remember if the second data section is for terrain or object graphics?

==================

This is rather not good news for me, since it means I probably should look into improving my own algorithm as well in terms of the compression ratio, if Lemmings is indeed so unforgiving.  Good thing that at least so far no one has run into similar problems yet with myvgaspec.

Thanks for alerting me to this.

[Mindless]

You should only run into that problem if your compressor compresses in MANY one byte chunks, so basicly your compressor should never cause a problem.

---

Terrain is first in the vgagr# files, then objects.

---

On another note, I've decompressed and recompressed some of the Lemmings DATs, namely "main.dat" and "vgagr0.dat" and my compression method produces files that are a often few kB smaller than the Psynosis compression!

MAIN.DAT - 56,472 bytes
recompressed - 53,445 bytes

LEVEL000.DAT - 3,722 bytes
recompressed - 3,717 bytes

VGAGR0.DAT - 24,464 bytes
recompressed - 22,928 bytes

VGASPEC0.DAT - 28,655 bytes
recompressed - 28,146 bytes

ADLIB.DAT - 12,988 bytes
recompressed - 12,790 bytes

[ccexplore]

You should only run into that problem if your compressor compresses in MANY one byte chunks, so basicly your compressor should never cause a problem.

I'm not 100% sure.  Keep in mind that all along, I simply test my compression algorithm by doing a comparison with the original decompressed data and the re-decompressed (is that a word?  ) data.  I haven't really tested it out in LemEdit/Custlemm like you did, with the exception of the example I made for myvgaspec.

From what I know about how the game decompresses, there should be no reason why it would choke on too many one-byte chunks.  So whether the problem is because the data is too large or because there are too many one-byte chunks, there's definitely something that is worth investigating.

On another note, I've decompressed and recompressed some of the Lemmings DATs, namely "main.dat" and "vgagr0.dat" and my compression method produces files that are a few kB smaller than the Psynosis compression!

Good job!  B)

Of course, the computers back then are a lot slower and has less RAM, so it's quite possible that Psynosis has to stick with a less efficient algorithm in order to ship the game on time.  

[Mindless]

Alright, I think I've worked out all the bugs... it probably needs more error checking, but I'm gonna release it anyway...
A working version of my ex-anticompressor: http://it.travisbsd.org/lemmings/files/lemmingsdatcompressor/lemmingsdatcompressor_1_1_1.zip

[Mindless]

I've been rummaging thru main.dat and here's what's what.

Code: [Select]

main.dat section 0:
 lemming animations

main.dat section 1:
 impending doom (nuke) counter
 terrain destruction masks (nuke, mine, bash, maybe others)

main.dat section 2:
 skill numbers

main.dat section 3:
 brown background
 lemmings holding signs
 music/fx sign

main.dat section 4:
 purple text
 blinking lemming eyes
 scroller (lemmings and reel)
 difficulty selector sign

main.dat section 5:
 unknown

main.dat section 6:
 skill panel graphics
 green numbers and letters

And here's an empty main.dat (all 0x00's).  Invisible Lemmings!
http://it.travisbsd.org/lemmings/_misc/empty_main.zip

[ccexplore]

Cool!  B)  The final missing piece of information finally falls into place.

Did you figure this out by examining the data in the actual file, or did you ask Mike for help?

If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.  I presume the format should be either some version of a planar bitmap.  Most likely a 3-bit one, possibly with an additional mask.

Is section #5 (decompressed) large?  If so it's most likely the intro screen (main menu) as a single bitmap.

Thanks!  B)

P.S.  Oh, and don't forget to do it for the Xmas Lemmings, which have different graphics in particular the lemmings animations.

[Mindless]

Cool!  B)  The final missing piece of information finally falls into place.

Did you figure this out by examining the data in the actual file, or did you ask Mike for help?

I'm just messing with the bits in the file, I haven't seen Mike around for a long time...

If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.  I presume the format should be either some version of a planar bitmap.  Most likely a 3-bit one, possibly with an additional mask.

Well, at the moment I'm working on the Lemmings animation data which is 2bpp planar bitmaps with black being transparent (no mask).

Is section #5 (decompressed) large?  If so it's most likely the intro screen (main menu) as a single bitmap.

Thanks!  B)

P.S.  Oh, and don't forget to do it for the Xmas Lemmings, which have different graphics in particular the lemmings animations.

Hmm? I'm numbering the data sections from 0 to 6, so section 5 decompressed is 758 bytes.

The intro screen is composed of a brown background with seperate planar bitmaps for the Lemmings holding signs and the Lemmings logo.

On a similar note, it's interesting that the shadow behind the Lemmings logo and the Lemmings holding signs is dynamic, therefore if you change the logo graphic or the sign graphics, the shadow on the intro screen is created at runtime to match the graphic.

[Mindless]

Well, at the moment I'm working on the Lemmings animation data which is 2bpp planar bitmaps with black being transparent (no mask).

Apparently this only applies to the first two animations, which are the walking animations.  They are 2bbp planar, black as transparent, 16x10 pixels, 8 frames each.

Some of the other animations (mining, basing, exploding?) require more planes. (possibly different dimensions, 8x10?)

[ccexplore]

Hmm? I'm numbering the data sections from 0 to 6, so section 5 decompressed is 758 bytes.

I was just venturing a wild guess for the section you listed as "unknown".  Well, 758 bytes is clearly not the intro graphics.

What happens when you modify the data of that section?  Where the game messes up/crashes/changes should give us a clue about that section

On a similar note, it's interesting that the shadow behind the Lemmings logo and the Lemmings holding signs is dynamic,

Wow, didn't think the programmers/Mike would pull something that fancy (well okay, it's not that hard, but still).  Cool.

[ccexplore]

If possible, I'd love for you to give us the offsets to the various bitmaps you found in main.dat.

Alternatively, since we can calculate, you just need to give for each section the sequence of bitmaps, and their dimensions, how many bpp, and whether there's a mask.  (This is assuming there are no unused or non-graphics bytes between one bitmap and the next.)

Thanks again, actually I'll probably look into this too as soon as I have a chance.

[ccexplore]

Some of the other animations (mining, basing, exploding?) require more planes. (possibly different dimensions, 8x10?)

I would expect them to be larger, not smaller......

One thing I've planned in mind (but will probably take some time to implement, if ever) currently is an app that helps explore this sea of uncharted bits.  I envision it to be like this:  There are 5 columns:  one that interprets the bits as just monochrome bitmaps (in effect, displaying the planes individually), next column interprets them as 2 bpp, the next as 3 bpp, another as 4 bpp, and yet another as 5 bpp (4 bpp+mask, in essense).  You can set the program to start interpreting from a particular offset, and also to set the width.  The program will then go thru a certain number of scanlines (say, 300) assuming that you have a sequence of bitmaps of the specified width.  If you guess right, then you'll get something legible, otherwise you get garbage and should try another width.

This should hopefully help with figuring out how exactly the bitmap are laid out.

[Mindless]

That would be a useful program!

Anyway, as to my comment about the smaller animations, that was because I was missing a frame.

Code: [Select]

main.dat section 0:
 lemming animations -  black = transparent
   [0x0000] walking (r)       8 frames, 16x10 px, 2bpp planar, 40 Bytes/frame
   [0x0140] jumping (r)       1 frame,  16x10 px, 2bbp planar, 40 B/f
   [0x0168] walking (l)       8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x02A8] jumping (l)       1 frame,  16x10 px, 2bpp planar, 40 B/f
   [0x02D0] digging          16 frames, 16x14 px, 3bpp planar, 84 B/f
   [0x0810] climbing (r)      8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0990] climbing (l)      8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0B10] drowning         16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x0D90] post-climb (r)    8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x0F10] post-climb (l)    8 frames, 16x12 px, 2bpp planar, 48 B/f
   [0x1090] brick layer (r)  16 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x1570] brick layer (l)  16 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x1A50] bash (r)         32 frames, 16x10 px, 3bpp planar, 60 B/f
   [0x21D0] bash (l)         32 frames, 16x10 px, 3bpp planar, 60 B/f
   [0x2950] mine (r)         24 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x30A0] mine (l)         24 frames, 16x13 px, 3bpp planar, 78 B/f
   [0x37F0] fall (r)          4 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x3890] fall (l)          4 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x3930] pre-umbrella (r)  4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3AB0] umbrella (r)      4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3C30] pre-umbrella (l)  4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3DB0] umbrella (l)      4 frames, 16x16 px, 3bpp planar, 96 B/f
   [0x3F30] splat(ter?)      16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x41B0] exit(ter?)        8 frames, 16x13 px, 2bpp planar, 52 B/f
   [0x4350] burn(er?)        14 frames, 16x14 px, 4bpp planar, 112 B/f
   [0x4970] blocker          16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4BF0] shrug(ger?) (r)   8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4D30] shrug(ger?) (l)   8 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x4E70] exploder         16 frames, 16x10 px, 2bpp planar, 40 B/f
   [0x50F0] explosion         1 frame,  32x32 px, 3bpp planar, 384 B/f
The jumping animation refers to the action that a Lemming does when it reaches a height of terrian that it can scale w/o being a climber.

[Mindless]

I completeded disecting section 0, and I also wrote a tool something like you mentioned above, if anyone wants it, I'll put it up.

[ccexplore]

I completeded disecting section 0, and I also wrote a tool something like you mentioned above, if anyone wants it, I'll put it up.

Cool!  I don't know if I'll ever do anything with it, but it's cool that you've completed the entire section 0.  And yeah, put up the tool.

[Mindless]

Code: [Select]

main.dat section 1:
 terrain destruction masks
   [0x0000] bash (r)          4 frames, 16x10 px, 1bpp planar, 20 B/f
   [0x0050] bash (l)          4 frames, 16x10 px, 1bpp planar, 20 B/f
   [0x00A0] mine (r)          2 frames, 16x13 px, 1bpp planar, 26 B/f
   [0x00D4] mine (l)          2 frames, 16x13 px, 1bpp planar, 26 B/f
   [0x0108] nuke              1 frames, 16x22 px, 1bpp planar, 44 B/f
 nuke counter
   [0x0134] "9"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x013C] "8"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0144] "7"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x014C] "6"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0154] "5"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x015C] "4"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0164] "3"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x016C] "2"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x0174] "1"               1 frame,  8x8 px,   1bpp planar, 8 B/f
   [0x017C] "0"               1 frame,  8x8 px,   1bpp planar, 8 B/f

Tool: http://it.travisbsd.org/lemmings/_misc/gxplr.zip
Limitations: Many. Loads palette from ground0o.dat.

[ccexplore]

Code: [Select]

main.dat section 1:
 &#A0;terrain destruction masks
 &#A0; &#A0;[0x0000] bash (r) &#A0; &#A0; &#A0; &#A0; &#A0;4 frames, 16x10 px, 1bpp planar, 20 B/f
 &#A0; &#A0;[0x0050] bash (l) &#A0; &#A0; &#A0; &#A0; &#A0;4 frames, 16x10 px, 1bpp planar, 20 B/f
 &#A0; &#A0;[0x00A0] mine (r) &#A0; &#A0; &#A0; &#A0; &#A0;2 frames, 16x13 px, 1bpp planar, 26 B/f
 &#A0; &#A0;[0x00D4] mine (l) &#A0; &#A0; &#A0; &#A0; &#A0;2 frames, 16x13 px, 1bpp planar, 26 B/f
 &#A0; &#A0;[0x0108] nuke &#A0; &#A0; &#A0; &#A0; &#A0; &#A0; &#A0;1 frames, 16x22 px, 1bpp planar, 44 B/f

<drools...>Ooh ooh ooh, I had been wanting this for such a long time.  This is heaven (well, it'll come in handy someday anyway in some unexpected way, I'm sure), thanks!

<- (well, for me, anyhow )

[ccexplore]

By the way, you can probably drop the word "planar" from future listings.  Because of the way the VGA/EGA hardware works, it simply doesn't make sense programming-wise to use anything else if you know you'll work with a specific graphics mode.  So all bitmaps should be planar.

[ccexplore]

Well, here's my attempt at section #2.  Unfortunately, most of it is still indecipherable junk to me, and I also found things that doesn't seem to be used in the game.  Anyhow:

Code: [Select]

main.dat section 2:
 skill number digits
    [0x1900] right "0", 1 frame, 8x8, 1bpp
    [0x1908]  left "0", 1 frame, 8x8, 1bpp
    [0x1910] right "1", 1 frame, 8x8, 1bpp
    [0x1918]  left "1", 1 frame, 8x8, 1bpp
    [0x1920] right "2", 1 frame, 8x8, 1bpp
    [0x1928]  left "2", 1 frame, 8x8, 1bpp
    [0x1930] right "3", 1 frame, 8x8, 1bpp
    [0x1938]  left "3", 1 frame, 8x8, 1bpp
    [0x1940] right "4", 1 frame, 8x8, 1bpp
    [0x1948]  left "4", 1 frame, 8x8, 1bpp
    [0x1950] right "5", 1 frame, 8x8, 1bpp
    [0x1958]  left "5", 1 frame, 8x8, 1bpp
    [0x1960] right "6", 1 frame, 8x8, 1bpp
    [0x1968]  left "6", 1 frame, 8x8, 1bpp
    [0x1970] right "7", 1 frame, 8x8, 1bpp
    [0x1978]  left "7", 1 frame, 8x8, 1bpp
    [0x1980] right "8", 1 frame, 8x8, 1bpp
    [0x1988]  left "8", 1 frame, 8x8, 1bpp
    [0x1990] right "9", 1 frame, 8x8, 1bpp
    [0x1998]  left "9", 1 frame, 8x8, 1bpp
Each digit is actually only 4 pixels wide, so they made two versions of each digit, one with the digit on the left half of the 8x8 box, the other with the digit on the right half of the 8x8 box.  This makes it convenient to construct any 2-digit number within an 8x8 box, like the numbers in the skill toolbar.

Code: [Select]

 mystery 8x16 characters
    [0x19A0] "%", 1 frame, 8x16, 3bpp
    ...
(characters included, in order)
% 0 1 2 3 4 5 6 7 8 9 . A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

These 8x16 characters takes up all the space up to the end of section #2.  The intended palette is clearly not the one from groundXo.dat.  I'm not able to figure out where they are actually used; maybe they aren't.  Note: they are not the green letters you see when playing a level.

[ccexplore]

Only partially figured out parts of sections #3 and #4 so far:

Code: [Select]

main.dat section 3:
 Lemmmings holding signs
   [0x9488]  F1 sign, 1 frame, 120x61, 4 bpp
   [0xA2D4]  F2 sign, 1 frame, 120x61, 4 bpp
   [0xB120]  F3 sign*, 1 frame, 120x61, 4 bpp
   [0xBF6C]  Difficulty Selection sign**, 1 frame, 120x61, 4 bpp
   [0xCDB8]  Exit to DOS sign, 1 frame, 120x61, 4 bpp
   [0xDC04]  F4 sign, 1 frame, 120x61, 4 bpp
*with the music/fx set to blank
**with the difficulty set to FUN
This is the end of section 3, contains the "Lemmings holding signs" graphics you see on the main menu screen.

Code: [Select]

main.dat section 4:
 purple text
    [0x69B0] "!", 1 frame, 16x16, 3bpp
    ...
 (contains the ASCII characters, in order, from '!' to "~")

This is end of section 4, contains all the purple text characters you see throughout the game.

[ccexplore]

I seem to do best with the font graphics in all sections so far:

Code: [Select]

main.dat section 6:
 Green text characters:
 [0x1900]  "%", 1 frame, 8x16, 3bpp
 ...

% 0 1 2 3 4 5 6 7 8 9 - A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
Again, located at end of section.  Notice parallel with end of section 2 (note correction:  my earlier description of end of section 2 listed the character '.' after '9'.  It should be a '-' instead just like here, the '-' is the minute-second separator in the display of time remaining).  These characters are the ones actually used in the game when playing a level.

[Shvegait]

I think I know what the other set of characters is for. Could they be the green text when playing the game in IBM PS2 or "High Performance" mode? (These characters are different from the ones when playing with PC Compatible or AMSTRAD modes).

[ccexplore]

I think I know what the other set of characters is for. Could they be the green text when playing the game in IBM PS2 or "High Performance" mode? (These characters are different from the ones when playing with PC Compatible or AMSTRAD modes).

That sounds quite plausible.  So you got the PS2/High Perf mode to work?  Is it DOSBox or just plain Windows DOS?

[Shvegait]

DOSBox. I don't think they actually "work", but they at least load up to the level screen, which is enough to see that the letters are in a different font. If I recall correctly, one of the modes doesn't have mouse (clicking) support, one of them will freeze between levels, in one of them the level won't even start (but the graphics will load)... Weird stuff happens for sure.

[Mindless]

Code: [Select]

main.dat section 3:
 main screen
   [0x0000] brown background             1 frame, 320x104, 2bpp
   [0x2080] Lemmings logo                1 frame, 632x94,  4bpp

[Mindless]

Code: [Select]

main.dat section 4:
 blinking eyes
   [0x0000] blink1         8 frames, 32x12, 4bpp, 192 B/f
   [0x0600] blink2         8 frames, 32x12, 4bpp, 192 B/f
   [0x0C00] blink3         8 frames, 32x12, 4bpp, 192 B/f
   [0x1200] blink4         8 frames, 32x12, 4bpp, 192 B/f
   [0x1800] blink5         8 frames, 32x12, 4bpp, 192 B/f
   [0x1E00] blink6         8 frames, 32x12, 4bpp, 192 B/f
   [0x2400] blink6         8 frames, 32x12, 4bpp, 192 B/f
 scroller (lemmings and reel)
   [0x2A00]  left Lemming 16 frames, 48x16, 4bpp, 384 B/f
   [0x4200] right Lemming 16 frames, 48x16, 4bpp, 384 B/f
   [0x5A00] reel           1 frame,  16x16, 4bpp, 128 B/f
 difficulty selector sign
   [0x5A80] mayhem sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x5E4C] taxing sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x6218] tricky sign    1 frame,  72x27, 4bpp, 972 B/f
   [0x65E4] fun sign       1 frame,  72x27, 4bpp, 972 B/f

[Mindless]

Code: [Select]

main.dat section 6:
 skill panel graphics
   [0x0000] skill panel  1 frame, 320x40, 4bpp, 6400 B/f
That about raps it up, except for section 5, which still completely perplexes me.

[ccexplore]

Actually, I think we're still missing the graphics for the music options (you know, the F3 one where you select between no music [blank], music [music note], or just sound effects ["FX"]).

And technically, there are still bytes at the start of section 2 where we can't find a purpose for, although that does mean we could care less about it for now.

Maybe section 5 is only used in one of the "High Perf" or whatever machine options.  How thoroughly have you tested that section?  I assume you've identified the general contents of each section by zeroing it out and see what parts of the game are affected.  Have you try this:  zero out section 5, pick some level, and then nuke it?  Just a total random (and probably wrong) hunch, see carefully if anything changes.

[Mindless]

Oh, oops, I forgot about the beginning of section 2...

Section 5 doesn't seem to have any effect on the VGA game, unless it effects the special graphics levels. I've fillled it with both 0x00's and 0x0F's and nothing seems to change, I played the first few levels without noticing anything wrong, and even with the whole main.dat fillled with 0x00's the explosion pixels are still there.

The thing I've noticed is that section 5 is the only one that has a decompressed size which is not divisible by 8, leading me to believe that it is not graphical data.

[Mindless]

Code: [Select]

main.dat section 2:
 skill panel (high performance)
   [0x0000] skill panel       1 frame,  320x40, 4bpp

[ccexplore]

Actually, I think we're still missing the graphics for the music options (you know, the F3 one where you select between no music [blank], music [music note], or just sound effects ["FX"]).

Never mind, that was my fault, I wasn't paying attention to the offsets (or actually, it's really gxplr's fault for the way it handles the +/-).  It turns out that I actually only read the section up to 0xEA50 or so, but gxplr wraps back to 0 at that point because it's still trying to read a 120x61x4bpp bitmap, which in the process causes it to go off the end of section and so it chooses to wrap around to 0.  So I didn't even realize I haven't actually reach the end of section yet.

Anyway, it turns out the music and FX icons are after the main signs, so here we go, the corrected listing for the end of section 3:

Code: [Select]

main.dat section 3:
 Lemmmings holding signs
   [0x9488]  F1 sign, 1 frame, 120x61, 4 bpp
   [0xA2D4]  F2 sign, 1 frame, 120x61, 4 bpp
   [0xB120]  F3 sign*, 1 frame, 120x61, 4 bpp
   [0xBF6C]  Difficulty Selection sign**, 1 frame, 120x61, 4 bpp
   [0xCDB8]  Exit to DOS sign, 1 frame, 120x61, 4 bpp
   [0xDC04]  F4 sign, 1 frame, 120x61, 4 bpp
   [0xEA50]  music note icon for F3 sign, 1 frame, 64x31, 4 bpp
   [0xEE30]  "FX" icon for F3 sign, 1 frame, 64x31, 4 bpp
*with the music/fx set to blank
**with the difficulty set to FUN

This time, I used a calculator and so we have indeed reach the real end of section 3.

[ccexplore]

The thing I've noticed is that section 5 is the only one that has a decompressed size which is not divisible by 8, leading me to believe that it is not graphical data.

Certainly possible.  What I had in mind when I asked about nuking is that instead of storing the explosion's scattered pixels/particles as bitmaps (which hardly makes sense), they could just store a table of (x,y)s and/or velocity vectors to represent the explosion pixels and how they move over time.  But you said you tested explosions, so I guess that isn't it.

Perhaps one way to find out if it is nongraphical data would be to compare the data with the corresponding section in cgamain.dat I guess.  If it's completely unrelated to graphics, then that section should be identical in both files.

[ccexplore]

Perhaps one way to find out if it is nongraphical data would be to compare the data with the corresponding section in cgamain.dat I guess. &#A0;If it's completely unrelated to graphics, then that section should be identical in both files.

Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

Could it have something to do with the copy protection thing?  How does that work in a real, non-cracked version anyway?

[Mindless]

...(or actually, it's really gxplr's fault for the way it handles the +/-). &#A0;It turns out that I actually only read the section up to 0xEA50 or so, but gxplr wraps back to 0 at that point because it's still trying to read a 120x61x4bpp bitmap, which in the process causes it to go off the end of section and so it chooses to wrap around to 0. &#A0;So I didn't even realize I haven't actually reach the end of section yet.

I should have listed that in the limitations. It's to prevent crashes, as you've probably guessed. &#A0;

[Mindless]

Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

Could it have something to do with the copy protection thing?  How does that work in a real, non-cracked version anyway?

http://groups.google.com/groups?q=%22russell.dat%22+lemmings&qt_s=Search

[ccexplore]

I've collected the information on main.dat into one document:

http://www.geocities.com/guestlevels/lemmings/maindat.txt

There are some slightly missing info that I still need to fill in, but everything else is there, including all the tables we posted here for the offsets to the various graphics and animations.

I'm too lazy to manually insert line breaks, so to best view this document, you should save it (right click on the link and choose "Save Target") to your machine and then open it with Notepad or Word.  Make sure to turn word-wrap on in Notepad.

[Mindless]

Intro Screen Palette

Code: [Select]

00 [0x000000] r:0   g:0   b:0   black
01 [0x201008] r:128 g:64  b:32  browns
02 [0x180C08] r:96  g:48  b:32  "
03 [0x0C0004] r:48  g:0   b:16  "

04 [0x08021F] r:32  g:8   b:124 purples
05 [0x100B24] r:64  g:44  b:144 "
06 [0x1A1629] r:104 g:88  b:164 "
07 [0x26232F] r:152 g:140 b:188 "

08 [0x001400] r:0   g:80  b:0   greens
09 [0x001804] r:0   g:96  b:16  "
10 [0x001C08] r:0   g:112 b:32  "
11 [0x002010] r:0   g:128 b:64  "

12 [0x343434] r:208 g:208 b:208 white
13 [0x2C2C00] r:176 g:176 b:0   yellow
14 [0x10142C] r:64  g:80  b:176 blue
15 [0x382024] r:224 g:128 b:144 pink

[ccexplore]

Great!  maindat.txt is now updated with the palette information, so it should have everything someone would need for extracting all the bitmaps out of main.dat.

[ccexplore]

DOSBox. I don't think they actually "work", but they at least load up to the level screen, which is enough to see that the letters are in a different font. If I recall correctly, one of the modes doesn't have mouse (clicking) support, one of them will freeze between levels, in one of them the level won't even start (but the graphics will load)... Weird stuff happens for sure.

Actually, I just tried the "PS/2" mode and it seems to work fine in DOSBox, maybe it's an XP/Win98 difference?

I also ran both the "High Performance" and the "PS/2" mode in plain DOS Windows and see the flashing behavior when you're in a level.  This seems to prove my theory that the game tries to do the palette switching trick, so that separate color palettes can be used in separate areas of the screen.  This also explain why they can afford a different green text font and skills panel for those modes.

Of course, the timing is no longer correct for modern PCs, and it just plain confuses DOSBox.

[ccexplore]

Actually, I just tried the "PS/2" mode and it seems to work fine in DOSBox

Note that "work fine" means doesn't crash.  The colors are indeed a little messed up, but that's to be expected since palette switching tricks are not emulated correctly by DOSBox.

[Mindless]

Ok, I just checked and they are indeed identical.  main.dat section #5 and cgamain.dat section #3 are the same.

I finally solved the mystery.  It contains the PC speaker sounds that play if you don't have enough memory to load ADLIB.DAT (but ADLIB.DAT is loaded before MAIN.DAT, so the game never works in that case).  Interestingly, I can't find any normal way to get the PC speaker sounds to play.  They don't play normally, even if you don't have an AdLib-capable sound card.  I was only able to get them to play by modifying the game to skip AdLib initialization.  (Of course, maybe this is just a bug in the version of the game that I'm using.  Edit: It was.)

Edit 2:  There are 3 different versions of Lemmings for DOS with the VGALEMMI.EXE files dated 1991-05-10, 1991-05-24, and 1991-07-19.  I was using the version dated 1991-05-24, which has a bug that makes PC speaker sound effects impossible.  Incidentally, it also has the bug where the music starts over at track 1 every time you fail a level.

Edit 3: An easy way to tell which version you have is based on the message when you exit the game.  If it says "Have a nice DOS!" then you have the buggy version.

[jkapp76]

I'd love to hear PC-speaker music!