LemSet - Custom graphic set compiler for DOS Lemmings (Sourcecode released)

[namida]

(EDIT: This topic was originally for an extraction tool to convert to Lemmini/jLevelBuilder formats; hence the first few posts are about that. For the download of LemSet, click here.)


Here's a little tool I wrote that'll extract the graphics from DOS Lemmings into Lemmini-friendly format. This might be useful for those who want to create precisely-adjusted levels for DosLemm in jLevelEditor, or those who simply prefer the low-res graphics.

It should be noted that the trigger area images it creates ([style]om_#.gif), while they are accurate to how DOS Lemmings performs and to the images extracted (try copying/pasting them over the image), Lemmini does not seem to like them. You'll probably want to use these ones for displaying trigger areas in jLevelEditor if you're creating levels for DOS, but if your goal is to use the graphics in Lemmini itself or create levels to be played in Lemmini, use Lemmini's existing versions of these files. I included a copy of them (in the "default" folder) just in case you don't have them.

This program will create the INI files. However, as I didn't work out the conversion table for trap sounds, the trap sounds it uses are all hardcoded; so if there are any gaps in the graphic set files you provide (or any diversion from the standard ordering), some traps may end up having the wrong sound. If someone would like to make a list for me of what sound effect ID in DOS = what sound effect ID in Lemmini, I'll be more than happy to work that into the code of the program, but I can't be bothered working it out myself.

And yes - this program can convert graphic set 9 (Xmas Lemmings) to Lemmini format! As far as the trigger area problem bug mentioned above goes, I created a custom trigger area graphic for the set 9 exit that SHOULD work; it's in the Default folder along with all the actual default ones. (The exit is the ONLY object in Set 9 that has a trigger area that does anything; and as we know, Lemmini completely ignores trigger areas with no effect.) You're on your own for converting the Xmas Lemmings levels, though... =P

Also - while the GROUNDxO.DAT files can (and should) be used as-is, the VGAGRx.DAT files must first be extracted using ccexplore's ldecomp or Mindless's Lemmings Tools. VGAGRx.DAT will extract into two sections; the first should be saved as VGAGRx.DAT.0 and the second as VGAGRx.DAT.1. Mindless's tools will automatically suggest these filenames.

Source code can be found here.

[namida]

Just to keep updated, I have successfully written code to extract DAT files. The existing program would be a pain in the ass to add this into (believe me, it is VERY sloppily coded), but I'm working on a new program which is more tidily coded and thus easier to add features to. My plan is for it to support not just extracting graphics to Lemmini, but more or less being a full-featured manager for Lemmings data files (basically, capable of editing as much as possible within the DAT files, apart from levels (since there's already three level editors out there) - I'll probably include basic functionality for levels but not full-blown editing, stuff along the lines of changing skills/title/etc). So you'll (finally) be able to create your own graphic sets. =D

I'm currently working on the code to compress them again. Initially I plan to just use faux-compression (ie: repeatedly using "store exact bytes"), but the algorithm is quite simple so at some point I'll probably develop a proper compression algorithm.

Just a warning: If you're going to start creating now (I wouldn't reccomend it, I'm notorious for not finishing what I start), the most likely format I'd use would be 24-color bitmaps. Not sure how I'll handle transparency, most likely using death magenta (it's worth noting that Lemmings only uses 18-bit color values, so you can use 255,0,255 for transparent and, say, 254,0,254 if you want the actual color - both would come out the same after converting to 18-bit anyway, but they can be detected seperately *before* the conversion). No idea what I'm going to do about objects yet. Of course, if you have any suggestions for what could work well, fire away. Keep in mind that I'm quite averse to using pre-existing libraries to load files, I prefer to write my own code so I know exactly how it works; this means that I'm probably not going to want to work with complex formats and would rather stick to simple ones like BMP.

[namida]

So, at the moment, it's uncompressing all 10 default VGAGR files in a little under 11 seconds. Not bad, considering how inefficient my code tends to be. xD

Working on the compression algorithm now. Initially aiming for a "just store the damn thing in a way that will decompress" (using 00's or 111's, for thsoe who know the format), but before I go on to write other code I want to write an actual proper compression algorithm.

At this point, it seems to generate the bitstream perfectly, but there's a glitch somewhere in the transferring it to bytes...

EDIT: Bug located, and after close examining, it's actually nowhere near as severe as I thought! The encoding is fine, the transferring to bytes is fine - the problem is it just cuts off what it writes to file a byte or two early. This should not be hard to fix.

... Oh god. The problem was that I forgot to include the header when calculating the compressed data size. Stupid mistake really, but hey, easily fixed once I worked it out.


EDIT: Success! I wrote a little snippet to test my various routines - basically, the code instructed it to "decompress all VGAGR files, recompress the extracted data into new files, then decompress those new files". (The last part simply because, if I know my decompression works - which it does, I can tell by comparing it to the output from ccexplore's or Mindless's tool - the easiest way to test if my compression works is to attempt to decompress it again.) The output from the first decompression is identical to that from the second, both of which are also identical to the output from ccexplore's/Mindless's tools. =D AWESOME. Next job, compression algorithm to replace the faux-compression currently used. I already have a very good idea of how I'm going to write it.


EDIT AGAIN: Success! It now compresses the files. My algorithm usually results in files about 200 or so bytes larger than the original files - but one major exception is VGAGR5.DAT, where it beats the original by nearly 2KB. Is that file known for being inefficiently compressed or anything?

[namida]

As posted over in the Lemmings Discussion forum, I've had success to some degree! Now, a lot of this is hardcoded, it required a bit of manual hex editing, and currently my compression algorithm is not very much liked by LemEdit or CustLemm (however, Mindless's decompression tool has no problem with it - didn't test with Lemmix yet) - so I had to output uncompressed data then manually compress it with Mindless's tool. So, this is mostly a proof-of-concept at this stage, but I'm working on an actual tool that can easily be used.

Nonetheless, I present to you the first ever DOS-compatible custom graphic set for Lemmings. =D

Works as-is with LemEdit and any DOS version of lemmings (and SHOULD work as-is with Lemmix too); if you want to use jLevelBuilder to mess around with it, use the extractor tool in the first post.

EDIT: For those who can't (or don't want to) download it and run it themself, but want to see it in action instead of just a screenshot, here: http://www.youtube.com/watch?v=dQ8U-6AgnKE

[LJLPM]

Good job! 

[namida]

Thanks!
Next goal is to ditch all the hardcoded data and make it actually read from a file (or take user-entered values). I'll probably work on fixing up the compression algorithm after that; it's not as major IMO since we already have a program that can take care of that.

[namida]

Some interesting finds so far:

1) ccexplore's documentation mentions that the default styles, terrain pieces only ever use the upper 8 pallete entries. I can confirm that they are NOT restricted to this - they most certianly can use the lower 8 too. Likewise, the mask can be completely seperate from the bitmap data, even though the default styles all use one of the bitmap planes as a mask.

2) Those who are familiar with Lemmings' mechanics probably expected this - but there are two types of trigger areas that aren't documented, specifically, "turn lemmings left" and "turn lemmings right". These are almost certianly the same effects used by the game when you place a blocker; but you can attach those effects to objects too! (Note: I haven't yet tested how well Lemmix handles this, only tested in the actual DOS game. At any rate though, it wouldn't be a major modification to Lemmix to add support if it's not already there.) You can also make an object place a steel area; this could be useful if you want larger steel areas than the level format inherently handles (or a larger quantity of them), though it's the first two that I'm expecting might see some really creative usage.

3) ccexplore's documentation mentions that the second bit of animation_flags seems to do something. From what I can tell, it simply tells if the object animates at all (kind of redundant when you can just work it out from the frame data, but that's the only thing I can come up with).



I'm wondering - is there any known information about the CGA version? And if not, could anyone provide me with a copy of the CGA version so I can experiment? Wouldn't mind adding support for that too. (My program already handles EGA, not that that's a huge task.)

[Simon]

Very nice!

After you describe it, it sounds logical to have special objects with directional funneling capability, but I've still did not expect them.

-- Simon

[ccexplore]

2) Those who are familiar with Lemmings' mechanics probably expected this - but there are two types of trigger areas that aren't documented, specifically, "turn lemmings left" and "turn lemmings right". These are almost certianly the same effects used by the game when you place a blocker; but you can attach those effects to objects too!

Yep, that's exactly how blockers work.  I hope Lemmix does it that way too, though there's no way to tell for sure without testing, and given that it already handles certain aspects of trigger areas incorrectly (eg. 0-valued triggers not canceling steel areas), it's a toss up as to what the actual result is.

In fact in addtion to "turn left" and "turn right" there is also a (admittedly far less useful) third value, 10 I think, which is used to represent the "neutral" central column in the blocker's field.  It has no effect on lemmings besides that its presence would prevent blocker assignments nearby, because the game checks for and disallows blocker assignments that would overlap with existing blocker fields (any "turn left", "turn right" as well as "neutral").

I'm wondering - is there any known information about the CGA version? And if not, could anyone provide me with a copy of the CGA version so I can experiment? Wouldn't mind adding support for that too. (My program already handles EGA, not that that's a huge task.)

The "alternate version" download from abandonia.com should have all versions of EXEs: vgalemmi.exe, cgalemmi.exe and tgalemmi.exe. 

IIRC the CGA hardware only supports one color graphics mode which is 320x200 with basically 4 fixed set of colors (ie. 2 bit per pixel).  The layout of pixels in the graphics files (after decompression) will most probably mirror the video memory layout, so with a little Internet research and some experimentation you should be able to work it out.

[namida]

Awesome. Well, first thing is to get this completely working for DOS. First version will support the full features of graphic sets, but will require the use of an external tool to compress the output (it won't require an external tool to decompress, only to compress - but no GIF support yet). It'll generate the sections for you. After that, I'll probably work on fixing up the compression algorithm - are you aware of any oddities that might be causing Lemmings / LemEdit to reject them, yet your tool, Mindless's tool and my own (all of which also decode the official files perfectly) can decompress them no sweat? Upper limits on the offsets or something that are lower than the file format allows for?

In terms of coding, I've just coded loading and saving INI files. There's actually only 10 values per object that need to be defined (and 6 of these are to do with trigger areas, while a couple of others are set to fixed values for windows, so you generally don't need 10 for each object), so it should be a quite simple format. That aside, the INI file can also be used to define the pallete - alternatively, you can use a 16x1 pallete.bmp file with one pixel for each color. Any color that isn't in the pallete becomes transparent. (It converts to 18-bit before matching so if, say, you have 252,252,252 in your pallete and use 255,255,255 in your image, it will count that as a match.)

(I'm aware scanning the images and building a pallete based on what's used would be easiest, but that's a feature I can add later. For now, I just want to get out something that works.)

[ccexplore]

are you aware of any oddities that might be causing Lemmings / LemEdit to reject them, yet your tool, Mindless's tool and my own (all of which also decode the official files perfectly) can decompress them no sweat? Upper limits on the offsets or something that are lower than the file format allows for?

Mostly "no".  I vaguely recall having observed cases of graphics issues in DOS Lemmings (but nothing that causes outright rejection) if the compression on VGAGrX isn't good enough, but I don't have any concrete details to share around this.  Decompressed size shouldn't matter from what I recall seeing in the code.  I suggest not worrying too much about it for now, just get something that works and see if anyone reports any problems.

[namida]

Well, LemEdit gives an error message, while the games just black screen...
Anyway, I've noticed a way in which my algorithm could be MAJORLY improved, so I'll probably work on both of those things at the same time. But for now... yeah. Focus is on getting it to output non-hardcoded-value-based GROUNDxO.dat files. Which isn't a hard task, it's just a task I haven't completed yet. xD

[ccexplore]

Well, LemEdit gives an error message, while the games just black screen...

What happens if you try to use the assets in Lemmix (either level editor or player)?  That may provide some insights as well.  It doesn't sound to me like the issue has anything to do with compression or decompression, especially if the data decompresses identically across multiple tools.  I strongly suspect you have bad data in your groundXo file.  If the problems were with the graphics files (ie. vgagrX.dat), I would expect at worst corrupted graphics being displayed but shouldn't result in error messages or black screens.

[namida]

Not possible. As I said (or at least thought, maybe I forgot to write it), if I take the output VGAGR file from my app, decompress it with Mindless's tools, then recompress it again with the same, it works perfectly - hence the download.

I tried with the Lemmix player. No problems, it loads perfectly.

The only thing I can think of that may help is that LemEdit gives an error screen with some information. I can't work out exactly what the information's referring to, but maybe you can?

[Simon]

Lix displays tileset and level properly. The L1 file reading code was written according to the same documentation as yours.

Johannes, the Droidlings author, made some L1 tilesets himself and published them (link to that topic). Those tilesets were broken in Lix, but ran fine in Lemmix. They have never been tested in Custlemm or Lemmings 1 itself.

Therefore, even though Lix loads your set nicely, it's not a seal of guarantee for matching the L1 file type definitions. There's even a slight chance for the descriptions to have subtle errors or to leave out specifications of irregular forbidden cases. But most likely, Lix and Lemedit misinterpret, next likely is that Lemmix misinterprets, only then that the description has errors. Custlemm is a cracked L1 executable and should be 100 % accurate.

I can't do much with the Lemedit error, it looks like debugging info for people with its source.

-- Simon