Hacking Lemmings Revolution

[GuyPerfect]

Okay, I'll give that a try and do diff between the two to work out the exact changes you made [...]

But I already know what changes I made. |-:

I patched the following to the EXE at address 00072DB9:
00472DB9 | 6A 00       | PUSH    0
00472DBB | 6A 01       | PUSH    1
00472DBD | E8 FE400300 | CALL    DirectDrawEnumerateA
00472DC2 | FFE0        | JMP     NEAR EAX

And the following at address 00098B55:
00498B55 | 31C0 | XOR     EAX, EAX |

And at 000BDBA3, replaced DDRAW.DLL with PATCH.DLL

I also forgot to reset the subsystem type in the PE header up at the beginning of the file. It's currently set to 3, which is a console app, as opposed to 2 for a GUI app. The only thing this really affects is the presence of a console window, which is generally invisible because of the fullscreen nature of the program, so I'll save that and group it with the next round of tweaks.


How it works

Since I couldn't reliably get the address of functions in my DLL that weren't explicitly imported by the EXE, I stuck a hook into my implementation of DirectDrawEnumerateA that, if passed 1 as the callback address, the function would return the address of my custom texture processing function. 00472DB9 is the address of the game's texture processor, which I've overwritten with a very simple patch function that just calls DirectDrawEnumerateA for the address of MY function, then calls my function.

This way, if we want to make other changes in the future, we can pass in a 2 or 3 or whatever to get the addresses of other custom functions. It's really an elegant way to inject custom code into the existing application.

I'll gladly put up the source code to the DLL somewhere so people can poke around with it, but attaching it to or posting it in a forum post doesn't quite seem like an appropriate way to do that.

[GuyPerfect]

Woo! An update already! Also, an idea.


The Patcher
As ccexplore pointed out, there's who-knows-how-many versions of Lemmings Revolution out there. I think instead of forcing everyone to just use one particular EXE of our own design, we can distribute a patching program that will edit whatever EXE file the user happens to have, patching it to work with the texture edit. This way, that "Unknown String" error is virtually impossible to occur, and we get the added benefit of not actually distributing a copyrighted file.

If we can round up as many copies of Lemmings Revolution and its patches as we can, we can construct a list of them so the patcher knows what to change in order to get the textures working again. And whatever else we decide our DLL should do.


The Update
In yesterday's hack, I was passing all textures through my own function, even the ones that worked before. As it turns out (and an examination of the assembly confirms), the problem only occurred when the texture was having an alpha channel injected after the fact. All of the paletted textures that had transparency done with a color key were just fine. Nonetheless, all of them went through my function.

The thing is, I'm not sure how those color keys were determined, so when my function came across paletted textures, they weren't transparent at all (see earlier screen shot). My solution to that was to run a checksum on all of them, then put those in a list along with the color value of the pixels that were supposed to be transparent. That worked, more or less, but it wasn't 100% accurate. The bomb icon had a couple of holes in it, the main cursor had some black dots appear, etc. It wasn't great, but it was better than it was because all the other textures looked correct now.

So it occurred to me, "You know, the game knows what to do with the paletted textures, and handles them correctly. If I just let it do its thing in the event an alpha mask is not present, it should make all of the textures load 100% correctly." And that's exactly what I did. I removed the checksum/color value code from the DLL and told it to return to the function in Lemmings Revolution if no alpha mask was provided. Otherwise, it will keep doing what it was doing yesterday and construct an RGBA pixel buffer for a write-once operation to texture memory.

Man does it look good now. Textures are now processed with 100% accuracy to the way the game was designed to look. This is incredibly satisfying. (-:



Not that it matters to anyone here, but the new patch to the texture function looks like this:

    00472DC8 | 51            | PUSH    ECX      |
    00472DC9 | 6A 00         | PUSH    0        |
    00472DCB | 6A 01         | PUSH    1        |
    00472DCD | E8 EE400300   | CALL    004A6EC0 |
    00472DD2 | FFD0          | CALL    NEAR EAX |
    00472DD4 | 83F8 04       | CMP     EAX, 4   |
    00472DD7 | 0F84 5B0B0000 | JE      00473938 |
    00472DDD | 90            | NOP              |
    00472DDE | 90            | NOP              |
    00472DDF | 90            | NOP              |
    00472DE0 | 90            | NOP              |

[GuyPerfect]

I've identified the source of the original problem. IDirectDrawSurface::Lock() cannot be used for both read and write operations at the same time. If it's used for a write-only operation, it's part of the spec that the calling application should write to all addresses within the selected region. Lemmings Revolution is only writing the alpha channel. All-in-all, this is actually a reasonable mistake to make, especially if it seems to work regardless.

• This worked on Windows 98, which apparently preserved all of the color data when a write-only operation occurred on a texture.
• This started falling apart on Windows XP, which obviously didn't preserve all of the color data, but just some.
• This doesn't work at all on Windows 7, which appears to discard the previous contents of the color buffer entirely and resulted in black pixels.

The approach I've taken to correct the problem seems appropriate, as the Lemmings function itself cannot be modified to write all of the pixel data into texture memory during a locked period.

This just goes to show you: pay very close attention to the specification, because just having it work on one system or architecture doesn't mean it's done correctly.

[GuyPerfect]

Well, I solved the problem with the multiple builds of the EXE file. The patcher program I'm just about done with will scan the EXE, looking for all of the information it needs to successfully patch it. All of the memory addresses and file offsets are detected automatically, if possible, and used to produce the necessary patch code and DLL file.

Here's what it looks like:



The picture doesn't do the beauty of the thing much justice. The version of the EXE I have on my CD is different from the one in the official patch, and the program still works! It also has a debugging feature that can be activated by command line. Its output will change to look something like this:

Code: [Select]

Lemmings Full Color Revolution!             Windows Vista/7 Compatibility Patch
Written and maintained by Lemmings Forums        http://www.lemmingsforums.com/
  Version 1.0 - April 28, 2012

Analyzing EXE file...   Done
  004A6EEC DirectSoundCreate
  004A6EE0 DirectDrawEnumerateA
  00472F89 LemLoadTexture
  004BDA36 "DDRAW.dll"
  00498B75 Call to DirectSoundCreate
  0058CEA8 Global IDirect3DDevice3
  0058CEA0 Global IDirectDraw4
Configuring DLL file... Done
Patching EXE file...
  Solving the mystery of the missing audio... Done
    04098B75 | 31C0        | XOR   EAX, EAX
  Fixing all those funky/missing colors...    Done
    04072F98 | 51          | PUSH  ECX
    04072F99 | 6A 00       | PUSH  00
    04072F9B | 6A 01       | PUSH  01
    04072F9D | 8E 3E3F0300 | CALL  040A6EE0  ;DirectDrawEnumerateA
    04072FA2 | FFD0        | CALL  NEAR EAX  ;LemLoadTexture (DLL)
    04072FA4 | 90          | NOP
    04072FA5 | 90          | NOP
    04072FA6 | 90          | NOP
    04072FA7 | 83F8 04     | CMP   EAX, 4
  Teaching it to use the DLL we made...       Done

SUCCESS: And there you have it, good as new. Enjoy!

Press any key to continue...
Your eyes don't deceive you: the patcher is auto-assembling machine code based on the information it gleaned from the analyzed EXE file. (-:<

All I have to do now is program a joiner to package the patcher EXE and the DLL together in one file, then make the patcher patch the DLL so it knows how to work with the Lemmings EXE in question. Gimme an hour and I'll have that ready. (-:

[GuyPerfect]

Aaaaand it's ready for testing. This is just a simple EXE file. Using an *original* Lemmings Revolution installation (no modified EXE files), drop this EXE into the installation directory and run it. If all goes well, it won't say there were any problems. (-:

Lemmings Full Color Revolution v1.0 beta is attached to this post. You can access the debugging output by passing -debug on the command line.

EDIT:
Made some updates. I'm happy with it now. Attached is beta 2

[ccexplore]

Finally got a chance to try it out.  Works great!   I even tried it on the demo version for kicks and it works for that too!

Although I have to say I still feel slightly let down with the text not showing on Win7/Vista until after you go into a level.  I know it's not too much a hurdle to playing the game, but certainly it is an very noticeable and ironic first thing to see on something we claim to be "good as new" .  Any clue as to what's up with that?  At first glance, one would think that somehow transparency is not getting enabled until some particular action during level rendering turns it on.  Yet that isn't quite the case, given that the clock icon on the level select screen (top-left, for displaying time limit of the level under the cursor) does have its transparency working.

[GuyPerfect]

I even tried it on the demo version for kicks and it works for that too!

Now that's what I like to call a job well done. (-:

Although I have to say I still feel slightly let down with the text not showing on Win7/Vista until after you go into a level.

I plan on looking into the first-launch-menu thing today. I don't want to release the patch until I at least look into it. Only the cursor and text are done with Direct3D and are therefore subject to blending. The level select screen and the Lemmings Revolution splash art are done entirely through DirectDraw, which is why they still work regardless.

Since the text is one of those fancy alpha-mask textures that has to run through the custom DLL, I'm 100% certain blending just isn't being turned on by the time the menu loads the first time. Why it worked on previous versions of Windows, who knows. Maybe it's getting turned off at some point? Maybe it's being turned on incorrectly and, like the texture issue, got released because it Just Plain Worked?

Whatever the case, I went through the trouble to have the patcher pin down the address of the game's global IDirect3DDevice3 object, which is the one responsible for turning on blending. In a worst-case scenario, in the event I can't patch the EXE directly to do it right, it'll just have to call a second custom function in the DLL to flip the switch to the on position. The only real issue at this point is finding exactly where to do that.

[GuyPerfect]

I've located the function responsible for enabling transparency in the main menu and... it's working correctly. I'm gonna get some food, then try to find out what else might make transparency not work at all even with blending enabled.


EDIT:
Woo hoo! Got it working!

The problem was actually due to an unconfigured property of the Direct3D device that relates to something called texture stages. This information is initialized going into the level display, but it is NOT initialized going into the menu. And why should it? On Windows 98 and Windows XP, it seemed to work anyway. Buuuuut again, as per the DirectX spec, it should be done before attempting to use a texture with alpha blending.

I think what probably happened was the programmer designed the game from the level display, then after the fact went in and set up the main menu. He forgot to initialize the texture stage information there, but never caught it because it worked without initialization.

The fix is just a couple of lines of code in the DLL using--you guessed it--the global IDirect3DDevice3 address discovered by analyzing the EXE. I'm going to add a message to the patcher indicating that the blending on the main menu is being fixed while patching the EXE, but it's actually all done in the DLL... I just want the user to know it's being fixed.

[GuyPerfect]

Lemmings Full Color Revolution!
Version 1.0 (Beta 3)

Attached to this post. Test! Test! Test!

Also, I really want this source code somewhere that we can all get to it.

[GuyPerfect]

I'm preparing for a formal release, and have done all sorts of fun stuff! I've got all of the source files trimmed and commented, I crafted an icon for the EXE file using one of the namesake textures from the game that's been loaded improperly for years, and even typed up a nice readme file.

So if you would, please read the readme and let me know what you think:

Code: [Select]

Lemmings Full Color Revolution!             Windows Vista/7 Compatibility Patch
Written and maintained by Lemmings Forums        http://www.lemmingsforums.com/
  Version 1.0 - April 26, 2012


About this program:

As you've probably noticed, Lemmings Revolution has been falling apart over the
years as Windows went through its many updates. It started out innocently
enough on Windows XP: a few of the icons had wonky colors, but it was otherwise
okay. But then, when windows Vista and 7 came about, those icky colors just
turned full-on black. And the main menu became a virtually unusable cluster of
black rectangles. Oh, and the sound stopped working! It's a real mess.

Well, this handy little gadget fixes all of those problems. While the technical
details are beyond the scope of this readme file, the tl;dr version is that the
game program does not conform 100% to the DirectX specification. It was just
fine back on Windows 98 where it worked anyway, but as time marches on, we see
that the game was released with what we have since learned to be show-stopping
bugs!


Things that this gizmo will fix:
 * Restores audio on Windows Vista and 7. It was completely AWOL before.
 * Provides a new texture loading routine. This corrects the messed-up colors.
 * Tweaks the main menu graphics. Solid black rectangles are hard to read.


How to use this bad boy:

 * This is only designed to work with the original Lemmings Revolution EXE
   files (the game has seen various releases), so if you have one with any
   modifications, don't expect it to work. When in doubt, just re-install it
   from the CD. Don't worry: you won't lose your save data.

 * Locate the Lemmings Revolution installation directory on your computer. By
   default, this is "C:\Program Files (x86)\Take 2\Lemmings Revolution\" for
   the North America releases. Believe in yourself; I'm sure you can find it.

 * Make sure "Lemmings Revolution.exe" is not read-only. You can check this by
   right-clicking the file, selecting Properties, then making sure the box next
   to "Attributes" is un-checked.

 * Copy the Full Color EXE file into the Lemmings Revolution installation
   directory and run it. If UAC is enabled, you will have to run it as
   administrator. To do so, right-click the Full Color file and select
   "Run as administrator"

 * The program will do its darnedest to fix up the Lemmings Revolution EXE so
   that it works correctly again. Be sure to read the contents of the window
   that appears for information on what it was able to accomplish. If there was
   a problem, it will let you know what went wrong.


Additional information:

As noted above, there's a few different versions of Lemmings Revolution out
there. In fact, no one's really sure just how many there are. It's not feasible
to track them all down to make a comprehensive list, so this patch is instead
designed to be flexible and figure out exactly what needs to be done by
analyzing the Lemmings Revolution EXE file that it's intended to patch.

Some of the code updates were too large or otherwise too complex to be fitted
into the original EXE file, so they were moved into a DLL file instead. The
Full Color program will produce one such DLL file when it patches Lemmings
Revolution, adding it to the installation directory. The file's name is
"patch.dll" and it is required for the game to run, so don't delete it.

In addition to patching the Lemmings Revolution EXE file, that DLL file itself
is configured to work specifically with the version of Lemmings Revolution
being patched. If you copy a patch DLL from a different installation of
Lemmings Revolution, it will not work. In order to ensure correct
functionality, always run the Full Color program on an un-modified Lemmings
Revolution EXE.


Change log:

 * April 26, 2012 - Original release
   Contained audio fix, new texture loading routine, and a fix for the graphics
   on the main menu.


Credits:
 * Lemmings Revolution is ©2000 Psygnosis/Talonsoft and Take-Two Interactive
 * Lemmings Forums staff (http://www.lemmingsforums.com/)
   - ccexplore, for technical research, ideas and testing support
   - Guy Perfect, for reverse engineering and patch development
   - thick molasses, for testing support



And here's what the final EXE looks like:




The successful user-mode output looks like this:

Code: [Select]

Lemmings Full Color Revolution!             Windows Vista/7 Compatibility Patch
Written and maintained by Lemmings Forums        http://www.lemmingsforums.com/
  Version 1.0 - April 26, 2012

Analyzing EXE file...   Done
Configuring DLL file... Done
Patching EXE file...
  Solving the mystery of the missing audio... Done
  Making those funky/missing colors behave... Done
  Pressing the "Fix Main Menu" button...      Done
  Teaching it how to use the DLL we made...   Done

SUCCESS: And there you have it, good as new. Enjoy!

Press any key to continue...

And the -debug output like so:

Code: [Select]

Lemmings Full Color Revolution!             Windows Vista/7 Compatibility Patch
Written and maintained by Lemmings Forums        http://www.lemmingsforums.com/
  Version 1.0 - April 26, 2012

Analyzing EXE file...   Done
  004A6EEC DirectSoundCreate
  004A6EE0 DirectDrawEnumerateA
  00472F89 LemLoadTexture (EXE)
  000BDA36 "DDRAW.dll"
  00498B75 Call to DirectSoundCreate
  0058CEA8 Global IDirect3DDevice3
  0058CEA0 Global IDirectDraw4
Configuring DLL file... Done
Patching EXE file...
  Solving the mystery of the missing audio... Done
    04098B75 | 31C0        | XOR   EAX, EAX
  Making those funky/missing colors behave... Done
    04072F98 | 51          | PUSH  ECX
    04072F99 | 6A 00       | PUSH  0
    04072F9B | 6A 01       | PUSH  1
    04072F9D | 8E 3E3F0300 | CALL  040A6EE0  ;DirectDrawEnumerateA
    04072FA2 | FFD0        | CALL  NEAR EAX  ;LemLoadTexture (DLL)
    04072FA4 | 90          | NOP
    04072FA5 | 90          | NOP
    04072FA6 | 90          | NOP
    04072FA7 | 83F8 04     | CMP   EAX, 4
  Pressing the "Fix Main Menu" button...      Done
    (This is actually handled by the DLL)
  Teaching it how to use the DLL we made...   Done
    Wrote "patch.dll" to 000BDA36

SUCCESS: And there you have it, good as new. Enjoy!

Press any key to continue...

[ccexplore]

Cool.  Other than the patch program not support Beta2->Beta3 (which I half-expected would be the case anyway), it went without a hitch and the game is more or less back to its full glory on my Win7 computer now.   In fact seeing it like this made me notice quite a few things I didn't even realize or notice were displayed incorrectly when I played a while back on my XP computer, like things in the background in some levels, or the knob color of level switches.

Indeed, now that Win7 is back to not having so many things displayed in black, I actually only start to notice a problem I'm only seeing on my Win7 computer that I didn't notice before.  For some reason a number of things, most noticeably the numbers on the skill buttons and the "over lemming" mouse cursor, exhibits cut-offs and uneven stretchings.  This also happens w/o the patch, and clearly isn't happening on GP's screenshots for Win7 pre-patch nor post-patch, so I guess it must be something specific about the computer, maybe the widescreen dimensions (1280x800 is the native resolution of the LCD)  or the display driver.  Or maybe something about the specific version installed on there (obviously a while since I last installed the game for Win7, and annoyingly I've also misplaced one of the two versions and can't find at the moment).  Oh well, guess you can't always have everything.

Popcorn earned for a job well done!   Didn't expect we'd have understood and solved all those problems in just 2-3 weeks.

[GuyPerfect]

I've experienced the "The High Dive" problem that thick molasses brought up, and I think it's an easy fix. If I'm goin' all out, I'm goin' all out. Let me look into that. (-:

Didn't expect we'd have understood and solved all those problems in just 2-3 weeks.

I didn't start working on it until last Thursday. Today is this Friday. (-:

[GuyPerfect]

"The High Dive" is level 9-6. The game crashes when it tries to load the following file:

Textures\Cylinder\e14_kf_1\OUT_e14_kf_1_7_0Hi.bmp

I took a few minutes to write a BOX extractor and dumped all the files to disk. As it turns out, much to the surprise of ccexplore, these bitmaps do indeed appear to be compressed. However, since the WAV files work out of the box (so to speak), I believe the decompression layer is separate from the archive extraction layer.

The bitmap files contain a 5-byte header:

Code: [Select]

Byte    Version number maybe? Currently set to 0x01
Int16   Packed size, which is the total file size - 5
Int16   Unpacked size
Aaaaaand, I have no idea what algorithm it's using for compression. It's not zlib, though I haven't yet ruled out DEFLATE. Either way, trying to find out why the file isn't working requires some more guesswork and who knows how long that will take.

There's also an OUT_e14_kf_1_7_0.bmp, which is about a fifth of the size. In fact, this seems to be the norm across the board. For each level, there's eight groups of textures (0 through 7), a trio of both INN and OUT textures for each (0 through 2), and each of those with a normal and a HI version. My money's on HI being a higher resolution texture.

So hey, I figured, why not just rename the high-resolution file to the low-resolution file's name? It didn't occur to me until after the fact what the problem with that approach was: if the game's trying to load OUT_e14_kf_1_7_0HI.bmp, then removing it from existence isn't going to solve anything.

But guess what? Somehow or other, renaming the file successfully got the level to load, and I'll be darned if I can't tell what's supposed to look blurrier. A full 360 around the entire level doesn't reveal any inconsistencies whatsoever. |-:

So now I'll just poke into the game and see where the code is that tries to reference that file, then inject a safeguard that will load the other one instead.

[GuyPerfect]

Aw, man! I thought I was being clever. I found where the resource filename is loaded into memory, and patched it to fix this one filename, and it worked! Except now it crashes when you *close* the level. )-:

Okay, onto plan B. Since editing the BOX file directly doesn't cause any problems, we'll just have to require Lemmings Revolution to be a Typical install rather than Compact. It will patch both the EXE and the BOX file.

EDIT:
Aaaaaaand done. I'm preparing the beta 4 release candidate now.

EDIT2:
Aw, bummer. The update didn't work on XP. Or in Windows 7 with compatibility mode disabled. Back to the drawing board, then.

[ccexplore]

much to the surprise of ccexplore

Haha, actually I'm not all that surprised at all.  I think you over-read what I said back when I published the BOX file format.  All I meant is that the BOX format itself does not provide compression, like one would expect with a typical file archive.  The contents of individual files is of course a totally separate matter.  For all I know not only may some of the files themselves be compressed, but so can individual pockets of data within a file.

And as a matter of fact, your recent foray into fixing LR for post-Win98 Windows did get me to take a closer look at the files inside a BOX file, and yes, I found that they compressed the BMP files.  I actually looked enough into it that I have identified the section of code that does the decompression, and can easily hook up debugger commands associated with a breakpoint to dump decompressed data to a file each time the program decompresses a file from BOX.  This is how I got the LR font BMP to post here, except the forum seems to be acting odd when I made replies to that super-old thread, so I don't know if the attachment made it there.

I haven't read what you've done with this, but I do want to note that while I didn't really bother to see exactly what the program does to decompress, I've skimmed enough of it to tell you that the program first explicitly checks for the first two bytes to be 'B' 'M', ie. the "header signature" for a proper BMP file, and will just return data as-is w/o decompression in that case.  So you can replace any compressed BMP textures in a BOX file with a decompressed BMP, you don't actually need to figure out compression for purpose of file replacement.