Author Topic: 3D Lemmings file formats  (Read 30785 times)

0 Members and 1 Guest are viewing this topic.

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #30 on: June 05, 2019, 11:25:27 PM »
Here's some interesting finds I came across while investigating the possible values for the Signs field in the PC version.

Yes, in the PC version. Keep that in mind as you look at the third one.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #31 on: June 05, 2019, 11:51:29 PM »
Here are the possible values for some of the style-related fields. Note that some of this needs further investigation, especially interactive objects; and "Walls" seems to be a very inaccurate name (but is what the game calls it). Try modifying it on Fun 20. "Signs" is also somewhat misleading, they're more like decorative elements drawn onto blocks (or sometimes in open space).

All values here are decimal

=== TEXTURESETS ===
00 - Pyramid
01 - Castle (???)
02 - Spaceship
03 - Circus
04 - Lemgo
05 - Maze
06 - Army
07 - Confectionary
08 - Computer
09 - Golf
10 - Pyramid 2
11 - Castle 2
12 - Spaceship 2 (/ Lemlab?)
13 - Circus 2
15 - Maze 2
16 - Army 2 (Snowy)
18 - Computer 2
19 - Combination of Pyramid and Golf?
21 - Castle 3 (???)
22 - Space 3 (Dot to Dot)
28 - Computer 3 (Breakout?)
36 - Army 3 (Corkscrew Digger)
38 - Computer 4 (Lemmtris?)
46 - Golf 2 (Land Ahoy)
48 - Computer 5 (Lemmtris?)
95 - Maze 3

=== OBJECTS ===
00 - Pyramid
01 - Castle
02 - Maze
03 - Circus
04 - Lemgo
05 - Maze
06 - Army
07 - Candy
08 - Maze 2
09 - Golf
10 - Pyramid 2
15 - Maze 3
16 - Army 2 (Snowy)
18 - Computer
38 - Pacman Ghosts
95 - Warning Signs

=== SIGNS ===
00 - Pyramid Splitter, Credit Signs ("One Giant Step", "Oh No", "Lemmings 3D") - front only
01 - Castle
02 - Credit Signs ("Programming By" x2, "Graphics By", "Raytracing By") - front and back
03 - Credit Signs ("Produced By", "Eclipse", "Sony", "Playstation") - front and back
04 - "Paradise This Way" replaces "Keep Off Grass", Credit Signs ("Hi Ho", Waterski, "Oh No" (dup), Juggle) - front and back
05 - Maze
06 - Army
07 - Candy
08 - Duplicate of 04
09 - Castle 2, also has a LemLab-looking splitter entrance?
11 - Castle 3, mostly identical to 09 but "Keep Off The Grass" isn't replaced and has Castle splitter instead of LemLab-y
95 - Maze 2

=== LAND ===
00 - Dirt / sand
01 - Grass
02 - Black diagonal tiles
03 - Red and yellow triangle tiles
04 - Lemgo Red
07 - Chocolate
08 - Circuit board
09 - Grass 2
15 - Grass 3
17 - Chocolate 2
18 - Dirt / sand 2
28 - Blue
38 - Solid blue

=== SEA ===
00 - Quicksand
01 - Dark blue water
02 - Normal blue water
03 - Green slime
04 - Lemgo water - blue
05 - Lemgo water - red
06 - Icy water
07 - Liquid chocolate

=== ANIMOBJECTS ===
00 - Egyptian Torches (*)
01 - "Clockwork Games" Flags
02 - Flashing Lightposts (*)
03 - Rings Of Fire
04 - Lemming Flags
05 - Lemming Flags
06 - Checkered Flags
07 - Candycane Torches (*)
08 - Neon Exit Sign
09 - "Exit" Flags
11 - Pillar Torches (*)
13 - Juggling Lemmings
22 - Rocket Flame
23 - "Clockwork Games" Flags (duplicate)
90 - "Block Here"
91 - "Turn Here"
92 - "Bomb Here"
93 - "Build Here"
94 - "Build Here" (duplicate)
95 - "Bash Here"
96 - "Mine Here"
97 - "Dig Here"
98 - "Climb Here"
99 - "Float Here"

Those marked with (*) are duplicated, with the second copy being upside down.

=== INTERACTIVE OBJECTS ===
00 - Flame blower
01 - Squasher
02 - Bear trap
03 - Trampoline
04 - Spring
05 - Teleporter
06 - Grabby slime? (I don't know how to describe this, but it's a standard trap...)
07 - Laser trap
08 - Rope slide

=== SKY ===
00 - Pyramids
01 - Castles
02 - Space
03 - Trees
04 - Lemgo constructs
05 - Stone wall with arches
06 - Ruins? (usually used with Army theme in official levels)
07 - Plain sky
08 - Space Red
09 - Golf course
12 - Space, bigger stars
16 - Snowy trees
17 - Chocolate mountains
18 - Space (blue)
27 - Pinkish sky

=== WALLS ===
00 - Pyramid
01 - Castle
02 - Space
05 - Maze
07 - Pyramid 2 (seems identical to 1)
08 - Computer
11 - Castle 2 (???)
18 - Digits

For all: 255 seems to be used to signify "none", although it has no special treatment (as far as I can tell) compared to any other invalid value.


If you create copies of the respective files in GFX, you can use other numbers from 0 to 127 - but any above that don't work. Presumably, L3D loads it as a signed value, but doesn't know how to handle negative numbers so just loads nothing. I don't know if traps remain functional when doing this.
« Last Edit: June 06, 2019, 10:38:09 AM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #32 on: June 06, 2019, 10:42:09 AM »
Here's some interesting finds I came across while investigating the possible values for the Signs field in the PC version.

Yes, in the PC version. Keep that in mind as you look at the third one.

Here's (I think) all of these kinds of signs. The "Oh No" one that appears twice in this image, appears twice in the data. Actually it appears three times - because the entire far-right column appears twice (and the "Oh No" also appears elsewhere). The back is duplicated a total of 16 times in the data. The three in the first column in this image, due to technical limitations could not appear in the same level as the back of the sprites - though it would be pretty easy to patch that.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #33 on: June 06, 2019, 08:18:18 PM »
Being an exit appears to be a special property of Block #1, which is automatically an exit on the side that (unrotated) faces Z=31, as long as that face is solid. If I copy Block #1's data to another block verbatim, it is no longer an exit. If I copy another block's data to Block 1, Block 1 remains an exit.

Block 0 remains "empty" no matter what the BLK file says. Blocks 2, 3 and 4 also seem to have special handling, though I haven't figured out what yet. I do notice block #2 is a splitter in the level I've been testing with, but I don't know if that's a fixed rule.

This is, at this stage, only based on testing with one level (specifically Mayhem 6 "Family Tree").

EDIT: Splitter does indeed appear to be an inherent property of Block #2, assuming the side faces are not solid. EDIT: No, even if they're solid - as long as they're not indestructible, if you bash away the block then the empty space acts as a splitter.



For the 3rd byte of block definitions, this is a bitwise value - I've found 0x10 makes the block nonsolid, while 0x02 sets the block as metal. I haven't figured out what the other flags do yet. Mayhem 6's splitter has 0x10, 0x08, 0x02 and 0x01 set - but I don't notice any difference when disabling 0x08 or 0x01, or enabling 0x04 (which I have seen enabled on other blocks).

The second byte in a face definition also appears to be at least partially bitwise, though I'm still figuring out how it interacts with the first byte exactly - and I haven't even looked at the third. 0x80 seems to be an "allow transaprency" flag.



Blocks #5 through #8 also appear to be unused in the majority of levels, although I haven't noticed anything special about them so far.
« Last Edit: June 06, 2019, 09:33:03 PM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #34 on: June 07, 2019, 10:51:36 PM »
Anything further I discover, assuming I don't want to raise it here as a matter of "can anyone else help figure this out?", will go into the "notes" folder in L3DEdit's Bitbucket repo.

https://bitbucket.org/namida42/l3dedit/src/master/notes/

These may be updated without notice as new information is discovered, but due to how Git works, you can always recover older versions of the documents.

These already contain information beyond what's listed here, so I recommend checking it out if you're interested in the formats. Of course, I'm mostly only going to pay attention to formats that are editor-relevant.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #35 on: June 15, 2019, 02:51:50 AM »
Lemmings 3D stores the replay data in memory, in exactly the same format as the demos for the practice levels (and four others, that will play if you leave the main menu idle too long).

With a debugger copy of DOSBox, you can dump the memory and extract the data from it, and create your own replay files this way. Here's proof - a replay file for my custom level Space Station. Replace the existing REP.000 in your L3D "REPLAYS" folder with this one, boot up the title screen, sit back and wait (it takes a few minutes for demo mode to kick in).

If you don't want to actually wait for the demo mode to start, or don't have a copy of L3D handy, this tease is also available in Youtube form: https://www.youtube.com/watch?v=ZDm2kZUWSRs

It goes without saying that watching this replay, either via Youtube or via an actual copy of Lemmings 3D, will spoil the solution to Space Station.

EDIT: This has now lead to L3DReplay.
« Last Edit: June 15, 2019, 09:47:55 AM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline Pooty

  • Posts: 359
    • View Profile
Re: 3D Lemmings file formats
« Reply #36 on: June 17, 2019, 02:48:41 AM »
I have learned the following information about the game's water.

Code: [Select]
0x118   Water direction towards -Z (signed byte)
0x119   Water direction towards -X (signed byte)
0x11A   {Still unknown}

0x11B (flags):
00000000 = Lemgo-style rendering, animated
00010000 = Lemgo-style rendering, not animated
00100000 = Other-style rendering, no animation
00110000 = Seems to be the same as the previous one.
SEGA Master System version
100% on 110/120 levels (92%). Other levels [Lemmings lost]:
Fun 03 [3], 06 [2], 18 [5]   
Taxing 19 [5], 27 [1], 28 [3]
Tricky 15 [5], 17 [2]
Mayhem 19 [7], 26 [10]

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #37 on: June 18, 2019, 09:19:16 AM »
I've looked at the remaining unknown bytes (or bit flags, in some cases). Some of them really have no pattern, and we might have to observe closer. However, there's a lot that seem to be the same on all but a few levels.

I'm going to list a few groups of levels, that share currently-unknown values in the data with each other, but different from all levels outside the group. To be very clear - each group is independent. And I realise this is a long shot, but also figured it doesn't hurt to ask.

And just to be clear - it's entirely possible some (or even all) of these are just huge nothingburgers, possibly remnants from the official editor or of features that were removed in the final game, or something like that. So please don't spend too much effort on thinking of it - this is more a "if anything comes to mind, let me know".

Group 1: "Bounce Bounce", "Play Time"
Group 2: "A Shortcut Through The Forest", "Play Time", and all practice levels
Group 3: "This Is The Army", "Stilt Walkers" and "Let The Race Begin"
Group 4: "Alilemms", "Rescue Team" and "The Hanger"
Group 5: All enclosed levels, plus "The Prisoner" and "Claustrophobic" (these last two still have the cameras in enclosed areas, so perhaps it's some setting that relates to that..)
Group 6: "The Bean Machine", "A Shortcut Through The Forest", "Hole in Ten", "Fore!", "Lemlab", "Spaghetti Junction", "Ricochet", "Motherboard", "Birthday Cake", "Brechin's Staircase", "The Hanger", "The Catacombs", "Hole In One, Two, Three", and all practice levels
Group 7: "The Bean Machine", "Alpine Assault Course", "Alilemms", "Jelly Climber", "Oh No! More Pyramids!", "Garden Maze", "King Coder's Tomb", "Rescue Team", "The Arena", "Snake", "The Five Arches", "Brechin's Staircase", "Ski Jump", "Family Tree", "Raiders of the Lost Lemming", "Jelly Belly Islands", "Castle Peralus"
Group 8: "Attack Of The 50ft Lemmings", "Jelly Climber", "A Head Above The Rest", "Play Time", "The Arena", "T Minus 5 And Counting"
Group 9: "A Shortcut Through The Forest", "Stilt Walkers", and all practice levels
Group 10: "Bounce Bounce", "It's A Classic", "Lemlab", "Rescue Team" and "Tuther Circus Level" (with "Picky Platform" having a unique third value, instead of the same one as all other remaining levels)
Group 11: "The M-A-Z-E", "Rescue Team", "Team Work", "Death Slide" and "Slippery"
Group 12: "Alpine Assault Course", "Slippery Maze", "The M-A-Z-E", "Ski Jump" and "The Hanger"

Additionally, can anyone think of any attributes unique to any of the following levels?
- "Take A Dive"
- "Bounce Bounce"
- "Jelly Climber"
- "Lemlab" (other than the laser trap)
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #38 on: June 24, 2019, 08:22:14 AM »
The save file (C:\LM3D.CD\LM3D.SAV) has a checksum or CRC of some kind - the first two bytes of the file. RevEng was not able to crack it, although it's very possible I just wasn't using it correctly. I'm wondering if anyone has any ideas here?

I could possibly try and get some CPU logs from DOSBox, if that helps. I have a fair idea of when L3D calculates this checksum (but I wouldn't know the first thing about understanding the CPU instructions that it follows to calculate it).

Beyond this checksum, the format of the save file looks pretty simple, so reading it won't be hard - but I'd like to be able to write it (so I can create a proper "all levels unlocked" save - using the actual level titles, where custom levels are in play - instead of relying on one created via the RASPUTIN cheat).
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: 3D Lemmings file formats
« Reply #39 on: June 24, 2019, 09:27:03 AM »
I could possibly try and get some CPU logs from DOSBox, if that helps. I have a fair idea of when L3D calculates this checksum (but I wouldn't know the first thing about understanding the CPU instructions that it follows to calculate it).

I haven't worked with DOSBox's debugging features, but if the CPU logs shows executed CPU instructions in assembly language format I could probably read it, though to be practical the log output needs to be loop-aware (it gets a bit annoying to read as you can imagine, if I have to manually skip through, say, 10000 iterations of some sequence of instructions executed in a loop producing that many lines in the logs).  So get the logs and we'd see how things go.  Alternatively, if the CPU logs help show the general memory addresses where the suspected CPU instructions are located, and DOSBox has a disassembly feature to convert the bytes in a memory region into assembly code, that will probably work better for me.  (And even without a disassembly feature, if DOSBox can at least dump the raw bytes of the memory region, I could load that into a debugger to look at disassembly.)

The checksum might not be CRC per se but just some other algorithm.  It can really be anything that's mildly tricky (or maybe not even that) to reverse engineer.  Honestly I'm kinda surprised they even bothered, it's just a savegame in a singleplayer game that already has a cheat code anyway.

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #40 on: June 24, 2019, 09:31:39 AM »
Here's a small sample of what comes up in the CPU logs (NOT actually from generating the checksum; this is just in general what logs look like). Unfortunately, it's meaningless to me, so I have no idea if it will be of any help.

Code: [Select]
1EA3:00005686  jnc  000056AF ($+27)            (down)                 EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000B92  call 00000CEE ($+159)                                  EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000B92  call 00000CEE ($+159)                                  EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000D2F  movzx ax,[0B34]                 ds:[0B34]=120F         EAX:00001D92 EBX:0000000D ECX:00040000 EDX:0000005C ESI:0000099E EDI:000071B0 EBP:00005C2D ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000D4D  movzx ax,[0B37]                 ds:[0B37]=1B04         EAX:00003387 EBX:00000018 ECX:00040000 EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000D84  pop  cx                                                EAX:0000149D EBX:0000001E ECX:00040000 EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B95  jnc  00000BBB ($+24)            (no jmp)               EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B95  jnc  00000BBB ($+24)            (no jmp)               EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B97  mov  ax,[233C]                  ds:[233C]=002F         EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
1EA3:000055C8  push si                                                EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1EA3:000055C8  push si                                                EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000BAF  popa                                                   EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BAF  popa                                                   EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BB0  bts  word [2452],06             ds:[2452]=0180         EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BB6  mov  si,0B54                                           EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BC7  mov  [0B14],si                  ds:[0B14]=0B65         EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:000018D3  mov  al,[C97A]                  ds:[C97A]=0000         EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
F000:00001440  push ds                                                EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F2 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:0
F000:00001460  callback 0023  (int 74 ret)                            EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007CE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
F000:00001464  cli                                                    EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007CE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
01ED:000018D3  mov  al,[C97A]                  ds:[C97A]=0000         EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1

The debugger build of DOSBox can certainly do memory dumps, and can also set watchpoints to break if a certain memory address is changed. I've managed to locate a copy of the save file in memory, although it appears this is only used by the game when loading / saving the file - still, this should at least be useable to break at the right time (via memory watchpoints).

The memory address where the file's contents can be found is 0x65460. I don't know what that translates to in terms of segments / etc; I just know that it's consistently at that address in a full memory dump, both when dumped via the debugging build of DOSBox and when (in standard DOSBox) dumped via a modified version of L3DReplay that extracts the entire memory instead of just the replay data.

Here's a CPU log. The checksum should be recalculated (at least) twice during these logs, unless they were too far back to be logged.

L3D is quite inefficient about this. When you complete a level, it first writes your new records for that level, then calculates the checksum. Then, it unlocks the next level, and calculates the checksum again. It also appears to write to file both times.
« Last Edit: June 24, 2019, 11:02:31 AM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: 3D Lemmings file formats
« Reply #41 on: June 24, 2019, 06:35:19 PM »
At the moment I'm not confident that I can make good use of the CPU logs.  Normally the leftmost column would be the memory address of the instruction being executed, but in the logs they don't line up with the jumps.  Maybe in the logs they have more to do with the location of instruction data as allocated by the emulator, rather than the instruction addresses as seen in emulation?  It might help if there's documentation on how to read the DOSBox CPU logs but I'm not finding any so far from a web search.

The best bet would probably be to find some way to locate the executed instructions in memory, do a memory dump, then use an external tool to disassemble the raw memory dump back to assembly code.  At work right now, will update later on how one might go about this.

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #42 on: June 24, 2019, 08:28:53 PM »
I wouldn't have a clue how to locate the executed instructions, but I can get some memory dumps for you - I'll do this a bit later.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: 3D Lemmings file formats
« Reply #43 on: June 24, 2019, 09:18:53 PM »
Further probing of web search results found a potential answer to the CPU logging issue:

https://www.vogons.org/viewtopic.php?f=32&t=7323&start=20#p162190

I assume the "core=normal" setting mentioned there is one of the settings in DOSBox.conf.  Sounds like you need to set that in order to ensure the CPU log is complete.  Can you try that and upload the resulting CPU log?  It's likely DOSBox will run (even) slower with the combination of that setting plus CPU logging enabled, but it's necessary.

It's still not ideal that the CPU log, as I feared, is just a linear sequence of executed instructions, but I can still try to work with that.  For now you can hold off on making memory dumps.

One more thing:  if you are able to set a data breakpoint on when the checksum memory location will be written to, do that and capture the CPU log at the time it hits that breakpoint.  Also, in case the log doesn't include the instruction for the breakpoint-triggering write (depends on whether it breaks before or after the write completed), copy whatever's displayed in the debugger window at the time the breakpoint is hit, I assume it should show the instruction being executed that triggered the data breakpoint.  Copy that info to end of CPU log.

[sidenote: for the memory dump thing I was requesting but holding off for now, you would take the address of the instruction the data breakpoint was hit, and grab maybe around 32000 bytes before and after the that address as the range to dump]

Offline namida

  • Administrator
  • Posts: 12398
    • View Profile
    • NeoLemmix Website
Re: 3D Lemmings file formats
« Reply #44 on: June 24, 2019, 09:26:24 PM »
Here you go. This should have the right info - I do notice that EAX's value is that of the checksum, and that it changes (through many different values) before the breakpoint (which is set to fire when the memory byte at 0x00065460 changes).

Getting a memory dump is trivial (the logs are much trickier) so I included one of those at the time of the breakpoint too, though it's just the entirity of the virtual memory space (or at least - it starts from the beginning of said space, and goes well beyond anything I've seen L3D actually use). I can also do memory dumps by segment / offset, if that's any help, but I would be clueless as to what the segments actually are or how they work so you'd need to be able to tell me exactly what values I'm using.

Quote
It's likely DOSBox will run (even) slower with the combination of that setting plus CPU logging enabled, but it's necessary.
No slower than usual for the debugger build of DOSBox, it seems. DOSBox with default settings is a bit slow when playing L3D in general, but a few config changes sorted that out (the biggest one being setting cycles to "max" - any specific value seems to either be laggy or have sound glitches, but "max" works perfectly).
« Last Edit: June 24, 2019, 09:40:02 PM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)