Lemmings Forums

Lemmings Boards => Tech & Research => Topic started by: Pooty on April 08, 2013, 10:13:57 PM

Title: 3D Lemmings file formats
Post by: Pooty on April 08, 2013, 10:13:57 PM
Hello, you. /waves

So, I've been busy for the last week or so. I've been trying to understand the technical stuff that makes 3D Lemmings work. I've made a good amount of progress, but I'm going to have to cut it short for now as I must focus on my university studies for the next few weeks.

I'll dump what I've learned so far in this thread. I've even made a level that you can try out, which I've posted in the level design forum (https://www.lemmingsforums.net/index.php?topic=1591.0). The level was planned using Minecraft because imagining a puzzle in 3D was a lot harder than I expected. Also, the level was made solely through hex editing, so I needed something to reference as I was going along.

Anyway, here's what I know so far. These aren't particularly tidy notes since these were made on the go, but I'm hoping that if you squint your eyes, you can make some sense of them. :P

[Edit 2019] This post has been updated with notes compiled by me and namida. More detailed notes have been put up in the L3DEdit repository (https://bitbucket.org/namida42/l3dedit/src/master/notes/), and may be more up to date than what's listed here.

===Technical overload begins here===

LEVEL.###
This file is in the LEVELS folder, and is where almost all of the level data is stored. Normally, this file is compressed using the RNC compression algorithm version 1 (http://segaretro.org/Rob_Northen_compression). It's a common enough algorithm, so any tool that can decompress RNC1 files will work here. Also, the file will load whether it's compressed or not. The uncompressed file size is around 65Kb, which is not a lot these days.

Level Information
Code: [Select]
Integers are stored in little endian.

0000      Byte                 Timer (Seconds)
0001      Byte                 Timer (Minutes)
000A-001B Struct (2 members)   Skill ID? (Byte), Number of this skill (Byte)
0022-???? Struct (3 members)   LandVertexX (Byte), LandVertexY (Byte), ??? (Byte)
00E2      Int16                Release Rate
00E4      Int16                No. of Lemmings
00E6-00E7 Int16                To Be Saved
00E8      Byte                 Texture Set
00E9      Byte                 Land Graphic
00F0-010F String               Level name (String requires null termination)
0110      Byte                 Non-interactive object set
0111      Byte                 Sign Set
0112      Byte                 Sea Graphic
0113      Byte                 Animated non-interactive object set
0114      Byte                 Interactive object type (e.g. traps)
0115      Byte                 Sky Graphic
0117      Byte                 Walls Set
011D-013C String               Comment (Not used in game, often used to store level author)
0143      Byte                 Theme index (see “Music & Wallpaper” section)
0151      Byte                 Music index
0153-0156 Array (4 elements)   Level border definition (kills lemmings that leave the border). Order: -X, -Z, +X, +Z
0159      Byte                 Preview Camera Pivot Y
015A      Byte                 Preview Camera Pivot Z
015B      Byte                 Preview Camera Pivot X
015C      Byte                 Kill ceiling (level border definition for +Y)
015D-017C Struct (5 members)   Initial camera positions (see "Camera Positions" section)
0200-7DFF                      Level Block Data
8200-FDFF                      Objects Data
Refer to the attached image to see how some of these relate to the objects seen in the game.

You'll notice there are gaps in there. There are plenty of things I couldn't figure out. For example, I don't know why the water moves the way it does. In level 2, the water is animated. I couldn't work out how this could be done.

Camera Positions (015D-017C)
There are four cameras. Data for each camera is stored in 8 bytes of data as follows.
Code: [Select]
Integers are stored in little endian

+00   Signed Int16  Camera Y Position
+02   Signed Int16  Camera Z Position
+04   Signed Int16  Camera X Position
+06   Byte          Camera rotation. 4 values.
+07   Byte          Unknown

Level Block Data (0200-7DFF)
Each level block data element is stored as a single 16-bit integer in little endian.

Code: [Select]
[MSB] XXXXXXYYYYZZWWWW [LSB]

X   6 bits   Block graphic ID (refer to BLK file)
Y   4 bits   Block type
Z   2 bits   Rotation. The block will rotate counterclockwise (90 degrees * Z)
W   4 bits   Block segments (blocks are split into 4 vertical segments from top to bottom, and all true bits must be consecutive)

The order in which the blocks are stored are along the Z axis (blue), Y axis (green), and then the X axis (red). The level size is stored in memory as 32 blocks long, 32 blocks wide, and 16 blocks tall. However, there are glitches in the game that prevent the full field from being used.
(https://i.imgur.com/qd1LBAR.gif)

Objects Data (8200-FDFF)
The objects data also uses 2 bytes to store data, and the order in which the game objects are stored are exactly the same as the level blocks. The definition of the two bytes, however, are different. Editing the second byte has done nothing as far as I've seen, so the following information relates only to the first byte.

Code: [Select]
00     Draw nothing
01-1F  Static Object (from OBJ.xxx)
20-3F  Signs (from SIGNS.xxx, first 6 bits = type, last 2 bits = rotation)
40-4F  Unknown
50     Animated Object (from ANIMOBJ.xxx)
51     Animated Object using four large Static Object frames (scrolling through frames 05-08)
60-67  Interactive Object (from TRAP.xxx)
70-C7  Wall (7#-C# represents the direction)
D0-E7  Half Wall (D#-E# represents the direction)
F0-F7  glitchy wall

Music & Wallpaper
Memory address 0143 is where the music and the wallpaper data is stored per level. The wallpaper refers to the image that you see before you start the level. If you've played the PlayStation and Saturn versions of the game, this image will also appear after you finish the level (win or lose).
Code: [Select]
01  Castle
02  Egypt
03  Space
04  Sweets
05  Golf
06  Computer
07  Army
08  Maze
09  Circus
0A  Lemgo (coloured blocks)
The alternative songs for each theme can be chosen at the memory address 0151. From what I've seen, values from 00-03 are accepted. Other values can result in some strange glitches.

BLK Files
These files are also stored in the LEVELS folder, and are linked to the level with the same file extension, e.g. LEVEL.000 is linked with BLK.000. This file represents a dictionary of blocks that can be placed in the level. There are 64 block definitions per level.

Some block indexes have special rules.
Code: [Select]
Block 0     Entrance. This requires the top two segments to be solid and the bottom two segments to be empty. There is an effective limit of 4 entrances per level.
Block 1     Exit. The lemmings interact with the +Z face unless the block is rotated.
Block 2     Splitter block.
Blocks 3-5  Unknown
Block 6     One-way block. It does not automatically put in any one-way block graphics, and they must be implemented using the Face Data below.
Blocks 7-8  Unknown

In official levels, normal block data begins from index 09.

Each block consists of 22 bytes.
Code: [Select]
00-01  Unknown
02     Block flags
03     Unknown
04-06  Data for +Z face
07-09  Data for -Z face
0A-0C  Data for +X face
0D-0F  Data for -X face
10-12  Data for +Y face
13-15  Data for -Y face

Block Flags (offset +02)
Code: [Select]
01   Double-sided rendering. The inside of the block will be rendered with the same textures as the outside of it (example: bridge in level 10).
02   Steel. The block is indestructible.
04   Liquid. Lemmings will drown if standing on this block.
08   Unknown
10   Non-solid for lemmings. The camera interacts with this block, but the lemmings pass through it.
20   Unknown
40   Slippery. Lemmings will slide on the top of this block.
80   Non-solid for camera. The lemmings interact with this block, but the camera passes through it.

Face Data (Offset +04-15)
These represent the graphics that each face of the block will use.
Code: [Select]
+00   Texture Index
+01   Texture Modifier Flags
+02   Shading

The graphics used by each face is stored in the level's texture file. The offset for each graphic is set to the value of the Texture Index multiplied by 4096 (0x1000). However, if any of the flags below are set, this can change the behavior in a currently unknown way.

Texture Modifier Flags
Code: [Select]
01-10   Unknown
20      Face will animate between 4 consecutive face textures.
40      Face will animate using special rules (see below).
80      Face will treat colour 0 as transparent.

The special rules of 40 depend on the value given for the Texture Index. Any indexes not listed below are unknown.
Code: [Select]
00      Four-frame animation from frame 12-15
02      Eight-frame animation from frame 00-07
06      Seems identical to 02

Shading
This byte reduces the brightness of the face texture.

Palette Data
The in-game palette is stored in a file called GFX/LM3D.PAL. The file consists of 256 RGB triplets. The colour values range between 00-3F.

Other files in the GFX folder (*.000, *.001, etc.)

Other files in the folder are stored as a single channel image. By linking these files to the palette, you'll get the graphics as they're seen in-game. The width of each type of graphics file are as follows:
Code: [Select]
AnimObject  64
Bgrd        320 (it’s the “3D Lemmings” background that you can see in the main menu)
Land        128
Obj         64
Sea         64 (not confirmed)
Signs       64
Sky         1024
Texture     64
Traps       64
Walls       64

The contents of each file is described below.

AnimObject
Code: [Select]
000   Egyptian Torches
001   Flag bearing the Clockwork Games logo
002   Flashing lightposts
003   Rings of fire
004   Lemming flag
005   Another lemming flag (different from 004)
006   Chequered flag
007   Candy cane torches
008   Neon exit sign
009   Exit flag
011   Pillar torches
013   Juggling lemmings
022   Rocket flame
023   Duplicate of 001
090   "Block Here"
091   "Turn Here"
092   "Bomb Here"
093   "Build Here"
094   Duplicate of 093
095   "Bash Here"
096   "Mine Here"
097   "Dig Here"
098   "Climb Here"
099   "Float Here"

Land
Code: [Select]
000   Dirt / sand
001   Grass
002   Black diagonal tiles
003   Red and yellow triangle tiles
004   Lemgo Red
007   Chocolate
008   Circuit board
009   Grass 2
015   Grass 3
017   Chocolate 2
018   Dirt / sand 2
028   Blue
038   Solid blue

Obj
Code: [Select]
000   Pyramid
001   Castle
002   Maze
003   Circus
004   Lemgo
005   Maze
006   Army
007   Candy
008   Maze 2
009   Golf
010   Pyramid 2
015   Maze 3
016   Army 2 (Snowy)
018   Computer
038   Pacman Ghosts
095   Warning Signs

Overlay
This file stores the cracks that you see when a lemming is bashing, mining or digging through terrain. It also masks the terrain debris that appear during these processes so that they don't appear so blocky. These are both stored as 1 bit-per-pixel images.
Code: [Select]
0x0000 - 0x1FFFF   Terrain cracks
0x2000 - 0x2FFFF   Debris masks

Sea
Code: [Select]
000   Quicksand
001   Dark blue water
002   Normal blue water
003   Green slime
004   Lemgo water - blue
005   Lemgo water - red
006   Icy water
007   Liquid chocolate

Signs
Code: [Select]
000   Pyramid Splitter, Credit Signs ("One Giant Step", "Oh No", "Lemmings 3D") - front only
001   Castle
002   Credit Signs ("Programming By" x2, "Graphics By", "Raytracing By") - front and back
003   Credit Signs ("Produced By", "Eclipse", "Sony", "Playstation") - front and back
004   "Paradise This Way" replaces "Keep Off Grass", Credit Signs ("Hi Ho", Waterski, "Oh No" (dup), Juggle) - front and back
005   Maze
006   Army
007   Candy
008   Duplicate of 04
009   Castle 2, also has a LemLab-looking splitter entrance?
011   Castle 3, mostly identical to 09 but "Keep Off The Grass" isn't replaced and has Castle splitter instead of LemLab-y
095   Maze 2

Sky
Code: [Select]
000   Pyramids
001   Castles
002   Space
003   Trees
004   Lemgo constructs
005   Stone wall with arches
006   Ruins? (usually used with Army theme in official levels)
007   Plain sky
008   Space Red
009   Golf course
012   Space, bigger stars
016   Snowy trees
017   Chocolate mountains
018   Space (blue)
027   Pinkish sky

Texture
Code: [Select]
000   Pyramid
001   Castle (???)
002   Spaceship
003   Circus
004   Lemgo
005   Maze
006   Army
007   Confectionary
008   Computer
009   Golf
010   Pyramid 2
011   Castle 2
012   Spaceship 2 (/ Lemlab?)
013   Circus 2
015   Maze 2
016   Army 2 (Snowy)
018   Computer 2
019   Combination of Pyramid and Golf?
021   Castle 3 (???)
022   Space 3 (Dot to Dot)
028   Computer 3 (Breakout?)
036   Army 3 (Corkscrew Digger)
038   Computer 4 (Lemmtris?)
046   Golf 2 (Land Ahoy)
048   Computer 5 (Lemmtris?)
095   Maze 3

Traps
Code: [Select]
000   Flame blower
001   Squasher
002   Bear trap
003   Trampoline
004   Spring
005   Teleporter
006   Grabby slime trap
007   Laser trap
008   Rope slide

Walls
Code: [Select]
000   Pyramid
001   Castle
002   Space
005   Maze
007   Pyramid 2 (seems identical to 1)
008   Computer
011   Castle 2 (???)
018   Digits

BombNumb.gfx
This file contains the graphics for the bomber counter, the camera number label, the ‘?’ when a lemming runs out of builder bricks, and the highlight lemming cursor. It is arranged in this format:
12
34
5?


Cogs.gfx
The cogs appear when the game is loading. I think the image width is 88 pixels, but I’m not certain. Also, I’m not sure where the palette data for this comes from.

Lemmings.fnt
This 32 pixel-wide file is where the animated lemmings used in the password screen comes from.

Minilemm.gfx
This file is also 32 pixels wide, but this file represents the lemming icons at the bottom of the game screen.
Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on April 08, 2013, 10:39:47 PM
This is awesome! I love the technical expertise we have in this community.
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on April 09, 2013, 02:21:25 AM
I'm not a super-ultra-technical type, but this makes for fantastic reading nonetheless. :thumbsup:

I'm still hoping to obtain rips of the sprites for the lemmings themselves in, say, PNG format. Is that any closer, now?
Title: Re: 3D Lemmings file formats
Post by: ccexplore on April 09, 2013, 02:47:06 AM
I'm still hoping to obtain rips of the sprites for the lemmings themselves in, say, PNG format. Is that any closer, now?

Well, between the info on compression, palette data and "other files in the gfx folder", at least some graphics and textures can definitely be extracted.  Not having played the game though, I'd think for the lemmings, you'd need the 3D models as well?  If so then that information is currently not documented it seems.

I've even made a level that you can try out, which I've posted in the level design forum. The level was planned using Minecraft because imagining a puzzle in 3D was a lot harder than I expected. Also, the level was made solely through hex editing, so I needed something to reference as I was going along.

Minecraft actually sounds like a very good idea.  Like 3D Lemmings, I haven't actually played Minecraft, but based on what little I know about it, sounds like it might be plausible to make use of Minecraft as a poor-man's level editor so you don't have to write one from scratch, as long as someone implements a way to convert from Minecraft representation to the Lemmings 3D game files' representations?  (Granted, I don't know how much information there is out there on Minecraft's own file formats.)
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on April 09, 2013, 03:09:05 AM
Not having played the game though, I'd think for the lemmings, you'd need the 3D models as well?
In Lemmings 3D, the lemmings don't have 3D models - just sprites, shown from half a gazillion* angles, that are scaled as needed. :P

*Totally scientific, I swear!

I'm guessing that they're contained in the files lemm.mhc and lemm128.mhc (the latter, I assume, being the high-resolution lemmings), from the directory lemm.
Title: Re: 3D Lemmings file formats
Post by: Pooty on April 09, 2013, 04:21:53 AM
I'm guessing that they're contained in the files lemm.mhc and lemm128.mhc (the latter, I assume, being the high-resolution lemmings), from the directory lemm.

Excellent guess. I think you're right. :thumbsup:

The demo and winterland versions of the game contains a surprising amount of uncompressed data. lemm128.mhc is not available on those versions, but lemm.mhc is.

I opened it up in Paint Shop Pro as a raw image of 26 pixels wide, and...


Almost there, but I can't quite get it right. Maybe the rest of the data in the file is important somehow?

Edit: I've just looked at the contents of OBJ.038.
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on April 09, 2013, 06:01:03 AM
Whoah! Could there have been some sort of Pac-Man knock-off theme at some point? :o (EDIT: I mean, the game's "Lemgo" set was originally called "Lego", with no apparent regard for trademarks, wasn't it? The music for those stages even retains that name... Or perhaps the Pac-Man style stuff is just from some sort of test?)

Also, excellent work with the lemming sprites, so far! :D
Title: Re: 3D Lemmings file formats
Post by: ccexplore on April 09, 2013, 06:47:20 AM
I opened it up in Paint Shop Pro as a raw image of 26 pixels wide, and...
<snip>
Almost there, but I can't quite get it right. Maybe the rest of the data in the file is important somehow?

I don't have the tools at the moment to do this myself, but try opening it with a different width and see what happens.  Even if it doesn't divide evenly into the total number of pixels (it's always possible some of the bytes are filler or otherwise unused by the game).

I'd start with 24 because it's divisible by 8.  If there's a "correct" width it's likely to be lower than the current 26, given the leftward skew of the graphics right now as you go from top to bottom.
Title: Re: 3D Lemmings file formats
Post by: Pooty on April 09, 2013, 12:17:29 PM
I don't have the tools at the moment to do this myself, but try opening it with a different width and see what happens.  Even if it doesn't divide evenly into the total number of pixels (it's always possible some of the bytes are filler or otherwise unused by the game).

I'd start with 24 because it's divisible by 8.  If there's a "correct" width it's likely to be lower than the current 26, given the leftward skew of the graphics right now as you go from top to bottom.

I've tried other values, but for some reason, 26 was arguably the closest to looking correct.

I've had another look at it, and I've realised that there is a second layer of compression. Just before the lemming gets drawn for the first time, there is a repeated pattern of 40 00 40 00 ... (in hexadecimal). The 40 is the offset for the current line in the sprite (in this case, 64 pixels), and the 00 is the number of pixels at the end that are set to the transparent value 0. Since the sprite is 64 pixels wide, this essentially means to draw a blank line.

The first non-blank line reads:
Code: [Select]
1D 1C 6E 6F 6F 72 6E 73 6F
This means: begin drawing the sprite at 0x1D pixels across (29 pixels), and leave 0x1C pixels at the end (28 pixels). Add these two values together, and take the result away from 0x40. The result is 7, so there are 7 pixels that are not blank. The next 7 bytes are the palette values that fill this gap.

I got started with that idea, and it definitely seems to work. Unfortunately, I made a mistake just after the shoulders, so I present to you half a lemming. :P


Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on April 09, 2013, 02:58:09 PM
Attach that file (or a zipped copy) and I'll take a whack at it. Cracking data formats is my cutie mark. (-:
Title: Re: 3D Lemmings file formats
Post by: Pooty on April 09, 2013, 04:09:45 PM
Here you go. I've also attached the palette file.
Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on April 09, 2013, 04:45:52 PM
Thanks, though I'm a step ahead of you. I forgot I had the disc image on my hard drive still because I missed it when backing everything up to DVD archives. (-:

Your earlier work was a good start. I've figured out the remainder of the file. The format is as follows:


Code: [Select]
File
===============================================================================
Manifest    Entry[568]    Cell manifest (0x08E0 bytes)
CellData    Byte[]        Cell data
===============================================================================

Entry
===============================================================================
Animation    Byte      The animation this cell belongs to
Flags        Byte      For 128px cells, 0x80 means "last cell in animation"
Offset       UInt16    The location in CellData of the Cell structure
===============================================================================

  Offset is stored as a 16-bit value to save space. To resolve the actual
  address, first add 0x08E0 (for the manifest), then add Animation * 0x4000 to
  resolve the actual offset of the cell in the file.

Cell
===============================================================================
LineOffsets    UInt16[]    Offsets relative to the Cell structure per Scanline
LineData       Byte[]      Scanline data
===============================================================================

  The LineOffsets are relative to the beginning of the Cell structure, not the
  beginning of LineData.

  The number of elements in LineOffsets depends on the file being processed.
  For LEMM.MHC, it's 64, and for LEMM128.MHC, it's 128.

Scanline
===============================================================================
Lead      Byte      Number of transparent pixels on the left side of the line
Trail     Byte      Number of transparent pixels on the right side of the line
Pixels    Byte[]    Width - Lead - Trail bytes of pixel data for the line
===============================================================================

  Pixel values represent palette indexes.

Attached to this post are ZIP files containing every sprite from both the 64 and 128 pixel files. For the rest of you, some sample rips:

Title: Re: 3D Lemmings file formats
Post by: Pooty on April 09, 2013, 04:52:15 PM
Bloody hell, that was fast. Nice one. :thumbsup:
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on April 09, 2013, 07:13:20 PM
AIEE!! Awesome! Thanks very much! :thumbsup:

This also seems to confirm a suspicion I had about the "Rasputin" cheat - it looks from this like it really does have unique sprites for assassinated lemmings (it was a little tricky to tell in-game due to the scaling), since there are indeed separate sprites for stunned-on-landing lemmings in here. :o

EDIT: I should clarify, those thanks are directed at both of you. :D I'm going to get some of these printed up as standees, I think, hehe...
Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on April 10, 2013, 06:37:18 AM
I'm going to get some of these printed up as standees, I think, hehe...

*Imagines a desk littered with assassinated Lemmings*

On an unrelated note, Rasputin was always the name of my Bulbasaur. I am not joking.
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on April 21, 2013, 09:04:11 PM
Haha! :P

@Pooty - Any more progress on this, lately? I must admit to being curious as to whether you'll be looking at the PSX version, too, since I know there's a wallpaper in that which isn't found in at least the MS-DOS version (I have no idea about the Saturn one, as I've never encountered it), which is used when you complete the Practice levels.
Title: Re: 3D Lemmings file formats
Post by: Pooty on April 21, 2013, 11:33:26 PM
No progress from me, sadly. I've been focusing on my university work. I'm in my last 2-3 weeks of the university semester, so not too long now. :)
Title: Re: 3D Lemmings file formats
Post by: Mikex62 on May 03, 2013, 01:02:32 AM
Has anyone tried to find results for sounds?
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on May 03, 2013, 01:27:36 AM
I had a go at getting at the sound effects a little while ago. You can open them in Audacity if you import them as raw audio, but I never figured out what frequency they're stored at (in fact, in my brief messing around with it, it seemed that each one may be different), so I couldn't get them to sound right...
Title: Re: 3D Lemmings file formats
Post by: Mikex62 on May 04, 2013, 01:28:26 AM
I took a crack at it and turned out I needed the byte order to be "Default endianness." Here is some of the samples I converted. :thumbsup:
Title: Re: 3D Lemmings file formats
Post by: Pooty on May 04, 2013, 02:20:19 AM
Nicely done. I'd expect that the endianness that would work would be little endian, since the rest of the game seems to use that. But I haven't looked into it much, so don't take it from me.

I did have a little play about just now with the VOXFX sound files, and like Prob Lem said, some of the playback rates of the files seem to be different. Some files worked at 11025 Hz, and others worked at 22050 Hz.

I've got around a week left of university for this year, so I can resume looking into 3D Lemmings more then.
Title: Re: 3D Lemmings file formats
Post by: Mikex62 on May 04, 2013, 03:06:54 AM
I just notice something when slowing down this file down 75%! ??? It sounds like a normal human! I put a comparison of both, the in-game one and the slowed down one in the attachments.
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on May 04, 2013, 04:18:10 PM
Unfortunately, the real lemmings were too busy colonising some more islands, and couldn't make it to the recording studio, so they had to assign some human stand-ins. ;)
Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on May 04, 2013, 11:54:27 PM
I just notice something when slowing down this file down 75%! ??? It sounds like a normal human!

Uh... um... Hmm...

...


... Just how exactly did you think they made the voices?
Title: Re: 3D Lemmings file formats
Post by: Mikex62 on May 05, 2013, 09:41:14 PM
They got someone else instead of the first Lemmings voice actor, said the phases weirdly, then sped the pitch and speed up. I still think they did not too bad but not too good.
Title: Re: 3D Lemmings file formats
Post by: Prob Lem on May 06, 2013, 11:26:25 AM
They got someone else instead of the first Lemmings voice actor
Well, naturally. The voices in the first Lemmings game were provided by Scott Johnston's mother, but by the time Lemmings 3D was made, DMA Design had sold the rights to Psygnosis, who had Clockwork Games make that particular installment. :P
Title: Re: 3D Lemmings file formats
Post by: DragonsLover on May 09, 2013, 12:20:02 AM
Awesome stuff here! :thumbsup:

I know the textures are easy to extract: each pixel takes reference to the game's palette.
The Pac-Man stuff is interesting. Wonder if there's some sort of hidden level somewhere or activated thru a cheat code.
And the idea of porting Lemmings 3D to Minecraft has been in my head for some times. Everything is there to make a Lemmings mod:
- The hotbar has 9 slots which could be perfect for the 9 Lemmings skills.
- The world is made of cube.
- There's already the textures of a block being mined.
- There's water and lava.
- We can use slabs for the builder.
- Making a 3D Lemmings model wouldn't be hard. There's already a Lemmings skin available too.
- In Creative mode, we could build any kind of awesome levels possibles.
- We could use the explosion of a Creeper to make the bomber's one (I just imagine with the Armageddon!).
- Making a texture pack would be easy if we can manage to extract the textures of the original game.

Everything else would be mostly some coding to make everything to work.
Title: Re: 3D Lemmings file formats
Post by: GuyPerfect on May 09, 2013, 02:19:47 AM
Everything else would be mostly some coding to make everything to work.

You saved the hardest part for last and made it sound trivial. (-:
Title: Re: 3D Lemmings file formats
Post by: Clam on May 09, 2013, 11:03:04 AM
Virtual Lemming mode is covered too :). You could even 'be the lemming' and do the digging yourself!

Here's a hilarious attempt at recreating Lemmings in Minecraft (only the 2d version mind you): http://www.youtube.com/watch?v=QxLBQU3D_jo

And I already use that Lemmings skin myself :D
Title: Re: 3D Lemmings file formats
Post by: namida on June 05, 2019, 01:10:35 AM
After doing a bit of experimentation with level files, I've found that blocks extend up right up until the start of object data; and objects until the end of the file.

However, the segment outside of the range mentioned in Pooty's post, corresponds to X = 31, which displays extremely glitchy in game (I haven't experimented to see if it functions correctly). Actually, the same is true for Z = 31 too (except where Y = 0; I suspect the glitching does still occur there, but outside the visible area), but this doesn't present as a consecutive block of bytes that appear to do weird stuff. So in practice, these shouldn't be used. Interestingly enough, aside from not being able to get the camera high enough to see the top of it, Y = 15 works no problem.

The attached screenshot shows what happens from one angle, although changing the angle of view can give different results. The blocks responsible for the glitchiness, if they were unglitched, would be a line along the very bottom of the level at the very back, similar to the line along the bottom near the front and the two midair ones.
Title: Re: 3D Lemmings file formats
Post by: namida on June 05, 2019, 11:25:27 PM
Here's some interesting finds I came across while investigating the possible values for the Signs field in the PC version.

Yes, in the PC version. Keep that in mind as you look at the third one.
Title: Re: 3D Lemmings file formats
Post by: namida on June 05, 2019, 11:51:29 PM
Here are the possible values for some of the style-related fields. Note that some of this needs further investigation, especially interactive objects; and "Walls" seems to be a very inaccurate name (but is what the game calls it). Try modifying it on Fun 20. "Signs" is also somewhat misleading, they're more like decorative elements drawn onto blocks (or sometimes in open space).

All values here are decimal

=== TEXTURESETS ===
00 - Pyramid
01 - Castle (???)
02 - Spaceship
03 - Circus
04 - Lemgo
05 - Maze
06 - Army
07 - Confectionary
08 - Computer
09 - Golf
10 - Pyramid 2
11 - Castle 2
12 - Spaceship 2 (/ Lemlab?)
13 - Circus 2
15 - Maze 2
16 - Army 2 (Snowy)
18 - Computer 2
19 - Combination of Pyramid and Golf?
21 - Castle 3 (???)
22 - Space 3 (Dot to Dot)
28 - Computer 3 (Breakout?)
36 - Army 3 (Corkscrew Digger)
38 - Computer 4 (Lemmtris?)
46 - Golf 2 (Land Ahoy)
48 - Computer 5 (Lemmtris?)
95 - Maze 3

=== OBJECTS ===
00 - Pyramid
01 - Castle
02 - Maze
03 - Circus
04 - Lemgo
05 - Maze
06 - Army
07 - Candy
08 - Maze 2
09 - Golf
10 - Pyramid 2
15 - Maze 3
16 - Army 2 (Snowy)
18 - Computer
38 - Pacman Ghosts
95 - Warning Signs

=== SIGNS ===
00 - Pyramid Splitter, Credit Signs ("One Giant Step", "Oh No", "Lemmings 3D") - front only
01 - Castle
02 - Credit Signs ("Programming By" x2, "Graphics By", "Raytracing By") - front and back
03 - Credit Signs ("Produced By", "Eclipse", "Sony", "Playstation") - front and back
04 - "Paradise This Way" replaces "Keep Off Grass", Credit Signs ("Hi Ho", Waterski, "Oh No" (dup), Juggle) - front and back
05 - Maze
06 - Army
07 - Candy
08 - Duplicate of 04
09 - Castle 2, also has a LemLab-looking splitter entrance?
11 - Castle 3, mostly identical to 09 but "Keep Off The Grass" isn't replaced and has Castle splitter instead of LemLab-y
95 - Maze 2

=== LAND ===
00 - Dirt / sand
01 - Grass
02 - Black diagonal tiles
03 - Red and yellow triangle tiles
04 - Lemgo Red
07 - Chocolate
08 - Circuit board
09 - Grass 2
15 - Grass 3
17 - Chocolate 2
18 - Dirt / sand 2
28 - Blue
38 - Solid blue

=== SEA ===
00 - Quicksand
01 - Dark blue water
02 - Normal blue water
03 - Green slime
04 - Lemgo water - blue
05 - Lemgo water - red
06 - Icy water
07 - Liquid chocolate

=== ANIMOBJECTS ===
00 - Egyptian Torches (*)
01 - "Clockwork Games" Flags
02 - Flashing Lightposts (*)
03 - Rings Of Fire
04 - Lemming Flags
05 - Lemming Flags
06 - Checkered Flags
07 - Candycane Torches (*)
08 - Neon Exit Sign
09 - "Exit" Flags
11 - Pillar Torches (*)
13 - Juggling Lemmings
22 - Rocket Flame
23 - "Clockwork Games" Flags (duplicate)
90 - "Block Here"
91 - "Turn Here"
92 - "Bomb Here"
93 - "Build Here"
94 - "Build Here" (duplicate)
95 - "Bash Here"
96 - "Mine Here"
97 - "Dig Here"
98 - "Climb Here"
99 - "Float Here"

Those marked with (*) are duplicated, with the second copy being upside down.

=== INTERACTIVE OBJECTS ===
00 - Flame blower
01 - Squasher
02 - Bear trap
03 - Trampoline
04 - Spring
05 - Teleporter
06 - Grabby slime? (I don't know how to describe this, but it's a standard trap...)
07 - Laser trap
08 - Rope slide

=== SKY ===
00 - Pyramids
01 - Castles
02 - Space
03 - Trees
04 - Lemgo constructs
05 - Stone wall with arches
06 - Ruins? (usually used with Army theme in official levels)
07 - Plain sky
08 - Space Red
09 - Golf course
12 - Space, bigger stars
16 - Snowy trees
17 - Chocolate mountains
18 - Space (blue)
27 - Pinkish sky

=== WALLS ===
00 - Pyramid
01 - Castle
02 - Space
05 - Maze
07 - Pyramid 2 (seems identical to 1)
08 - Computer
11 - Castle 2 (???)
18 - Digits

For all: 255 seems to be used to signify "none", although it has no special treatment (as far as I can tell) compared to any other invalid value.


If you create copies of the respective files in GFX, you can use other numbers from 0 to 127 - but any above that don't work. Presumably, L3D loads it as a signed value, but doesn't know how to handle negative numbers so just loads nothing. I don't know if traps remain functional when doing this.
Title: Re: 3D Lemmings file formats
Post by: namida on June 06, 2019, 10:42:09 AM
Here's some interesting finds I came across while investigating the possible values for the Signs field in the PC version.

Yes, in the PC version. Keep that in mind as you look at the third one.

Here's (I think) all of these kinds of signs. The "Oh No" one that appears twice in this image, appears twice in the data. Actually it appears three times - because the entire far-right column appears twice (and the "Oh No" also appears elsewhere). The back is duplicated a total of 16 times in the data. The three in the first column in this image, due to technical limitations could not appear in the same level as the back of the sprites - though it would be pretty easy to patch that.
Title: Re: 3D Lemmings file formats
Post by: namida on June 06, 2019, 08:18:18 PM
Being an exit appears to be a special property of Block #1, which is automatically an exit on the side that (unrotated) faces Z=31, as long as that face is solid. If I copy Block #1's data to another block verbatim, it is no longer an exit. If I copy another block's data to Block 1, Block 1 remains an exit.

Block 0 remains "empty" no matter what the BLK file says. Blocks 2, 3 and 4 also seem to have special handling, though I haven't figured out what yet. I do notice block #2 is a splitter in the level I've been testing with, but I don't know if that's a fixed rule.

This is, at this stage, only based on testing with one level (specifically Mayhem 6 "Family Tree").

EDIT: Splitter does indeed appear to be an inherent property of Block #2, assuming the side faces are not solid. EDIT: No, even if they're solid - as long as they're not indestructible, if you bash away the block then the empty space acts as a splitter.



For the 3rd byte of block definitions, this is a bitwise value - I've found 0x10 makes the block nonsolid, while 0x02 sets the block as metal. I haven't figured out what the other flags do yet. Mayhem 6's splitter has 0x10, 0x08, 0x02 and 0x01 set - but I don't notice any difference when disabling 0x08 or 0x01, or enabling 0x04 (which I have seen enabled on other blocks).

The second byte in a face definition also appears to be at least partially bitwise, though I'm still figuring out how it interacts with the first byte exactly - and I haven't even looked at the third. 0x80 seems to be an "allow transaprency" flag.



Blocks #5 through #8 also appear to be unused in the majority of levels, although I haven't noticed anything special about them so far.
Title: Re: 3D Lemmings file formats
Post by: namida on June 07, 2019, 10:51:36 PM
Anything further I discover, assuming I don't want to raise it here as a matter of "can anyone else help figure this out?", will go into the "notes" folder in L3DEdit's Bitbucket repo.

https://bitbucket.org/namida42/l3dedit/src/master/notes/

These may be updated without notice as new information is discovered, but due to how Git works, you can always recover older versions of the documents.

These already contain information beyond what's listed here, so I recommend checking it out if you're interested in the formats. Of course, I'm mostly only going to pay attention to formats that are editor-relevant.
Title: Re: 3D Lemmings file formats
Post by: namida on June 15, 2019, 02:51:50 AM
Lemmings 3D stores the replay data in memory, in exactly the same format as the demos for the practice levels (and four others, that will play if you leave the main menu idle too long).

With a debugger copy of DOSBox, you can dump the memory and extract the data from it, and create your own replay files this way. Here's proof - a replay file for my custom level Space Station (https://www.lemmingsforums.net/index.php?topic=4258.msg75753#msg75753). Replace the existing REP.000 in your L3D "REPLAYS" folder with this one, boot up the title screen, sit back and wait (it takes a few minutes for demo mode to kick in).

If you don't want to actually wait for the demo mode to start, or don't have a copy of L3D handy, this tease is also available in Youtube form: https://www.youtube.com/watch?v=ZDm2kZUWSRs

It goes without saying that watching this replay, either via Youtube or via an actual copy of Lemmings 3D, will spoil the solution to Space Station.

EDIT: This has now lead to L3DReplay (https://www.lemmingsforums.net/index.php?topic=4273.msg75758#msg75758).
Title: Re: 3D Lemmings file formats
Post by: Pooty on June 17, 2019, 02:48:41 AM
I have learned the following information about the game's water.

Code: [Select]
0x118   Water direction towards -Z (signed byte)
0x119   Water direction towards -X (signed byte)
0x11A   {Still unknown}

0x11B (flags):
00000000 = Lemgo-style rendering, animated
00010000 = Lemgo-style rendering, not animated
00100000 = Other-style rendering, no animation
00110000 = Seems to be the same as the previous one.
Title: Re: 3D Lemmings file formats
Post by: namida on June 18, 2019, 09:19:16 AM
I've looked at the remaining unknown bytes (or bit flags, in some cases). Some of them really have no pattern, and we might have to observe closer. However, there's a lot that seem to be the same on all but a few levels.

I'm going to list a few groups of levels, that share currently-unknown values in the data with each other, but different from all levels outside the group. To be very clear - each group is independent. And I realise this is a long shot, but also figured it doesn't hurt to ask.

And just to be clear - it's entirely possible some (or even all) of these are just huge nothingburgers, possibly remnants from the official editor or of features that were removed in the final game, or something like that. So please don't spend too much effort on thinking of it - this is more a "if anything comes to mind, let me know".

Group 1: "Bounce Bounce", "Play Time"
Group 2: "A Shortcut Through The Forest", "Play Time", and all practice levels
Group 3: "This Is The Army", "Stilt Walkers" and "Let The Race Begin"
Group 4: "Alilemms", "Rescue Team" and "The Hanger"
Group 5: All enclosed levels, plus "The Prisoner" and "Claustrophobic" (these last two still have the cameras in enclosed areas, so perhaps it's some setting that relates to that..)
Group 6: "The Bean Machine", "A Shortcut Through The Forest", "Hole in Ten", "Fore!", "Lemlab", "Spaghetti Junction", "Ricochet", "Motherboard", "Birthday Cake", "Brechin's Staircase", "The Hanger", "The Catacombs", "Hole In One, Two, Three", and all practice levels
Group 7: "The Bean Machine", "Alpine Assault Course", "Alilemms", "Jelly Climber", "Oh No! More Pyramids!", "Garden Maze", "King Coder's Tomb", "Rescue Team", "The Arena", "Snake", "The Five Arches", "Brechin's Staircase", "Ski Jump", "Family Tree", "Raiders of the Lost Lemming", "Jelly Belly Islands", "Castle Peralus"
Group 8: "Attack Of The 50ft Lemmings", "Jelly Climber", "A Head Above The Rest", "Play Time", "The Arena", "T Minus 5 And Counting"
Group 9: "A Shortcut Through The Forest", "Stilt Walkers", and all practice levels
Group 10: "Bounce Bounce", "It's A Classic", "Lemlab", "Rescue Team" and "Tuther Circus Level" (with "Picky Platform" having a unique third value, instead of the same one as all other remaining levels)
Group 11: "The M-A-Z-E", "Rescue Team", "Team Work", "Death Slide" and "Slippery"
Group 12: "Alpine Assault Course", "Slippery Maze", "The M-A-Z-E", "Ski Jump" and "The Hanger"

Additionally, can anyone think of any attributes unique to any of the following levels?
- "Take A Dive"
- "Bounce Bounce"
- "Jelly Climber"
- "Lemlab" (other than the laser trap)
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 08:22:14 AM
The save file (C:\LM3D.CD\LM3D.SAV) has a checksum or CRC of some kind - the first two bytes of the file. RevEng (http://reveng.sourceforge.net/) was not able to crack it, although it's very possible I just wasn't using it correctly. I'm wondering if anyone has any ideas here?

I could possibly try and get some CPU logs from DOSBox, if that helps. I have a fair idea of when L3D calculates this checksum (but I wouldn't know the first thing about understanding the CPU instructions that it follows to calculate it).

Beyond this checksum, the format of the save file looks pretty simple, so reading it won't be hard - but I'd like to be able to write it (so I can create a proper "all levels unlocked" save - using the actual level titles, where custom levels are in play - instead of relying on one created via the RASPUTIN cheat).
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 09:27:03 AM
I could possibly try and get some CPU logs from DOSBox, if that helps. I have a fair idea of when L3D calculates this checksum (but I wouldn't know the first thing about understanding the CPU instructions that it follows to calculate it).

I haven't worked with DOSBox's debugging features, but if the CPU logs shows executed CPU instructions in assembly language format I could probably read it, though to be practical the log output needs to be loop-aware (it gets a bit annoying to read as you can imagine, if I have to manually skip through, say, 10000 iterations of some sequence of instructions executed in a loop producing that many lines in the logs).  So get the logs and we'd see how things go.  Alternatively, if the CPU logs help show the general memory addresses where the suspected CPU instructions are located, and DOSBox has a disassembly feature to convert the bytes in a memory region into assembly code, that will probably work better for me.  (And even without a disassembly feature, if DOSBox can at least dump the raw bytes of the memory region, I could load that into a debugger to look at disassembly.)

The checksum might not be CRC per se but just some other algorithm.  It can really be anything that's mildly tricky (or maybe not even that) to reverse engineer.  Honestly I'm kinda surprised they even bothered, it's just a savegame in a singleplayer game that already has a cheat code anyway.
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 09:31:39 AM
Here's a small sample of what comes up in the CPU logs (NOT actually from generating the checksum; this is just in general what logs look like). Unfortunately, it's meaningless to me, so I have no idea if it will be of any help.

Code: [Select]
1EA3:00005686  jnc  000056AF ($+27)            (down)                 EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000B92  call 00000CEE ($+159)                                  EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000B92  call 00000CEE ($+159)                                  EAX:00000000 EBX:00000BC9 ECX:00040000 EDX:00000000 ESI:0000C8DA EDI:000071B0 EBP:00000400 ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:0 PF:1 IF:1
01ED:00000D2F  movzx ax,[0B34]                 ds:[0B34]=120F         EAX:00001D92 EBX:0000000D ECX:00040000 EDX:0000005C ESI:0000099E EDI:000071B0 EBP:00005C2D ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000D4D  movzx ax,[0B37]                 ds:[0B37]=1B04         EAX:00003387 EBX:00000018 ECX:00040000 EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000D84  pop  cx                                                EAX:0000149D EBX:0000001E ECX:00040000 EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B95  jnc  00000BBB ($+24)            (no jmp)               EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B95  jnc  00000BBB ($+24)            (no jmp)               EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000B97  mov  ax,[233C]                  ds:[233C]=002F         EAX:0000149D EBX:0000001E ECX:0004002F EDX:00000016 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:1 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
1EA3:000055C8  push si                                                EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1EA3:000055C8  push si                                                EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EA DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
01ED:00000BAF  popa                                                   EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BAF  popa                                                   EAX:00000021 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007EE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BB0  bts  word [2452],06             ds:[2452]=0180         EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BB6  mov  si,0B54                                           EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000A4C EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:00000BC7  mov  [0B14],si                  ds:[0B14]=0B65         EAX:00000003 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007FE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:1 OF:0 AF:1 PF:1 IF:1
01ED:000018D3  mov  al,[C97A]                  ds:[C97A]=0000         EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
F000:00001440  push ds                                                EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F2 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:0
F000:00001460  callback 0023  (int 74 ret)                            EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007CE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
F000:00001464  cli                                                    EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007CE DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
01ED:000018D3  mov  al,[C97A]                  ds:[C97A]=0000         EAX:00000000 EBX:00000014 ECX:0004002F EDX:00000007 ESI:00000B54 EDI:000071B0 EBP:0000161C ESP:000007F8 DS:10D3 ES:10D3 FS:0000 GS:0000 SS:2622 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1

The debugger build of DOSBox can certainly do memory dumps, and can also set watchpoints to break if a certain memory address is changed. I've managed to locate a copy of the save file in memory, although it appears this is only used by the game when loading / saving the file - still, this should at least be useable to break at the right time (via memory watchpoints).

The memory address where the file's contents can be found is 0x65460. I don't know what that translates to in terms of segments / etc; I just know that it's consistently at that address in a full memory dump, both when dumped via the debugging build of DOSBox and when (in standard DOSBox) dumped via a modified version of L3DReplay that extracts the entire memory instead of just the replay data.

Here's a CPU log. The checksum should be recalculated (at least) twice during these logs, unless they were too far back to be logged.

L3D is quite inefficient about this. When you complete a level, it first writes your new records for that level, then calculates the checksum. Then, it unlocks the next level, and calculates the checksum again. It also appears to write to file both times.
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 06:35:19 PM
At the moment I'm not confident that I can make good use of the CPU logs.  Normally the leftmost column would be the memory address of the instruction being executed, but in the logs they don't line up with the jumps.  Maybe in the logs they have more to do with the location of instruction data as allocated by the emulator, rather than the instruction addresses as seen in emulation?  It might help if there's documentation on how to read the DOSBox CPU logs but I'm not finding any so far from a web search.

The best bet would probably be to find some way to locate the executed instructions in memory, do a memory dump, then use an external tool to disassemble the raw memory dump back to assembly code.  At work right now, will update later on how one might go about this.
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 08:28:53 PM
I wouldn't have a clue how to locate the executed instructions, but I can get some memory dumps for you - I'll do this a bit later.
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 09:18:53 PM
Further probing of web search results found a potential answer to the CPU logging issue:

https://www.vogons.org/viewtopic.php?f=32&t=7323&start=20#p162190

I assume the "core=normal" setting mentioned there is one of the settings in DOSBox.conf.  Sounds like you need to set that in order to ensure the CPU log is complete.  Can you try that and upload the resulting CPU log?  It's likely DOSBox will run (even) slower with the combination of that setting plus CPU logging enabled, but it's necessary.

It's still not ideal that the CPU log, as I feared, is just a linear sequence of executed instructions, but I can still try to work with that.  For now you can hold off on making memory dumps.

One more thing:  if you are able to set a data breakpoint on when the checksum memory location will be written to, do that and capture the CPU log at the time it hits that breakpoint.  Also, in case the log doesn't include the instruction for the breakpoint-triggering write (depends on whether it breaks before or after the write completed), copy whatever's displayed in the debugger window at the time the breakpoint is hit, I assume it should show the instruction being executed that triggered the data breakpoint.  Copy that info to end of CPU log.

[sidenote: for the memory dump thing I was requesting but holding off for now, you would take the address of the instruction the data breakpoint was hit, and grab maybe around 32000 bytes before and after the that address as the range to dump]
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 09:26:24 PM
Here you go. This should have the right info - I do notice that EAX's value is that of the checksum, and that it changes (through many different values) before the breakpoint (which is set to fire when the memory byte at 0x00065460 changes).

Getting a memory dump is trivial (the logs are much trickier) so I included one of those at the time of the breakpoint too, though it's just the entirity of the virtual memory space (or at least - it starts from the beginning of said space, and goes well beyond anything I've seen L3D actually use). I can also do memory dumps by segment / offset, if that's any help, but I would be clueless as to what the segments actually are or how they work so you'd need to be able to tell me exactly what values I'm using.

Quote
It's likely DOSBox will run (even) slower with the combination of that setting plus CPU logging enabled, but it's necessary.
No slower than usual for the debugger build of DOSBox, it seems. DOSBox with default settings is a bit slow when playing L3D in general, but a few config changes sorted that out (the biggest one being setting cycles to "max" - any specific value seems to either be laggy or have sound glitches, but "max" works perfectly).
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 09:58:24 PM
Ok, I started off at the end of the log since presumably the log ends at the point where the checksum value is written to memory.  Based on that, tell me if what I said below tracks with the savegame data being processed at the time:

Code: [Select]
1E58:0000745F  push fs                                                EAX:00000000 EBX:00001600 ECX:00000000 EDX:00000007 ESI:00000020 EDI:000003EE EBP:08100014 ESP:000007EE DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:00007461  mov  fs,[3D00]                  ds:[3D00]=6546         EAX:00000000 EBX:00001600 ECX:00000000 EDX:00000007 ESI:00000020 EDI:000003EE EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:00007465  mov  di,0002                                           EAX:00000000 EBX:00001600 ECX:00000000 EDX:00000007 ESI:00000020 EDI:000003EE EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:00007468  mov  cx,0742                                           EAX:00000000 EBX:00001600 ECX:00000000 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:0000746B  xor  ax,ax                                             EAX:00000000 EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1

1E58:0000746D  add  ax,fs:[di]                 fs:[0002]=5551         EAX:00000000 EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:1 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:00007470  ror  ax,03                                             EAX:00005551 EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
1E58:00007473  xor  ax,cx                                             EAX:00002AAA EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1
1E58:00007475  add  di,0002                                           EAX:00002DE8 EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000002 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:1 IF:1
1E58:00007478  loop 0000746D                                          EAX:00002DE8 EBX:00001600 ECX:00000742 EDX:00000007 ESI:00000020 EDI:00000004 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1

1E58:0000747A  mov  fs:[0000],ax               fs:[0000]=6313         EAX:00006DF2 EBX:00001600 ECX:00000000 EDX:00000007 ESI:00000020 EDI:00000E86 EBP:08100014 ESP:000007EC DS:1088 ES:1088 FS:6546 GS:2658 SS:25D7 CF:0 ZF:0 SF:0 OF:0 AF:0 PF:0 IF:1

1) So the checksum in hexadecimal is 6313, ie. first byte (hex) is 13 and second byte is 63?
2) In the loop, according to the logs the following values were the first few processed into the checksum calculation:
Code: [Select]
fs:[0002]=5551
fs:[0004]=4441
fs:[0006]=4F43
fs:[0008]=4552
fs:[000A]=0000

Translating to individual bytes, they would be (in hex):

51 55 41 44 43 4F 52 45 00 00

================

If above tracks with the savegame data, the next step is simply to translate the above calculation (from 1E58:00007465 to 1E58:00007478) from assembly to some pseudolanguage which you can understand well enough to implement yourself.
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 10:00:34 PM
Those are indeed the first 10 bytes (excluding the checksum) of the save file data. Excluding the 2-byte checksum, the entire size of the save file is 0x0E85 (3717) bytes - as far as I can tell, this never varies, although I don't know for sure that every byte is used.

Regarding pseudocode: I understand the basic bitwise operations (eg. AND, OR, XOR, NOT, bitshifts), but not the more advanced stuff like... I think "bit rotate" was one I heard of? If the pseudocode resembles any of BASIC, Pascal or C#, I should be able to understand it. Even if not, it's all pretty similar really, so I should still be able to make sense of it.
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 10:19:21 PM
Yeah, ROR is bitwise rotate right.  Anyway, here's pseudocode for above:

Code: [Select]
count = 1858
checksum = 0

do
   checksum = checksum + <next 2 bytes read as a 16-bit little-endian word>
   (equivalently:  checksum = checksum + <next byte> + <next next byte> * 256)
   checksum = checksum AND 65535   (ie. keep only lower 16 bits)
     
   checksum_last3bits = checksum AND 7
   checksum_shifted3bitsright = checksum div 8  (integer division dropping remainder, so 7 div 8 = 0, 8 div 8 = 1, 9 div 8 = 1)
   checksum_rotatelast3bitsintofirst3bits = checksum_last3bits * 8192
   checksum = checksum_shifted3bitsright OR checksum_rotatelast3bitsintofirst3bits

   checksum = checksum XOR count
   count = count - 1
while (count > 0)

The business in the middle is implementing the bitwise rotate with more commonly supported bitwise operators in high-level languages.  In case bitwise shifts are not available, I did those above using integer division and multiplication instead, you're welcome to just use shifts if they are available in your language (right shift by 3, and left shift by 13).

Based on the code, it expects 3716 bytes after the 2-byte checksum.  Maybe it's an off-by-1 error that the file size (excluding checksum) is 3717 rather than 3716?  The checksum algorithm is hardcoded to process exactly 3716 bytes.

[edit: fix missing the discarding all but lowest 16 bits after the add]
Title: Re: 3D Lemmings file formats
Post by: namida on June 24, 2019, 10:21:49 PM
Ah, so rotate is essentially just a bitwise shift that wraps around? That's simple enough then. I don't know if FreePascal supports it, but that's very easily replicated anyway.

EDIT: To clarify, when doing this step: "checksum = checksum + <next 2 bytes read as a 16-bit little-endian word>" - I assume we're immediately discarding anything above the 16th bit?

EDIT: Answer to the above question is "yes", and it works - I've successfully written code that repairs the checksum on a save file with an incorrect one. Next step, of course, is to actually generate the all-levels save file. :)

Here's the relevant Pascal code, in case that's more readable to anyone else reading this topic in the future. I've stripped out the parts that just relate to selecting the save file via an Open File dialog and actually opening / closing the file stream, I assume anyone who's familiar with Pascal will have little difficulty doing this.
Code: [Select]
var
  F: TFileStream;
  Buffer: array[0..3718] of Byte;

  Checksum: Word;
  RemainingCount: Word;
  WordPtr: ^Word;
[...]
        F.Read(Buffer[0], 3719);
        WordPtr := @Buffer[2];
        RemainingCount := 1858;
        Checksum := 0;

        while (RemainingCount > 0) do
        begin
          Checksum := (Checksum + WordPtr^) and $FFFF;
          Checksum := ((Checksum and $FFF8) shr 3) or ((Checksum and $0007) shl 13);
          Checksum := Checksum xor RemainingCount;
          Dec(RemainingCount);
          Inc(WordPtr);
        end;

        F.Position := 0;
        F.Write(Checksum, 2);
Title: Re: 3D Lemmings file formats
Post by: ccexplore on June 24, 2019, 11:21:37 PM
EDIT: To clarify, when doing this step: "checksum = checksum + <next 2 bytes read as a 16-bit little-endian word>" - I assume we're immediately discarding anything above the 16th bit?

Ah yes, sorry, I forgot about that part with the add, that the mathematical sum can have more bits than the operands.  The CPU is running in 16-bit mode so AX is only 16 bits, thus you only keep lower 16 bits of the sum.  Will edit my post above for clarity.
Title: Re: 3D Lemmings file formats
Post by: namida on July 09, 2019, 07:35:58 AM
With the one exception of BGRD.000, all files with a number as the extension in Lemmings 3D Winterland's GFX folder, are either unique to Lemmings 3D Winterland, or are identical to the same-name file in the full verison of Lemmings 3D.

This means that if you want to use the Lemmings 3D Winterland graphics in custom levels, you can safely just copy all the numbered files (except BGRD.000) into regular Lemmings 3D - and yes, they work! :) (I'll add the relevant presets in the next L3DEdit update, for now, you'd have to enter the numbers manually.)

No numbers in the 60s, 70s or 80s are used for graphic files by either regular or Winterland L3D.
Title: Re: 3D Lemmings file formats
Post by: namida on July 10, 2019, 01:05:12 AM
Here's a minorly interesting discovery: The 4th level in Lemmings 3D Winterland, instead of having the author's name in the comment field, has the text "EARLY TRICKY". I don't recall this level's solution, so I'm not sure off-hand how accurate that is, but it's what's there.
Title: Re: 3D Lemmings file formats
Post by: namida on July 11, 2019, 12:09:30 AM
And now an actual feature: The byte at 0x0157, known for the 0x01 flag marking the bottom of the level as slippery (if combined with the "bottom of level is solid" flag), also has an effect on the 0x02 flag - this makes land areas slippery. This single flag affects all land areas, I don't believe there's a way to set it on a per-land basis, though I'm going to investigate a bit more. (EDIT: If this is configurable per-land, it's not as part of the known land data, nor part of different flags in the 0x0157 byte.)

This is particularly interesting, as this is a feature that isn't used in any official levels, not even in Winterland. The levels that have ice at the bottom of them (Alpine Assault Course, Slippery Maze, Ski Jump, Snowy Islands and Arctic Obstical Course) all have a sea graphic that resembles ice, and use the 0x01 flag alongside "bottom of level is solid" - and don't contain any land areas full stop. (In fact - if you place land areas in a level with that combination of flags, but not the 0x02 "land is slippery" flag, the land areas are exempt from the slipperyness. Since none of the above levels have that flag set, land will be non-slippery on them.)
Title: Re: 3D Lemmings file formats
Post by: namida on July 22, 2019, 01:31:05 AM
It seems all but certain that the byte at 0x0140, known to have an effect on the amount of faces that can be rendered, is actually part of a two-byte value (0x013F-0x0140). It in turn seems that the range of accepted values here are 0x000F to 0x044C. All values outside this range are identical, but I'm not sure exactly how it's calculated - I thought it was "always 0x0300, or something close to it", but that might just be the cap rather than an exact value, due to what I've observed re: replay breakage (see below).

EDIT: Values above 0x0439 reduce the maximum number of lemmings you can have in the level (actually, even 0x0439 and a bit below probably does, but since you shouldn't have more than 80, and 80 lemmings works with a value of 0x0439...). At 0x044C, the limit is 65 lemmings.

It also seems at a glance - though I didn't check this too in-depth - that the rule "0x013E = 0x0140" in official levels, extends to this as a two-byte value, ie: in official levels, the two bytes 0x013D-0x013E, always have the same value as the two bytes 0x013F-0x0140. I still have not found any effect of changing 0x013D and/or 0x013E in a level, though.

I've also discovered, that replays working seems to depend on whether the value is above or below 0x02AC. Take this with a grain of salt, as this is based on testing a single level with a single replay. If a replay is recorded on a level where the value is 0x02AC or above, then the level is modified to make this value below 0x02AC (or vice versa), the replay breaks. This explains the replay breakage I saw with L3DEdit - all were in cases where I modified a level created with an old version of L3DEdit, using a newer version. Older versions left 0x0000 in these bytes, while newer ones write 0x0400 (which will be changed to 0x044C 0x0439 in the future, if I don't find any replay breakage or other negative side effects of such a change). EDIT: Based on further testing, there seems to be nothing special about 0x02AC outside of the context of that particular level. I've seen breakage on a different level when changing from 0x0400 to 0x0439. Assume any change to this value can break replays.
Title: Re: 3D Lemmings file formats
Post by: namida on July 23, 2019, 05:06:41 AM
On further investigation, it appears any change whatsoever to 0x013E can break replays (I had one break when changing a level from 0x0400 to 0x0439) - so I'll have to consider what's more important, pushing the limit very slightly further (and this is on the order of maybe 10 or so extra blocks, I believe), or not breaking replays (which to be fair, there aren't all that many of yet).

Of course, Option C is "preserve existing value for existing levels unless the user specifically asks to change it, use 0x0439 for any new levels"... I might go with that one.
Title: Re: 3D Lemmings file formats
Post by: namida on August 06, 2019, 02:25:08 AM
It appears the interaction between the face render limit and the lemming count is not as straightforward as I thought. I've had a level where, with a face render limit of 0x0439, only 78 lemmings were allowed to spawn. Additionally, decreasing this to 0x0437 - which I would've expected to allow 80 lemmings again - only allowed 79. This is on a level that previously worked fine with 80 lemmings @ 0x0439, and has less blocks now - but a couple more objects, and lots of small lands instead of one large one.

Given this, I'll likely leave L3DEdit's default at 0x0400 (1024), which I have never seen cause problems, but with that being said the next update allows editing this value anyway. :)
Title: Re: 3D Lemmings file formats
Post by: namida on October 16, 2019, 03:26:31 AM
I've started a TCRF page (https://tcrf.net/3D_Lemmings_(DOS)) for Lemmings 3D PC version, and put up some of the most interesting finds there. Would be great if others could help complete it. :)
Title: Re: 3D Lemmings file formats
Post by: ccexplore on October 16, 2019, 10:44:21 AM
Hmm, really, no TCRF page for DOS Lemmings 1? :o I'm not sure but I feel like I remember seeing some hidden text in the game's EXE as well as in adlib.dat.  Maybe I'll start the page if I can confirm.  If nothing else, there is at least that one unused trap in the blue Crystal style that we all know about.  Oh, and all those flag objects across all 5 styles that are unused due to no 2-player mode on DOS.
Title: Re: 3D Lemmings file formats
Post by: ccexplore on October 16, 2019, 07:45:17 PM
Hmm, really, no TCRF page for DOS Lemmings 1?

Alright, I went ahead and created that page (https://tcrf.net/Lemmings_(DOS)) now.  I didn't realize how much of a pain it is to register and then create the page from scratch, maybe that's why no one bothered.  I hope to find more to add there soon.
Title: Re: 3D Lemmings file formats
Post by: namida on July 14, 2021, 07:55:45 AM
TITLE.MHC's format isn't documented anywhere.

Code: [Select]
No header. File immediately begins with first cell.

CELL STRUCTURE:
0x00 ~ 0x01 -- Size of cell data (including these bytes)
0x02 ~      -- Image data

Image data is usual format, except any instance of 00 is repeated a number of times indicated by the following byte. If the quantity is 00, the number of times is "until the end of the current line".

First attachment is L3D's TITLE.MHC file, second attachment is Winterland's.