Author Topic: Lemmings 2 Data Formats  (Read 18034 times)

0 Members and 1 Guest are viewing this topic.

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Lemmings 2 Data Formats
« on: December 01, 2021, 12:22:35 PM »
I'm currently working on something similar to the Lemmix player for Lemmings 2: The Tribes. (MS-DOS Version)

It's in the early stages so far, so still working on parsing the various file formats.

I have managed to fully parse and load the following data types; (With help from various notes/docs by geoo89 and GuyPerfect - Much appreciated)

- IFF Files
- RAW Bitmaps
- Styles
- Levels
- Non IFF Form files
- SBLAST.VOC
- All UI related files

I was wondering if anyone had done any work parsing .ANM files? (e.g. ARK.ANM)
Any info, would be greatly appreciated :)

Offline WillLem

  • Posts: 3523
  • Unity isn't sameness, it's togetherness
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #1 on: December 02, 2021, 01:57:23 AM »
This sounds like a great idea! Will it be a simple Windows-ready hosting of the OG, or will you be adding QOL features (such as replay capabilities) to make it more accessible?

Apologies that I can't offer more than encouragement at this stage!

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #2 on: December 02, 2021, 09:15:59 AM »
The goal is to initially host the OG. However, all of the data is unpacked, converted then cached into something more friendly and modern to facilitate easy modding/addition of styles/levels/animations/sound and graphics data etc.

The player and tools will be windows initially and run on .NET Core. Aside from a thin wrapper around OpenGL to push the frame buffer, it's all managed. All rendering is done in a custom and fast blitter.

As for QOL elements, I am open to suggestions. I like the idea of storing replays :) (I'm already considering multiple fast-forward speeds for the impatient of us)

Now that I have all of the data caching working, I am underway with the player.

Progress: (A little rough around the edges)

- Menu/UI rendering
- Level rendering inc. objects with sub-components
- Rudimentary AI (They just get a random animation on spawn currently - red dot is collision)

« Last Edit: December 02, 2021, 09:26:40 AM by ravenix83 »

Offline namida

  • Administrator
  • Posts: 12422
    • View Profile
    • NeoLemmix Website
Re: Lemmings 2 Data Formats
« Reply #3 on: December 02, 2021, 08:53:48 PM »
Are you intending to replicate L2's physics exactly, or just to create a game that feels the same to the user (but might not exactly match how L2 works)?

Just because - if it's the latter case, I would suggest comparing your project to Lemmini rather than Lemmix, as one of Lemmix's most defining features is that it attempts to reproduce L1's physics exactly. (NeoLemmix and Lix would be even worse comparisons due to the additional features in the former, and open-source-overhaul nature of the latter.) That's not to knock it in any way - an L2-like game with QoL features and playable on modern platforms could definitely breathe some new life into the game - I'm just suggesting a wording that might give people a more accurate idea of what you're aiming for.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #4 on: December 02, 2021, 10:01:23 PM »
It's a fair point. My intention from the start is to recreate the physics exactly (As far as possible). As to if I achieve that, is another matter entirely, given the number of skills to replicate. Currently using frame-by-frame analysis to determine physics.

The last thing I want to do is misrepresent or mislead.

How would you suggest I categorise it, if exact physics replication is a primary goal but not a certainty?


Offline geoo

  • Administrator
  • Posts: 1481
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #5 on: December 03, 2021, 08:29:24 PM »
Exciting to hear about this! I'm afraid I don't have any more information about the .ANM file format.
Is this needed for anything beyond the intros? If I recall, Lemmix started out as an editor with a playtest function, which I think is really what the community needed back then considering the focus has always been custom content creation, and the LemmixPlayers only came after (I might be wrong about this though).

Quote
It's a fair point. My intention from the start is to recreate the physics exactly (As far as possible). As to if I achieve that, is another matter entirely, given the number of skills to replicate. Currently using frame-by-frame analysis to determine physics.
I believe Lemmix was able to replicate the L1 physics exactly by using a disassembly and translating parts of it into Pascal, basically. I guess EricLang and ccexplore would be the people to know more about this. I don't know if anyone looked into disassembling L2, if anyone, I guess ccexplore would be the person to ask, though he hasn't been active in a while.

The main value of this is that people could use your L2 replica to do valid L2 challenges (e.g. save max amount of Lemmings, or use the minimum amount of skills to solve a level) with the benefit of the QoL features, as has happened with Lemmix in the past. At the same time, L2 has a whole bunch of weirdness that you might not want to replicate (and without a disasm probably wouldn't be able to replicate): For example, pausing can get lemmings stuck in cannons; you can pause in 1-frame intervals to bypass trigger areas of traps (e.g. to save 100% in Sports 1); the rules determining which lemming under the cursor is assigned a skill are dumb, etc...

So unless you want to replicate all of this, it might make sense to make some deliberate choices to get rid of some glitches along the way (e.g. use divers or hoppers to glitch into a wall).

As for QoL features: Replays are definitely a big thing for any community creating custom content. Multiple fast forwards sound useful too. Some more challenging features (that were definitely well appreciated when added to other engines) are savestates and going back 1 frame/second/some other interval. (Taking them into account already early during the design process might make them easier to implement in the future.)
Loading up individual levels via commandline parameters (and batch verifying validity of replays, if replays are implemented) should also prove useful for level designers. Might also be worthwhile checking in with kaimitai who was working on a new level editor/modding tool: https://www.lemmingsforums.net/index.php?topic=5769.0

Quote
The player and tools will be windows initially and run on .NET Core. Aside from a thin wrapper around OpenGL to push the frame buffer, it's all managed. All rendering is done in a custom and fast blitter.
Any chance this might reasonably run on Linux, e.g. using mono? Even within this forum, it feels like L2 is somewhat niche, and at least two people who appreciate L2 are using Linux :)

Either way, excited to hear about how this is progressing! Not sure I can be of much help, as I think you've already found all the documentation that is out there, but maybe I have some other stuff lying around or some obscure knowledge that occasionally could be useful...

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #6 on: December 03, 2021, 09:03:39 PM »
The .ANM files are for the intros and seem to be keyframe based. I've done about 30% of the work for it. But, as it's not critical, it's moved down the to-do list :)

I have been reading into and watching videos on known glitches. From my own experience there are quite a few undesirable quirks. I think you maybe right in that it would be better to make some deliberate choices about what is replicated.

Savestates are on the list. Not sure about rewinding, but it can go on the nice to have list :) (Savestates would help with that in the near term)

As for Linux support, it would work as its .Net Core 3.1 which works on most distros. I think the only snagging point would be the audio library. (I'd be happy to find a suitable cross-platform alternative, just need to dust off my Ubuntu VM)

Apologies, but the link you referenced is discussing L2 style limitations. Is this the right link?

I'm now at a point where all style, level objects and animations are working ( Just chains to go :) )

Offline geoo

  • Administrator
  • Posts: 1481
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #7 on: December 03, 2021, 09:25:42 PM »
Wow, you're making fast progress!

If you have questions about any specific glitches or behavior, feel free to ask!

As for Linux support, it would work as its .Net Core 3.1 which works on most distros. I think the only snagging point would be the audio library. (I'd be happy to find a suitable cross-platform alternative, just need to dust off my Ubuntu VM)
Nice, happy to hear that! ;)


Apologies, but the link you referenced is discussing L2 style limitations. Is this the right link?
Yes, in the topic kaimitai mentioned they were working on an editor/modding tool. I don't know if this still progressing or on ice, but I just thought it might make sense to make sure not to duplicate efforts or write incompatible tools.

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #8 on: December 04, 2021, 12:31:32 AM »
Thank you :)

Any info you have on known glitches would be fantastic. I have pulled pieces together, but don't like to miss stuff ;)

I have managed do an initial implementation for the chain simulation. Just making sure it's stable at any simulation speed.
« Last Edit: December 04, 2021, 12:44:38 AM by ravenix83 »

Offline namida

  • Administrator
  • Posts: 12422
    • View Profile
    • NeoLemmix Website
Re: Lemmings 2 Data Formats
« Reply #9 on: December 04, 2021, 06:38:15 PM »
Quote
I believe Lemmix was able to replicate the L1 physics exactly by using a disassembly and translating parts of it into Pascal, basically. I guess EricLang and ccexplore would be the people to know more about this. I don't know if anyone looked into disassembling L2, if anyone, I guess ccexplore would be the person to ask, though he hasn't been active in a while.

It was more like a clean room implementation - ccexplore used disassembly to figure out the exact physics, then described these to EricLang, who then implemented them with his own code.

This did initially lead to a couple of inconsistencies between DOS and Lemmix. Later patches by ccexplore and myself fixed some of these ("fixed" in the sense of "made Lemmix behave the same way as DOS") - most notably the nuke glitch but two other particularly notable ones are the pause-for-time glitch, and an inaccuracy in how long traps take to reset.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #10 on: December 04, 2021, 10:28:14 PM »
In that case, seeing as I'm not familiar with DOS era assembly, I will be relying on frame analysis and using that to derive physics behaviour. So, to clarify on your original point. It is probably safe to say it won't be wise for me to frame it as being like Lemmix if exact accuracy derived from asm, was its defining feature.

Also, I will be looking to avoid the original L2 glitches.

For geoo - Removed all Windows dependencies. Now running on Linux (Surprisingly with higher framerates :thumbsup: - capping at 5k instead of 4.5k with vsync off)

Only dependencies are OpenGL 4.1 and OpenAL.

Offline Simon

  • Administrator
  • Posts: 3938
    • View Profile
    • Lix
Re: Lemmings 2 Data Formats
« Reply #11 on: December 05, 2021, 07:44:01 AM »
I'm the second Linux user with interest in L2.

Lovely project, I wish you the best as you push it forward. L2 tooling looks daunting to create, there are so many tiles, skills, ..., to support, and rewriting the L2 game is the holy grail.

Physics: I have no strong opinion on how close to DOS L2 is best. As soon as you change the behavior, it all becomes a question of taste anyway. Ultimately, one person has to decide what bug(fix) goes it and what doesn't. Let's see where this project takes you.

Quality of life: Replays and deterministic physics are good. Players like to share their replays with the level designers, so they can fix backroutes. Even during realtime play, consider to put all user actions into an internal replay first, and only then let the game display the outcome of that replay.

Savestates and replays are the ideal first quality-of-life features to make. Both together then lead to backwards framestepping: Lix and NeoLemmix implement backwards framestepping with frequent short-term savestates for performant short physics rollback (loading a savestate from ~1 second ago, then recompute forward to the desired frame using an internal replay), and more spaced-out long-term savestates to support occasional longer rollbacks.

Have you published the project source code? I'd be happy to compile and run it on Linux. Some years ago, I've compiled Nepster's NeoLemmix editor in C# with Mono. Thus, even if you haven't written any build documentation yet, I might get somewhere to give you feedback.

-- Simon
« Last Edit: December 05, 2021, 08:30:58 AM by Simon »

Offline namida

  • Administrator
  • Posts: 12422
    • View Profile
    • NeoLemmix Website
Re: Lemmings 2 Data Formats
« Reply #12 on: December 05, 2021, 05:50:49 PM »
Quote
Lix and NeoLemmix implement backwards framestepping with frequent short-term savestates for performant short physics rollback (loading a savestate from ~1 second ago, then recompute forward to the desired frame using an internal replay), and more spaced-out long-term savestates to support occasional longer rollbacks.

To be precise, in NeoLemmix's case, it creates a state every 10 seconds (as measured by the in-game clock, which in turn operates on the basis of 1 second = 17 frames), including at frame zero (because loading a state is much quicker than re-rendering / etc the level from scratch). Each time a state is loaded, all states newer than it are cleared. NeoLemmix will discard most states that are over one minute old, but will keep the ones that were saved on a xx:00 or xx:30 mark. Once states become three minutes old, they're only kept if they're on an xx:00 mark (these ones are never discarded due to age).

I believe Lix's algorithm is more complicated, primarily because whereas NL's was designed from the ground-up for singleplayer, I believe Lix's was originally designed for sync purposes in online multiplayer and then adapted to also help with single-player backstepping at a later date.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline RavenNine

  • Moderator
  • Posts: 58
    • View Profile
Re: Lemmings 2 Data Formats
« Reply #13 on: December 06, 2021, 11:43:09 AM »
@Simon - Thanks for the support. As for source code, it's currently not published. However, it is built as a self contained package including .NET runtime that will "just work" as long as OpenGL 4.1 and OpenAL is available on the system. Mono isn't used. Instead it is .NET Core 3.1. Testing how well this method works on other distros will be useful. Currently running on Ubuntu 20.04.

@namida - That sounds like a more sensible option. In terms of performance and reliability, I think 10 second snapshots are going to be the way to go as it needs to store the state of the world bitmap.

I have only just recently join this community and I'm finding it very welcoming and supportive :)

Offline namida

  • Administrator
  • Posts: 12422
    • View Profile
    • NeoLemmix Website
Re: Lemmings 2 Data Formats
« Reply #14 on: December 06, 2021, 04:56:45 PM »
Quote
as it needs to store the state of the world bitmap.

You need to store a lot more than this - basically the state of any in-game object, of the terrain map, and of attributes about the game itself (like remaining time, number of lemmings saved).

You're also going to have to think about how the fan will interact with save states - the fan very, very much blurs the line between UI and gameplay tbh. (Although, can you speed the fan up while paused? If so, perhaps saving the fan's state is not necessary.)

For reference - here's a list of the things saved in NeoLemmix. I've added the comments here myself (sometimes replacing the ones from the actual source code) to explain what they're for, if they're not obvious. There's also some redundancy in here - some is for performance reasons, a lot relates to outdated code and doesn't actually need to be in here anymore but just hasn't been tidied up yet.

Quote
      LemmingList: TLemmingList; // The lemmings and their current states / positions / etc
      SelectedSkill: TSkillPanelButton; // I believe no longer actually used for anything
      TerrainLayer: TBitmap32; // Visual copy of the terrain layout
      PhysicsMap: TBitmap32; // Physics copy of the terrain layout (ie: stores things like which pixels are steel, which are solid, which are one-way, etc)
      ZombieMap: TByteMap; // NeoLemmix has a type of lemming called a "zombie", this stores a map of which pixels are infectious (due to being too close to a zombie)
      CurrentIteration: Integer; // Or basically, "time elapsed since the start of the game, measured in frames rather than seconds"
      ClockFrame: Integer; // Number of frames since last whole second. Could really just be calculated by "CurrentIteration mod 17"
      ButtonsRemain: Integer; // Another NeoLemmix feature is locked exits which require pushing buttons to unlock; this stores the number that remain
      LemmingsToRelease: Integer; // Number of lemmings that exist on the level but have not spawned from an entrance yet
      LemmingsCloned: Integer; // Number of times the Cloner skill (which creates a clone of an existing lemming) has been used
      LemmingsOut: Integer;
      fSpawnedDead: Integer; // Kinda misleading name; it's how many lemmings were already zombies when they spawned (as opposed to becoming zombies later).
      LemmingsIn: Integer; // Number of lemmings saved
      LemmingsRemoved: Integer; // Number of lemmings removed from gameplay for any reason (death or saving)
      NextLemmingCountdown: Integer; // How many frames until the next lemming spawns from an entrance
      DelayEndFrames: Integer; // Gameplay will not terminate for this many frames, even if there are no lemmings left (used when the last lemming alive is killed by a triggered trap, so that the full animation plays)
      TimePlay: Integer; // Either "number of whole seconds remaining" (if level has a time limit), or "negative of the number of whole seconds taken" (if it does not).
      EntriesOpened: Boolean; // Whether or not the hatches have finished opening
      Gadgets: TGadgetList; // The various interactive objects (like exits, entrances, traps, etc) and their current states
      CurrSpawnInterval: Integer; // "Spawn interval" is Lemmings lingo for an inverse release rate (ie: low number = fast, high number = slow). NL can present either an RR or an SI to the user (it's an option), but generally uses SI internally.

      CurrSkillCount: array[TBasicLemmingAction] of Integer;
      UsedSkillCount: array[TBasicLemmingAction] of Integer;

      UserSetNuking: Boolean; // Has nuke been activated?
      ExploderAssignInProgress: Boolean; // Are the bomber countdowns from the nuke still in the process of being assigned?
      Index_LemmingToBeNuked: Integer; // Which lemming is the next one to start the nuke countdown for?

And then - lemmings in particular have about this many fields again per lemming. Objects, not quite so much, but still a few each.
« Last Edit: December 06, 2021, 05:05:44 PM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)