Lemmings Music Formats-Sound Images, Tony Williams

[Borg_Number_One]

Hi.

Reffering to these threads / posts...:
http://eng-forum.lemmingswelt.de/cgi-bin/yabb/YaBB.cgi?board=tech_id;action=display;num=1114127516;start=15

http://eng-forum.lemmingswelt.de/cgi-bin/yabb/YaBB.cgi?board=tech_id;action=display;num=1114127516;start=30

...I decided to start a new thread to summarize the facts about the:

+ compressed files in the Lemmings games:
+ Adlib Music Format / Adlib version of the "Sound Images"-Music Format by
Tony Williams

Compression

The PC games:
+ Lemmings
+ Oh No! More Lemmings
+ X-Mas Lemmings '91
+ X-Mas Lemmings '92
+ Holiday Lemmings '93
+ Holiday Lemmings '94

seem to use the same engine/system:


1.)
All .dat files in the game directories are compressed with/in a "special" and really unknown compression method.

But there is a really new tool,
called "lemleveldecompressor",
which is able to unpack the .dat files.

http://it.travisbsd.org/lemmings/files/lemleveldecompressor.zip


Thanks go to: "ccexplore" and Mindless ( http://it.travisbsd.org/lemmings - Mindless' Homepage)

2.)
The commercial released .exe files (not the demos)
of the above listed lemmings games,
are packed generally with PKLite and/or LZ91/LZEXE91

You can use following tools to unpack the executables:

a)
+ http://fabrice.bellard.free.fr/lzexe.html - homepage of the LZEXE91 packer/unpacker
-->
http://fabrice.bellard.free.fr/lzexe/lzexe91e.zip
http://fabrice.bellard.free.fr/lzexe/lzexe91.zip

b)
+ http://www.pkzipstore.com/dr/v2/ec_MAIN.Entry16?SP=10024&PN=29&xid=39892&V1=30011352&V2=30011352&V3=1&V4=38&V5=0&S1=&S2=&S3=&S4=&S5=

PKLITE by PKWARE

Using the "-x" switch, PKLITE can also unpack executables again,
which were packed in normal way by PKLITE.

But it is not able to unpack "unextractable" PKLITE compressed files again.
-->take FS by SMT instead.

c)
+
http://protools.reverse-engineering.net

-->
http://protools.reverse-engineering.net/files/utilities/fs.zip - File Scanner by SMT
(it is able to detect and unpack many (exotic) compressed files, such as LZEXE91 and even the "extra variant" of PKLITE, which is normally unextractable.)


Lemmings' Adlib Music Format by Tony Williams

All the Lemmings Games mentioned above ("Lemmings" - "Holiday Lemmings '94")
use (slightly different versions of) a Sound/Music system by Tony Williams - the "Sound Images" Sound/Music system.
In each mentioned Lemmings game you can find a file,
called: "adlib.dat".
You can also unpack these files with the "lemleveldecompressor".
If you take a HEX/text editor and open the unpacked adlib.dat file, than you can see strings like:

+ "Sound Images"
+ "Tony Williams"


Roman Dolejsi,
the author of RDOSPLAY - a multi Adlib/FM Synthesis format player
( http://sorry.vse.cz/~roman/dimension/rdos/rdosplay/ )

could successfully rip the tunes from:
"Holiday Lemmings '93"
and he could figure out the one type/variant/version of the "Sound Images" music format,
which was used in "Holiday Lemmings '93".

http://sorry.vse.cz/~roman/dimension/rdos/rdosplay/

-->
http://sorry.vse.cz/~roman/dimension/rdos/rdosplay/rply021b.zip

-->
rply021b.zip-->files\lem


Because of Roman losts the sources of RDOSPLAY and because of he has no time anymore,
he cannot help anymore (to figure out the slightly different types/variants/versions of the "Sound Images" music format.



However, I have got news.

I reached to figure out where Tony Williams work and he was very lucky and happy to hear the news about
the efforts to figure out the sound format and
my plans about adding his "Sound Images" formats/format versions to the ADPlug project: http://adplug.sf.net.

Well, he will check its old computers, FDs, HDs, CDs, backups, etc. to send me detailed information about its developed "Sound Images" versions and the music format specifications..


I am sure, that we all will enjoy the music/tunes from the different above listed lemmings games in their original format, very soon.

[Mindless]

Cool B)

[Proxima]

All .dat files in the game directories are compressed with/in a "special" and really unknown compression method.

Even more unknown than Compression Method One?  

[Borg_Number_One]

Hi.

Today, some hours ago Tony Williams sent me all necessary stuff to add
"Sound Images (Generation 2)"-Format(s) to ADPlug.

I also informed Simon Peter (ADPlug author) and he was very happy/lucky about the news.


However, Simon studies and the next one year he has no time for ADPlug anymore.

So, I am thinking about to manage the further development of ADPlug and to add "Sound Images" support.

Simon agreed to my idea to continue the development of ADPlug.

[Mindless]

Hmm... what ever happend to this?

[Timballisto]

So, we might be able to play lemmings music without the game?  I assume we're nowhere close to editing music yet but whatever.

[ccexplore]

Technically I have already done something like it.  Since you can capture the raw commands to the sound card in DOSBox, I wrote a program eons ago to playback what was captured (you need to run it in DOSBox for it to work though however).  It works, but of course, the capture can't capture things like proper looping and such, so the music would stop playing at wherever you stop the capturing at.

Tim, I'm rather surprised you actually brought up the question about editing the DOS music.  My experience so far here indicated that very few if any people ever composed their own music from scratch for any Cheapo levels/styles!  It would seem that most people download their MIDIs from somewhere.  Indeed, I can only remember Ahribar's attempt as the only instance of non-downloaded music.

It's rather questionable then how useful it would really be to be able to edit the music.  And keep in mind that what Borg's trying to do is to playback music stored exactly in the format that Lemmings used to store them.  We are talking about a format that is so proprietary to the Lemmings games, even Lemmings 2 doesn't seem to use the same format anymore.  So it's unlikely you will be able to easily incorporate music from other DOS games even if you have the ability to edit the Lemmings music fully, unlike Cheapo where you do find many MIDIs available on the Internet for download.

[Borg_Number_One]

Hi.

1.)
With the help of ccexplore's and mindless' lemmings dat decompressor :
http://it.travisbsd.org/lemmings/files/lemmingsdatdecompressor/lemmingsdatdecompressor_1_1_1.zip

I was able to unpack adlib.dat in many Lemmings games.
The uncompressed adlib.dat files contain the player/playing code and the Lemmings tunes.

All the news were already send to the ADPlug author Simon Peter.


For those who cannot wait, I captured the RAW Adlib commands which were generated by "XMAS/Christmas Lemmings" (Holiday Lemmings 1993) while playing the game.
http://bnobtc.pix-art.com/new-raw-adlib-tunes.rar

The RAW format can be played with ADPlug (http://adplug.sf.net) and/or with RDOSPLAY ( http://sorry.vse.cz/~roman/dimension/rdos/rdosplay/).

RDOSPLAY is a DOS application and it needs DOS sound driver or direct hardware access to an available real sound card with FM/OPL2/OPL3/Adlib chip.
So you either need a real DOS and a DOS compatible sound card or you can take
+ DOSBOX emulator
+ VMware
+ MS VirtualPC
+ VDMSound or
+ SoundFX
if you want to start RDOSPLAY in Windows 200/XP/...


2.)
RDOSPLAY is already able to play the extracted Lemmings tunes from unpacked adlib.dat files.
I already tried to play Lemmings tunes from other Lemmings games which use an adlib.dat file, but only the extracted tunes from Holiday Lemmings sound decent in RDOSPLAY.

I have been in contact with the author of RDOSPLAY (Roman Dolejsi) for many years.
But he lost the RDOSPLAY sources.

So he allowed me to disassemble RDOSPLAY with "intelligent/interactive" disassemblers which produces more userfriendly code than common disassembler programs.
If he has time, then he will check the sources.

I also send the sources to the ADplug author Simon Peter.


3.)
ccexplore, because Simon and me would like to add support for the compressed adlib.dat files in different Lemmings games (adlib.dat's embedded tunes will be handled as sub-tunes in ADplug),
we would need the sources of your lemmings dat decompressor.

Why?
The users should not spend time unpacking the adlib.dat file.
And if an user does not know the lemmings dat decompressor then he should not spend time to look for methods about: "How to unpack the adlib.dat file?"
In future the users just should simply be able to put the different adlib.dat files into Winamp's/ ADPlug's playlist without uncompressing the adlib.dat files.

So, it would be great, if you could send me the sources of lemmings dat decompressor.


PS:
How did you figure out the .dat (de)compression method?

[ccexplore]

How did you figure out the .dat (de)compression method?

Disassembling and checking it out on a debugger (after undoing the pklite packing of the EXE, of course; there are utilities out on the Internet for that). &#A0;It's not as bad as it sounds. &#A0;Basically you can start with the string constants that references compressed files (eg. levelXXX.dat). &#A0;That would likely lead you to the functions that handle the file. &#A0;The File I/O operations can be identified thru DOS interrupt instructions. &#A0;Then the steps that follows after File I/O would naturally be the decompression.

RDOSPLAY is already able to play the extracted Lemmings tunes from unpacked adlib.dat files.
I already tried to play Lemmings tunes from other Lemmings games which use an adlib.dat file, but only the extracted tunes from Holiday Lemmings sound decent in RDOSPLAY.

I can't imagine why that would be the case. &#A0;I ended up disassembling also adlib.dat in Jan 06, and I don't exactly find it all that advanced in its adlib usage. &#A0;Is this a general RDOSPLAY problem or specific to adlib.dat (and if the latter, is it possibly related to the extraction process)? &#A0;What sorts of problems are you experiencing with the non-Holiday tunes? &#A0;Maybe I can be of help.

Incidentally, don't know if you know this already, but the various adlib.dat's (and tandysnd.dat's) from the various Lemmings game are actually self-contained driver+music data packages. &#A0;All you need to do to use it (in DOS) is to decompress it, load it into offset 0 of some allocated segment, and set an interrupt handler pointing to offset 0 of that segment. &#A0;Then you can call interrupts into the driver just like Lemmings does, in order to play the music (though you do need to continuing call an "update" interrupt to keep it going). &#A0;I don't know how that would apply to RDOSPLAY (if at all), but thought you might want to know.

ccexplore, because Simon and me would like to add support for the compressed adlib.dat files in different Lemmings games (adlib.dat's embedded tunes will be handled as sub-tunes in ADplug),
we would need the sources of your lemmings dat decompressor.

Sure, I'll e-mail you. &#A0;Come to think of it, I don't know why I didn't just put it up for downloading.

(edit: &#A0;turns out I have already uploaded ldecomp.zip, just that I never bothered to include the source file. &#A0;I have now added that to the zip, download at http://www.geocities.com/guestlevels/lemmings/ldecomp.zip.  Ignore the comment crap at the start of the file--you and others have my permission to incorporate any parts of the source into whatever you want.)

Hope it's readable, ldecomp.exe was originally meant more as a "concept demo" anyway.

[Borg_Number_One]

How did you figure out the .dat (de)compression method?

Disassembling and checking it out on a debugger

That is what I just wanted to know.

(after undoing the pklite packing of the EXE, of course; there are utilities out on the Internet for that)

I know about executable packers, as you could see abouve/in previous threads.

RDOSPLAY is already able to play the extracted Lemmings tunes from unpacked adlib.dat files.
I already tried to play Lemmings tunes from other Lemmings games which use an adlib.dat file, but only the extracted tunes from Holiday Lemmings sound decent in RDOSPLAY.

I can't imagine why that would be the case.  I ended up disassembling also adlib.dat in Jan 06, and I don't exactly find it all that advanced in its adlib usage.  Is this a general RDOSPLAY problem or specific to adlib.dat (and if the latter, is it possibly related to the extraction process)?  What sorts of problems are you experiencing with the non-Holiday tunes?  Maybe I can be of help.

It seems to be that the author of RDOSPLAY (Roman Dolejsi) just tested the extracted tunes from "Holiday Lemmings 1993" (X-MAS/Christmas Lemmings)
So, when I try to play other extracted tunes then RDOSPLAY pruduces weird sound.

Incidentally, don't know if you know this already, but the various adlib.dat's (and tandysnd.dat's) from the various Lemmings game are actually self-contained driver+music data packages.

I know this fact, as I already wrote above/in previous threads.

All you need to do to use it (in DOS) is to decompress it, load it into offset 0 of some allocated segment, and set an interrupt handler pointing to offset 0 of that segment.  Then you can call interrupts into the driver just like Lemmings does, in order to play the music (though you do need to continuing call an "update" interrupt to keep it going).  I don't know how that would apply to RDOSPLAY (if at all), but thought you might want to know.

When I try to use MS' debug.exe to load the unpacked adlib.dat, then nothing happens while debugging/executing the unpacked adlib.dat file.

Before I start to run/debug the unpacked adlib.dat file in MS debug, I also changed:
+ the instruction point:

Code: [Select]

rip
IP 0100
: 0000
+ the "cs"

Code: [Select]

rcs
CS 159E
: 15AE
...but nothing seems to work.

Could you also send me the sources from all unpacked adlib.dat files, you already disassembled/decompiled?


Thank you very much for the Lemmings dat decompressor sources.

[ccexplore]

All you need to do to use it (in DOS) is to decompress it, load it into offset 0 of some allocated segment, and set an interrupt handler pointing to offset 0 of that segment. &#A0;Then you can call interrupts into the driver just like Lemmings does, in order to play the music (though you do need to continuing call an "update" interrupt to keep it going). &#A0;I don't know how that would apply to RDOSPLAY (if at all), but thought you might want to know.

When I try to use MS' debug.exe to load the unpacked adlib.dat, then nothing happens while debugging/executing the unpacked adlib.dat file.

I'm not sure what you're trying to do.  I said it is a self-contained driver, not quite a program that you directly execute.  You call into it as software interrupt and supply various parameters in the registers.

The interrupt interface was actually documented.  There is a demo of Lemmings 2 floating around somewhere on the net (don't remember the URL), and very oddly, it includes a text file that has some internal memo from DMA that has info on the interrupt interface.  Although, it's not that hard to figure out from scratch.

Could you also send me the sources from all unpacked adlib.dat files, you already disassembled/decompiled?

I only disassembled Lemmings and ONML, but I'm sure all the various adlib.dat's are structured the same.  You can use my somewhat annotated disassembly of the Lemmings adlib.dat as a reference.  Check e-mail.

One thing you do need to know that's outside of adlib.dat, is the "update rate", essentially the smallest unit the driver counts time in, also equivalent to the frequenty at which the "update" function in the driver is called.  Since the "update" function is called from the external program, the update rate used is not found in adlib.dat.  Based on what I know from lemmings, I believe the update rate might be roughly 72 Hz.

[Timballisto]

So is there ever going to be a way to tell how to edit the lemmings music?  Or for that matter make your own using the same format?  I'd be really, really interested in making my own music for the original lemmings.  From what I could get from the above posts, the only lemmings that works so far for playback is Xmas, right?  ...I wish I understood how this stuff worked.

[ccexplore]

So is there ever going to be a way to tell how to edit the lemmings music? &#A0;Or for that matter make your own using the same format? &#A0;I'd be really, really interested in making my own music for the original lemmings.

May I refresh your memory on this post in the "Lemmings Remake" thread!

So in a way I have already edited lemmings music.  That being said, right now the state of affairs is a little like your request on edit the graphics sets:  the knowledge is there but no one's motivated enough to want to go thru the grunt work to write a program that makes the editing easy and achievable by non-programming types.

I did promise to help customize graphics and music for the special graphics levels in the Lemmings Remake though, so perhaps at some point I'll try to get a MIDI->adlib converter of some sort.

In the meanwhile, think Cheapo instead.

[ccexplore]

So is there ever going to be a way to tell how to edit the lemmings music? &#A0;Or for that matter make your own using the same format? &#A0;I'd be really, really interested in making my own music for the original lemmings.

May I refresh your memory on this post in the "Lemmings Remake" thread!

So in a way I have already edited lemmings music. &#A0;That being said, right now the state of affairs is a little like your request on editing/creating Lemmings graphics sets: &#A0;the knowledge is there but no one's motivated enough to want to go thru the grunt work to write a program that makes the editing easy and achievable by non-programming types.

I did promise to help customize graphics and music for the special graphics levels in the Lemmings Remake though, so perhaps at some point I'll try to get a MIDI->adlib converter of some sort.

In the meanwhile, think Cheapo instead.