Tree with text/image files > binary format

[Simon]

Hi,



Files have names. They can be a number, but that's strange. Consider an indexing file that assigns numbers to graphic tile filenames.

A dir can contain other dirs, e.g., one for each interactive object. It's a powerful database structure.

-- Simon

[ccexplore]

Sorry, to an outsider with little context of what your latest rant-of-the-day is about, both diagrams look just as complicated even though I'm sure the underlying thing you're trying to say is much simpler.

[namida]

Check #lix chatlogs. For quite a while, Simon has been trying to persuade me to use a format similar to those of Lemmini / Lix for graphic sets, rather than a binary one that's more comparable to the DOS format (although the current format, unlike the DOS one, has no limit on number of terrain pieces / objects and uses non-planar 32 bit bitmaps, with a slight hack of not storing RGB values for pixels where the alpha is zero).

The graphic set editing tool (as well as its predecessor, LemSet) both are capable of importing such a format and turning it into a binary format that NeoLemmix can use; Simon is saying that these formats should just be supported directly in NeoLemmix.

[Simon]

Yeah, I suggest to support the file tree directly from the game/editor. There's an indirection through the new binary format right now, which seems unnecessary.

Tree and binary blob have been designed as equivalents. One can be transformed by gstool into the other without loss of information. The idea is not to abolish gstool; gstool is valuable to manage trees.

I won't design sets these days, but others are following the project closely. There is a culture evolving here. One day, others might like to import the Neolemmix format. The simpler the distributed data is, the better.

Hnn, I don't want to be perceived by ccx as too ranting >_>;; Most of the ranted-on issues aren't time critical, so I'll be glad to spread them out in the future more.

-- Simon

[namida]

In which case, the only details they need to know are how to decompress the usual Lemmings DAT format (which is used in many file types in both DOS/regular Lemmix and NeoLemmix), and how to read an ARGB image that skips the RGB if A = 0. This means no reliance on specific libraries being available for their preferred programming language (or variant of such), and no need to learn how to read and write current graphics formats (which, aside from BMP, I think we can agree are pretty complex if you don't have a library doing it for you). The metainfo section is documented, and using a single file for a graphic set is simpler when it comes to building all-in-one EXEs, or further down the track, (ideally) a single file which can be loaded into a general player EXE.

I think you're forgetting that NeoLemmix works on a more similar principle to the original games, in that one level = one graphic set + maybe a single "vgaspec" image. In a project like Lix, where graphic sets can essentially be mixed and matched, your way of doing things is a lot easier to handle. In something like NeoLemmix where each graphic set is essentially self-contained, this kind of format works better. When doing it your way, generally several graphics formats must be supported (at the very least BMP and PNG would be expected), which also raises the question of how to handle transparency on those that don't inherently support it, or which technically do but use of it is rare (BMP itself is a great example of this latter case). In a graphic set editor, you can always fall back on asking the user (as is currently done in GSTool when importing object frame strips in BMP format, or from PNG files that don't contain an alpha layer); this is a lot less practical in the game. By using a single, proprietary-but-documented format, it's possible to ensure the graphic sets contain all the info they need unambiguously, and none of the info they don't.

The one case I could see your suggestion being useful is during development of a graphic set, to avoid having to rebuild it when new things are added. Since I myself am the only person who's built completely new NeoLemmix graphic sets so far - and I assume that, at least in the near future, most other people's use of custom sets will be limited to conversions from other engines (but we'll see) - it's hard to say whether anyone else would find such a feature useful. If they're primarily using the graphic set tool directly, more than the INI/PNG structure, then it makes little difference.


Ultimately, I'd guess this is something I could consider if there's a significant amount of interest from people who are actually going to make use of it. Until then, I find the current system of a single file per graphic set (or two files, in cases where the old ground/vgagr format is still being used) is both tidier and easier to implement in the game.

[Simon]

In which case, the only details they need to know are how to decompress the usual Lemmings DAT format (which is used in many file types in both DOS/regular Lemmix and NeoLemmix), and how to read an ARGB image that skips the RGB if A = 0.

Having to implement L1 compression/decompression from scratch is a huge chore. There's no popular library available for this.

The uncompressed file format is simpler than PNG, yes.

The L1 compression format is an obscure, bad format. Everybody who wants to hack L1 data must write code for it, but that doesn't make it reasonable to use it for newer things.

This means no reliance on specific libraries being available for their preferred programming language (or variant of such), and no need to learn how to read and write current graphics formats (which, aside from BMP, I think we can agree are pretty complex if you don't have a library doing it for you).

libz+libpng is installed about everywhere, and sits in every package manager. If you can't compile libz+libpng on some obscure system, chances are you won't do graphics set development on there either, because no user tools exist on that system.

Programming languages either link to C libraries directly, or get common libraries ported to them quickly.

Also, people interested in IRS graphics set programming will be interested in game development -- why would they hack graphics sets otherwise -- and need much more complicated libraries in the first place than a simple and common image library.

If they are only interested in the graphics, but not in a game, then they'll probably be writing server-side code, like a level database. Servers tend run on some unix variant with libpng readily installed.

The metainfo section is documented

Very good. :-)

(404 right now? You'll be filling it in soon I guess.) correct link now.

and using a single file for a graphic set is simpler when it comes to building all-in-one EXEs, or further down the track, (ideally) a single file which can be loaded into a general player EXE.

Compiling entire packs into one single file is another design problem. A tree of levels again is much more admissible for the same reasons as the tree of images.

When doing it your way, generally several graphics formats must be supported

That's more work for the implementor, but a massive improvement for the user.

which also raises the question of how to handle transparency on those that don't inherently support it [...] this is a lot less practical in the game. By using a single, proprietary-but-documented format, it's possible to ensure the graphic sets contain all the info they need unambiguously, and none of the info they don't.

Correct, the tree must invent some additional convention.

I thought gstool compiles trees to binary, and unpacks binary to tree, with these two actions being two-sided inverses. (In particular, Binary -> tree -> binary gives you an identical binary as before.) If that's the case, then you have already a convention for the tree, which is well-supported.

What's the exact state of the art here?

My suggestion founds on this assumption, that the tree is 100 % exchangable with the binary blob, except for how game/editor cannot load it directly.

The one case I could see your suggestion being useful is during development of a graphic set, to avoid having to rebuild it when new things are added. [...] If they're primarily using the graphic set tool directly, more than the INI/PNG structure, then it makes little difference.

Then you assume gstool to be more powerful than all the third-party tools combined that hack on PNGs and text files.

Batch file processing, text editors, both GUI and stream editing, graphics tools, both GUI and on command-line, and version control will all be unhappy with a single binary.

People will want to use many tools on the set, gstool certainly being one of them.

Ultimately, I'd guess this is something I could consider if there's a significant amount of interest from people who are actually going to make use of it.

Fair.

Until then, I find the current system of a single file per graphic set (or two files, in cases where the old ground/vgagr format is still being used) is both tidier and easier to implement in the game.

Easier for you, okay. Other programmers have to write an L1 decompressor, or modify existing source, instead of being able to delegate it to a popular library.

I'm mainly concerned with the user's workflow, secondly with ease for developers. (If ease for developers were the primary concern, I'd ask to scrap the binary format entirely, so they have one less format to support.)

Tidier, we've already talked about this on IRC. I find dirs and common formats very tidy, unlike splodging things together into one opaque blob.

In the other world, one tileset is one dir. That's at least as easy to handle as one binary blob.

Alright, now this has turned out way more rant-y again; the basic question is at "what's the exact state of the art" above.

-- Simon

[namida]

(404 right now? You'll be filling it in soon I guess.)

Nope; just that I typed the wrong link in that post. It's been up there since before a NeoLemmix update supporting it was even released.

This one should work better.