Author Topic: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates  (Read 6522 times)

0 Members and 1 Guest are viewing this topic.

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
[DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« on: October 28, 2019, 06:12:24 PM »
We recently received a style submission. I will call this style "Style A". This style was created in old-formats by one author, and converted and submitted by a different author. This style in question was almost identical to "Style B", from the same author who created Style A (but Style B was submitted, a long time ago, by the author themself). I didn't compare the exact physics, but at a quick glance, it differs only slightly in terms of shading. On top of this, both of these are in turn just a derivative of Style C (which was among some of the first NL custom styles), by a different author altogether.

I feel that this is an excessive level of duplication. Style C vs Style B, there is some merit - while they're intended to be (though aren't always) physics-compatible, there's a significant difference in visual. As in, one is almost DOS-style, and the other is more full-color, modern style. However, A vs B is literally just a slight difference in brightness. As such, Style A was rejected from the download - it would have been utterly nonsensical IMO to accept Style A at the same time as we're removing orig_dirt_md.

My concern however, is that when I brought this to the attention of the user who converted and submitted Style A, they brought to my attention some other styles that are partial or wholly duplicates with recolorings, such as GigaLem's Xmas variants of ONML styles, or Mobius's "tancastle" style. To be fair - these are generally a lot more distinguished from the source styles than Style A and Style B were from each other; these two were so close that orig_dirt and orig_dirt_md were more different than these two were. But I also wonder if we really need even that - we're already dealing with a very over-bloated styles download, can we trim some of the fat? We currently have about 42MB of styles (to be fair, this also includes sound effects; but on the flipside, this is zipped size), consisting of a total of 166 styles.

Ideally, eventually NL will get back its download-on-demand features so new users will only have to download a few core styles (the official ones, probably including L2 / L3, and maybe some of the most commonly-used custom ones like the Lemmings Plus ones, the Lemmini Castle set, etc) and those that are rarely / never used will just sit on a server doing nothing instead of bloating the download / user folders. However, this is likely still quite some time away, and I think we need to do something in the meantime. I suspect that when Nepster made the call to start including all styles with NL's download, it was not envisioned that the number of styles available would explode like this - there were maybe 40 or so styles back then. (Indeed, if I'm being perfectly honest, the potential for this was a reason I was averse to moving to these kind of simple formats. Having somewhat of an effort-based barrier to entry helped keep the quantity down and the quality up; though it's undeniable that the flipside is true too, that there have been some excellent styles created that might not have happened under the more-difficult old-formats way of doing things.)

I have implemented a feature for the next update (V12.7.0-RC6 or V12.7.0-stable) that can be used to specify replacement pieces / styles for any removed ones - this can work either on the level of entire styles, or just single pieces (I've made use of the single-piece replacement feature to deal with a duplicate piece in one of my styles, for example).

Perhaps a good starting point is figuring out which of the current styles are actually used by anything. For example - GigaLem's Xmas variants of OhNo styles, I'm fairly sure the only use those saw was in Holiday GigaLems, which I don't believe GigaLem has any intention of bringing over to new-formats, so do we really need to keep the styles? Unfortunately though, unlike Lix, NL doesn't have a single central repository of levels, so every pack would need to be tracked down and checked to determine this. (It would also not be very practical to maintain a central repository, unless it's based on user-submission, in which case it likely won't catch every existing level.)

The alternative option may be to go back to how we used to do things - NL comes only with a few core styles, and users must download any extras on their own. Content authors would need to point out what styles are used in their packs, and ideally, where to get them. Downside of this, is that if someone's download link breaks, the style is lost, and all content that relied on it is broken. Also, we now have a whole bunch of content that uses a mix of "core" and "non-core" styles, and creators of this would need to go back and figure out what's included vs what needs to be downloaded.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline IchoTolot

  • Global Moderator
  • Posts: 3611
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #1 on: October 28, 2019, 06:29:30 PM »
Quote
The alternative option may be to go back to how we used to do things - NL comes only with a few core styles, and users must download any extras on their own. Content authors would need to point out what styles are used in their packs, and ideally, where to get them. Downside of this, is that if someone's download link breaks, the style is lost, and all content that relied on it is broken. Also, we now have a whole bunch of content that uses a mix of "core" and "non-core" styles, and creators of this would need to go back and figure out what's included vs what needs to be downloaded.

I am 100% against this. I would rather download 1000 styles than tracking everything down manually and deal with tons of broken links and new topics about missing styles errors like in the old days (Seriously there was one every week or so).

I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

For the older near duplicates: Exactly list them, examine the usage and then decide over a possible exclusion.

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #2 on: October 28, 2019, 06:41:22 PM »
Quote
I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

One possible issue here: We can't stop people from posting their custom styles on the forums, we can only exclude them from the NL download, or at most, make NL specifically recognize those styles' names and use the original even if the duplicate is present. (Okay, technically, as forum admins, in theory we could delete posts of such styles and even ban people who repeatedly post them, but this feels ridiculously heavy-handed in practice - and this is coming from someone who was perfectly okay with preventing new posts in the pre-V10 levels releases forum to discourage new pre-V10 content.) Thus, would this "quality control" just end up leading to a de-facto version of the old ways, just with a much larger amount of "core" content?
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline IchoTolot

  • Global Moderator
  • Posts: 3611
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #3 on: October 28, 2019, 06:48:31 PM »
Quote
I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

One possible issue here: We can't stop people from posting their custom styles on the forums, we can only exclude them from the NL download, or at most, make NL specifically recognize those styles' names and use the original even if the duplicate is present. (Okay, technically, as forum admins, in theory we could delete posts of such styles and even ban people who repeatedly post them, but this feels ridiculously heavy-handed in practice - and this is coming from someone who was perfectly okay with preventing new posts in the pre-V10 levels releases forum to discourage new pre-V10 content.) Thus, would this "quality control" just end up leading to a de-facto version of the old ways, just with a much larger amount of "core" content?

I am totally against any censorship. If people want to post duplicate styles let them, it's their right. It's only their problem to maintain then and the standard library won't support those styles.

Only the size of the standard library download is the issue here and needs to be prevented from explode any further. The user can hord up as many tilesets as they like, just that standard download is the issue.


Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #4 on: October 28, 2019, 06:54:56 PM »
Quote
I am totally against any censorship. If people want to post duplicate styles let them, it's their right.

Yes, I agree - my point about being able to delete the posts is saying "This is physically possible, but it would be ridiculous to actually do it".
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline Strato Incendus

  • The King of Shimmiers (crowned by Flopsy ;D )
  • Posts: 1754
  • #RIP Spearer/Grenader (2020 - 2021)
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #5 on: October 28, 2019, 08:41:43 PM »
I appreciate namida's attempt to keep things anonymous here - me, personally, I am fine with everyone knowing that the user who sent the tileset in question to Nessy was me. ;) (I'm not giving away whose tileset we're talking about here, though I think there's enough information on the forums already to figure out for those who are interested.)

I think we already agreed a while ago not to create any new versions of tilesets which are merely recolourings, IIRC? This case was special, because the tileset I converted is actually an interim version, i.e. an older, different looking version of a tileset which, in its final state, has become part of the official styles download by now. I had used this style as it was being developed, and wanted the levels created during that time period to still look the same way they were originally created.

Of my own accord, I never would have gone out of my way to actively recolour any existing tilesets. ;)

So I think this is more comparable to providing backwards-compatibility for old content, rather than encouraging the creation of new content?

Of course, with graphic sets, this is a more difficult case than with whole levels, because any graphic set that can be used to play old levels can also be used to create new levels with. While with different versions of NeoLemmix, creators have to jump through some hoops to create content for outdated versions, any style that's available in the style folder, even if it's just intended for playing, can be just as easily accessed in the level editor as any other, "established" tileset.

Thus, I acknowledge these two things aren't strictly comparable. But in general, I'd be on the side of "let's allow the existing recolourings we have or have had in the past, but let's not create any completely new ones." I mean, there's nothing stopping anyone from taking e.g. the original pillar tileset and re-painting it in pink camouflage, but really, what's the point? ;)

Removing existing recolourings, meanwhile, like the festive Rock and Brick tileset, the golden Crystal tileset (does that still exist?), or mobius's tanCastle, would end up breaking levels using them, and thus be equivalent to a cull.
I already had levels break simply due to just single pieces being removed from tilesets by the original creator of that tileset.



This is also something we should consider creating some sort of "code of conduct" for: If you create a graphic set and release it to the public, starting from that point, anyone can create levels with it. (If it's a preliminary version, one can explicitly advise people not to create levels with it yet, just like when new skills are getting introduced which aren't finalised yet.) Adding further pieces to a tileset is no problem, then - it used to be, in Old Formats, in which new pieces could only be added at the end of the list, otherwise they would have messed up any levels using this tileset, because the order was relevant. Now that levels only refer to each piece by name, adding stuff is a lot easier.

But removing pieces from existing graphic sets, especially when it comes to terrain, can break a bunch of levels without the level designers noticing it. The style gets updated with the next download, and suddenly certain levels don't work anymore.

So the question is really: "Who 'owns' a graphic set?" Just the original creator? If we think in terms of intellectual property, the graphic-set designer owns the tileset, but anyone who designed levels with that tileset owns the level. Therefore, a graphic-set designer removing pieces from an existing tileset violates level designers' intellectual property.

Unless we want to think of graphic sets more like "licensing platforms", such as Steam, or online trading card games - where the players (in this case, this includes level designers) don't own anything, but only get the permission to use the content, but this can be revoked at any time, potentially.

Of course, we can easily fall down a rabbit hole here if we apply the same logic to level packs - does a player "intellectually own" their solution? And thus, when the pack creator fixes a backroute and destroys the solution, is this also a violation of intellectual property? :D Players put a lot of time and brain power into trying to solve levels, that's a fact.
« Last Edit: October 28, 2019, 10:11:12 PM by Strato Incendus »
My packs so far:
Lemmings World Tour (New & Old Formats), my music-themed flagship pack, 320 levels - Let's Played by Colorful Arty
Lemmings Open Air, my newest release and follow-up to World Tour, 120 levels
Paralems (Old Formats), a more flavour-driven one, 150 levels
Pit Lems (Old Formats), a more puzzly one, 100 levels - Let's Played by nin10doadict
Lemmicks, a pack for (very old) NeoLemmix 1.43 full of gimmicks, 170 levels

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #6 on: October 28, 2019, 09:04:59 PM »
Quote
So I think this is more comparable to providing backwards-compatibility for old content, rather than encouraging the creation of new content?

If there are physics differences between the two styles (which is even more likely given the status as an "early version" of the style that's later been treated as an entirely separate style), then having both of them exist is a very strong potential source of confusion and this takes priority over what content it will or won't break. If there are no physics differences, then nothing will break as long as NeoLemmix correctly redirects from one style to the other when loading levels.

If there are some pieces that exist in one version but not the other, they should be merged into the primary version.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline Proxima

  • Posts: 4568
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #7 on: October 28, 2019, 10:35:30 PM »
I'm glad this is being discussed. As some of you remember, I was very upset when the cull of dirt_md was announced, and it wasn't the cull itself that was hurtful, it was the lack of any consultation -- one minute everything's fine, the next, some of the content I'm working on is scrapped and I never got to have any say in the matter.

So maybe we can start over and have a proper discussion this time.

The real problem here is the bloat of the styles folder, not the existence of recolours -- there are over 160 styles and only a handful of them are recolours. Also, most individual users don't need anywhere near all the styles -- if I trimmed my folder to the styles used by the packs I am playing, it would be well under half that size.

So, what's to be done? Namida's suggestion of download-on-demand seems to solve everything, if it works more or less like I imagine (downloading a pack automatically comes with a download of any styles you need but don't have installed). The only problem is, as namida says, it may not be coming for some time.

As an interim measure, I would suggest that packs include a text file that lists all styles they use, and namida creates (I imagine this wouldn't be too hard) a tool for generating such files. Then the end user can easily determine which styles they personally don't need, and delete them.

If we agree that download-on-demand will happen eventually, then what is the harm in allowing recolours to exist? As with all less-used styles, they will only bloat the end user's style file if that particular user chooses to opt in.

This has all felt like a huge slap in the face to me, and I can't help asking: is it really that important to you to have 160 styles in your styles folder rather than 166? Will that really solve any problems?

Offline Dullstar

  • Posts: 2092
    • View Profile
    • Leafwing Studios Website (EXTREMELY OUTDATED)
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #8 on: October 29, 2019, 03:21:18 AM »
I would like to see download-on-demand return. There's a lot of styles currently, of varying quality.

On the topic of recolors: I know we have recoloring for lemming sprites. Out of curiosity, how complicated would recolor support for entire styles be? It could be a way to allow recolor styles to exist without causing excessive bloat, although a way to distinguish them from their parent sets would probably be best. I envision a system where you would select the parent style you want and then would have the option to choose a recolor from there. Low-effort recolors could be a potential problem created by this, though, so there would probably have to be some somewhat stringent standards for what's enough to be accepted into the automatic downloads (though I'd say any from the official games I think should be allowed to be made).

I actually don't think the filesize is a huge problem, at least once they've been downloaded. What I think is the bigger problem is that the list of styles available in the editor is bloated and hard to browse. I'd suggest two possibilities for dealing with this (that are not mutually exclusive): 1) allow the user to create a "favorites" list of styles so they can quickly find the ones they use the most, or 2) allow the user to hide unwanted styles.

Offline IchoTolot

  • Global Moderator
  • Posts: 3611
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #9 on: October 29, 2019, 12:13:38 PM »
Quote
On the topic of recolors: I know we have recoloring for lemming sprites. Out of curiosity, how complicated would recolor support for entire styles be? It could be a way to allow recolor styles to exist without causing excessive bloat, although a way to distinguish them from their parent sets would probably be best. I envision a system where you would select the parent style you want and then would have the option to choose a recolor from there. Low-effort recolors could be a potential problem created by this, though, so there would probably have to be some somewhat stringent standards for what's enough to be accepted into the automatic downloads (though I'd say any from the official games I think should be allowed to be made).

I think Dullstar is onto something here and it could be the solution for Proxima's/Strato's problems while also reducing the file/folder clutter!  :thumbsup:

Possible solution: Don't treat recolors as a new standalone style in a new folder, but have an option inside the editor to choose an existing variant/recoloring of the parent style. Variants are stored in a textfile inside the parent style's folder.

It's just the question how the recoloring is acieved. We want to avoid having to duplicate all piece files inside the folder. Rather than that a textfile could be made where a color variant is specified. An example for the specifications would be: Change the following colors to these colors now.
It is be quite a bit of work to define all the nessesary color swaps in this textfile as there are probably many nessesary ones, but it's way better filenumber/folder wise than to duplicate everything.
This way the variants are also 100% identical physics wise and only provide a recoloring.

Example for the dirt style would be: You create a dirt level and then choose the genesis/etc variant from a selection field that comes up in the editor when recolor variants exist. All variants and how they are generated are stored in a textfile inside the style's folder.
This way things like the golden crystal style could be added without having another nearly identical style folder.


Offline Strato Incendus

  • The King of Shimmiers (crowned by Flopsy ;D )
  • Posts: 1754
  • #RIP Spearer/Grenader (2020 - 2021)
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #10 on: October 29, 2019, 01:26:21 PM »
Indeed this is something I had thought of, too - I'll have to check again, though, I may actually have combined e.g. gigalem_treemd and gigalem_treemdbright on the same level. In such cases, a blanket approach of recolouring all pieces, either to just gigalem_treemd or just the bright version, wouldn't suffice to replicate the physics in the same way as standard tileset mixing does.

More generally speaking, it would be impossible to mix several different recoloured tilesets on the same level. This in turn would make them much less versatile than standard standalone tilesets, and feel like a step back to version 1.43, at least as far as these specific styles are concerned.

Or am I misunderstanding something here? ;)
My packs so far:
Lemmings World Tour (New & Old Formats), my music-themed flagship pack, 320 levels - Let's Played by Colorful Arty
Lemmings Open Air, my newest release and follow-up to World Tour, 120 levels
Paralems (Old Formats), a more flavour-driven one, 150 levels
Pit Lems (Old Formats), a more puzzly one, 100 levels - Let's Played by nin10doadict
Lemmicks, a pack for (very old) NeoLemmix 1.43 full of gimmicks, 170 levels

Offline IchoTolot

  • Global Moderator
  • Posts: 3611
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #11 on: October 29, 2019, 01:53:57 PM »
Quote
Or am I misunderstanding something here? ;)

Yes.

The option to choose the style variant would have to be coupled with the terrain/object choosing bottom bar and not with the main theme.

Let's say, you choose a main style, then select (for example) orig_dirt in the terrain/object bar below and opon choosing that another field appears where you can choose the variant which is read from the mentioned textfile by the editor. The images on the selection bar change on the fly. So you can choose tiles from orig_dirt and/or (both is then possible) the genesis variant.

So mixing between terrain/objects would be indeed possible after my example suggestion based on Dullstar's idea.

Offline Strato Incendus

  • The King of Shimmiers (crowned by Flopsy ;D )
  • Posts: 1754
  • #RIP Spearer/Grenader (2020 - 2021)
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #12 on: October 29, 2019, 01:56:31 PM »
Thanks for the explanation! ;)

In that case, this does sound like a great idea indeed! I'm fully in support of this approach! :thumbsup:
My packs so far:
Lemmings World Tour (New & Old Formats), my music-themed flagship pack, 320 levels - Let's Played by Colorful Arty
Lemmings Open Air, my newest release and follow-up to World Tour, 120 levels
Paralems (Old Formats), a more flavour-driven one, 150 levels
Pit Lems (Old Formats), a more puzzly one, 100 levels - Let's Played by nin10doadict
Lemmicks, a pack for (very old) NeoLemmix 1.43 full of gimmicks, 170 levels

Offline Flopsy

  • Global Moderator
  • Posts: 955
  • Lix Nerd
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #13 on: October 29, 2019, 03:04:55 PM »
I am in support of what has been mentioned above about having a dropdown menu for different styles within styles.

However I would like to call attention to this topic

https://www.lemmingsforums.net/index.php?topic=4486.msg77811#msg77811

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #14 on: October 29, 2019, 05:35:10 PM »
I am in support of what has been mentioned above about having a dropdown menu for different styles within styles.

However I would like to call attention to this topic

https://www.lemmingsforums.net/index.php?topic=4486.msg77811#msg77811

I went into more detail about this and proposed something that might in general help with such cases in that topic, but I'll quickly summarize here: This topic is really about styles that are, as a whole, duplicates or near-duplicates of other styles. Single pieces that do properly fit in multiple styles aren't such a big deal.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #15 on: October 30, 2019, 06:22:26 PM »
Quote
The alternative option may be to go back to how we used to do things - NL comes only with a few core styles, and users must download any extras on their own. Content authors would need to point out what styles are used in their packs, and ideally, where to get them. Downside of this, is that if someone's download link breaks, the style is lost, and all content that relied on it is broken. Also, we now have a whole bunch of content that uses a mix of "core" and "non-core" styles, and creators of this would need to go back and figure out what's included vs what needs to be downloaded.

I am 100% against this. I would rather download 1000 styles than tracking everything down manually and deal with tons of broken links and new topics about missing styles errors like in the old days (Seriously there was one every week or so).

I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

For the older near duplicates: Exactly list them, examine the usage and then decide over a possible exclusion.

One thing that I was reminded of while thinking about a different matter: This situation (with missing styles) only arose because of content creators ignoring the repeatedly-given advice, "your NXPs should include a copy of ALL non-standard styles used in your packs". Of course, this is not practical in new-formats where styles included with packs will start overwriting each other (possibly with out of date versions), but yeah, issues back then occurred primarily due to blatant disregard for best practices.

Another piece of advice I have repeatedly given, relating to the current formats, is that packs should be ZIPped such that they can be extracted to the base NL folder (ie: the ZIP will contain a "levels" folder, inside which is a "InsertPackNameHere" folder; it might also contain at the base level a "music" or "styles" folder if needed). This has been even more severely ignored; I don't think I can name a single pack outside of my own that is ZIPped in such a manner, and I've pretty much given up on trying to get this message through. The NL installer needed signifcant kludges in its coding to deal with this. At least most packs are zipped in such a way that you at least only have to extract it once; but I've seen a few where you have to "extract these files to levels, then extract these ones to music, etc"... I didn't even try to make the installer account for this; instead either rejecting or re-packing affected packs (depending on their value).

The point here: Any solution must be enforced somewhere along the line - be it in NL itself, or a server that content is uploaded to. It's not going to be good enough to merely advise people "do this to avoid problems", because no one will do it. Even things that must be done to avoid content breaking are often semi-neglected by some (but not all) users.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline IchoTolot

  • Global Moderator
  • Posts: 3611
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #16 on: October 30, 2019, 06:57:29 PM »
Quote
Another piece of advice I have repeatedly given, relating to the current formats, is that packs should be ZIPped such that they can be extracted to the base NL folder (ie: the ZIP will contain a "levels" folder, inside which is a "InsertPackNameHere" folder; it might also contain at the base level a "music" or "styles" folder if needed). This has been even more severely ignored; I don't think I can name a single pack outside of my own that is ZIPped in such a manner, and I've pretty much given up on trying to get this message through. The NL installer needed signifcant kludges in its coding to deal with this. At least most packs are zipped in such a way that you at least only have to extract it once; but I've seen a few where you have to "extract these files to levels, then extract these ones to music, etc"... I didn't even try to make the installer account for this; instead either rejecting or re-packing affected packs (depending on their value).

I didn't even know this was a thing to be honest.

Also, at least for me, I never ship the music together with the pack as then the download just to get an update becomes too large, therefore I have 2 components. And for these I personally always prefer unzipping inside the levels and music folder respectively as that seems more intuitive for me (pack into levels, tracks into music). I never even had the installer in mind and didn't know there needed to be bigger adjustments.

Quote
The point here: Any solution must be enforced somewhere along the line - be it in NL itself, or a server that content is uploaded to. It's not going to be good enough to merely advise people "do this to avoid problems", because no one will do it. Even things that must be done to avoid content breaking are often semi-neglected by some (but not all) users.

True words.

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #17 on: October 30, 2019, 07:15:46 PM »
Quote
Also, at least for me, I never ship the music together with the pack as then the download just to get an update becomes too large, therefore I have 2 components. And for these I personally always prefer unzipping inside the levels and music folder respectively as that seems more intuitive for me (pack into levels, tracks into music). I never even had the installer in mind and didn't know there needed to be bigger adjustments.

Seperate ZIPs is okay; it's about where those ZIPs should be extracted to - even before the installer, this was a thing simply to keep it simple - the user always just needs to extract to the NL folder, and not worry about what things go where in terms of subfolders.

Long term, this should become a non-issue user-side, as NL itself downloads and extracts the packs. But this is not a feature NL has yet; and at any rate this would make it more important to have consistency creator-side.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #18 on: October 31, 2019, 02:04:15 AM »
Flopsy made another topic asking about a specific case (duplicating single pieces across a few styles, rather than duplicating styles as a whole).

Most important post is the last one:

Had some uncertianty on Discord around the original question so just to be clear:

- Don't create styles that are, as a whole, slight visual variations on existing ones. I'm probably going to grandfather in existing styles of this nature (including, retroactively, orig_dirt_md), but we definitely won't be accepting new ones (at least for now; and this extends to the Genesis styles that weren't already present).
- It's fine to duplicate a few individual pieces between styles when they really do fit into every style - some official styles do this already (eg. steel blocks between orig_dirt and orig_pillar). For example, Flopsy's original question above would be a case where this is okay.

In terms of "what happens later if/when we get an ability like the one suggested in reply #2" - to be on the safe side, when duplicating pieces between styles, try to keep the name the same in all styles. But I envision that "fixing" levels where this is relevant will be as simple as "run Cleanse Levels in a version of NL that's new enough to have this feature".

In terms of "what's going to happen long term" - I do really agree that once we get on-demand downloading of styles back in place, this is a much less significant issue. The on-demand downloading is a priority feature for V12.8, so once that's out, we can possibly go back to allowing this. Before that though, a discussion needs to be had around physics consistency - whether NL enforces it somehow, or if it's on a basis of "not keeping it accurate to the source style is grounds for your style being removed", or whether it's just "use these styles at your own risk, don't assume they're physics-accurate to the source material".

I'm going to close this topic; further discussion around styles acceptance / tidyup goes in the original topic on this subject: https://www.lemmingsforums.net/index.php?topic=4483.0
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #19 on: October 31, 2019, 02:06:57 AM »
A question I have for those who care about these "palette swap" styles (I'm going to use this term from now on, even though it's not accurate on a technical level, because it carries the right implication):

Do you envision yourself using the original version of a style and the palette swapped version of a style, together in the same level*? Or would it be acceptable for a setup of "you can only pick one version or the other"?

* other than cases where this is only done because of a piece that exists in the original but not the palette-swap version
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #20 on: October 31, 2019, 08:59:21 AM »
I mean, I can conceive of level visual designs that might intentionally use both together in the same level, like maybe one side of the level is where your normal lemmings enter and gathered in, and the other side is zombie-lemmings infested, and a suitably palette-swapped set of terrain pieces (and maybe even objects) used to depict the zombie side differently from the non-zombie side (eg. maybe grayscale for the zombie side).

But on the whole, I expect such mix usage to be relatively rare.

That said, perhaps the level file format could even be extended to support applying per-element palette swaps, meaning in extreme case every single individual piece of terrain and object in the level can each have their own independent palettes/recolors.  In practice going to such extreme direction should be rare and perhaps somewhat discouraged (I imagine it might increase memory consumption, especially for objects), but in theory one could consider implementations along those lines, and then it would certainly allow level designers to use same style with multiple palettes applied all within same level.

Even the extreme case I just mentioned might not actually be so crazy design-wise, though again probably not common.  So maybe not literally tweaking every single terrain and object pieces, but imagine a vertical scrolling level where visually, you want to depict the lighting being the brightest at the top and darkest at the bottom, as if the lemmings are descending deep down into a cave or the abyss starting from near the surface.  The cleanest way (but not actually achievable currently, I think) to do this is to apply a single continuous gradient of opacity, overlaid over the entire level area.  But an alternative approximation of same effect could be to recolor each terrain and object to be brighter/darker based on their vertical positioning.

Offline Strato Incendus

  • The King of Shimmiers (crowned by Flopsy ;D )
  • Posts: 1754
  • #RIP Spearer/Grenader (2020 - 2021)
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #21 on: October 31, 2019, 09:53:21 AM »
I for one am pretty sure that I have in fact already used the original and the to-be palette-swapped version of a style on the same level. ;) So enabling this is definitely something I'd support!
My packs so far:
Lemmings World Tour (New & Old Formats), my music-themed flagship pack, 320 levels - Let's Played by Colorful Arty
Lemmings Open Air, my newest release and follow-up to World Tour, 120 levels
Paralems (Old Formats), a more flavour-driven one, 150 levels
Pit Lems (Old Formats), a more puzzly one, 100 levels - Let's Played by nin10doadict
Lemmicks, a pack for (very old) NeoLemmix 1.43 full of gimmicks, 170 levels

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #22 on: October 31, 2019, 06:04:15 PM »
Quote
That said, perhaps the level file format could even be extended to support applying per-element palette swaps, meaning in extreme case every single individual piece of terrain and object in the level can each have their own independent palettes/recolors.  In practice going to such extreme direction should be rare and perhaps somewhat discouraged (I imagine it might increase memory consumption, especially for objects), but in theory one could consider implementations along those lines, and then it would certainly allow level designers to use same style with multiple palettes applied all within same level.

I think it would be easier to just keep them as separate styles in this case.

Quote
The cleanest way (but not actually achievable currently, I think) to do this is to apply a single continuous gradient of opacity, overlaid over the entire level area.  But an alternative approximation of same effect could be to recolor each terrain and object to be brighter/darker based on their vertical positioning.

This is currently achievable, though it would need to be done with a large no-effect object, which would need to be specifically made to fit the level's height (it could be resizable for the horizontal axis). Of course, this would not affect the lemming sprites, due to them being drawn in front of all objects. Note that the object would need to specifically be "no effect" and not "background", as the latter gets drawn to the lowest layer no matter what.

I don't advise actually doing this. The impact on performance would be very significant, and the fact that it's a no-effect, not a background, also means that neither clear physics mode nor "disable backgrounds" option will hide it in any way.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline ccexplore

  • Posts: 5311
    • View Profile
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #23 on: October 31, 2019, 06:44:56 PM »
and the fact that it's a no-effect, not a background, also means that neither clear physics mode nor "disable backgrounds" option will hide it in any way.

Wait, really? :o I'm surprised clear physics mode would still show no-effect objects, why would it if by definition the object would have no physics effects?

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #24 on: October 31, 2019, 10:21:10 PM »
Quote
Wait, really? :o I'm surprised clear physics mode would still show no-effect objects, why would it if by definition the object would have no physics effects?

Because objects that are legitimately no-effect have usually been marked as background objects. No-effect has generally been used for animating parts of objects that don't as a whole animate, such as exit tops; where it would be more confusing for part of an object (visually speaking) to suddenly disappear.

We have secondary animations now, but very few styles outside of the official ones have actually been updated to use them yet; most still use no-effect objects for this purpose.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #25 on: November 03, 2019, 01:45:39 AM »
As I have said I was likely going to do, I'm going to grandfather in any styles that would fall under this, but already exist in NL's styles download - but no new styles that are just slight tweaks of other styles will be accepted until further notice.

This also means that orig_dirt_md will be restored (but the other Genesis styles, as they did not already exist in NL's download, will not be made available as part of the standard styles download).
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline Dullstar

  • Posts: 2092
    • View Profile
    • Leafwing Studios Website (EXTREMELY OUTDATED)
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #26 on: November 05, 2019, 03:38:29 AM »
I'm not sure how likely it is that support for palette swap styles will be added, but during the discussion regarding high-res styles, I found myself wondering how high-res would interact with palette swaps that were designed with the low-res version of the style in mind, under the assumption that the high-res style may be likely to use more colors than the original style. I'd assume one of the two would have to take precedence, either using whatever the fallback for missing high-res graphics is, or skipping the palette swap process.

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #27 on: November 05, 2019, 05:33:05 PM »
Quote
I'm not sure how likely it is that support for palette swap styles will be added, but during the discussion regarding high-res styles, I found myself wondering how high-res would interact with palette swaps that were designed with the low-res version of the style in mind, under the assumption that the high-res style may be likely to use more colors than the original style. I'd assume one of the two would have to take precedence, either using whatever the fallback for missing high-res graphics is, or skipping the palette swap process.

People have shown interest in being able to mix different palette swaps of the same style together in a single level, so the simplest way to do this would be to keep having them as separate styles. If one has high-res and one doesn't, then upscaling will kick in for the one that doesn't. Essentially, NL itself will have no concept of a "palette swap", it's just another style.

This should be, as mentioned, fine once we have on-demand downloading instead of having to download and extract all the styles up-front.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #28 on: December 14, 2019, 10:45:34 PM »
https://www.youtube.com/watch?v=YG2OmQ1B96U

From V12.8.0 onwards:
- NL's main download will include a few styles: "default", "xmas", "special", "orig_sega", and all the L1 / ONML styles. (And even orig_sega I'm on the fence about. One could easily argue that if its inclusion is justified, so is the inclusion of the L2 / L3 styles.)
- Other styles can be downloaed either manually, or through NL. NL can do this automatically when encountering a level that needs the styles; and also has a menu to manually download any available style.
- An "all styles" download will continue to be provided. This download will be updated periodically; not necesserially every time a single style is updated.
- Additionally, "special" will be split off, so that it only contains pieces from official levels. All pieces in there belonging to other users will be moved into a "[username]_special" style, with the aliasing system used to prevent breakages of existing levels. (Note that this only applies to the game. Before the levels will be useable in the editor after the V12.8.0 update, they'll need to be run through Cleanse Levels.)

This mostly averts the problem with the styles download being huge. I need to give some thought to how sounds might be better handled in this system, but everything apart from that is pretty much good to go now.
« Last Edit: December 15, 2019, 02:02:30 AM by namida »
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)

Offline namida

  • Administrator
  • Posts: 12399
    • View Profile
    • NeoLemmix Website
Re: [DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates
« Reply #29 on: February 15, 2020, 09:59:57 PM »
With V12.8.0's release and the return of online downloading, I no longer feel this is a serious problem. Just remember - if you're creating / maintaining such a style, it's your responsibility to keep it up to date with the normal version of the style, and it is not the normal version author's responsibility to tell you when / what they update; it's yours to pay attention to it.
My Lemmings projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)