[SUG][NeoLemmix Level Packs] Difficulty Ratings Scale

[WillLem]

Suggested Difficulty Scale for Custom NeoLemmix Levels/Packs

After discussion, the majority of us seem to agree, more or less, that the following scale is appropriate:

(I've placed the various ranks of Lemmings Redux onto this suggested scale).

Beginner - brand new players - Gentle
Easy - begins to stretch the player with basic puzzles - Quirky
Medium - introduces more complex puzzles and level design concepts - Zany / Manic
Hard - expects the player to have a solid grasp of the game, puzzles become particularly challenging - Lunatic
Advanced - requires more in-depth knowledge of a wide range of solving techniques, strategies and tricks
Extreme - challenges even the most accomplished solvers


(Previous OP):

I'd like to propose the following sliding-scale difficulty rating for creators to assign to their packs, which may help to give a more specific idea of what a player can expect based on their ability. It may also help players to identify their ability level, and choose packs accordingly.

If adopted, creators would be encouraged to apply the ratings as part of any level pack upload in the following manner:

Pack Name [Difficulty 0-3] - this would indicate that the pack features levels which range in difficulty from beginner through to advanced, and therefore can be enjoyed by more casual players as well as more experienced players who wish to take a break from more challenging packs! It may also serve as an ability-boosting pack for less experienced players.

Pack Name [Difficulty 3-5] - this would indicate a more difficult pack that is best attempted by advanced players who are looking for further challenges.

Pack Name [Difficulty 4-6] - this would indicate a pack that is best attempted only by very experienced players who have established themselves as master solvers.

Pack Name [Difficulty 0-6] - this would indicate that the pack spans the full range of difficulty ratings.

Here are my proposed difficulty ratings explained:

0) Beginner - this rating indicates that the pack includes tutorial levels for the absolute beginner who has no prior experience of playing the game, and needs to be introduced to the skills, objects and their basic functions, and general gameplay.

Examples of levels in this category: Just Dig!, Only Floaters Can Survive This, You Need Bashers This Time.
Examples of techniques: Basic application of a few skills.
Examples of objects: The levels may feature fire, water and traps, but these will be very easy to avoid and may not even form part of the main route.

1) Novice - this indicates levels which expect that a basic tutorial will have been completed, and the player is able to navigate their way through reasonably easy levels with plenty of available skills and various possible solutions.

Examples of levels in this category: Smile If You Love Lemmings, Keep Your Hair On Mr. Lemming, Don't Let Your Eyes Deceive You.
Examples of techniques: Basic application of a number of different skills.
Examples of objects: The levels may feature fire, water and traps, but these will be very easy to avoid and may not even form part of the main route. The levels may also gently introduce objects such as teleporters and pickup skills.

2) Intermediate - packs with this rating will be generally easygoing packs that players of all ability levels can enjoy; the levels will be easy to solve and fun to play through, serving the purpose of keeping a player's basic skills well exercised. There may be the occasional challenge, but these will be reasonably solvable in the first attempt.

Examples of levels in this category: We Are Now At LEMCON ONE, Konbanwa Lemming San, Lemmingology.
Examples of techniques: Considered application of a number of different skills.
Examples of objects: The levels may feature fire, water and traps that require some thought to avoid. The levels may make use of teleporters and pickup skills to enhance the level, but not necessarily make it more difficult.

3) Advanced - this rating indicates that some of the levels will be more challenging and require above-average ability and a less casual approach to the game; solutions will be less obvious, and will include more advanced tricks, techniques and strategies more familiar with experienced players.

Examples of levels in this category: Turn Around Young Lemmings!, I Have A Cunning Plan, Compression Method 1.
Examples of techniques: Advanced application of a number of different skills, introduction of lateral-thinking puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require considered strategy to avoid. The levels may make use of teleporters and pickup skills to enhance the level and increase the difficulty of the puzzle to some degree.

4) Expert - this rating suggest that the levels are of a very high standard of difficulty, requiring much more precise use of multiple skills and combinations of advanced techniques which would not necessarily be obvious to a beginner or a casual player. It will generally take several attempts to complete a level, and the levels will usually have a very limited number of possible solutions.

Examples of levels in this category: The Great Lemming Caper, Save Me, Lemming Tomato Ketchup Facility.
Examples of techniques: Precise application of a number of different skills, increasingly complex puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

5) Master - packs in this rating will include levels of extreme difficulty, requiring established knowledge of most if not all advanced tricks and solution methods; levels will have very complex and precise solutions, and only the very minimum number of skills and time required for the intended solution. The levels will also expect perfect timing, knowing when as well as how to apply a particular skill.

Examples of levels in this category: It's Hero Time!, No Added Colours Or Lemmings, The Race Against Cliches.
Examples of techniques: Very precise application of a number of different skills, complex puzzles and strategies comprising very specific solutions.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

6) Legend - these levels are only for players of the very highest solving ability. They will normally feature several combinations of extremely complex and painstakingly precise techniques, as well as pixel-perfect timing. These levels will often introduce brand new techniques not seen before, and use them in such a way to increase the difficulty of the puzzle considerably.

Examples of levels in this category: Anything from the final ranks of Lemmings Plus Alpha or Lemmings United.
Examples of techniques: Precise and unusual application of a number of different skills, extremely complex puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

[namida]

Difficulty is subjective. What one person finds hard, another might not. On top of that, it's generally very hard for creators to accurately judge their own packs, especially relative to other people's packs (as opposed to their own earlier / later packs).

While I encourage creators to try and indicate the difficulty of their pack, I don't think a "standardized" system is going to work out all that well - and we certianly won't be enforcing one.

A more realistic approach, though this would be a "distant future" thing if it happens at all, would be a content database packs can be downloaded from, that would also accept user ratings re: difficulty. While still not super-reliable, players could then at least use these to get a more useful indication.

[Proxima]

I fully support the idea of some sort of database or overview page that makes it easier for users to find content, and including difficulty ratings so that it's easier for users to find content appropriate to their skill level, so they can get more enjoyment out of the game and to help those who want to to gradually work their way up the ladder until they are ready to tackle the really tough stuff.

I think it would be great if we (as a community) could put our heads together, come up with a standardised system we can agree on, and apply it to existing packs (with each pack having a rating for its start and end points, since most packs have some progression from easier to harder within the pack). We might disagree on exactly where a certain pack belongs, but hopefully our ratings would average out to sensible decisions.

The current system of leaving it to the pack author is better than nothing, but has three obvious problems: some pack authors don't bother at all; pack authors are not the best at judging their own difficulty; and the lack of any discussion means that different pack authors mean different things by, e.g. "Easy" (see for example Lemmings Destination, which is rated as "Easy - Very Hard" by its author -- the first rank would be between 5 and 6 on WillLem's scale).

Since I've thought about this so much, maybe I'll have a go at writing up my own proposal for a difficulty scale to use as a starting point. I don't think we should start off from WillLem's, because it doesn't match well to the range of usermade content -- we don't need the first six gradations to cover the range from "Just Dig" to "It's Hero Time" when the vast majority of usermade content is harder than anything in the original games.

[WillLem]

I agree that any such system ought to be compiled by the community, by gathering data and feedback from various users from a range of skill levels.

My proposal is mainly a template: I'm sure that the range of difficulty is wider than I've suggested and I think that really anything harder than the original games would be considered "Master" or "Legend" by my standards, but there are probably subcategories within and even beyond those that I haven't encountered yet.

I'd like to be part of the discussion for whatever system gets implemented, simply because I'm relatively new to NeoLemmix and I'm probably about average ability when it comes to solving; I think my perspective could be very useful.

I also think that Namida's idea of a content database is definitely a good proposal for implementation and could go hand in hand with this idea (downloading everything needed for a level pack from one place); collating all of the packs together and grouping them by difficulty would be a great development.

[WillLem]

Proxima has suggested that ranks, rather than packs, should have universally-agreed difficulty ratings, and I think this is a great idea. Of course, they could still be titled whatever the user wishes, but they could be assigned a number or letter based on their difficulty.

So, you could have:

Fun (1)
Tricky (2)
Taxing (3)
Mayhem (4)

and:

Flib (2)
Blah (3)
Jibble (4)
Wumpus (5)

or even:

Snib (4)
Weff (5)
Blopsy (6)

We'd just have to agree (as a community) on what exactly 1, 2, 3, 4, 5 and 6 mean. Here's what I suggest:

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

Of course, a "5" level essentially becomes a "4" level once a trick or method becomes common knowledge, but this has always been true of Lemmings games, and indeed many puzzle games. To some extent, the same could happen for "6" levels as well, although there are certain levels that stand the test of time in this regard, and are always considered extremely difficult.

The easiest way to assign a number to a rank is to give the most difficult level in the rank a number rating, and then simply apply it to the entire rank. Of course, difficulty within ranks can fluctuate, but it gives a clear indicator for the general standard of the rank (and, by extension, the pack), which is all this is meant to be.

---

N.B. The rating system should always indicate the designer's intention, because it's impossible to take into account how a group of players will approach solving the level. For example, a designer could assign their level a "5", but there may be several possible alternative solutions or backroutes. The level is still, however, a "5" due to what the intended solution is. If a player is savvy enough to craft out a different solution, it doesn't make the level itself any less difficult.

[mantha16]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

[Strato Incendus]

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

Interesting categorization! I think all my New-Formats packs (Lemmings World Tour is both Old and New Formats) will be in the 3-5 range.
Lemmicks might be considered in that range as well.
Pit Lems might be 2-4 or maybe 2-5, though I can think of very few trick-based levels in Pit Lems that are so obscure that they would warrant a five. Pit Lems is the only one of my packs with a regularly high number of any-way-you-want levels, because back then I thought this was what the community wanted, and what they meant by "puzzle".

Meanwhile, I've used the crash of my old laptop as an incentive to do a new and clean download of the entire Lemmings Plus Series for New Formats, and started playing Lemmings Plus I, this time with the intention to do it continuously. I've breezed through in one session well into the third rank, until the large number of consecutive any-way-you-want levels started boring me to death. I know that eventually, once the first real puzzle pops up in that pack, it will hit me out of nowhere and most likely stump me. I just hope that with all the experience (and knowledge of obscure tricks) I've picked up in the meantime, I will be able to make further progress this time than during my last attempt.



But yeah, any-way-you-want levels, or those that leave you with many spare skills at the end - levels which you label as difficulty 1 or 2 - I just hate those. A pointless waste of the player's time. If you can do basically anything, it feels like nothing you do matters. Any decision you make on a level can be the correct one. And if you make a false decision, you have plenty of skills left to salvage the attempt without even having to rewind.
Usually, not even the requirement of having to save every lemming puts much of constraint on the "level's indifference". (This was kind of what I was poking fun at in Lemmings World Tour, with my "Great Level of Indifference" - an X-of-everything level that also had a save requirement of 0 - until somebody set the minimum save requirement in New Formats to 1, apparently .)
Even worse, when a bunch of levels can be solved in the same way (e.g. with Basher-cancel-staircases), it also gets repetitive. Combine that with the large number of remaining spare skills and feels like I'm backrouting every level.

I assume Lemmings Plus I was namida's very first pack, so it's by no means a fair comparison - I'd have to compare it to Paralems, which was my very first pack, and Lemmings Plus I is definitely much higher in quality than that. At that point, I guess it was also something revolutionary in and of itself to simply get any type of new custom Lemmings levels in the classic tilesets with the classic skills. But in direct comparison to the other packs we have available for New Formats now, this one so far doesn't seem to have aged well. I expect it to get really challenging on the top ranks, of course... it just takes ages to get there.



Maybe this means that a pack shouldn't span the entire difficulty spectrum from 1 to 6? Because then no beginner will be able to finish it, and any advanced player will be bored for a very long time by the early ranks until the pack finally gets up to their skill level?

Or maybe the difficulty curve of my own packs is just too flat - starting out fairly challenging compared to other packs which start with any-way-you-want levels, but then basically remaining at that level of difficulty, maybe only increasing level complexity and "obscurity" in the course of the ranks...

The pack with the best difficulty curve in my opinion is still SubLems, which starts very easy and becomes fairly hard at the end... but I wouldn't say it goes all the way up to 6. Otherwise, I for one wouldn't have been able to complete it.

[Proxima]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

Just to make it clear, we are not suggesting that it should be required to order packs from easy to hard. Just that many packs cover a wide range of difficulty, so that it's hard and maybe less meaningful to assign a rating to the pack as a whole, and more meaningful to assign ratings to the individual ranks. For a pack like Sammings where each rank has similar difficulty, the ranks would get much the same rating and that's perfectly okay.

You make a valid point that difficulty can also vary a lot from level to level, but if we tried to assign difficulty ratings to individual levels, assigning the ratings would be a lot more work (so fewer people would bother) and there would also be a lot more disagreement -- I remember when DireKrow and I were playing LPI/Psycho, we had a lot of disagreement with each other and namida over which levels we found hard. I have a hunch that if we looked at ranks as a whole, our evaluations would be a lot more similar. For instance, pretty much everyone who's played LPI will probably agree that Psycho is (overall) harder than Danger; there might be disagreement over how much harder.

[Proxima]

We'd just have to agree (as a community) on what exactly 1, 2, 3, 4, 5 and 6 mean. Here's what I suggest:

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

I think this is on the right lines, provided of course that the categories are not interpreted rigidly. Levels can have one specific solution and be very easy: "Just dig", "All or Nothing", "Rules to fall". Conversely, levels can be open-ended, with more skills than necessary or even allowing more lemmings to die than are necessary for the solution, and still be very hard: "Survival of the Craftiest", "Path of Wickedness", "Labyrinth of Despair" (the Lix level, not the NL level it's based on). In general, difficulty is a combination of length of solution; obscurity of tricks involved; temptingness of red herrings ("entropy"); how much leeway you have in terms of skills, save requirement and time; and difficulty of execution. Higher difficulty in any one of these factors could push a level up a category even if the others are not present.

Maybe this means that a pack shouldn't span the entire difficulty spectrum from 1 to 6?

I mean... who said it should? There are packs such as the Lix community set, which does span the entire difficulty range (in fact, I would say its six ranks correspond fairly closely to WillLem's 1-6) because it aims to be a "flagship" pack putting together the entire community's best efforts. Finishing the pack is expected to be a substantial commitment. Revenge of the Lemmings is in a similar boat.

But other than those community packs, I can't think of any pack that literally spans the entire range. Namida's LPI is probably 1-5, while his harder LP packs might be 2-6 or even 3-6.

[WillLem]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

I wouldn't suggest that any such system be enforced, or even obligatory. I'd simply suggest that it be there as a universal indicator of the pack's difficulty, should the author want that to be taken into account by players, and particularly - for the purposes of compiling a leaderboard.

Choosing not to give your pack a difficulty rating is as valid a creative decision as any, and certainly shouldn't preclude anyone from playing your pack - it can simply be a "surprise", which I'm sure many players would enjoy from time to time

However, those who pride themselves on creating particularly challenging levels, or those who simply want to attract players by giving an indication of how difficult (or not) the pack is expected to be, can do so in a more standardised way that the majority of the community agree on and understand. It would only take a few creators to begin using the system and it would either catch on or not; I wouldn't expect everyone to use it, in any case.

Lemmicks might be considered in that range as well.

For sure, I'd probably even put Lemmicks into "4 and up" territory - most of those levels require a specific solution that you can either figure out or not, I don't remember there being any levels that the player can just riff with. As we've previously discussed, I think a rank or two of 2 - 3 rated levels would not have gone amiss!

But yeah, any-way-you-want levels, or those that leave you with many spare skills at the end - levels which you label as difficulty 1 or 2... A pointless waste of the player's time. If you can do basically anything, it feels like nothing you do matters. Any decision you make on a level can be the correct one. And if you make a false decision, you have plenty of skills left to salvage the attempt without even having to rewind.

I totally disagree. The best way to learn anything is to get creative with it. These kind of levels allow the player to do that; they can explore the game at their own pace, in their own way, and this can be its own reward. Of course, this is obviously best for beginners and novices, but even experienced players can stumble upon new tricks, new ways of doing things, ways to optimise skill usage, by being given the freedom to explore in a less restricted setting. Hardcore puzzles are great, and offer a lot of satisfaction when you complete them, for sure, but easier levels have their place. They also help to give a player momentum and, as Mantha said, a bit of "relief" in between the more difficult ones.

Even worse, when a bunch of levels can be solved in the same way (e.g. with Basher-cancel-staircases), it also gets repetitive

To be fair, I'd consider Basher-cancelling a comparitively advanced trick that I absolutely wouldn't expect beginners to know about, and certainly wouldn't feature in a "1-2" rated level, or even a "3". Also, repetitiveness is possible at any level of difficulty.

if we tried to assign difficulty ratings to individual levels, assigning the ratings would be a lot more work (so fewer people would bother)

Absolutely. I think the simplest way to assign a rating would be to rate the most difficult level in the rank. That way, the rank is (in theory) never more difficult than its given rating, although it can of course be less so.

Levels can have one specific solution and be very easy: "Just dig"

That's why I added "and relatively obscure" to the definition of a "4" level

That said, I agree that the definitions should be treated as loosely as possible, and it is very much a spectrum, as you've discussed.

In general, difficulty is a combination of length of solution; obscurity of tricks involved; temptingness of red herrings ("entropy"); how much leeway you have in terms of skills, save requirement and time; and difficulty of execution.

Absolutely. I'd expect that the definitions could be further quantified by these sorts of things, to a greater or lesser extent. Also, if an author has created a level which would otherwise be straightforward if not for, say, a tight time limit or use of red herrings, then I would expect that they would know to rate it a bit higher in terms of difficulty even though it might otherwise "fit" the definition of a lower rating.

As always, the definitions I've suggested are merely a starting point on which to build, or perhaps from which to create a comparitive starting point based on whatever factors arise from discussion.

[Strato Incendus]

They also help to give a player momentum and, as Mantha said, a bit of "relief" in between the more difficult ones.

Oh, believe me, I'm the first one to advocate for breather levels in between difficult ones! Some other people criticise that as a "fluctuating difficulty curve"; I think it helps maintaining player motivation and allows the cognitive resources to regenerate.

However, such a "breather-level" pattern is usually not how these any-way-you-want levels are employed. They're usually back to back, clogging up the lower ranks. Meaning, and experienced player could basically take the chance and just skip rank 1, possibly also rank 2. But of course, we don't do that, because we don't want to be "philistines" .

Both issues boil down to the same thing, though:
You simply can't starve people of their dopamine hits for too many levels in a row. If you do, they'll quit. Both too many easy levels AND too many difficult levels in a row lead to this same result.
Where your personal level of tolerance in this regard lies, that of course will differ from player to player.

[Proxima]

Meaning, an experienced player could basically take the chance and just skip rank 1, possibly also rank 2. But of course, we don't do that, because we don't want to be "philistines" .

Who's "we"? I've often seen experienced players, such as Icho, jump in at the top rank of multi-rank packs, and no-one ever criticises them for it.

Both issues boil down to the same thing, though:
You simply can't starve people of their dopamine hits for too many levels in a row. If you do, they'll quit. Both too many easy levels AND too many difficult levels in a row lead to this same result.
Where your personal level of tolerance in this regard lies, that of course will differ from player to player.

Sure, but it seems like you complain a lot about levels that are too easy for you, when you are simply not the intended audience, just as you are not the intended audience for levels, ranks or packs that are too hard for you. There is nothing wrong with authors producing easy content, just as there's nothing wrong with authors producing difficult content. What really matters is having some kind of system to help players find content at the level they want to play at.

[mantha16]

dont get me wrong i like the idea of a rating system its just i rate all mine as easy and icho said some of my levels arent as easy as i think so id have no idea how to rate them lol

[Dullstar]

I often think it's hard to rate your own levels' difficulties. You know the solution, after all, and it might be more (or less) well-hidden than you thought. My tendency seems to be to underestimate difficulty.

[mantha16]

maybe testers could offer suggested ratings?

[WillLem]

What really matters is having some kind of system to help players find content at the level they want to play at.

Yes, exactly! +1 for this.

i rate all mine as easy and icho said some of my levels arent as easy as i think so id have no idea how to rate them lol

Your levels are definitely not "easy", I'd say they're intermediate and up. In the above suggested scale, I'd say yours are usually around the "4" mark, with the occasional one in "3" territory when you're feeling generous (or haven't backroute-proofed the level!)

My tendency seems to be to underestimate difficulty.

I think most creators do, to be honest. It definitely seems to be a recurrent theme/issue in these conversations.

maybe testers could offer suggested ratings?

This is a great idea. It would help to encourage use of a universal ratings system as well, since creators, testers and players would need to be aware of the general definitions.

[WillLem]

OK, so... just to pad these definitions out a bit based on discussion so far:

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, and/or with very obvious solutions.
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills.
3 - Levels with fewer possible solutions and requiring more specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done if the skillset is limited. Design features such as tight time limits, red herrings, limited skillsets and difficult save requirements may start to appear at this level of difficulty, but will usually be there mainly to enhance an otherwise straightforward level.
4 - Levels requiring one very specific and relatively obscure solution, with few if any clues for the player and usually only providing the necessary skills for that solution. Levels with this rating will usually not provide alternative solutions, and will be designed specifically to prevent anything but the intended solution. Design features will be more unforgiving, and expect a high degree of skill from the player.
5 - Levels requiring one very specific solution involving very advanced (or even brand new) tricks and methods not widely known to the community, and only providing the necessary skills for that solution. These may also be particularly complicated levels requiring a lot of time and effort to solve.
6 - Indicates levels with an extreme degree of complexity, with the expectation that only the highest-skilled players will be able to solve the level.

What does everyone think? Should there be more levels of difficulty? Are the definitions too loose/not loose enough?

And, importantly - would this system be usable?

[Strato Incendus]

Okay, based on those definitions, I get the impression that I might finally be able to approach level 6 territory with Lemmings Open Air... ^^ But considering there is one level from Pit Lems that kaywhyn is currently stuck on, maybe even that pack has a few select "outliers".

I'm not sure I will ever make any difficulty 1 or 2 levels again. Unless I'm making tutorial levels - which I actually quite enjoy. But the problem is, about halfway through designing those levels, I usually also want to find some way to still make them interesting to more experienced players, as well... which at best results in a tight skill limit and/or saving everyone, and at worst includes some simple but not-quite-obvious skill combination that will definitely come as a surprise for complete newcomers.

But yes, complete any-way-you-want-it levels, I think that's what original DMA Lemmings is for. The Fun rank does an excellent job at making X-of-everything levels... well... fun! While a hastily-made X-of-everything level will usually result in an ONML Tame level. You youself made "Tame Gone Wild" for a reason!

So I wish we would include fewer of those open-ended levels in our custom packs, and just put more emphasis on pointing complete newbies to original Lemmings first (to be played in NeoLemmix, of course ), for learning all the basic skills. This would free up much more design space for custom levels, instead of every custom pack having a first rank of 20 to 30 no-brainer levels.

[Proxima]

But yes, complete any-way-you-want-it levels, I think that's what original DMA Lemmings is for. The Fun rank does an excellent job at making X-of-everything levels... well... fun! While a hastily-made X-of-everything level will usually result in an ONML Tame level. You youself made "Tame Gone Wild" for a reason!

So I wish we would include fewer of those open-ended levels in our custom packs, and just put more emphasis on pointing complete newbies to original Lemmings first (to be played in NeoLemmix, of course ), for learning all the basic skills. This would free up much more design space for custom levels, instead of every custom pack having a first rank of 20 to 30 no-brainer levels.

I am starting to get seriously annoyed by this. You do not have the right to prescribe to other authors what difficulty level they should aim at. Just enjoy content at your level, and accept that some content isn't for you. (If you want to complain that we don't have a good system to help players find content, I would be totally on your side, but that complaint should not be directed at pack authors.)

Of course I agree that "hastily made" levels tend to be bad; that's true at any difficulty. But looking at the content that's been produced recently, I can see a lot of high-quality easier levels, such as Sammings and the early ranks of Lemminas. If you are still of the opinion that most usermade easy levels are more Tame-like than Fun-like, then -- and I mean this seriously -- maybe you should list the packs you've played / looked at, so that I can see which ones you haven't looked at and make recommendations? It would also be an interesting study, to help us look at how designing tendencies have changed over time.

As for "every custom pack having a first rank of 20 to 30 no-brainer levels", that hasn't been true for years -- the later Lemmings Plus packs start with a first rank that's mostly easier linchpin puzzles, and packs such as Nessy's, Nepster's and IchoTolot's likewise jump straight into medium or hard puzzles. In fact, I can't think of a large multi-difficulty pack that does start with a "Tame" rank, unless I look all the way back to Lemmings Plus I, and that is now seven years old.

[Crane]

To add to this, my pack "Holiday Lemmings '16" (which I still need to convert to the latest NeoLemmix version, because some levels are now impossible due to the deadly ceiling)... it only has 30 levels and the difficulty climbs quickly, but the first three levels start with X of everything to get you into the groove - however, each has replayability in the form of a talisman.

Level 1 - 20 of everything... easy enough starter level with some slightly tricky terrain to navigate.  Comes off as a classic 20-of-everything level. Has a talisman that challenges you to complete it without any builders.
Level 2 - 10 of everything... starts with a deadly drop (and more than 10 lemmings).  Completing it in under 1 minute nets a talisman.
Level 3 - 5 of everything... skills are more limited.  Has two talismans: complete with 100% (difficult because there's a steel floor under the trapdoor, so stopping the lemmings from going to the left requires 3 Builders), and complete with no Bashers (requires the use of bombers and careful skill conservation).

Level 4 - 3 of everything... doesn't have a talisman, and barely counts as an X of everything level, but you have to use what you have so 3 lemmings can help each other reach the exit through a large steel obstruction.

----

To mention another pack... I helped convert some levels in Revenge of the Lemmings to the Picnic rating which is largely Fun and Tame-like levels.  A couple of levels that stand out is "Hunting Season" and "Singularity" because the intended solution in their harder versions is disguised in some way.  The former, "Wish you had them?", has no Builders and it's not obvious how you get to the exit without them.  For the latter, "Duality", the limited skills mean you can't just brute-force your way to the top with Builders, and uses, in my opinion, an absolutely beautiful sacrifice as part of the intended solution.

And this is just me... when I played the Fun levels and realised they repeated in later difficulties (as well as the first 13 Tricky levels), I started to wonder with anticipation when the next repeat will appear!  In fact, the only non-special Fun level that doesn't repeat is level 3: "Tailor-made for Blockers", probably because it was actually tailor-made!  I tried to follow the same philosophy with the Picnic levels in Revenge of the Lemmings with, at the time, only "Picnic 29" not repeating (although a harder version was made for it later).

Finally, with all these easyish levels in Picnic, you can just skip straight to Hootenanny if you don't want to play them.

----

As mentioned before, one problem with difficulty is that, as a level designer, it's hard to accurately measure the difficulty of one's own level.  I thought the difficulty of my level "Dropship 12" was on par with high Mayhem or Havoc, given it uses a variant of a famous Mayhem technique, but feedback from others suggest it is one of the hardest levels I have ever made.  Granted, such overestimation or underestimation of difficulty is generally not too extreme... only one or two levels out.

Though not with Lemmings, I have used a similar system for rating difficulty in the form of stars.  Usually rated from 1 to 5 stars (although 0 can easily be added to this), and "Flashing 5-star" for something insanely difficult.  I've noticed a lot of packs rate the highest difficulty as "Extreme".

[Strato Incendus]

@Proxima: Fair enough; I just happen to currently be playing Lemmings Plus I again, so it's very prominent in my memory. NepsterLems also starts with these open-ended levels. I have indeed played a good chunk of Lemminas; haven't looked at Sammings yet, though, so thanks for the suggestion!

And that's also what my post was - a suggestion, a wish, and a description of what I prefer. Not a prescription.

[mantha16]

personally I deliberately try to design easier packs as i feel theres easier pack deficit (in terms of new content) but I dont even create what I consider tutorial levels at least not on purpose.  I assume people that download original content that aren't meant to replicate original lemmings games have some level of knowledge so don't need tutorial levels.  Plus I don't consider difficulty rankings so.  I'm interested in theme hence packs like Sammings and Sammings2.  I'm interested in style hence All The Styles and the yet to be resurrected TVLems and 6 Nations Lems.

I kinda get a bit annoyed by bitching about other peoples packs I mean sure offer constructive criticism but if someone doesn't like a pack then they don't have to play it. Besides which builderfests, x of everything levels are great for challenges and for replaying, like if I replay one of those levels I try for a different solution everytime.

I do think difficulty ratings would be really useful for people I just dont think that authors are the best judge of difficulty since they know the solutions.  Maybe players could offer their views and after a majority vote the pack moved to a sub-thread for whatever difficulties are agreed on. 

Obviously opinions will vary

[WillLem]

This discussion kinda petered out, but I've had a number of comments on the advertised difficulty of my packs recently (and changed Lemminas to "Medium" as a result), so I think it's worth reviving this topic.

It is difficult to know exactly where to place your pack on the difficulty scale if there is no difficulty scale. All it would take is for us to come up with an agreed list of criteria which designers can refer to, make it official (i.e. give it its own sticky topic), and suggest that designers use it wherever possible.

Of course, some designers will ignore it, or not know about it, and still others may get it slightly wrong and make an incorrect guess as to the difficulty of their pack. But, if the ratings and their criteria are sufficiently detailed and clearly explained, then this is less likely to happen. And those occasional packs which miss the mark will likely get picked up on anyway.

So far, I've made a few suggestions which received some interest but didn't really take off. And, I get it, maybe I'm not the person to make such a scale, but as long as custom content is regularly being created, and to encourage new users to find packs at their own ability level, I do think it needs to get done.

Some initial suggestions:

The scale should be viewed from the point of view of a complete beginner, so "Easy" would mean can't-miss, any-way-you-like, open-endedness, etc. Most of us would find Mayhem "Easy" nowadays, but we grew up playing Lemmings; the Mayhem levels are still "Hard" to a beginner, even in NeoLemmix.

The difficulty rating given to a pack should indicate its maximum difficulty, ideally. Whilst level designers may wish to include "Fun-style" levels early on in their packs, that doesn't necessarily need to be indicated in the pack's given rating.

[Gronkling]

Custom packs aren't generally played by complete beginners, if it's intended for complete beginners then "Difficulty: Beginner" like the introduction pack is fine. Most people who are going to be playing custom packs will have beaten the original games I imagine. Stuff that's about the same level as the original games would be "Easy", more standard custom levels "Medium", things that require an in depth knowledge of the game "Hard", and stuff that even the best solvers on the forum find really difficult "Very hard".

[WillLem]

Custom packs aren't generally played by complete beginners

Whilst that might be true (but is impossible to know for certain), difficulty ratings shouldn't preclude any level of ability.

Most people who are going to be playing custom packs will have beaten the original games I imagine.

I'm aware of several people on the forums who haven't beaten the original games fully yet. Also, again - your statement, whilst not entirely unfounded, is impossible to know for sure and therefore shouldn't form the basis for a standard, forum-wide difficulty rating.

The only thing we can be certain about is that there are people who have never played Lemmings before, and there are people who are very experienced at playing Lemmings. The only way to have all difficult ratings covered is to cover everything from Beginner and up.

[Proxima]

The only way to have all difficulty ratings covered is to cover everything from Beginner and up.

That's certainly true; but your original conclusion that difficulty ratings should be assigned from the point of view of a beginner simply doesn't follow. To a beginner, Fun 1 to 20-ish are "easy", late Fun and most of Tricky is "medium", Taxing and Mayhem are "hard", and everything from mid-tier DoveLems upwards is an indistinguishable porridge of "impossible". Such ratings are of no use to non-beginners wanting to know how hard a pack really is.

[WillLem]

I'm absolutely happy for this not to be done in the way that I've suggested, I'm just providing a starting point. The key message here is that something needs to be done about difficulty ratings.

I suppose the problem is that no two people are going to regard any particular pack as being the same level of difficulty, which is why I suggested viewing every pack from the POV of a beginner. OK, most people on the forums are not beginners - I get that - but we all have been, and I remember really struggling with Oh No! More Lemmings because of its advanced level of difficulty, which is way beyond that of original Lemmings.

There comes a point when levels' difficulty stops being quantifiable in degrees, as well. The levels just start to seem outright impossible. I'd say that here is where it's not really worth having a scale, because it no longer becomes useful. Here, we just need the rating to say "beginners, casual players and those of around average ability will likely find this pack basically impossible, whereas those of way above average and advanced ability will find it a suitable challenge".

Therefore, my suggested scale would be:

Beginner / Easy
Novice / Medium
Expert / Hard
Master / Advanced
Genius / Extreme
Legend / Super-Extreme*

*The latter of these is really an "extra" rating to denote that even highly skilled players will find the pack an extreme challenge could be useful. Beyond that, though, it just gets into silly territory where difficulty ratings start to become meaningless anyway.

It's worth noting that, whilst a lot of custom content is harder than the original games, once solved it's just as "Easy". That's why, again, I'd suggest using a beginner's POV to design whatever scale we use.

[IchoTolot]

We already have a scale that is widely used: Easy - Medium - Hard - Extreme

So I don't think there is action needed in that regard. Maybe mods should be able to hand out/change ratings as sometimes creators misjudge their packs, but other than that the current system seems fine to me. It's just that some packs could need a re-evaluation.

I don't agree that viewing every pack from the POV of a beginner is a good thing here as most stuff would then simply fall in Hard or Extreme. The original games aren't really a good scale anymore on top of that.

[Proxima]

OK, so... what do we do, then?

Firstly, we relax and have a cup of tea. I'm on your side here even if we disagree about the details, and I don't like being shouted at

Beginners really only need to know whether content is beginner-appropriate or not, so the first two ratings should be "Beginner" (suitable for complete beginners) and "Easy" (accessible to complete beginners and will begin to stretch them).

But intermediate/advanced players also need to have rankings that help them find content at a level most suitable for them. And for such players, the gulf between late DoveLems and late United is much wider than the gap separating late DoveLems from "Just dig", so lumping them into the same rank doesn't help anyone.

On a quick side note:

Lemmings Destination is a perfect example of a pack which describes its early ranks as "Easy" when they're far more comparable to the later puzzles in ONML, which itself I would probably regard as "Advanced". We can say ONML is not Advanced now, because we've all beaten it, but I still remember struggling as early on as the Crazy rank.

Lemmings Destination is a one-off example of a pack whose difficulty is clearly mislabelled. Everyone agrees on this. But I think it's the only example of a really clear mislabel, not a symptom of a wider problem.

And yes, ONML is not "advanced". "We struggled on it once" just isn't a good metric. I struggled on Fun 24 the first time. Lemmings is a puzzle game, and ideally all levels should be a bit of a struggle at first.

I was going to continue by making my own proposal for a difficulty scale, but then Icho posted, so I'll frame my proposal as a reply to Icho instead.

[Gronkling]

How I judge it:
Beginner - For new players who may still have trouble with some tricky/taxing levels
Easy - About the same level as the original games
Medium - Quite a few hard puzzles, definitely a step up from the original games, but doesn't require an in depth knowledge of how everything works. This is the standard level for most custom packs so makes sense to be in the middle.
Hard - Puzzles are very difficult, and generally an in depth knowledge is expected
Very Hard\Extreme - For if even the most advanced solvers in the forum find it difficult to solve

These ratings just seem pretty much intuitive and use the most common words for difficulty levels.

[Proxima]

We already have a scale that is widely used: Easy - Medium - Hard - Extreme

Sure. The problem is that right now, we can't agree on what those labels should mean (WillLem is proposing a recategorisation that would assign the same labels to different difficulties), and four ranks doesn't seem to be enough (e.g. Sammings and PimoLems are both currently labelled "Medium").

And... while I was writing up a proposal, Gronkling stepped in. Mine would be pretty much the same as his, except I think "Very Hard/Extreme" should be split up into two ranks -- we could call them "Advanced" and "Extreme".

Some examples of content I would put in each rank:

Beginner - Fun/Tricky; early LPI; early GemLems
Easy - Taxing/Mayhem/Crazy; early DoveLems; most of Sammings; early Lemminas
Medium - Wicked/Havoc; late DoveLems; mid-tier PimoLems; late Lemminas
Hard - late PimoLems; LPI/Psycho; early NepsterLems

... and I won't give examples for Advanced versus Extreme because I'm not at the level where I've played very much of that

[WillLem]

Firstly, we relax and have a cup of tea. I'm on your side here even if we disagree about the details, and I don't like being shouted at

Haha Who's shouting? Bold text is just to bring attention to a particular point, whereas CAPS LOCK IS FOR SHOUTING (that's why it's called SHOUTY CAPS)! Multiple exclamation points can also indicate a stronger or more incredulous tone, but I didn't even use those!!!!!

I've revised my earlier post slightly, and I agree with the following:

Beginners really only need to know whether content is beginner-appropriate or not, so the first two ratings should be "Beginner" (suitable for complete beginners) and "Easy" (accessible to complete beginners and will begin to stretch them).

(Suggested difficulty scale)
These ratings just seem pretty much intuitive and use the most common words for difficulty levels.

while I was writing up a proposal, Gronkling stepped in. Mine would be pretty much the same as his, except I think "Very Hard/Extreme" should be split up into two ranks -- we could call them "Advanced" and "Extreme".

All agreed.

Quick discussion of this particular point, for what it's worth:

And yes, ONML is not "advanced"... I struggled on Fun 24 the first time. Lemmings is a puzzle game, and ideally all levels should be a bit of a struggle at first.

Yes, but it's worth acknowledging the degrees between struggling on Fun 24 and struggling on, say, Crazy 2. The latter is a much more specific puzzle, and whilst it's a trick we're all familiar with, it's still definitely not "Easy". On Gronkling's suggested scale, I'd probably place ONML somewhere between Medium and Hard, even in retrospect.

To move the discussion forward a bit, though, I'm happy to concede that a lot of custom packs have compressed the lower end of the scale considerably, so Gronkling's suggestion is a good step up from the one I suggested. And, Proxima's suggestion seems an ideal further refinement as well

[WillLem]

We all seem to agree then, more or less, that the following scale is appropriate:

(I've placed the various ranks of Lemmings Redux onto this suggested scale).

Beginner - brand new players - Gentle
Easy - begins to stretch the player with basic puzzles - Quirky
Medium - introduces more complex puzzles and level design concepts - Zany / Manic
Hard - expects the player to have a solid grasp of the game, puzzles become particularly challenging - Lunatic
Advanced - requires more in-depth knowledge of a wide range of solving techniques, strategies and tricks
Extreme - challenges even the most accomplished solvers

The question now becomes - on what basis should these difficulty ratings be applied to packs?

I would suggest that only the difficulty level of the final rank needs to be indicated, since it can be assumed in the vast majority of cases that the pack will work its way up to that level of difficulty from one of the lower ratings on the scale. It also signals to the player the expected solving ability required in order to fully complete the pack.

Maybe, though, it could be appropriate in some cases to indicate that a pack does specifically include a rank of tutorial levels.

Thoughts?

[namida]

I would suggest that only the difficulty level of the final rank needs to be indicated, since it can be assumed in the vast majority of cases that the pack will work its way up to that level of difficulty from one of the lower ratings on the scale

Lemmings Plus Omega II and Lemmings Plus Alpha are of similar difficulty at the end. However, they are very, very different at the start - Lemmings Plus Omega II starts off fairly easy (probably about late-Tricky difficulty for the first few levels, though it does ramp up quite quickly), while Lemmings Plus Alpha is already post-Mayhem from the very first level. The idea that only the final rank's difficulty should matter is relying on the assumption that everyone who plays the pack intends to play it the whole way through, but some people might indeed want to just play the easier, early levels in a lot of packs.

I'm personally of the opinion that it's hard to give a very meaningful measure of difficulty. The best way would be to analyze feedback, or even better, directly have discussion with the players over what the difficulty is like - but this is very tricky to do for a new pack, even if it has been tested by a varied-skill-levels team of testers. However, I'd definitely say that if there is one, it would be more useful to reflect the whole pack, or even better to give a range; rather than to only reflect "what's the hardest this pack ever gets?"

[WillLem]

some people might indeed want to just play the easier, early levels in a lot of packs
---
it would be more useful to reflect the whole pack, or even better to give a range

Agreed, I suppose a range is more useful. Also, since most packs are ranked, it does make sense to indicate where on the scale it starts, and where it finishes.

Perhaps packs which cover a smaller range of difficulty could only mention their top level. For instance, in the examples you've given it may be appropriate to list Lemmings Plus Alpha as "Extreme", whilst listing Lemmings Plus Omega II as "Medium - Extreme".

[namida]

some people might indeed want to just play the easier, early levels in a lot of packs... it would be more useful to... give a range

Quoting like this is quite misleading. You've strung together segments that are not even from the same paragraph, let alone the same sentence, here.

[Gronkling]

Ideally you'll find the easiest levels quite easy, the medium levels match your skill level and the hardest levels push you. I think if you want a single point to judge difficulty on, the average "medium" difficulty of a pack makes more sense than the most difficult. Range is probably better though.

[WillLem]

Quoting like this is quite misleading. You've strung together segments that are not even from the same paragraph, let alone the same sentence, here.

My apologies, there was no intention to mislead here at all. I've adjusted the original message to include the full quote.

[WillLem]

What do people think about the idea of representing the various stages of difficult as numbers (or letters)?

So, the above scale could be represented thus:

1 - Beginner
2 - Easy
3 - Medium
4 - Hard
5 - Advanced
6 - Extreme

That way, packs could be labelled [Difficulty: 1-4] or [Difficulty: 5-6]

I'm not sure how I feel about this idea. I think I'm suggesting it on the grounds that it would save space in the topic title whilst indicating the range of difficulty presented by the pack, but words are always more helpful, particularly if someone isn't familiar with the suggested scale.

[WillLem]

I think we need an easier way of gauging the difficulty of our own levels. Maybe something like this helps:



Obviously, this graph was done very quickly and is by no means representative of all possible levels. However, it can be a useful indicator of where your level probably is on the difficulty scale.

In the above example, if the level provides 5 skills in total, and all 5 are required in the intended solution, then it's likely to be a "Medium" difficulty level (or possibly higher).

However, if there are 35 skills in total, and only 1 is required, it's likely to be a "Beginner" level.

Again, this graph does not suit all levels. Levels which give red herring skills or skills which are of absolutely no use in completing the level are obvious anomalies, as are levels which may only require a small number (<5) of skills, but for which the solution is extremely obscure. In these cases, however, the creator is likely to be more aware of how difficult their level is.

I'd suggest taking a look at your level: if it has a total of 10 or more skills, all of which must be used in a very particular way, it's likely to be quite a hard level. 5 or more, and it's likely to be Medium difficulty.

Comments, suggestions, agreements, elaborations?


EDIT: Perhaps "necessary" assignments must be further defined as "non-trivial". So, a gap requiring 15 Builders to get across when the level only provides 15 Builders is obviously not a "Hard" level, but the assignments are trivial, so the graph no longer applies in quite the same way. Such cases must see the gap as "1 obstacle requiring N of the same skill, with N being equal to the number of that skill which is provided", redefined as 1:1 for the purposes of this graph.


EDIT 2: It occurs to me that I could be equating difficulty with lack of open-endedness. I do genuinely think that the minute a level requires a particular skillset, and only provides that exact skillset, it becomes a difficult level. Oh No! More Lemmings (from Wild onwards) is a very difficult pack for anyone who has never played it before, and most custom levels are even harder.

I think what's happening is that the community are brushing open-ended levels away as "trivial" or "beginner", and only gauging the relative difficulty of the remaining puzzle levels, starting with "Easy" as something which the creator would expect most people to be able to solve. This seems problematic, but I can't quite put my finger on why.

[namida]

I think what's happening is that the community are brushing open-ended levels away as "trivial" or "beginner", and only gauging the relative difficulty of the remaining puzzle levels, starting with "Easy" as something which the creator would expect most people to be able to solve. This seems problematic, but I can't quite put my finger on why.

Most open-ended levels are trivial or beginner difficulty. Note that the key word here is "most", not "all".

[Strato Incendus]

I appreciate this graph, WillLem, but according to that, nearly all of my smaller levels would be considered "hard", because they usually provide exactly the number of skills the intended solution requires.

[WillLem]

Most open-ended levels are trivial or beginner difficulty. Note that the key word here is "most", not "all".

Maybe. But they needn't be, with a bit of imaginative map-making.

I appreciate this graph, WillLem, but according to that, nearly all of my smaller levels would be considered "hard", because they usually provide exactly the number of skills the intended solution requires.

Exactly. All of your levels that I have played so far are at least what I would consider "Hard".

[Proxima]

Replying to posts in the Lemmings Surges topic:

WillLem is right that the currently used difficulty labels are insufficient. While there are technically six ratings in use (Beginner, Easy, Medium, Hard, Very Hard, Extreme), the last two are very rarely used, and "Hard" is used for a huge range of difficulties. It's easier than it was for novice players to find content at their level, but for mid-range players such as Strato, WillLem and myself, it's impossible to tell whether a pack claiming to be "Hard" is at or below our level or way beyond us.

That said, I'm not really sure what we can do about it. If we tried to make a standardised difficulty scale using existing packs to illustrate each rating, that doesn't work for authors who haven't played the packs in question. Theorycrafting based on "number of assignments" or anything similar doesn't really capture the essence of difficulty.

Maybe we should encourage more of the people who play packs, rather than the authors, to indicate where they believe a pack's difficulty would fall on a scale?

[Armani]

Another question that came to my mind : Imagine there is a custom level pack named "A".  A's first rank has [medium] difficulty overall and final rank has [hard] difficulty overall. But the easiest levels of A have [easy] difficulty and the hardest levels of A have [very hard] difficulty. Then between [Medium-Hard] and [Easy-Very Hard], which one is more appropriate indication?

"Hard" is used for a huge range of difficulties.

I think this is very true. Most of the 'not-easy' packs claim themselves having "hard" difficulty.

Maybe we should encourage more of the people who play packs, rather than the authors, to indicate where they believe a pack's difficulty would fall on a scale?

As I play quite a lot of packs rating themselves as "hard", I'm more than willing to share my opinion on whether the pack's difficulty is indicated properly / which pack is harder between pack A and pack B or something like that once we establish difficulty scale everyone can agree with.

[IchoTolot]

Just to be clear:

My main problem in the pack topics was mostly that the scale here was already seen as 100% forum wide accepted and other users who were not part of the discussion were already told "Hey this is what we use!" despite no great forum wide agreement!

What do people think about the idea of representing the various stages of difficult as numbers (or letters)?

So, the above scale could be represented thus:

1 - Beginner
2 - Easy
3 - Medium
4 - Hard
5 - Advanced
6 - Extreme

That way, packs could be labelled [Difficulty: 1-4] or [Difficulty: 5-6]

I'm not sure how I feel about this idea. I think I'm suggesting it on the grounds that it would save space in the topic title whilst indicating the range of difficulty presented by the pack, but words are always more helpful, particularly if someone isn't familiar with the suggested scale.

This was the last post before the advertisement campaign through new pack topics began!

I highly veto against making it already marked as widely accepted at this point!



But back to the problem at hand:

1.) If we come to a scale widely accepted: I think it's best to use moderator powers to roll out the scale widely and base the rating on author+user experience. Packs already rated for a long time should receive the same/similar rating if no big objections in some cases are made.

2.) My problem with the currently suggested scale: The advanced case. What is advanced? It is more unclear than you think! It could come just after medium and before hard for example (from here on you need more "advanced" stuff).
We need a term that is crytal clear here! To adjust the scale another rank is needed. We could add one between "hard" and "extreme", but alse between "medium" and "hard" could work. We just need a term that is crystal clear in its meaning so that every user can see right away where it stands in the scale without seeing the whole scale before. "Advanced" does not achieve this.

3.) Just using plain number like the suggested 1-4 for example does not have much information value. New players don't know the scope of the scale or if 4 still means easy. And no, they won't likely read the topic for the scale even if it's pinned! That's why we need clear terminology!

[IchoTolot]

Now another post about level difficulty and why you cant simply put a graph on it.

From my experience the following holds true: For a good evaluation on a packs difficulty you won't get around having veteran users play the pack!

Simply saying if we have many skills and most/all of them have to be uses = hard level does not reflect the nature of levels.

Of course there are quite a few cases where this indeed fits, but you still would have too many exceptions.

I have seen many cases like that, where the majority of skill assignments (even if they were many) are crystal clear. Like if you have a gap with no terrain around it a builder has to go there and you have a skill that is 100% locked in place. Like that levels can get easy very fast if a lot of the skills are effectively locked in place.

Lemmings is PSPACE-complete (https://arxiv.org/abs/1202.6581) and it is very hard to judge its difficulty on easy countable variables. Let's just say you can programm chess computers, but a computer solving all kinds of lemming levels is a far distant dream if not even impossible.

A better approach would be using more vague but also more fitting terms like "entropy".

As I would double post my exaplanations, here I will simply link an older topic where we already discussed the matter: https://www.lemmingsforums.net/index.php?topic=3341.0

My first post in there describes the terminology.

We also had another topic discussion the option to reduce said entropy: https://www.lemmingsforums.net/index.php?topic=4023.0


As said term is hard to grasp the best difficulty evaluation is to have some experienced players go through the packs and judge their difficulty. Also, more experienced pack authors who played and solved many levels are more likely to judge the pack's difficulty closer to the experienced difficulty by the user from my experience.

[WillLem]

This post isn't as long as it looks, there are a lot of quotes in here! Still, grab a cuppa first if you're up for reading it

That said, I'm not really sure what we can do about it ... Theorycrafting based on "number of assignments" or anything similar doesn't really capture the essence of difficulty.

Sure, but that's why we're having this discussion, to try to "capture the essence of difficulty" and then make it into something workable.

Maybe we should encourage more of the people who play packs, rather than the authors, to indicate where they believe a pack's difficulty would fall on a scale?

Now you're talking. That's a brilliant idea! The only problem I can see with it is that it necessarily relies on the availability of players in such a position, meaning that there could be some length of time between a pack's release and its difficulty rating being assigned. A minor problem, maybe, but a problem nonetheless.

I still think that encouraging designers to get a handle on the difficulty of their own levels is important, not just for the sake of rating the pack, but for their own benefit as content creators; it can only help to improve and refine their output, as Strato has demonstrated.

Imagine there is a custom level pack named "A".  A's first rank has [medium] difficulty overall and final rank has [hard] difficulty overall. But the easiest levels of A have [easy] difficulty and the hardest levels of A have [very hard] difficulty. Then between [Medium-Hard] and [Easy-Very Hard], which one is more appropriate indication?

I always think that a pack should indicate its hardest difficulty; I'd go with "Medium-Very Hard" for the example you've given. Reason: it shows that the player can expect even the first rank to get fairly tough, and the pack as a whole to go into "Very Hard" territory by the end. It just seems more useful to the player.

I'm more than willing to share my opinion on whether the pack's difficulty is indicated properly / which pack is harder between pack A and pack B or something like that once we establish difficulty scale everyone can agree with.

Great stuff! I'm sure your opinion will be very highly valued

I highly veto against making it already marked as widely accepted at this point!

Fine, but you could have said something much sooner (i.e. when this conversation was last being had). I took the lack of replies following mine as a sign that there were no further disagreements or discussions to be had.

I think it's best to use moderator powers to roll out the scale widely and base the rating on author+user experience.

Awesome! This would be very helpful

Packs already rated for a long time should receive the same/similar rating if no big objections in some cases are made.

Why? Longevity has no value when there has been no prior, explicitly agreed understanding. In fact, the problem has come about for this very reason, so any previous scale should automatically be dismissed and re-assessed based on new understanding.

2.) My problem with the currently suggested scale: The advanced case
...
We could add one between "hard" and "extreme", but alse between "medium" and "hard" could work. We just need a term that is crystal clear in its meaning so that every user can see right away where it stands in the scale without seeing the whole scale before. "Advanced" does not achieve this.

Fair enough. I think that adding something between "Medium" and "Hard" might be better in this instance, since the tendency is for creators to underestimate difficulty; increasing terminology towards the lower end of the scale is therefore more useful.

I also agree that terminology (as opposed to characters) is far more useful; I'd be happy to scrap any notion of numbered/lettered ratings.

From my experience the following holds true: For a good evaluation on a packs difficulty you won't get around having veteran users play the pack!

I'm not entirely sure what you mean here to be honest...

Lemmings is PSPACE-complete (https://arxiv.org/abs/1202.6581)

My brain turned into melted marshmallow whilst trying to read and understand this

Let's just say you can programm chess computers, but a computer solving all kinds of lemming levels is a far distant dream if not even impossible.

Ah, OK. That makes sense. Thanks

A better approach would be using more vague but also more fitting terms like "entropy".

I see what you mean here. It basically adds finesse to my proposal that the "skills:intended uses" ratio is a good indicator of difficulty, since it accounts for exactly how those skills are intended to be used, and how well concealed those uses are in the level's design.

I still think that any level which only provides the skills required to solve it should almost never be considered "Easy." Beginner/tutorial levels are an obvious exception, as are levels for which the skills have obvious "use this skill here" points (e.g. a level with a swimming pool, a gap and a pillar which provides 1 Swimmer, 1 Builder and 1 Basher, or similar setup).

Even "Medium" difficulty should really only start to introduce the kind of puzzles which require a very particular solution to be used. Exactly how difficult that solution is to find would then move the level higher up the scale.

Your point about "Trickery" is also compelling: levels which rely on tricks to gain a "Hard" rating are almost always no longer "Hard" when a player knows the trick, meaning that the rating is entirely based on the player's level of experience (which, I would suggest, should not necessarily be the case).

As you've said: how well the trick is concealed, or the manner in which it's used in the level contribute far more to the level's difficulty than simply the trick's presence in the level. Such levels are, therefore, even harder to rate in terms of difficulty.

the best difficulty evaluation is to have some experienced players go through the packs and judge their difficulty. Also, more experienced pack authors who played and solved many levels are more likely to judge the pack's difficulty closer to the experienced difficulty by the user from my experience.

Agreed, again. This is a very good idea, and if players are happy to do this, then all we need to do is agree on a scale and begin to implement it as closely as possible.

It's worth remembering, though, that experienced/high-ability solvers are more likely to underestimate difficulty; expanding the scale to incorporate more degrees at the lower end of the scale ought to combat this, otherwise a lot of levels will end up getting lumped together as "Easy", which is exactly the problem we have now.

[IchoTolot]

What about replacing "Advanced" with "Harder"

"Harder" is clearly more difficult than hard and I would argue clearly lower than "Extreme".

[WillLem]

What about replacing "Advanced" with "Harder"

"Harder" is clearly more difficult than hard and I would argue clearly lower than "Extreme".

Maybe if we decide which levels of difficulty need a definition first, then the words will come naturally.