kaywhyn's and Icho's Advice for Level/Pack Creators - Always Make a Solving Replay!

[kaywhyn]

Yea, I feel the need to make this, especially as it's pretty long overdue. I think I remember Icho wanted to make a topic on this from a while back as well. It's going to be a rant of sorts, so make of it what you will!

Before I go on this rant, do note that I have yet to make and release a level pack of my own (it's currently a WIP), but I've been making levels for NL for the past 4, almost 5, years. This is long enough that I can offer this sound advice for any NL level/pack creator, and that is always make a solving replay for any completed levels you make and keep a master replay set for your levels/pack! In this way, in case a player who plays your level/pack has doubts on its solvability, which is really easy to do especially on really difficult levels as a lot of the time they give the appearance of an "impossible" level, you at least have proof that the level is 100% possible and that you the author didn't make a mistake when you made the level. You would also be able to send a replay of the level if requested by the player. This practice of always making a solving replay for completed levels is one that I've been doing right from the start of my level designing career, ever since I got into level making all those 4 years ago. Yes, I know that testing your own levels can sometimes be a lot of work, but it's a really good habit! We fortunately also have the mass replay check tool, and it's always a good idea to run replays of your levels/pack through it to make sure that everything is fine on your end!

All rightie, now starts my rant. Nothing irritates me more than when I'm playing levels made by someone else that turned out to be impossible after all, especially as I have wasted a lot of time in the past playing such levels that turned out to not be solvable! Icho can also tell you from his experiences, and he can sing you a song about playing impossible levels, asking the author for a replay, only to not get one because he/she didn't have one for the level saved.

Recently, I privately pre-tested a pack that's now cancelled and encountered a level that, while it was solvable via the normal solution, it had an impossible talisman! => I wasted about an hour trying to obtain something that was made impossible by the level design. This makes me wonder if the creator somehow didn't test the level properly for the talisman and/or put that talisman by mistake, as it was the only one for the pack. At the same time, the only way I saw that the talisman would be possible is if the Lemmings were able to step up into the containment area and hence be safe as they would no longer be in danger, but instead the Lemmings turn around because the wall is too high for them to be able to ascend it, and you can't afford to let that happen since they would walk out the level side and die.

So yes, this advice extends to and includes making a solving replay for talismans as well should you include any for your level/pack! I even asked if Armani was able to obtain the talisman, and when I was supplied his replay for the level, he didn't as well. The reason for this is I know he always goes for them, so the fact that he didn't obtain the talisman either supports that the talisman is indeed impossible to obtain.

Another thing that recently occurred last week that finally got me to make this topic was when a recently registered member who had been lurking silently on the Forums for the past several years posted and inquired about two levels in another user's level packs topic that they couldn't solve because the levels seem to not provide enough skills in order to make them solvable. While I haven't yet posted in the topic that the user posted in, I silently took a look at the levels they mentioned and I too couldn't solve them => I wasted a lot of time on two levels that don't seem to be solvable at all! Once again, another case of an author not properly testing the levels!

I definitely can't emphasize enough the importance of making a solving replay for levels you make and then keeping a master replay set for your levels/pack safe somewhere on your machine. Heck, make a backup of your replays/master replay set so that in case you accidentally delete it you at least still have a copy! I can honestly say that for many of my levels that I made all those years ago that I have begun to forget some of the details of my intended solutions, and this makes me very glad that I have made solving replays for them! I myself have yet to encounter someone who has played my levels and asking me for a solving replay because they're not sure about their solvability. 

[IchoTolot]

Totally agree here! 

A confirmation from the author that the level is indeed solvable, or a replay checker result file would be enough for me as a confirmation, but yeah level and replay should always come together in pairs!

[Mindless]

This is truly a problem that spans back to the beginning.  LARSPACK, which is likely the first Customize Lemmings level pack ever published, has an unsolvable level.  Oops!  Of course, they didn't have replay files back then, but the underlying problem is the same: "Did you test it?"

This is also why I'm trying (with limited success) to get authors to submit solutions for their level packs on the Lemmings Level Database.  It's not fun playing a difficult level when you're not sure there is actually a solution.

[Simon]

The first benefit is proof of solvability.

The second benefit is that you document the intended solution.

You'll forget a few of your own intended solutions within 5-10 years. Or you pass maintainership of your pack to another community member. In both cases, your documented solution helps judge backroutes.

-- Simon

[WillLem]

The three engines I currently maintain (SuperLemmix, RetroLemmini and NeoLemmix CE) can all be hard-coded to autosave successful replays. It's already default behaviour for all, but maybe we should just go a step further and make it non-optional?

Another idea might be to require the presence of a working .nxrp file in order to load a level. On first loading the level, we run the MRC for just that level and its detected replay (this would take milliseconds and would happen silently). If it passes, we can display "Verified Solvable" on the preview screen. If it fails or the replay is missing, we display "Unverified" instead. The user can still play the level if they wish.

Better yet, adapt the .nxlv format to contain replay data. The loading procedure then reads this data directly, and tests it to see if the level (and its talismans) are solvable. Getting the data into the level file would be easy enough: each time the level is played and solved, add the data to the level file. If data exists already, overwrite it. Create new sections for talisman data. Don't overwrite if the level isn't solved.

[Armani]

afaik, the impossible talisman that kaywhyn pre-tested wasn't because the author intentionally added an impossible talisman to their level, but rather because a bug fix ended up breaking the talisman. (Previously, Lemmings were able to step up higher than 6 pixels under very specific conditions.) Fortunately, that bug was fixed before the deadline, but since the level had been created before the fix, this issue occurred.

Another idea might be to require the presence of a working .nxrp file in order to load a level. On first loading the level, we run the MRC for just that level and its detected replay (this would take milliseconds and would happen silently). If it passes, we can display "Verified Solvable" on the preview screen. If it fails or the replay is missing, we display "Unverified" instead.

To me, this sounds like: If you don't provide solution replays when publishing your level, your level will be given a "Unverified" label on the preview screen.

I don't intend to release all solutions at the same time I publish my levels. (For all my previous level packs, I've always released the solution replays after some time had passed.) So I also don't want my levels to be marked as "Unverified" just because I didn't include the solutions right away.

Better yet, adapt the .nxlv format to contain replay data.

To me, this basically means the same thing: requiring authors to provide a solution together with the level, unless the replay data is somehow encrypted.

People on the Lemmings Forum have done a great job ensuring that their published levels remain solvable. The honor system has worked well so far.(outside of the broken talisman, it's just one person so I think it's more likely on their end) I believe we should continue trusting level authors to ensure their own levels are solvable, rather than enforcing something systematically.

[WillLem]

afaik, the impossible talisman that kaywhyn pre-tested wasn't because the author intentionally added an impossible talisman to their level, but rather because a bug fix ended up breaking the talisman.

Whatever the reason, the level was unsolvable, causing the player unnecessary annoyance and wasting a significant amount of their time. If there's a simple way to help prevent this, we should do it.

To me, this sounds like: If you don't provide solution replays when publishing your level, your level will be given a "Unverified" label on the preview screen.

Yes, exactly. If you fail to provide verification for your level, it should be marked as unverified.

I don't intend to release all solutions at the same time I publish my levels.

Why not?

People on the Lemmings Forum have done a great job ensuring that their published levels remain solvable. The honor system has worked well so far.(outside of the broken talisman, it's just one person so I think it's more likely on their end) I believe we should continue trusting level authors to ensure their own levels are solvable, rather than enforcing something systematically.

You're right that it works the majority of the time, but there will always be examples that slip through even when it isn't the author's fault. Unsolvable levels have been a problem since the beginning of custom level creation; if we can introduce a way to ensure that levels are solvable, we should do it. And as others have pointed out, there are other benefits to ensuring a solving replay is provided/available (i.e. documenting the intended solution).

Authors who (for whatever reason) don't want to provide verification can choose not to, but their levels should then be marked as unverified. They would still be playable, and if it's a well-established author such as yourself, I imagine the "Unverified" marker wouldn't really matter so much.

If people dislike the idea of it outright saying "Verified/Unverified", we could make it a symbol that's either monochrome (when unverified) or full color (when verified) like when a talisman is completed, and put it in the top left corner of the preview screen out of the way but visible.


I suppose the counterpoint would be: if the author doesn't provide a replay, the player can simply choose not to play the level. This in itself might help to incentivise authors to provide solving replays.

But, as we've seen, in practice this doesn't really happen. Players like to have new levels, that's what the custom engines are for. Built-in failsafes should really be an expected part of this.

[Armani]

Whatever the reason, the level was unsolvable, causing the player unnecessary annoyance and wasting a significant amount of their time. If there's a simple way to help prevent this, we should do it.

You're right that it works the majority of the time, but there will always be examples that slip through even when it isn't the author's fault. Unsolvable levels have been a problem since the beginning of custom level creation; if we can introduce a way to ensure that levels are solvable, we should do it. And as others have pointed out, there are other benefits to ensuring a solving replay is provided/available (i.e. documenting the intended solution).

I want to emphasize again that this incident began with the level of one specific user. And I think most people here are already aware that this user has caused other issues in the past, even aside from releasing an unsolvable level this time. I've been on this forum for six, almost seven years now and I've never felt that unsolvable levels were a problem serious enough to require any sort of drastic measure. In fact, despite having played countless levels over the years, this is the first time I've ever encountered an unsolvable one.

While unsolvable levels may have been a problem at the very beginning of custom level creation, they have become extremely rare as time has passed and as level design culture has developed. The history of Lemmings custom level design is long, and of course various issues have come up along the way. But NeoLemmix has grown through many discussions and community consensus and from that the NeoLemmix philosophy was formed.

Whenever a new designer published levels containing elements considered "bad design," the community explained why those elements were problematic and offered advice on how to improve. We haven't ignored or disincentivized the levels just because that's the simple method. I think the same approach is sufficient here as well. We can simply continue advising authors on why maintaining a working replay is beneficial for everyone (documenting the intended solution, providing it when asked etc...) And in fact, kaywhyn has already done an excellent job of this in this very thread.

Yes, exactly. If you fail to provide verification for your level, it should be marked as unverified.

Authors who (for whatever reason) don't want to provide verification can choose not to, but their levels should then be marked as unverified. They would still be playable, and if it's a well-established author such as yourself, I imagine the "Unverified" marker wouldn't really matter so much.

For me the marker actually would matter. It's true that long-time forum members wouldn't doubt the solvability of my levels but new users join from time to time, and I would not want my levels to carry a label that might unintentionally suggest something is wrong with them. The word "unverified" can easily create the wrong impression, even if that's not the intention.

Why not?

I prefer not to release all solutions at the same time I publish my levels because I enjoy seeing how far players can get without having the answers available from the start. For me, part of the fun of creating levels is watching people try to solve them on their own first and many players also enjoy the challenge of approaching a new pack without immediate spoilers. That said, whenever someone tells me they're stuck on a particular level and asks for the solution I always provide it. This has been the case for every pack I've ever released without exception.

Also, it's worth noting that on this forum, it's actually quite rare for authors to release full solution replays upfront. Only a small number of packs include solutions from the start. Most authors publish their levels without replays and later respond by confirming whether a player's submitted solutions are intended. My approach is very much in line with that long-standing community standard.

I suppose the counterpoint would be: if the author doesn't provide a replay, the player can simply choose not to play the level. This in itself might help to incentivise authors to provide solving replays.

I see where you're coming from but I feel that this would effectively turn into a disincentive for authors who choose not to include a replay. I think providing solution replays should remain entirely up to each author and they shouldn't be penalized in any way for their choice.

[Dullstar]

I prefer not to release all solutions at the same time I publish my levels because I enjoy seeing how far players can get without having the answers available from the start. For me, part of the fun of creating levels is watching people try to solve them on their own first and many players also enjoy the challenge of approaching a new pack without immediate spoilers. That said, whenever someone tells me they're stuck on a particular level and asks for the solution I always provide it. This has been the case for every pack I've ever released without exception.

Adding to this: It's REALLY difficult to gauge the difficulty of a level you've just created, because you already know how it's supposed to be solved, and you can get all sorts of feedback as a result of people attempting the level; even if you only get back a set of replays, you'll see that the hardest levels (for that player at least) remain unsolved. If someone spoils themselves, though, then they can generate their own replay for the level, and you lose that feedback unless they explicitly tell you that they had to look it up.

[namida]

I don't think there's anything wrong with it (even if it's not what I'd do myself) if an author never chooses to release their replay files, let alone just delays them a bit after the release of the pack (which I did do with my own packs) - but they personally should still have them, and be able to run a mass replay check and confirm every level is solvable.