[SUG] Update contest format to include/encourage more players/designers

[WillLem]

Someone has to say it.

Pretty much the same people always win LOTY and LDC. Here are the top 3 designers in every contest for the past 5 years. If listed twice, the designer placed in the top 3 with 2 separate levels. "/" denotes joint winners:

2023 LOTY: NieSch, Armani/Icho, (3rd place TBA)
LDC 29: Armani, kaywhyn, Icho
LDC 28: Kingshadow, Armani, Simon
2022 LOTY: Armani, Icho, Icho
LDC 27: Simon, Simon, NeiSch
LDC 26: NeiSch, NeiSch, Armani
LDC 25: Armani, Icho, (no 3rd place)
2021 LOTY: Nessy, Icho, Icho
LDC 24: tan x dx, Icho, Kingshadow
LDC 23: Kingshadow, Armani, (no 3rd place)
LOTY 2020: DireKrow, Armani, Icho
LDC 22: Armani, Nessy, (no 3rd place)
LDC 21: Icho, Armani/Armani, (no 3rd place)
LOTY 2019: Icho/Namida, Crane, Icho
LDC 20: DireKrow, Armani, (no 3rd place)
LDC 19: Icho/Flopsy, Armani, (no 3rd place)
LDC 18: Namida, Crane, (no 3rd place)

I've put the designers that only appear once in bold. There are only 3 unique designers out of a possible 51 (top 3 multiplied by 17 contests)

Here's the final stats for all 51 award slots:

Armani, Icho: 13 appearances each
NieSch: 4 appearances
Simon: 3 appearances
Crane, Kingshadow, Namida, Nessy: 2 appearances each
Flopsy, kaywhyn, tan x dx: 1 appearance each
(3rd place not awarded 7 times)

Firstly, congratulations to Armani and Icho for setting and maintaining the high standards of level creation that these contests enjoy - let's be clear that the quality of their levels is not in question here at all, nor is the value of their participation!

However, these 2 designers account for more than half of the total possible appearances in the top 3 of all contests. Honestly, we can probably be doing more to make sure other designers get a look-in, particularly when it comes to making it into the top 3.

In fairness, the actual 1st-place winner of each contest is more or less different each time, but the point here is that the pool itself is often very similar. This is particularly true when looking at the final round of each contest; it's nearly always the same 4 or 5 designers every time, with maybe the occasional wild card from someone who made a particularly good level that contest.

What I'd suggest is adapting the format of these contests slightly with the goals of (a) encouraging more participation, and (b) increasing the likelihood of different designers making it into the top 3. I'm sure that Armani and Icho will always be strong contenders, and rightly so, but we need to see other names up there more often than we are doing at the moment, otherwise more and more people will tune out, and this will only exacerbate/perpetuate the problem.

My hope is that the following suggestions will open the contests up a bit and ensure that, even if Armani and Icho always make the top 3, maybe that 3rd person will be someone different each time, and more designers will benefit from the encouragement of having had one of their levels make it to the finals.



On to the suggestions:

There were 7 contests in which a 3rd place wasn't awarded. Awarding Gold (1st), Silver (2nd) and Bronze (3rd) ought to be standard in every contest - this alone would ensure that more designers get to see their name in the top 3 slots.

In the case that 2 of the same designer's levels make it into the top 3, either choose only the one that got the most votes or (probably better) count them as a joint award for that slot, and award the next slot to the designer with the next highest amount of votes. Or, to put it another way, we're awarding the designer rather than the individual levels in this particular case, and so we can place equal recognition on both levels. However it ends up being formulated, the end result should be that no fewer than 3 designers end up in the final round. Example:

       Armani gets 8 votes for Level A and is awarded Gold (1st place)
       Icho gets 7 votes for Level B, 6 votes for Level C and is awarded Silver (2nd place) for both levels combined
       (SomeoneElse) gets 5 votes for Level D and is awarded Bronze (3rd place)

Better yet, only allow 1 entry per designer per contest, with the possible exception of LOTY. (EDIT: This suggestion has been updated in response to Armani's post)

If keeping the "3 rule" format, allow all 3 winners of the previous contest to choose 1 rule each, rather than the top spot winner choosing all 3. This in itself would likely incentivise more participation; making it into the top 3 suddenly means more, and a designer who manages to make it that far is rewarded for doing so. This is particularly meaningful when there are only 1 or 2 votes in the difference between the top 3, which is sometimes the case.

Alternatively, the "3 rule" format could either be reduced to 2 rules (maintain an element of choice whilst reducing workload and possible confusion for participants) or scrapped altogether in favour of a single rule (easiest to follow/maintain and could help make contests shorter, so they could be run more often and therefore provide more chances to win, but could also have the undesired side-effect of reducing participation due to lack of interest in the rule).

In many cases, LDC levels that placed in the top 3 also end up in the LOTY top 3 for that year. Perhaps implement a contest entry condition preventing the same level from entering 2 different contests. The level has already been awarded an accolade previously - giving it another award, whilst perhaps deserving, prevents other designers/levels from receiving recognition. Preventing this is an easy way to ensure that different levels (and, perhaps therefore different designers) get a fresh chance with each contest. And, if the same designer is awarded a LOTY top 3 place for a different level than their LDC top 3 place, it's a chance for that designer to be recognised for more of their work, arguably making the award more valuable to them personally. Conversely, perhaps we deem it necessary to bestow more recognition on a particularly good LDC winner. One way to achieve this would be to have a separate "Contest Level Of The Year" award, which pits all of that year's contest winners against each other for one final vote.

I hope these suggestions are useful. If nothing else, it might be worth trialling a few of them for the next few contests and see if it amounts to at least an uptick in participation. Or, if people don't think there's a problem with the way things are now and this just comes across as complaining, that's fine. Feel free to ignore this post and I'll go back to ignoring the Forum contests (it's not as if I haven't got plenty else to be getting on with!)

But seriously, I'm making these suggestions in the hopes that it might benefit other designers who are discouraged from participating due to thinking there's no chance of winning, or even simply due to confusion regarding the format. There's actually very little for me to gain here personally, I'm just being a voice.

[Armani]

The whole point(or the most important point) of Level Design Contest imo is enjoying the process of creating the levels itself under particular restrictions. The challenge comes from making a level that meets all the given conditions and that's the fun part. So for me, only allowing 1 entry per designer per contest is just reducing the opportunity of having that fun and seems like the tail is wagging the dog. Also, it's worth mentioning that people were overwhelmingly in favor of allowing entering all 3 entries instead of 2(The poll ended 7:1):
https://www.lemmingsforums.net/index.php?topic=4573.msg79097#msg79097


Other than that, I'm not particularly against anything here.

There were 7 contests in which a 3rd place wasn't awarded. Awarding Gold (1st), Silver (2nd) and Bronze (3rd) ought to be standard in every contest - this alone would ensure that more designers get to see their name in the top 3 slots.

In the case that 2 of the same designer's levels make it into the top 3, either choose only the one that got the most votes or (probably better) count them as a joint award for that slot, and award the next slot to the designer with the next highest amount of votes. Or, to put it another way, we're awarding the designer rather than the individual levels in this particular case, and so we can place equal recognition on both levels. Example:

Especially I think these two are worth a try.

[WillLem]

The whole point(or the most important point) of Level Design Contest imo is enjoying the process of creating the levels itself under particular restrictions. The challenge comes from making a level that meets all the given conditions and that's the fun part.

Agreed, so we should do as much as we can to ensure that more people get to enjoy this, not just pretty much the same people each time

So for me, only allowing 1 entry per designer per contest is just reducing the opportunity of having that fun and seems like the tail is wagging the dog.

I had to look up "tail wagging the dog" because I don't know what that means. The most relevant thing I could find is that it's a large group of people doing something to satisfy a small group of people.

Assuming that is what you mean, I'm not sure how that applies here, since I'm advocating for the many Forum members who don't participate in level design contests, and asking the tiny group of people who always participate and even smaller group who regularly make it into the top 3 to consider a different approach to the contests in the hopes that more people will participate (and, thus, get to enjoy the benefits of the contest that you mentioned).

But OK, maybe reducing entries to only 1 per designer isn't the answer here (and, I wasn't aware that there had been a vote on this!) - In that case then, I'd change that particular suggestion to one of the following:

Designers can enter as many levels as they like, but can only proceed into the final round with 1 level. The designer gets to choose which level, or we automatically default to whichever of their levels gets the most votes. Ultimately, there should be 3 designers in the final.

Designers can enter as many levels as they like, but again, since we always want a total of 3 designers to progress into the final round, we might need to group a designer's levels together if 2 or more receive a high number of votes. So, for example, in this pre-final situation:

   Participant A has 8 votes for Level A and 7 votes for Level B
   Participant B has 6 votes for Level C
   Participant C has 5 votes for Level D
   Participant D has 4 votes for Level E
   Participant E has 3 votes for Level F

The outcome should be that Participant A progresses to the final round with both of their levels, Participants B and C progress with each of their levels, and Participants D and E are eliminated.

These are essentially unpacking the second suggestion in the OP a bit more, tbh, to which you've already expressed agreement. Basically, whatever is decided, the point is that the final round should always consist of a group of levels by no fewer than 3 designers, however many levels that ends up being. The goal here is to get more designers into the final round, to encourage more participation, so that more people can enjoy the benefits of having entered the contest.

Winning isn't everything, of course, and the joy of designing a well-made level can absolutely be its own reward, we agree on that. But, don't forget that it's much easier for a person in your position to adopt that particular mindset, because your levels have received recognition and acclaim on the Forums, and you have won 3 contests and reached the top 3 in another 9. Imagine how you'd feel if your highest-quality levels never even made it to the final. Surely you must accept the possibility that this might kill some of the joy of entering these contests, particularly if that consistently happens to the same designer over and over again, in spite of the obvious quality of their levels and the hard work and effort they've put in to try and make it a bit further this time. People who enter contests are usually motivated by winning until they win, then they get the luxury of being motivated by participating at all

Don't get me wrong here, I'm not suggesting that we lower the standards or move the goalposts - I absolutely expect that yourself, Icho, NieSch, Simon and some of the others who have consistently performed well in contests over the years will continue to shine during these events - we need that to keep things competitive and give participants something to aim for, and you have my congratulations for doing so well at setting and maintaining these standards. I've updated the OP the reflect this sentiment - this post is not about winner-bashing, it's about re-assessing the contest format to allow more people into the winners' circle.

So, we need to make sure we're acknowledging as many designers as possible, and grouping a designer's levels (in the event that they end up with 2 or more highly-voted levels) to allow other designers into the top slots is, I think, a reasonable adjustment which doesn't upset the format too much.

[Armani]

Sorry WillLem! My English is not as good as I would like and looks like I didn't use the idiom in appropriate context.

What I meant was:
That particular tweak you suggested(1entry per designer) was based on good intention(encourage people to participate in contests) but would possibly end up getting something not good to what is more important.(reduced opportunity of level designing)

Though awarding 3rd place is not hard to do(if we decide to do so, the mixed round with 3levels becomes the final and that's it) and an easy way to let more designers get appearance in the high rounds.
Always letting 3 different designers to progress into the final round needs more refined and extra rules, and probably more works to do for the contest organizers, but still something we could try.

[IchoTolot]

Comments on the suggestions so far:

There were 7 contests in which a 3rd place wasn't awarded. Awarding Gold (1st), Silver (2nd) and Bronze (3rd) ought to be standard in every contest - this alone would ensure that more designers get to see their name in the top 3 slots.

In the case that 2 of the same designer's levels make it into the top 3, either choose only the one that got the most votes or (probably better) count them as a joint award for that slot, and award the next slot to the designer with the next highest amount of votes. Or, to put it another way, we're awarding the designer rather than the individual levels in this particular case, and so we can place equal recognition on both levels. However it ends up being formulated, the end result should be that no fewer than 3 designers end up in the final round.

Making sure that the top three are always awarded should be no problem.
I am also open for puting the focus more on the designer rather than the level for the final placement.

For LOTY specifically I would like to keep the focus on the individual levels though.

If keeping the "3 rule" format, allow all 3 winners of the previous contest to choose 1 rule each, rather than the top spot winner choosing all 3. This in itself would likely incentivise more participation; making it into the top 3 suddenly means more, and a designer who manages to make it that far is rewarded for doing so. This is particularly meaningful when there are only 1 or 2 votes in the difference between the top 3, which is sometimes the case.

Also open for this and I think this is a good thing.

Alternatively, the "3 rule" format could either be reduced to 2 rules (maintain an element of choice whilst reducing workload and possible confusion for participants) or scrapped altogether in favour of a single rule (easiest to follow/maintain and could help make contests shorter, so they could be run more often and therefore provide more chances to win, but could also have the undesired side-effect of reducing participation due to lack of interest in the rule).

I would oppose puting the rule count down to 1 as this can severely reduce participation when the rule is not to everyone's liking.
I tend to say that the rule count is not the main thing to adress your issue.

Also more frequent contests can be counterproductive --> design fatique. So I would be careful here.

In many cases, LDC levels that placed in the top 3 also end up in the LOTY top 3 for that year. Perhaps implement a contest entry condition preventing the same level from entering 2 different contests. The level has already been awarded an accolade previously - giving it another award, whilst perhaps deserving, prevents other designers/levels from receiving recognition. Preventing this is an easy way to ensure that different levels (and, perhaps therefore different designers) get a fresh chance with each contest. And, if the same designer is awarded a LOTY top 3 place for a different level than their LDC top 3 place, it's a chance for that designer to be recognised for more of their work, arguably making the award more valuable to them personally. Conversely, perhaps we deem it necessary to bestow more recognition on a particularly good LDC winner. One way to achieve this would be to have a separate "Contest Level Of The Year" award, which pits all of that year's contest winners against each other for one final vote.

Here I stand strongly by the term "LEVEL of the year". Doesn't matter where that level originates.

I would also not want to split the contest up.



In my opinion, one of the main reasons (besides level design experience) why mostly the same names pop up: Consistency in terms of entering the contests.
In my case, I simply entered every single contest and prepared levels for it. Even if I missed the final round regularly, just the fact I enetred every single time and threw my name+level in the ring makes a huge impact on the result.

[Simon]

WillLem, what exactly do you want to see instead?

You count the number of designers who have shown (= placed 1st, 2nd, or 3rd) exactly once in the past n contests, and you value a high count for this. Let's assume Nessy did 50 % worse than he really did (2 showings), i.e., assume he showed only once. This will improve your metric by 33 % or by 66 %, depending on who takes the now-open slot. Is that desirable?

I don't particularly mind switching from levels to desigers for top 3, but it won't fix your problem either. Your proposal brings a change only when you see 1 or 2 designers sharing top 3 levels. Your example is that someone at random will get 3rd under your proposal. But the much more likely outcome is that one of the frequent showers gets promoted from 4th to 3rd.

There are only 3 unique designers out of a possible 51 (top 3 multiplied by 17 contests)

3 may be small, but more than 6 or 7 would be odd, too, when we have prolific core contributors.

Certainly, I don't expect 51 different people showing exactly once. I would expect that if we had 1,000,000 designers participating in those 17 contests. I'm more surprised that Icho and Armani tie exactly than that there are exactly 3 exactly-one-point scorers.

If we ignore strength (= a designer's ability to create levels that show in the top 3), I expect the number of showings per designer to follow a Poisson distribution: Many people score low, some people get medium scores, and a few people get high scores. Your data is roughly shaped like one: Many people have no hits (and you aren't listing them), then you have 7 with one or two hits, then you have 2 people with more hits but not the maximum, and 2 more with the maximum.

I'm not sure if the Poisson distribution is really the appropriate model after we begin looking at designers' strengths. Poisson describes independent events scattered across a-priori equal targets.

-- Simon

[WillLem]

Sorry WillLem! My English is not as good as I would like and looks like I didn't use the idiom in appropriate context.

I assure you, your English is always great. It's just not an idiom I'm familiar with so I had to look it up! Thanks for explaining what you meant

That particular tweak you suggested(1entry per designer) was based on good intention(encourage people to participate in contests) but would possibly end up getting something not good to what is more important.(reduced opportunity of level designing)

Yes, having given it more thought, 1 entry per designer probably isn't the way to go. Agreed.

Making sure that the top three are always awarded should be no problem.
I am also open for puting the focus more on the designer rather than the level for the final placement.

Glad to have your agreement on these points, hopefully this helps to encourage participation and gives more designers a decent shot at placing in the top 3.

I would oppose puting the rule count down to 1 as this can severely reduce participation when the rule is not to everyone's liking.
...
Also more frequent contests can be counterproductive --> design fatique. So I would be careful here.

Yes, agreed. I realise that wasn't the best suggestion. I'm glad that my general point seems to have been made well enough, though, and that we might get to see a refreshing update to the contest format in other areas.

Here I stand strongly by the term "LEVEL of the year". Doesn't matter where that level originates.

OK, fair enough. I still think it would be better to have unique levels win each contest, but I guess we can't agree on everything. Let's move on from that, then.

In my opinion, one of the main reasons (besides level design experience) why mostly the same names pop up: Consistency in terms of entering the contests.

Agreed, this is pretty much why I made this post in the first place. It's as much a comment that the same people tend to enter as it is that the same people tend to win.

Maybe there should be some small but significant reward for anyone who enters a certain number of contests. Some Forums have things like reputation meters for users and other similar "levelling up" accolades. It's mostly superficial, but it does give users a sense of belonging and progress which might be a good way to encourage more Forum interaction in general.

NOTE: This suggestion is somewhat off-topic, admittedly. It needn't be responded to seriously here; I can always make a Site Discussion suggestion topic about that if the idea still feels strong in a few days or weeks.

WillLem, what exactly do you want to see instead?

I suppose it would be good to see a larger variety of users entering the contests in the first place, and a few more people (be they regulars or first-time entrants) getting into the final round having not done so previously.

You count the number of designers who have shown (= placed 1st, 2nd, or 3rd) exactly once in the past n contests, and you value a high count for this. Let's assume Nessy did 50 % worse than he really did (2 showings), i.e., assume he showed only once. This will improve your metric by 33 % or by 66 %, depending on who takes the now-open slot. Is that desirable?

Apologies, but I'm not sure I understand the question. Could you rephrase?

I don't particularly mind switching from levels to desigers for top 3, but it won't fix your problem either. Your proposal brings a change only when you see 1 or 2 designers sharing top 3 levels. Your example is that someone at random will get 3rd under your proposal. But the much more likely outcome is that one of the frequent showers gets promoted from 4th to 3rd.

Yes, I accept that's a possibility. But, one of those frequent showers might never have made it to the final round - the proposed change increases the chance that they will, by however small a margin.

And, in response particularly to the highlighted comment: this happens in 4 out of the 17 contests listed in the OP, which I'd say is a significant enough amount to warrant the proposal even without further argument. However, to hopefully add more weight to it, the fact that entrants can post multiple levels increases the designer's chances of winning if they enter for all 3 rules; grouping a designer's levels (should >1 of theirs make it to the final round) levels out this playing field, which I'd say is a very significant update to the format.

Possible scenario, to illustrate this:

Current Scenario: Designer A enters for all 3 rules, and is a regular contestant with a proven track record of strong levels that make it very far in the contest. Designer B can think of a good level for one of the rules, but is put off entering at all because they see that Designer A has entered 3 levels, and the chances of one or more of those nudging their own level down the list seems too high to go to the trouble of creating and maintaining a contest entry.

Possible Scenario, with proposed update: Designer A enters for all 3 rules as above. This time, Designer B knows that even if all 3 of Designer A's levels makes it all the way to the final (admittedly unlikely, but possible), they still have a shot at making it there themselves because Designer A will only take up 1 slot rather than all 3. They go ahead and enter their level, deeming it worth the effort.

3 may be small, but more than 6 or 7 would be odd, too, when we have prolific core contributors.

Perhaps, but I'd be willing to bet that we get to at least 5 unique designers over the next 15 contests with the proposed changes (i.e. those to which Icho has already agreed - always award 3 places, 3 designers in the top 3 rather than 3 levels, and allow all 3 to choose a rule each), which would be a significant enough improvement IMHO.

[Dullstar]

It might be interesting to go a bit further and examine a few more things:

How many designers enter? How far do they normally get?

If we think recognition is helpful for encouraging participation, then I think there might be some merit to having another voting category that explicitly excludes levels from the strongest competitors (as determined by past contest results).

[IchoTolot]

As we are getting close to the results of the current contest:

Should I start awarding the top 3 designers here (with it being possible that one person has multiple levels in the final bunch), or again go purely by levels?

EDIT: Actually it will be best to start a new system (if it is voted for) with the next contest.

[Proxima]

I voted No. It seems to be very rare that the same designer has more than one level in the top 3 (only once out of the 12 contests in WillLem's OP) so the change wouldn't do much to solve the problem (even if we agree that there is a problem) -- and it would make the voting system more complex and potentially lead to needing even more voting rounds, when we have too many as it is.

[WillLem]

it would make the voting system more complex and potentially lead to needing even more voting rounds, when we have too many as it is.

It shouldn't need more voting. In fact, the top 3 can be determined with a single voting round. Award first place to most votes, second place to second most votes, third place to third most votes. No need for more voting rounds.

[namida]

it would make the voting system more complex and potentially lead to needing even more voting rounds, when we have too many as it is.

It shouldn't need more voting. In fact, the top 3 can be determined with a single voting round. Award first place to most votes, second place to second most votes, third place to third most votes. No need for more voting rounds.

The issue is, let's assume half the users like Level A a lot and the other half hate it. Half the users like Level B a lot and the other half hate it. Everyone agrees Level C is very good, just not quite as much as their preference out of A and B. In such a case, my view is that C should win, but it would likely place 3rd in a single round.

Maybe we need to look for a poll addon that allows STV style voting...

[WillLem]

The top 3 should be all different designers. If the same designer has 2 levels in the final, we need to bring in whoever got the next biggest amount of votes to grab third place. It might mean that it's worked out differently for different contests, but there will never need to be more voting rounds: just grab whoever got the most votes from the previous round, even if they were eliminated in that round. That's just a suggestion, I'd ultimately be happy with any system that awards 3 designers: that's the goal here. The "how" can be figured out.

Again, bottom line, the top 3 should be 3 different designers. If Designer A has 2 levels in the final, group those levels. Or, whatever works. If we continue with the system of the same designer grabbing more than one of the 3 top spots, we reward less people. That's the problem which needs to be solved, IMHO, by whatever means works for everybody.

[IchoTolot]

In my point of view, I don't care if this would result in an extra voting round - even if the last 2 levels are from the same designer, I would still be curious what level would win here.


But fundamentally I still see both the regular and LOTY contests still as a "Level Design Contest" first. With the goal of finding the the best level/s regardless of its designer.

If a designer makes multiple extremely good levels both of those sould be rewarded - as those were the best levels. The amount of people getting in the top 3 I see as a side note.

[WillLem]

In my point of view, I don't care if this would result in an extra voting round - even if the last 2 levels are from the same designer, I would still be curious what level would win here.

Proxima wants to avoid this if possible, and I agree.

Here's a hypothetical voting round:

Designer A's Level A gets 7 votes
Designer B's Level A gets 6 votes
Designer B's Level B gets 5 votes
Designer C's Level A gets 4 votes
Designer D's Level A gets 3 votes

From this single round of voting, we can determine 1st Place (Designer A), 2nd Place (Designer B) and 3rd Place (Designer C). We can also determine that Designer B's Level A got more votes than their Level B.

We don't need more voting rounds.

But fundamentally I still see both the regular and LOTY contests still as a "Level Design Contest" first. With the goal of finding the the best level/s regardless of its designer.

Yes, that's why there's a problem. I'm proposing that we shift emphasis to the designer.

Let's say Designer A enters 3 levels into the contest, and they're a popular and highly regarded level designer with a track record of excellent, contest-winning levels. In theory, they could take up all 3 of 1st, 2nd and 3rd Place with just those 3 levels.

Perhaps what's needed is a system of "3 designers will be rewarded regardless of level placement". So:

Designer A enters 3 levels. The levels get 9, 8 and 7 votes respectively. The designer is awarded 1st place.
Designer B enters 1 level. The level gets 6 votes. The designer is awarded 2nd place.
Designer C enters 1 level. The level gets 5 votes. The designer is awarded 3rd place.

The point being that we're always interested in rewarding 3 designers, regardless of how many levels they've entered and how those levels have placed in the voting. We know from polling results how the levels themselves have performed in the contest, perhaps that's enough feedback.

As long as designers can enter more than 1 level, there is an unfair advantage given to any designer that enters +(<=1) level than any other designer. Rewarding the designer rather than the level addresses this and re-balances the contest.

If a designer makes multiple extremely good levels both of those sould be rewarded - as those were the best levels.

There is 0 chance of the levels going unrewarded in my proposal. The difference is that the designer only takes 1 of the top 3 slots, regardless of how many of their levels make it to the final. It would look like this:

Designer A - 1st Place - Level A (9 votes), Level B (7 votes)
Designer B - 2nd Place - Level A (8 votes)
Designer C - 3rd Place - Level A (6 votes)

Notice that Designer A's levels have been grouped in 1st place, because that designer got the most votes for one of their levels (Level A), and the other level (Level B) got more votes than the level submitted by Designer C. If their Level B had gotten, say, 5 votes, then that level wouldn't appear in the final or in the above list.

The point is that the following placement of the above levels is wrong, because it awards 2 medals to the same designer:

Designer A - 1st Place - Level A (9 votes)
Designer B - 2nd Place - Level A (8 votes)
Designer A - 3rd Place - Level B (7 votes)

Both of the above lists reward both of Designer A's levels, but Designer C misses out on the 3rd place slot when implementing a "reward levels only" system. The fact that Designer A's Level B received more votes that Designer C's level should be irrelevant here; we're rewarding Designer A by both giving them first place (for their Level A), and mentioning their other level in the results. Designer C gets to be awarded 3rd place.

This, IMHO, is the best way to balance the contest results.

The amount of people getting in the top 3 I see as a side note.

Yes, again, this is the source of the problem. This topic exists because of this exact viewpoint. If the contest isn't adjusted to shift focus to the designer to some extent, then nothing has really changed and the contest will continue to be unbalanced in favour of only the most prolific and popular designers, as it demonstrably has been for quite some time.

[WillLem]

Here's a more extreme example to illustrate what I'm proposing:

Designer A's Level A gets 100 votes
Designer A's Level B gets 98 votes
Designer B's Level A gets 99 votes
Designer C's Level A gets 97 votes
Designer D's Level A gets 1 vote
Designer E's Level A gets 1 vote
...
Designer Z's Level A gets 1 vote

Should Designer C be awarded 3rd place, or Designer A?

[IchoTolot]

Here's a more extreme example to illustrate what I'm proposing:

Designer A's Level A gets 100 votes
Designer A's Level B gets 98 votes
Designer B's Level A gets 99 votes
Designer C's Level A gets 97 votes
Designer D's Level A gets 1 vote
Designer E's Level A gets 1 vote
...
Designer Z's Level A gets 1 vote

Should Designer C be awarded 3rd place, or Designer A?

I would still say Designer A.

Again in my book the contest is purely about levels, not designers and I would not call it a problem just a different focus.

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!

The best/most popular levels win. The author is second fiddle here.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

But that gave me another idea:

How about we only post the levels without giving the authors name!

The authors will only be named after the final vote! This would remove next to all influence of the authors.

[Proxima]

In DROD contests, we do as WillLem is suggesting -- if the same designer places twice in the top 3 then their lower entry isn't counted and the 4th place moves up to 3rd.

This does make me wonder. Instinctively, I was against doing this for Lemmings contests, but why is that, if I'm okay with it for DROD?

I guess the main reason is that I don't see what it would achieve. There are no prizes for 2nd and 3rd place anyway. So I don't see how changing the way results are presented would do anything to encourage more designers to take part. It's always the case that as a new entrant into contests, you have a low chance of winning against the Icho/Armani juggernaut; but not no chance, and the best way to win a contest is to have patience and keep trying. Not everyone has the patience and time to do this.

[namida]

be unbalanced in favour of only the most prolific and popular designers, as it demonstrably has been for quite some time.

How is it unbalanced in their favor though? They win because they consistently submit the best levels. There's no special rule that gives IchoTolot or Armani an advantage over other levels; they're just really good at making levels.

Honestly, this is almost starting to feel like a "let's give other people participation trophies" and/or "let's list more than the top 3" angle in practice (as in practice, all we'd be doing is listing 4th place but calling it 3rd). Indeed, perhaps it's better to make no change to how places are awarded, but to adopt the suggestion of keeping the entries anonymous until results are announced (assuming a practical and reasonable way to handle updates can be found). Maybe that should be trialed for one or two contests before deciding on whether to make it permanent?

To be clear, the only idea here that I'm strongly opposed to (and even this, it's up to Icho and kaywhyn to make the final call, not me), is the proposal of determining multiple of the top places via a single voting round (rather than an elimination format). Mass-voting is okay in the earlier rounds but I really think (given that STV-style voting isn't feasible) elimination is the only right way to handle the closing rounds.

[WillLem]

Again in my book the contest is purely about levels, not designers and I would not call it a problem just a different focus.

Those who often win are unlikely to see a problem with the current system

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!

The best/most popular levels win. The author is second fiddle here.

It obviously isn't "100% not true". The most prolific and popular designers tend to win the contests; the data proves that. It's probably at least 30% true, otherwise why would this topic need to exist? I have very little stock in the level design contests themselves; my goal is to help improve Forum interaction.

How about we only post the levels without giving the authors name!

The authors will only be named after the final vote! This would remove next to all influence of the authors.

Very good idea, I'd be up for at least trying this for a contest or two to see how it goes.

However, I'd still want to see 3 names in the final results.

I guess the main reason is that I don't see what it would achieve. There are no prizes for 2nd and 3rd place anyway. So I don't see how changing the way results are presented would do anything to encourage more designers to take part.

The reward is being placed in the top 3, and having your name displayed there. A bronze medal is still a medal, and many people would benefit from having one of their levels recognised alongside the usual winners.

It's always the case that as a new entrant into contests, you have a low chance of winning against the Icho/Armani juggernaut; but not no chance, and the best way to win a contest is to have patience and keep trying. Not everyone has the patience and time to do this.

Right, which is why we need to re-balance the way the results are gathered. Those with multiple levels in the contest shouldn't have an automatic advantage over those with only one level in the contest. Any system that allows designers to enter more than one level optionally and then rewards the same designer twice is therefore unfair, or at least unbalanced. Ideally, all designers would enter the same number of levels; in that system, rewarding the same designer more than once would at least be more fair, but still problematic IMHO.

How is it unbalanced in their favor though? They win because they consistently submit the best levels. There's no special rule that gives IchoTolot or Armani an advantage over other levels; they're just really good at making levels.

Their reputation as good level designers perhaps affords their levels a certain amount of bias when it comes to voting. There's no way to prove this really, but I would be interested to see what would happen if we did hold a contest where the level author wasn't disclosed until the end of the contest.

Honestly, this is almost starting to feel like a "let's give other people participation trophies" and/or "let's list more than the top 3" angle in practice (as in practice, all we'd be doing is listing 4th place but calling it 3rd).

I'm disappointed that this has come up. I've been trying to avoid that particular way of viewing what I'm suggesting here because it cheapens what I'm actually trying to achieve. My goal is simple: I want more people to feel encouraged to participate in the contests, thus (hopefully) increasing Forum interaction and preventing stagnation of the contest format. One way of doing this is to award the top 3 designers, rather than the top 3 levels.

There may be other ways.

Indeed, perhaps it's better to make no change to how places are awarded, but to adopt the suggestion of keeping the entries anonymous until results are announced (assuming a practical and reasonable way to handle updates can be found).

This would certainly help to remove any possible bias during the voting stages and I'm all for it, but I'd personally still want to see 3 designers being awarded first, second and third place rather than 3 levels.

If we keep awarding levels rather than designers, we can end up with a situation where a single designer takes all 3 of the top spots. Even if it rarely (if ever) happens in practice, it's still a serious flaw in the contest format IMO.

The only way to fix it would be to allow one level per designer per contest. But, people don't want that either. So, the only way to keep the end results fair and balanced is to reward the best 3 designers regardless of level placement. We don't need to call it "first", "second" and "third" place if that's bothersome; instead we can award Gold, Silver and Bronze, with a simple rule that the same designer can't be awarded more than one medal. It isn't the same as arbitrarily bumping 4th place up to third, it's literally looking at which 3 designers' levels appear at the top of the voting list, ignoring the number of levels themselves, and then making the awards.

Maybe this is one to at least try and see how it plays out in practice. It's not really asking a lot, and those who participated in the contests might enjoy the recognition they might otherwise not have had.

Mass-voting is okay in the earlier rounds but I really think (given that STV-style voting isn't feasible) elimination is the only right way to handle the closing rounds.

Why? I'm not sure I understand the difference it makes. If anything, it just means that the same people have to keep voting for the same levels. Or, if the level they voted for doesn't make it into the final round, they then only have the choice to either vote for a level they didn't think should win, or not vote.

Ideally, there should be one voting round. The basis of your vote should be "I think this level should win the contest". Otherwise, why vote at all?

I feel less strongly about this than I do about rewarding 3 designers rather than 3 levels, though. The voting system itself is probably a different conversation for a different topic. All I will say here is that multiple rounds are a bit of a bore, and - importantly - sporadic Forum users are likely to miss entire rounds, which is another glaring flaw in the contest format.

[namida]

If we keep awarding levels rather than designers, we can end up with a situation where a single designer takes all 3 of the top spots. Even if it rarely (if ever) happens in practice, it's still a serious flaw in the contest format IMO.

And as long as it's three different levels, I don't really see the issue. Why does a level become less worthy of winning just on the basis that an entirely seperate level from the same author was also really good?

Why? I'm not sure I understand the difference it makes. If anything, it just means that the same people have to keep voting for the same levels. Or, if the level they voted for doesn't make it into the final round, they then only have the choice to either vote for a level they didn't think should win, or not vote.

Ideally, there should be one voting round. The basis of your vote should be "I think this level should win the contest". Otherwise, why vote at all?

I already explained this in reply #11.

[WillLem]

And as long as it's three different levels, I don't really see the issue. Why does a level become less worthy of winning just on the basis that an entirely seperate level from the same author was also really good?

The issue is that we're only looking at levels by 1 designer. If you don't think that's a problem, then we simply disagree and there isn't really anywhere that the conversation can go from there.

I already explained this in reply #11.

I disagree with what you've said in that explanation. People should vote for whichever level they think should win out of all the available levels. Whichever level gets the most votes places top in the list, and so on.

If we allow multiple votes per user, that should solve the problem outlined in your reply. Here's how:

There are 100 users. Level A divides people; 50 love it, 50 hate it. It gets 50 votes. Level B is more universally liked, and gets 70 votes. Assuming that Level B was the second choice of the 50 people that loved Level A, 20 of the people that hated Level A helped nudge Level B into the winning position.

[WillLem]

OK, maybe the contests simply aren't for me. I'm done attempting to provide anything constructive to this debate, or to the contest format.

My suggestions aren't getting any support, and the valid points I've raised regarding the contest's problematic structure and voting system are being completely ignored. For those reasons, this isn't something I want to pursue any further.

[IchoTolot]

I noticed that this was simply locked, but I will make a final statement anyway.

First, I don't see a sign that any statements were ignored here, just discussed rather than automatically changes being implemented based on these.

Also, both methods of either awarding the designer or the level are valid contest methods. Neither of those are problematic in themself, but just have a different focus. Both work and you can make contests with both.

So why did we adapt the level focus?

- The reason I see why people choose to participate in a level design contest is not simply to get a medal/placing in the top 3. It is to create levels with interesting rulesets and play other levels with these rulesets. Winning is secondary. The joy and also excuse to make and play new levels takes the forefront.

- A hint that this holds indeed true is that in the past only 1 or 2 levels could be submittet and people actively wanted to be able to submit for all 3 rules just to A) create more levels and B) play more levels. Again, this shows the interest lies more in the levels not the designers.

As a result, the levels should be awarded in the case for this contest which I also see reflected in the poll.

Those who often win are unlikely to see a problem with the current system

This I highly find unfair to just accuse me of not wanting to change things because I win contests.

A) Over the time (even before you joined the forum) first namida and then myself consistently adapted the contest design to address widespread public desires. Adopting the 3 rule system, reworked the voting to reduce tiebreaker rounds, adding the possibility to update your entries, maling a compilation pack were all such adaptations as an example.

B) I really like making contest levels - so of course I create levels for all 3 rules. Yes, this increases the chance that one of my levels wins, but the core of the reason why I am doing this is not the victory, but because I like to create levels after rulesets - sometimes I even make entries with suboptimal chances to get far because I like a certain gimmik. And with consistently high effort being placed in creating these levels results come around.
I can exclude myself from voting even, but I highly doubt that this would bring any difference in the number of contest entries!

C)

"The most prolific and popular designers tend to win the contests; the data proves that."

I think you are interpreting the data wrong and that is why you see the contest design as problematic. "popular" I highly doubt that this has anything to do with it. People are not just voting like "ah that is Icho's level that one gets my vote" - I still have voting flops!. Again, Armani and myself entered all contests with 3 entries and gathered lots of experience in adapting to rules and make quality levels in terms off appearance, difficulty and creative solutions. Of course this will most likely show results as more time passes and it is quite hard to beat.

I really really try to make it to up to as many people as I can, but over the years I ever so often hit my limitations. I learned to accept that I can't satisfy everyone and that not everyone will approve the decisions that I made. But in the end decisions need to be made.

As a result from this (apart from ensuring a clear level top 3 every time):

I want to make a testrun with the anonymous playing/voting phase. If it goes poorly we can always scrap it again for the next contest or even change it back on the fly.

[Simon]

this isn't something I want to pursue any further.

Please don't lock it nonetheless. There will be more useful discussion.

-- Simon

[WillLem]

Please don't lock it nonetheless. There will be more useful discussion.

Unlocked so discussion can continue, but I have nothing more to add other that what's already been said.

Icho, apologies if the comments felt directed at you personally, that wasn't my intention. I hope that the topic might prove to be helpful at some point.

[Simon]

Thanks!

You count the number of designers who have shown (= placed 1st, 2nd, or 3rd) exactly once in the past n contests, and you value a high count for this. Let's assume Nessy did 50 % worse than he really did (2 showings), i.e., assume he showed only once. This will improve your metric by 33 % or by 66 %, depending on who takes the now-open slot. Is that desirable?

Apologies, but I'm not sure I understand the question. Could you rephrase?

My point was that you didn't pick the best metric (evaluation function) to turn your raw data into a snappy conclusion.

Your metric was: We count the number of different designers show exactly once across the first 3 places of the past ~20 contests. We want to increase this count.

My counterexample was: Nessy showed twice, and he's not so prolific. According to your metric, it would be better if Nessy showed only once instead of twice. That will bump your count (of designers who showed exactly once) from 3 to 4, i.e., we improve the metric by 33 %. This would suggest that we should somehow hamper designers from showing a second time in a later contest.

I thought: A better metric for your point is total number of different designers across the first 3 places of the past ~20 contests.

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!
The best/most popular levels win. The author is second fiddle here.

I'd only give it 95 % to be untrue. :-) The voting system has indeed a slight tendency to favor the same designer within the same contest.

This is because the initial voting buckets aren't random. You make three buckets, one per rule, and vote within each bucket. Later, survivors enter one big bucket for the semifinals. Assume designer X's level from bucket 1 survives. This doesn't affect buckets 2 and 3, but it makes all other levels from bucket 1 less likely to enter the semifinals. Now, because every designer can only enter once per rule, you have a correlation: Each of X's other levels (which cannot be in bucket 1) are individually more likely to enter semifinals than designer Y's levels individually (because Y can have one of his levels in bucket 1).

I don't think this slight tendency is a problem. If you still want to eliminate this slight tendency, don't make a bucket per rule. Make random buckets across all rules. Or change the voting system altogether.

-- Simon

[namida]

I like the bucket rule because it also gives an idea of the best levels in each category. This could be done with (maybe post-contest so it doesn't delay the overall results) an extra poll or two, though, if buckets were mixed.

[Dullstar]

It's already been brought up but I would like to explicitly mention support for the idea of changing the rule design from 1st place makes all 3 rules to each of the top 3 makes 1 rule; if someone is trying to earn that reward, hitting top 3 should be more attainable.

[namida]

To take this concept even further: How important are the mixed rounds, especially given that some rules inherently produce stronger levels than others?

Perhaps a way forward is to get rid of the three rules system, in favor of more frequent contests with a single rule. These in turn could be run in (near-)parallel - essentially, not much different from the current setup, except that each rule is treated as a seperate contest and we just find the winner for each rule, without comparing the rules to each other. This in turn works nicely with "each winner picks one rule", and gets rid of any debate over whether winners should be levels or authors (as we'd be back to one entry per author under such a setup, aside from LOTY which is almost universally agreed should be purely about levels). Voting in turn could be reduced to two or three rounds - all levels for a rule, (maybe) top 3, and finally top 2. Yes, for a set of three rules this would be six or nine polls, but each rule can be run in parallel rather than having to wait for the last one to finish voting, if desired.

And this still allows for creators to get the frequent opportunities with a variety of rules, that was a motivation for the "2 of 3 rules" and eventually "you can enter all 3" changes that were popular.

The one (very minor) concern that comes to mind is, we tend to like having old rules repeated once in a while. But are people with an opportunity to pick just one rule, going to want to sacrifice that on repeating an old one? (Or conversely, will we run into an issue of too many repeats?). Maybe this isn't a problem; if we go this route, I suggest at first just let whatever happens happens and see how it works out, but if we end up feeling that repeats are too rare or too common, one approach could be along the lines of "Rule 1 must always be original; if you win on this, you must come up with an original rule. Rule 2 must always be a repeat, if you win on this, you must pick a past rule to repeat. Rule 3 is free choice and you can go either way."

[WillLem]

+1 for what Dullstar and Namida said

[IchoTolot]

I will probably start polls at the end of next week when I am back from my trip. The first one will be about the 1 entry with 1 rule or multiple entries with multiple rules question.

Also, I will probably post those in a dedicated topic and not directly in the discussion topics.