Poll

How shall the voting in Level Design Contests be handled?

Base the distribution of levels on contest rules (as we did before)
6 (66.7%)
Pseudo-randomly distribute all levels evenly onto a reasonable number of groups (LOTY like)
3 (33.3%)

Total Members Voted: 9

Voting closed: March 26, 2019, 04:31:44 pm

Author Topic: Polls for possible changes to the contests  (Read 541 times)

0 Members and 1 Guest are viewing this topic.

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Polls for possible changes to the contests
« on: March 20, 2019, 02:32:01 pm »
As stated in the recent Contest Topic I want to do 2 polls regarding the rules of the contests:

Poll 1.) Change the voting round duration from 3 days normally and 4 days for finals to 2 days normally and 3 days for finals.

Poll 2.) Base the distribution of levels not on contest rules anymore and instead just pseudo-randomly distribute all levels evenly onto a reasonable number of groups.

I will leave each poll open for 3 days and make a decision based on the results. :)

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #1 on: March 20, 2019, 02:33:26 pm »
The first poll is up! :)

Offline Simon

  • Administrator
  • Posts: 2657
    • View Profile
    • Lix
Re: Polls for possible changes to the contests
« Reply #2 on: March 22, 2019, 06:39:57 pm »
Rename the contest from
Lemmings Forums Level Contest
to
Level Design Contest.

Reason: Other contests are on Lemmings Forums, too, are about levels, too, and are contests, too. Example: The contest where we minimize skills on a given level. Thus Lemmings Forums Level Contest is not precise.

I don't mind the author/rules/versioning in the filename, but please include the level name. The level name is the handle, not the author or the rule. When I have a loose replay called Nepster_r1v3, from what contest is that?

-- Simon

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #3 on: March 22, 2019, 06:52:21 pm »
Quote
Rename the contest from
Lemmings Forums Level Contest
to
Level Design Contest.

Reason: Other contests are on Lemmings Forums, too, are about levels, too, and are contests, too. Example: The contest where we minimize skills on a given level. Thus Lemmings Forums Level Contest is not precise.

Good idea! :)

I agree that Level Design Contest is more precise.

Quote
I don't mind the author/rules/versioning in the filename, but please include the level name. The level name is the handle, not the author or the rule. When I have a loose replay called Nepster_r1v3, from what contest is that?

How about: Author_Levelname_RuleVersion?

Offline Simon

  • Administrator
  • Posts: 2657
    • View Profile
    • Lix
Re: Polls for possible changes to the contests
« Reply #4 on: March 22, 2019, 07:14:47 pm »
Quote
How about: Author_Levelname_RuleVersion?

Good!

-- Simon


Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #5 on: March 23, 2019, 04:29:39 pm »
The poll resulted in the following:

3 days for regular votes and 4 days for finals:
    4 (40%)
2 days for regular votes and 3 days for finals:
    6 (60%)

So I will shorten the voting period to 2 days for regular votes and 3 days for finals.

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #6 on: March 23, 2019, 04:32:19 pm »
The next poll regarding the voting system is up! :)

Offline Proxima

  • Posts: 3344
    • View Profile
Re: Polls for possible changes to the contests
« Reply #7 on: March 23, 2019, 05:50:46 pm »
But for a big change I want 2 things:

1.) A majority for an overhaul determined by a poll.

2.) After that: A concrete plan to what the new system should be. As I don't know how which desired attributes a new system should have and what exact flaws shall be corrected - here the persons wanting a change shall help me out. I want concrete ideas!

Okay, sorry I haven't gotten round to this up to now, I've been very busy with my course, and getting up early every day has meant that whenever I've not been busy, I've been too tired to really do anything :P

I'll start with some thoughts, and then move on to concrete proposals.

The biggest problem with the current system is far too many voting rounds. The most recent contest vote required eleven rounds to pick out a winner from twelve levels. That requires a lot of unnecessary effort from both the organiser and voters; and it means voters are not participating equally, unless they visit the forum every few days and have enough time to look at which levels are up for voting and make a vote. That makes the results far less meaningful, since the winner is heavily dependent on the luck of who was able to participate. And this will be more of a problem with the current proposal to shorten the time for each round.

So, how can we reduce the number of rounds without moving to a completely different voting system?

Firstly, no tiebreakers before the semi-final round. For example, in the last contest, the Rule 2 vote was intended to pick three of five levels. Two clearly made it through; another two tied for the third slot. What's the harm in letting both of those two go through? They had less support than the two clear winners, so they're not going to beat them in the final either.

Secondly, reduce the survival rate. In the last contest, the survival rates from each rule ranged from 50% to 66%. Do we really expect that a level that placed third of five has any chance in the final? And if it somehow did win the final, would that be a fair result, when the first poll shows it was liked less than two other levels? I propose that survival rates should be roughly equal to current attrition rates, i.e. rules with 2, 3, 4, 5, 6 levels should go down to 1, 1, 2, 2, 2 survivors.

Thirdly, have at most three rounds in the mixed group. My second proposal would reduce the mixed group to at most 6 levels, so it shouldn't need more than three rounds in any case. (I'm tempted to propose reducing it to two, but it's nice to know the third-place winner and to have a dramatic showdown between only two finalists.)

I think that with these changes (instead of reducing the length of each round, not in addition), the voting process would be a lot smoother and more enjoyable, and voting in each round would feel a lot more meaningful.

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #8 on: March 23, 2019, 06:43:47 pm »
I considered your proposals and have made up another possible plan based on that. In the end I 100% cannot satisfy everyone though. As I think even the first poll showed that the forum is somewhat split about the process in general and in the end I will have to say: THIS is how we do it. Otherwise it is an endless discussion.

Ok, but now to the plan I have:

- We pseudo-randomly distribute all levels evenly onto a reasonable number of groups (LOTY like). This will ensure the group size or the rule doesn't matter anymore as this was brought up as a critique point.

- We bring down the groups to 1/2 levels in one round each. If there is a minor tie (like 2 or 3 levels) all the levels advance. Bigger ties will still need a tiebreaker (like 4 level tie for the 2nd place or so), as this will result in a very over sized mixed group. This should be rare case in general though.

- No tiebreakers before the determination of the final 3 levels - only if there is a major tie which will result in 5+ levels advancing from a group there will be an emergency tiebreaker. So only when we are voting for the final 3 and 2 levels tiebreakers can happen normally.

- The mixed group will ensure the last 3 levels with tiebreakers (to determine 1st, 2nd and 3rd place) and there will be at maximum just 1 round between the start of the mixed group and the semi-final with 3 remaining levels.

- The voting duration will stay at 3/4 days.


This way I will cover the following critiques:

- The group size and the rule doesn't matter anymore

- Fewer voting rounds

- Reducing of tiebreakers

- Reducing of the survival rate

- Only 3 major mixed group votes

- Enough time to vote

Please share your opinion on this plan! :)

Offline Simon

  • Administrator
  • Posts: 2657
    • View Profile
    • Lix
Re: Polls for possible changes to the contests
« Reply #9 on: March 24, 2019, 12:14:16 pm »
Hats off to Icho for being open (to change election system) after the poll (about duration of voting in existing election system that has many votings) even if it brazenly overrides the poll result. This is sensible: The main problem is that the elections take too long overall, and there has been incentive to vote (for shortest poll time) merely to accelerate the overall process.

Fewer rounds with long voting times is good. Reducing survival rate is good.

Tiebreakers should be rare. If survival rate is low, you can let tying levels pass instead of tiebreak round, in a pinch. That's good.

If the winning level has to beat levels of other design-rules, then it's unnecessary to group by design-rule in the first place. Yes, make random groups across all entries; in a pinch, prefer fewer groups with more levels each. Then vote within each bucket, then merge the survivors from all groups.

If a contest has < 10 entries, you don't even need groups; everything can go into a single poll. The only reason to make groups seems to be cognitive ease? It's easier to pick a favorite 3 from 8 than it is to to pick a favorite 5 from 23.

-- Simon

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #10 on: March 24, 2019, 01:08:12 pm »
Quote
Hats off to Icho for being open (to change election system) after the poll (about duration of voting in existing election system that has many votings) even if it brazenly overrides the poll result. This is sensible: The main problem is that the elections take too long overall, and there has been incentive to vote (for shortest poll time) merely to accelerate the overall process.

I initially thought the 2 main problems were a too long voting period and the group size inequality and the 2 polls here were the easiest way to address the problem.

I did not consider the solutions and points from Proxima before his post and just thought that this would be indeed better, having a still longer time to vote, but fewer rounds. I am more in favor of this proposal now.

Both polls, the 1st one and the current one, are very close and not one-sided. So I feel not very bad for deviating from the result.

Also I still want to ensure that everybody has indeed a chance to vote.

Quote
If a contest has < 10 entries, you don't even need groups; everything can go into a single poll. The only reason to make groups seems to be cognitive ease? It's easier to pick a favorite 3 from 8 than it is to to pick a favorite 5 from 23.

I would not draw the line at 10 though. I still think a group of 8 is a bit too big. My line would rather be at roughly 6. For 10 I would make 2 groups of 5. I am still in favor of smaller groups.



As a result from this I am more and more in favor for changing the system up to the plan I proposed earlier:

Quote
- We pseudo-randomly distribute all levels evenly onto a reasonable number of groups (LOTY like). This will ensure the group size or the rule doesn't matter anymore as this was brought up as a critique point.

- We bring down the groups to 1/2 levels in one round each. If there is a minor tie (like 2 or 3 levels) all the levels advance. Bigger ties will still need a tiebreaker (like 4 level tie for the 2nd place or so), as this will result in a very over sized mixed group. This should be rare case in general though.

- No tiebreakers before the determination of the final 3 levels - only if there is a major tie which will result in 5+ levels advancing from a group there will be an emergency tiebreaker. So only when we are voting for the final 3 and 2 levels tiebreakers can happen normally.

- The mixed group will ensure the last 3 levels with tiebreakers (to determine 1st, 2nd and 3rd place) and there will be at maximum just 1 round between the start of the mixed group and the semi-final with 3 remaining levels.

- The voting duration will stay at 3/4 days.

I already got 2 voices from the discord chat that they have no objections against this and if no flood of angry posts arrives saying that this is terrible, I think I will try this out. We can still make adjustments if needed. :)

Offline Proxima

  • Posts: 3344
    • View Profile
Re: Polls for possible changes to the contests
« Reply #11 on: March 24, 2019, 01:28:27 pm »
The new plan sounds very good to me. There's only one thing I don't like about it -- I would rather keep the first round being grouped by rule so that we establish the winner of each rule as well as the overall winner of the contest -- especially for rules like "Make a Tame level" where all entries have no chance of winning overall. But if people want to get rid of rule-grouping, I'm okay with that, and it's still a big improvement on the current system.

Offline Simon

  • Administrator
  • Posts: 2657
    • View Profile
    • Lix
Re: Polls for possible changes to the contests
« Reply #12 on: March 24, 2019, 05:31:45 pm »
Whatever cuts number of rounds is good. We can try those ideas.

For the record/future discussion, I believe we should prune much, much faster per round than (10 go into two groups of 5 with 2 or 3 surviving). I would run 10 entries like the French elect their president, with exactly two rounds: First round has all entries, best two advance to second round.

I admit that it's taste, some want a spectacle, and some want sub-winners among the levels for the same design rule, it's all good. Also the organizer should, to an extent, just do what makes themselves feel best.

-- Simon

Online namida

  • Administrator
  • Posts: 8813
    • View Profile
    • NeoLemmix Website
Re: Polls for possible changes to the contests
« Reply #13 on: March 25, 2019, 08:06:18 pm »
The new plan sounds very good to me. There's only one thing I don't like about it -- I would rather keep the first round being grouped by rule so that we establish the winner of each rule as well as the overall winner of the contest -- especially for rules like "Make a Tame level" where all entries have no chance of winning overall. But if people want to get rid of rule-grouping, I'm okay with that, and it's still a big improvement on the current system.

I agree with this. Some rules are likely to lead to more successful (I wouldn't necesserially say "better") levels than others, so I think it's worth running them separately.
My released level packs:
Lemmings Plus Series | Doomsday Lemmings

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #14 on: March 25, 2019, 08:36:00 pm »
The new plan sounds very good to me. There's only one thing I don't like about it -- I would rather keep the first round being grouped by rule so that we establish the winner of each rule as well as the overall winner of the contest -- especially for rules like "Make a Tame level" where all entries have no chance of winning overall. But if people want to get rid of rule-grouping, I'm okay with that, and it's still a big improvement on the current system.

I agree with this. Some rules are likely to lead to more successful (I wouldn't necesserially say "better") levels than others, so I think it's worth running them separately.

I think this is the main point to discuss still.

If some rules lead to better levels, then those from "worse" rules will drop out anyway during the contest. They will get into the mixed group sooner or later. I still think the equal group size with equal surviving rates is a higher goal to achive than delaying the inevitable mixing. ???

Offline Simon

  • Administrator
  • Posts: 2657
    • View Profile
    • Lix
Re: Polls for possible changes to the contests
« Reply #15 on: March 26, 2019, 06:39:04 pm »
For results among subsets of entries (all levels from a given design-rule), seriously consider ditching polls, in favor of (a) have each voter submit an ordered lists of all entries, or (b) have each voter distribute 20 points across all entries, with at most 10 points per entry and voter.

It will be radically different, but it will speed up the process enormously, and you can look at any subset you like.

You can still hold a French presidential election between 2-3 winners of the list/score round.

-- Simon

Online namida

  • Administrator
  • Posts: 8813
    • View Profile
    • NeoLemmix Website
Re: Polls for possible changes to the contests
« Reply #16 on: March 26, 2019, 07:13:04 pm »
The new plan sounds very good to me. There's only one thing I don't like about it -- I would rather keep the first round being grouped by rule so that we establish the winner of each rule as well as the overall winner of the contest -- especially for rules like "Make a Tame level" where all entries have no chance of winning overall. But if people want to get rid of rule-grouping, I'm okay with that, and it's still a big improvement on the current system.

I agree with this. Some rules are likely to lead to more successful (I wouldn't necesserially say "better") levels than others, so I think it's worth running them separately.

I think this is the main point to discuss still.

If some rules lead to better levels, then those from "worse" rules will drop out anyway during the contest. They will get into the mixed group sooner or later. I still think the equal group size with equal surviving rates is a higher goal to achive than delaying the inevitable mixing. ???

Yes - but at least the group-specific rounds act as kind of a "mini-contest" to play the levels from those rules against each other, which is a more equal playing field.

Perhaps a hybrid approach works here, to try and allow for this to some extent while not holding the contest up too much - if rule A has roughly the same number of levels as rules B and C combined, combine B and C into a single round of voting while keeping A separate. If all three are roughly equal, either do them in a single combined round or a round each, depending on which fits better with the total number of submissions.
My released level packs:
Lemmings Plus Series | Doomsday Lemmings

Offline IchoTolot

  • Moderator
  • Posts: 1970
    • View Profile
Re: Polls for possible changes to the contests
« Reply #17 on: March 26, 2019, 08:22:02 pm »
Given the vote ended 6 to 3, I think I keep the rule spilt up for now.

While trying to make the survival rates roughly equal, while them being lower as before so we will always end up with ~6 levels in the mixed stage.

The other speed up processes I will implement.


Something truly radical, like the list method Simon proposed, I may try out in a spin-off contest. ;)

I had some ideas for different voting styles and a ranking list was one of them. Another one was a "monarchy"/"wise senat" way in which 1 or ~3 people simply decide the entire ranking. Which could be an interesting twist, trying to appeal exactly that one person or the small group.

But as I said that would be a spin-off contest first where it's ok to goof around and experiment completely. :)

Online namida

  • Administrator
  • Posts: 8813
    • View Profile
    • NeoLemmix Website
Re: Polls for possible changes to the contests
« Reply #18 on: March 27, 2019, 08:34:41 am »
I can't say I'd be too keen on the single judge method; but the list method might be worth at least trying.

Trying to design around general community trends is one thing; but I think trying to design to a specific person's standards - even if that person is different each contest - might harm the quality of levels that get made. It can also be somewhat hard to predict in some cases, given that the kinds of levels a person enjoys making might not be the same as the kind they enjoy playing.
My released level packs:
Lemmings Plus Series | Doomsday Lemmings