I agree with you, Christian!

I'm afraid I can't toss any worthwhile suggestions as to how we can improve judging, but I hope the community can find ways to improve the already established methods, and that we can all voice our opinions and suggestions in a civil manner.

And more than 3 judges sounds like a good idea. I'd definitely suggest future contests consider that idea.

Having hosted the Christmas contest for two years now, one of the biggest things I learned for the judging stage was the need to view the work as a whole, and not just as the separate judging categories. It's Gestalt theory: the whole is greater than the sum of the parts. You have to break each entry down on a categorical level for scoring, but afterwards look at that score and see if it accurately reflects the entry as a complete piece.

Often I found that one entry might score higher on categorical basis, but another entry would just be better overall, so I'd have to adjust the scores accordingly.

Just some tips for those who might ever be judges or organizing a contest.

On a side note, I totally agree that theme interpretation should never be that much of a score.

Visit me: Website | Twitter | Facebook | YouTube | Google+

I also support this idea.
It may sound a little self-serving since I could have placed a little better in this BRAWL, but even when I won BRAWL last year with Odoriferous, I felt a little weird about it because there were two entries I topped which I thought might beat me.  And, even if they I were to top them, I was pretty sure that I would do so only slightly and they would follow directly after mine, and yet they were a few entries behind, which all felt weird.

If it can be managed I think more judges would be an excellent idea.  Also, as Flying Minifig mentioned, if possible getting an outsider to be added to the judging panel may be interesting as it could give some rather unbiased opinions.  I feel as it is judging deviates somewhat from the community's general opinion of the films.  But more judges may help to make the general opinion of the judges somewhat closer.
Of course, I doubt we can ever make everyone happy, but we might be able to make the system to some degree slightly better.

I think one of the points Christian was trying to make in his spreadsheet thingy with his fancy math is that the criteria didn't actually matter that much, as regardless of having several different points of criteria, they all tended to be rated relatively similar percentages.
In THAC X the judges simply picked their favourite films, and I think that may in general do better, but especially so with more judges, and possibly even more so with Aq's other idea of cutting off the high and low ratings, of course, I'm not sure how that would turn out, but it sounds like an interesting idea.  A good film is good because it's really good, and choosing it as good from a few categories is a bit weird.

Though, theme is an especially weird category to have lead, as I've always thought of the theme as more of a way to prove that it was made within the time constraints, as well as a sort of almost running gag throughout our films so we could obviously theme-drop.  I've never watched a BRAWL or THAC film and thought I liked it because of the way the theme was interpreted was especially good, but I've always liked it more because it was just a good film.

I've judged several brickfilming contests before and been given some random criteria, sometimes it does seem like a sort of roll of the dice since you're not quite sure exactly how much it should get in some certain category, so I tended to just lean toward more points or less points depending upon how much I liked the film.
I also feel that judging the film as a whole takes somewhat less time than thinking about specific categories.

I'm not complaining about BRAWL. I have been annoyed at the way judging has been treated for a while. This year's BRAWL was excellently run from what I saw. Questions were answered in a timely manner and the judging was prompt. This was general designed to get stuff going for future contest while people still cared about contest stuff. I have a few ideas for improving judging.

First, I think giving a score and ranking to every film is a huge waste of time. My suggestion is do the judging in stages. With a few judges select the pool of films that will be judge. I would not set a hard coded value or percentage because the amount of good entries will vary on the contest. After that go through all the select entries and rank them. Then announce a top 10 or top 5 or whatever is appropriate for the contest. That way being ranked would be more of an horror and it also eliminate a lot of the complain of films being under ranked.

Now as far as ranking systems I have two ideas.

The first in political science terms is called instant runoff voting. Each of the judges ranks the films from first to last. These rankings are tallied together. I think the easiest way to do this is to say a first place vote is one point a second place vote is two and so on. You tally up the point and eliminate the film with the most points. That film gets the last place ranking. From there the ballots are adjust removing the film that was just ranked. If the on the ballot the film was not the lowest ranked film the films that are lower ranked move up. This process is repeated until all the film are ranked and you have a winner. This entire process is easily automated and only takes each judge filling out one ballot. Statistically it leads to the judges being happier with the results.

The second is currently only a theory but it is based on the Elo ranking algorithm. This is used in a lot of things for example chess and most video games with ranking systems. How it works is all films would start out at the same level. For example, 1500. The judges would then compare one film against another and select the one they like better. From there the algorithm gives points to the winner and subtracts them from the looser. The amount of point is determined by the expected outcome of the face off. So for example, if a 1800 and a 1200 film face each other and the 1800 film wins it will not that gain that many points because it was heavily favored. However, if the 1200 film wins it will gain much more points. After a few rounds a clear ranking is developed.

Odysee ~ YouTube ~ Website ~ Twitter

Just throwing this out there:
I have nothing against Nathan Wells, he is an incredible brickfilmer, but I don't see why Blinders won BRAWL this year. When I looked at the spread sheet, I was shocked to see that Blinders had the highest score in theme interpretation. I've watched it over and over, but I can only find tiny references to development. Galan505's film, The Website, was obviously about development, it couldn't have been more blatant, and yet it got a 27 and Blinders got a 35? I feel like the judging was incredibly biased in theme interpretation this year. The score of theme interpretation was directly proportional to presentation. BRAWLING DURING BRAWL, while it was a joke film, got an 11.7 even though it was obviously about developing a cure for all diseases. THIS IS MESSED UP!!

There. Now look what you made me do. You made me rant.

Take it or leave it, I feel like my post is relevant to this thread.

A lot of good ideas are being tossed around in this thread! It's nice to see a discussion on the topic of judging. There will certainly be changes to the judging system next year. I haven't made a decision on what system to use yet (I don't need to for quite some time), so please feel free to keep suggesting judging systems/techniques you know of or thought of.

The theme interpretation isn't rated on how obvious/apparent the theme was in a film, rather its interpretation, how clever/unique it was. BRAWLING DURING BRAWL was not a serious film, it was a joke, intended to get last place, so don't take the rating on that seriously.

This seems to be the most familiar and straight-forward solution. I think it would be good to explore it further for future contests.

This is also an intriguing solution. It might be harder to convince the general public about it, since it seems a little more complex and uncommon (even though, as you point out, it is used in many ranking systems). I think if this method is used, it needs to be explained clearly so people aren't turned off by it's unfamiliarity.

YouTube | Twitter | Brickfilm Archival Project
Brickfilms Wiki | BiM: The Documentary

I feel like that if I had actually put effort into BRAWLING DURING BRAWL it would likely have gotten a much higher score in nearly every respect even if I left the story and theme basically untouched.  I could have lengthened it out and had it play itself more seriously with well-built sets, dramatic lighting, better minifigures, proper voice acting, nice music, and perhaps a few more jokes.  Maybe even a little bit more background showing how the doctor got his ambition, showing his wife and parents dying of disease.
Now, that would all likely be a much better film, however, the theme and story would be basically as they were, however I would guess get a higher score on all accounts, even theme and story.  They wouldn't be much different, yet they would be presented much better and seem much nicer (or maybe just not horrible).  And the film might've placed decently.

But the same would likely go the other way.  Imagine now We the Pumpkins Three, except instead it's called 3 Pumpkins.  The Pumpkins are just blank orange heads on unfitting red shirts, whilst just one of them has the correct head but it's turned the wrong way.  The elf looks kinda weird and it's just a normal yellow head turned round.  The set is just a few things of old grey.  The script doesn't rhyme, it's just me yelling into a mic making an intro and the elf's like, "yo can I have a face?" and the pumpkins would be like "yeah sure girl, we just need you to get us some stuff to develop it."  "K." and it says in comic sans "LATER" and then she comes back, she gets her face, she cries, and the pumpkins explain the moral less than eloquently without properly rhyming and then it ends.
All of it animated and shot just like BRAWLING DURING BRAWL.

Now this would all likely look disgusting, however, it would have exactly the theme interpretation and story as the actual film, though I have little doubt it would easily place last or very nearly there.

As Aquamorph pointed out, the criteria for the judging is a little meaningless.
I sometimes feel a little like every good brickfilm is just a bad brickfilm in disguise, that or every bad brickfilm is just a very good one in disguise.  A bad story can be made to look good, a good story can be made to look bad.  The whitest cleanest LEGO brick turns black in the dark, a black LEGO brick will shine brightly if you position it to reflect the light the right way.

Remember, this thread is about how to improve contest judging for the future, and not about the previous BRAWLs. Let's stay on topic.

YouTube | Twitter | Brickfilm Archival Project
Brickfilms Wiki | BiM: The Documentary