Topic: Some Concerns about THAC & Future Contests

Hello everyone!

As we all know, THAC wrapped up over a month ago and the results were recently posted. Before we get started I want to say Congrats to all the winners and the Top 10 who placed this year.
Now that we got that out of the way, I’d like to take some time to address a few concerns I had with the overall judging of the famous contest. THAC 2018 had over 60 qualifying entries this year, that number keeps growing as more people are becoming more interested in participating. There were only three judges to review all those films.

The Top 10 were announced and displayed in the contest winners’ forum post, however to some peoples’ surprise many of the films that made the top 10 didn’t seem to match what the judges were supposed to be looking. For anyone out there thinking that I’m writing this as a hit-piece, please know that that’s not the intention. I’m just an animator and film critic pointing some things out.

In last week’s Season opener of the Frame100 podcast, the hosts went through the Top 10 entries that got acknowledgment. Starting at the 10th spot was “Cease Fire” by Twickabrick, a very good animator (s). However what ended up winning THAC 2018  was  The Exquisite Corpse by RealBrick, Chris W., and Lucas. Both films were well animated, and interpreted the “Unlikely Alliance” theme in an original way! What’s important to understand here is that while both films were well done, these two films (In my opinion) should have been switched. Here’s why:

In the description for THAC XV, Nathen Wells (The current Host) listed what your brickfilm would be judged on:

“Entries will be judged on criteria including originality, production values (animation, set design, cinematography, sound, etc.), story, and creative interpretation of the theme by the judges.  Films will be discussed and ranked collectively by the judges until a consensus is reached. The top 3 films will eligible for the prize pool. Entries below 10th place will not be publicly ranked.”

That’s a direct quote from the original forum post. If this was, in fact, true, The films in the Top 10 should be ranked differently. I’m not going to get into every single brickfilm and explain why I feel it should be moved to “X” RANK. So I’ll use the first place entry & the 10th place entry to explain.

Starting with “Cease Fire”. The Animation in this film was spot on! It was smooth, no camera bumps or light flicker. The story was unique, and lighting and color correction brought the film to life. It had been argued one of the judges that Cease Fire didn’t have a lot of animation. (Source: Far & Away Future Productions).  The ENTIRE film was an animation. It had subtle movements that really made the minifigures look real. There was nothing to bring you out of that world in any way.  Yes, there was a lack of animation e.g (Walking, running, jumping, panning etc.). However, that’s not an element that should be considered when judging. Overall the film met the Judging standards listed above.

Now let’s talk about the 1st place entry “The Exquisite Corpse”.

This film was another well-animated entry. It had a dark twist that really sucked you into the sick romance of a man and his corpse. Which I personally found really enjoyable, I loved the taboo nature of the story and its characters.  For a film being as long as it was, it was even more impressive! However, let’s bring something to light.
This was animated by more than one person. This was a collaborative endeavor by three people. This, in my opinion, shouldn’t be judged using the same merits as let’s say “Brick city cops”. That was another entry animated by one person. The reason why this raises some concerns for me is simple. You have a film that is being animated by more than one person, which means less work per person (in theory). This means that a lot more ground can be covered in shorter amount time than just having one person do everything. This undermines the concept of THAC.  One of the other major points I wanted to hit was this quote specifically:

“Films will be discussed and ranked collectively by the judges until a consensus is reached.”
I understand that obviously, judges need to agree on the films they are representing. However, something to note: “SillyPenta,” The site Admin said during the frame100 podcast last week “BiM contests intentionally don’t judge on specific criteria”
This, in my opinion, is problematic. If you are a judge of a contest, you should follow the guidelines that you’ve listed. By not following those guidelines, you end up with results like “My gut tells me that I should go for “X”. Or “Oh I don’t like this, just because.”. These are not professional ways of critiquing films. I’m not saying that’s what happened here, but to me something just doesn’t pass the “smell test”.   

There were also only three judges for over 60 entries. This is also a problem, and here’s why:
If one judge particularly likes a certain film, chances are he will want that film placed high. With only two other people to “veto” that film, chances are they won’t. This also comes down to the nature of the relationship between judges, which is a post for another time.  The point I’m trying to make is that if these films aren’t being judged by strict guidelines, It doesn’t make any sense to enter a contest. When I enter a contest, I expect to know what it is exactly I’m being judged on so I can make a film that best fits the requirements.

It’s really discouraging to find out that the hard work in a film will go unnoticed because the entries aren’t being judged on the what was outlined in the contest rules. It makes me feel like there’s no point in putting out my best work if that’s not going be judged on its animation etc.

So, I pass the questions off to you fellow community members: What do you think of the current judging system? Is there any way we could as brickfilmers could help make these rules more clear for future events?
Am I being harsh? Do you feel like my concerns are justified? I would love to get a civil discussion started!

Austin-Frame5 Studios https://i.imgur.com/6kA5imJ.png

Re: Some Concerns about THAC & Future Contests

Not much to disagree with, I think that while THAC is a fantastic contest and I had a lot of fun, I see valid reasons for why the judging may need a little more tweaking. Props to Nathan Swihart though for going back through the THAC films and giving more feedback, though - it's been awesome and he is the real MVP here!

I didn't win, spoiler alert, but I am a little dismayed that my THAC entry will all but disappear into the void of entries. Knowing how close to the mark I was and what to do to improve as a filmer would be fantastic for THAC XVI.

Re: Some Concerns about THAC & Future Contests

I've helped judge a few contests here and there, and even hosted/organized EASTER one year. If there's one thing I've learned, it's that judging film is a very tricky art.

If, for example, one film has small and simple movements smoothed to perfection, and another film has larger, far more complex movements, yet contains a few imperfections, which do you rank higher? How do you compare the hand waving of a calm conversation with the mad dash of a panicked crowd at the approaching monster? Can you assign a completely objective value to that? And if one film has perfect animation, yet the concept is stale and lifeless, should that rank higher than an original and creative movie with blander sets?

Those are many of the questions that plague the minds of brickfilm judges. Hence, discussion.

Now to address the quote from SillyPenta, so far as I understand it.
Here's why most contest judging can't be that specific. Take two of our own events as examples: THAC is very unlike the Animation Challenge, and thus, judging cannot as straightforward. The Challenge looks to one thing: The animation itself. Creativity plays a part as well, but the Challenge can be judged by very specific criteria. THAC, on the other hand, cannot be so easily taken care off. Often, films carry various imperfections, the theme is more nebular and open to interpretation, and often, in my experience, these films become more (or sometimes less) than the sum of their parts. One judge may find something to be more creative than another, or one may see the animation of one as being better than another, and different tastes may begin to color individual rankings. Plus, some flawed films may simply be far more enjoyable than those whose presentation may be of a higher quality. That's why it's tricky.

The description guidelines for judging don't seem, at least here, to contradict that. No hint is given as to which factors overrule others, nor that the clear victor in one category will necessarily be the winner overall. If one person put all of their might into crafting the smoothest animation this world has ever seen, but his story wasn't original, the judges will need to discuss where the film should rank.

Some contests have gotten more specific in the past, however, that has it's own barrel of monkeys to deal with. As it is, the judges do try to collaborate and discuss things. This helps balance the many different factors. Could they do better? Probably. We're all human. I have never agreed 100% with the results in any contest, (even the EASTER that I ran) but that's life. Sometimes, life isn't fair. The judges can try their hardest, but nothing can ever be quite what is expected. Again, too many variables.

There's a difference between strict guidelines, and general guidelines. While THAC and other contests try, they certainly lean more toward the general guideline side of things. Why? Because it's impossible to judge this variety and number of films on incredibly specific criteria, as doing so cannot account for the number of variables involved.

As to collaborations, I'm for them. I think this year's winner is the first that I remember to really showcase that potential in a THAC, but it doesn't seem to be a problem for me. Many of the best animators in the world work in teams, some friends, some family, some across continents, some across the room. Anybody is free to collaborate, or not collaborate, as they please. A collaboration has great potential for good, but also bad. The jerk between animation styles can be a boon, or a curse. This is one year, and in the history of THAC, how many times has a team won? Some, but not all. Have I ever felt a hint of jealousy at others having a brother or friend help them on their film? Oh yeah. But that doesn't mean they should not be allowed to do it.


Now, all that being said, your concerns are certainly justified. Judges are flawed humans, and it does good for them, and us, to think over our process and look for areas of improvement. Sometimes, honestly, judges do a better job than other times. Sometimes, the films they are handed are more easily and obviously judged than at other times. I can't speak for this contest, having only seen a handful of the films, and having no part of the judging, but these are still great talking points that I hope are picked up by those with more involvement and familiarity with this specific contest. With the coming SHAC having four judges instead of THAC's three, maybe that'll help even things out a bit more.

Also, great job stating all this so civilly and carefully! That means a lot, and will help it be considered by those to whom it is addressed.

Re: Some Concerns about THAC & Future Contests

I'm going to be honest here when I say that I can only remember a handful of entries from this contest that did not make the top 10.

There were a few that I felt should have taken the place of others, but that's the case with every contest in the history of all things ever. All I can say about the entries is at least they were all entries I can remember from out of the 62 entries this year, and from a story standpoint, which is how I judge a film most highly, they were probably the strongest.

Now, I'm not going to point any names here, but I have to admit a lot of the entries this year had nonsense plots, if they made any sense at all. I'm not saying that's a bad thing - this is THAC after all, but when it comes to the judging process, it's not going to fair very well if little effort has been put into the story, regardless of how nice it looks.

One thing I can say is that this time, the people who worked hard to tell an intriguing and well told story did succeed this time around and for that reason, I think the judging was fair.

I might as well just add that I loved Exquisite Corpse for this very reason. I love how well it came together as one film. I definitely consider it a success and I have no problem with it being at number one.

Yes, you could say they had an advantage over others, seeing as 3 people did the work of 1 man, but did they really? Each segment was about the average length of the other entries and each member, at least from what I understand, were given little information about what the story would be apart from a basic synopsis. All this, and it still remains one of the strongest entries. More important than any of this though, is that it's a really good brickfilm, with one of the best stories, great animation, and honestly one of the most memorable entries of this year's THAC.

Another important thing to remember is that judges don't always judge brickfilms that stick most faithfully with the theme, but rather the ones that did things a little differently, and interpreted the theme in a way others hadn't thought of. From as far as I can remember, that's how all the contests have played out. In summary, Exquisite Corpse is going to be ranked higher than the one where Timmy and Jimmy hate each other, but find common ground in the end.

I don't mean any ill will to any of the the entries involved, though. Everyone should be proud of themselves for managing to complete a film at all within the 24 hour deadline. Congratulations to all of you. You managed to do something I couldn't this time around mini/bigsmile

Last edited by William Osborne (February 8, 2018 (09:24am))

Powerless - 2nd Place Winner of SHAC 2018!
https://goo.gl/stcePM                                               
YouTube Twitter "Be on your guard; stand firm in the faith; be courageous; be strong." 1 Corinthians 16:13

Re: Some Concerns about THAC & Future Contests

Woof, someone's got their pants on backwards.

Re: Some Concerns about THAC & Future Contests

Could be you.

Rsteenoven wrote:

Woof, someone's got their pants on backwards.

Austin-Frame5 Studios https://i.imgur.com/6kA5imJ.png

Re: Some Concerns about THAC & Future Contests

With any contest, the theme generally serves two purposes:

1. To prevent cheaters by adding yet another stipulation to the contest to make sure entries are made within the time limit/previously completed films aren't submitted as a "new work."

2. Spur creativity by giving all entrants one common ambiguous theme to base their films off of.

Themes, IMO, are more-so meant for those purposes, rather than as strict judging guidelines. And, no matter if you're having the judges freely rank the entries, or if they go off of a weighted number ranking system, it will always be inevitable that you’ll end up with two films competing for the same spot - one a better fit to the overall theme - the other a better overall brickfilm.

This has happened with nearly every brickfilming contest since I've been a member of the community and have paid attention to such things. And honestly, I usually tend to disagree with the final judging as well. But, at the end of the day, it's not whether we made a theme-fitting film, but rather, that we made the best film we could in the time limit.

I remember immediately disagreeing with the final rankings for THAC X back in the day – Especially the 1st place entry. But, after I thought about it more (and a bit of time passed) I eventually came to appreciate those brickfilms more than I initially did – the 1st place entry even becoming one of my favorite all time brickfilms!

Edit after posting:
Just thought I'd add that I wasn't the biggest fan of "The Exquisite Corpse," either. But, again, that could change in time.

I decided to host my own contest (potentially 2… more on that later) partially because of my want to give back to the community, and partially due to my own disagreements with past contest judgments. I’ve set my goals and hope to stick with them, even though I know some may not agree with the final rankings. I’m not saying you’re complaining, but, at the same time, you’ve laid down your opinion, but have no way to act upon it.

If you think there could be more improvements made to brickfilming contests, why not contact Nathan, or even host your own contest! At the end of the day, I don’t care who wins, so long as everybody had fun and all winners receive their prizes.

Last edited by Dyland (February 8, 2018 (04:16pm))

https://i.imgur.com/bvWvkc5.png

Re: Some Concerns about THAC & Future Contests

Disclaimer: I had nothing to do with judging on this year's THAC.

As far as judge quantity, based on past experiences I consider 5 the optimal number to balance out outlier opinions and hopefully end up at something more representative of a kind of objectivity. However, I do think it's better to have three trustworthy judges (past top-4 winners of BiM/Brickboard/Brick-a-Brack contests or bearing some kind of similar qualification) than to take an "any five will do" approach. I suspect that's what happened here (picking three good judges instead of tacking on two more who were just whoever was available), because THAC is historically a difficult contest to find qualified judges for - most people don't want to commit to non-participation, and it is a big time commitment to judge in. But I do think five judges is generally better if it's feasible.

As far as the concern of abiding by given criteria, I haven't listened to Penta on the podcast (Penta didn't run this contest) but I can say that in the early days of BiM contest judging, we took a more categorical approach breaking things down by animation, story etc. and weighting those categories. The feeling was, consistently, that this resulted in a system that essentially forced us to rank the films in a way that was not reflective of what anybody actually thought were the best films. The solution was to move to a more holistic judging approach. And for the most part, I think that has served us well. It doesn't mean you aren't taking those  same qualities of the films into account or aren't judging by them, as you infer, but it allows you to account for the extent to which a film is equal to more or less than the sum of its parts because of how they work together to create the whole. The way different elements of a film's execution play off of each other makes a huge difference in the overall efficacy of the film, and if you are using weighted categories you can't really account for that.

To elaborate, I would not want a scenario like we had on early BiM contests, where for instance animation is always weighed with equal importance. In some films the animation is a huge part of what makes it stand out; in others, it isn't, but something else makes up for that entirely.

If I had to guess, the real source of the disconnect here between what you think the results should have been and what they are comes down to personal taste. You are correct that with three judges, that doesn't get very averaged out. However, any panel is going to ultimately have somewhat different collective taste from any one given person, whether that person is on the panel or not involved in judging.

http://i.imgur.com/wcmcdmf.png

Re: Some Concerns about THAC & Future Contests

As the designer of the current BiM judging system I feel I should respond to this. I posted a similar thread to this one back in 2014. I proposed a new system which is basically the system we use today.  http://www.bricksinmotion.com/forums/po … 3/#p330073 I do realize that it has been a while since this kind of thing has been explained so people are not familiar with how the judging system works. So I will explain it here. The current BiM judging system works like this. Judges are selected from a pool of respected member of BiM or the general film community. For bigger contests like the Summer Contest they like pulling in people from outside the brickfilming world for a fresh perspective. Once the judges are selected and the films submitted the judges will independently watch all the films and rank their top 15 films. The top 15 leaves a decent sized buffer for a top 10. I collect everyone's lists and place it into a spreadsheet. I have a program I wrote that then does instant run off voting from the rankings. IRV leads to rankings that mathematically the judges are most happy with. Films not in a judges rank list are given a ranking of 16. The sum of the rankings are added up for all the films and the one with the highest total is removed and given a rank. So if there were 20 films the film would be given a rank of 20. The the judges rankings get adjusted for the removal of that film. So if one of the judges had that film ranked 11 then their 12th ranked film will now be ranked 11th. Their 13th film will be ranked 12th and so one. This process is done until there is only one film left. This film is ranked first. This is how the IRV rankings are generated. This is not the final rankings however because this ranking list is then sent back to the judges and the talk to each other and agree to move the rankings around however they see fit. The IRV leads to a good starting point and for the most part the judges only move around a few films after this point. Once the judges are happy with the final rankings they are announced to the forum.

Frame5Studios wrote:

THAC 2018 had over 60 qualifying entries this year, that number keeps growing as more people are becoming more interested in participating. There were only three judges to review all those films.

This is fair I would totally prefer to have had more judges. THAC has traditionally been judged by a panel of three to five and most commonly three people. I personally prefer five but we couldn't make that happen so we made three work.

Frame5Studios wrote:

Both films were well animated, and interpreted the “Unlikely Alliance” theme in an original way! What’s important to understand here is that while both films were well done, these two films (In my opinion) should have been switched. Here’s why:

In the description for THAC XV, Nathen Wells (The current Host) listed what your brickfilm would be judged on:

“Entries will be judged on criteria including originality, production values (animation, set design, cinematography, sound, etc.), story, and creative interpretation of the theme by the judges.  Films will be discussed and ranked collectively by the judges until a consensus is reached. The top 3 films will eligible for the prize pool. Entries below 10th place will not be publicly ranked.”

That’s a direct quote from the original forum post. If this was, in fact, true, The films in the Top 10 should be ranked differently. I’m not going to get into every single brickfilm and explain why I feel it should be moved to “X” RANK. So I’ll use the first place entry & the 10th place entry to explain.

Art is subjective and because of that breaking a film into categories and giving scores to those categories is not a good solution. This is how THAC used to be judges and how judges scored all the categories was wildly different. I ran the number from past THACs that used this system and the judges has a standard deviation of 20 points on a 100 point scale. That is crazy. Art isn't as simple as creating a rubric and checking boxes. We have moved to a system where we are looking at the film as a whole and ranked it compared to another film. It's a subjective process and you cannot get around that.

Frame5Studios wrote:

Now let’s talk about the 1st place entry “The Exquisite Corpse”.
....
This was animated by more than one person. This was a collaborative endeavor by three people. This, in my opinion, shouldn’t be judged using the same merits as let’s say “Brick city cops”. That was another entry animated by one person. The reason why this raises some concerns for me is simple. You have a film that is being animated by more than one person, which means less work per person (in theory). This means that a lot more ground can be covered in shorter amount time than just having one person do everything. This undermines the concept of THAC.


Why should we judge a film by a team differently than a film made by one person? Are we going to start keeping track of how many people work on a film and dividing their score by that number? Does asking other people to voice act in your THAC film count as a team? This seems fundamentally unfair and unrealistic. People with more money for better equipment are also going to have an easier time. A faster computer for editing and a faster internet connection is a huge advantage. We aren't trying to judge films based on the resources they had to make the film we are judging the film.

Frame5Studios wrote:

“Films will be discussed and ranked collectively by the judges until a consensus is reached.”
I understand that obviously, judges need to agree on the films they are representing. However, something to note: “SillyPenta,” The site Admin said during the frame100 podcast last week “BiM contests intentionally don’t judge on specific criteria”
This, in my opinion, is problematic. If you are a judge of a contest, you should follow the guidelines that you’ve listed. By not following those guidelines, you end up with results like “My gut tells me that I should go for “X”. Or “Oh I don’t like this, just because.”. These are not professional ways of critiquing films. I’m not saying that’s what happened here, but to me something just doesn’t pass the “smell test”.

 
What's a professional way to judge a film? As I pointed out before splitting up judging a film into categories is not good. You have now multiplied the subjectivity. When it comes down to it the rankings are the films that the judges felt where the best overall films. We look at and discuss the originality, production values (animation, set design, cinematography, sound, etc.), story, and creative interpretation of the theme of a film. These things are not ranked evenly however.

Frame5Studios wrote:

There were also only three judges for over 60 entries. This is also a problem, and here’s why:
If one judge particularly likes a certain film, chances are he will want that film placed high. With only two other people to “veto” that film, chances are they won’t. This also comes down to the nature of the relationship between judges, which is a post for another time.  The point I’m trying to make is that if these films aren’t being judged by strict guidelines, It doesn’t make any sense to enter a contest. When I enter a contest, I expect to know what it is exactly I’m being judged on so I can make a film that best fits the requirements.

 
This is not true. The IRV makes it that outlier opinions are not given more weight than the value of the group as a whole.

Frame5Studios wrote:

It’s really discouraging to find out that the hard work in a film will go unnoticed because the entries aren’t being judged on the what was outlined in the contest rules. It makes me feel like there’s no point in putting out my best work if that’s not going be judged on its animation etc.

 
I'm sorry you feel this way but if you are only making films to get the attention of a handful of judges you might to reevaluate why you are making films in the first place.

Frame5Studios wrote:

So, I pass the questions off to you fellow community members: What do you think of the current judging system? Is there any way we could as brickfilmers could help make these rules more clear for future events?
Am I being harsh? Do you feel like my concerns are justified? I would love to get a civil discussion started!

I'm always open to ways to make the judging better and improve the system. One way is I think I need to document the process in a formal way that can get linked in future contests.

http://i.imgur.com/O9sIt89.png
"WTF is wrong with y'alls hearing?" -CK98 ~ YouTube ~ Website ~ Twitter ~ Google+

Re: Some Concerns about THAC & Future Contests

Thank you for sharing your thoughts on my post! I've read every contributing post and I have some more understanding! I appreciate the time you took in responding!



AquaMorph wrote:

-snip-

Last edited by rioforce (February 9, 2018 (01:23pm))

Austin-Frame5 Studios https://i.imgur.com/6kA5imJ.png

Re: Some Concerns about THAC & Future Contests

I think the general consensus is that we just need the judges to be highly intelligent robots with the ultimate wisdom who can judge all of the brickfilms to everyone's satisfaction. mini/tongue

Joking aside though, I think personally feel that Exquisite Corpse was one of the best entries and I felt it would be either that one or Pongowl's superb entry which would win, so I'm not fussed about the winner. I also don't have a problem with anybody doing collaborations during THAC because of the reasons that have been previously stated. Yes, some may find it slightly odd/annoying that the last two  THAC winners have been collaborations, but I personally don't have a problem with that. I think that the current judging system is fine, although maybe at some point in the near future with the BiM contests becoming more and more popular there should be a top 20. I also would like to commend nswihart on giving his feedback to many of the entries, and I definitely think that in the future everyone should be given feedback on their entry, as it really does help people to know where to improve and gives them a better chance for at winning in the next contest they enter.