Discussion about Better Competition Judging

(Shaderbytes) #1

So Here is my first brain dump on the matter. This is a open discussion and the goal here is not put any person or entity down , it is to rather find solutions to better practices. The floor is open to all speakers.

Identifying current problem areas :

• Input field to small

This leads to subjectivity that is very high, The more judges the better. Sketchfab hope to get the community involved in this process in the future. It is just not as easy to simply introduce a new system. None the less we want more people and more variety.

• Input field are not knowledgeable in the required aspects of the competition subject

This obviously is also a major problem , two solutions here :

  1. Pick people who have more knowledge in relation to the subject duh, or at least give their vote a higher weight towards the final results.

  1. When that is not possible , there are some steps that can be taken to at least get a good idea of how to judge when this is the case.

  2. Identify the style.

  3. Identify the aging and or environment influence on the subject.

  4. Google the subject and collect a few results from the search.

For example lets take a random subject - a rally car. From the above :

For style - Geometric / Rendering / Texture style - is it :

• Realistic

• Semi Realistic

• cartoon

• abstract/other

Environment/aging :

Was it standing on a showroom after a wash and polish or was it driving in a muddy forest?
Is it new or old?

For newer materials , you dont have wear/aging to concern yourself about so it is easier to judge such a material. If takes less work to achieve so here you need to be very strict on diffuse , roughness , normal etc…

For older materials, based on reference images , identify common wear and tear areas. Note the color , the roughness and the normal mapping of the wearing affect. These are more complex materials and no wear and tear is 100% the same. Again based on reference images you can identify common traits here regardless of them all looking “different”

A good example here was in the last skateboard competition. Wheels on skateboards face a same direction and for the most part spend most of there time accumulating expected wear and dirt patterns , mostly in one direction or area. For example here are some reference images of skateboard wheels ( mostly lighter colored wheels ) notice the surface area that touches the ground, it is way darker than the wheel walls. so this is a common trait.

This is tricky as mentioned , the wheels could be 3 weeks old or 3 years old. But at least with what is mentioned above if a entrant just used some random wear and tear effect/brush on the object , regardless of what level they do it , if it is not consistent with the expected general wear and tear it has to be noted as that.

based on narrowing down the search criteria to something more specific now go and collect some reference images.

(Most users are collecting reference images to create their submissions- the least a judge can do is the same effort… Currently user spend two weeks laboring to make something cool and a judge spends 2 minutes looking at some of the entries and makes a very uneducated selection - this is unacceptable , I feel sorry for all the participants that have been victim to this to date )

• Blind results

This is a big one , no judge should tell another judge what they chose. This WILL always create a seeded mentality , no argument here … any scientific experiment will tell you the same. Judges need to submit their top three choices to a neutral/unseen/inaccessible repository.

• Self subjection awareness

Identify if you have a bias towards a certain art style. Factor this in by listing your bias among other styles and after making your selection you need to do some self evaluation at that point looking at the list and see if your bias is affecting your decision making too much. If it is then do a second set of top 3 where you specifically choose styles outside your bias. At that point you have it narrowed down pretty well but I dont know how to influence you anymore towards a better decision from this point… flip a coin haha. Im just joking if you have any ideas let me know.

An example of this , again based on the rally car…

You may have a bias to clean showroom environments… so you could overlook outdoor muddy action work because of this. Let say it is a texturing comp and the outdoor scene is done way better from a technical point of view … most others will be able to see that but you wont because of your bias.

This is fine , many people have such bias … the point here is to at least do the contest entrants a favor and identify your bias , and take the steps towards making decisions that are based on the competition criteria ( texturing in the example ) and not on your bias.

anyway that is my thoughts for now , hoping to hear more from others.

Here is a growing list of a scoring system , if you feel there can be more let me know, remember that the point is to keep the list condensed , covering the most important aspects of judging. As mentioned earlier there are many small nuances that could be considered but you need to weigh those up against the points i suggested below and think where they fit in , are they more important ? do some points need to be divided into smaller accuracy etc… also not all points will be used in every competition

  1. How well does the scene portray the competition criteria? 0-100…

  2. How much effort do you think was put into creating the scene? 0-100…

  3. How accurate is the environmental aspect of the materials 0-100…

  4. How balanced /accurate is the aging and wearing aspects of the scene? 0-100…

  5. Originally 0-100…

  6. Composition and lighting? 0-100…

  7. Post process balance? 0-100…

  8. Topology 0-100 ( how clean and neat are the edge loops, are there any lighting artifacts due to bad topology etc )

  9. Poly count in regards to real time display 0-100 ( unless there was a poly count restriction in place already )

  10. UV quality? 0-100 ( packing / island shape / seams / tiling )

  11. Texture quality ( not dimension of texture , 4k is obviously higher quality than 2k - rather quality of painting, all channels like diffuse , roughness ec… )

[ENDED] Sketchfab Texturing Challenge: Skateboard
[ENDED] Sketchfab Texturing Challenge: Skateboard
(Ceslav Sukstul) #2

Fully agree with everything in here. People have to understand that to be judge it’s first of all responsibility and it must be done as accurate as possible. Same as we taking car to repair service and expecting professional analysis and work.
Second thing in judging there isn’t just I like this and don’t like. Work must be divided into categories (for yourself).
For example is artist created art by him/herself? If not how much is used from internet and how much is really done by artist? You not judging who downloaded coolest image but work of the artist. Downloaded images must help an artist but not do his/her work.
As mentioned above do homework and gather references. You will easily see if artist used references for his/her work or not. Texture same as model must tell the story. How, where and why object was used.
Now we have very powerful software such as Substance Painter which helps us a lot with procedural texturing. But still without manual touch of artistic hand this kind of texturing is lack of storytelling.
And last is keep an eye on the mistakes. Such as seams, wrong distribution of the dirt/dust/scratches. Braking of the textures on the edges especially where seams should be. It easy to spot when one side is scratched and then other side like cut with knife is less scratched or even color difference. Such things need manual artist work to fix them or in some cases knowledge how to fix automatically.
And of course details balance. If one part of the model looks more used than another it’s already not good. It’s again more to storytelling.

So ending this post I would remind that to be a judge is not just privilege but and responsibility and requires some work.

(Shaderbytes) #3

yeah I think a scoring system card needs to be created for the judging process . then judges have to use the card on all entries they review.

A scoring system can include factors like , original artwork. So for example if you took an image of the internet for something in your design , then you would score low on that one aspect , if you did your own design you would score high etc…

The scoring system needs to cover many other aspects of judging and Although it can become more and more difficult or tedious to consider every nuance I think over time a condensed system can be achieved with the most weighted aspects of scoring being the only ones included.

Each aspect can require a percentage value like 0-100 . Using this system will require more effort but as you mentioned it deserves such effort. Also it is then an established system and a great guideline to any potential judge , regardless of their bias or the type of day they are having, not full proof, but I believe a scoring system or sorts needs to be created.

(Shaderbytes) #4

(EDIT) list moved to first post

(Ceslav Sukstul) #5

Yes something like this. Because without any criteria it’s hard to understand what actually mean texturing skills or modelling skills. Which subject is exactly the main concentration point. Is it whole work as one, or clean work without or with less errors etc.
I think judges must provide explanation why they pick one work and not another. I mean of course to the team organising competition not artists. This way it would show that judges done proper work not just “like don’t like”.
So I think points system would be best in this case. Even maybe not 100 points system but just 10 points would be enough.

(Shaderbytes) #6

yeah the 0-100 is a percentage , so 0-10 would be the same thing. If you think i meant 100 scoring questions of coarse that is too many :wink:

Anyway I also thought some more points could be added , although these would be subject to the competition type for example , if it was a modelling comp then.

1.Topology 0-100 ( how clean and neat are the edge loops, are there any lighting artifacts due to bad topology etc )
2.poly count in regards to real time display 0-100 ( unless there was a poly count restriction in place already )

For the general items i listed before I included “post process balance” as this can be overused and abused to often.

(Shaderbytes) #7

Actually I think a scoring card like this should become public space and any person should be able to complete the card… 3 entries have to be scored at minimum. Sketchfab could create a webpage with a navigation of all the entries with the card questions below.

So then it is easy to complete such a form and submit it. There should also perhaps be some sort of incentive offered to users who do participate in the judging process. like 50 submissions == 3 months pro or something like that

(Ceslav Sukstul) #8

Yes all depends on the challenge subject. Even modelling can have different categories. Like low/mid/high poly, topology, optimisation, UV layout, texture maps quantity and resolution etc.
Same goes to texturing. For example in skateboard challenge some people used just one UV set and I used few just because it was allowed. So rules must be more strict to not allow too much or things will get messy and hard to judge.
And I agree with first post that first of all judges must have skills in subject they will judge. But of course sometimes it isn’t possible, but again this is work for team who organise challenge, to find professional judges not just let it go by itself.

(Shaderbytes) #9

sure , i think once a list of potential scoring options is finalized and available , then per competition they can stipulate which of those options will be used in the judging process.

In regards to judges having knowledge of the subject , i did try and address that issue in the first post , basically saying , if you dont have the knowledge , follow the rule system for best judging … categorize the scene - collect reference images according to that category , then reference those while using the scoring card. Obviously not as good as having a true person with extensive knowledge concerning the subject , but the above can produce better results in light of knowing many times having a knowledgeable person will not be an option

(Shaderbytes) #10

I like your ideas you presented in regards to uv and texture quality etc… maybe compile a list of scoring points for such things ( in the same fashion as I did ) so I can amend that to the full list of possible scoring aspects

(Ceslav Sukstul) #11

To be honest about UV’s it’s pretty simple. How much wasted space is there and how good packing is. It’s almost same because better your UV packing less of wasted space will be. There is more detailed look as well. Such as is UV’s straightened when needed or just left as is. Straightened UV’s can be better packed and leave less wasted space.
So in the end if model has 3 UV sets need to look if it really was necessary or could be done with 1-2 sets. Again if there are any restrictions.

(Shaderbytes) #12

dont try make sense of how things up winning here … it will drive you mad … being a good human does not make you a good judge of humans. Same thing here i believe … being a good artist does not make you a good judge of art. Even with a good judging panel things still become subjective … Ive seen this in vans off the wall comps and all sorts of other competitions etc… so cheer up :wink: better luck next time.

The only good solution is if a massive amount of people are allowed to vote . I mean scientifically this is the best manner to achieve more accurate statistics/results … larger input field… Now this is better but even that is subject to be flawed because popular people can say get more people to vote for them regardless of having a worse entry than others. etc…

(Norgeotloic) #13

Congrats on winning the challenge @spliter !

And don’t worry about your English (which is more than OK!) : the important part is your texturing work, not your English proficiency :wink:

@shaderbytes I 100% agree with you: science and statistics are not always the better solution !

To keep away from the “popular people = more votes” paradigm, I guess that one good hypothetical way to allow everyone to vote would have to make the models “anonymous” by hiding the uploader’s name from the model during the competition, not displaying them in the uploader’s list of models, and allowing people to vote from a “neutral” page, on which the models would be randomly displayed for instance…

But that would create a logistics nightmare, and most importantly, I think that the final models would tend to get judged on a “Wow factor” and “Sexyness level”, more than on the challenge’s rules…

So keep up judging, judges, and thanks for the fun challenge :slight_smile: !

(Shaderbytes) #14

Yes a blind test with a greater input field would be even better , something to keep in mind , but as you mentioned not an easy task for a company to set up, so we are stuck with biased results from a very few subjective unknowing judges.

But going back to the “WOW” factor and this is what @asicson mentioned , when I used to do photography as a hobby many years ago I was part of a photo club , where we also did assignments ( like comps ) and the person who facilitated it immediately on the first meeting gave heed to say dont just take photos of beautiful things - and then mistake a beautifully taken photo for a photo of a beautiful thing. He mentioned he is on the look out for that and will not be swayed by the subjects beauty but will look to all the important aspects of photography when judging the image.

Also when having review meetings of work people had done with their photography assignment , when he displayed such an image to the group , right before he got into any of the technical aspects of the image he would identify the subject as inherently beautiful and would ask the members not to be influenced by this and to start looking past that and look at the technical aspect of photography as he unpacked them. He was a good judge :wink:

No offense @spliter , great job , just addressing the concerns raised and it is a valid point. I’ve been on the masters program for years and this is the first time i put my name down to volunteer for judging, it will also be the last time i do it. Their were two actual skatboarders on this judging team ( i used to skate for years ) I think the other judge still skates ,- none of our choices were selected.

Let hope sketchfab grows in this area so judging becomes better in the future :wink:

(Ceslav Sukstul) #15

Yes it’s not easy task to make things right because you can’t control it and can’t control judges. But I think dipper understanding of subject what you judging is important and knowledge where to look for mistake or just see them. Everyone can see his way if work looks nice or not but more professional people can split work into categories and judge by it. I mean if person is artist he/she knows how to split lets say painting into categories. If person is professional in different field he/she won’t be so objective because and will stay just with I like this or I don’t like.
I don’t want to make things escalate here because it won’t help anyone and would be unprofessional. And judges decision is final anyway. I just told my short opinion without trying “burn anyone on fire”.
And I respect any work of any artist and if I would point on things, which I think are wrong, in first place I would just harm artist not judges opinion and that would be unprofessional and unrespectful.
If for example anyone would like to know why I think how I think I can give short explanation in private but I won’t do that in public for the reason above. The artist won and I respect that.

(Shaderbytes) #16

This thread will be dead and forgotten soon so @asicson or any other user … I started a civil discussion regarding competition judging here :