So Here is my first brain dump on the matter. This is a open discussion and the goal here is not put any person or entity down , it is to rather find solutions to better practices. The floor is open to all speakers.
Identifying current problem areas :
• Input field to small
This leads to subjectivity that is very high, The more judges the better. Sketchfab hope to get the community involved in this process in the future. It is just not as easy to simply introduce a new system. None the less we want more people and more variety.
• Input field are not knowledgeable in the required aspects of the competition subject
This obviously is also a major problem , two solutions here :
- Pick people who have more knowledge in relation to the subject duh, or at least give their vote a higher weight towards the final results.
When that is not possible , there are some steps that can be taken to at least get a good idea of how to judge when this is the case.
Identify the style.
Identify the aging and or environment influence on the subject.
Google the subject and collect a few results from the search.
For example lets take a random subject - a rally car. From the above :
For style - Geometric / Rendering / Texture style - is it :
• Semi Realistic
Was it standing on a showroom after a wash and polish or was it driving in a muddy forest?
Is it new or old?
For newer materials , you dont have wear/aging to concern yourself about so it is easier to judge such a material. If takes less work to achieve so here you need to be very strict on diffuse , roughness , normal etc…
For older materials, based on reference images , identify common wear and tear areas. Note the color , the roughness and the normal mapping of the wearing affect. These are more complex materials and no wear and tear is 100% the same. Again based on reference images you can identify common traits here regardless of them all looking “different”
A good example here was in the last skateboard competition. Wheels on skateboards face a same direction and for the most part spend most of there time accumulating expected wear and dirt patterns , mostly in one direction or area. For example here are some reference images of skateboard wheels ( mostly lighter colored wheels ) notice the surface area that touches the ground, it is way darker than the wheel walls. so this is a common trait.
This is tricky as mentioned , the wheels could be 3 weeks old or 3 years old. But at least with what is mentioned above if a entrant just used some random wear and tear effect/brush on the object , regardless of what level they do it , if it is not consistent with the expected general wear and tear it has to be noted as that.
based on narrowing down the search criteria to something more specific now go and collect some reference images.
(Most users are collecting reference images to create their submissions- the least a judge can do is the same effort… Currently user spend two weeks laboring to make something cool and a judge spends 2 minutes looking at some of the entries and makes a very uneducated selection - this is unacceptable , I feel sorry for all the participants that have been victim to this to date )
• Blind results
This is a big one , no judge should tell another judge what they chose. This WILL always create a seeded mentality , no argument here … any scientific experiment will tell you the same. Judges need to submit their top three choices to a neutral/unseen/inaccessible repository.
• Self subjection awareness
Identify if you have a bias towards a certain art style. Factor this in by listing your bias among other styles and after making your selection you need to do some self evaluation at that point looking at the list and see if your bias is affecting your decision making too much. If it is then do a second set of top 3 where you specifically choose styles outside your bias. At that point you have it narrowed down pretty well but I dont know how to influence you anymore towards a better decision from this point… flip a coin haha. Im just joking if you have any ideas let me know.
An example of this , again based on the rally car…
You may have a bias to clean showroom environments… so you could overlook outdoor muddy action work because of this. Let say it is a texturing comp and the outdoor scene is done way better from a technical point of view … most others will be able to see that but you wont because of your bias.
This is fine , many people have such bias … the point here is to at least do the contest entrants a favor and identify your bias , and take the steps towards making decisions that are based on the competition criteria ( texturing in the example ) and not on your bias.
anyway that is my thoughts for now , hoping to hear more from others.
Here is a growing list of a scoring system , if you feel there can be more let me know, remember that the point is to keep the list condensed , covering the most important aspects of judging. As mentioned earlier there are many small nuances that could be considered but you need to weigh those up against the points i suggested below and think where they fit in , are they more important ? do some points need to be divided into smaller accuracy etc… also not all points will be used in every competition
How well does the scene portray the competition criteria? 0-100…
How much effort do you think was put into creating the scene? 0-100…
How accurate is the environmental aspect of the materials 0-100…
How balanced /accurate is the aging and wearing aspects of the scene? 0-100…
Composition and lighting? 0-100…
Post process balance? 0-100…
Topology 0-100 ( how clean and neat are the edge loops, are there any lighting artifacts due to bad topology etc )
Poly count in regards to real time display 0-100 ( unless there was a poly count restriction in place already )
UV quality? 0-100 ( packing / island shape / seams / tiling )
Texture quality ( not dimension of texture , 4k is obviously higher quality than 2k - rather quality of painting, all channels like diffuse , roughness ec… )