Refraction in AR (and reflection)

hi,

  1. can a transparent refraction materials see-through and show the real world in AR?

this model, when seen in AR the material doesn’t show the world through: Refraction Demo: S**t happens - Download Free 3D model by Sketchfab (@Sketchfab) [3b07b79] - Sketchfab

  1. how to use the real world as environment for reflections for AR models?

this model, when seen in AR does not show the real world as reflections:

thanks

Hi,

both features 1 & 2 are not supported for now

Thanks for the prompt reply,

please update if implemented.

@paul_sketch - re: refraction in AR - are there any updates or best practices that you can share?

i cannot for the life of me get any refractive materials from any software to render correctly in AR. i’ve tried blender, keyshot, dimension, vectary, sayduck, model-viewer, three.js, sketchfab, written my own gltf by hand - i’m really stuck.

Mike Bond at Adobe, who co-authored the new transmission extension for GLTF, hinted that SketchFab had successfully implemented refraction: https://github.com/mrdoob/three.js/issues/16977#issuecomment-749793318

transmission/refraction in WebGL & AR appears to be “cutting edge” - at least in the sense that GLTF only announced their extension a few weeks ago. none of the software listed above actually supports the new GLTF extensions - babylon.js is the only viewer (other than Khronos’s own) that does. is Sketchfab ahead of the curve here? i think this is an incredibly valuable problem to solve - so many real-world products have refractive materials.

unfortunately, this quick test suggests that SketchFab is not supporting the three new GLTF 2.0 PBR extensions correctly (definition not KHR_materials_transmission or the windshield would be see through)

the .usdz export is completely off:

the “pony cartoon” (i cant have more than 2 links in this post because i am new here) appears to work (i can see through the windshield)…although maybe this is just alpha transparency and not refraction?

1 Like

Hi,

  • Implement importing parameters for the new GTLF extensions is on the near roadmap
  • refraction/refraction in sketchfab viewer is working but not in AR.

“GLTF transmission support” in AR (Reflection/Refraction in AR) is a very complex topic It means it need access and process:

  • back camera image (refractions)
  • front camera image (reflections/refractions)

And it needs to process lighting informations from those (used for pbr computation) each frame, as PBR shading that is used in the “Transmission” specifications needs all those informations,

2 Likes