How does Sketchfab determine when to autoplay audio?


(Andgokevin) #1

Hi!

I’m developing Supermedium, a fully VR browser (http://store.steampowered.com/app/803010/Supermedium/), that prominently features Sketchfab content.

For some sites such as Apollo 11 and Tomb of Nefertari, the sound only autoplays the first time or on reload, but not on navigation through Supermedium (which is a bit unique). If I could understand how Sketchfab determines when to autoplay audio, I could perhaps dig into the issue.

Thanks!


(Paul Sketch) #2

Two things:

  • Tab visibility: tab must be visible to play

  • “User Gesture” to get user authorisation on page to play audio. As there is no W3C api to know that sadly, It tries to play audio, and if it doesn’t you should get a popup asking for user gesture. If code below return “false”, we can play.

         var a = document.createElement('audio');
      try {
          a.pause();
          a.play();
          return a.paused;
      } catch (error) {
          return true;
      }

(Andgokevin) #3
  • Tab visibility: what if the tab was made active after the site was already loaded? The audio should start playing. At the moment, I don’t think it does.

  • User gesture: that’s actually not required for desktop browsers (which Supermedium is). Sites can play audio when they want on desktop browsers. Which is why it sometimes works on Supermedium even if the user never touches the mouse or keyboard.

Can you check if there is a handler for visibilitychange to start playing audio? Thanks!


(Paul Sketch) #4

Not sure to understand…
You can check “tab switching” between sketchfab model with sound to see it in action.
(all comes from chrome limitations of 6 audio context maximum per browser, and the resume/suspend audiocontext not bypassing that limits, so we have to deallocate/allocate full audio on tab switch, which means on/off audio)

check chrome 66 autoplay policy changes.


(Andgokevin) #5

check chrome 66 autoplay policy changes.

Not all browsers are Chrome though. So the use case is we’ve built Supermedium, a fully VR browser, where it makes sense to autoplay audio without gesture because the user is in a headset, not on their mouse and keyboard. Actually have many users using and loving Sketchfab VR’s mode in headset (versus just the 2D mode).

At the moment, we’ve been working around that Sketchfab doesn’t automatically enter VR on vrdisplayactivate event like spec says, and it’s currently requiring a user gesture to autoplay audio, even though it isn’t strictly necessary on all browsers.

Wondering if there’s a way to request or force autoplay the audio without doing a workaround?


(Paul Sketch) #6

Sorry I wasn’t clear enough, it’s browser limitations, not on our side, so it require gesture ONLY if the browser require gesture.

We do try to play and IF we’re not allowed we have to popup a message asking user a gesture.

Please read the code poster above, that’s exaclty the code we use to check if we can play or not.
On your browser I would Test just with that code and check if you get correct answer. if it works, I would then check with tabvisibility events and see If I get correct events on the frames that have sounds.

Slighty OT:
We have to do that code, as it’s feature detection instead of “browser sniffing” because of the huge variety of cause and lack of spec on that exact subject.
(chrome autoplay, firefox mobile special settings allowing autoplay or not, etc.)
So we have to do it on desktop too.

We’ve been asking Webaudio, chrome, etc for years a way to “detect if we’re allowed” without getting any clear and dedicated API endpoint to do that.


(Paul Sketch) #7

From specs webvr 1.1:
“vrdisplayactivate: A user agent MAY dispatch this event type to indicate that something has occured which suggests the VRDisplay should be presented to” ?


(Andgokevin) #8

Check with tabvisibility events and see If I get correct events on the frames that have sounds

Alright. The browser can definitely autoplay audio without gesture (based off of Gecko), it works sometimes on Sketchfab. Can you clarify what I should look into with regards to tab visibility? You mentioned tab switching and visibility events. When does the audio code run? What happens if the audio code runs when the tab is not visible? If the tab becomes visible, will the audio code reattempt?

From specs webvr 1.1:

Yeah, sorry for the off-topicness on that. But interesting to talk about. It’s may, but more in WebVR, it increasingly should be should. The spec was written, though not many people were really consuming WebVR, not much content was out there, and not much was connected. It’s evolving a bit more now.

vrdisplayactivate is what powers link traversal (Firefox, Oculus, Edge, Supermedium), and is what Supermedium uses when navigating to a Sketchfab model. Without it, any VR site going to Sketchfab will essentially hit a dead end as Sketchfab doesn’t auto-enter VR. So by adding it, Sketchfab becomes connected to the VR Web. Many tools/sites follow this pattern (e.g., all three.js/A-Frame/Within content).

Thanks!


(Andgokevin) #9

Shortened to three questions:

  • When does the audio code run?
  • What happens if the audio code runs when the tab is not visible?
  • If the tab becomes visible, will the audio code reattempt?

(Paul Sketch) #10
  • tab is visible and Browser can play sound (test code above)

yes.


(Rémy Bouquet) #11

We do use it from our vr browser btw, but under certain conditions only with url params. Last time I checked there was no proper way to know if the calling context was already presenting VR or not. Without this we have no way of knowing if we have to call presentation. If this changed, please let me know.

Sketchfab is mainly for plain 3D, VR traffic is actually quite marginal, we won’t start VR presentation right away when displaying a model.


(Andgokevin) #12

Thanks for the response. I’ll release a workaround for our browser for the audio. We entered some weird cases where the audio never plays.

Without this we have no way of knowing if we have to call presentation. If this changed, please let me know.

If you get the vrdisplayactivate event, you should start presentation. The browser will fire that event if there was a previous presentation. If not, then it doesn’t do anything. Other browsers such as Firefox / Oculus Browser / Samsung Internet support this, and some frameworks such as three.js / A-Frame have this logic built in.

VR traffic is actually quite marginal.

It’s a pain to use WebVR in 2D browsers today such that I agree no one bothers to use in VR. We launched Supermedium on Steam / Oculus to make it seamless and quick to visit many VR sites, including Sketchfab, without leaving headset or typing URLs. We’ve been driving VR usage.