The HTML3D component in Amazon Sumerian enables you to render HTML into a 3D scene. For many use cases, it can massively accelerate building your experience. HTML3D is useful for placing UI into the environment, for example, creating a TV that plays video from Amazon Prime Video, or tagging objects in an augmented reality (AR) experience.
However, some Amazon Sumerian features, such as Post Effects and virtual reality (VR), don’t mix well with HTML3D. This article explains what’s happening behind the scenes to help you understand when to use and when to avoid using HTML3D.
How Does It Work?
The HTML3D content isn’t part of the image rendered by WebGL. In the 3D scene, we create a proxy object that represents the HTML3D element. The proxy object, a simple plane with special shaders, renders as transparent black pixels plus a depth value. This cuts a transparent hole into the rendered image, allowing other content on the containing page to be visible.
At the same time, the HTML defined in the HTML3D component is added to the containing page. CSS 3D transforms move, scale, and rotate the HTML to precisely fill the hole in the rendered image. The browser composites the page, layering the rendered image on top of the HTML content. This creates the illusion that the HTML is in 3D.
Post Process Effects
Post process effects (Post Effects) are special image effects applied to the rendered image immediately after the render is finished, but before the image is presented to the browser. These effects aren’t applied to the HTML content.
Post Effects ignore the alpha channel of the rendered image. They read the RGB values of the rendered image and rewrite the new RGB colors with an opaque alpha value of 1.0. Here’s what you would see if you performed a blur operation on the rendered image.
The HTML3D content is still being transformed and positioned, but there is no longer a transparent hole in the rendered image.
When entering VR mode, only the WebGL canvas is sent to the head-mounted display (HMD). Just like Post Effects, the rendered image is treated as RGB values only. In VR, you’ll see a black plane where your HTML3D content appeared in the 2D browser view.
When entering VR, we capture a screenshot of the HTML by rendering it into a canvas as SVG, then converting it to an image. The screenshot is applied to the plane as a texture. Instead of a black plane, you’ll see an image of the HTML in 3D.
Whenever the HTML is changed or resized or has a new style applied, the screenshot is updated.
Rendering the HTML to an image and converting it to a texture on a plane is a slow operation that can take 200 milliseconds or longer. This delay is unacceptable for most VR use cases, where a hitch in frame rate can cause users to experience nausea.
The HTML3D component in Sumerian can be an effective way to add UI and text labels to your 3D scene. However, be aware that using Post Effects conflicts with HTML3D. Rendering static or infrequently updated content in VR is possible, but isn’t recommended for dynamic, rapidly updated HTML content. The type of HTML content you use will be limited by browser security and performance bottlenecks.
The HTML3D component provides a lot of capabilities for Sumerian users, but it’s good to know its limitations as you start building your scene.