Intermediate Tutorial


Capturing Picture-In-Picture Screenshots With Amazon Sumerian


15 minutes

Posted on: June 17, 2019

Tags:

screenshot new
Learn Sumerian
Capturing Picture-In-Picture Screenshots With Amazon Sumerian

Tags

screenshot
new

In this tutorial you will learn about:


In this tutorial, you will learn how to capture a screenshot of your Amazon Sumerian scene and create a picture-in-picture display. We’ll make a simple scene that uses custom scripts to capture the image rendered to the canvas with the HTMLCanvasElement.toBlob() API. Note: This exercise is best for desktop devices and is not supported by the Safari browser on iOS.

You’ll learn about:

  • Capturing screenshots with toBlob()
  • Executing scripts after the render is complete with RenderSystem.onPostRender()
  • Sending the results of toBlob() to an HTML <img> element

Prerequisites

Step 1: Set Up the Scene

In this step, we create a scene with a single cube, an empty entity named _SCRIPTS, and an HTML entity.

  1. Open the Create entity menu to add a Box entity. Note: This entity will only serve as an object to view within the canvas. Therefore you can select any entity or object you prefer.

    Create a Box

  2. Use the Create entity menu again to add an empty Entity. Rename it _SCRIPTS. This empty entity will act as a container holding the scripts that capture user input, generates a screen shot, and saves it as an image.

    Create Empty Entity

  3. Use the Create entity menu to add an <> (HTML) entity. Rename it “ScreenShotDisplay”. This HTML entity will be the window that displays the last captured screenshot.

    Create HTML Element

Step 2: Modify ScreenShotDisplay to Create a Picture-in-Picture Window

In this step, we update the HTML element to create a picture-in-picture window over the canvas.

First, we adjust the HTML component settings to ignore the camera Transform. Then we adjust the CSS and HTML to create a wrapper <div> element that contains an instructions <div> and a picture <div>. The instructions container will have a bit of text, and the picture container will contain an <img> element that we’ll use to display our screenshot.

  1. Select the ScreenShotDisplay entity, and then under the HTML component, uncheck Move with Transform.

    Uncheck Move with Transform

  2. Select Open in Editor to open the Sumerian Text Editor to edit the CSS and HTML for this entity.

    Uncheck Move with Transform

  3. Replace the default HTML text with the following. This code defines the location, size, and contents of the ScreenShotDisplay entity.

    <style>
        .wrapper {
                position: absolute;
                top: 10px;
                left: 10px;
                height: 25vh;
                width: 25vw;
                background: #00000F;
            }
    
        .stretch{
                position: absolute;
                top: 0px;
                left: 0px;
                height: 100%;
                width: 100%;
        }
    
        .picture {
                border: 1px solid #000000;
        }
    
        .instructions{
            position: absolute;
                top: 30%;
                left: 0px;
                height:50px;
                width: 100%;
                text-align: center;
                background: #ffffff;
        }
    </style>
    
    <div class="wrapper">
        <div class="instructions">
            <p> Press 'p' to take a screenshot.</p>
        </div>
        <div class="stretch">
            <img id="pictureID"/ class="stretch picture">
        </div>
    </div>
    

    Your scene should now look similar to the following, with a Box in the center of the canvas and a picture-in-picture element in the upper-left corner.

    Result

Step 3: Add a Script to Capture User Input

In this step we will add a custom script to the empty _SCRIPTS entity to detect when the user presses the p key, which will then emit an event message to take a screenshot.

  1. With the _SCRIPTS entity selected, add a Script component by pressing Add Component, and then selecting Script.

    Script Component

  2. Click + (plus icon) to add a new script.

    Add Script

  3. Select a Custom script.

    Custom Script

  4. Select the script, rename it “KeyboardInput”, and then press Edit in Text Editor.

    Rename Script

  5. Copy and paste the following script to replace the default script in the Text Editor.

'use strict';

function setup(args, ctx) {

    ctx.key = function(e){
        if(e.key == 'p'){
            console.log("Take Screenshot.");
            sumerian.SystemBus.emit("RecordScreen");
        }
    }
    ctx.domElement.addEventListener("keyup", ctx.key);
}

function cleanup(args, ctx) {
    ctx.domElement.removeEventListener("keyup", ctx.key);
}

This script uses the setup function to define a callback function, ctx.key. The callback function is called every time a keyup event is fired. If the p key is pressed, the function emits a RecordScreen event.

Step 4: Add a Script to Capture Screenshots

We will add another custom script to the _SCRIPTS entity that will take the screenshot. This script listens for RecordScreen events, and updates RenderSystem to record a screenshot after the rendering is complete.

  1. Add a Custom script, and then rename it to “ScreenShotter”.

  2. Copy and paste the following script to replace the default script.

'use strict';

// Configure RenderSystem to capture screenshot onPostRender
function recordScreen(args, ctx, data){

    // Cache the onPostRender function
    ctx.cachedPostRender = ctx.rendersystem.onPostRender;

    // Write a new onPostRender function
    ctx.rendersystem.onPostRender = function(){

        // Capture the rendered canvas as a png
        ctx.domElement.toBlob(function(blob){
            let url = URL.createObjectURL(blob);

            // Send blob to the HTML image element
            ctx.picturePlaceholder.src = url;		
            ctx.picturePlaceholder.onload = function(){
                // Clean up blob
                URL.revokeObjectURL(url);
            }
        },'image/png');

        // Restore previous onPostRender function
        ctx.rendersystem.onPostRender = ctx.cachedPostRender;
    }
}


function setup(args, ctx) {
    // Cache a reference to the RenderSystem and HTML image element
    ctx.rendersystem = ctx.world.getSystem('RenderSystem');
    ctx.picturePlaceholder = document.getElementById('pictureID');

    // Set up listener for RecordScreen
    ctx.recordScreen = function(data){
        recordScreen(args, ctx, data);
    }
    sumerian.SystemBus.addListener('RecordScreen', ctx.recordScreen);
}

function cleanup(args, ctx) {
    sumerian.SystemBus.removeListener("RecordScreen", ctx.recordScreen);
}

This script creates a handler that responds to RecordScreen events. When it detects the event, it temporarily overrides the onPostRender method of the RenderSystem to capture a screenshot of the canvas using the toBlob API.

Step 5: Test the Scene

We now have a complete scene that will take a screenshot when you press the p key. We’ll set the screenshot as the source of the image element in our HTML entity.

Press the Play button, and then press the p key. Your scene should look similar to the image below.

Rename Script

What’s Happening?

Immediately after the render is complete, onPostRender() is called on every active system. By default, this function is defined as an empty function onPostRender() {}. To capture a screenshot, we override the function with a new function.

Before we write a new onPostRender function, we store a reference to the current version. Then, at the end of our custom function, we restore the cached version. As a result, on the frame where we capture a screenshot, we execute a custom onPostRender function, then return to the normal onPostRender behavior on the next frame.

function recordScreen(args, ctx, data){

          // Cache the current onPostRender function
          ctx.cachedPostRender = ctx.rendersystem.onPostRender;

          // Write a new onPostRender function
          ctx.rendersystem.onPostRender = function(){

              // Screen-capture code

              // Restore previous onPostRender function
              ctx.rendersystem.onPostRender = ctx.cachedPostRender;
          }
      }

The ctx.domElemetoBlob() does all the work of copying the pixels rendered into the canvas. In this example we specify the mimeType as 'image/png' which stores the image in memory as a .png file.

The inline callback function takes the bytes stored as a blob, converts them to an object URL, and sets the URL as the source of the image element. We perform an additional cleanup step, revoking the object URL once it is loaded in the image.

    ctx.domElement.toBlob(function(blob){
        let url = URL.createObjectURL(blob);
        ctx.picturePlaceholder.src = url;

        ctx.picturePlaceholder.onload = function(){
            URL.revokeObjectURL(url);
        }
    },'image/png');

In this tutorial, you learned how to create a system that takes screenshots and display them as a picture-in-picture element. You now have all the building blocks you need for capturing the result of a rendered frame. With the ability to now capture screenshots, there are many actions you can take with the captured image: You could use the captured screenshot in your scene as a texture; have the browser save the image to the users filesystem; upload the image to Amazon S3 using the AWS Javascript SDK. You could also keep it in memory and use it as part of your web-application.

For related topics, see the following tutorials:

Back to Tutorials

© 2019 Amazon Web Services, Inc or its affiliates. All rights reserved.