Create Free-Floating Text on Surface Textures in Amazon Sumerian

By Leo Chan | Posted October 21, 2019


Augmented Reality (AR) and Virtual Reality (VR) applications often require text to be displayed in a 3D environment. Examples include UI elements such as heads-up displays (HUDs), text annotations, signage, instructions, or text captioning. Amazon Sumerian provides an HTML entity and an HTML 3D entity that allow for all the text layout and design capabilities of a web browser. This works well when rendering a 3D Sumerian scene in a web browser, on a mobile device, or on a desktop. However, because they’re rendered using a browser Document Object Model and then overlaid on the scene we need to use a different approach to AR and VR devices.

This article shows a technique that renders text to a geometry’s texture map. Because this is standard texture mapping, this technique works for all Sumerian scenes, including AR and VR. To summarize, we show you how to create text in AR and VR.

The layout and formatting provides multiple font choices, coloring, and word wrapping are included. This is achieved using standard Sumerian scripting and standard script property UI widgets.

You’ll learn about:

  • Creating scripts and script UI
  • Live editing during preview mode
  • Creating textures and materials in scripts
  • Using shared objects among scripts

Step 1: Create and Set Up the Scene

  1. Navigate to the Sumerian Dashboard, and create a new, empty scene. Name the scene “Surface Text”.

  2. At the top of the canvas, choose Create Entity.

  3. In the dialog box that appears, under the 3D Primitives section, choose Quad.

  4. With the Quad selected, in the Inspector panel, choose Add Component, and then choose Script.

  5. In the Script component, choose the + (plus icon), and then choose Custom (Preview Format).

Step 2: Copy the displaySurfaceText Script

  1. Open the Text Editor, and press the “J” shortcut.

  2. Rename the script you created earlier from Script to “displaySurfaceText”.

  3. Select the displaySurfaceText script and replace its contents with the following.

//
// This code is for reference and demonstration purposes only. It is offered 'as-is' with
// no warranties, express or implied.
//
// (c) Amazon Web Services, 2019
// authors: Leo Chan
//

import * as s from 'module://sumerian-common/api';
import * as si from 'module://sumerian-common/internal';

export default class SurfaceText extends s.Action {

  static get PROPERTIES() {
    return {
      textureSize: {
        order:        0,
        type:         s.type.Float,
        control:      s.control.Select,
        description: 'Texture dimensions in pixels. Smaller values will save memory, larger values will result in smoother text',
        options:      [ 256, 512, 1024, 2048, 4096 ],
        default:      512,
      },
      textOffset: {
        order:        1,
        type:         s.type.Vector2,
        description: 'Text offset, scaled from 0 to 1',
        default:      [ 0, 0 ],
      },
      backgroundOpacity: {
        order:   2,
        type:    s.type.Float,
        control: s.control.Slider,
        min:     0,
        max:     1,
        default: 0,
      },
      backgroundColor: {
        order:   3,
        type:    s.type.Vector3,
        control: s.control.Color,
        default: s.color.White,
      },
      textColor: {
        order:   4,
        type:    s.type.Vector3,
        control: s.control.Color,
        default: s.color.White,
      },
      fontFamily: {
        order:        5,
        type:         s.type.String,
        description: 'CSS Font Family. See https://developer.mozilla.org/en-US/docs/Web/CSS/font-family for details.',
        default:     '"Helvetica Neue", Helvetica, Arial, sans-serif',
      },
      fontStyle: {
        order:        6,
        type:         s.type.String,
        control:      s.control.Select,
        description: 'Sets whether a font should be styled with a normal, italic, or oblique face from its font family.',
        options:      [ 'normal', 'bold', 'italic', 'oblique'],
        default:     'normal',
      },
      fontSize: {
        order:        7,
        type:         s.type.Integer,
        description: 'Font size in texture pixels',
        control:      s.control.Slider,
        min:          1,
        max:          512,
        step:         1,
        default:      50,
      },
      textAlign: {
        order:      8,
        type:       s.type.String,
        control:    s.control.Select,
        options: [ 'left', 'center', 'right' ],
        default:   'center' ,
      },
      verticalAlign: {
        order:      9,
        type:       s.type.String,
        control:    s.control.Select,
        options: [ 'top', 'middle', 'bottom' ],
        default:   'middle',
      },
      text: {
        order:        10,
        type:         s.type.String,
        description: 'Text to display, use “\n” for multi-line text',
        default:     'SurfaceText :)',
      }
    };
  }

  static get SIGNALS() {
    return {
      onSuccess: {
        description: 'Triggered when text rendering is complete',
      },
      onFailure: {
        description: 'Triggered when there is an error in text rendering ',
      },
    };
  }

  start(ctx) {
    ctx.autoStop = false;

    // Start the displaySurfaceText action function when this action starts
    ctx.start(
      ctx => this.displaySurfaceText(ctx, {text: this.text})
    );

    // Set up an entity event listener to update text. Call it by getting the entity
    // this script is on and calling entity.event('updateText').emit('New Text')
    ctx.entity.event('updateText').listen(ctx, text => this.displaySurfaceText(ctx, {text}));
  }

  // ___________________________________________________________________________________________

  //
  // displaySurfaceText: renders text on a texture
  // with a transparent background onto the entity.
  //
  // @param {Object} ctx - The calling action's context object
  // @param {Object} options - Text configuration options with following properties
  //   entity: (required) The ctx.entity object the script is on
  //   text: The text to render. Use the character string \n for newlines
  //   textureSize: Integer dimension of the texture. Use a power of two (64,128,256,512,1024,...)
  //   fontFamily: See https://developer.mozilla.org/en-US/docs/Web/CSS/font-family. For example, '"Helvetica Neue", Helvetica, Arial, sans-serif',
  //   fontStyle: 'normal'|'bold'|'italic'|'oblique'
  //   fontSize: Font size string, e.g., '128px'
  //   textAlign: 'left'|'right'|'center' - center is the default
  //   verticalAlign: 'top'|'bottom'|'middle' - middle is the default
  //   fillStyle: Color to use for the text, in hex or common color name. 'white' is the default.
  //
  displaySurfaceText(ctx, options) {

    // Use the 'default' values set in the SurfaceText PROPERTIES as default values
    const defaultOptions = Object.entries(SurfaceText.PROPERTIES).reduce(
      (accumulator, entry) => {
        accumulator[entry[0]] = entry[1].default;
        return accumulator;
      }, {});
    // Collect options for this object's properties and the passed in options object
    const in_options = {
      textureSize: this.textureSize,
      textOffset: this.textOffset,
      textColor: this.textColor,
      textAlign: this.textAlign,
      verticalAlign: this.verticalAlign,
      backgroundColor: this.backgroundColor,
      backgroundOpacity: this.backgroundOpacity,
      fontFamily: this.fontFamily,
      fontStyle: this.fontStyle,
      fontSize: this.fontSize + 'px',
      entity: ctx.entity,
      text: options.text
    };
    this.options = {};

    //
    // numberToHexString
    // Convert a normalized number [0,1] to hexadecimal string representation
    //
    // @param {Number} n
    //
    const numberToHexString = ( n ) => Math.floor(
        Math.max( 0, Math.min( 1, Number( n ))) * 255
      )
      .toString( 16 )
      .padStart( 2, '0' );

    //
    // vector3ToHex
    // Convert a Vector3 normalized RGB color to hexadecimal string representation
    //
    // @param {Vector3} vector3ToConvert
    //
    const vector3ToHex = (vector3ToConvert) => '#' +
      numberToHexString(vector3ToConvert.x) +
      numberToHexString(vector3ToConvert.y) +
      numberToHexString(vector3ToConvert.z);

    //
    // wordWrap: insert newlines for wordWrap using the provided canvasContext
    //
    // @param {Object} canvasContext 2D canvas context used to measure the text
    // @param {String} text String to word wrap. This may contain the string \n which is treated as a newline
    //
    const wordWrap = (canvasContext, text) => {
      const lines = [];
      const newlineLines = text.split('\\n');
      for(let line of newlineLines) {
        const space  = ' ';
        const words = line.split(space);

        if (words.length <= 1) {
          lines.push(line);
        } else {
          let lineLast   = words.shift();   // pop the first element of the words array
          let lineLength = 0;

          for (let word of words) {
            lineLength = canvasContext.measureText(lineLast + space + word).width;
            if( lineLength < canvasContext.canvas.width) {
              lineLast += (space + word);
            } else {
              lines.push(lineLast);
              lineLast = word;
            }
          }
          lines.push(lineLast);
        }
      }
      return lines;
    };

    // Let's cascade our options from defaults to custom options
    Object.assign(this.options, defaultOptions, in_options);

    // Additional computed default options
    this.options.font = options.font || `${this.options.fontStyle} ${this.options.fontSize} ${this.options.fontFamily}`;
    if (this.options.textColor) {
      this.options.fillStyle = vector3ToHex(this.options.textColor);
    }

    // Create a shared canvas per entity to render text. We can't share it globally, unfortunately, because
    // the material uses a reference to the canvas (set in setImage) and defers its usage until it renders.
    const entityCanvasContext = ctx.entity.value('text-canvas-channel');
    let canvasContext = entityCanvasContext.get();
    if(!canvasContext) {
      entityCanvasContext.set(document.createElement( 'canvas' ).getContext('2d'));
      canvasContext = entityCanvasContext.get();
    }

    if(!canvasContext) {
      console.log("[displaySurfaceText] Error: can't get canvas context! Aborting surface text render.");
      ctx.signal(this.onFailure);
      return;
    }

    // Configure the canvas
    canvasContext.canvas.width = this.options.textureSize;
    canvasContext.canvas.height = this.options.textureSize;
    canvasContext.font = this.options.font;
    canvasContext.textAlign = this.options.textAlign;
    canvasContext.fillStyle = this.options.fillStyle;
    canvasContext.textBaseline = this.options.verticalAlign;
    canvasContext.text = this.options.text;

    // Render the text on the canvas
    const lines = wordWrap(canvasContext, this.options.text);
    const lineHeight = options.lineHeight || parseFloat( this.options.fontSize ) * 1.2;
    let x = 0;
    let y = 0;
    if( this.options.textAlign === 'left' ) x = 0;
    else if( this.options.textAlign === 'right' ) x = canvasContext.canvas.width;
    else x = Math.floor(canvasContext.canvas.width / 2);

    if( this.options.verticalAlign === 'top' ) y = 0;
    else if( this.options.verticalAlign === 'bottom' ) y = Math.floor(canvasContext.canvas.height - lineHeight * (lines.length-1));
    else y = Math.floor((canvasContext.canvas.height - lineHeight * (lines.length - 1)) / 2 );

    x += Math.floor(this.options.textOffset.x * canvasContext.canvas.width);
    y += Math.floor(this.options.textOffset.y * canvasContext.canvas.height);

    canvasContext.clearRect( 0, 0, canvasContext.canvas.width, canvasContext.canvas.height );
    canvasContext.fillStyle = vector3ToHex(this.options.backgroundColor) + numberToHexString(this.options.backgroundOpacity);
    canvasContext.fillRect( 0, 0, canvasContext.canvas.width, canvasContext.canvas.height);
    canvasContext.fillStyle = this.options.fillStyle
    for( let i = 0; i < lines.length; i ++ ) {
      canvasContext.fillText( lines[ i ], x, y + i * lineHeight );
    }

    // Get or create texture and material on the entity for the text
    const reusableTexture = ctx.entity.value('text-texture-channel');
    let texture = reusableTexture.get();
    if(!texture) {
      reusableTexture.set(new si.Texture( null,
        {
          wrapS: 'EdgeClamp',
          wrapT: 'EdgeClamp',
          premultiplyAlpha: true,
        },
        this.options.textureSize,
        this.options.textureSize));
      texture = reusableTexture.get();
    }
    if(!texture) {
      console.log("[displaySurfaceText] Error: cannot get texture. Aborting text render");
      ctx.signal(this.onFailure);
      return;
    }

    texture.setImage( canvasContext.canvas, canvasContext.canvas.width, canvasContext.canvas.height );

    // Assign texture to a clone of the existing material on the entity. We clone so that entities that happen
    // to share a material don't all end up with the same text.
    const reusableMaterial = ctx.entity.value('text-material-channel');
    let material = reusableMaterial.get();
    const meshRendererComponent = this.options.entity ? si.Entity.forPublicEntity(this.options.entity).meshRendererComponent : null;
    if(!material) {
      if(meshRendererComponent) {
        reusableMaterial.set(meshRendererComponent.materials[0].clone());
        material = reusableMaterial.get();
      } else {
        console.error("[displaySurfaceText] Error: entity not provided to SurfaceTextDisplayer object. Aborting Text Render.");
        ctx.signal(this.onFailure);
        return;
      }
    }
    if(!material || !meshRendererComponent) {
      console.log("[displaySurfaceText] Error: cannot find material on the entity. Aborting text render");
      ctx.signal(this.onFailure);
      return;
    }
    meshRendererComponent.materials[0] = material;

    // Set material properties
    material.blendState.blending = 'TransparencyBlending';
    material.renderQueue = si.RenderQueue.TRANSPARENT;
    material.uniforms.opacity = 1;
    // Assign the texture to the diffuse map - other possibilities include
    // PBR Maps:PBR_SPECULAR_MAP, PBR_GLOSSINESS_MAP, PBR_SPECULAR_GLOSSINESS_MAP,
    // Classic Maps: SPECULAR_MAP, AO_MAP, TRANSPARENCY_MAP, REFLECTION_MAP
    const mapName = (material.shader.name === 'ShaderLib.pbr') ? si.Shader.BASE_COLOR_MAP : si.Shader.DIFFUSE_MAP;
    material.setTexture(mapName, texture);

    // For visual state machines, transition when complete
    ctx.signal(this.onSuccess);
  }
}

Step 3: Understanding the Script

This is a fair bit of code, so let’s walk through it from top to bottom.

  1. The first import lines and export default class lines import the Sumerian modules used in the script and set up a default exported action for Sumerian to run. See the scripting API documentation for details.

  2. The PROPERTIES section sets up the Script component UI. See the scripting API Action documentation for details.

  3. The SIGNALS section defines the State Machine outbound transitions that are used with this script. See the scripting API Action documentation for details.

  4. Next we set ctx.autoStop = false. Typically, scripts stop after all their child actions complete. In our script, however, we want to be able to live edit the text. That is, we want to make changes to the PROPERTIES during preview mode and have the script restart after each change. To do this, we need to override the default auto-stop behavior because PROPERTIES changes will only restart actions that aren’t stopped. The line ctx.autoStop = false; in our start function makes our root action not stop after all its child actions have completed.

  5. Next we use the ctx.start function to start a set of child actions at the same time. In our case, we only start one child action. We use the Action Function syntax, using JavaScript big arrow notation, as described in the scripting API documentation. This calls the function displaySurfaceText when the root script Action instance is started (and restarted).

  6. displaySurfaceText is where the actual rendering of the text occurs. It starts by defining a few helper functions to convert colors from a numerical value in the range of 0 to 1 to a hexadecimal format used by the canvas document element. It also defines a helper wordWrap function to handle the wrapping of lines of text within the texture.

  7. After copying the configuration options and assigning default values, we create a canvas document element to handle the rendering of the text to a texture. We use ctx.entity.value to store and retrieve a canvas that can be shared across action restarts, such as when the PROPERTIES of the SurfaceText script change. See the scripting API for more details.

  8. The code starting at the // Render the text on the canvas comment then sets up the text rendering and renders it to the canvas.

  9. Once the text is rendered, we create a texture to put it on by using the texture.setImage() function. Similar to the shared canvas, it would be wasteful to re-create the texture every time there is a PROPERTIES change, so we use the ctx.entity.value to create a single texture per entity that we can re-use across restarts (that is, PROPERTIES changes). See the scripting API documentation for more details.

  10. Next we clone the existing material on the entity on which we’ll use the texture. This allows us to inherit all the material properties from the material on the entity, while preventing our text texture from showing up on other entities that might be sharing the entity’s original material.

  11. Finally the ctx.signal(this.onSuccess) line will trigger the onSuccess transition when this script is used in an Execute Script action in a State Machine.

Step 4: Test Your Scene

  1. Press the play button at the bottom of the canvas. The surface text will show up only in preview mode, because scripts are only run in preview mode.

  2. Click the Inspector panel to enable live editing.

  3. On the left side of the Entities panel, select the Quad you created.

  4. Change some of the properties of your script in the Script component. Observe how these edits are shown in the viewport. You can change the text, placement, color, opacity, font, font size, and other properties. Try it out and experiment! Also try increasing the Transform component Scale property to make the text bigger. For the Font Family property, any CSS font-family property string will work.

Optional: Try It on Different Surfaces

Most commonly you’ll be putting text on Quads (flat surfaces), but you could use this script on any surface. Try it on a Sphere entity and see what happens.

Tip: You can instantiate a script on an entity that doesn’t have a Script component on it by simply dragging the script from the Assets panel onto the entity’s Inspector panel. This will add the Script component and instantiate the dropped script at the same time.

This article’s JavaScript script uses the Amazon Sumerian Engine API. To learn more about the API and what you can achieve by starting with the API, see the Introduction and Quick Start guides.

Leo Chan

Leo Chan is a Senior Specialist Solutions Architect working with AR/VR technologies. He loves working at the intersection of Technology and Art and has built tools such as Maya and Houdini for the Film and Video Gaming industries and created block buster content with studios including Pixar and Electronic Arts.