Intermediate Tutorial


Creating Multiplayer XR Web Applications


30 minutes

Posted on: November 22, 2018

Learn Sumerian
Creating Multiplayer XR Web Applications

Tags

aws integration
cognito
publish
website
customize
appsync
react
new

In this tutorial you will learn about:


In this 4 part tutorial you’ll be building an Amazon Sumerian scene that has:

  • Authentication
  • Custom splash screen / loading indicator while the engine loads your scene
  • Interaction using React components within your web application
  • Interaction with other users that have their scene open.

You will build React components that communicate with entities in your scene. You’ll add the ability to attach a modifiable name to a moving sphere in your scene as well as change its color and add the functionality to see the sphere’s from other user’s scenes in your scene.

You will learn about the following:

  • Amplify
  • Sumerian scripting
  • React components
  • Sumerian message passing / SystemBus
  • HTML3D components
  • AWS AppSync

Prerequisites:

  • If you have not yet installed and configured the AWS Amplify CLI, check out the video walkthrough.

Part 1: Create a Sumerian scene, create an Amplify application and setup a splash screen / loading indicator React component

Step 1: Complete the Getting Started with Amazon Sumerian, AWS Amplify, and the React Component tutorial.

Note: When you reach “Step 2: Create a Sumerian Scene” in that tutorial, create a scene using the “Default Lighting” template and skip building the room and leave your scene empty for now. We’ll be building a completely different scene in this tutorial.

Step 2: Implement the first React component

The first component we’ll be implementing is IndeterminateLoading which will be displayed while our scene is loading.

Open App.js and implement the IndeterminateLoading and SumerianScene components.

function IndeterminateLoading() {
    return <img src={logo} className="App-logo" alt="logo"/>;
}

class SumerianScene extends Component {

    async componentDidMount() {
        await this.loadAndStartScene();
    }

    render() {
        return <div
            id="sumerian-scene-dom-id"
            style={{width: "100%", height: "100%", position: "absolute"}}
        />;
    }

    async loadAndStartScene() {
        await XR.loadScene(this.props.scene, "sumerian-scene-dom-id");
        const controller = XR.getSceneController(this.props.scene);
        this.props.onLoaded(controller);
        XR.start(this.props.scene);
    }

}

And modify the App component.

class App extends Component {

    constructor(props) {
        super(props);
        this.state = {
            loading: true,
            sceneController: null
        };
    }

    sceneLoaded(sceneController) {
        this.setState({
            loading: false,
            sceneController
        });
    }

    render() {
        return (
            <div className="App">
                {this.state.loading && <IndeterminateLoading/>}
                <div style={{visibility: this.state.loading && 'hidden'}}>
                    <SumerianScene scene='scene1' onLoaded={(controller) => this.sceneLoaded(controller)}/>
                </div>
            </div>
        );
    }
}

Start a local development server for your Amplify application by running the following commands:

cd sumerian-amplify-app
npm start

This should have launched a browser window opened with http://localhost:3000/.

You should now see a rotating atom / React icon while your scene loads.

Part 2: Create Sumerian entities and implement React components that interact with them

Open up the empty Sumerian scene created in Part 1 with the Sumerian scene editor.

Step 1: Add a sphere, HTML 3D and light entity to the Scene

  1. Above the canvas, choose Create Entity.
  2. Select Sphere under the 3D Primitives category.
  3. Rename the entity to LocalSphere.
  4. Above the canvas, choose Create Entity again.
  5. Select HTML3D under the Others category.
  6. In the Entities panel on the left side of the editor drag and drop the newly created Html3d Entity on the newly created Sphere to make the Html3d Entity a child of the Sphere.
  7. Select the Html3d Entity.
  8. In the Inspector panel on the right side of the editor expand the Transform component.
  9. Uncheck Uniform Scale, set Translation(0, 0.55, 0), Rotation(270, 0, 0), Scale (1, 0.2, 0.5).
  10. In the inspector panel on the right side of the editor expand the HTML 3D component.
  11. Change Width to 175.
  12. Click the Open in Editor button.
  13. In the text editor that was just opened add an class attribute to the <p> element. After you’ve made the change save it, close the text editor and return to the main Sumerian editor.
<p class="sphere-name" style="font-size: 24px; margin: 0;">
    undefined
</p>

Step 2: Setup event listeners

  1. Select the LocalSphere in the Entities panel on the left side of the editor.
  2. Click the Add Component button in the Inspector panel on the right side of the editor.
  3. Select Script from the pop-up menu.
  4. Within the newly added Script component click the + (Add Script) button and select Custom from the pop-up menu.
  5. Within the newly added Script instance click the pencil icon (Edit Script) button.
  6. In the the previous tutorial, Getting Started with Amazon Sumerian, AWS Amplify, and the React Component, you imported the SumerianScene component from the aws-amplify-package using the following line:

     import { withAuthenticator, SumerianScene  } from 'aws-amplify-react';
    

    Before you continue, make sure to remove SumerianScene from the import statement to avoid any compiling errors.

     function hexToRgb(hex) {
         var result = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(hex);
         return [parseInt(result[1], 16), parseInt(result[2], 16), parseInt(result[3], 16)];
     }
    
     function updateColor(ctx, value) {
         let rgb = [...hexToRgb(value).map(value => value / 255), 1];
         ctx.entity.getComponent('MeshRendererComponent').materials[0].uniforms.materialDiffuse = rgb;
     }
    
     function updateName(ctx, value) {
         const nameElement = ctx.entity.children().top.find(entity => entity.dom3dComponent).dom3dComponent.domElement.getElementsByClassName('sphere-name')[0];
         nameElement.textContent = value;
     }
    
     function setup(args, ctx) {
         sumerian.SystemBus.addListener('updateSphereColor', (value) => updateColor(ctx, value));
         sumerian.SystemBus.addListener('updateSphereName', (value) => updateName(ctx, value));
     }
    
  7. In the text editor that was opened replace the existing setup function with the following code snippet. After you’ve made the change save it, close the text editor and return to the main Sumerian editor.

Step 3: Set the camera to get a top view of the board

  1. Select the Default Camera in your scene from the hierarchy panel.
  2. In the Camera component, uncheck the Follow Editor check box.
  3. Set the Transform component to have Translation (0, 10, 0), Rotation(-89, 0, 0) and Scale (1, 1, 1) to get a good top view of the entire board.

Save and then re-publish your Sumerian scene.

Step 4: Implement a React component to change the name of the Sphere

Open App.js add LabelledInput, TextInput and SceneControls components.

function LabelledInput({label, type, onChange}) {
    return (
        <label>
            {label}
            <input type={type} onChange={(event) => onChange(event.target.value)}/>
        </label>
    );
}

function TextInput({label, onChange}) {
    return <LabelledInput label={label} type="text" onChange={onChange}/>
}

class SceneControls extends Component {

    emit(channelName, value) {
        if (((this.props.sceneController || {}).sumerian || {}).SystemBus) {
            this.props.sceneController.sumerian.SystemBus.emit(channelName, value)
        }
    }

    updateSphereName(value) {
        this.emit("updateSphereName", value);
    }

    render() {
        return (
            <div>
                <TextInput label="Sphere name" onChange={(value) => this.updateSphereName(value)}/>
            </div>
        );
    }
}

And modify the App component’s render method.

class App extends Component {

    ...

    render() {
        return (
            <div className="App">
                {this.state.loading && <IndeterminateLoading/>}
                <div style={{visibility: this.state.loading && 'hidden'}}>
                    <SceneControls sceneController={this.state.sceneController}/>
                    <SumerianScene scene='SumerianAmplify' onLoaded={(controller) => this.sceneLoaded(controller)}/>
                </div>
            </div>
        );
    }
}

You should now see an input labelled “Sphere name” above you scene, changing the input’s value updates the text in Html3d Entity above the Sphere.

Step 5: Implement a React component to change the color of the Sphere

The last React component we’ll implement, ColorInput, will allow you to change the color of the sphere added to the scene.

Open App.js and implement ColorInput.

function ColorInput({label, onChange}) {
    return <LabelledInput label={label} type="color" onChange={onChange}/>;
}

And modify SceneControls by adding the updateSphereColor method and modifying the render method.

class SceneControls extends Component {

    ...

    updateSphereColor(value) {
        this.emit("updateSphereColor", value);
    }

    render() {
        return (
            <div>
                <TextInput label="Sphere name" onChange={(value) => this.updateSphereName(value)}/>
                <ColorInput label="Sphere color" onChange={(value) => this.updateSphereColor(value)}/>
            </div>
        );
    }
}

You should now see an input labelled “Sphere color” above you scene, changing the input’s value updates the color of the the LocalSphere.

Tip: If you’re unable to pan, rotate or zoom your scene add the following CSS rule to App.css. This may happen after adding the HTML3D entity to your scene.

#Dom3dSystem-root {
  pointer-events: none;
}

You just built an Amplify XR application with React components that can interact with your Sumerian scene!

Part 3: Adding a WASD controlled sphere to your Sumerian scene

Step 1: Add a plane for the sphere to move on.

  1. Create a new entity in your scene by pressing the Create Entity button and selecting Quad from the 3D Primitives section.
  2. Rename the entity to Board.
  3. Set the Transform component to have Translation (0, -1, 0), Rotation (-90, 0, 0) and Scale (0, 0, 0).
  4. (optional) Update the texture on the Material component to customize the board.

Step 2: Add physics to the LocalSphere and the Board

  1. Select the LocalSphere entity and add the Rigid Body component without changing any of the default properties.
  2. Add the Collider component to the LocalSphere entity and set the Shape as Sphere.
  3. Add the Collider component to the Board entity and set the Shape as Infinite Plane.

Step 3: Add a state machine to the LocalSphere entity

  1. Select the LocalSphere entity and add the State Machine component.
  2. Create a new behavior using the + (Create Behavior) button in the component.
  3. Edit the behavior by pressing the edit button (pencil icon) next to the behavior.
  4. Rename the state to Default and add WASD Keys action to the state.
  5. Create 4 more states for each action.
  6. Add respective Key Up action to each state for each WASD keys.
  7. Create state transitions by dragging arrows from the source WASD to their corresponding state on Key Press and back on the Key Up action.
  8. Add Apply force on rigid body to the WASD states with Force values to correspond to W(0, 0, -5), A(-5, 0, 0), S(0, 0, 5) and D(5, 0, 0) depending on the state.

Save and then re-publish your Amplify scene and try to control your LocalSphere with the WASD keys.

Part 4: Integrating AppSync for multiplayer support

Step 1: Create the AppSync API that will be used as the backend to support multiplayer message exchange

  1. Use this template to create an AWS CloudFormation stack. To learn more about creating CloudFormation stacks, see the AWS Setup tutorial. The stack template provided above will already contain the following AppSync schema:

     type Mutation {
         transformPlayer(
             playerId: String!,
             playerName: String!,
             color: String!,
             transformData: AWSJSON!
         ): AWSJSON
     }
    
     type Query {
         noop: String
     }
    
     type Subscription {
         subscribeToTransformPlayer: AWSJSON
         @aws_subscribe(mutations: ["transformPlayer"])
     }
    
     schema {
         query: Query
         mutation: Mutation
         subscription: Subscription
     }
    
  2. In your CloudFormation dashboard select the AmazonSumerianAppSyncTutorialStack and note the AppSyncApiUrl from Stack Outputs. You will need this when configuring the Sumerian scene.

  3. Next, we need to update our authenticated access IAM role to allow Sumerian access. To do this, open the configuration file located at amplify/backend/amplify-meta.json, find the authRoleName and copy the value of the role to your clipboard (it should look something like sumeriandemo-012345678910-authRole).

  4. Open the IAM dashboard in the AWS Console, click on Roles, & search for the authRoleName role we copied to the clipboard. Once you find the role, click on it.

  5. Next, under the permissions tab, click on Attach Policies, select the AWSAppSyncInvokeFullAccess policy from the list and click on Attach Policy. This allows your Sumerian scene to invoke AppSync for posting or subscribing to mutations.

Step 2: Bundle up AppSync client to use in the Sumerian scene

The next step is to bundle AppSync Mobile SDK so that it is referable in a Sumerian scene. To do that,

  1. Create a new project and initialize it with npm, accepting the defaults, as follows:

     mkdir appsync && cd appsync
     touch index.js webpack.config.js
     npm init
    
  2. Edit your package.json file and be sure it includes the following:

     "scripts": {
       "build": "NODE_ENV=production webpack --config webpack.config.js --progress"
     },
     "dependencies": {
       "apollo-cache-inmemory": "^1.1.0",
       "apollo-client": "^2.0.3",
       "apollo-link": "^1.0.3",
       "apollo-link-http": "^1.2.0",
       "aws-sdk": "^2.141.0",
       "aws-appsync": "^1.0.0",
       "es6-promise": "^4.1.1",
       "graphql": "^0.11.7",
       "graphql-tag": "^2.5.0",
       "isomorphic-fetch": "^2.2.1",
       "ws": "^3.3.1"
     },
     "devDependencies": {
       "babel-core": "^6.26.3",
       "babel-loader": "^7.1.2",
       "babel-plugin-es6-promise": "^1.1.1",
       "babel-plugin-transform-class-properties": "^6.24.1",
       "babel-plugin-transform-object-rest-spread": "^6.26.0",
       "babel-preset-env": "^1.7.0",
       "webpack": "^4.26.0",
       "webpack-cli": "^3.1.2"
     }
    
  3. From a command line, run the following: npm install

  4. Now add the following code to your index.js file:

     'use strict';
    
     import { AUTH_TYPE } from 'aws-appsync/lib/link/auth-link';
     import AWSAppSyncClient from 'aws-appsync';
     import gql from 'graphql-tag';
    
     const appsync = {
         AWSAppSyncClient: AWSAppSyncClient,
         AUTH_TYPE: AUTH_TYPE,
         gql: gql
     };
    
     if (typeof window !== 'undefined') {
         window.appsync = appsync;
     }
    
     export default appsync;
    
  5. And the following code to your webpack.config.js file:

     const webpack = require('webpack');
    
     const webpackConfig = {
       context: __dirname,
       entry: { appsync: './index.js' },
       output: {
         path: `${__dirname}/dist`,
         filename: 'appsync.js'
       },
       module: {
         rules: [
           { test: /\.js?$/, use: 'babel-loader', exclude: /node_modules/ },
         ],
       }
     };
    
     module.exports = webpackConfig;
    
  6. Now from the command line, run the following:
     npm run build
    
  7. Copy the file dist/appsync.js to the /public directory in your Amplify app that was created in the previous part of the tutorial.
  8. Run amplify publish again to publish the appsync.js file along with your Amplify web app.

Step 3: Create an entity to initialize the environment in your Sumerian scene

  1. Now that the infrastructure is set for AppSync, go back to your Sumerian scene to integrate the Multiplayer experience.
  2. Create a new entity in your scene by pressing the Create Entity button and selecting Entity from the Others section.
  3. Rename this entity to IntEnvEntity.
  4. Add a Script component on it for initializing the AppSync integration.
  5. Press the + (Add Script) button in the Script component followed by choosing Custom from the menu.
  6. Open the script editor by clicking the edit button (pencil icon).
  7. Add the appsync.js file published in your Amplify web app from the previous step to the current script as an External Resource.
    1. The URL might contain https://...cloudfront.net/appsync.js. Alternatively, the URL could end in amazonaws.com. This will depend on whether you chose a prod or dev deployment. The the URL will be found in the terminal after the $ amplify publish command completes.
    2. If the Sumerian script editor complains that URLs need to be https or protocol agnostic (e.g. //cdnjs.cloudflare.com/...) your Amplify hosting is configured in development mode. This means that you’ll need to add CloudFront hosting so that your Amplify web app can be served using HTTPS (recommended) or use the protocol agnostic URI syntax //.
  8. Replace all the existing code with the following:

     'use strict';
    
     const appSyncUrl = 'https://*******.appsync-api.us-west-2.amazonaws.com/graphql';
     const appSyncRegion = 'us-west-2';
     const localClientId = String(Math.random()).replace('.', '');
     const remoteClientIdEntityMapping = {};
    
     function hexToRgb(hex) {
         var result = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(hex);
         return [parseInt(result[1], 16), parseInt(result[2], 16), parseInt(result[3], 16)];
     }
    
     function updateColor(entity, value) {
         let rgb = [...hexToRgb(value).map(value => value / 255), 1];
         entity.getComponent('MeshRendererComponent').materials[0].uniforms.materialDiffuse = rgb;
     }
    
     function updateName(entity, value) {
         const nameElement = entity.children().top.find(entity => entity.dom3dComponent).dom3dComponent.domElement.getElementsByClassName('sphere-name')[0];
         nameElement.textContent = value;
     }
    
     function updateRemotePlayer(entity, remoteData) {
         entity.setTranslation(remoteData.transformData.x, remoteData.transformData.y, remoteData.transformData.z);
         updateColor(entity, remoteData.color);
         updateName(entity, remoteData.playerName);
         entity.appSyncDemoLastUpdate = Date.now();
     }
    
     function addRemotePlayer(world, remoteClientIdEntityMapping, remotePlayerPrefabId, remoteData) {
         world.loader.loadInstance(remotePlayerPrefabId).then((resource) => {
             const material = new sumerian.Material(sumerian.ShaderLib.texturedLit);
             resource.getComponent('MeshRendererComponent').materials[0] = material;
    
             world.addEntity(resource);
             // resource.scriptComponent.setup(resource); // Uncomment if you add a script component to the remote prefab
             remoteClientIdEntityMapping[remoteData.playerId] = resource;
             updateRemotePlayer(resource, remoteData);
         });
     }
    
     function setup(args, ctx) {
         // Used to cleanup spheres after 5s of inactivity
         ctx.entity.appSyncDemoIntervalId = setInterval(function() {
             const now = Date.now();
             for (let key in remoteClientIdEntityMapping) {
                 if (now - remoteClientIdEntityMapping[key].appSyncDemoLastUpdate > 5000) {
                     remoteClientIdEntityMapping[key].removeFromWorld();
                     delete remoteClientIdEntityMapping[key];
                 }
             }
         }, 1000);
         const credentials = ctx.world.getSystem("AwsSystem")._sdkConfig.credentials;
         const transformPlayerSubscription = appsync.gql(`
             subscription TransformPlayerSubscription {
                 subscribeToTransformPlayer
             }`
         );
         const transformPlayerMutation = appsync.gql(`
           mutation TransformPlayerMutation($playerId: String!, $playerName: String!, $color: String!, $transformData: AWSJSON!) {
             transformPlayer(playerId: $playerId, playerName: $playerName, color: $color, transformData: $transformData)
           }`
         );
    
         const client = new appsync.AWSAppSyncClient({
             url: appSyncUrl,
             region: appSyncRegion,
             auth: {
               type: appsync.AUTH_TYPE.AWS_IAM,
               credentials: credentials
             }
         });
    
         const remoteTransformPlayerWrapper = function(loader, remotePlayerPrefabId) {
             return function(response) {
                 let remoteData = JSON.parse(response.data.subscribeToTransformPlayer);
                 if (remoteData.playerId === localClientId) { // Skipping echo
                     return;
                 } else if (remoteData.playerId in remoteClientIdEntityMapping) { // Remote player exists. Woohoo translate it
                     updateRemotePlayer(remoteClientIdEntityMapping[remoteData.playerId], remoteData)
                 } else { // Remote player doesn't exist. Create one.
                     console.log('Adding remote player', remoteData);
                     addRemotePlayer(loader, remoteClientIdEntityMapping, remotePlayerPrefabId, remoteData)
                 }
             }
         }
    
         const localTransformPlayerWrapper = function(client) {
             return function(playerName, playerColor, translation) {
                 client.mutate({
                     mutation: transformPlayerMutation,
                     variables: {
                         playerId: localClientId,
                         playerName: playerName,
                         transformData: JSON.stringify(translation),
                         color: playerColor
                     }
                 }).catch(err => console.log('Error transforming player', err));
             }
         }
    
         client.hydrated().then(function (client) {
             ctx.worldData.transformPlayer = localTransformPlayerWrapper(client);
             const observable = client.subscribe({ query: transformPlayerSubscription });
             observable.subscribe({
                 next: remoteTransformPlayerWrapper(ctx.world, args.remotePlayerPrefab.id),
                 complete: console.log,
                 error: console.log,
             });
         });
    
     }
    
     function cleanup(args, ctx) {
         window.clearInterval(ctx.entity.appSyncDemoIntervalId);
         for (let key in remoteClientIdEntityMapping) {
             remoteClientIdEntityMapping[key].removeFromWorld();
             delete remoteClientIdEntityMapping[key];
         }
     }
    
     // Defines script parameters.
     //
     var parameters = [
         {type: 'entity', key: 'remotePlayerPrefab', description: 'Remote Spawn'}
     ];
    
  9. Update the appSyncUrl in the script to the one you created in Step 1.

Step 4: Create an entity that will be used to spawn remote players

  1. Duplicate the LocalSphere entity and rename it to RemoteSphere.
  2. Remove the Collider, Rigid Body, Script, and State Machine components from it.
  3. Drag the RemoteSphere to the Default Pack in the Assets panel.
  4. Delete it from the scene.
  5. Add the reference of the RemoteSphere to the Script component of the InitEnvEntity by dragging it from the Asset Panel on to the Drop entity area for Remote Player Prefab.

Step 5: Update script to echo local player position to AppSync

  1. Edit the script on the LocalSphere and replace it with:

     function hexToRgb(hex) {
         var result = /^#?([a-f\d]{2})([a-f\d]{2})([a-f\d]{2})$/i.exec(hex);
         return [parseInt(result[1], 16), parseInt(result[2], 16), parseInt(result[3], 16)];
     }
    
     function getColor(ctx) {
         return ctx.entity.appSyncDemoSphereColor;
     }
    
     function updateColor(ctx, value) {
         let rgb = [...hexToRgb(value).map(value => value / 255), 1];
         ctx.entity.appSyncDemoSphereColor = value;
         ctx.entity.getComponent('MeshRendererComponent').materials[0].uniforms.materialDiffuse = rgb
     }
    
     function getName(ctx) {
         return ctx.entity.appSyncDemoSphereName;
     }
    
     function updateName(ctx, value) {
         const nameElement = ctx.entity.children().top.find(entity => entity.dom3dComponent).dom3dComponent.domElement.getElementsByClassName('sphere-name')[0];
         ctx.entity.appSyncDemoSphereName = value;
         nameElement.textContent = value;
     }
    
     function setup(args, ctx) {
         ctx.entity.appSyncDemoSphereName = 'undefined';
         ctx.entity.appSyncDemoSphereColor = '#FFFFFF';
         ctx.entity.appSyncDemoIntervalId = setInterval(function() {
             if (ctx.worldData.transformPlayer) {
                 ctx.worldData.transformPlayer(getName(ctx), getColor(ctx), ctx.entity.transformComponent.getTranslation())
             }
         }, 1000);
    
    
         sumerian.SystemBus.addListener('updateSphereName', (value) => updateName(ctx, value));
         sumerian.SystemBus.addListener('updateSphereColor', (value) => updateColor(ctx, value));
     }
    
     function cleanup(args, ctx) {
         window.clearInterval(ctx.entity.appSyncDemoIntervalId);
     }
    
  2. Re-publish your Sumerian scene.

You just built a Sumerian scene that can be interacted with by multiple sessions using AppSync. View your scene using the hosted URL. Make sure to also load the scene in two different browser windows and move each sphere to see the multiplayer interaction. You can change the color and name of your local sphere using reusable React components and others in the sphere can see those changes in near realtime in their scene. Access control to the scene is restricted to just users who have signed up with your service by integrating Amplify and Amazon Cognito.

To learn more about AWS Amplify, check out the docs or the Github repository.

Back to Tutorials

© 2019 Amazon Web Services, Inc or its affiliates. All rights reserved.