Getting Started with WebVR and Spatial Audio

With new found freedom, I can finally explore some creative endeavors that I have put off. As it so happens, one of those revealed itself in an email that landed in my inbox on 12/31. “Envelop for Live” beta now available it read. I have been waiting for what felt like an eternity for that. After attending several Envelop performances over the years, I always wanted to be involved with the software but didn’t have the time or knowledge. Now I can finally explore this beautiful sound sculpting tool. Spatial audio has started to become more popular recently with the advances in VR and what better way to explore both of them than with a little web project. Lets get started.

Envelop for Live (E4L) is an open source audio production framework for spatial audio composition and performance.

If you’re curious about the details of the speaker installation they have built, you can view the completed Kickstarter page here. For the more tech-savvy, I encourage you to read the documentation and papers provided on the EnvelopForLive Github page.

Project Setup

Prerequisites:

JavaScript: Intermediate
DAW: Intermediate
OS Used for code examples: OSX

Software:

  • Ableton Live 9
  • Max For Live
  • Max 7
  • Text Editor
  • Terminal
  • NodeJS
  • Google Chrome

Hardware:

  • Headphones
  • WebVR capable device (I’m using a Pixel XL) or Emulator

For this project, I’m going to try out some newer JavaScript tooling that I haven’t used before. The first being Webpack as I have always been a Browserify guy. I followed this guide to get started.

The first dependency we’re going to add is ThreeJS. ThreeJS is an abstraction layer on top of WebGL and other web APIs. We’ll use it to give us a head start setting up WebVR.

Note: Still following the Webpack guide here, replacing lodash with three.

npm i -S three

I like to use SimpleHTTPServer when I’m working on small static apps. This starts an HTTP server on port 8000 by default. Pretty nifty little tool that is anywhere you can run python. Or if you’re feeling fancy and want auto reloads, you can try the webpack-dev-server. Only problem here is that neither of them do HTTPS out of the box. This will become an issue for us when testing the WebVR site on device.

python -m SimpleHTTPServer

After some trial and error with a couple ThreeJS examples, I landed my first commit with the initial Webpack setup. I like simplicity and speed when it comes to bundlers and my first impressions of Webpack were very positive! I will likely use it over Browserify moving forward.

WebVR Setup

Now that we have a simple framework to work with, lets update our ThreeJS example file to be a WebVR template. I’m going to follow this CSS Tricks tutorial but augment it in a way that gives us some nicer application structure to work with. There are currently a few other out of the box WebVR solutions, notably Aframe and Primrose. I’m using ThreeJS due to the rich community currently surrounding it.

Let’s walk through setting up our class after trying out the tutorial. First, we need to setup our dependencies and the ThreeJS WebGL environment. I’m not using the WebVR polyfill for this guide but if you want to support other devices like iOS, you can enable it. It may take some additional tweaking on your behalf.

/app/threeDScene.js

Setup the Webpack dependencies.
import * as THREE from 'three';
import '../vendor/VRControls';
import '../vendor/VREffect';

I’m not using the node_modules/three/examples VRControls and VREffects because I needed to modify the exports in order for them to work properly with Webpack. I believe there are ways around this but I couldn’t figure it out off the top.

Create a class container for managing the 3D Scene.
class threeDScene {
    constructor() {
        this.scene = new THREE.Scene();
        this.camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 1, 10000);
        this.renderer = new THREE.WebGLRenderer({ 
            antialias: true 
        });

        this.renderer.setPixelRatio(window.devicePixelRatio);

        document.body.appendChild(this.renderer.domElement);
    }
}
Setup the vr controls in the constructor.
this.controls = new THREE.VRControls(this.camera);
this.controls.standing = true;

this.effect = new THREE.VREffect(this.renderer);
this.effect.setSize(window.innerWidth, window.innerHeight);

this.vrDisplay = {};

navigator.getVRDisplays().then((displays) => {
    if (displays.length > 0) {
        this.vrDisplay = displays[0];
    }
});

document.querySelector('#startVR').addEventListener('click', () => {
    this.vrDisplay.requestPresent([{source: this.renderer.domElement}]);
    this.isShowingVR = true;
});
Setup the window size change events so that our render is always fullscreen in the constructor.
window.addEventListener('vrdisplaypresentchange', this.onResize.bind(this));
window.addEventListener('resize', this.onResize.bind(this));
Add onResize as a class function.
onResize() {
    this.effect.setSize(window.innerWidth, window.innerHeight);
    this.camera.aspect = window.innerWidth / window.innerHeight;
    this.camera.updateProjectionMatrix();
}
Here I took the cubes code from the css tricks guide and modified it to work with a class function.
cubes() {
    this.cubesMesh = [];

    for (let i = 0; i < 100; i++) {

        let material = new THREE.MeshNormalMaterial();
        let geometry = new THREE.BoxGeometry( 50, 50, 50 );
        let mesh = new THREE.Mesh( geometry, material );

        // Give each cube a random position
        mesh.position.x = (Math.random() * 1000) - 500;
        mesh.position.y = (Math.random() * 1000) - 500;
        mesh.position.z = (Math.random() * 1000) - 500;

        this.scene.add(mesh);

        // Store each mesh in array
        this.cubesMesh.push(mesh);
    }
}
And lastly lets setup our animation loop and fire the 2 functions in the constructor.
animate() {
    let cubes = this.cubesMesh;

    //If we don't change the source here, the HMD will not move the camera.
    if(this.isShowingVR) {
        this.vrDisplay.requestAnimationFrame(this.animate.bind(this));    
    }
    else {
        requestAnimationFrame(this.animate.bind(this));    
    }

    // Every frame, rotate the cubes a little bit
    for (let i = 0; i < cubes.length; i++) {
        cubes[i].rotation.x += 0.01;
        cubes[i].rotation.y += 0.02;
    }

    // Render the scene
    this.controls.update();
    this.effect.render(this.scene, this.camera);
}
Last 2 lines of our constructor.
this.cubes();
this.animate();

index.js

Our index.js should be extremely simple at this point.

import * as ThreeDScene from './threeDScene';
let threeDScene = new ThreeDScene();

Compile your code and run the webserver. For mobile testing, you’ll need one of the newer Chrome betas with WebVR enabled. All of that information can be found at webvr.info.

If you don’t have a mobile device that you can point at your dev machine, you can use the WebVR API Emulation chrome extension for Desktop Chrome. I noticed some issues in Chrome 55 but Chrome 57 (Canary build at time of this writing) is working great. When testing cutting edge web APIs, its always best to try out these builds too. You can view my completed code here.

Now that we have conquered our first battle in getting WebVR setup, lets get our audio setup going.

Envelop For Live(E4L) Setup

If you’ve worked with Ableton before, setting up E4L will be a breeze. I recommend starting here with this great introductory video by Christopher Willits.

All of the nitty gritty details can be found on the E4L Github page. You’ll want to follow the guides there and if you run into any issues you can join the E4L Facebook Group to interface with the larger community.

Setting up Max to talk to the browser over WebSockets

Max is an extremely powerful audio/visual tool that I’m slowly starting to wrap my head around. I have a couple musician friends who have been using it for years but it always seemed too difficult out of the box and felt like it had a steep learning curve. I could tell immediately that has changed as soon as I opened Max 7. Max 7 seems to be a revolution in the UI and portability of the Max ecosystem. Having a package manager alone seems huge! I haven’t been following Max closely for that long so its possible this is all just new to me but playing around I immediately feel a sense of accomplishment. Lets take a look at how we can connect to the Max data for E4L.

After downloading Max, I saw a blip for their new product, Miraweb. Of course I thought, oh shiny but that actually took me down the wrong path and created a lot of latency in the project. Big thanks to Rama Gottfried, one of the E4L creators for helping me and supplying the Max patch that we will use to achieve this. The reason Mira didn’t work in my opinion was that we had to decode the OSC message in order to proxy it through the WebSocket. If we didn’t have to decode there and instead decode in the browser, the Mira approach might be more successful.

The solution we landed on after some trial and error was proxying the UDP socket to another port that I could write a server to listen in on and send over a websocket that was separate from Mira. I was able to achieve most of this in a few lines of code thanks to the OSC.js library. E4L typically runs the UDP signal on port 3333. Open the UDPPatch.maxpat provided with this example. Modify the port nubmer to something other than 3333 as node.js will not allow you to listen on the same port. I’ve used 57121 in my example. It’s important to keep in mind we are creating a Node.js server here and that requires a different structure than our Webpack application bundle.

Install OSC.js
npm i osc -S

UDPServer.js

const osc = require('osc');

// Create an osc.js UDP Port listening on port 57121.
var udpPort = new osc.UDPPort({
    localAddress: "0.0.0.0",
    localPort: 57121
});

// Listen for incoming OSC bundles.
udpPort.on("bundle", function (oscBundle, timeTag, info) {
    console.log("An OSC bundle just arrived for time tag", timeTag, ":", oscBundle);
});

// Open the socket.
udpPort.open();
Test out your server.
node UDPServer.js

If everything worked, you should be able to move a coordinate in the Envelop Max for Live plugin and receive a UDP message with the coordinates of the speaker array. If you want information from the other plugins, you can modify the regex like filter in the provided maxpat file. Now how do we get these coordinates into our WebVR instance? Easy, WebSockets. For that we are going to use the great node.js library Engine.io.

Install Engine.io
npm i engine.io -S
npm i engine.io-client -S

I created a new class file called maxToBrowser.js in my app folder for dealing with the OSC data that will be coming over the websocket.

import * as eio from 'engine.io-client';

class maxToBrowser {
    constructor(threeDScene) {
        const socket = new eio.Socket('ws://localhost:1337/');

        socket.on('open', function(){
            socket.on('message', function(data){
                let oscMessage = JSON.parse(data);
                console.log(oscMessage)
            });
        });
    }
}

I created a custom EventEmitter class to help us work with the asychronous flow. I’m well aware new connections will cause memory leaks. We’ll fix that in the next tutorial.

const EventEmitter = require('events');
class MyEmitter extends EventEmitter {}
const myEmitter = new MyEmitter();
Setup Server side Engine.io
const engine = require('engine.io');
const server = engine.listen(1337);

server.on('connection', function(socket){
    myEmitter.on('jungle', function(data) {
        socket.send(JSON.stringify(data));
    })
})
Send OSC data over websocket
myEmitter.emit('jungle', { 
    0: oscBundle.packets[0],
    1: oscBundle.packets[1]
});

Make sure you run Webpack to update your client code and restart the UDPServer. Now when testing with the Envelop Max for Live plugin, you should see the positioning data coming through in the console of Chrome. Getting there!

Using the Max data in our ThreeJS scene

Let’s say we want to modify the coordinate of the spheres that we have added to our VR scene. All we need to do is pass the threeDScene into our maxToBrowser class and then we can access the cubes coordinates.

Update app/index.js
import * as ThreeDScene from './threeDScene';
import * as MaxToBrowser from'./maxToBrowser';

let threeDScene = new ThreeDScene();
let maxToBrowser = new MaxToBrowser(threeDScene);
Update app/maxToBrowser.js
threeDScene.cubesMesh[0].position.x = oscMessage[0].args[0] * 1000;
threeDScene.cubesMesh[0].position.y = oscMessage[0].args[1] * 1000;
threeDScene.cubesMesh[0].position.z = -250;

threeDScene.cubesMesh[1].position.x = oscMessage[1].args[0] * 1000;
threeDScene.cubesMesh[1].position.y = oscMessage[1].args[1] * 1000;
threeDScene.cubesMesh[1].position.z = -250;

If you did everything right, you should end up with something like this. The finished code can be found on Github.

How cool is that? The possibilities for fun are limitless!!!

Next time we’ll take a look at using more inputs, coordinating ThreeJS camera movement with the Envelop system, recording audio with E4L and using Omnitone by Google for the playback. If you have questions or feedback, feel free to reach out on Twitter. Thanks for reading.

=.+.+.=

Divination

Pinegrove - Recycling

09 January 2017