WebVR and Spatial Audio - Part II

In this tutorial we will explore building and experimenting with visualizations of 3d audio in virtual reality using Envelop for Live. Here is the the first getting started tutorial where we laid the foundation for using WebVR and E4L together.

Project Setup

Completed Code: Github

Overview:

  1. Import 3D Model Into Three.js
  2. Map Three.js Scene to Blender Layer Objects
  3. Setup E4L with Speaker Decoder
  4. Parse Incoming Gain Level Stream
  5. Recording the 3D Scene
  6. Upload 3D Scene to YouTube / Facebook in 360 Format

Prerequisites:

JavaScript: Advanced
DAW: Intermediate
OS Used for code examples: OSX

Software:

  • Blender
  • Ableton Live 9
  • Max For Live
  • Max 7
  • Text Editor
  • Terminal
  • FFMPEG
  • NodeJS
  • Google Chrome

Hardware:

  • Headphones
  • PC

Import 3D Model Into Three.js

Three.js has an extremely flexible import system that allows you to use complex models from other software platforms such as Blender. The export step from Blender to Three.js is pretty straight forward and can be followed here. Make sure you follow the Blender settings as shown in the screenshots or you’re going to have a bad time. One thing they don’t mention is the geometry section. Change it to be Geometry and not BufferGeometry. You’ll also want to give your Blender layers descriptive names as to recall them later in JavaScript. For mine, I used Speaker01-24 since we are modeling the 24 channel Envelop decoder.

Once you have exported the file from Blender into JSON format, modify the JSON file into a .js file and add module.exports to it. This allows our model to be used with Webpack.

Update EnvelopModel.js file.
module.exports = {
    "object":{
Import Blender 3d model into threeDScene.js.
import * as EnvelopModel from '../EnvelopModel.js';

Instead of loading the cubes into the scene as we were before, lets add our 3D model to the scene.

/app/threeDScene.js

Replace cubes function with model loader setup.
setupSpeakerModel() {
    let loader = new THREE.ObjectLoader();
    loader.parse(EnvelopModel, (blenderModel) => {
        this.scene.add(blenderModel);
    });
}

You should now see the model being loaded in the VR scene instead of the cubes. If you have trouble doing the export from Blender or other 3D software, it’s always best to tinker with the export settings until you find something that works.

Map Three.js Scene to Blender Layer Objects

Next lets map our 3d scene objects into a JavaScript object so that we can easily manipulate their properties. When you import a scene into Three.js from Blender, all of the layers become children of the scene. You can iterate over the children looking for the layer names that you desire and add those objects to your map.

/app/maxToBrowser.js

mapSceneToModel(scene) {
    this.speakers = {};

    scene.children[0].children.forEach((child) => {
        if(child.name.indexOf("Speaker") != -1) {
            this.speakers[child.name] = child;
        }
    });

    //Initialize speaker properties to display transparency.
    for(let speaker in this.speakers) {
        this.speakers[speaker].material.transparent = true;
        this.speakers[speaker].material.opacity = .1;
    }
}

Setup E4L with Speaker Decoder

This time instead of using the positional data from the audio inputs in our MaxToBrowser class, we are going to use the gain levels from the 24 channel decoder. In order to get the gain levels out of the decoder, we need to alter the envelop satellite Max patch. Over in the E4L Facebook group, I was able to get some advice and it was recommended that I copy the osc.streamer patch for this task. The osc.streamer patch takes audio channel information as inlets and sends it over UDP. I copied that patch and modified it ever so slightly to send the UDP data to our server that we created in the first tutorial. The UDP object is sending the information as messages and not in a bundle as in the first tutorial. We could likely optimize this into a better format but we aren’t doing performance tuning this time around. If you have trouble grasping the concepts here, the 2 patches used are in the max folder of the git repo.

E4L decoder patch preview.

E4L decoder

Turn up gain level in UDP Patch.

UDP Patch

You can modify either of these patches to your liking while not inside the Envelop project in Max. I would recommend setting up the OSC messages to be similar enough to your Blender layers as to not cause trouble parsing and passing the data around. If you use one computer for the audio and the other to render the 3d scene, you will get much better performance.

UDPServer.js

Modify UDP server to receive messages instead of bundles.
udpPort.on("message", function (message, timeTag, info) {
    myEmitter.emit('message', message);
});

myEmitter.on('message', function (data) {
    socket.send(JSON.stringify(data));
})

Parse Incoming Gain Level Stream

Now that we have an object representing our Blender elements and the audio level data coming over the WebSocket, lets connect the data to the opacity of the objects. This connection should raise and lower the opacity of the object mapped to the corresponding audio level. The example code below shows 24 gain levels being mapped to 24 spheres, channeling the input via the Blender layer names map.

/app/maxToBrowser.js

socket.on('message', function(data){
    let oscMessage = JSON.parse(data);
    let channel = "Speaker" + oscMessage.address.substr(18, oscMessage.address.length);
    self.speakers[channel].material.opacity = oscMessage.args[0];
});

Test things out and hopefully you should see the audio levels of the decoder manipuatling the opacity of your obects in realtime. This is starting to look pretty cool!

Note: Remember that in order to test, you must re-run webpack to bundle the JavaScript code.

Recording the 3D Scene

We now have 3d elements being directly manipulated by audio levels from E4L. How do we share it with our friends? Well if you want to be one of the new kids on the block, we can create a 360 video that can be uploaded to Facebook or Youtube for viewing in VR. You could also create a 2D video as well but what’s the fun in that?

Many thanks to James Pollack for his awesome writeup on how to record 360 video in your browser using the CubemapToEquirectangular class. This class manipulates the canvas in such a way that we can record 4k videos in the browser, download the output, manipulate the metadata and then upload it to Facebook or YouTube as 360 video.

/vendor/CubemapToEquirectangular.js

Manipulate class to append canvas element. If not appended, MediaRecorder will not work.
 if (provideCubeCamera) {
        this.getCubeCamera(2048)
    }

let cubeDOM = document.body.appendChild(this.canvas);

/app/record.js

Create Record class to gather the output of the new canvas camera in 360 format.
import * as CubemapToEquirectangular from '../vendor/CubemapToEquirectangular';
import * as download from 'downloadjs';

class Record {
    constructor(renderer, camera, scene) {
        this.renderer = renderer;
        this.camera = camera;
        this.scene = scene;
        this.cubeMap = new CubemapToEquirectangular(this.renderer, true);
        setTimeout(() => { this.start(); }, 5000);
    }
    /**
     * Start recording using the MediaRecorder and MediaStream APIs.
     */
    start() {
        let canvasStream = this.cubeMap.canvas.captureStream();

        const options = {
            audioBitsPerSecond: 128000,
            videoBitsPerSecond: 2500000,
            mimeType: 'video/webm'
        };

        this.outputStreamBlobs = [];
        let createMediaStream = window.webkitMediaStream || window.MediaStream;
        let outputStream = new createMediaStream();

        [canvasStream.getVideoTracks()].forEach(function (stream) {
            stream.forEach(function (track) {
                outputStream.addTrack(track);
            });
        });

        this.outputRecorder = new MediaRecorder(outputStream);
        this.outputRecorder.onstop = this.stop.bind(this);
        this.outputRecorder.onerror = function(err) {
            console.log(err);
        }
        this.outputRecorder.ondataavailable = this.data.bind(this);
        this.outputRecorder.start(10); // collect 10ms of data
    }
    /**
     * Add blob into Array 
     * @param  {Event} event
     */
    data(event) {
        if (event.data && event.data.size > 0) {
            this.outputStreamBlobs.push(event.data);
        }
    }
    /**
     * Stop MediaRecorder API and dump in memory chunks to file that will be downloaded.
     */
    stop() {
        this.outputRecorder.stop();

        let superBuffer = new Blob(this.outputStreamBlobs, { type: 'video/webm;codecs=h264' });
        download(superBuffer, 'testME.webm', 'video/webm'); 
    }
}

index.html

Add 2 buttons to start and stop our recording
<button id="startButton">Start Capture</button>
<button id="saveButton" >Stop Capture</button>

/app/threeDScene.js

Add event listeners to start and stop recording
document.getElementById('startButton').addEventListener('click', (event) => {
    this.record = new Record(this.renderer, this.camera, this.scene);
    this.capturing = true;
});

document.getElementById('saveButton').addEventListener('click', (event) => {
    this.capturing = false;
    this.record.stop();
});

Clicking record should now append another canvas element to the body. This is the canvas that the 360 video is made from. You should position your camera before pressing record and not manipulate the scene at all while it is recording besides the incoming audio data stream. When you click stop, the browser should ask you to download a webm file.

Upload 3D Scene to YouTube / Facebook in 360 Format

Lastly, all we have to do is to convert the video file into a format compatible with Facebook and YouTube.

Convert webm video file to mp4 in ffmpeg.
ffmpeg -fflags +genpts -i testme.webm -r 24 360VideoOutput.mp4

After the video has finished converting with FFMPEG, open the Spatial Media Metadata Injector and re-save the file with the spherical video declaration. Then upload to your network of choice!

Spatial Metadata

Thanks for reading. Next time we’ll add audio to the recording and embed the 360 playback to complete this series of tutorials.

=.+.+.=

Divination

16 February 2017