VR/AR prototyping for everyone

Published in

Inborn Experience (UX in AR/VR)

15 min readNov 27, 2017

Hello again, guys! Today I want to speak to you about modern VR/AR prototyping. We will talk about Framer.js, A-Frame and AR.js particularly. This article is mostly for those, who are familiar with Framer.js but not so familiar with VR and AR.

Not so long ago I made a Framer prototype which included Sketchfab models and showed how you could control and adjust a model.

And I think that it’s a good approach if we are talking about simple actions and models. But what if we need to achieve something more complicated. To do that, we will need to use something more sophisticated like three.js for instance. But! It’s too hard to set up it and adjust, depends on the task of course.

No. We will use A-Frame. It’s a really awesome 3d-framework supported by Mozilla.

First things first. Let’s start with a small bite of VR history. I know that it could be a long journey before we get practical advice, so if you don’t want to read the theory just use the table of contents below 😉

VR history
A-Frame
Prototyping VR/AR
AR.js and A-Frame
Framer and AR.js

How many realities do we have now?

At the moment I could separate them into four different ones:

They all different but basically about the same.

What is virtual reality and what is behind it?

It’s a very interesting question if we take origins of the word — the etymology. Late 14c., “influencing by physical virtues or capabilities, effective with respect to inherent natural qualities,” from Medieval Latin virtualis, from Latin virtus “excellence, potency, efficacy,” literally “manliness, manhood” (see virtue).

In Nowadays the meaning is “being something in essence or effect, though not actually or in fact”.

Virtual reality has a very long history, that didn’t start with Oculus Rift. Humankind dreamed about virtual reality a long time ago. And these dreams started to turn into reality in 1838 with the invention of the stereoscope.

1838–1935

The first careful steps towards virtual reality. Of course, in those days, people didn’t have words as “VR” or “AR”. But they already had this understanding that they could deceive the mind.

In 1935 Stanley G. Weinbaum described VR glasses in “Pygmalion’s Spectacles” novel

1956 Sensorama

Later in 1956, Morton Heilig made the Sensorama — reality simulator — which imitated wind, stereo picture, smell etc. optional.

It looks and feels very awkward, but it’s another step forward. Some sort of cheap attractions like CINEMA 99D in amusement parks.

17D? Are you serious? How about 999D? No problem, mate 🤗

Like in any industry, marketing will try to sell you something you don’t really need or will try to exaggerate and describe the product better than it is.

1968 sword of Damocles

It’s widely considered to be the first virtual reality (VR) head-mounted display (HMD) system. It was created in 1968 by computer scientist Ivan Sutherland with the help of his student Bob Sproull. Sword of Damocles could reach very simple vertex 3d models but it was a real 3d graphics, not a 2d which pretended to be a 3d. But it was so heavy and due to it has a lot of similarities to the sword of Damocles, it got that name — the Sword of Damocles.

1978 Aspen Interactive Movie Map

Or the old fashioned Google maps. Another experiment which I could say also was related to VR. The Aspen Movie Map was a revolutionary hypermedia system developed at MIT by a team working with Andrew Lippman in 1978 with funding from ARPA. Basically, it was a stop-motion tour through shooted step-by-step pictures, a user was capable to choose a drive direction. But the map also included a 3d map of Aspen.

1987 Data glove and eye phone

Then from “Atari Research” department came the new team “VPL research” (Visual Programming Languages) who was one of the first companies that developed and sold virtual reality products. They developed the Data Glove, the EyePhone and the Data Suit.

And then the 90’s came

A long time virtual reality was considered only in terms of academic knowledge, profit for science and medicine, not for a broad mass. But the picture is started to change when 90’s came due to they came with a completely different and strong game market.

There was a lot of marketing scam products among these also products from such a big corporations as Sega and Nintendo.

But there was also a very first step towards to a customer like Virtuality. Of course, it developed not as a home console it was a big attraction with HMD, data gloves, special chair or cockpit — gaming machine. We could say that it was a solid VR with sensors and sometimes with a legit and true 3d graphics.

Market’s Rat Racing

Like I mentioned above VR came into the market but greedy corporations and their marketing divisions spoil it. Obviously, that hardware and software weren’t prepared for the immersive VR experience at your home.

The result of the Market’s Rat Racing was a disappointment in VR. Actually, a lot of the concepts that was prepared for the serial manufacturing was never released.

Look closer at Sega VR, Nintendo VirtualBoy and Atari Jaguar VR.

While game companies tried to sale their “VR” products Silicon Graphics Inc., (SGI) started developing OpenGL in 1991 and released it in January 1992. Open Graphics Library (OpenGL) is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. It’s a very important point not only because game developers now could write a code without worrying about drivers compatibility, also because it’s open-source (it also everywhere now — power of open-source projects) and without OpenGL we would not have WebGL decade later.

Keep calm and do VR

There were good exceptions, still not so good in terms of tactical and technical characteristics but we are getting closer to the actual VR.

Period from 1995 to 2011 was an HMDs competition. Most distinctive were VFX1 Headgear in 1995. This HMD had sensors, speakers, cyber puck joystick and simple installation.

VFX1 could cost you $695, characteristics 263 x 230 LCD capable of 256 colors.

Also, just watch the commercial. It’s always funny how actors overreacting in these commercials.

Next device what I want to mention is SEOS 120/40 HMD. Unfortunately, I didn’t find any video, but you could read this revivew. Which is more interesting is the price and characteristics: $70,000 – 1280 x 1024 LCD 120 degree view.

During the 2000s appeared a lot of different HMDs. You could just look at this table — http://sed.inrialpes.fr/ARCHIVE/PRV/CATALOG/hmd.html and another one http://3dj.at/data/vr/hmd/list/hmdlist.html

Well, enough with HMDs experiments.

2006 WebGL

WebGL evolved out of the Canvas 3D experiments started by Vladimir Vukićević at Mozilla. Vukićević first demonstrated a Canvas 3D prototype in 2006. By the end of 2007, both Mozilla and Opera had made their own separate implementations.

In early 2009, the non-profit technology consortium Khronos Group started the WebGL Working Group, with initial participation from Apple, Google, Mozilla, Opera, and others.

So now we have a union library for 3d graphics on the web. Thanks to Mozilla, it’s also interesting because later we will talk about A-Frame — comprehensive 3d framework supported by Mozilla.

Let’s take a little break just to mention Palmer… lucky… who came up on KickStarter with his VR headset Oculus Rift DK1. And this is also an essential step because it gave a new impulse to the whole VR. People again start to believe in the bright and shiny VR future.

Mobile VR era begins

In 2014 Google designed Google Cardboard. It basically a folded piece of carton but if you putt your phone inside you could achieve no jokes VR (depends on your phone hardware of course). Only for 5 buсks — VR for everyone indeed. Of course, Google Cardboard has limitations comparing to fancy and expensive VR headsets, but it’s in your pocket, mobile and everyone can afford it.

In 2014 we already had Three.js and some other JS-libraries working with 3d. But in 2015 A-Frame appears.

2015 A-Frame

A-Frame is an open-source web framework for building VR experiences. It is primarily maintained by Mozilla and the WebVR community. It is an entity component system framework based on Three.js where developers can create 3D and WebVR scenes using HTML.

Why A-Frame is so cool?

A-frame has visual editor https://aframe.io/aframe-inspector/example

A-frame has a really cool and easy markup language. You don’t need to write a long and complicated setup. Instead, you could just write a scene as if you write an HTML page.

You also have VR-mode button in the right-bottom corner

A-Frame has a lot of ready to use components — sound, layout, images, AR and much more https://aframe.io/aframe-registry
Due to A-Frame based on Three.js you could use it as well in much comprehensive way, expanding the capabilities of the A-Frame.
Good documentation and strong community https://aframe.io/docs/0.7.0/introduction Also A-Frame weeks https://aframe.io/blog

Also In 2016 Mozilla and Google announced the version 1.0 release of the WebVR API https://hacks.mozilla.org/2016/03/introducing-the-webvr-1-0-api-proposal

Prototyping VR/AR

With web-VR we have also mobile VR, VR experience everywhere. With such tools as A-Frame we have the opportunity to create VR experience very easily. Even by designers who barely familiar with code.

I made my own example of the simple VR scene. Because I was preparing my speech for Framer meetup all examples will be around Framer. I used several A-Frame components such as:

A designer has a lot of prototyping tools in nowadays, but most of them are “closed”. And we can’t do something new and fresh before developers will bring us a new version with new features. We have just a few exceptions now— Sketch app because it allows writing additional plugins and Framer Studio because it’s JS-based which means you could do anything you could imagine.

About Sketch app plugin — https://blog.prototypr.io/sketch-plugin-sketch-to-vr-4e23ced47e6 and what a surprise it using A-Frame 😋

Also, Marvelapp has several cool articles about designing for VR — https://blog.marvelapp.com/vr-prototyping-marvel

And Framer. Framer.js it’s a sophisticated tool, basically JS frmawork for designers with already prepared awesome functions for beautiful animations and flows. That is why I think that integrate VR experience into Framer without using Framer functions it’s a sin. We can’t use A-Frame for VR in Framer, it will be absolutely pointless because Framer works with HTML DOM, A-Frame just translate HTML markup into the <canvas>. But we have the cool module for VR https://blog.framer.com/comprehensive-guide-to-prototyping-vr-in-framer-29b7dd1bdd2a from jonastreub

But basically, at the moment, it’s a 360 degrees photo without stereoscopic effect. You can’t use your prototype with cardboard but all Framer animations and functions are available.

But, A-Frame has a very cool add-on AR.js https://aframe.io/blog/arjs

This add-on works with A-Frame and all components including. And about month ago I start to think how we could achieve augmented reality with Framer.js and A-Frame because in this case we don’t need stereoscopic effect and we could use all that Framer.js could offer for us.

AR.js and A-Frame

“Finally!” think impatient reader. Yep. Let’s dig into AR. I’ll show you how to achieve AR with A-Frame and later with Framer.

Augmented reality it’s when you bring something virtual in reality — 3d models in our case. And this add-on makes it easy with A-Frame https://github.com/jeromeetienne/ar.js

How easy is it? Only 10 lines of code — https://medium.com/arjs/augmented-reality-in-10-lines-of-html-4e193ea9fdbf

To achieve AR we will need:

Hiro-marker. It’s a special mark for a camera. By this marker, AR.js will know how it should project a scene. AR.js has a few templates for this marker, but you can create your own marker. Marker not necessary should be printed. Read more here https://medium.com/arjs/how-to-create-your-own-marker-44becbec1105

A-Frame HTML markup. Add AR.js to A-Frame

I tested the library on Codepen, just to make sure that it works properly. To test the library I made two simple models, dioramas of London and Kyiv Framer communities. But of course you can use and build a test scene with standard a-frame primitives.

Next we need to include our models inside <a-marker> tag. And… that’s all😛

But, just to make sure that we could change models attributes I also made three buttons —rotate models, zoom models and switch models.

On the image below I marked the structure of the our AR-page

Also, not any server is good for storing assets. I used https://glitch.com definitely recommended

And the JS part. I will us jQuery but it’s not necessary, you also could use document.querySelector()

We will have three scale states indicated by the flag var

Switch button. We just hide one model and show another. Indicated by the flag

And the final result

I think that it’s really cool. But what if we need prototype a mobile app? Or if you need something more complex than three buttons? With just a simple HTML and JS even with jQuery it could be difficult to achieve because you will need to write controls and animations for them from the scratch. Much better to use for this purpose already exist framework which includes all of this. I talking about Framer of course.

Framer and AR.js

There are several ways to include A-Frame into Framer. We could include <script> tag into index.html with AR.js scr from the Framer root folder and write HTML inside Framer app, but it’s a bad way because you will need to take all your A-Framer code under quotes. You will have to struggle with dependencies and asynchrony; and you will lose all syntax highlight for A-Frame markup — No thank you, also you disrupt the integrity of a Framer project.

The best way here will be an i-frame. HTML inside HTML. I found a lot of benefits here

You will have a separate—A-Frame markup and your Framer project — convenient to edit.
You will have syntax highlight.
No worries about JS libraries dependencies.
Solid AR group with markup, canvas and videolayer

A-Frame to Framer through i-frame

The structure of the project inside will be looks like that:

Let me explain what is happening here. We have Framer project folder (root folder) and inside the folder we have a lot of Framer’s stuff like index.html, app.coffee, framer, images and modules folders. Inside the root folder, we create a new folder “A-Frame” or “aframe” or something like that, it doesn’t matter, files inside that’s what’s important. Inside “aframe” folder we will store iframe.html which will contain links to A-Frame.js and AR.js and contain A-Frame murkup. In addition inside “aframe” folder we could include “models” folder which will store models and textures, but in the example below I will use my old github links to the model and texture.

Now let’s build a prototype. We will make a simple prototype where by side swipes we will rotate and scale a model. It’s only a demonstration, you could do with a model not only these actions, actually almost anything.

That is what will do. Later I also will make a record from my old iPhone 6 to test the prototype on the real device.

Design

I wanted to bring immersive experience, that’s why I decided to remove all visible buttons from the view. Instead, we will use swipes on the left side — rotate, on the right side — scale. Simple and clear, but of course we will need, also to show a user how to use it, teach him. Hopefully, any types of events like swipe or drag very simple to achieve with Framer.

I made my design in the design mode in Framer.

So far we have four screen— “Please rotate your device”, hidden side controls, “Use Hiro-marker”, coach-screen “Swipe to close”.

Adding an A-Frame i-frame in Framer

First thing what we will do it’s clear the console every time you reboot your prototype. Also, let’s rotate our device and write listener for orientation

Next we create “ARlayer” Framer layer. Inside the layer, we put a HTML code which is simple <iframe> with “src” which is path to our iframe.html. Also it’s very important to keep layer always on the back due to videolayer should be on the background, and UI on the top, that is why we also write ARlayer.sendToBack()

Our layers structure will look like that:

And the last part here. Create an iframe var, get our iframe by the tag name. Also, let’s create a var for the model “londonModel”, to control the model; now it’s “undefined” because our iframe didn’t load yet.

To get iframe content properly we should wait till iframe will be loaded. And only after iframe content will be loaded we could grab it and also rewrite “londonModel” var.

Now we are able to control and pass new parameters, change current.

Part with functions for controllers also is not too big

How to test it

Important issue here is how to upload a prototype and how to test it.

Recommended browser on desktop — Chrome.
If you test it on iPhone, make sure that you have iOS11 because of webRTC.
Browser on iPhone — only Safari. Because only Safari has access to media sources or webRTC. Other browsers don’t support it as well and Framer preview app.
Localhost from Framer doesn’t work, unfortunately. Instead install Prepros it has all you need — appropriate localhost, livereload and livepreview in browser. Just drag-n-drop your project in the app. But but make sure that automatic compilation for .coffee and .js files are muted.