The current state of Augmented Reality and React (Native)

My personal journey through Expo and Viro

I was extremely excited to try out React Native, lots of the features appealed to me. Ability to code javascript on iOS, Live previews on your device and on the iOS simulator. Cross-platform coding from the language we learned in school seemed like a no-brainer so I had to give it a shot. Ooo also, if I’m doing a mobile app, I might as well do it in augmented reality. Wishful thinking, I didn’t realize yet.

Things started off well, I started googling

and found some promising results.

Very cool, this video showed me how to create an Augmented Reality app, connecting to Google Poly Api, which allowed users to search any Augmented Reality object on the platform and see it augmented in their environment.

8pm, night before I present: This was going to be easy…class is going to enjoy watching this demo.

So, I start following along and I download Expo (which is sick btw, if your not trying to do anything in AR)

Expo is a really neat software that offers both a command line environment and online interface to work with React Native. Similar to other React Native API’s. They provide a lot of built in components that you can plug right into your JSX file to make the coding proccess more effiecent. For example, in order to use AR, you can easily add the THREE-EXPO component, which allows you to access three.js from expo, without having to stretch to far.

I code along

and continue watching the video. Things were going well, although the expo platform updates so often, it took a bit of reorganizing to follow the flow of the video. And then we get up to the cool — part accessing the camera. I could not wait to see how this worked in action. So I set it all up.

Get this error — expo no longer supports augmented reality check out www. for more information

Boom!

So I check out their API docs and listed as the first option is ‘AR’, since its the first option im feeling pretty good about it.

9:00pm: I realized AR has been removed from the expo library, so I was no longer able to use it. Close down the video. And started getting very frustrated

Needed to find another solution…behold this message from Andrew Santos

Master Andrew Santos mentioned that I previous student worked in a platform called Viro React. It looked pretty easy to use, so I checked out their documentation.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

homebrew install command

brew install node

node install command

yum install watchman

install watchman

npm install -g react-native-cli

install the react command line interface

npm install -g react-viro-cli

install the Viro Command Line Interface

npm install -g react-viro-cli

And started working off the documentation

react-viro init ViroSample

Downloaded the Viro app

‎Viro Media

Here’s the link to download the app, it allows you to test out your mobile experience as you are developing. Unike typical iPhone apps, AR, requires the use of the phones camera so you can’t use the xCode simulator. So downloading the app here is a crucial step. Everything was off to a great start

10pm: Smooth Sailing

Off the bat everything was working well, the Viro sample project, worked as promised and the next step in the documentation was to add components. This looked pretty easy, since its react it all worked by importing different materials

./HelloWorldSceneAR.js
import {
  ...
    ViroBox,
  ViroMaterials,
} from 'react-viro';

and then I created a box under the ViroText component

<ViroBox position={[0, -.5, -1]} scale={[.3, .3, .1]} materials={["grid"]} />
ViroMaterials.createMaterials({   grid: {     diffuseTexture: require('./res/grid_bg.jpg'),
 }, 
});

and added materials to it.

Adding 3D objects to the scene

Continuing to in the documentation I started adding 3D objects you need to import components Viro3DObject, ViroAmbientLight and ViroSpotLight.

Add the following to your import statement

import {
    ...
  Viro3DObject,
  ViroAmbientLight,
  ViroSpotLight,
} from 'react-viro';

Once your done importing create a new Viro3DObject and import the provided files by viro into your project as the source. Specify the position, scale and type — vrx is a converted 3d object file type

<Viro3DObject source={require('./res/emoji_smile/emoji_smile.vrx')}            position={[-.5, .5, -1]} scale={[.2, .2, .2]} type="VRX" />

Position the object in space, give it a scale and specify the type — note their are a lot of different object types, and they are all handled differently in viro. For more information on 3D objects, view this doc

Create an AR Plane

For it to be AR It needs to be positioned well in space, and Viro offers a great solution. In order to do so import ViroARPlaneSelector into the import statement of your react app. And next add the following code into your ViroARPlaneSelector

<ViroARPlaneSelector />

In order to nest the objects in space into the ViroPlane you just wrap the object into the ViroARPlane Selector. And now your all set with a ‘working’ augmented reality project using react. Now that we have some objects in the scene we can spice it up.

Image Recognition

I finished the documentation and started working on image recognition and tying in

Now the fun ended, both my devices stopped working and after doing some research this problem seems inevitable at least a difficult fix for the time being. So I went to sleep, woke up early this morning, to try to get it to work before the blog presentation this afternoon.

I realized the payoff wasn’t even worth it and I was stressing out over this blog post to much and this framework.

After an hour or so additional time spent trying to get the app to work this morning, I decided it was time to call it and instead write a short story on my short-lived experience with augmented reality and javascript.

So how’s the current state of augmented reality and react — not great.

Thanks