AR/VR Development

Verge3D allows for creating web-based Augmented Reality (AR) and Virtual Reality (VR) experiences running on top of the browser technology called WebXR (eXtended Reality on the Web). In addition to that you can use USDZ format and Apple AR Quick Look technology to preview your models in AR mode on iPhones and iPads.

Contents

Virtual Reality (VR)

Setting Up

HTC VIVE and Meta Quest devices should work with WebXR out of the box. Use Google Chrome browser for PC-connected devices such as HTC VIVE Pro. Use the browser which comes with your headset in case of standalone devices, e.g run your VR apps in Quest Browser on Meta Quest 1-3.

WebXR requires a secure context. Verge3D apps must be served over HTTPS, or from the localhost URL.

Creating VR Apps

The VR mode can be checked with check VR mode and set up using enter VR mode puzzle.

Minimal VR puzzles

Interaction with 3D objects is performed by using the gaze-based reticle pointer automatically provided for VR devices without controllers (such as cardboards).

Gaze-based VR reticle pointer

For VR devices with controllers, interaction is performed by the virtual ray casted from the controllers.

Controller-based VR ray

You can use the standard when hovered or when clicked puzzles to capture user events as well as VR-specific on session event.

How to move in VR mode

Upon entering the VR mode, you will get the special stereo camera which will be attached to a VR headset. It means you can't use any controls or Puzzles to change the camera's position (or orientation). The only way to move the camera in VR is parenting it to some other object (e.g character) and moving this object instead. Check out the camera control object for more info.

In general, you will need to specify the movements by Puzzles or JavaScript, e.g by using a character object and some physics which prevents the character to move beyond the scene bounds (or obstacles). However, if you use the sitting or standing VR positioning mode, there is a Puzzles library called VR Controls which can make the things a lot easier.

For usage example, check out the following demo from the Asset Store: Virtual Reality.

Augmented Reality (AR)

Setting Up

You can run your Verge3D-based augmented reality applications on mobile devices with Android or iOS/iPadOS operating systems.

WebXR requires a secure context. Verge3D apps must be served over HTTPS, or from the localhost URL.

Android

To enable augmented reality, you need an Android device which supports ARCore technology and latest Google Chrome browser. You also need to install Google Play Services for AR. The installation of this package is prompted automatically upon entering AR mode for the first time, if not pre-installed.

Installing Google Play Services for AR

iOS/iPadOS

For iPhones/iPads you have two choices:

  1. install WebXR Viewer browser
  2. preview assets in USDZ format using the conventional Safari browser

Mozilla's WebXR Viewer is a Firefox-based browser application which supports the AR technology on Apple devices (starting from iPhone 6s). This app is available from the Apple App Store. This solution has an advantage, that you can use the same WebXR technology (as well as Puzzles logic) that you use to run AR on Android devices. The disadvantage is obvious — you (as well as your users) should install WebXR Viewer browser manually.

With USDZ assets, you can use the Apple AR Quick Look technology to run simplified augmented reality experience in the default Safari browser. This approach requires you to use completely different rendering engine integrated in iOS, as such you can't make complex materials or use Puzzles logic for your apps running in AR mode.

Creating WebXR-based AR apps (Android, WebXR Viewer)

The AR mode can be set up for any Verge3D app using the enter AR mode puzzle.

Minmal AR puzzles

Upon entering AR mode you will be able to position your 3D content in the "real" coordinate system, which is aligned with your mobile device. In addition to that, you can detect horizontal surfaces (tables, shelves, floor etc) by using the detect horizontal surface AR puzzle.

Also, to see the real environment through your 3D canvas, you should enable the transparent background option in the configure application puzzle.

Enabling background transparency in puzzles

For usage example, check out the following demo from the Asset Store: Augmented Reality.

Creating USDZ-based AR apps

To enter AR Quick Look mode you need to create USDZ asset first. To do this, you can export your scene (or a single model) in runtime (on the fly) or use your favorite modelling suite to make it ahead of time.

Preparing USDZ asset in runtime

This approach is based on using export to USDZ puzzle to generate USDZ (see Puzzles snippet below).

Preparing USDZ asset in modelling suites

USDZ export supported natively in Blender (starting from version 3.5), 3ds Max (requires USD extension to be installed), and Maya.

USDZ is different from USD, being a container which combines USD models with media data, such as PNG, JPEG, MP3. Thus, modelling suites which able to export to USD not necessary support USDZ.

If you use macOS/iOS/iPadOS, there are plenty of tools available on Apple website to convert your models stored in OBJ/FBX/glTF/USD formats to USDZ.

Loading USDZ

To preview USDZ asset, exported by the modelling suite or some other tool, you need to create an HTML link with the image inside it:

<a id="enter_AR_button" rel="ar" href="model.usdz"> <img src="media/enter_AR_button.png"> </a>

The link should reference a USDZ model asset, e.g model.usdz. The image should reference picture which shows some button with icon or text, e.g "Enter AR". In the example above, the image located at media/enter_AR_button.png inside the application directory.

When the user clicks on such link on Apple device, the system will detect USDZ asset referenced by the link and open an AR Quick Look mode so that you can position the model in the real world:

Apple AR Quick Look

With Puzzles, the same link can be created on demand:

Creating USDZ link using puzzles

To generate and load USDZ assets in runtime use the following code:

Creating USDZ asset in runtime using puzzles

Check out the differences — we used export to USDZ puzzle here and assigned additional download attribute to the link.

For usage example, check out the following demo from the Asset Store: Augmented Reality.

How to move in AR mode

Upon entering the AR mode, you will get the special camera which will be attached to the phone (or the AR headset). It means you can't use any controls or Puzzles to change the camera's position (or orientation). The only way to move the camera in AR is parenting it to some other object and moving this object instead. Check out the camera control object for more info.

Chrome Debugging

For faster iterations during development of WebXR-based apps (but not USDZ-based), you can use Chrome's port forwarding feature to run your AR or VR app in mobile browser. This will save you from re-uploading it to a remote web server each time.

What's Next

To find out how to script your AR/VR app with Puzzles, read the following section. Also, check out the Physics Guide if you're going to simulate a realistic virtual environment with colliding objects and/or character.

For usage example, check out the following demo from the Asset Store: Snowballs VR.

Got Questions?

Feel free to ask on the forums!