Stereolabs ZED - Unity plugin

This package brings the many mixed reality features of the ZED and ZED Mini depth sensors into Unity. It comes with prefabs to make adding advanced features to your project as simple as drag-and-drop, helper scripts to simplify custom integration, and numerous examples to see the ZED/ZED Mini in action and learn how it works.

Features include:

Pass-through augmented reality with an Oculus Rift, HTC Vive and other headsets
Third person mixed reality
Real-time depth occlusion between the real and virtual
Adding virtual lights and shadows to real objects
Inside-out positional tracking
Spatial mapping
Real-time plane detection

Getting Started

Step 1: Check Compatibility

Make sure you have the following:

Hardware:

ZED or ZED Mini depth camera (ZED Mini preferred for AR pass-through)
NVIDIA GPU with CUDA capability 3.0 or higher (a GTX 1060 or higher is recommended for AR pass-through)
Dual core processor clocked at 2.3GHz or higher
4GB RAM or more
USB 3.0
Compatible VR headset (required for pass-through AR)

Supported VR headsets:

Oculus Rift
HTC Vive
HTC Vive Pro*
Windows Mixed Reality headset* (via SteamVR only)

*Stereolabs' ZED Mini mounting bracket was designed for the Oculus Rift and original HTC Vive. It can fit on some WMR headsets like the Samsung Odyssey, but you will need to create a custom attachment for the Vive Pro or other WMR headsets.

Software:

Windows 7-10
NVIDIA CUDA† (download)
Unity 2017.1 or above (download)
ZED SDK (download)
ZED Unity plugin (download)

† ZED SDK v2.7 requires CUDA 10. Future SDK versions will require future versions of CUDA that may not be backwards-compatible.

Step 2: Installation

Install the required version of NVIDIA CUDA and restart your computer. If you have existing versions of CUDA installed that aren't compatible with the current ZED SDK, uninstall them first.
Install the ZED SDK and restart your computer.

Step 3: Import the Plugin

Open Unity and create a new project.
Go to Assets -> Import Package -> Custom Package. Navigate to where you downloaded the ZED Unity Plugin and import it.
Leave all files checked and click Import. (You can save space by unchecking the Examples folder on import, but we recommend the examples for first-time users.)
If you're prompted to upgrade your scripts, click "I made a backup, go ahead!" (We won't tell)
If you're prompted to change some settings, click "Accept All".

Example Scenes

After importing the plugin, pick and try some of the example scenes. Each is designed to demonstrate a main feature of the ZED. Some contain prefabs and example scripts that can be repurposed for your projects.

Planetarium

A beautiful display of the ZED plugin's basic mixed reality features, viewable with or without a headset. Watch as the planets are properly occluded by the real world. Move your hand under the sun and see its warm glow light up your skin. Move, rotate and scale the planets with the keyboard, or with your VR controller if using the Oculus Integration or SteamVR plugins.

Movie Screen

Similar to the Planetarium sample, but now playing a 2D movie on a movable, scalable 3D screen. Demonstrates how 2D content can easily be displayed in a 3D, mixed reality scene.

Drone Battle

Defend yourself from attack drones that spawn around your room and shoot at you. Block lasers with your hand, and shoot back with the spacebar, or VR controllers if using the Oculus Integration or SteamVR plugin. The scripts show how to make use of ZEDSupportFunctions.cs to spawn objects in empty areas (DroneSpawner.cs) and shoot projectiles that can hit the real or virtual world (Projectile.cs).

Dark Room

Your office is now a night club! This scene darkens the real world, then casts a laser light show on your walls and floors, complete with tunes and a disco ball. Shows the usefulness of ZEDManager's Camera Brightness feature, and creative ways to use ZEDLight.cs.

Simple Plane Detection

Run the scene and hold down spacebar to see if you're looking at a valid surface where a bunny could stand. Release the spacebar and a bunny will fall from the sky and land on that surface with proper physics.

VR Only Plane Detection (requires VR HMD and Oculus/SteamVR plugin)

Aim your controller at the floor and pull the trigger to place a bunny. Then, a baseball bat will appear in your hand. Smack the bunny and send it flying as far as it can go - you'll see the distance once it lands. Shows how plane detection can fit into a proper game, and how to use ZEDSupportFunctions's HitTestOnRay to check for planes using world space.

Greenscreen

Aim your ZED at a greenscreen and hit Play to see your subject standing in a small town in the desert. You'll see that the nearby crates still have all the proper occlusion, but the greenscreen background is replaced with the virtual background. Attach the ZED to a controller or Vive Tracker and specify it in ZEDControllerTracker to move the ZED around in your tracked VR space.

Creating Your First Project

Make a new scene and delete the main camera.
In the Project tab, go to Assets -> ZED -> Prefabs.
If you plan on doing pass-through AR, drag and drop ZED_Rig_Stereo into the scene. Otherwise, use ZED_Rig_Mono instead to save performance.

Right-click the Hierarchy and click 3D Object -> Sphere to put a basic sphere into the scene. Position it about a meter in front of the ZED camera rig.
Select the Directional Light in your scene. Add a ZED Light component.
Set the Directional Light to point straight down by setting its X Rotation to 90 in its Transform component.
Run the scene. You'll see the sphere floating in the real world, and casting a shadown below it.

Adding AR Passthrough

This assume you have a VR headset (Oculus Rift, HTC Vive, HTC Vive Pro or Windows Mixed Reality headset) with a ZED Mini mounted on front.

If you used ZED_Rig_Mono in the last scene, replace it with ZED_Rig_Stereo.
Go to Edit -> Project Settings -> Player.
Enable Virtual Reality Supported. In Unity 2017.1 or 2017.2, this is in the Other Settings section. In 2017.3 and above, it's in XR Settings.
Run the scene and put on the headset.

If using a WMR Headset:

The ZED Unity plugin supports WMR headsets via SteamVR only, which requires different settings than described in Unity's WMR Quick Start Guide. If you have trouble getting it to run, make sure that:

Your platform (in Build settings) is set to PC, Mac & Linux Standalone (Not Universal Windows Platform)
OpenVR is listed in your Virtual Reality SDKs list, ideally at the top of the list
Windows Mixed Reality for SteamVR is installed on your PC

Understanding ZED_Rig_Mono and ZED_Rig_Stereo

The ZED Rig prefabs replace the regular Camera in a Unity scene, and handle everything the user sees, both real and virtual. It captures the virtual world while also projecting the real world in 3D so that the final output will be a combination of the two with proper occlusion and shadows. When using the ZED or ZED Mini in a scene, you will almost always want one of these two prefabs in it.

The two rigs differ slightly in hierarchy but have three main parts:

Root object: Holds the ZEDManager component, allowing you to change numerous camera settings (listed in the next section) via the Inspector. When not in AR pass-through mode, this object is the one that moves when Tracking is enabled.
Camera object(s): Holds a Unity Camera, and the ZEDRenderingPlane component, which handles how the Camera renders. It also sets up the Frame object with the proper transform, materials and material settings. In AR pass-through mode, this is what moves around.
Frame object(s): Quad(s) positioned in front of the camera, textured with video from the real-life ZED. Rendered with a material that offsets the depth of each pixel based on its depth as calculated by the ZED SDK, allowing real-time occlusion and lighting effects.

Which prefab to use depends on whether you are doing AR pass-through. If so, choose ZED_Rig_Stereo. If not, choose ZED_Rig_Mono. (ZED_Rig_Stereo will work as well but its second, unused Unity camera will cost performance.) Always delete the default Unity camera in a scene before dropping in one of the ZED rig prefabs.

ZED_Rig_Mono

The mono rig is perfect for third-person mixed reality, or any kind of augmented reality where a headset isn't used. It has a single Unity camera that outputs the final video stream to the main display like any other. The "real" part of the video feed comes from the ZED's left camera.

ZED_Rig_Stereo

The stereo rig is necessary for AR pass-through, but will have the same behavior as the mono rig if no headset is being used. It has two Unity cameras and two Frame objects - one for each eye. The two cameras are a child object of an empty "Camera_eyes" object, that represents the center of the user's head.

With VR enabled in the Player settings, you can just drag the camera into the scene and run it. When a headset is detected, the prefab will position Camera_eyes using Unity's built-in VR/XR functions. As such, you do not need any other plugins to use pass-through AR unless you want to add controller support.

Key Scripts

ZEDManager.cs

ZEDManager is the central script of the plugin, and the primary way a user like yourself interacts with the ZED within Unity. It sets up and closes the connection to the ZED, adjusts parameters based on user settings, enables/disables/handles features like tracking, and holds numerous properties, methods and callbacks a developer can find useful. ZEDManager is attached to the root object of both ZED rig prefabs.

Input:

Camera ID: You can connect up to four ZED or ZED Mini cameras to a single PC at once. This value decides which camera you'll connect to. Note that the order of cameras is defined by the order in which Windows recognizes the devices.
Depth Mode: The accuracy of depth calculations. Higher settings mean more accurate occlusion and lighting but costs performance. Note there's a significant jump in performance cost between QUALITY and ULTRA modes.
Input Type: The ZED SDK can take input from one of three methods:
- USB: Input from a live camera attached to your ZED. You can specify the resolution and FPS of that camera. 720p and 60FPS is the best balance for AR pass-through.
- SVO: Load a recorded SVO file taken during a previous session with a live ZED. This file acts as if a live ZED were attached. Specify the path to the .SVO file, whether to loop it once finished, and whether to play each frame based on its time stamps or sequentially.
- Stream: Input from a ZED on a remote device that's actively streaming its camera input. Set the IP and Port to connect to. See the Streaming section of ZEDManager to broadcast a stream.

Motion Tracking:

Enable Tracking: When enabled, the ZED will move/rotate itself using its own inside-out tracking, regardless of whether an HMD is attached. In AR pass-through mode, it will combine its tracking with the HMDs and allow our timewarp feature for latency compensation.
Enable Spatial Memory: When this and Tracking are enabled, the ZED will remember points in the scene to relocalize itself when there is tracking drift. Tracking is more accurate but may "jump" when it relocalizes.
Path Spatial Memory: If using Spatial Memory, you can specify a path to an existing .area file to start with some memory already loaded. .area files are created when using Spatial Mapping (see ZEDSpatialMapping.cs).

Rendering:

Depth Occlusion: When enabled, virtual pixels can be covered up by real pixels, allowing a virtual cube to be behind your real table, for example. Turn off to make virtual objects always appear over the real world.
AR Post-Processing: Whether to apply additional post-processing effects to make the pass-through experience feel more realistic. Requires extra performance but is usually worth it.
Camera Brightness: Lower to make the real world darker. Note that this doesn't change the actual input from the camera, but the final material, meaning that re-lighting the real world will make it brighter realistically (see the Dark Room sample).

Spatial Mapping

Note: In plugin version 2.7.1 and earlier, the Spatial Mapping interface was in a standalone class called ZEDSpatialMappingManager.

Lets you scan your environment into a mesh. Useful for collisions where geometry must be persistent, for building navmeshes that AI can use to navigate, and for saving the meshes for later. It is also how you create a .area file used by the Spatial Memory feature of ZEDManager.

During runtime, click Start Spatial Mapping to start the scan. Once you've scanned enough, hit "Stop Spatial Mapping" to finish the mesh (and save it if needed).

Resolution governs the detail of the mesh. Medium is recommended not only for performance, but because it tends to average out minor depth errors.
Range governs how far a surface can be and still get scanned. Medium is recommended not only for performance, but to ensure accuracy.
Mesh Filtering governs how much the mesh is smoothed when finished. Low is usually best, but increasing it can sometimes help with some tough-to-scan surfaces.
Texturing saves the surface texture in addition to the geometry. Note that this will add significantly to the time it takes to finalize the mesh.
Save Mesh causes the mesh to get saved as an .obj, .bin, or .ply file to the finalized path. It also saves the .area file used for spatial memory if both Enable Tracking and Spatial Memory is enabled on the ZEDManager.

Read more here, or try the included Spatial Mapping sample scene.

Recording:

At runtime, you can record an .SVO video file of the ZED's input, to be played back later and used as if it were input from a live ZED.

SVO File: Where the recorded file will be saved. Specify a filename that ends in .svo.
SVO Compression: Which method of compression to be used. AVCHD_BASED achieves small file size and is generally recommended.
Start/Stop Recording: Press this at runtime to start recording to the specified file.

Note: In plugins v2.7.1 and prior, this function was handled by the now-deprecated ZEDSVOManager component.

Streaming:

You can broadcast your ZED's video so that other devices can use it as input.

Enable Streaming Output: Check this to enable streaming. It will automatically start when you run the app.
Codec: The compression used to encode the output video.
Port: The port on which to broadcast the stream.
Bitrate: How much information to send at once. Lower settings results in lower video quality but is easier on the network.
GOP: Maximum GOP size for the codec. Setting this to -1 removes the limit.
Adaptative Bitrate: Enable to automatically increase and decrease the bitrate based on performance.

Advanced Settings:

Fade In At Start: Disable to remove the fade-in effect you see when the ZED is first connected.
Grey Out Skybox on Start: Removes color and color emissions from the skybox, which can cause unrealistic lighting effects in an AR scene. Leave this checked when the real environment is more prominent than the virtual environment, such as pass-through AR. Turn it off when virtual elements are predominant, such as greenscreen VR capture.
Don't Destroy On Load: Enable to set the ZED rig's DontDestroyOnLoad value, which will prevent its destruction when you change scenes.
Show Final AR Rig: In pass-through AR mode, the plugin uses a second, hidden AR rig to make final adjustments to the image before it's sent to the headset. It's hidden by default because it can be quite confusing to see, but this setting lets advanced users observe and modify this rig. See ZEDMixedRealityPlugin below for more details.
AR Layer: The second AR rig needs to see nothing but the canvases in front of it. To accomplish this while making it simple to understand for users, we assign the quad objects in the AR rig to the layer specified here, and it's the only layer that the cameras in that rig can see. Assign this to an unused layer, and make sure not to put other objects in it.

Useful public members of ZEDManager.cs include:

public event OnZEDManagerReady OnZEDReady: A callback that fires after the ZED is finished initializing. Useful for having a script wait until the user can see the world before doing something.
public static ZEDManager GetInstance(sl.ZED_CAMERA_ID): Returns the ZEDManager instance in the scene that's connected to the specified camera ID (1 - 4).
public static List<ZEDManager> GetInstances(): Returns all active ZEDManager instances in the scene.
public Transform GetLeftCamera(): Returns the the left camera component in the rig. There is also a right-camera counterpart, but it's best to use this method as both ZED rig prefabs (ZED_Rig_Mono and ZED_Rig_Stereo) use the left camera.

ZEDRenderingPlane.cs

ZEDRenderingPlane configures and updates the Frame objects in the camera rig that display the real world, and is responsible for actually mixing the real and virtual images. You usually don't need to interact with it, but as it's central to the plugin's functionality, it can be useful to know its role. The specifics of what it does are lengthy and complex, but some of its responsibilities include:

Scaling and positioning the Frame/Canvas objects properly
Creating materials assigned to the Frames from images/depth from the ZED camera and properties from ZEDManager
Applying post-processing
Updating the images, depth and light information in real time

ZEDCamera.cs

ZEDCamera is the main interface between Unity and the ZED SDK and primarily consists of extern calls to our native Unity wrapper. It's based off the Camera.cpp class in our C++ SDK. It's not a monobehaviour and can't be attached to a GameObject. It is mostly handled by ZEDManager and a developer usually doesn't need to interact with it. However, it has some functions that can be useful to advanced users:

public int ImageWidth/ImageHeight: Use to get the resolution of the ZED image.
public Matrix4x4 Projection: Projection matrix that represents the real camera's own projection matrix. ZEDRenderingPlane applies it to Unity cameras to make them match the ZED's input, and you can do the same to make additional mirrored cameras.
public Vector3 GetGravityEstimate(): If using a ZED Mini, uses its IMU to estimate which way is down.
public sl.ERROR_CODE RetrieveImage((sl.ZEDMat mat, sl.VIEW view, sl.ZEDMat.MEM mem, sl.Resolution resolution): Advanced. Creates a ZEDMat instance with an "image" version of the specified view, intended to be user-viewable. Passing VIEW.DEPTH, for example, will return a grayscale depth map that lacks precision but is easily viewable. You will need to use the ZEDMat's IntPtr and some conversion to get a Unity-friendly Texture2D from this.
public sl.ERROR_CODE RetrieveMeasure(sl.ZEDMat mat, sl.MEASURE measure, sl.ZEDMat.MEM mem, sl.Resolution resolution): Advanced. Similar to RetrieveImage, but with measured values instead of an imprecise, user-viewable version. For example, passing VIEW.DEPTH will return an array of floats for the depth to each pixel in view. Note that it's often simpler to use the static functions in ZEDSupportFunctions.cs to retrieve individual depth values.

ZEDMixedRealityPlugin.cs

ZEDMixedRealityPlugin.cs is used when in AR pass-through mode. It creates and updates a second camera rig (hidden by default) that adds our video timewarp latency compensation to the final image the camera sees. ZEDManager creates it automatically and there's rarely a reason to change it, but you may occasionally want to access it to understand the final output or add custom effects.

Like the ZED rig, it consists of two cameras and two quads. The quads hold the output of the cameras in the rig. But unlike in the rig, the quads aren't children of the cameras. When you turn your head, instead of the quads being "locked" to the camera's rotation, the script has them trail behind according to the ZED's latency. The end result is that the position of objects you see in the real world stay synchronized with reality, despite the latency. This reduces discomfort significantly.

To view this rig at runtime, enable Show Final AR Rig in ZEDManager's Advanced Settings.

Other Useful Scripts

ZEDLight.cs

Add this component to any Point, Spot or Directional light to cause it to illuminate the real world. For directional lights, it will also add shadows on the real world if enabled in the light. Note that in Forward rendering (the default and recommended setting) you can only have 8 such lights at a time. In deferred rendering, you do not need to attach the ZEDLight component for lights to work, and there is no limit to how many you have.

Additional Resources:

Want more details? Our Documentation has overviews and tutorials for getting started with specific features.

Not sure where to start with attaching your ZED Mini to your VR headset? See our guides for Oculus Rift and HTC Vive.

Got a problem you just can't seem to solve? Contact our support at [email protected].

Made something amazing you'd love to share with us? Email our developer evangelist at [email protected].

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

Stereolabs ZED - Unity plugin

Getting Started

Step 1: Check Compatibility

Step 2: Installation

Step 3: Import the Plugin

Example Scenes

Planetarium

Movie Screen

Drone Battle

Dark Room

Simple Plane Detection

VR Only Plane Detection (requires VR HMD and Oculus/SteamVR plugin)

Greenscreen

Creating Your First Project

Adding AR Passthrough

If using a WMR Headset:

Understanding ZED_Rig_Mono and ZED_Rig_Stereo

ZED_Rig_Mono

ZED_Rig_Stereo

Key Scripts

ZEDManager.cs

ZEDRenderingPlane.cs

ZEDCamera.cs

ZEDMixedRealityPlugin.cs

Other Useful Scripts

Additional Resources:

0 comments on commit `448b9d7`

Commit

There are no files selected for viewing

Stereolabs ZED - Unity plugin

Getting Started

Step 1: Check Compatibility

Step 2: Installation

Step 3: Import the Plugin

Example Scenes

Planetarium

Movie Screen

Drone Battle

Dark Room

Simple Plane Detection

VR Only Plane Detection (requires VR HMD and Oculus/SteamVR plugin)

Greenscreen

Creating Your First Project

Adding AR Passthrough

If using a WMR Headset:

Understanding ZED_Rig_Mono and ZED_Rig_Stereo

ZED_Rig_Mono

ZED_Rig_Stereo

Key Scripts

ZEDManager.cs

ZEDRenderingPlane.cs

ZEDCamera.cs

ZEDMixedRealityPlugin.cs

Other Useful Scripts

Additional Resources:

0 comments on commit 448b9d7

0 comments on commit `448b9d7`