Runway In Store

Except for the cool but expensive physical installations, what other technologies could bring more possibilities and convey the brands' attitudes in a more creative way? The emerging technology, Augmented Reality is incorporated by some forward-thinking retail brands into the customer experience, which has the following advantages:

Enables customer engagement

Triggers more customer-generated contents on social media

Is scalable and easy to change themes

Costs much less than physical installations

Runway in store is a physical playground that encourages customers in the retail store to move around and interact with the augmented visuals that are projected on the wall or an LED screen, where a Kinect v2 is used to capture the real-time the human image and drive a real-time movement of a 3D character in Unity. By choosing Nike as the brand, the AR installation tracks human motions with multiplayer features and metaphorical explorations of the augmented 3D visuals.

Customers then could save and share the looks to social media, so that the brand can employ social media as a means of reaching out and connecting with potential customers.Fortunately, this idea drew the attention from Verizon who gave us lots of support to move this project forward.

Initially I wanted to build a mobile AR app, that could enable users to track their foot movement and see the triggered effects on the screen which led to three technical directions for detecting human movement: Machine learning: Vision framework (iOS) or YOLO with CoreML (iOS), Vuforia 3D object recognition, Kinect with creative coding.

My first mobile prototype uses CoreML vision framework which processes a live feed from the camera and extracts information from each frame using both built-in and external machine learning models. However, the real-time object detection results are not reliable even if there are no big moves; The marker-based AR framework Vuforia has size limitations to the scanned object (which ideally would be the shoes), and the 3D object recognition may also lose track sometimes.

Customers then could save and share the looks to social media, so that the brand can employ social media as a means of reaching out and connecting with potential customers.Fortunately, this idea drew the attention from Verizon who gave us lots of support to move this project forward.

So I finally decided to go with Kinect v2 which can capture the RGB images and sense the depth data. I also chose Unity 3D as the developing platform that enables 3D visual programming. Here is how the space setting looks like: customers enter the playground facing a Kinect and a wall to be projected or a big LED screen. The Kinect will track human movements and 3D augmented visuals will be showing on top of the RGB images.

I designed the implemented an algorithm for the augmented 3D visuals with motion trails and particles. All of the parameters could be adjusted and customized easily such as the color, particle lifespan, texture and so on. The algorithm works like this:

Layer 0: Kinect RGB image as background image.

Layer 1: Augmented 3D visual effects. Use Kinect v2 to drive a real-time movement of a 3D character (model).

• Map the Kinect skeleton data to a rigged mesh
• Use the Skinned Mesh Renderer component to render Bone animations, where the shape of the Mesh is deformed by predefined animation sequences.
• Use vertices of an animating skinned mesh as emitting points.
• Computes the velocity and acceleration of each emitting points and generates particle effects accordingly.