Adding AI to Mixed Reality

Over the last few months I have had the privilege of helping numerous Microsoft Partners get started with building Artificial Intelligence into their Mixed Reality applications.

It might help you to understand what I mean by Artificial Intelligence, as it is a heavily overloaded term used to describe everything from an algorithm to predict what you might like to buy next, through to a future technology that will run the world and take all of your jobs. For the purpose of this article I will limit the term of AI to describe a set of algorithms to help determine a result of a specific query with a certainty high enough to be useful by the customer making the query. For example, given a sentence spoken by a customer, the algorithm has an 80% (or greater) confidence that the intention of the sentence was to order a specified item for delivery at a given time.

One of the aspects of almost all AI is that software developers are no longer working with clear binary results (1 or 0, on or off), instead, with AI algorithms, the result is a percentage of certainty of correctness, often termed confidence.

Working with this confidence, the application can modify the experience for the customer.
You might be asking why this is interesting for a Mixed Reality application?
With the example I just provided, of understanding the intention of a spoken command, a Mixed Reality application can become far more useful. If you have ever worn a headset, VR, AR, or MR, you will know that the controls for input are limited. Either using hand controllers or simple hand gestures is often not enough to control a complex application or environment. Speech is a great way to enhance the interface. When the speech can be in the form of natural language input that an algorithm can translate into an intention the application can act upon the experience for the customer is greatly improved.

In the one week workshops, the developers learn how to use computer vision services to recognize the objects that a camera is seeing, translate text between languages, understand the intention of a natural language command, and even build their own machine learning algorithm from scratch. The developers then take these lessons and build out a demo or proof of concept application they can take back to their workplace.

One thing that is becoming clear is that while 5 years ago you would have struggled to find things you use every day that utilize some form of AI, in the coming years you will find it hard to find any technology that doesn’t take advantage of some form of AI.

Dr. Neil Roodyn