Mixed Reality

Mixed reality (MR) was defined in 1994 by Paul Milgram and Fumio Kishino as "...anywhere between the extrema of the virtuality continuum". Microsoft has adopted the term for its own platform, Windows Mixed Reality, to help group Virtual and Augmented reality. The technology has been accessible to developers and businesses for a few years now, yet, the clear majority of applications belonging to this not-so-new market are still mainly composed by single person experiences.

This is surely appropriate for a lot of business cases, such as visualisation of data that should only be accessible by the interested party, or training on specific single-user scenarios.

It is also not surprising that a person wearing a headset, gives the impression of “digital isolation” and inspires the idea that MR is more suitable for solo experiences and one-to-environment interactions. An impression that is often misleading.

If we talk about games and entertainment, for example, several multi-player experiences show that it is possible to create immersive and engaging virtual experiences, where users interact with each other and share the same content.

In the business environment, however, not only does there seem to be a resistance for adopting MR, but it is evident to us, that there is a lack of understanding of how this technology could improve collaboration and productivity within teams, possibly involving clients as well.

What could these scenarios be? The list is literally never-ending.

Obviously, any kind of situation that requires cooperation and involves expensive set-ups or dangerous situations, could potentially be recreated in a digital version. Reducing costs and risks is surely an objective of most companies, if not all, and MR could easily be leveraged to achieve better results with lower risks and costs.

Thinking out of the box is paramount for business oriented people (and developers), wishing to work with bleeding edge technologies. Developers who want to engage and experiment in building mixed reality applications, for cooperative scenarios can really go wild, since not much has so far hit the public eye in a significant way. At nsquared, we have brainstormed several scenarios related to what we do best: facilitating cooperation in the work space and improving the outcomes of business meetings.

Our development facility, featuring multiple Microsoft HoloLens’ and immersive headsets allowed us to experiment and develop several ideas and solutions that would be compelling to many businesses using meeting spaces, multi-user interaction, and visual data. Difficult to explain in words, even more difficult to understand the potential of this technology unless you try first hand.

If you are interested in seeing how mixed reality could change the way you do business, contact us and book a hands-on demonstration.

nsquared solutions

Adding AI to Mixed Reality

Over the last few months I have had the privilege of helping numerous Microsoft Partners get started with building Artificial Intelligence into their Mixed Reality applications.

It might help you to understand what I mean by Artificial Intelligence, as it is a heavily overloaded term used to describe everything from an algorithm to predict what you might like to buy next, through to a future technology that will run the world and take all of your jobs. For the purpose of this article I will limit the term of AI to describe a set of algorithms to help determine a result of a specific query with a certainty high enough to be useful by the customer making the query. For example, given a sentence spoken by a customer, the algorithm has an 80% (or greater) confidence that the intention of the sentence was to order a specified item for delivery at a given time.

One of the aspects of almost all AI is that software developers are no longer working with clear binary results (1 or 0, on or off), instead, with AI algorithms, the result is a percentage of certainty of correctness, often termed confidence.

Working with this confidence, the application can modify the experience for the customer.
You might be asking why this is interesting for a Mixed Reality application?
With the example I just provided, of understanding the intention of a spoken command, a Mixed Reality application can become far more useful. If you have ever worn a headset, VR, AR, or MR, you will know that the controls for input are limited. Either using hand controllers or simple hand gestures is often not enough to control a complex application or environment. Speech is a great way to enhance the interface. When the speech can be in the form of natural language input that an algorithm can translate into an intention the application can act upon the experience for the customer is greatly improved.

In the one week workshops, the developers learn how to use computer vision services to recognize the objects that a camera is seeing, translate text between languages, understand the intention of a natural language command, and even build their own machine learning algorithm from scratch. The developers then take these lessons and build out a demo or proof of concept application they can take back to their workplace.

One thing that is becoming clear is that while 5 years ago you would have struggled to find things you use every day that utilize some form of AI, in the coming years you will find it hard to find any technology that doesn’t take advantage of some form of AI.

Dr. Neil Roodyn

Augmented Reality: what could be, should be

Tony Stark operates his Iron Man suit from his office, Aloy from Horizon Zero Dawn overrides Tall Necks, and Shuri drives a car remotely in Black Panther. Pokémon GO overlays your environment to show creatures, and Vuforia allows you to see and interact with digital content. All of these are powered by Augmented Reality, though the first three fictional examples are a bit different, aren’t they: they physically influence the real world, not just overlay it, or provide additional data. This is what needs to change about Augmented Reality development.

Just as the remote control revolutionised the use of the television, augmented reality should revolutionise our lives: allowing us to potentially remote control everything! Wave your hand at the TV to change the channel, or grip and turn your hand mid-air towards your air conditioner to turn it up or down. These kinds of gestures can surpass things like voice control because they are universal – no translation needed!

In Sydney, Australia, at nsquared solutions, I work with the team to create some truly amazing applications and experiences. We turn what was the ‘future’, into the ‘modern’, and show businesses and people how they can work together better, by harnessing technology to its full purpose, so to make our lives easier, and allow people to be more collaborative.

More recently I’ve been working on an application called nsquared space planner, which is usable with both the Microsoft HoloLens, and the new Mixed Reality Immersive Headsets. Though, it provides the user with a different experience, depending on their device. The core premise to the app is to help people design and plan their spaces. The app provides you with various furniture options (over forty), which can be placed in your space, so that you can see what it might look like in your space, in actual scale. The included tools allow for changing the various material styles of the furniture, and the ease of placing them in the world, using position and rotation. In this way, it helps people be more productive, by taking away the logistical headaches that would otherwise be associated with such a task.

What if you could take it a step further though, and whilst overlaying your surroundings, also interact with the real world? Make that lamp brighter or turn the stereo up! Have your Italian friend speak to you and receive live translation. Or mute your smartphone, because you’re too busy having fun – I mean, planning and designing your space.

The good news is that this is all very possible. We are already doing it at nsquared, in Sydney! Object shape, and image, recognition is available to devices like the HoloLens. So is translation of hundreds of languages, and the ability to interact with your lamp remotely.

So, the next time you are watching Tony Stark transform his environment with augmented reality, you will be able to also, and it won’t be so futuristic. From now on, let’s aim to work together better. 

Elliot Moule

References
Coogler R, Black Panther (Marvel Studios 2018).
Favreau J, Iron Man (Marvel Studios 2008).
nsquared space planner, (nsquared solutions, 2018).
Tatsuo N, Pokemon GO (Niantic 2016).
van der Leeuw M, Horizon Zero Dawn (Guerilla Games 2017).
Vuforia (PTC Inc 2011).