Adding AI to Mixed Reality

Over the last few months I have had the privilege of helping numerous Microsoft Partners get started with building Artificial Intelligence into their Mixed Reality applications.

It might help you to understand what I mean by Artificial Intelligence, as it is a heavily overloaded term used to describe everything from an algorithm to predict what you might like to buy next, through to a future technology that will run the world and take all of your jobs. For the purpose of this article I will limit the term of AI to describe a set of algorithms to help determine a result of a specific query with a certainty high enough to be useful by the customer making the query. For example, given a sentence spoken by a customer, the algorithm has an 80% (or greater) confidence that the intention of the sentence was to order a specified item for delivery at a given time.

One of the aspects of almost all AI is that software developers are no longer working with clear binary results (1 or 0, on or off), instead, with AI algorithms, the result is a percentage of certainty of correctness, often termed confidence.

Working with this confidence, the application can modify the experience for the customer.
You might be asking why this is interesting for a Mixed Reality application?
With the example I just provided, of understanding the intention of a spoken command, a Mixed Reality application can become far more useful. If you have ever worn a headset, VR, AR, or MR, you will know that the controls for input are limited. Either using hand controllers or simple hand gestures is often not enough to control a complex application or environment. Speech is a great way to enhance the interface. When the speech can be in the form of natural language input that an algorithm can translate into an intention the application can act upon the experience for the customer is greatly improved.

In the one week workshops, the developers learn how to use computer vision services to recognize the objects that a camera is seeing, translate text between languages, understand the intention of a natural language command, and even build their own machine learning algorithm from scratch. The developers then take these lessons and build out a demo or proof of concept application they can take back to their workplace.

One thing that is becoming clear is that while 5 years ago you would have struggled to find things you use every day that utilize some form of AI, in the coming years you will find it hard to find any technology that doesn’t take advantage of some form of AI.

Dr. Neil Roodyn

Experiences with Microsoft’s Azure Face API

In the last few weeks I have been working with Microsoft’s Azure based Face API.

If you have never used the API you might well be surprised by how extensive the information about each face returned by the API can be. Here is just a small part of the information that comes back:
1. The coordinates of the face inside the scene.
2. The amount the face is tilted.
3. A guess of the person’s age.
4. How much the person is smiling.
5. Whether the person is wearing glasses or not.
6. Whether the person has facial hair or not.
7. Whether the person is male or female.
8. A guess at the emotional state of the person.

All the above as well as very detailed information about positions of features in the face can be obtained.

The way in which the API is used has been designed to be very straightforward.

To be able to recognize a face, the Microsoft engine in Azure needs to have some sample images of the face. The set of samples is called the training set, and the project I worked on started by sending a set of images to Azure for each of the people we wanted to recognize later in our project.

When the time came to recognize people, we set up a camera connected to a PC and every few seconds sent the current camera image to Azure asking the Face API to tell us if any faces were in the image.

If a single person walked up to the camera, the response would be that there is one face in the image we had sent. The Face API is quite capable of picking up many faces in a single image (for instance where the image shows a number of people seated around a table).

Once we know there are faces in an image, we need to use a different function in the Azure Face API where we send just the area around a face to Azure and ask whether that face belongs to someone in our training sets. The response we get back is not just a yes/no response, but a probability of how likely it is that the face we sent matches someone. Generally, we would choose the highest probability match (if there is one).

In our project we wanted a PC app to trigger an activity whenever someone the app knew came into range of the camera. In effect we would also know when they had left as we would stop seeing them through the camera.

The Face API made it easy for us to set up the project and begin testing. At that stage, we began to realize it was not all quite so simple.

The first sign was that people who walked past the camera in profile were not recognized. Actually, they weren’t even detected as faces! After some investigation it was possible to determine a list of circumstances that were likely to have an impact on whether someone was going to be matched.

The first step in getting a match as noted above is to detect that there is a face in an image. This step, we discovered can be affected by quite a few things. Here is a partial list:
1. A person’s head should not be turned away from the camera by more than about 45 degrees.
2. If the camera is positioned too far above the mid-line of the face, no face is detected. Similarly, even if the face and camera are at the same level but the person turns their face too far up or looks down too far, no face is detected.
3. If the face is tipped too far from vertical with respect to the camera, a face will not be detected.
4. The mouth should not be covered.
5. The nose should not be covered.
6. Each eye should be visible or at most obscured just by no more than a finger width.
7. Ears, forehead and chin do not need to be visible.
8. Placing a hand against the side of the head or chin does not prevent detection.
9. Beards, moustaches and glasses do not prevent detection.
10. Strong backlighting (e.g. A large window behind a person) can make detection impossible.

Even if a face is detected, the face may fail to match against the training set due to other problems:
1. If the place/camera where the training set was collected is different to the where the recognition is to be done, the success rate in matching may be lowered.
2. If the resolution of the cameras used for training and for recognition are very different, the success rate in matching may be lowered.
3. If the camera resolution is high (e.g. 1920x1080), matching is easily achieved at 2 metres distance from the camera. If the camera resolution is low (e.g. 640x480), matching at 2 metres from the camera becomes difficult.
4. If the facial expression at recognition time is too different to the expression used in the training set (e.g. mouth open at recognition, while the training images all had mouth closed), recognition may fail.

Achieving a reliable result in a project once you know more about the characteristics of the API becomes not just a matter of putting some code together. The project design may need to juggle with the position of the camera, perhaps using more than one camera. Some thought will also need to go into lighting and possibly devising techniques to compensate for perfectly normal face obscuring activities such as people simply turning their heads.

Peter Herman

Wood Staining and Finishing for New and Old Timber

A lot of our furniture that we own, in and around the home, is made from wood, although this maybe less so the case in modern times. Often, when old wooden furniture starts looking worn, people will tend to dispose of it, but with just a little bit of time, effort, and some simple tools, you can make your furniture look new again. Alternatively, you might be working on a little project at home that could benefit aesthetically, and functionally, from a coat of stain and varnish. By conditioning, and varnishing, your timber, you helping to it keep its shape and extend its lifetime.

You will need some tools along the way, including sandpaper or sanding blocks (usually from 240GSM up to 150 GSM), a suitable bristle brush, rags, mineral turpentine, empty jars, and tack cloths. Tack cloth is a slightly adhesive cloth that will help pick up any little imperfections like dust that may settle on the wood between coats.


Sanding/preparation process
If you want to recondition existing furniture, or whatever it may be, the first thing you will have to do is remove the existing coats of varnish and stain. If you want to keep the existing colour, then you will simply need to sand off the layer of varnish. I recommend using 180-grit sandpaper to begin with, then make your way down to 240. If you are removing a layer of varnish, using a suitable wet/dry sandpaper and soaking it with water will stop the varnish from sticking to it. If you would like to give it some new colour, keep sanding until you remove the stained layer and get down to the natural colour of the timber. Remember, with everything you do, always work in the direction of the grain: be it sanding, staining, or varnishing. Once you have finished sanding, clean off the dust with a rag, and then clean off again, with the tack cloth.

1. First make sure all the parts are sanded down to ensure a smooth surface before conditioning. Depending on condition of the timber start with a 160-grit sandpaper and finish on a 240 grit
2. Wipe off the surface with a rag and tack cloth to remove any dust
3. Mix wood conditioner using a clean paddle-pop
4. (Optional) Apply wood primer on all surfaces of the wood with a flood coating, wait 2 hours before flipping over and applying on other side
     a. Wait at least 6 hours before staining


Staining
Next, it is time to stain the timber if you have chosen to do so. If you would like a natural finish, or to keep the previous colour, you can skip this and go straight to varnishing. There are water based and oil based stains and varnishes, but stick to using one type for the stain and the varnish. There are also stain and finish (varnish) cans you can buy, which will also varnish your timber at the same time, but you may also need to do extra coats of varnish if you feel that you have reached the desired colour in the wood. Before you begin, make sure you are working in a well-ventilated environment that won’t be affected by the stain. Stain is near impossible to remove from clothes, and difficult to remove from skins, so wear gloves, and work somewhere that won’t be affected by any splashing.

1. Mix wood stain using a paddle-pop, or if the stain has not been used in a while shake vigorously an hour before use
2. Apply wood stain on top side and end grains on the sides
3. Wipe off any stain that has dripped onto the underside
4. Wait 5-10 minutes
5. Wipe off wood stain using a rag. Ensure this is done in a circular fashion (wax on, wax off)
6. Wait 2 hours until stain is touch dry
7. Flip wood onto the other side
8. Apply wood stain on the new topside
9. Wipe off any stain that has dripped down onto the sides, but take care not to take off any on the top’s edges
10. Wait at least 6 hours before doing another coat
11. Repeat this process so there are 2-3 layers of staining done (3 recommended)


Varnishing
Now for the varnishing, make sure you have wiped the surface off with the tack cloth. If you’re working with mostly flat surfaces, start off with using a brush, and try and keep the coats even on the surface, but don’t go back and try to touch up the varnish if you spot something at the end. If there’s an imperfection, wait for it to cure and sand it off, and try again on the next coat. You will want around 2-3 coats with the brush and finish off with spray lacquer (of the same type); this will help you get a nice smooth finish. If you’re working with furniture with contoured surfaces you may find it easier to exclusively use spray lacquer. Leave the brush to soak in mineral turps between layers, and make sure the surface is touch dry before flipping it over and doing the other side.

1. Mix polyurethane clear coat while taking extreme care not to introduce and air bubbles
2. Apply coat to the top and sides, but do not go back over areas that may not be exactly even; doing this afterwards may compromise the finish
3. Wipe off any lacquer that has dripped onto the bottom side
4. Wait 2-3 hours
5. Flip wood back over
6. Apply coat on new topside
7. Wait at least 6 hours
8. Use 180+ GSM sandpaper and sand the surface moderately and even out any blotches
9. Use 240+GSM and be gentle after the first coat, careful not to remove any of the stain!
10. Repeat until 2-3 coats have been applied
11. Sand down once more
12. Wipe off dust with tack cloth
13. Apply an even coat of spray lacquer on top
14. Wait 2-3 hours
15. Flip over
16. Spray even coat on other side
17. Repeat until at least 3 layers of spray lacquer have been applied; until surface is uniform and sand lightly between each layer
18. OPTIONAL coat the table surface with a layer of beeswax timber finish
19. Rub off beeswax with a rag in a circular fashion once it is dry


That concludes this condensed guide on how to complete wood staining and finishing on timber for new timber products, or give new life to your furniture at home. As always, remember to work in a well-ventilated space, and if you don’t want something to get dirty from splashes or spills, cover it up or work in a different area.

Charlie Ho Si

Business Applications for Mixed Reality

The vast majority of virtual, augmented and mixed reality software in the market right now is for entertainment.

A valid case can be made saying that games, social interactions and 360 videos are the reason for the success of these new technologies within the consumer market, but what I have yet to see, are widespread applications that leverage those technologies to benefit productivity and collaboration in the workplace, while also reducing costs for an enterprise.

Let me give you some examples:
If a person wants to learn how to drive a car, the process begins with a test and then practising driving. Maybe starting in a parking lot of a factory on Sunday, like I did when I was younger, to minimise risks of accidents and then carefully moving to the road. What if driving schools had an Immersive driving experience, where the driver can learn the movements, how to control the car, in complete safety. The driver could also be challenged to drive in different weather conditions, like snow, rain and fog, some of which might not be possible to depending on where you live, and all in total safety. Nothing will replace the real experience, by all means, but the driving school could benefit by reducing the risk to staff and students, reducing the cost of insurance, and car usage, while improving the learning experience.

Let me take you through another example, something I have been secretly working on for some time, at nsquared, and from looking at the most recent results, we are now confident to talk about it. We call it ‘nsquared screens’. It’s an Immersive application, replicating an environment very similar to control rooms which they have at NASA, airport control towers, stock market trading offices and mall security rooms. Those environments can be very pricey to set up. We have replaced the hardware setup, and recreated it as an immersive application, where you can have the data displayed in multiple “floating screens”. Not only does it become much more affordable to create and sustain a control room of this kind, but it would also come with you when you travel for business.

Mixed reality is compelling, and can be incredibly entertaining, however, if you are as passionate as we are about this technology, imagine the ramifications that Mixed Reality can have improving your professional life and the productivity of your enterprise. At nsquared we surely are in a premium position to make this happen.

Stefano Deflorio

References
Northeast Guilford High School (2018). Distracted Driving Awareness Event. [image] Available at: https://www.flickr.com/photos/ncdot/34126511795 [Accessed 16 April 2018].

Stefano Deflorio. (2017). nsquared screens.

Microsoft (2018). Microsoft Mixed Reality [image] Available at: https://winblogs.azureedge.net/win/2016/12/Windows10-MR-Devices-1024x576.jpg [Accessed 16 April 2018].

Artificial Neural Networks

When you read about artificial neural network (ANN), the first thing you learn is that an artificial neural network is like the human brain: it can be trained to perform a certain task. Like how our brain is composed of neurons, that process information received either from the outer world or from other neurons, ANN has artificial neurons that work the same way. In the case of a human brain, when a person touches a kettle of boiling water, input is the touch sensation, and output is a signal from the brain, to remove the hand from the kettle. Similarly, for an ANN that is trained for image recognition, when input is an image of a furry puppy, the output is the word “puppy" or “dog", depending on how it was trained.

Figure 1 - Artificial Neural Network (ANN)

Figure 2 - Biological Neuron

Although, in recent years, ANN has been proven to achieve exceptional results in a particular task, it is yet to reach the capabilities of a human brain, where a single network performs multiple tasks. An ANN is created and trained for a purpose, and with enough data and training, no doubt, it can outperform human brains in executing a task. AlphaGo can be taken as an example, which is the first computer program to defeat a human world champion of the game Go. Other such tasks, where ANN has shown better results, include image and object recognition, and voice recognition. However, the challenge for ANN lies in training one network that can learn and carry out multiple tasks. It would be absolutely amazing to see an ANN that is powerful enough to recognise a person, learn to play computer games, and write songs as well, and I believe, that is the next stepping stone for ANN.

At nsquared, we are excited to be working with cognitive services and machine learning systems, to improve the way we work together better. For an experimental project that I worked on, I created a UWP app that produces drawings of objects using Tensorflow, an open-source machine learning library. In this project, I worked with sketch-rnn, which is a neural network, based on a type of ANN, called recurrent neural network. I used pre-trained models, that were available online, as well as experimented with training my own ANN using existing datasets.

Sabina Pokhrel

References
Burnett, C. (2018). Artificial neural network. [image] Available at: https://commons.wikimedia.org/wiki/File:Artificial_neural_network.svg [Accessed 28 Mar. 2018].

DeepMind. (2018). AlphaGo | DeepMind. [online] Available at: https://deepmind.com/research/alphago/ [Accessed 28 Mar. 2018].

Looxix (2018). Neuron - annotated. [image] Available at: https://commons.wikimedia.org/wiki/File:Neuron_-_annotated.svg [Accessed 28 Mar. 2018].

Steinberg, R. (2018). 6 areas where artificial neural networks outperform humans. [online] VentureBeat. Available at: https://venturebeat.com/2017/12/08/6-areas-where-artificial-neural-networks-outperform-humans/ [Accessed 28 Mar. 2018].

GitHub. (2018). tensorflow/magenta. [online] Available at: https://github.com/tensorflow/magenta/tree/master/magenta/models/sketch_rnn [Accessed 12 Apr. 2018].

Designing for Mixed Reality

At nsquared, we are working very closely with Microsoft to build training material to help you build the best Mixed Reality (MR) applications. One of the topics which we cover in our training material, is how to design for MR, and knowing when to optimise your 3D assets.

Designing for MR can be a daunting challenge, but crucial to the successful performance of an application. The main goal of optimisation, is finding a balance between beautiful 3D assets and making sure assets do not hinder the performance of an application. Optimisation can be considered and applied when modelling, UV mapping, texturing and exporting 3D assets, but it is important to understand when to use the correct method.

Generally, when building 3D assets for game engines, such as Unity, it is important to keep the polygon count as low as possible. The lower the polygon count, the more efficiently an application can run, reducing lag. This is even more important when building 3D assets for MR as the performance requirements are higher.

Before reducing the polygon count of your 3D assets, there are a few key questions you need to keep in mind. This ensures that you are creating a well optimised application, while ensuring that the user experience is maintained.

Ask yourself the following questions:

The key is to understand what needs to be detailed, and what does not. Knowing which assets need more detail will help you prioritise the assets that need the greatest number of polygons, so time and effort is not spent on assets of lower importance.

Bots Everywhere!

As a kid, I used to wonder if I could make an intelligent clone of myself: which would look like me, sound like me and do all the work which I never enjoy doing or find boring to do. As I grew up and entered the professional space, "scheduling meetings" was one of them. Although Outlook was helpful to show the available times for everyone, I wished to not even have to switch my laptop ON, for this. I know I am sounding too lazy, but I am what I am :D.

At nsquared Solutions, the place where I work on amazing tech, and produce and develop awesome outcomes, I got an opportunity to turn my wish to reality: working on automating the "scheduling meetings", into a fun and interactive application. I am glad to mention that I used the Microsoft Bot Framework along with my C# skills, as I wanted the Bot to be up and running as soon as possible. Though, the Microsoft Bot Framework also works with Node.js. It supports a variety of platforms and can be hooked up with channels like Skype, Teams, Facebook, and Slack, to mention a few. Coupled with LUIS (Language Understanding Intelligent Service), the Bot gets more power-packed as it binds the natural language and creates models which improve on usage.

Whenever I talk about Bots, I always think of Sci-fi movies with super complex and intelligent digital beings. When I started to work on the Bot, my expectation was for it to be intelligent and get even smarter over time. I expected it to deliver new experiences, removing barriers between people and technology. And I wanted it to help us make sense of the huge amount of data that is all around us, to deliver those better experiences. The success in adopting AI to solve real-world problems hinges on bringing a comprehensive set of AI services, tools and infrastructure to every developer, so they can deliver AI-powered apps of the future that offer unique, differentiated and personalised experiences. As a summary from my experience, I can say that it is simple for existing developers without deep AI expertise to start building such personalised data-driven app experiences of the future.

Tripti Wani