Test Driven Development (TDD)

What is TDD? How is it different from Unit Tests? How many tests should one write when using a TDD approach? - These and many more questions come to our mind when we think or decide to take a TDD approach.

TDD stands for Test Driven Development and is different from writing Unit Tests Unit Tests refer to what you are testing, -where-as TDD describes when you are testing. To simplify this, with Unit tests we test and verify the smallest possible unit of behavior, wherein with TDD the tests drive the development. We can say that Unit tests are a part of a TDD approach where we write tests before writing the code. It can include Unit tests, functional tests, behavioral tests, acceptance tests, etcetera.

The idea looks simple in theory, and represents a fundamental change to approaching software development.

Red-Green-Refactor cycle:
The key to TDD is the Red-Green-Refactor cycle. Write tests that fail, fix the code and run the tests again - repeat this until they pass. The below diagram explains it well:

Source: https://centricconsulting.com/case-studies/agile-test-driven-development/

Workflow:
RED - Write a failing test which captures the requirements.
GREEN - Implement the functionality by writing just enough code to pass the test.
REFACTOR - Refine/improve the code without adding any new functionality.

And then repeat the whole cycle.

In my opinion, it is always helpful to see the code tested upfront using this painless testing approach. It gives a sense of confidence to us before we start end-to-end tests for the project. Not only this, I believe it gives a fair indication of code coverage, fewer defects and easy maintenance as well.

As important as it is to start with this approach, it is equally important to make modifications, whilst continuing using the approach. Considering a real-world situation, applications change and over time a method may be removed/added/modified. While making any modifications in the code at a later stage in time, we should run all the tests written until now to ensure that we did not break any functionality while adding/modifying code. In my experience, this reduces the testing time by more than half.

It is very important to monitor the code coverage though the development/maintenance cycle of the application. With code coverage, we get to know if any code is not being called by a test. The two scenarios that will be applicable here are: the code is missing a corresponding test, or it is dead code and needs to be removed.

As writing-running-fixing tests consumes time, it is very tempting to put the writing of tests on the back-burner. The most pressing and difficult aspect of this is to keep the discipline and continue the practice. When it comes to shipping robust, high quality products the benefits of this approach are rewarding!

At nsquared we are working using TDD on our latest products. If you want to find out more about how we work, please get in touch.

Tripti Wani

Adding AI to Mixed Reality

Over the last few months I have had the privilege of helping numerous Microsoft Partners get started with building Artificial Intelligence into their Mixed Reality applications.

It might help you to understand what I mean by Artificial Intelligence, as it is a heavily overloaded term used to describe everything from an algorithm to predict what you might like to buy next, through to a future technology that will run the world and take all of your jobs. For the purpose of this article I will limit the term of AI to describe a set of algorithms to help determine a result of a specific query with a certainty high enough to be useful by the customer making the query. For example, given a sentence spoken by a customer, the algorithm has an 80% (or greater) confidence that the intention of the sentence was to order a specified item for delivery at a given time.

One of the aspects of almost all AI is that software developers are no longer working with clear binary results (1 or 0, on or off), instead, with AI algorithms, the result is a percentage of certainty of correctness, often termed confidence.

Working with this confidence, the application can modify the experience for the customer.
You might be asking why this is interesting for a Mixed Reality application?
With the example I just provided, of understanding the intention of a spoken command, a Mixed Reality application can become far more useful. If you have ever worn a headset, VR, AR, or MR, you will know that the controls for input are limited. Either using hand controllers or simple hand gestures is often not enough to control a complex application or environment. Speech is a great way to enhance the interface. When the speech can be in the form of natural language input that an algorithm can translate into an intention the application can act upon the experience for the customer is greatly improved.

In the one week workshops, the developers learn how to use computer vision services to recognize the objects that a camera is seeing, translate text between languages, understand the intention of a natural language command, and even build their own machine learning algorithm from scratch. The developers then take these lessons and build out a demo or proof of concept application they can take back to their workplace.

One thing that is becoming clear is that while 5 years ago you would have struggled to find things you use every day that utilize some form of AI, in the coming years you will find it hard to find any technology that doesn’t take advantage of some form of AI.

Dr. Neil Roodyn

Experiences with Microsoft’s Azure Face API

In the last few weeks I have been working with Microsoft’s Azure based Face API.

If you have never used the API you might well be surprised by how extensive the information about each face returned by the API can be. Here is just a small part of the information that comes back:
1. The coordinates of the face inside the scene.
2. The amount the face is tilted.
3. A guess of the person’s age.
4. How much the person is smiling.
5. Whether the person is wearing glasses or not.
6. Whether the person has facial hair or not.
7. Whether the person is male or female.
8. A guess at the emotional state of the person.

All the above as well as very detailed information about positions of features in the face can be obtained.

The way in which the API is used has been designed to be very straightforward.

To be able to recognize a face, the Microsoft engine in Azure needs to have some sample images of the face. The set of samples is called the training set, and the project I worked on started by sending a set of images to Azure for each of the people we wanted to recognize later in our project.

When the time came to recognize people, we set up a camera connected to a PC and every few seconds sent the current camera image to Azure asking the Face API to tell us if any faces were in the image.

If a single person walked up to the camera, the response would be that there is one face in the image we had sent. The Face API is quite capable of picking up many faces in a single image (for instance where the image shows a number of people seated around a table).

Once we know there are faces in an image, we need to use a different function in the Azure Face API where we send just the area around a face to Azure and ask whether that face belongs to someone in our training sets. The response we get back is not just a yes/no response, but a probability of how likely it is that the face we sent matches someone. Generally, we would choose the highest probability match (if there is one).

In our project we wanted a PC app to trigger an activity whenever someone the app knew came into range of the camera. In effect we would also know when they had left as we would stop seeing them through the camera.

The Face API made it easy for us to set up the project and begin testing. At that stage, we began to realize it was not all quite so simple.

The first sign was that people who walked past the camera in profile were not recognized. Actually, they weren’t even detected as faces! After some investigation it was possible to determine a list of circumstances that were likely to have an impact on whether someone was going to be matched.

The first step in getting a match as noted above is to detect that there is a face in an image. This step, we discovered can be affected by quite a few things. Here is a partial list:
1. A person’s head should not be turned away from the camera by more than about 45 degrees.
2. If the camera is positioned too far above the mid-line of the face, no face is detected. Similarly, even if the face and camera are at the same level but the person turns their face too far up or looks down too far, no face is detected.
3. If the face is tipped too far from vertical with respect to the camera, a face will not be detected.
4. The mouth should not be covered.
5. The nose should not be covered.
6. Each eye should be visible or at most obscured just by no more than a finger width.
7. Ears, forehead and chin do not need to be visible.
8. Placing a hand against the side of the head or chin does not prevent detection.
9. Beards, moustaches and glasses do not prevent detection.
10. Strong backlighting (e.g. A large window behind a person) can make detection impossible.

Even if a face is detected, the face may fail to match against the training set due to other problems:
1. If the place/camera where the training set was collected is different to the where the recognition is to be done, the success rate in matching may be lowered.
2. If the resolution of the cameras used for training and for recognition are very different, the success rate in matching may be lowered.
3. If the camera resolution is high (e.g. 1920x1080), matching is easily achieved at 2 metres distance from the camera. If the camera resolution is low (e.g. 640x480), matching at 2 metres from the camera becomes difficult.
4. If the facial expression at recognition time is too different to the expression used in the training set (e.g. mouth open at recognition, while the training images all had mouth closed), recognition may fail.

Achieving a reliable result in a project once you know more about the characteristics of the API becomes not just a matter of putting some code together. The project design may need to juggle with the position of the camera, perhaps using more than one camera. Some thought will also need to go into lighting and possibly devising techniques to compensate for perfectly normal face obscuring activities such as people simply turning their heads.

Peter Herman

Business Applications for Mixed Reality

The vast majority of virtual, augmented and mixed reality software in the market right now is for entertainment.

A valid case can be made saying that games, social interactions and 360 videos are the reason for the success of these new technologies within the consumer market, but what I have yet to see, are widespread applications that leverage those technologies to benefit productivity and collaboration in the workplace, while also reducing costs for an enterprise.

Let me give you some examples:
If a person wants to learn how to drive a car, the process begins with a test and then practising driving. Maybe starting in a parking lot of a factory on Sunday, like I did when I was younger, to minimise risks of accidents and then carefully moving to the road. What if driving schools had an Immersive driving experience, where the driver can learn the movements, how to control the car, in complete safety. The driver could also be challenged to drive in different weather conditions, like snow, rain and fog, some of which might not be possible to depending on where you live, and all in total safety. Nothing will replace the real experience, by all means, but the driving school could benefit by reducing the risk to staff and students, reducing the cost of insurance, and car usage, while improving the learning experience.

Let me take you through another example, something I have been secretly working on for some time, at nsquared, and from looking at the most recent results, we are now confident to talk about it. We call it ‘nsquared screens’. It’s an Immersive application, replicating an environment very similar to control rooms which they have at NASA, airport control towers, stock market trading offices and mall security rooms. Those environments can be very pricey to set up. We have replaced the hardware setup, and recreated it as an immersive application, where you can have the data displayed in multiple “floating screens”. Not only does it become much more affordable to create and sustain a control room of this kind, but it would also come with you when you travel for business.

Mixed reality is compelling, and can be incredibly entertaining, however, if you are as passionate as we are about this technology, imagine the ramifications that Mixed Reality can have improving your professional life and the productivity of your enterprise. At nsquared we surely are in a premium position to make this happen.

Stefano Deflorio

References
Northeast Guilford High School (2018). Distracted Driving Awareness Event. [image] Available at: https://www.flickr.com/photos/ncdot/34126511795 [Accessed 16 April 2018].

Stefano Deflorio. (2017). nsquared screens.

Microsoft (2018). Microsoft Mixed Reality [image] Available at: https://winblogs.azureedge.net/win/2016/12/Windows10-MR-Devices-1024x576.jpg [Accessed 16 April 2018].

Artificial Neural Networks

When you read about artificial neural network (ANN), the first thing you learn is that an artificial neural network is like the human brain: it can be trained to perform a certain task. Like how our brain is composed of neurons, that process information received either from the outer world or from other neurons, ANN has artificial neurons that work the same way. In the case of a human brain, when a person touches a kettle of boiling water, input is the touch sensation, and output is a signal from the brain, to remove the hand from the kettle. Similarly, for an ANN that is trained for image recognition, when input is an image of a furry puppy, the output is the word “puppy" or “dog", depending on how it was trained.

Figure 1 - Artificial Neural Network (ANN)

Figure 2 - Biological Neuron

Although, in recent years, ANN has been proven to achieve exceptional results in a particular task, it is yet to reach the capabilities of a human brain, where a single network performs multiple tasks. An ANN is created and trained for a purpose, and with enough data and training, no doubt, it can outperform human brains in executing a task. AlphaGo can be taken as an example, which is the first computer program to defeat a human world champion of the game Go. Other such tasks, where ANN has shown better results, include image and object recognition, and voice recognition. However, the challenge for ANN lies in training one network that can learn and carry out multiple tasks. It would be absolutely amazing to see an ANN that is powerful enough to recognise a person, learn to play computer games, and write songs as well, and I believe, that is the next stepping stone for ANN.

At nsquared, we are excited to be working with cognitive services and machine learning systems, to improve the way we work together better. For an experimental project that I worked on, I created a UWP app that produces drawings of objects using Tensorflow, an open-source machine learning library. In this project, I worked with sketch-rnn, which is a neural network, based on a type of ANN, called recurrent neural network. I used pre-trained models, that were available online, as well as experimented with training my own ANN using existing datasets.

Sabina Pokhrel

References
Burnett, C. (2018). Artificial neural network. [image] Available at: https://commons.wikimedia.org/wiki/File:Artificial_neural_network.svg [Accessed 28 Mar. 2018].

DeepMind. (2018). AlphaGo | DeepMind. [online] Available at: https://deepmind.com/research/alphago/ [Accessed 28 Mar. 2018].

Looxix (2018). Neuron - annotated. [image] Available at: https://commons.wikimedia.org/wiki/File:Neuron_-_annotated.svg [Accessed 28 Mar. 2018].

Steinberg, R. (2018). 6 areas where artificial neural networks outperform humans. [online] VentureBeat. Available at: https://venturebeat.com/2017/12/08/6-areas-where-artificial-neural-networks-outperform-humans/ [Accessed 28 Mar. 2018].

GitHub. (2018). tensorflow/magenta. [online] Available at: https://github.com/tensorflow/magenta/tree/master/magenta/models/sketch_rnn [Accessed 12 Apr. 2018].

Augmented Reality: what could be, should be

Tony Stark operates his Iron Man suit from his office, Aloy from Horizon Zero Dawn overrides Tall Necks, and Shuri drives a car remotely in Black Panther. Pokémon GO overlays your environment to show creatures, and Vuforia allows you to see and interact with digital content. All of these are powered by Augmented Reality, though the first three fictional examples are a bit different, aren’t they: they physically influence the real world, not just overlay it, or provide additional data. This is what needs to change about Augmented Reality development.

Just as the remote control revolutionised the use of the television, augmented reality should revolutionise our lives: allowing us to potentially remote control everything! Wave your hand at the TV to change the channel, or grip and turn your hand mid-air towards your air conditioner to turn it up or down. These kinds of gestures can surpass things like voice control because they are universal – no translation needed!

In Sydney, Australia, at nsquared solutions, I work with the team to create some truly amazing applications and experiences. We turn what was the ‘future’, into the ‘modern’, and show businesses and people how they can work together better, by harnessing technology to its full purpose, so to make our lives easier, and allow people to be more collaborative.

More recently I’ve been working on an application called nsquared space planner, which is usable with both the Microsoft HoloLens, and the new Mixed Reality Immersive Headsets. Though, it provides the user with a different experience, depending on their device. The core premise to the app is to help people design and plan their spaces. The app provides you with various furniture options (over forty), which can be placed in your space, so that you can see what it might look like in your space, in actual scale. The included tools allow for changing the various material styles of the furniture, and the ease of placing them in the world, using position and rotation. In this way, it helps people be more productive, by taking away the logistical headaches that would otherwise be associated with such a task.

What if you could take it a step further though, and whilst overlaying your surroundings, also interact with the real world? Make that lamp brighter or turn the stereo up! Have your Italian friend speak to you and receive live translation. Or mute your smartphone, because you’re too busy having fun – I mean, planning and designing your space.

The good news is that this is all very possible. We are already doing it at nsquared, in Sydney! Object shape, and image, recognition is available to devices like the HoloLens. So is translation of hundreds of languages, and the ability to interact with your lamp remotely.

So, the next time you are watching Tony Stark transform his environment with augmented reality, you will be able to also, and it won’t be so futuristic. From now on, let’s aim to work together better. 

Elliot Moule

References
Coogler R, Black Panther (Marvel Studios 2018).
Favreau J, Iron Man (Marvel Studios 2008).
nsquared space planner, (nsquared solutions, 2018).
Tatsuo N, Pokemon GO (Niantic 2016).
van der Leeuw M, Horizon Zero Dawn (Guerilla Games 2017).
Vuforia (PTC Inc 2011).