Then we set up a distance from a virtual camera (the human eyes) to this frame being 50 cm. As we tested these settings with the device, we’ve found that, because with HoloLens the user sees visuals in a stereo mode, the person naturally recognizes the distance to the frame and understands that the QR code needs to be at the same distance from the eyes as the rectangle is to be captured and recognized by the system. So, the system receives some additional information concerning the space that a QR code should occupy within the frame, as well as the area where this code can be found. So, the computation time is reduced significantly.
Another unexpected finding about HoloLens UI was that if you want to have your 3D models projected the way it is done with regular 3D screens, you, most likely, won’t be able to make it work correctly. The thing is that, in case the model gets projected on the edge of one of the HoloLens screens, the user starts noticing the boundary between the real world and the virtual one. Therefore, you will have to do some post-processing on your side to blur the picture on the edges of the screens.
And how things looked from the code side
Technology-wise, our aim was to make our system perform the following key actions:
- identify the product;
- pull relevant information from the server and prepare statistics;
- visualize the statistics;
- communicate with the user and respond to interactions.
Speaking about functionality implementation, there was no difficulty associated with points #2 and #3 from our list above. These features are easily enforced with Unity3D, as it offers comprehensive, ready-to-use classes and components that you can employ. The item #4 is also easy to implement using the HoloLensInputModule and applying the business logic of interaction.
And what about #1? Well, this is where we had the most fun. So, I would like to get into the details a bit.
Whether it is about a warehouse or a store, the goods are identified and differentiated with the help of machine-readable codes. For our system, we chose QR code and used ZXing.Net library to enable code reading. As we started, it took up to 1.2 seconds for the system to take a picture, search for the QR code, and have it recognized. Well, things were definitely too slow, even for real-time processing. Since 0.4 seconds out of this time were used to take a picture, and this operation is asynchronous, we could only try to influence the remaining 0.8 seconds.
To speed up performance, we reduced the picture area that the system processed to search for the QR code by 9 times and implemented some optimizations for ZXing.Net, such as removed detection for other types of codes and added a range of the required QR code sizes. As a result, we made our system perform calculations within 0.103 seconds. And to ensure smooth animation, you need to have 30-60 FPS (33-16.7 ms).
Since namespace System.Threading is not available with Unity3D for HoloLens, you won’t be able to establish parallel processing. When looking for ready-to-use solutions, we found that Unity’s coroutines is the best fit. We managed to split method performance into several sub-calls. For example, picture processing was broken into three separate stages:
private void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
_photoProcessState.result = result;
_photoProcessState.photoCaptureFrame = photoCaptureFrame;
StartCoroutine(ProcessFrame());
}
private IEnumerator ProcessFrame()
{
yield return null;
PreprocessPhoto(_photoProcessState); // #1
yield return null;
DetectCodes(_photoProcessState); // #2
// return;
}
The first yield was called to avoid the overuse of the frame time, since Unity3D performs some generic actions before it executes the callback function. Method #1 is performed over 0.03 seconds approximately, while #2 takes up to 0.06 seconds. This way, you can split performance into some shorter parts to bypass the exceeded frame time and avoid animation lags.
We can also divide the process of code detection into some smaller chunks and perform “heavy” calculations without the influence on high FPS. To make things even faster, use modules from the unity asset store or WinRT API, which is accessible from C# scripts only.
So, why it is worth bringing mixed reality technology to brick-and-mortar?
In short, with HoloLens managers can effectively combine the digital experience with the benefits of the in-store presence. Mixed reality HoloLens technology gives an immediate possibility to analyze the product-related information right when looking at the shelf. Therefore, retailers can benefit from a great combination of extensive amount of data and visualisation technologies used against the real state of things.
