Utopia or reality? Mixed reality in industrial practice
03 May 2018
Mixed reality in the process industry – dream or reality? (Source: Tran - stock.adobe.com)
Wherever software supports people at work, the question of the best man-machine interface arises. Can mixed reality solutions simplify users’ work in the future?
In other words: How and where can the information the person needs be presented so that it can be understood, is easy to find and can be used in practice? This also applies in the daily work of automation engineering, and especially in the process industry, whether in the operation of the plant or in servicing and maintenance work. Can solutions from the field of mixed reality simplify users’ work in the future?
Since many people are not necessarily wholly familiar with the terms virtual, augmented and mixed reality, here are a few definitions. Virtual reality is understood as the complete simulation of a virtual reality. Augmented reality, on the other hand, extends physical reality by adding virtual aspects. Finally the term “mixed reality” is used at present to describe all media that lie between the poles of physical and virtual reality; thus it is used as a general or umbrella term (Fig. 1). The mixed reality spectrum is thus delimited on one side by physical and on the other by virtual reality.
Typical application cases
Before discussing whether it is already technically possible to use mixed reality in the process industry today – and if so, how – the question arises of what the potential applications might be. Are there applications which might benefit from a man-machine interface of this kind, either now or in the near future? The answer is – yes. There is a great potential in plant maintenance. In the extensive plants of the process industry, it can prove difficult to even find the components that need to be maintained. In this situation it could be envisaged that a mixed reality solution would, so to speak, “navigate” an employee through the plant. Directly on the spot it could then display the information on the components requiring maintenance, and provide access to the necessary documents. And while all this was going on, the technician would have both hands free to concentrate on his actual task.
The mixed reality spectrum. (Source: Rösberg)
There is also a potentially useful application at the control level. Instead of displaying the information about the distributed control system on big screens at a central point, it could be displayed on head-mounted devices (HMDs). Thanks to the associated freedom of mobility it would then be possible to compare the apparent status of the plant with the measured values from individual devices on site.
Another potential area of application is repetitive processes – monotonous but important tasks. Here, too, there is a use for mixed reality solutions. When performing such activities, people tend to tire and make mistakes more easily. Appropriate mixed reality concepts could help employees to work more efficiently, carrying out the unvarying processes in an automated form and documenting all the steps. There are certainly many other potential areas of application for mixed reality solutions apart from the examples mentioned here; thus it is definitely justifiable to examine the question of technical feasibility.
Where will the future take us?
Rösberg Engineering GmbH from Karlsruhe, Baden has taken up this challenge. The enterprise has specialised in the process industry and offers software solutions that support plant engineers and operators in their work throughout the life cycle of the plant. The company from Karlsruhe also provides services in the modernisation and building of process plants. Of course, as automation experts they always focus on the question of what trends will influence the sector in the future, and address the aspect of how far these can realistically be implemented. For this reason the company entrusted Stefan Stegmüller, a student from the Information Technology dual study course, with the task of investigating the present state of technology in this area for his bachelor thesis. He also examined the practical use of a mixed reality solution by designing an experiment that reproduced a real situation in a plant, with the aim of assessing the time and resources necessary to realise the solution, and identifying potential difficulties.
Plant Assist Manager: system description. The software supports the user in performing and documenting process steps and workflows. (Source: Rösberg)
Mixed reality: A use case
The use case chosen for the research project was based on the Plant Assist Manager (PAM) developed by the automation experts (Fig. 2). This software supports users in performing and documenting workflows such as the filling of tank trucks. The workflows are optimised, automatically documented and then displayed, for instance, on mobile devices. With the help of a special checklist the user is guided through the work processes and thus has all relevant information available at a glance. For this application, an HMD would be ideal as the man-machine interface. The bachelor thesis focused on testing the feasibility and usefulness of this approach.
To do this, Stegmüller worked with an experimental construction that was a scaled-down version of a real tank filling plant. When filling tanks with chemicals, it is essential to avoid all mistakes, as they may be dangerous. Here support is provided by the relevant checklists. The aim was to navigate the user through the filling process using the Microsoft HoloLens.
Practical benefits and challenges
Stefan Stegmüller investigated the practical use of the Microsoft HoloLens for the process industry. (Source: Rösberg)
The advantages of this solution are obvious: employees no longer have to switch to and fro between the task to be performed and the tablet with the required instructions; they have all the information directly “in view” while working. They can confirm completed actions by voice or gesture command and have both hands free to work with. Moreover, operating in this way is much more intuitive if the necessary work instructions are directly linked to the relevant plant component, or the valve to be opened.
Major challenges were posed by the choice of the right tracking method, i.e. the question of how best to anchor the virtual content in the real environment. This is essential in order for the displayed information to be shown at the right point. Basically the following methods are available: magnetic or infra-red tracking; tracking with visible light; or a system using inertia (Inertial Measurement Unit, IMU for short). In tracking with visible light, a differentiation is additionally made between tracking with natural features, referenced tracking or model-based tracking. Stegmüller reports: “In the case I am concerned with, I mainly considered the possibilities of tracking with natural features, meaning prominent features of perceived natural images, and tracking with reference marks such as QR codes, for instance. The final choice fell at first on QR codes and this was subsequently modified to markings in the form of images with triangles of random size, sequence and rotation. This variant proved to be advantageous as these perceived images can be uniquely identified by the algorithm used. When using QR codes as anchor points for virtual content, mistaken identities can occur during evaluation, due to their basic structural similarity.”
Seamless interaction
Another big challenge in implementing the system was the interplay between individual components such as PLC, databases, software and the HoloLens: what components are compatible, and where do interfaces still need to be developed? The concepts initially developed for the system had to be re-thought and adapted several times. During the research project it became clear that the use of a mixed reality solution in industrial practice is definitely realistic. Of course, the input in terms of resources has to be weighed against the benefits in each case. Stegmüller sums up: “In the next few years a lot will be happening in this field, and I can certainly envisage mixed reality headsets like the HoloLens opening up practical applications in the process industry in the foreseeable future. Practical tests have shown that this man-machine interface and tools such as the Plant Assist Manager ideally complement one other. We look forward to the first applications in industrial practice.”
Contact Details and Archive...