Computer vision makes crane navigation smarter

The objective of the research collaboration between Konecranes and the University of Helsinki, which started in 2020, is to develop safe and sustainable technologies for the cranes of the future, with automation and AI assisting the crane operator. Konecranes donated funding to the university for the research project. The research will naturally support Konecranes’ product development with scientifically tested data, but it will also have wider benefits and other applications, such as the navigation of autonomous vehicles or drones.

The technology being developed will have an important role in observing and measuring material flows in the material handling operations of a factory, for example, where the crane is just one part of the equation. It helps measure targets to be lifted and observe their movements in a dynamic environment shared by people, other moving machines, and steady objects. This information can be transferred to enterprise resource planning systems and storage management systems to seamlessly support the overall management of material flows within a facility.

Currently, laser technology is widely used for similar perception of physical surroundings, However, the use of camera technology is significantly more cost-effective and will therefore be more easily available for a wider group of users.

The technology being developed will have an important role in observing and measuring material flows in the material handling operations of a factory, for example, where the crane is just one part of the equation.

Deep learning is integral in today’s computer vision research

There are three distinct challenges when it comes to navigating a crane and understanding its environment. One has to do with 3-dimensional object detection, another with the positioning with low-cost one lens cameras, and the third with reflective surfaces in a factory environment. The project seeks to tackle these challenges using a computer vision technique called visual SLAM (simultaneous localization and mapping), in which cameras are used to generate a continuously updating map of a machine’s surroundings as well as information on its exact position.

3D object detection is needed to recognize people and obstacles around the crane to avoid collisions. People, who from the crane camera’s point of view are small, distant, and constantly moving, are not easily identified. In this area, AI and deep learning have an important role. To put it simply, deep learning involves a computer model learning by example, by using for instance large amounts of image or sound input to perform classification tasks.

“Computer vision research today almost always includes elements of deep learning, because it generates significantly better recognition results. Deep learning has developed considerably during the past ten years, and there could be another leap ahead,” says Laura Ruotsalainen, Associate Professor at the Department of Computer Science at the University of Helsinki, who leads the research project.

For 3D object detection to be possible, there needs to be visual data with sufficient information. Cameras in cranes are positioned high up in a somewhat peculiar angle and in this project only have one lens. This creates difficulties with depth perception, which is necessary in order to understand the movement of the crane. Determining distances with one-lens cameras is one of the areas the research team is working on.

The third research area is finding ways to get sufficient information from the images captured by the camera even when there are reflections in the images. Reflective surfaces in a factory setting can hamper the crane’s ability to correctly map its surroundings and detect obstacles. This research area serves as an example of the kind of real-world information about surroundings and constraints that Konecranes can provide to the research project.

“We maintain a regular dialogue between Konecranes and the research team about different use cases, practical problems encountered particularly when it comes to machine vision, and about the practicalities that exist when we are talking about the industrial application of the research. We have provided data and image material and received updates on the progress and findings made by the research team,” says Sami Terho, Senior Specialist, Crane Intelligence at Konecranes.

Helping the crane operator to better observe the environment and adding automation can increase the efficiency of crane operations.

Academic research and application-oriented development in tandem

Konecranes also carries out its own research with a more practical viewpoint in tandem with the academic research. According to Sami Terho, the two approaches complement each other. The university research team is building a theoretical foundation that Konecranes can learn from when developing new technologies and automation solutions.

“We have identified many practical applications. One is the possibility to measure material streams using the crane – what is moved, when, and where. This would give the crane a larger role in material handling. On the other hand, helping the crane operator to better observe the environment and adding automation can increase the efficiency of crane operations,” Terho says.

Safety aspects are naturally also a top priority. If the crane detects a human on its path, it sounds an alarm or stops automatically, which can help in avoiding serious accidents more easily.

Konecranes also cooperates with Aalto University and has donated a crane, named ‘Ilmatar’, to the university for use in research and teaching. The crane serves as part of an open innovation environment that can also be used by, for example, startups, small and mid-sized enterprises, or other parties interested in innovating new applications and devices. The environment includes a digital twin of the crane, enabling research and development from anywhere in the world.

Looking to the future

The research continues on many fronts. The solutions to the challenges concerning the use of computer vision in cranes are yet to be discovered, but both the university research team as well as Konecranes’ own R&D team look confidently ahead.

“We are happy to collaborate with Konecranes because the company appreciates the nature of this research – and does not expect ready-to-use practical business applications tomorrow. Science can take time. But a company that looks to the future and wants to develop its technologies in the long run is easy and rewarding to work with,” Laura Ruotsalainen concludes.