Augmented reality for future automation
CIRM Director Kazuma Waimatsu is a Senior Executive Officer at Furuno Electric Co. Ltd. He is charge of Marine Electronic Products and Director of the Marine Electronic Products Division; Here he writes about Augmented reality for future automation.
The number of ships crossing the world’s oceans is increasing every day and the seas are becoming more congested than ever. The size of the ships is also increasing, so it is becoming more and more crucial to be aware of the situation around the vessel directly from the bridge in order to improve safety and security during any operation.
At Furuno we have taken on this challenge and created a system to support seafarers’ visual perception during operations using Augmented Reality (AR) technology which will also be the first step toward the automated ship of the future.
Our company is involved in the supply of marine electronic equipment and services to the commercial shipping, fishing and recreational boat industries. From our foundation in 1948, we first developed fish detection equipment before expanding into the commercial shipping and recreational boat business.
Since then, Furuno have developed a full range of products and expanded into the global market. Wev currently have a 41% share of the global commercial navigation radar market.
Among our latest flagship products is the Furuno ENVISION AR NAVIGATION (AR 100M Type) , which was recently developed to provide substantial navigation support through a system that leverages the latest technology
Until now seafarers have had to keep in their heads a variety of navigational information including radar, Automatic Identification System (AIS), Electronic Chart Display Information System (ECDIS), chart, route and visual information.
But superimposing all these information sources over the actual live video imagery allows to be easily aware of the surrounding area and to share the information in one simple system.
Basically, the actual images taken by a camera placed at the front section of the ship, overlayed with ship operational data presented through AR technology, can be used as an auxiliary operational decision support function to avoid risks and ensure safe navigation.
The combined data which is displayed includes the ship’s position, route, speed and gyro compass location information, in addition to AIS and radar tracking information, the fixed ECDIS route and chart information.
It offers a visual support for stress-free operation by recognising vessels and obstacles in visually challenging circumstances such as night and poor weather conditions.
The project was initiated by Furuno’s development division, which had posed the following question: “Can augmented reality technology be useful for commercial shipping?
AR technology had already been widely used in the car and gaming industry, but there were not many use cases in shipping. Furuno quickly came up with a concept and introduced it for the first time at a maritime exhibition.
It soon attracted interest from a number of companies. Mitsui OSK Lines showed a particularly strong interest and said: “We would really like to try out this technology.” Furuno then embarked on a joint development project.
During development, one of the difficulties was deciding on the type of information and configuration style. Another problem was how to display the real-time data in a stable and easy-to-see way when the ship is rolling or pitching.
Also, the further forward the camera was placed, the more distorted the image became. With the image distortion, we had a tough time trying to clearly overlap the images and information. But despite these difficulties, we succeeded in developing the product.
A prototype was installed on two vessels and further improved following feedback from navigation officers. After a year and a half of development, the product was finally launched in May 2019.
As a joint development project conducted with MOL, it was decided to deploy AR navigation on 21 VLCCs operated by the company.
From early 2020, the system has been offered to shipping companies worldwide. While being complete and revolutionary, AR navigation is only a first step and we intend to develop various additional solutions within the Furuno ENVISION series as visual support for navigation and watchkeeping.
Augmented reality for navigation is our first response in a long road towards the realisation of the first fully automated ship. Since January 2021, we have established a special division to accelerate the development of sensors that will enable the creation of the world’s first fully automated ship by 2025. It is this great alliance of researchers and developers that will make this project a reality.
In Japan in 2020, under the sponsorship of the Nippon Foundation, MEGURI2040, a joint project to develop an unmanned ship, also began.
Here as well, Furuno played an important role in this project, being tasked with using its expertise in sensor technology to detect the area around a vessel.
Until now, the ship’s master has had to carry out the recognition, decision making and operational tasks, but our system is able to provide assistance with the recognition part. We aim to develop a series of technologies that could correctly assess the situation around the vessel and, while avoiding collisions with other vessels, navigate autonomously.
At Furuno, we see three stages in the route toward the realization of autonomous vessel operation. Firstly “Recognition support” supports the recognition of the surrounding situation, the “Decision support” which connects the recognized information with the “judgment” of how to operate the vessel, and finally “ship operational support” connects the judgment with the actual operation of the vessel. AR Navigation is the technology behind the “Recognition support” that is the first step along that path.
AR Navigation is not the final product. Based on mutual requests and customer feedback and recommendations, we aim to further develop the product.
For example, at present, AR navigation is limited to images of the front of the ship, but from now on our research and development team is working on a new solution so that images from all directions can be seen.
To be able to have simultaneous images from all directions, without any blind spots, the study of where and how many cameras need to be placed is necessary.
There are other technical issues such as how to simultaneously combine images from multiple cameras.
This will not be easy, but we intend to meet these technical challenges with the precious help of our clients and partners.
Behind the development of AR Navigation is Furuno’s passionate desire to realize safe navigation.
We have a grand ambition to achieve zero accidents. This does not only apply to automated ships, but also and especially for manned ships.
The first step is to make it easier for crew members to have better and more accurate situational awareness around the vessel. This is why we want to provide new solutions with our most innovative products and functions.
Furuno is moving full steam ahead towards the realisation of totally safe and secure navigation. The journey continues!
Founded in Japan in 1938, the company now has many subsidiaries worldwide with a truly impressive global workforce. https://www.furuno.co.jp/en/
