Advanced Navigation Aids System based on Augmented Reality Available online at www sciencedirect com ScienceDirect International Journal of e Navigation and Maritime Economy 5 (2016) 021 – 031 Origina[.]
Available online at www.sciencedirect.com ScienceDirect International Journal of e-Navigation and Maritime Economy (2016) 021 – 031 Original article Advanced Navigation Aids System based on Augmented Reality* Jaeyong OH 1, Sekil PARK 2, Oh-Seok KWON 3 3 Dept Korea Research Institute of Ships and Ocean Engineering, Korea, ojyong@kriso.re.kr Korea Research Institute of Ships and Ocean Engineering, Korea, skpark@kriso.re.kr of Computer Engineering, Chungnam National University, Korea, oskwon@cnu.ac.kr, Corresponding Author Abstract Many maritime accidents have been caused by human-error including such things as inadequate watch keeping and/or mistakes in ship handling Also, new navigational equipment has been developed using Information Technology (IT) technology to provide various kinds of information for safe navigation Despite these efforts, the reduction of maritime accidents has not occurred to the degree expected because, navigational equipment provides too much information, and this information is not well organized, such that users feel it to be complicated rather than helpful In this point of view, the method of representation of navigational information is more important than the quantity of that information and research is required on the representation of information to make that information more easily understood and to allow decisions to be made correctly and promptly In this paper, we adopt Augmented Reality (AR) technologies for the representation of information AR is a 3D computer graphics technology that blends virtual reality and the real world Recently, this technology has been widely applied in our daily lives because it can provide information more effectively to users Therefore, we propose a new concept, a navigational system based on AR technology; we review experimental results from a ship-handling simulator and from an open sea test to verify the efficiency of the proposed system Keywords: navigational aids system, augmented reality, IBS(Integrated Bridge System) Copyright གྷ 2016, International Association of e-Navigation and Ocean Economy Hosting by Elsevier B.V This article is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/) Peer review under responsibility of Korea Advanced Institute for Intelligent Maritime Safety and Technology * This is a revised version presented at the 3rd Ai-MAST held at Rivera Hotel in Daejeon, Korea, November 12-14, 2015 http://dx.doi.org/10.1016/j.enavi.2016.12.002 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality I Introduction The marine traffic environment has recently witnessed significant increases in traffic and trends toward larger and faster ships, resulting in a greater risk of marine accidents Data from the Korean Maritime Safety Tribunal has revealed a steady rise in marine accidents over the last five years, with 80% of such accidents being caused by errors during operation Among all accidents, collision accidents were highest in number and mainly caused by human error such as negligence, and the number of marine accidents continues to increase along with the resulting damages (Ministry of Oceans and Fisheries, 2015) In order to prevent or help mitigate this, real-time support is provided to ship navigators through the Vessel Traffic Services System (VTS) and the Aids to Navigation System, and related legislation around the world is being carried out to prevent marine accidents Additionally, navigation systems with advanced IT technologies are being developed to provide various data to navigators for safer operation; however, such complex equipment could be more of a burden for navigators and interfere with safe operation It cannot be assumed that all navigation equipment provides useful information to navigators, and thus there is a need to analyze the current limitations of different types of navigation equipment and solutions to such limitations from the perspective of the navigators According to the bridge operations analysis results by STCW (The International Convention on Standards of Training, Certification and Watchkeeping for Seafarers), the largest problem with currently operating navigation equipment is the provision of excessive and unnecessary information or inappropriate methods of data provision In particular, the alert functionality with respect to emergency situations is currently provided through RADAR and ECDIS, but such alert messages are delivered through text messages or alarm sounds which are ineffective and not intuitive, so this aspect requires further improvement Additionally, an effective means of information delivery for limited screen space needs to be further studied (Jeong J.S, 2012) In this respect, this paper proposes an AR (Augmented Reality) technology-based navigational aid system that can be utilized in bridge environments as an intuitive means of information delivery Experimentation was conducted to examine the application potential of the proposed system II Analysis of Previous Study As mentioned above, the complexity of the navigation equipment in the bridge can interfere with safe operation; thus, navigational aid systems developed in the future need to support the decision-making process of the navigator through effective information provision methods rather than by simply increasing the amount of information delivered Moreover, the intuitiveness of the navigation data provided is a critical element that can resolve issues caused by excessive information This paper aims to improve upon this by utilizing augmented reality technology Augmented Reality is a field of VR (Virtual Reality) that fuses virtual objects or information to the real environment through computer graphics technology so that virtual objects appear to originate from the existing environment This technology allows the display of information by overlapping the information upon images of the real world Considering that data obtained 22 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality through watch-keeping takes up a significant portion of the ship operation data provided, this method of utilizing augmented reality as a method of information provision in a bridge may be very effective Various studies are currently being carried out regarding such augmented reality technologies along with the recent development of mobile devices, and applications to a diverse range of industries including aerospace, automobile, and ships are increasing rapidly The augmented reality display technology, HUD (Heads-Up Display) system is being used in fighter jets and commercial airliners to contain the time of the pilot and enhance the capabilities of the pilot (Kim K.H 2008) Currently, attempts to apply this HUD technology to the automobile industry are being carried out in order to minimize visual interference to the driver while providing essential information while driving Generally, ships are relatively slower than aircraft or automobiles However, navigational equipment has been diversified with the development of new technologies Therefore, ships officers are requesting the efficient display of information on the bridge, and various navigational aids systems are being researched and developed using augmented reality technology as a means of displaying data In the European e-Navigation project, ACCSEAS, a wearable navigational aids system was developed using HMD (Head-Mounted Display), as shown in Figure (a), in order to enhance the target recognition speed of the navigator Though simulation experiments were conducted, further research using the position data of the ship and navigator is still necessary for accurate registration (ACCSEAS, 2015) Additionally, various navigation support applications have been developed recently for mobile devices, as shown in Figure (b), and such applications are being used in the navigation of small ships, like yachts (B&G, 2015) Figure 1: User Interfaces of AR Navigation System (a) ACCSEAS Project, (b) Mobile application of SeaNav, B&G Source: ACCSEAS (2015), B&G (2015) However, these systems all have different user interfaces, different methods of displaying information, and different types of data expressed with no agreed upon standard, causing inconveniences to the user and restrictions in their usage Also, the indiscreet provision of inaccurate data can result in serious problems such as marine accidents, further necessitating research on what kind of data is to be shown and how such data will be delivered (Olivier Hugues, 2010) In this paper, a survey of navigators was conducted to determine key elements necessary for navigation, and we then propose an augmented reality-based user interface which can effectively express such data Moreover, a navigational aid system to implement the proposed 23 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality interface is detailed, and we review experimental results from ship-handling simulator and an open sea test to verify the efficiency of the proposed system III Navigational Aids System based on AR 3.1 User Requirements As the first step in designing and implementing an augmented reality-based navigational aid system, a survey was carried out to analyse user requirements This survey was composed of questions to determine necessary navigation data, effective methods of presenting such data, and the arrangement or location of the presentation of the data The survey was completed by 20 experienced ship officers Analysis of the survey results revealed that most respondents preferred navigation data to be presented and arranged in ways similar to how they are presented in current navigation equipment and desired new functionality including the display of distance to a fairway Table 1: List of the Maritime Service Portfolios (MSPs) – taken from Annex of NCSR 1/28) Items Details Unit Type GPS UTC/LAT/LON/SoG/CoG date/time/degrees/knots/degrees $xxRMC heading heading degrees $xxHDT STW speed through water knots $xxVHW AIS AIS messages - $xxVDM echo Sounder depth meter $xxDPT tidal Current tidal current direction/speed degrees/knots $xxVHW wind wind direction/speed degrees/knots $xxVWR RoT rate of turn degrees/min $xxROT rudder indicator rudder indicator degrees $xxRSA RADAR target RADAR tracked target - $xxTTM CCTV camera color image, pan/tilt degrees/degrees JPEG Ownship attitude roll/pitch degrees/degrees - Figure 2: User Interface of AR Navigation System 24 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality In addition, respondents showed positive responses toward the augmented reality system being developed, stating that it could be used effectively in the entry and departure of ports On the other hand, responses showed that integration with the ECDIS system is necessary for the provision of accurate navigation data and that data regarding fairways, anchorage, land area, and planned routes are necessary for safe operation Table shows the necessary navigation data for the augmented reality navigational aid system obtained from the survey results 3.2 Design of User Interface The information for the navigational aids system were categorized into 1) data regarding the ownship in operation, 2) data regarding traffic ships, and 3) ECDIS data based on the survey response analysis results As shown in Figure 2, data regarding the ship in operation are mainly displayed through the ODD (Overhead Data Display) in the bridge, composed of the location, bearing, speed, and rate of turn (RoT), and displayed on the upper portion of the augmented reality display using an intuitive interface, such as a graph Data of traffic ships are displayed as overlays on their corresponding positions, where detailed data was additionally displayed when the user selected the data The detailed data was generated based on the AIS data, and the targets of RADAR and AIS were distinguished using symbols The ECDIS data was displayed selectively on the location of the ship in operation and followed the standard of ENC (Electronic Navigational Charts) symbol Additionally, the user was able to select whether to display each data, and it was designed so that the user is able to adjust the data display color 3.3 System Configuration The hardware of the system is composed of a PTZ (Pan/Tilt/Zoom) camera, AHRS (Attitude and Heading Reference System), NMEA Combiner, and user console with an additional joystick device for camera control The software system is composed of a data manager module, user interface module, registration module, and augmented image rendering module Figure shows the data flow between each module The first stage of the system collects navigational data through the NMEA combiner, and the ENC data and planned route data are extracted through the ECDIS integration module Following this, the user interface module uses the collected navigation data to generate the display component elements, processes user input such as through the joystick, and finally generates the augmented image in real time through the registration and rendering modules for provision to the navigator In this experiment, a simulator system was also used to conduct tests in various maritime conditions, and a simulator integration module was included for this The detailed specifications of each module are as follows 25 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality Figure 3: Configuration of Navigation Aids System based on AR (a) Data Manager Module The data manager module processes navigation data collected from various navigation devices of the bridge and performs integration and management functions The collected navigation data in our experiment followed the NMEA-0183 standard, was synchronized based on the GPS time (UTC, Universal Time Coordinated), and was shared with other modules through Shared Memory And, multi-threading was used for performance enhancement to allow parallel processing of the input navigation data (b) ECDIS Module ENC are vector type charts that conform to International Hydrographic Organization (IHO) specifications, as contained in Publication S-57 S-57 includes various feature object classes and their attributes The proposed system extracts and uses elements necessary for the augmented reality navigational aids system from the ENC data of ECDIS Data determined through analysis of the requirements is extracted from ENC, and the extracted data is then stored in a separate database system and displayed selectively depending on the location of the ownship in operation In addition, the distance to the next waypoint and the estimated arrival time data are calculated from the route plan inputted through ECDIS (c) Unser Interface Module The user interface module is the user interface (GUI) provided to the navigator and performs the task of user input processing through mouse and joystick operation Additionally, display components are generated using the collected navigation data, and the option of whether to display each component along with its colour is reflected (d) Augmented Image Rendering Module In the augmented image rendering module, an augmented image is generated by integrating the collected navigation data and ENC data In order to generate the augmented image, the augmented elements are arranged in a virtual 3-dimensional space and integrated with the real world image in real time In this study, a 3-dimensional graphics engine (Unity3D) that supports GPU acceleration was used to enhance the performance of the image rendering Additionally, the 26 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality registration process of matching the position of the virtual object with the position of the real world image needs to be performed in advance for the virtual reality system In this module, registration is carried out by positioning the camera in the virtual space according to the ownship position (latitude and longitude) and attitude (pitch and roll) data obtained in real time through the GPS and AHRS sensors installed on the ship (e) Simulator Integration Module A ship-handling simulator was used in this study in order to assess the applicability of the implemented augmented reality-based navigation support service and system for various maritime environments The simulator integration module virtually generated a camera image and sensor data, and the generation period and types of data used were identical to the actual ship environment IV Experiments In this study, a prototype of the AR navigational aid system was manufactured based on the implemented technology Using the prototype, simulated and real ship application experiments were performed For each experiment, each function of the prototype was tested for normal operations, application feasibility to the bridge environment was evaluated, and usability as navigation equipment was assessed Table shows the specifications of the prototype used in the experiment Table 2: Specification of Prototype system Specification PC system CPU Intel i7 3.0GHz RAM 16GB GPU NVIDIA GTX750 Operating System Windows pro 64bit Monitor LCD monitor, 23 inch PTZ Camera AXIS P5534E (image resolution : 1280×720 pixels) 4.1 Experiment Conditions The subjects of all the experiments were experienced ship officers After being sufficiently informed of the system instructions before the experiments were carried out, the navigator subjects were instructed to use the system for port entry and departure situations After the experiment was finished, opinions on the usability of the system and additional requirements were collected through surveys and interviews For the simulator experiment, we utilized an FMB (Full Mission Bridge) class ship-handling simulator system from the Korea Research Institute of Ships and Ocean Engineering (KRISO) In order to make the experiments as realistic as possible, the experiment scenario was designed using marine traffic analysis data from Busan North Port (Safe Tech Research, 2013) For the experiment at open sea, the prototype system was installed on the 27 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality training ship, “Saenuri” of Mokpo National Maritime University, and the experiment was carried out using entry and departure conditions at Mokpo Port The detailed experiment conditions are shown in Table 3, and the experiment environment is shown in Figure Figure 4: Experiments for usability test (a) AR navigation system with ship-handling simulator system, (b) Saenuri at Mokpo harbour, (c) prototype of AR navigation system Table 3: Experiments condition Simulation test Sea test target harbor Busan port entry Mokpo port entry ownship Tanker SAENURI traffic-ships 15 vessels > 30 vessels weather dense fog light fog # of subjects 12 4.2 Experiment Results The experiment included the process of identifying problems during the operation of the prototype system along with each function Experiment results showed that the hardware and software systems operated in a stable manner in the simulator and real ship environment, and the average augmented image refresh rate, which determines the performance of the entire system, was measured to be above 30Hz In addition, the data manager module processed all the received data without data loss Figure shows the user screens of the simulator and real ship experiment The ownship information was displayed as a graph in the upper portion of the screen by integrating the navigation data as shown in Figure (a), and the azimuth circle data shown below the graph operated according to the bearing of the ship in operation The traffic ship information was directly displayed on the location of the corresponding ship by integrating the RADAR and AIS data as shown in Figure (b) This feature is more intuitive than existing navigation equipment in determining information of traffic ships; however, there were cases when the overlay on the position of the other ships actually made recognition difficult The ECDIS data was displayed the ENC symbols as shown in Figure (c) Since the same size symbols were used, it was found that there were difficulties in identifying its distance Meanwhile, the bearing and distance information to the next waypoint calculated from the route plan data were displayed according to the ownship information 28 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality Figure 5: Screenshot of Experiments (a) simulation test, (b) sea test Figure 6: Results of experiment (a) ownship information, (b) trafficship information, (c) ECDIS information 4.3 Analysis of User Survey After the simulator experiment, a survey was carried out on the officers with regard to the satisfaction level for each function of the augmented reality system The survey questions were composed of scale items and miscellaneous opinion items Analysis of the survey responses reveals that levels of satisfaction regarding the ownship information was highest in items asking the satisfaction level for each function, as shown in Figure 7, while the level of satisfaction of the traffic ship information was relatively low Additionally, the level of satisfaction for the entire system showed high points for efficiency and effectiveness, while legibility and clarity obtained lower than average points The relatively low level of satisfaction for legibility and clarity was thought to be due to the difficulty in distinguishing information, as the information overlapped in multitudes when other ships were concentrated near the horizon during port entry and departure The officers who participated in the experiment were interviewed They gave mostly positive feedback with regard to the effectiveness of the augmented reality navigation equipment; however, they also expressed concern with regard to the additional workload necessary to use the new navigational equipment Additionally, for the information display method in the augmented reality image, responses varied depending on the inclination or preference of the respondent It is evident from the data that development of user interfaces optimized for each user is necessary 29 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality (a) (b) Figure 7: Survey results of AR navigation system (a) satisfaction of functions, (b) satisfaction of AR system V Conclusion Although various types of navigation equipment with advanced technology are available to help enhance operation safety for navigators, further research on the method of efficiently displaying and servicing information rather than indiscriminately providing a bounty of information is necessary In this regard, this paper proposed a navigation aid system based on augmented reality technology, and the proposed system provides various overlaid navigation information on images from cameras to support swift and accurate decision-making by officers The ship-handling simulation and real vessel application experiments were carried out to verify the functionality of the system Through our experiments, stable operation of the prototype hardware and software in a real ship environment was verified In a survey conducted after the experiment, most participants gave positive feedback with regard to the augmented reality navigation aid system but expressed concern with the increase in workload from the new navigation equipment There were also cases when registration errors occurred during augmented image registration due to location errors from the GPS, and elements that need improvement were identified, including difficulty in distinguishing data due to excessive amount of information displayed on the horizon As research and development continue, various navigation aid systems based on transparent displays will be further developed and utilized in the bridge to maximize the effect of augmented reality technologies Such use of augmented reality technology is expected to prevent marine accidents and increase ship operation efficiency in the future Submitted: March 8, 2016 Accepted: June 25, 2016 VI Acknowledgements The Contents of this paper are the results of the research project of Ministry of trade, industry & energy (10041790, Development of Advanced Ship Navigation Supporting System based on Oncoming International Marine Data Standard) 30 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality References B&G (2015), http://www.bandg.com Jeong, J S (2012), “A Study on the User Requirements for the Support of a Safe Navigation of the Incoming Vessels to Port Waterways and the Outgoing Vessels”, Research report of Korea Research Institute of Ships and Ocean Engineering, pp 73-83 Jung M (2015), “Analysis of User Requirement for the Improvement of ECDIS to Enhance Navigational Safety and Work Efficiency”, Journal of Navigation and Port Research, Vol 39, No 3, pp 141-147 Jung M (2016), “User Requirement Analysis of ECDIS for the Development on S-Mode Guideline”, Journal of Navigation and Port Research, Vol 40, No 3, pp 89-95 Kim K H (2008), “Application of Head-Up-Display Technology to Telematics”, Electronics and Telecommunications Trends, Vol 23, No 1, pp 153-162 Mads Bentzen (2015), “ACCSEAS Final Report” Ministry of Oceans and Fisheries (2015), “Causal analysis of marine accident”, http://www.mof.go.kr Olivier Hugues (2010), “An Experimental Augmented Reality Platform for Assisted Maritime Navigation”, ACM Virtual Reality International Conference, pp 87-92 SafeTechResearch (2013), “Maritime Safety Audit for New Marina Terminal of Busan North Port” There is no conflict of interest for all authors 31 ... Interface of AR Navigation System 24 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality In addition, respondents showed positive responses toward the augmented reality system. .. specifications of each module are as follows 25 OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality Figure 3: Configuration of Navigation Aids System based on AR (a) Data... PARK, KWON / Advanced Navigation Aids System based on Augmented Reality (a) (b) Figure 7: Survey results of AR navigation system (a) satisfaction of functions, (b) satisfaction of AR system V Conclusion