Machine Learning for Automated Detection of Shipwreck Sites from Large Area Robotic Surveys

Past Expedition

Dates

May 23 - June 3, 2022 / June 5 - 23, 2023

Location

Atlantic Ocean

Overview

Due to its maritime history and strategic location, Thunder Bay National Marine Sanctuary contains almost 100 known shipwreck sites, with over 100 shipwrecks still left to be found. Shipwrecks can help us better understand our past, but discovering and exploring them is expensive, time-consuming, and labor intensive. By advancing and training the capabilities of marine robotic systems to search for and survey shipwreck sites autonomously, scientists aim to increase the efficiency and decrease the costs associated with such exploration efforts. In 2022 and 2023, a multi-institute team of researchers conducted expeditions in Thunder Bay to do just that.

Figure 1: Map of survey regions A, B, C, and D, where C and D are exploratory areas. Shipwreck sites and exploratory sites from year 1 and year 2 are displayed. *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 194 KB).

Part 1: May 23 – June 3, 2022

The main goal of the first year of field expeditions was to collect datasets to develop machine learning methods for shipwreck detection from sonar imagery. The team also conducted preliminary imaging surveys for close-range inspection of shipwreck sites with robotic systems.

Field expeditions were focused in Thunder Bay National Marine Sanctuary (TBNMS), which is unique in its abundance of known shipwreck sites of varying type, size, and wreck conditions. The team conducted eight shore-based day operations throughout the sanctuary on R/V Storm between May 23-June 3, 2022. Each mission consisted of traveling to the initial starting point, deploying robotic systems, and conducting surveys in a local area. When the mission was complete, we moved to the next planned site.

To collect side-scan sonar data, we deployed Michigan Technological University’s IVER-3 autonomous underwater vehicle (AUV). Surveys covered a total of 8.04 square kilometers (3.1 square miles) over the course of field trials. Figure A shows an example of the high-resolution side-scan sonar imagery we collected with the AUV over the Grecian site.

Figure A. Grecian site imaged by the IVER autonomous underwater vehicle equipped with side-scan sonar. — Figure A: *Grecian* site imaged by the IVER autonomous underwater vehicle equipped with side-scan sonar. *Image courtesy of Machine Learning for Automated Detection of Shipwreck Sites from Large Area Robotic Surveys.* Download largest version (jpg, 1.36 MB).

Working with maritime archaeologists at the sanctuary, we created high-resolution labels of shipwreck sites, distinguishing between background, ships, debris, and moorings. Figure B shows a sample label. Labeled data allows us to train and evaluate machine learning methods for detecting shipwreck sites in side-scan sonar imagery.

Figure B. Grecian site imaged by the IVER autonomous underwater vehicle equipped with side-scan sonar. — Figure B: Example label of side-scan sonar imagery to create a machine learning dataset for shipwreck detection. Labeling was supervised by maritime archaeologists who are familiar with the sites. *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 1.36 MB).

We also conducted preliminary deployments of a remotely operated vehicle (ROV) for close-range imaging surveys of shipwreck sites. Over the course of the expedition, we had a total of 13 ROV deployments with three different ROV platforms. With data collected by the ROV, we are able to create large-scale 3D reconstructions of shipwreck sites. Figure C shows initial 3D reconstructions obtained from data collected by the ROV.

Figure C. Preliminary 3D reconstruction of Monohansett obtained with data from an ROV. — Figure C: Preliminary 3D reconstruction of *Monohansett* obtained with data from an ROV. *Image courtesy of iCORE Lab, Louisiana State University.* Download largest version (jpg, 402 KB).

The data collected during the Year 1 field expeditions will lay the foundation for developing and training machine learning methods in Year 2 of the project.

By Katie Skinner, Assistant Professor, University of Michigan

Part 2: June 5 – 23, 2023

Building on the previous year’s work, the objectives of the second year of this expedition were to validate and demonstrate our shipwreck detection and exploration pipeline, and to continue to collect sidescan data of shipwreck sites in Lake Huron. In addition, a secondary mission was to perform exploratory searches for shipwrecks in areas with high potential for new discovery.

Field work took place in the Thunder Bay National Marine Sanctuary. Figure 1 shows an overview of the planned survey regions, though modifications were made to these survey regions to accommodate weather and vehicle operating status each day.

Operations consisted of shore-based day expeditions between June 5-9, June 13-16, and June 20-21, 2023, during which we visited 15 sites. Each mission consisted of traveling to the initial waypoint, deploying robotic systems, and conducting surveys in a local area. When the mission was complete, we moved to the next planned waypoint. For remotely operated vehicle operations, we used moorings to ensure safe operation.

As a collaboration between the University of Michigan, Louisiana State University, and Michigan Technological University, the full detection pipeline involved an initial survey from an IVER3 autonomous underwater vehicle (AUV) equipped with a side-scan sonar followed by our novel shipwreck detection model which produced GPS coordinates of potential wreck sites (see Figure 2).

Figure 2: Georeferenced side-scan sonar image of Barge No. 1 and the shipwreck detection model prediction polygon (in green). — Figure 2: Georeferenced side-scan sonar image of *Barge No. 1* and the shipwreck detection model prediction polygon (in green). *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 682 KB).

These coordinates were passed along to an autonomous planning and navigation algorithm on a BlueROV, which was used to collect up-close camera and depth data for 3D reconstruction. Figure 3 shows the survey path that was automatically generated by algorithms developed by the Louisiana State University team.

Figure 3. Path planned for autonomous survey of Barge No. 1. — Figure 3. Path planned for autonomous survey of *Barge No. 1.* *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 72.6 KB).

Once the survey was planned, the modified BlueROV, Dory, was deployed to conduct an autonomous survey of the site. Figure 4 shows deployment of the Dory AUV from the R/V Storm research vessel, and Figure 5 shows imagery collected during an autonomous survey of the Barge No. 1 shipwreck site.

Figure 4: AUV Dory is deployed from R/V Storm for an autonomous shipwreck survey. — Figure 4: AUV *Dory* is deployed from R/V *Storm* for an autonomous shipwreck survey. *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 4.94 MB).

Figure 5: Sample imagery collected during an autonomous survey of Barge No. 1. — Figure 5: Sample imagery collected during an autonomous survey of *Barge No. 1.* *Image courtesy of Field Robotics Group, University of Michigan.* Download largest version (jpg, 374 KB).

Overall, this field expedition successfully demonstrated the full pipeline developed by the multi-institute team to deploy robotic platforms to automatically detect shipwreck sites from sidescan sonar imagery, and then perform planning and navigation to carry out autonomous surveys of detected shipwreck sites. The collected data will be used to compute 3D models of surveyed sites.

By Anja Sheppard (Ph.D. Student, University of Michigan) and Katie Skinner (Assistant Professor, University of Michigan)

Published February 8, 2024