The Engineering Department and ROV Team spent the better part of several days fabricating equipment to allow quick and safer repairs.

The Engineering Department and ROV Team spent the better part of several days fabricating equipment to allow quick and safer repairs. Click image for larger view and image credit.


Technicians work to repair the satellite antenna in the VSAT dome.

Technicians work to repair the satellite antenna in the VSAT dome. Click image for larger view and image credit.


Hurry up and wait!


Jeremy Potter
Expedition Manager
NOAA Office of Ocean Exploration and Research

camera icon View a slideshow of various images taken depicting onboard preparations for the ROV leg of the expedition.

I have a nasty habit of incessantly checking my BlackBerry when I’m preoccupied with something at work. Final preparations for a major expedition certainly qualify. That’s probably why it took all of about 5 seconds from the time my phone buzzed until I saw the foreboding message from ship’s Commanding Officer (CO): “EX VSAT Failure”

EX is NOAA’s short-hand for the NOAA Ship Okeanos Explorer. The VSAT is a large satellite antenna that is the basis for our telepresence capability. The VSAT is essential equipment that allows us to send data and information - including ROV video from the seafloor – to anyone with an internet connection around the world. This capability allows most of the Scientist Team to participate from shore. Only a very few scientists are actually onboard the ship.

Losing our telepresence capability would easily have a cascade of implications for the Expedition. Since I was just about to step onto my first flight, I had plenty to think about. How bad is it? Can it be repaired before we sail on July 7? What do we do if it can’t?

Timing of the failure couldn’t have been much worse. It was the Saturday just before the July 4th holiday. Personnel joining the ship were already leaving home to catch flights. The manufacturer and key trouble-shooting/repair contacts were off for the long weekend. We were supposed to leave a remote port in less than 4 days. Based on single email, I was willing to bet that we’d be sailing without any telepresence capabilities.

By the time I arrived in Puntarenas, a few key members of the ROV and Telepresence Teams were already on board and working with ship personnel to trouble-shoot the system. The hope was to gather as much preliminary information as possible to have it ready when the manufacturer and their technical personnel were available on July 5th. The more organized and proactive we were the better chance we’d have of minimizing any lost time at-sea. No one likes sitting around in port.

Fast forward to the evening of July 10 . . .the EX was still at anchor offshore Puntarenas. The VSAT was still down. Twice daily conference calls with the ship, shore-side support, and manufacturer representatives had been occurring for five days. Every time we thought we were close to a fix, something else popped up and surprised us. Repair technicians who originally told us they could fix it in a few hours had already spent at least 50 hours up in the VSAT dome. Every hour that passed by was another hour of time lost at-sea. Ship and mission crew were tired and frustrated.

Just before the ship had to get underway, repair technicians successfully fixed one of the key problems. Though it wasn’t enough to enable telepresence, on board mission personnel implemented a contingency plan to provide partial capability.

Just before the ship had to get underway, repair technicians successfully fixed one of the key problems. Though it wasn’t enough to enable telepresence, on board mission personnel implemented a contingency plan to provide partial capability. Click image for larger view and image credit.


As the ship got closer and closer to a go/no-go departure decision. VSAT repairs stretched well into the early morning hours.

As the ship got closer and closer to a go/no-go departure decision. VSAT repairs stretched well into the early morning hours. Click image for larger view and image credit.


Eventually we couldn’t wait any longer, the sailing board was set for departure at noon on July 11 . . .working VSAT or not. With the deadline looming, the repair technicians found and fixed a key problem. Unfortunately, it wasn’t enough to enable telepresence. Time had run out, the technicians went to shore and the ship got underway.

Fortunately the story doesn’t end there. About 1-hr after weighing anchor, Telepresence Lead Webb Pinner ambled into Forward Lounge with a slight smirk on his face. He’d just worked out an elegant contingency plan. Though far from the desired solution, he successfully restored some passable capability that even the repair technicians flown in from the States didn’t even consider.

Things may not always work out like we want; but we continue to succeed because we’ve got people that just get stuff done.

Sign up for the Ocean Explorer E-mail Update List.