Tag Archives: qa

Just because it’s production, doesn’t mean it’s not a test

Posted on by 0 comment

Back in July I wrote about the week long sailing trip that ended after 1 day with engine failure and dramatic action.  Since then our old sailboat has been stuck in Anacortes, WA while the local marine service company diagnosed and repaired the engine.  My wife also delivered our first child during that time so we were a little busy anyway.  They declared the engine good to go last week and I scheduled sea trials and pickup for Tuesday (8/24).  We packed a cooler full of food, some clothes, sleeping bags and drove up to Anacortes to meet the boat.  After a slightly expensive lunch at the marina restaurant, with masterful drinks poured by the same bartender we were served by the last time, we met the Travelift about to splash the boat.

The engine fired up just fine and sounds much better than it used to.  It runs and idles smoother, doesn’t smoke, runs cooler, etc.  So we headed out for sea trials in the bay and cruised around for about 45 minutes at different RPMs, heating and cooling the engine to stress it a little looking for any problems.  The boat runs great!  Under engine power we move about 1 knot faster than before too.  I think the engine has been running poorly for quite a while before it failed.  Anyway, satisfied that the boat engine was performing well, we headed back in to the dock.  I paid the bill, we loaded out provisions and headed out with just enough time to make it to Deception Pass for slack tide.

In the immortal words of Captain Ron — “Well, the best way to find out is to get her out on the ocean Kitty, if anything’s gonna happen, its gonna happen out there.”

20 minutes out, a new sound develops from the engine compartment.  It sounds like metal rattling–a very distinct, sharp sound.  Down in the engine compartment it’s a very loud sound, an exhaust leak from somewhere.  A couple phone calls and we turn back to Anacortes.  We clearly aren’t making it to Deception Pass tonight.  The engine is not quote right yet.  Mechanic shows up and determines that the head gasket is leaking, might have been a defective gasket.  But it’s solid copper and a new one is several days away.  Another mechanic joins us at 8:30am the next morning and finds out that the head bolts loosened during the sea trial and subsequent motoring.  He tightens then up and its running fine again.  So out for another trial, then back to cool the engine and check the bolts again–still good!

So we finally leave Puget Sound’s own Bermuda Triangle for home.  We pass through Deception right on time and continue south towards Coupeville on Whidbey Island.  In another moment of calamity on our eternal 3-hour tour, we are moving along at over 6 knots when the boat suddenly stops dead in the water and pitches forward.  Jason who was in the galley, flies forward into the head and falls down while dishes go flying.  A quick check of the depth sounder (showing 2.8ft) confirms my fears..  we hit a sand bar.  It turns out the navigator (me) was too preoccupied on his cell phone dealing with plans for the night and talking to the car dealer about the Mazda’s coolant leak, to notice that we were about 100 yards outside the marked channel.  Reversing the engine does nothing to help and the current is pushing us against the sand bar pretty hard.

If you read the previous post, you’ll remember that the dinghy saved the day when the engine failed..  Well, another notch on the dinghy’s stern is due after I threw it off the bow, mounted the Yamaha motor, and used it as a mini tugboat to spin the sailboat around into the current to push off the sand bar.  I’m contemplating renaming the sailboat and dinghy to “The Problem” and “The Solution” respectively.

We made it safely to Coupeville and had a wonderful afternoon and evening.  My wife and baby drove over to meet us for dinner and the next morning we shoved off early for Everett.  We got a little wet on this last run due to rain but made it home safe, locked the boat down, hopped in the car and went home.  It’s a series of mini-adventures I will never forget.

On the plus side, our little old sailboat is now better equipped, I have a new found respect for the dinghy, and I got to go boating once more before summer ends, even if it did cost us a lot more money than we had planned.

Every Cruise is a Shakedown Cruise (in IT terms, every Production environment is also a QA environment)

Posted on by

It’s the morning of day 2 on a 7 day sailing trip in the San Juan Islands of Puget Sound.  We are 43 nautical miles from our homeport, and I’m sitting at the table watching a diesel mechanic take apart the little engine on our boat.

Over the 4th of July weekend, we spent nearly 3 full days getting the boat ready for this trip.  Washed inside and out, installed new convenience items, changed the oil, checked the transmission fluid, batteries, electrical systems, etc.  We taken several short and long trips with our Cal 2-29 over the past 5 years and there hasn’t been a single trip over 24 hours that didn’t require a repair of some kind.  Once, the bilge pump sucked water INTO the boat and we had to re-plumb the bilge pump system with makeshift hoses available at the nearby port.  Another time, while docking in Friday Harbor, my wife leaned too hard on a stanchion, causing it to break off and sending her into the cold Puget Sound water.  Twice, an over-zealous helmsperson switched from reverse to forward gear while the engine was at speed and tore the flex coupling on the prop shaft in half.  Both times we were close to docking so we just drifted into port and made repairs.  After that we thought we had finally seen the last of the major issues for a while.

On Tuesday morning, we left too early to fuel up so I brought a 5-gallon can of Diesel on board.  35 nautical miles later that proved to be a good idea, when we almost ran out of fuel, while navigating the tight and dangerous Deception Pass.  We refueled without stopping using a makeshift funnel made out of a plastic water bottle.  Afterwards, the engine was clearly turning more than 2000 rpm based on sound and boat speed but the tachometer was showing 600-800 and bouncing wildly.  Something to look at later since the engine seemed okay.

An hour later, on the west side of Fidalgo Island, entering the Strait of Juan de Fuca, we were planning our final destination for the day when the engine began to lose power for an unknown reason. Finally, we saw what seemed to be unusual black smoke from the exhaust.  At that point we shut down the engine to check on things.  We were a few hundred yards from a rock wall, which was cause for some concern, but we had a little time to assess the situation.

At first glance, the alternator belt was very loose but it didn’t make sense because the bolt that allows for adjustment had clearly not moved.  It turned out that the bolt on the other end of the mounting arm, the one that secures the arm to the engine block, had sheared off and the arm was free of the engine.  Since the engine is an old diesel, which does not require any power or electronic systems to run, we decided we’d try and remove the belt and go without the alternator until we can repair it.  We also found a few random bolts and screws in the engine compartment.

While working to secure the belt out of the way with zip-ties we noticed the starter solenoid had pretty much fallen off of the starter, the spring was visible even.  The bolts had come loose and one was missing, plus reattaching would require a lot of work due to the location of the bolts.  Well, being a single cylinder small diesel, the Farymann A30M can be started with a hand crank when warm, so we secured the solenoid out of the way and figured we’d fire it up with the crank and get to a nearby marina.

Hand cranking failed to produce a running engine, and we really don’t know why, we may have needed the glow plug on which we forgot about until a long time after giving up.  It was looking like we were going to have to call Vessel Assist, when I remembered a story I heard about someone pushing their sailboat with their dinghy lashed to the side of the boat near the stern.  So we secured the dinghy, fired up the Yamaha 2.5hp motor, and amazingly we were moving along at 4knots just in time to move away from the rock wall that was now only about 100 yards away.  An hour later we dinghy-motored our little 35 year old Cal into Flounder Bay on the northwest corner of Fidalgo Island.  Some steaks, corn on the cob, and a healthy dose of Captain Morgan over the next few hours helped the mood and the day was done!

At this point we’ve found that not only was the alternator and starter solenoid loose from the engine, one of the two engine mounts was about 30 minutes of running from falling off also.  It’s likely the loose engine mount added vibration, which caused the other bolts to loosen, causing more bolts to fail completely–a multi-stage failure of sorts.  Today, our goal is to work with the marine service tech to get the engine put back together and tightened up, then see if the engine will run, and assess anything we find there.  At $92.50 per hour, this could be a costly day.

This experience, and the previous ones we’ve had as well, reminded me that you need to be prepared for anything, especially when your life depends on it.  When your customers (internal or external) depend on your IT systems, you should be prepared for anything to go wrong, and you might have to patch things together to get it going until you can fix it the right way.  And that’s okay.  Remember, duct tape and zip-ties can pretty much fix anything!  😉

And it’s only been 24 hours since the trip started.

Follow up here