Mid-Day Outage

Our outage today--approximately 90 minutes between 11:00am and 12:30pm lasted longer than it should have, because we initially mis-diagnosed it.

Because it presented exactly like the upstream fiber outage earlier this month, we wasted time having FirstLight confirm the condition of their handoff to us. Only after they were able to verify it did we look inside the network to discover we had no power to our principal server cabinet in Ripton. It appears that we suffered a premature failure of an AC breaker there, which was easily corrected once we discovered the problem. We are addressing methods of prevention and faster diagnosis of any future occurrence. This will include some combination of redundant AC circuitry, more robust UPS capacity, and networked signalling of power events.

Event Summary

At approximately 6:00pm last evening our upstream provider, FirstLight, suffered a discontinuity in an underground location in South Burlington. As a consequence we immediately lost our Internet connectivity. We were in touch with the FirstLight Network Operations Center immediately. They had technical personnel locating and working to repair immediately. The fault was between an underground vault and a utility pole drop in deeply frozen ground. As a consequence, the ground and conduit had to be thawed with steam in order to guard against further damage, and the process was slow. It was 8:20am this morning by the time the fiber repair was complete.

Update from FirstLight Network Operations Center, 11:26pm:

"Crews remain on site. OTDR was shot and it places the issue underground in a vault. Suspected frozen conduit. No ETR at this time another crew is standing by as they work to dig out the vault and gain access to the conduit. This likely may be a while as steam may be required to get to the heart of the problem.

MAJOR FIBER CUT

Our Upstream feed from FirstLight is down. The FirstLight NOC is currently trying to locate a fiber cut. As soon as we know more, we will post here, and on the Voice Status Line (802) 382-8700.

Power outage

Most of a foot of wet and heavy snow from the early season storm of the last two days resulted in a four hour grid power outage in Ripton this evening--from around 3:11pm to 7:11pm. Major network nodes and most customers were unaffected, but limited battery life at the Heppell and Flinner barn relays meant downstream customers in the village center and Bread Loaf areas were out for the last hour or more of that period.

Wet n' Wild Windstorm

We survived yesterday's windstorm with little damage. There were a few brief outages at more remote sites owing to power interruptions at minor relay points, but the only significant problem occured shortly before Noon  at the Pearl Lee location. Cabling to the tree-mounted 900 AP there was whipped and torn by the gale force winds, resulting in an outage for three customers near the intersection of North Branch Road, the Lincoln Road, and Pearl Lee Rd. Weather and parts logistics kept us from restoring service until about 1:20pm today.

LIGHTNING DAMAGE

We lost a power distribution unit supplying almost all infrastructure at the Hanson tower location early this morning. It failed at about 6:50am, almost certainly because of cumulative  damage from yesterday and last night's rounds of severe thunderstorms. The fact that it took down multiple network devices simultaneously-including our primary Ripton backhaul link- foiled normal alerting and slowed diagnosis. We were able to retrieve and configure a backup PDU and restore service to most Ripton customers at approximately 11:45am. A single AP failed to come back up, and it was close to 1:00pm before the final handful services was restored. We apologize for the outage and thank our customers for their understanding.