Mid-Day Outage

Our outage today--approximately 90 minutes between 11:00am and 12:30pm lasted longer than it should have, because we initially mis-diagnosed it.

Because it presented exactly like the upstream fiber outage earlier this month, we wasted time having FirstLight confirm the condition of their handoff to us. Only after they were able to verify it did we look inside the network to discover we had no power to our principal server cabinet in Ripton. It appears that we suffered a premature failure of an AC breaker there, which was easily corrected once we discovered the problem. We are addressing methods of prevention and faster diagnosis of any future occurrence. This will include some combination of redundant AC circuitry, more robust UPS capacity, and networked signalling of power events.

Event Summary

At approximately 6:00pm last evening our upstream provider, FirstLight, suffered a discontinuity in an underground location in South Burlington. As a consequence we immediately lost our Internet connectivity. We were in touch with the FirstLight Network Operations Center immediately. They had technical personnel locating and working to repair immediately. The fault was between an underground vault and a utility pole drop in deeply frozen ground. As a consequence, the ground and conduit had to be thawed with steam in order to guard against further damage, and the process was slow. It was 8:20am this morning by the time the fiber repair was complete.

Update from FirstLight Network Operations Center, 11:26pm:

"Crews remain on site. OTDR was shot and it places the issue underground in a vault. Suspected frozen conduit. No ETR at this time another crew is standing by as they work to dig out the vault and gain access to the conduit. This likely may be a while as steam may be required to get to the heart of the problem.

MAJOR FIBER CUT

Our Upstream feed from FirstLight is down. The FirstLight NOC is currently trying to locate a fiber cut. As soon as we know more, we will post here, and on the Voice Status Line (802) 382-8700.