Tuesday, May 21, 2019

Lightning Damage

The severe lightning of Sunday night (5/19) took out a minor point to point link between the Hanson tower and the Heppell location, cutting service to six customers, primarily in the Breadloaf area. Unfortunately, the damage was to an older radio, for which we do not have replacements on hand. We immediately ordered more current replacement radios for the link, which are arriving later today (Tuesday), along with a backup reserve unit. We will change out units on both the tower and Peddlars Bridge ends of the link tomorrow, and shoud have service restored by mid-day.

Saturday, April 6, 2019

Blip, Scheduled Maintenance

We experienced some grid power anomalies last night around 11:10pm, and another interruption and failure to reboot of key equipment in our core rack. This caused a network outage until we manually restored power slightly less than an hour later. We had already planned and announced a maintenance window to upgrade our UPS equipment in that rack, scheduled for about 16 hours from now (3:00-5:00am, Sunday 4/7/2016). We expect this upgrade will supply conditioned bridging power for both minor power fluctuations and for the brief disruption of AC power during the automatic transfer to backup generation. Further, in the event of a full shutdown and simultaneous hard reboot of all rack equipment, it should be able to supply full startup peak power, which the previous implementation appears to have failed to do, dating from the relocation of our primary RADIUS server to that location.

We are taking the opportunity to update firmware and add some functionality to our core routing server during the maintenance window.

Friday, March 22, 2019

Mid-Day Outage


Our outage today--approximately 90 minutes between 11:00am and 12:30pm lasted longer than it should have, because we initially mis-diagnosed it.

Because it presented exactly like the upstream fiber outage earlier this month, we wasted time having FirstLight confirm the condition of their handoff to us. Only after they were able to verify it did we look inside the network to discover we had no power to our principal server cabinet in Ripton. It appears that we suffered a premature failure of an AC breaker there, which was easily corrected once we discovered the problem. We are addressing methods of prevention and faster diagnosis of any future occurrence. This will include some combination of redundant AC circuitry, more robust UPS capacity, and networked signalling of power events.


Wednesday, March 6, 2019

Event Summary


At approximately 6:00pm last evening our upstream provider, FirstLight, suffered a discontinuity in an underground location in South Burlington. As a consequence we immediately lost our Internet connectivity. We were in touch with the FirstLight Network Operations Center immediately. They had technical personnel locating and working to repair immediately. The fault was between an underground vault and a utility pole drop in deeply frozen ground. As a consequence, the ground and conduit had to be thawed with steam in order to guard against further damage, and the process was slow. It was 8:20am this morning by the time the fiber repair was complete.

Tuesday, March 5, 2019

Update from FirstLight Network Operations Center, 11:26pm:

"Crews remain on site. OTDR was shot and it places the issue underground in a vault. Suspected frozen conduit. No ETR at this time another crew is standing by as they work to dig out the vault and gain access to the conduit. This likely may be a while as steam may be required to get to the heart of the problem.
MAJOR FIBER CUT

Our Upstream feed from FirstLight is down. The FirstLight NOC is currently trying to locate a fiber cut. As soon as we know more, we will post here, and on the Voice Status Line (802) 382-8700.