Power Again

Power to our main server rack in Middlebury was interrupted again this morning a few minutes before 9am--compounded by a battery failure in one of our rackmount UPSes . Once more, this appears to be related to ongoing electrical work elsewhere in the building, but the battery failure was key. We were able to restore power and full network availability by 10:20am.
We just ordered a new "hot spare" backup UPS and replacement batteries and should have them in place this weekend.

Email Server Issue

GMAVT is having problems with the email server this morning. At this time we are unable to send or receive ripton-coop.net and nbnworks.net emails. As of two minutes ago, they report that they are working on the problem and we expect resolution shortly.

This issue was resolved within 30 minutes.

POWER!

Power to our main server rack in Middlebury was interrupted this morning--incredibly, both our main grid power AND our backup power. This appears to be related to ongoing electrical work elsewhere in the building, and we are still working out the details. Fortunately, we were able to restore power and full network availability by 10am.
We will continue to investigate the causes and take appropriate measures.

Concentrator Issue

We had a problem this evening with a routing device at the W. Cornwall tower which required an onsite fix. All subscribers might have had trouble connecting for a period of one to two hours.

Post Hurricane Irene Report

We're pleased to report that the network weathered the week's disasters without incident, although a handful of users at the far end of secondary links with limited backup battery power would have been unable to connect, owing to the extended power outages. We will be fitting those secondary relay locations with more robust battery backups before the end of the year.
Almost all users who had power--including their own generator backup--would have had uninterrupted access. We did see a handful of burned or defaulted routers owing to the grid power flickers. This a good time to remind all that a good quality UPS with a healthy battery in it will generally prevent these problems.

Our hearts go out to all of our neighbors who suffered--and the many who continue to suffer--from the terrible losses of this event.

For current road, bridge, and river status Google has stepped up with good online tool, which you can find here:

http://crisislanding.appspot.com/?crisis=2011_flooding_vermont

Micro-Outage

We experienced an outage this afternoon between approximately 4:51pm and 4:59pm. Our W. Cornwall backhaul radios were unreachable for this period, and this would have caused a system-wide outage. We recovered them with automatic processes in place. We are investigating the causes.

WELCOME

Welcome to the North Branch Networks Status Log. We will post news of significant network events here. This is the place to check for any network issues if you have Internet access. If not, call (802) 382-8700 for current status, particularly for any outage information. Any news is posted there first, and events are generally documented here more fully somewhat later.

PERFECT STORM

From Monday morning through 6:45pm Tuesday evening we experienced the longest, most widespread outage in our five-year history. A mega-snowstorm, which prevented vehicle access to our W. Cornwall tower location until it was cleared with a front end loader early Tuesday morning, a concealed equipment problem on the Battell roof pushed over the edge by the drifted snow there, and finally, a defect in a brand new unused spare backhaul radio we attempted to deploy to replace the unit at Battell conspired to lengthen our concerted efforts at restoration.

The only good news is that the discovery of the problem on the Battell roof has already solved a couple of nagging minor network issues, including the past month's problem with the W. Cornwall link, and has improved network performance perceptibly.

We are aware aware that constant broadband connectivity is a necessity, and that this kind of event is absolutely unacceptable. We are evaluating the lessons learned and implementing policy changes, which, in combination with this year's backhaul redundancy plan, will make a recurrence of this disaster impossible. Thank you for your continued patronage and support as we move forward.