Last Week

Early last week, our RADIUS server* started experiencing memory sector failures, which accelerated over the next several days. This occurred at the worst possible time, as I was out of town and also out of cell coverage, a very rare circumstance, and the problem was technically beyond support resources left here.

Although we had the problem controlled to some extent by Wednesday evening, allowing almost all users to connect owing to constant micromanagement, it was actually not until Friday mid-day that we had a more durable--but still temporary--fix in place.

We are now in process of building two new RADIUS servers, and will have redundancy and failover to the backup going forward. We will continue to host the first in our rack, and are trying to decide on a remote location for the backup. This will increase the reliability of the system by several orders of magnitude.

In addition, we will be addressing the problem of high level local backup support. As always, we thank you for your patience and understanding.

* Our RADIUS service stores subscriber credentials and other data. Every time a user attempts to log  in, an intermediary authentication device must check with the RADIUS server to validate the user's credentials before allowing it to connect to the network. Most of our voice service customers connect differently and would not have been affected, but we had a couple voice services connecting using RADIUS for monitoring, and those were impacted.

FIBER FU

Last evening, around 7:00pm, when many Riptonites were sitting down to an important School Board presentation, a fiber demarc device of our fiber provider for the Middlebury-Ripton link failed. Normally, we expect immediate notification from that provider in such an eventuality, since we do not have "visibility" into their network segments. Absent that notification, we wasted many hours troubleshooting, tearing down, and reprovisioning multiple devices on our network before finally concluding, by process of elimination, that the problem was not on our end. Support from our fiber provider was able to confirm the failure of their equipment and locate it, and we were able to help them restore connectvity shortly after 3:00am.

Later today, we will confer with that provider's engineering staff, confirm that normal emergency notification procedures are now in place, and ask them to support our process for automatic restoration.

Additionally, we continue work to add a replacement wireless link into Ripton for total redundancy.

Glitch Reboots

This morning, Sunday November 1st, between 9:24 and 9:48, we experienced three reboots of a principal Ripton router causing three brief outages of less than three minutes each to Ripton subscribers. We apologize--the reboots were caused by a misconfiguration which was entirely our error. The problem was corrected and network operation is now fully normal.

TRANSITION

Yesterday, under duress, we completed a major transition. As of a few minutes before Noon, September 1st, we are backhauling service to Ripton through our own fiberoptic drop on Old Town Road. This eliminates the W. Cornwall wireless site which has been so problematic recently, causing the weekend outage of late July as well as a pair of much briefer interruptions.

Unfortunately, yesterday once again the W. Cornwall site owners failed to meet their contractual commitments; some of our equipment there was powered down a few minutes before midnight Monday night, even though there was an obligation to power it for another week. We had planned to announce a maintenance window in the early morning hours of either Wednesday or Thursday to complete the transfer during a time when an outage would have the least impact on our customers. As a result of the situation in W. Cornwall, we were forced to effect the transition on the fly Tuesday morning, and did not complete it until a few minutes before Noon.  All of our Ripton customers would have been unable to connect until then.

Please accept our sincere apologies for the difficulties of this transition, and thank you all for your loyalty.  Please know that these difficulties are behind us; the new fiber feed will be not only much more reliable, but is already bringing significantly more bandwidth up the mountain. It is also easily scalable, unlike the former path, so we can continue to increase bandwidth as the need arises. Additionally, we now begin work on implementing another high capacity wireless link into Ripton, so that we can provide a unique level of redundancy there.

We will continue to fight to provide your best Internet option. Once again, thanks to all of our Ripton customers for your patience and your patronage.