Email Server Certificates

OOPS! We missed the first annual renewal of our GeoTrust certificates--we thought that they would renew automatically, but we were wrong. Holders of nbnworks.net and ripton-coop.net email addresses are likely getting warnings from their email programs about the expired certificates, but most should be able to click through the warnings and connect. The problem should disappear tomorrow, or Tuesday at the latest.

New Link in a Hurry, Upgrade Report

Around 1:30pm this last Friday (11/1), the backhaul radio link from Peddlars Bridge to Old Town Road in Ripton failed. After informing and  consulting with the six affected customers in the village area , rather than expend time and energy to repair a link that we had scheduled for replacement, we elected to accelerate configuration and testing of two new high throughput radios and schedule the build for Saturday. Good weather allowed us to build a complete new high capacity direct link from the Hanson tower to the Leeds barn on Old Town Road and commission it by few minutes after 5:00pm on Saturday. We were also able to prepare cabling for replacement of the Leeds to Grip link sometime in the next week after new radios for that link arrive. Shortly after that we will deploy a new AP at the Grip location serving Village customers. This will significantly increase capacity in and out of the Village area.

At this point, we are happy to report that we have replaced all but one of the original 900 MHz radios served by the three original sector APs on the Hanson tower with our new high-capacity beam-forming solution. The replacement of the Grip AP and associated village clients, mentioned above, will mark the replacement of the second of the four older branch 900 APs and clients. At that point nearly 90% of our Ripton customers will be migrated to one of the new high speed platforms!

Rite of Autumn

In what has begun to seem like a tradition with the first frost on the mountain, one of the oldest network links, the Leeds to Grip radio required a reboot this morning. Four customers were without service from 6 through shortly after 8am this morning. Service was restored with the reboot. The radio responsible is scheduled for replacement before the snow flies.

Radius Blip

Our Radius authentication system became unstable around 9:00am this morning, and new logon requests were being refused as a result. A reboot of the Cornwall concentrator server at approximately 9:10 solved the problem, but caused a brief interruption of service to most customers. Service is now fully normal.

TELJET OUTAGE

We suffered an extended outage from our principal upstream provider, TelJet Longhaul, from approximately 10am through 2:25pm today. Although we have contracted significant additional bandwidth from another fiber provider, Level3, we are still in process of implementing BGP and load balancing in our Network Operations Center, and were unable to switch the majority of customers over before TelJet restored service. This incident is precisely why we have gone to multiple providers and we will accelerate the completion of our network reconfiguration so that this cannot happen again.

Incidentally, TelJet was quite recently acquired by Tech Valley Communications and we suspect that this outage is related to the integration of those systems. We are--unhappily--still waiting for a response from their support team.

VoIP Problem

Our handful of VoIP (Voice Service) customers are experiencing problems with incoming calls. Callers to those numbers are receiving busy signals, even when customers are on hook (not using the phone). We expect resolution shortly from our voice hosting partner, and will post as soon as we have an update.

Link Restored

It took longer than expected, but repairs to the link damaged last evening were finally completed at 4:30 this afternoon, restoring service to the five affected customers in the village area of Ripton.

Lightning Strike

Last evening's violent lightning storms took out one radio link in the Ripton network.  At 5:10pm the Peddlars Bridge to Old Town Road Link went down. We are replacing damaged radios and expect to have the link up this (Saturday) morning, restoring service to the five customers affected.

Hidden Problem Corrected, Upgrade Report

We had been noticing a perceptible slowdown in network performance and some stream buffering over the last few weeks. Nothing was showing up in any of our monitoring, however, except a generally diminished volume of traffic which we were inclined to attribute to glorious Spring weather. Wrong!---online use never slows down. Yesterday we discovered a subtle routing problem and corrected it, and speeds and volumes have recovered. We are continuing to raise the performance bar, and have now rolled out equipment to essentially triple last year's subscription speeds to a substantial portion of the network, and are on track to reach 100% well before the end of the year.

Email Server Restored

The email server is back up. Thanks for your patience.

Email Server Down

The email server for nbnworks.net and ripton-coop.net accounts has been down for the last 90 minutes. Engineers are working to restore function, and we will update here as soon as we have more information. Thanks for your patience.

NAS glitch

Around 8:10 last evening, a problem emerged with our Network authentication concentrator in West Cornwall. We were able to diagnose and correct this by 8:40 or so, but customers may have experienced an interruption of service between those times. We are taking steps to prevent a recurrence, and we thank you for your patience and understanding.

Core Rebuilding

We had an overnight scheduled maintenance early Monday morning, announced for 1am through 5am,  for major Network Operations Center changes, including wholesale replacement of our principal firewall and reconfiguration of our Radius server. Unfortunately, several problems--most resulting from what appears to be a bug in the new firewall software--prevented us from having full function of the Radius server until shortly after 5pm Monday. Although this did not not affect access for most customers, some customers would not have been able to log back in to the network from the logged off state during this time. We apologize for the disruption.

Outage and Repair

At very close to Midnight last evening we experienced a near complete network blackout owing to ice damage to cabling outside our Network Operations Center in Middlebury. Repairs were finally complete at approximately 6:10am.
Thank you for your patience and a belated Happy New Year!