New Link in a Hurry, Upgrade Report

Around 1:30pm this last Friday (11/1), the backhaul radio link from Peddlars Bridge to Old Town Road in Ripton failed. After informing and  consulting with the six affected customers in the village area , rather than expend time and energy to repair a link that we had scheduled for replacement, we elected to accelerate configuration and testing of two new high throughput radios and schedule the build for Saturday. Good weather allowed us to build a complete new high capacity direct link from the Hanson tower to the Leeds barn on Old Town Road and commission it by few minutes after 5:00pm on Saturday. We were also able to prepare cabling for replacement of the Leeds to Grip link sometime in the next week after new radios for that link arrive. Shortly after that we will deploy a new AP at the Grip location serving Village customers. This will significantly increase capacity in and out of the Village area.

At this point, we are happy to report that we have replaced all but one of the original 900 MHz radios served by the three original sector APs on the Hanson tower with our new high-capacity beam-forming solution. The replacement of the Grip AP and associated village clients, mentioned above, will mark the replacement of the second of the four older branch 900 APs and clients. At that point nearly 90% of our Ripton customers will be migrated to one of the new high speed platforms!

Rite of Autumn

In what has begun to seem like a tradition with the first frost on the mountain, one of the oldest network links, the Leeds to Grip radio required a reboot this morning. Four customers were without service from 6 through shortly after 8am this morning. Service was restored with the reboot. The radio responsible is scheduled for replacement before the snow flies.

Radius Blip

Our Radius authentication system became unstable around 9:00am this morning, and new logon requests were being refused as a result. A reboot of the Cornwall concentrator server at approximately 9:10 solved the problem, but caused a brief interruption of service to most customers. Service is now fully normal.

TELJET OUTAGE

We suffered an extended outage from our principal upstream provider, TelJet Longhaul, from approximately 10am through 2:25pm today. Although we have contracted significant additional bandwidth from another fiber provider, Level3, we are still in process of implementing BGP and load balancing in our Network Operations Center, and were unable to switch the majority of customers over before TelJet restored service. This incident is precisely why we have gone to multiple providers and we will accelerate the completion of our network reconfiguration so that this cannot happen again.

Incidentally, TelJet was quite recently acquired by Tech Valley Communications and we suspect that this outage is related to the integration of those systems. We are--unhappily--still waiting for a response from their support team.

VoIP Problem

Our handful of VoIP (Voice Service) customers are experiencing problems with incoming calls. Callers to those numbers are receiving busy signals, even when customers are on hook (not using the phone). We expect resolution shortly from our voice hosting partner, and will post as soon as we have an update.

Link Restored

It took longer than expected, but repairs to the link damaged last evening were finally completed at 4:30 this afternoon, restoring service to the five affected customers in the village area of Ripton.

Lightning Strike

Last evening's violent lightning storms took out one radio link in the Ripton network.  At 5:10pm the Peddlars Bridge to Old Town Road Link went down. We are replacing damaged radios and expect to have the link up this (Saturday) morning, restoring service to the five customers affected.