Another Switch Glitch

 We discovered this morning that one of the replacement switches that we installed in the maintenamce overnight eight days ago was not delivering consistent power to at least one, possibly more, network AP radios, causing customers to lose service. Upon investigation, we had also lost our primary configuration interface for the switch. Fortunately we had recovered one of the failed switches from two weeks ago and were able to reconfigure it and deploy. Seems fine so far.

Relief

We are happy to report that the provisional fix we described below--temporary switches and power injectors--has been completely successful. All network nodes are up and all customers should have access.

When the replacement switches arrive next week, we will announce a maintenance window for their installation.

Thank you all for your patience.

Interim Update

After polling our regular suppliers and scouring the web, we have located two appropriate replacement switches...in Lithuania!!  They should be here in about a week.

In the meantime, we're attempting to piece together a provisional system with some older simple switches and separate power supplies for virtually every radio in the system. With considerable effort and a good dose of luck, we could have most folks back up today.

We will update as things progress.

MELTDOWN

 We began Tuesday morning with the intention of replacing the 2.4 sector that had been acting up over the last several weeks. We ended the day with one of our two primary network switches partially functional and the other barely alive. As a result, three quarters of network customers are down.

It is still not clear precisely what caused the damage to the switches, and we will not know how quickly we can get replacements until later this (Wednesday) morning. We will post when we know more.

 Flaky AP

Over the last several days we have seen an issue develop with our Sector H2 Access Point radio. This causes brief disconnects of the nine clients on that sector with variable frequency.

After investigating and checking all probable causes short of replacing the entire unit on the tower the problem persists. We have ordered a suitable radio to be express shipped from California.

A Very Chaotic Day of Tower Work and More

 

We started out today with a primary goal of restoring the Hanson tower to Flinner barn point to point link, and additionally planned to replace a compromised cable to a dedicated service radio on the tower. By mid-afternoon both of those goals had been accomplished but not before a surprise network outage between around 10:30 and 11:45 in the morning. It appears that the root cause of some of our recent issues was a series of rolling port malfunctions in a core layer three switch; just such a problem caused a loss of connectivity to our primary backhaul radio. Fortunately, once we determined the problem--totally unrelated to our cabling work in process--we were able to bring online a redundant pair of backhaul radios already in place.

We have ordered a replacement switch on overnight delivery, and have relocated some equipment to available ports on another switch to guard against any recurrence before the new switch is in place.

The overall damage:

Several Old Town Road and Ripton village center customers experiencing choppy service through complete outage from approximately mid-day Saturday through 2:00pm this afternoon. A complete outage for all customers from approximately 10:30 to 11:45 this morning.

We apologize for the disruptions and truly appreciate your patience and understanding.

LINK DOWN

We lost the radio link from Hanson to the Old Town Road barn late this morning and have been unable to restore it from the ground. We do not have a tower climber available until Monday, unfortunately, so it will be mid-day Monday before it's restored.  This means that Ripton customers on Old Town Road and in the immediate Village area will be without service until then.   We apologize for the anticipated long outage and will post here when and if the situation changes.  Customers should remember that they can always check the NETWORK STATUS LINE at (802) 382-8700 for the most immediate updates. 

Same Radio, Fixed

 We were able to find a wiring fault causing the problems with the Hanson to Flinner link and repair it, restoring full service to the village and Old Town Road locations at about 4:30pm this afternoon.

Thanks to all for your patience.

Same Radio, Cont'd

 We continue to have (escalating) problems with the same radio. We are going onsite to see if we can effect a solution from the ground.

Another Blip

 Same radio just flapped and reassociated at half duplex. We got into it and locked it at 100FDX. We'll see how this goes.

Radio Problem

In the last half hour we discovered and mitigated a problem in a single radio link--the connection from Hanson to Flinner on Old Town Road. This appears to have caused intermittent connection problems for customers in the immediate Ripton village area and on Old Town Road beginning at around 11:30. The radio was not properly autonegotiating connection speed with our switch at Gigabit speed and we were able to manually set the ethernet speed to 100FDX temporarily, restoring a stable connection. This probably indicates a cabling or ethernet port problem on the radio. We are planning on being on the Hanson tower later in the week and will be investigating further.

 

Yesterday's Email Issues Resolved

Cluster B - Email Login

Resolved - Email services continue to run smoothly and services are fully restored.

Incident Start Time: 06-24-2020 15:45:00 EST
Incident End Time:06-24-2020 19:43:00 EST

Total Duration: 4 hours

We apologize for the email service interuption for nbnworks.net and ripton-coop.net email users and thank you for patience and understanding.

Email Server Issue

The eMail server handling both nbnworks.net and ripton-coop.net email accounts is currently not authorizing account holders. Techs are working to correct the issue.

eMail Server Problem Resolved

Our eMail server partners have finally effected a fix to the server authentication problems. Webmail, POP/SMTP, and IMAP should all be functioning for nbnworks.net and ripton-coop.net email accounts.

eMail Authentication

We are experiencing authentication problems for eMail. The problem is being addressed and we expect a resolution shortly. In the meantime, users of nbnworks.net and ripton-coop.net email accounts my be experiencing problems receiving email messages.

Fiber Repaired

As of approximately 5:45am this morning, Internet connectivity to our network was restored. Approximately three hours later, FirstLight Fiber messaged:

"At this time field services are hands off and we believe that the fiber outage in Burlington VT should be resolved as of 8:35am 11/2. REASON FOR OUTAGE Riverbed scouring due to flooding from heavy rains ripped out the 144 from its conduit, along with a couple of other 144 count fibers"

Apparently extensive fiber splicing through the night was required to complete the repair. As this is the second occurence of this type of outage in a single year, we will be strongly advocating for FirstLight to configure a failover to a feed from the south, through Rutland, which appears to be achievable with existing fiber routes.

Thanks to our valued customers for your patience and understanding.

MAJOR FIBER OUTAGE

The complete fiber feed into the network has been down since approximately 8:30 this morning. The situation is confusing and FirstLight as well as several other networks are experiencing multiple outages across New England due to the extremely high winds and heavy rain. The proximate cause of our outage appears to be due to flooded conduit in the Burlington area which has taken out multiple networks south of Burlington. We still have no estimated time of restoral from FirstLight. We will update the Status Line at (802)382-8700 as soon as we have more information and this site soon thereafter. Thanks to all for your patience and understanding.

Lightning Damage

The severe lightning of Sunday night (5/19) took out a minor point to point link between the Hanson tower and the Heppell location, cutting service to six customers, primarily in the Breadloaf area. Unfortunately, the damage was to an older radio, for which we do not have replacements on hand. We immediately ordered more current replacement radios for the link, which are arriving later today (Tuesday), along with a backup reserve unit. We will change out units on both the tower and Peddlars Bridge ends of the link tomorrow, and shoud have service restored by mid-day.

Blip, Scheduled Maintenance

We experienced some grid power anomalies last night around 11:10pm, and another interruption and failure to reboot of key equipment in our core rack. This caused a network outage until we manually restored power slightly less than an hour later. We had already planned and announced a maintenance window to upgrade our UPS equipment in that rack, scheduled for about 16 hours from now (3:00-5:00am, Sunday 4/7/2016). We expect this upgrade will supply conditioned bridging power for both minor power fluctuations and for the brief disruption of AC power during the automatic transfer to backup generation. Further, in the event of a full shutdown and simultaneous hard reboot of all rack equipment, it should be able to supply full startup peak power, which the previous implementation appears to have failed to do, dating from the relocation of our primary RADIUS server to that location.

We are taking the opportunity to update firmware and add some functionality to our core routing server during the maintenance window.

Mid-Day Outage

Our outage today--approximately 90 minutes between 11:00am and 12:30pm lasted longer than it should have, because we initially mis-diagnosed it.

Because it presented exactly like the upstream fiber outage earlier this month, we wasted time having FirstLight confirm the condition of their handoff to us. Only after they were able to verify it did we look inside the network to discover we had no power to our principal server cabinet in Ripton. It appears that we suffered a premature failure of an AC breaker there, which was easily corrected once we discovered the problem. We are addressing methods of prevention and faster diagnosis of any future occurrence. This will include some combination of redundant AC circuitry, more robust UPS capacity, and networked signalling of power events.