Another Switch Glitch

 We discovered this morning that one of the replacement switches that we installed in the maintenamce overnight eight days ago was not delivering consistent power to at least one, possibly more, network AP radios, causing customers to lose service. Upon investigation, we had also lost our primary configuration interface for the switch. Fortunately we had recovered one of the failed switches from two weeks ago and were able to reconfigure it and deploy. Seems fine so far.

Relief

We are happy to report that the provisional fix we described below--temporary switches and power injectors--has been completely successful. All network nodes are up and all customers should have access.

When the replacement switches arrive next week, we will announce a maintenance window for their installation.

Thank you all for your patience.

Interim Update

After polling our regular suppliers and scouring the web, we have located two appropriate replacement switches...in Lithuania!!  They should be here in about a week.

In the meantime, we're attempting to piece together a provisional system with some older simple switches and separate power supplies for virtually every radio in the system. With considerable effort and a good dose of luck, we could have most folks back up today.

We will update as things progress.

MELTDOWN

 We began Tuesday morning with the intention of replacing the 2.4 sector that had been acting up over the last several weeks. We ended the day with one of our two primary network switches partially functional and the other barely alive. As a result, three quarters of network customers are down.

It is still not clear precisely what caused the damage to the switches, and we will not know how quickly we can get replacements until later this (Wednesday) morning. We will post when we know more.

 Flaky AP

Over the last several days we have seen an issue develop with our Sector H2 Access Point radio. This causes brief disconnects of the nine clients on that sector with variable frequency.

After investigating and checking all probable causes short of replacing the entire unit on the tower the problem persists. We have ordered a suitable radio to be express shipped from California.

A Very Chaotic Day of Tower Work and More

 

We started out today with a primary goal of restoring the Hanson tower to Flinner barn point to point link, and additionally planned to replace a compromised cable to a dedicated service radio on the tower. By mid-afternoon both of those goals had been accomplished but not before a surprise network outage between around 10:30 and 11:45 in the morning. It appears that the root cause of some of our recent issues was a series of rolling port malfunctions in a core layer three switch; just such a problem caused a loss of connectivity to our primary backhaul radio. Fortunately, once we determined the problem--totally unrelated to our cabling work in process--we were able to bring online a redundant pair of backhaul radios already in place.

We have ordered a replacement switch on overnight delivery, and have relocated some equipment to available ports on another switch to guard against any recurrence before the new switch is in place.

The overall damage:

Several Old Town Road and Ripton village center customers experiencing choppy service through complete outage from approximately mid-day Saturday through 2:00pm this afternoon. A complete outage for all customers from approximately 10:30 to 11:45 this morning.

We apologize for the disruptions and truly appreciate your patience and understanding.

LINK DOWN

We lost the radio link from Hanson to the Old Town Road barn late this morning and have been unable to restore it from the ground. We do not have a tower climber available until Monday, unfortunately, so it will be mid-day Monday before it's restored.  This means that Ripton customers on Old Town Road and in the immediate Village area will be without service until then.   We apologize for the anticipated long outage and will post here when and if the situation changes.  Customers should remember that they can always check the NETWORK STATUS LINE at (802) 382-8700 for the most immediate updates.