We started out today with a primary goal of restoring the Hanson tower to Flinner barn point to point link, and additionally planned to replace a compromised cable to a dedicated service radio on the tower. By mid-afternoon both of those goals had been accomplished but not before a surprise network outage between around 10:30 and 11:45 in the morning. It appears that the root cause of some of our recent issues was a series of rolling port malfunctions in a core layer three switch; just such a problem caused a loss of connectivity to our primary backhaul radio. Fortunately, once we determined the problem--totally unrelated to our cabling work in process--we were able to bring online a redundant pair of backhaul radios already in place.
We have ordered a replacement switch on overnight delivery, and have relocated some equipment to available ports on another switch to guard against any recurrence before the new switch is in place.
The overall damage:
Several Old Town Road and Ripton village center customers experiencing choppy service through complete outage from approximately mid-day Saturday through 2:00pm this afternoon. A complete outage for all customers from approximately 10:30 to 11:45 this morning.
We apologize for the disruptions and truly appreciate your patience and understanding.
