We started out today with a primary goal of restoring the Hanson
tower to Flinner barn point to point link, and additionally planned to replace
a compromised cable to a dedicated service radio on the tower. By mid-afternoon
both of those goals had been accomplished but not before a surprise
network outage between around 10:30 and 11:45 in the morning. It appears that
the root cause of some of our recent issues was a series of rolling port malfunctions
in a core layer three switch; just such a problem caused a loss of connectivity
to our primary backhaul radio. Fortunately, once we determined the problem--totally
unrelated to our cabling work in process--we were able to bring online a redundant
pair of backhaul radios already in place.
We have ordered a replacement switch on overnight delivery,
and have relocated some equipment to available ports on another switch to guard
against any recurrence before the new switch is in place.
The overall damage:
Several Old Town Road and Ripton village center customers experiencing
choppy service through complete outage from approximately mid-day Saturday
through 2:00pm this afternoon. A complete outage for all customers from
approximately 10:30 to 11:45 this morning.
We apologize for the disruptions and truly appreciate your
patience and understanding.