Early last week, our RADIUS server* started experiencing
memory sector failures, which accelerated over the next several days. This
occurred at the worst possible time, as I was out of town and also out of cell
coverage, a very rare circumstance, and the problem was technically beyond
support resources left here.
Although we had the problem controlled to some extent by
Wednesday evening, allowing almost all users to connect owing to constant
micromanagement, it was actually not until Friday mid-day that we had a more
durable--but still temporary--fix in place.
We are now in process of building two new RADIUS servers,
and will have redundancy and failover to the backup going forward. We will
continue to host the first in our rack, and are trying to decide on a remote
location for the backup. This will increase the reliability of the system by
several orders of magnitude.
In addition, we will be addressing the problem
of high level local backup support. As always, we thank you for your patience
and understanding.
* Our RADIUS service stores subscriber credentials and
other data. Every time a user attempts to log in, an intermediary
authentication device must check with the RADIUS server to validate the user's
credentials before allowing it to connect to the network. Most of our voice
service customers connect differently and would not have been affected, but we
had a couple voice services connecting using RADIUS for monitoring, and those
were impacted.