Upstream Fiber Update

We have been assured that the FirstLight Fiber (DNS) issue has been mitigated. We have migrated most services that we can to a new set of FirstLight DNS resolvers. However, we are seeing some residual issues--some devices failed to migrate as they should have, and others with static IPs and DNS do not seem to be performing optimally on the old resolvers. We are addressing these issues on an individual basis.

Upstream Fiber Issue

We have been seeing some instability on our FirstLight upstream fiber feed, starting around 8:00pm this evening. The FirstLight NOC is aware of the issues and working to resolve them. Customers may be having difficulty reaching Internet destinations at the present time.

Ripton Power Mostly Up

As of late today, it appears that almost all of Ripton has grid power restored, although we are seeing a few stragglers apparently without power. We still have one minor AP at the Selden Mill location down without power. Other than that all nodes remain up and running.
(The Selden Mill location came back online with grid power early Monday evening).

Grid Power Still Mostly Down for Ripton

Most Ripton customers are still experiencing only sporadic connection to grid power. All NBN nodes are staying up on alternate power except the Leeds-Grip link into the village center area, where we still have limited backup battery reserve. We will be moving more robust battery/inverter capacity there from the school sometime in the next several weeks, as that capacity is no longer needed at the school.
Again, with the exception of customers in that limited area, and a single cutomer relayed through the Selden Mill Road repeater,customers with alternate power for their receiving units and routers should be able to connect.

We're Fine, but Power's Up, Down, Up, Down, UP....

RIPTON: Yesterday afternoon and last night's heavy snow caused widespread power outages with some periods of restoral, followed by outages again. It now appears that power was restored again to most of Ripton at about 11:55am. In any case, almost all Ripton customers with backup power have been able to stay connected as backup power at all of our major nodes performed as expected. Customers should remember that if they can keep their receiving units and routers powered with small battery backups they can stay connected through these power outages.
   The forecast calls for more heavy snow and mixed precipitation right through to the weekend, so we may see more of this....be prepared!

Ripton Link Weather Sensitivity Continues

We lost the W. Cornwall to Ripton link briefly (around 10 minutes, shortly before 6:00pm) again this evening and have recovered it and reconfigured it manually to a lower data rate in the interest of stability.  This link is obviously demonstrating some sensitivity to these marginal freezing temperature weather events.  We will be watching it closely and manually adjusting it until weather permits a total link replacement. Thanks for your patience during this time.

Another Wobble, more Serious

This afternoon the Cornwall to Ripton backhaul link (same link as in the previous post from mid-November) actually went down briefly at around 3:30pm today. It came back up rather quickly, but it took some time to retune it to a perfectly stable lower modulation rate at a little after 5:00pm. It appears that we had a rare weather anomaly--severe icing on the uphill end, perhaps. It seems to be recovering signal strength slowly, and we will be monitoring it closely overnight and bumping up the modulation rate if possible. In the meantime, Ripton customers will see somewhat slower rates and video streaming may be more problematic. We will post an update in the morning.
We are finishing up an upgrade plan that includes increasing bandwith to Ripton by a factor of 10, and implementing backhaul redundancy as well.  We'll be looking for investors at various levels to help make that happen...fair warning to deep pocketed customers concerned about the availability of quality, reliable, locally supported service!

Big Link Wobbles

We have been noticing a gradual drop in traffic volume from the Ripton area lately. It turns out the microwave link quality from W. Cornwall to the Hanson tower had been deteriorating. We were able to adjust power and modulation on that link this morning and quality is significantly improved, but we have accelerated a plan for replacement of the entire link with new, higher capacity equipment. Stay tuned for details.

Emergency Maintenance

We were forced to take down our edge routers for emergency reprovisioning shortly after Midnight this morning, causing a lack of access for all users. Work continued for the next several hours but service was fully restored well before dawn. We apologize for the lack of warning.

Packet drops

We have become aware of a small and variable amount of packet loss recently. The problem appears to be on the ethernet side at Oxford of the Bicentennial to Oxford link and appears to be on the tower. We are monitoring the situation.

LIGHTNING OUTAGE REPORT

This was a big one. Tuesday evening's violent thunderstorms included a direct strike on the W. Cornwall tower and significant damage to radios atop Bicentennial Hall. We were able to restore service through the night into Wednesday morning, but damaged gear in both locations failed again making diagnosis and repair even more problematic, and it was well into last evening before service was fully restored.

This season's expansion plans include redundant ring structure for our backbone radios which will make the possibility of this kind of outage nearly impossible. We continue to work to make NBN your best option for broadband service. Thanks for your patronage.

PLEASE REMEMBER TO CALL 382-8700 FOR CURRENT NETWORK STATUS  IF YOU ARE EXPERIENCING PROBLEMS!

Fiber Outage

One of our two fiber providers had a widespread outage--"most of Vermont"--between approximately 9:10 and 10:08am this morning. We were unable to shift all of our users over to our other provider by the time of restoral, so some users would probably have suffered almost nearly an hour without service. We are in the process of configuring the network so that all customers will automatically and nearly seamlessly move to the remaining upstream provider before this kind of rare event is likely to recur.

"Sweaty Backhaul" Revisited

We blew it--we missed the diagnosis. The instability issue in the Tuesday post persisted, intermittently, for the next two days. The problem turned out to be a faulty router power supply that was dropping power output level intermittently, which caused very confusing routing and authentication issues that appeared variously to be configuration problems, radio interference, or ethernet connectivity issues. We eventually solved the problem with a replacement of the power supply late Thursday afternoon.
Our New Years Resolution and promise to all: We are redesigning for all-DC power distribution and monitoring for all power parameters at major nodes, with automated alarm thresholds., to be instituted completely in 2014.

Sweaty Backhaul

Yesterday's severe temperature and humidity swing caused a major link from the W. Cornwall tower to the Ripton tower to become unstable, causing most Ripton customers to experience a "choppy" connection. The problem grew worse over the course of the day, and it was 1:30am this morning before we were finally able to stabilize it. We apologize for any inconvenience this may have caused, and thank you for your patience.