Skip to content

The Noise Floor and What To Do With It

Image

In previous articles (here and here), I have attempted to steer you away from trying to locate sources of interference and provided a framework for troubleshooting to test for and isolate problems stemming from noise or interference. I thought it would be good to next demonstrate an example of how this procedure can be put to use. So allow me to spin a terrifying yarn… a true story of a WISP that experiences an unexplained sector outage at the start of a holiday weekend.

To review, the term ‘noise floor’ is defined in Wikipedia as:

“The measure of the signal created from the sum of all the noise sources and unwanted signals within a measurement system, where noise is defined as any signal other than the one being monitored.”

To paraphrase, the noise floor is the murky, nasty stuff you want your operating frequencies to stay well above. Ideally, you want the noise floor to hover around -100 dBm or lower. Always. The greater the noise floor, the greater the level your legitimate operating frequencies need to be to guarantee optimal performance of your links. So what happens when the level of the noise floor changes to your network’s detriment?

Observe the spectrum analysis below. It is a snapshot from a dynamic baseline of an access point operating at 924.8 MHz:

As you can see, this sector was already experiencing less-than-ideal noise levels, mainly due to antenna height (70 meters) and its directionality. However, almost all customers were close enough to the tower that the receive signal strength (RSS) on both sides was well above the manufacturer’s recommended level for noise tolerance (20 dBm). Even with fluctuating interference (the red line) up to -65 dBm, average throughput was consistently tested and verified at the promised rates offered by the WISP’s service agreement.

Now for the scary stuff:

A week later, the same radio’s spectrum analysis was reporting an unprecedented 20 dBm rise in the noise floor, effectively severing all customer connections. So what to do? Run to the hills or stand and fight? The crux here was to quickly rule out anything within my domain of control that could be the cause so that I could start formulating a plan to solve the crisis. To do this I used the troubleshooting process I previously described in the Noise Detection article.

GENERAL ASSESSMENT

Scale: Single 90º sector at 924.8 MHz affecting ~20 customers within 2 km of 70m 4-sector tower.
Internal factors: No known changes or additions to the WISP customer’s wireless network.
External factors: No known tower (or other) construction projects in customer’s coverage area.
Timing: Interestingly enough but ultimately unhelpful, the abrupt noise level increase was recorded shortly after midnight on Canada Day (July 1).

TROUBLESHOOTING PROCESS

Step Test Result
1 Baseline analysis 20 dBm increase in noise floor in upper 900 MHz band
2 Configuration changes No changes to hardware or software within the 24 hr period in which problem occurred
3 Customer feedback Unnecessary; effects to affected customers already known (unable to connect)
4 Radio configuration Configuration correct
5 Radio operation and monitoring AP radio nominal operation; monitoring shows RSS flatline for affected CPEs, SNR levels through the roof
6 Hardware inspection and component swapping All hardware visually inspected and components swapped with spares; no effect
7 Parallel system testing Spare radio with panel antenna attached detected similar noise levels at original antenna height (70m). Noise floor levels decreased slightly at each 10m step down to 30m.

CONCLUSION AND SOLUTION

This real-life example describes every wireless operator’s worst nightmare in all-too-vivid detail. And as nightmares usually go, deploying the solution was no easier than admitting the cause of the problem. Given the affected sector’s height and the fact it was pointing into a densely populated area, it was concluded that the noise floor was caused by a newly installed, unknown system transmitting from somewhere in an approximate 250 km2 area south of the tower. As such, the sector’s channel was retired and the remaining three sectors were rotated to compensate. Service to 98% of customers was restored with similar or better performance.

Advertisements

Noise and Interference: Detection and Troubleshooting

Mitigating noise and interferenceObviously you know you have a problem when the support desk phones start ringing, or link monitor statistics begin to display wildly errant values such as high latency or SNR. But what or who is the culprit? The wireless troubleshooting techniques described here should only be followed once you have ruled out problems at the tower side such as flakey cabling, flapping switch or router ports, routing issues, false positives (ie. an unusual, but normal increase in user traffic), or unscheduled upstream or backhaul provider outages.

Following a logical troubleshooting procedure limits the downtime needed to test a wireless issue and will help you arrive at the solution sooner. So first, think about and note the following. By answering the questions below, you will find one area remains for which to focus your attention. And as I always stress, if you feel overwhelmed, do not feel shy calling your favorite IT consulting firm for help.

SCALE

Is the problem affecting the whole network, or a specific tower, sector or link? Or maybe a small group of customers in a single area are affected?

INTERNAL FACTORS

Are you aware of any recent configuration or equipment changes made to the affected portion of the network? If so, what are they?

EXTERNAL FACTORS

What’s going on in the environment? What is the adjacent competition doing, if anything? Are you aware of any new construction that could be presenting a new obstacle within the coverage area or link experiencing the issue?

TIMING

Is there a certain time that the problem occurs or started to occur? Is there a pattern to its occurrence or is it more random in nature?

Next, you will want to start with gathering and analyzing all the data at your disposal before assuming the system needs to be taken offline to conduct testing. Developing a test plan based on factual data will greatly reduce your scheduled maintenance window’s period, or prevent it altogether.

      1. Baseline analysis: check RSS, SNR, scan, latency and performance baseline data (log files, MRTG, Solarwinds, etc.). Note the average value for each, as well as the dates and times any dramatic changes in these values occurred.
      2. Changes: check with colleagues to determine if any changes in equipment and/or configuration have occurred since the times noted in step #1 above.
      3. Customer feedback (placed at step 3, but you may want to move it further down on the list, depending on your situation): If you’re a WISP operator, you are already fully aware of how slippery the slope can get when gaining customers’ perspectives. However this should not be overlooked, especially if the problem is affecting multiple customers. Correlating affected customers’ stories can quickly eliminate false information while solidifying the facts. Particular attention should be given to asking customers questions regarding the use of wireless equipment within their own home and immediate vicinity.
      4. Check radio configuration: access the affected radio(s) using the connection method with which you are familiar (telnet, HTTP). Check the obvious areas such as firmware version, power level, frequency, IP configuration (address, gateway, DNS, SNMP, SNTP, etc.), modulation schemes (if applicable), SSIDs, authentication (static, RADIUS, TACACS, keys), encryption, QoS, VLANs, etc.
      5. Check radio operation and monitoring tools’ data: link activity (TCP/IP stats such as Rx and Tx packets), errors (discarded and retransmitted packets), conduct wireless performance and ping tests and collect current RSS and SNR readings; compare these to the baseline in step 1.
      6. Spectrum scanning: requires taking the production system offline for the time it takes to perform the scans. Using the radio’s built-in spectrum analyzer is the most practical and unobtrusive means of sampling what’s going on in the RF environment. If the radio does not include a spectrum analyzer but the installation consists of a RF cable run into an enclosure or customer premises, a stand-alone spectrum analyzer (Anritsu, HP) can usually be attached to the antenna with the correct adapters (be sure to compensate for any loss added to the path between the radio and antenna). The benefit of using a stand-alone spectrum analyzer is its flexibility and ‘real-time’ viewing aspect.
      7. Hardware: inspect all connections for corrosion, water ingress, tearing, bending, cracking, etc. If suspect components are found, begin swapping out the easiest pieces (RF jumpers, connectors, filters, lightning arrestors, PoEs), retesting as per step 5 and 6 above after each change. Last to try swapping is the radio itself; verify the correct configuration and firmware is loaded on the swap unit prior to the exchange.
      8. Parallel system testing: at this point there are only a couple of components that remain to be tested; the tower or customer RF cable (LDF4-50, LMR400) and antenna. Since these items are not easily swappable, parallel system testing makes sense. It is best to conduct this testing using the same equipment as is installed in the production system. However, many operators and field services make use of a pre-configured ‘test rig’ that is relatively easy to deploy to save time. In this case, calculations should be done to compensate for the loss or gain in the path as a result of testing with dissimilar equipment before comparing to a baseline. Also, the active system’s radio(s) should be disabled during testing using the parallel system to prevent masking the results. If possible, it is highly recommended to perform tests in both vertical and horizontal polarities and at different elevations. If the parallel test system shows similar characteristics to the affected installed system, noise mitigation will be necessary (see next section). Make sure you have recorded all stats, scans and other test results so as to determine the course of action (ie. swapping out a faulty sector antenna) that will best reduce the effects of noise or interference as well as minimize downtime.

Locating Sources of Noise In Your Wireless Network… NOT!

Mitigate your RF noise and interference

OVERVIEW

If you have installed, audited, managed, serviced, or had anything remotely to do with broadband radios operating in the unlicensed ISM band (including WiFi), chances are you share an intimate relationship with noise and interference. Of course, both exist in the licensed world as well, but usually with much less of an impact. Each has a separate meaning but yield the same results: calls from customers complaining about their service. After all, it usually isn’t the radio’s hardware designers and engineers that are stuck with trying to figure out why performance has tanked or find the cause of an outage. It’s the WISP operators, staff or IT consultants who are looking for a practical way to solve an immediate problems.

Unfortunately there is no magic bullet – this is the mysterious and enigmatic world of wireless we’re talking about. However, you should be able to take what you have learned and methodologically apply it to your unique situation in order to lessen the negative effects of wireless noise, improving your customers’ overall performance. Most importantly, you’ll cement your reputation as a miracle worker!

ATTEMPTING TO LOCATE SOURCES OF INTERFERENCE

After numerous failed attempts, we have concluded that trying to locate sources of interference (ie. drive testing), especially when it comes to unlicensed frequency bands, is mostly a waste of time. The amount of effort put into trying to locate an interferer far exceeds any possible favorable outcome, for reasons that are explained below.

  1. Any affected sector(s) (ie. on which you have paying customers) must be disabled during the location process to help reveal and hone in on any potential source of interference that is mostly masked by said sector(s). Whether operating in an advertised maintenance window or an unexpected service outage, you can expect your inbound support call volume to rapidly increase as a result.
  2. The system behind the source of the inference needs to be fairly active in order to be detected and then confirmed. If the effects of the interference on your sector (or sectors) are at their worst during business hours it is probably due to the nature of the traffic propagating from the offending system. This emphasizes the first point above. You won’t be very popular for taking a coverage area offline mid day to scan for noise, regardless of how much the interference is affecting performance. End users usually take ‘slow’ a lot easier than ‘down’.
  3. Once you locate the source of the offending signal, who says the operator of the system needs to cooperate? Although FCC and (to a certain extent) CRTC regulations dictate operators of radios transmitting in the unlicensed bands need to play nice, little if any action is taken, nor do penalties exist to hand out to parties accused of inhibiting one’s wireless services. Note that this can also apply to licensed bands as well. Licensing of non-military designated frequencies is usually nothing more than a public registry to map out who gets to operate what frequency where.
  4. Lastly, the very nature of wireless is dynamic. This not only applies to fluctuating RSSI and SNR levels, but also to the center frequencies any wireless network operator chooses to use at any given time. In the most extreme case, we witnessed an operator who changed the operating frequencies of all the radios across his entire network on a weekly basis in attempts to improve performance or simply navigate around fluctuating noise concerns in his coverage area. Radios with their frequency hopping feature switched on further underscore this point.

In the end, it is best to develop a frequency plan based on solid data (path profiles, coverage plots, site survey measurements, radio and antenna specifications, etc.) and try to stick with it once deployed. Changes to the original design should only be made if they are absolutely necessary; that is, when all efforts to mitigate the noise or interference have been exhausted. Remember that changing the frequency of one sector could ultimately affect all sectors throughout your entire system, resulting in a mind-warping frequency juggling act that may have been best left alone. Be aware that interference is a two-way street. So extend the RF olive branch to your neighbors. Mutual cooperation will help improve one another’s network performance and stability, hence reducing cost and maximizing return on investment.

Finally, if you feel out of your depth, get an IT consulting firm who has wireless experience to help – it’s an investment you won’t regret.