Separation of Ambient Radio Noise and Radio Signals Received via Ionospheric Propagation

Abstract

Systems for atmospheric research and wireless communication use the High Frequency (HF) radio spectrum. At these frequencies, typically up to 20 MHz, the ambient electromagnetic noise is stronger than the noise generated by the receiver itself, thereby limiting the sensitivity of the instruments. Especially in urban areas, the noise level is high. In remote rural environments, where artificial noise sources are absent, a much lower noise level is observed. It has been shown that this noise arrives via ionospheric propagation and consists of impulsive noise from lightning and a background component that resembles additive white Gaussian noise. To establish the absolute field strength of this background noise component, a direction- and polarization-agnostic antenna is realized by adding the power of two orthogonal antenna elements in the digital domain. To suppress radio signals arriving via ionospheric propagation—of which the spectral and temporal aspects are not known a priori—a novel adaptive filter is demonstrated that separates the background noise from the radio signals in the joint frequency-time domain. This method is demonstrated using measurements from a polarimetric experiment on 7 MHz in a remote rural area in Catalonia. The results are submitted to the International Telecommunication Union for the validation of ambient noise models.

1. Introduction

At frequencies below approximately 20 MHz, knowledge of the level of the ambient electromagnetic noise is essential for the design and planning of long-range radiocommunication systems and scientific instruments for observation of the upper atmosphere. At these frequencies, the ambient noise is significantly higher than the noise generated in the receiving equipment, and therefore, determines the sensitivity limit of the system [1] (p. 766). Ambient noise originates in natural sources, such as lightning [2], the Sun [3] (pp. 37–44), planets [4,5], the Milky Way and other extraterrestrial bodies [3] (p. 115), [6], and in artificial sources, such as combustion engines, electronic and electrical appliances.

1.1. Ambient Electromagnetic Noise Models

To facilitate radio system design, the International Telecommunication Union (ITU) provides empirical models of the ambient noise [7]. Some of these models are based on measurements performed in the 1960s and 1970s [8]. The ambient noise levels are frequency dependent, and they may differ per country and region, due to differences in industrialization, population density [9], building attenuation [10], regulations, etc. In light of the increase in the number of electronic and electrical devices, which are the major source of unwanted electromagnetic noise emissions [11], since that time, it is essential to verify and update these models to the actual situation. To that end, a handful of ITU member states (Germany, UK, The Netherlands, Japan and USA) have completed ambient noise measurements in the last 20 years. Notwithstanding their considerable effort, the obtained samples must be considered sparse when compared to the ambition of the ITU to provide guidance for all locations of the Earth and all frequency ranges. Several other interesting ambient noise measurement campaigns have been conducted elsewhere, such as [12,13], but often without bringing measurement methods and calibration in line with the ITU requirements, and without submitting results to the ITU. Absorbing ambient noise values from dissimilar methods and with incomplete documentation into the ITU models does not provide consistent results and is rejected. Still, there is a great need for accurate, calibrated ambient noise measurements from a larger variety of countries and regions of the world, especially from the Lower- and Middle-Income Countries [14].

1.2. Remote Rural Noise as a Baseline for Spectrum Planning

Currently, in all geographical areas, measurements of the noise in remote rural environments have been very scarce. This is unfortunate, as these values are especially important: they represent a baseline ambient noise level, due to the absence of artificial noise. These baseline values are used in international radio spectrum management. Worldwide and regional spectrum planning is conducted by the ITU and its members to ensure that all radio applications can function side-by-side without mutual interference, resulting in frequency allocations and technical requirements. If the calculated interference field strengths are small compared to this baseline ambient noise level, their impact may be considered negligible.

1.3. Contributions of This Article

As the ITU reference database contained few remote rural noise measurements, and none from Spain, it was decided to reprocess the raw data of a previous scientific experiment to obtain the absolute ambient noise field strength in a remote rural location in the Spanish Pyrenees [15]. This is not a simple task: due to the propagation of electromagnetic waves in the ionosphere, many signals of radio applications arriving from distances of thousands of kilometers [16] are observed as well, embedded in the background noise. To solve this problem, a novel adaptive filter was designed that has proven to be very effective for the removal of these radio signals. The obtained ambient noise field strength values were submitted to Study Group 3 of the ITU for the validation of radio noise models.

2. Ionospheric Propagation of Noise

2.1. Evidence of Ionospheric Propagation of Noise

In large cities, the ambient electromagnetic noise mainly consists of artificial noise, electromagnetic pollution that is a byproduct of electrical and electronic appliances and gas discharges. Due to the high equipment density in cities and their proximity, noise levels are high [9]. This noise reaches the receiver by direct coupling, line-of-sight or groundwave propagation [17], depending on distance and frequency.

By contrast, in remote rural areas, few artificial noise sources exist and the distance to them is large, and hence, the noise level is significantly lower. At these locations, impulsive noise from the powerful electrical discharges of distant lightning storms is observed, arriving via line-of-sight, groundwave propagation [18] or even via ionospheric refraction [19]. In between these pulses, the background electromagnetic noise is observed.

In such locations, almost all background noise arrives through the ionosphere or via refraction in the ionosphere. This was proven earlier with observations during a sudden ionospheric D-layer absorption event, triggered by a solar X-ray flare [20]. Figure 1 shows reprocessed measurement data from those observations. The blue pixels are individual measurements; the blue line represents the floating mean.

The ambient electromagnetic noise level drops by approximately 12 dB during the X-flare, which implies that, in a remote rural location, 94% of the observed noise power arrives via the ionosphere.

2.2. Terrestrial and Extraterrestrial Background Noise

The background electromagnetic noise is a combination of extraterrestrial noise arriving through the ionospheric ‘iris’, a circular area in the ionosphere that is transparent for electromagnetic waves and terrestrial noise reflected by the ionosphere, as shown in Figure 2. Depending on the observation frequency and the peak plasma frequency of the ionosphere, the iris can be hundreds of kilometers wide, or fully closed.

In the HF frequency spectrum, the relationship between the highest frequency to be reflected by the ionosphere f, the peak plasma frequency of the ionosphere for the extraordinary wave fxF2 and the zenith angle θ is approximated by Martyn’s ‘Secant Law’ [21]:

$$f<fxF2 \sec \theta$$

(1)

This approximation assumes a flat earth and ionosphere, and is not precise for zenith angles larger than 75°, but it is adequate for most other angles [22]. So, the ionosphere reflects radio waves with a frequency f when the zenith angle θ is:

$$\theta >{\cos }^{-1}\left(\frac{fxF2}{f}\right)$$

(2)

For smaller angles, the ionosphere is transparent, and a circular iris opens above the receiver, increasing in size with increasing frequency. At higher frequencies, where the ionosphere is fully transparent, only ambient noise from extraterrestrial sources is received, as was discovered by Jansky in 1933 [6], which started radioastronomy. At frequencies below fxF2, where the iris is fully closed, only terrestrial noise is received. The transition from atmospheric noise to extraterrestrial noise with increasing frequency is gradual. The solid angle of the ionospheric iris can be calculated using:

$$\mathsf{\Omega }=4\pi {\sin }^2\left(\theta /2\right)$$

(3)

If we assume that the received noise power is proportional to the solid angle (which is simplification), and that the terrestrial noise is 10 dB higher than the extraterrestrial noise, the transition will follow a curve, like in Figure 3. A different relative level of the terrestrial noise will produce as slightly different transition frequency. This article focuses on the terrestrial component of the background noise, observed below fxF2.

2.3. Ionospheric Reflection of Artificial Noise from Cities

The terrestrial noise has a much higher field strength than the extraterrestrial noise and is probably caused by accumulated emissions of artificial noise sources in major cities. This has not been fully proven yet, but Obenberger power-line-generated noise reflected by ionospheric Sporadic E-layer clouds has been observed using the Long Wavelength Array [23], making this a plausible scenario. An overview of the most common ambient noise propagation mechanisms at HF is given in Figure 4, which is adapted from [24].

3. Method Used to Obtain the Background Ionospheric Noise

3.1. The Original Polarimetric Experiment

The La Salle Universitat Ramon Llull, Barcelona, Spain, and the University of Twente, Enschede, the Netherlands, performed a scientific experiment in the lower Pyrenees. In this experiment, described in [15], the polarizations of the downcoming ordinary and extraordinary wave components, caused by dual refraction in the ionosphere [25], and their isolation were studied, as well as the influence of the wave angle relative to the Earth’s magnetic field [26].

Six probes emitting waves with sequences of pre-programmed polarizations were deployed in a radius of 50–100 km around a central receiver [27]. The frequency of the experiment was 7.009 MHz, and the experiments were performed during daytime, when fxF2 exceeded this frequency. This was a prerequisite for receiving the signal of the probes and this was monitored throughout the experiment.

The receiver was installed in a remote rural location to achieve maximum sensitivity (low ambient noise). This location was found near Sant Marti de Sesgueioles, in Catalonia; coordinates: 41.682° north, 1.484° east. The groundwave path of 65 km to the major city of Barcelona was blocked by the Montserrat mountains, with a peak height of 1236 m. The distance to the smaller cities of Lleida and Tarragona was more than 70 km. The receiver was directly fed from deep-discharge batteries, to avoid noise from voltage convertors. This set-up was highly successful for the measurement of the polarization of Near Vertical Incidence Skywave (NVIS) signals. The antenna voltages where sampled and digitally stored, after offline data processing took place. Firstly, the individual probe frequencies were separated by spectral filtering. After that, for each of the probes, the processing was synchronized with the sequences of preprogrammed transmit polarizations, after which the ionospheric propagation could be analyzed. This proved successful [15].

3.2. A New Procedure to Obtain the Background Radio Noise

However, to obtain the background ambient noise, the data processing had to be significantly modified. Firstly, instead of looking at the polarization of the antenna signal, the overall power coming from the antenna needed to be measured in a direction- and polarization-agnostic way. This was done by changing the processing of the sampled antenna, as will be discussed in Section 3.3. This changed the antenna factor, which was therefore corrected, as described in Section 3.4.

Secondly, instead of filtering and retaining the known frequencies of the six constant probe signals, many dynamic radio signals needed to be suppressed, of which the spectral and temporal aspects were not known a priori. To this end, an adaptive filter was designed that acted on a frequency-time representation (spectrogram) of the received antenna voltages. The intermediate step to create this spectrogram is described in Section 3.5. The adaptive filter is described in Section 3.6.

Finally, the impulsive noise from far-away lightning strikes needed to be suppressed (Section 3.7), to only leave the background ambient noise for further analysis.

3.3. Antenna Modifications to Obtain the Background Noise

For the polarization measurement, two resonant half-wave Inverted Vee dipole antennas were mounted perpendicular to each other and fed with equal lengths of coaxial cable via two balance-unbalance transformers (baluns) with equal phase delay. As the antenna was entirely passive, no noise was generated in the antenna itself. The two antenna elements were connected to a dual channel synchronous direct-sampling receiver. After down-conversion, baseband data streams of 3000 24-bit IQ-samples/s were stored for post-processing. After filtering, the difference in amplitude and phase of the signal from each antenna element was calculated on a per-sample basis, which provided the polarization of the incoming signals.

For the background noise measurement, instead of deriving the polarization of the incoming signals, the total power, independent of polarization, needed to be obtained. This could be achieved by first calculating the output powers of the individual antenna elements and then adding them together; see Figure 5. This process has already been described in [24], and results in an omni-directional and polarization-independent antenna. The radiation pattern of this antenna differs from the short lossless vertical monopole (SLVM) above perfectly conducting ground that the ITU defines as a reference antenna for ambient noise [7] (p. 5). However, the SLVM has low sensitivity for waves arriving from near the zenith [24], and is therefore not suitable for reception for the measurement of ionospheric noise in this frequency range, where NVIS propagation [28] is dominant.

3.4. Modifications to the Overall System Calibration

The receiver channels are calibrated to provide absolute power values. The estimated expanded standard measurement uncertainty is <2 dB for a 95% confidence interval. The receiver noise figure is measured using a calibrated noise source and is 12 dB at 7 MHz.

Due to the modified antenna as discussed above, adding the power of 2 antenna elements, a system gain of 3 dB is realized over a single antenna. The ohmic losses in the coaxial feeder cables of 11 m length, combined with the losses of the balance transformers, are approximately 0.5 dB. Therefore, the measured power overrepresents that of a passive omni-directional antenna by 2.5 dB, and measurement values must be accordingly corrected. The receiver noise figure is also be lowered by 2.5 dB, to become 9.5 dB.

3.5. Modification to the Spectral and Temporal Filtering

In the polarimetric experiment, the signal of the probes contained the information wanted, and the background noise and other radio signals were suppressed by filtering in frequency and time. This could be easily done, as the precise frequency, bandwidth and timing of the probe signal transmitters was known.

For the background noise measurements, the probe signals and other radio signals must be suppressed, and the ambient noise retained. This is not an easy task: due to the propagation of electromagnetic waves in the ionosphere, many radio signals are also observed, arriving from distances of hundreds of kilometers, embedded in the background noise. The characteristics of these signals are unknown, and the ionospheric propagation is highly variable. To solve this problem, we will describe a novel adaptive filter that has proven very effective for their removal in these circumstances.

From the polarimetric experiment, approximately 816,000,000 ambient noise samples were stored. The samples represent 3 kHz of bandwidth and contain the ambient noise along with the signals of the six probes, each approximately 80 Hz wide, and other weak signals from radio services and scientific instruments, such as ionosondes. A spectrogram of 1 h of this data can be seen in Figure 6. The spectrogram was created using a 1000-point Fast Fourier Transform (FFT) with a Blackman-Harris window, chosen for best spectral resolution. The signals of the probes can be seen, their power spectral density (PSD) is more than 50 dB stronger than that of the background noise. Furthermore, other radio signals of unknown origin appear in the spectrogram.

3.6. A Novel Adaptive Filter for the Removal of Narrowband Radio Signals

To retain the background ambient noise and at the same time remove the radio signals, an adaptive filter is realized. Using the statistical information of the samples produced by each FFT, a dynamic threshold is set to distinguish the frequency components of the background ambient noise from the discrete radio signals. Based on this threshold, the radio signals are removed in the frequency domain.

The threshold is determined by sorting the frequency bins of each FFT on increasing PSD; see Figure 7. The median value is established, as well as the slope of the tangent at this point. The slope is determined from the data points at 40% and 60% of the sorted values. This was carried out by drawing a straight line through these two data points, but a more robust way would be to use linear regression on all data points between the 40% and 60% data points. A second line parallel to the tangent is drawn, adding a small margin (2.5 dB) to reduce the influence of small irregularities on the threshold. The threshold is then determined by the intersection of that line and the curve. The frequency bins to the right of this intersection point are removed. As a result, the corresponding time-frequency pixels in the spectrogram are removed, as depicted in Figure 8, and they will no longer contribute to the background noise measurement. An example with real measurement data is shown in Figure 9.

The margin of 2.5 dB is a compromise value, empirically selected. A higher value will reduce the sensitivity of the adaptive filter, whereas a lower value will increase the number of rejected bins containing background radio noise.

Comparing the spectrograms of Figure 6 and Figure 9 shows the effective removal of all radio signals, their modulation sidebands and even their signal impurities. Careful observation determines that narrowband artificial noise is also removed, as can be observed around 7.008 MHz between 17:00 and 17:16 h local time. The advantage of this method is that correct removal of unwanted high frequency signals can be directly observed and verified in the spectrogram and no further calibration is needed.

3.7. Removal of Impulsive Noise

Due to the nature of this adaptive filter, wideband impulsive noise, such as atmospherics caused by lightning, is not removed. These pulses simultaneously raise the PSD in all frequency-bins of a single FFT, and will therefore not be detected. These impulses can be discerned as narrow white vertical lines in Figure 9.

For our purpose, we are only interested in the background noise; therefore, the impulsive noise must be removed as well. To achieve that, a second filter is cascaded after the adaptive filter to blank the impulsive noise. This filter detects a sudden spike in the mean PSD of subsequent FFTs and removes the entire FFT that represents the pulse. The result is shown in Figure 10 in an amplitude-time graph, and in Figure 11 as a spectrogram. The white vertical lines have now disappeared. If desired, a second iteration could use the resulting background noise level to redefine the threshold and more precisely detect and filter the impulsive noise, but this is not carried out in our study.

The probability density graphs of the samples before and after adaptive filtering are given in Figure 12. The unfiltered probability density has a more prominent upper tail, containing the PSD of the beacons, atmospherics and other radio signals. As a result, the difference between the median and the mean PSD is very high: 31 dB. After filtering, the lower and middle parts of the probability density graph remain unaltered, but the upper tail has now become much smaller. The difference between the median and the mean PSD is now reduced to 2.9 dB.

4. Results

The raw measurement samples from the original polarimetric experiment covered two days, on 11 February 2016, from 07:45 to 18:02 local time, and on 12 February 2016, from 07:40 to 18:03 local time (UTC + 1). After processing the raw measurement samples as described in previous sections, the median PSD of the background ambient electromagnetic noise is calculated from the filtered time-frequency bins. The median ambient noise factor F_am is calculated using formula 6 of Recommendation ITU-R P.372-15 [7]:

$$F_{am}=P_N-C_{ant}-B+204$$

(4)

where F_am is the ambient noise factor in dB, P_N is the noise power measured at the receiver input in dBW, C_ant is the correction factor for the relative gain of the dual channel hybrid antenna-receiver system, as explained in Section 3.3, and B is the logarithm of the bandwidth b (in Hz):

$$B=10{ \log }_{10}\left(b\right)$$

(5)

The median ambient electromagnetic noise factor values were calculated on an hourly basis and are recorded in

Table 1. Measured median ambient noise factor F_AM.

Date	Local Time [UTC + 1]	PSDrx [dBW/Hz]	Cant [dB]	Famb [dB]
11 February 2016	07:00–08:00	−164.5	2.5	37.0
11 February 2016	08:00–09:00	−162.8	2.5	38.7
11 February 2016	09:00–10:00	−164.7	2.5	36.8
11 February 2016	10:00–11:00	−166.6	2.5	34.9
11 February 2016	11:00–12:00	−166.9	2.5	34.6
11 February 2016	12:00–13:00	−168.9	2.5	32.6
11 February 2016	13:00–14:00	−169.1	2.5	32.4
11 February 2016	14:00–15:00	−167.2	2.5	34.3
11 February 2016	15:00–16:00	−166.5	2.5	35.0
11 February 2016	16:00–17:00	−164.0	2.5	37.5
11 February 2016	17:00–18:00	−160.7	2.5	40.8
12 February 2021	07:00–08:00	−164.2	2.5	37.3
12 February 2021	08:00–09:00	−163.1	2.5	38.4
12 February 2021	09:00–10:00	−164.8	2.5	36.7
12 February 2021	10:00–11:00	−166.3	2.5	35.2
12 February 2021	11:00–12:00	−167.4	2.5	34.1
12 February 2021	12:00–13:00	−170.1	2.5	31.4
12 February 2021	13:00–14:00	−169.9	2.5	31.6
12 February 2021	14:00–15:00	−169.0	2.5	32.5
12 February 2021	15:00–16:00	−166.6	2.5	34.9
12 February 2021	16:00–17:00	−164.0	2.5	37.5
12 February 2021	17:00–18:00	−161.1	2.5	40.4

. The receiver noise figure is 9.5 dB, as was discussed in Section 3.3. This is two to three magnitudes lower than the ambient noise, therefore its influence on the measurement is negligible and a correction for receiver noise is not necessary. The ambient noise factors are calculated for each hour of the observations and plot versus time in Figure 13, to show the diurnal variation of the noise over the two days. Figure 14 provides a statistical representation of all the measurements. The median value is shown as a horizontal dash. The 10-th and 90-th percentile are shown as boxes. The vertical lines show the extreme values. The predicted ambient noise values, given by the ambient noise models of Recommendation ITU-R P.372-15 [7] (pp. 99–100), are added as dashed horizontal lines in Figure 13 and Figure 14.

5. Discussion

From these results, two observations stand out: (a) the observed values are somewhat higher than the predicted values for remote rural locations, given by Recommendation ITU-R P.372-15, and (b) both days show a comparable diurnal noise variation. To understand these observations, we must consider the propagation path of the noise to the receiver. To investigate this aspect, ionospheric measurements obtained with a DPS4D ionosonde are processed as described in the Supplementary Material. During our observations, the ionospheric iris was fully closed from approx. 08:40 to 18:00 local time, when fxF2 exceeded the observation frequency f_tx. This can be seen in Figure 15, which shows fxF2, measured by the ionosonde of the Ebro Observatory, located at 40.8° N, 0.5° E, on the same two days.Consequently, only terrestrial noise could be observed, and it was shown in previous research [20] that in that case, 94% of the received noise power arrives from ionospheric reflection.

Let us therefore assume that most of the background (non-impulsive) electromagnetic noise comes from an accumulation of unwanted emissions of electric and electronic devices in nearby major cities. The difference between the observed values and the predictions could then be explained by the reduced sensitivity of the SLVM antenna—the ITU reference antenna—for high elevation angles. Ambient noise generated by the city of Barcelona and other major cities in the region would arrive at angles higher than 75–80° [29], angles at which the SLVM has 10–15 dB less sensitivity [24]. Furthermore, we must remember that the ITU noise models represent the worldwide average, and therefore will not represent all individual countries and regions equally well.

The variations in the ambient noise do roughly correlate with the diurnal variation of the D-region absorption, which is maximal around local noon [16] (p. 222). This may indicate that D-layer absorption is one of the main causes of the background noise variation in a remote rural location. Higher values in the evening as compared with the morning could possibly be attributed to an increased use of home appliances in the evening.

6. Conclusions

A method to measure the background ambient electromagnetic noise, which is received via ionospheric propagation, was successfully demonstrated. This was done by reprocessing a previously recorded dataset consisting of sampled antenna voltages from two orthogonal Inverted Vee antenna elements, collected for a polarimetric experiment investigating ionospheric radio wave propagation.

To measure the background noise regardless of direction and polarization, a synthetic omnidirectional and polarization-agnostic antenna was realized by adding the power of both antenna elements in the digital domain. These characteristics cannot be realized using analogue antennas alone. A new adaptive filter algorithm was designed to reject the highly variable content of radio signals—typical for ionospheric propagation in the HF spectrum—from the recorded spectrum, to make the measurement of the significantly weaker background noise possible. The method selects small frequency-time blocks in the spectrogram and excises those blocks if a dynamically set threshold is exceeded.

Further improvement of the adaptive filtering may be realized with a more advanced radio signal detection mechanism, which is subject to further study. The current method only looks along a line parallel to the frequency axis, while visual inspection of the radio signals in the spectrogram would identify frequency-time-areas (rectangular forms) with increased PSD, rather than along a line. A detection method that seeks such areas would reduce false positives and allow a sharper detection threshold. It would also allow the deletion of small border areas around the identified area, in which signal power that is present slightly below the noise could now go undetected. This would further improve the accuracy of the results.

The first measurements processed using this method show a diurnal variation in the background ambient noise arriving via ionospheric propagation, the origins of which merit further research. The atmospheric (lightning) source of ambient noise is well-known, but the relative contribution of the ionospheric reflection of artificial noise generated by cities is largely unknown. This source could be further investigated using large antenna arrays for radio astronomy, such as the Long-Wavelength Array (LWA) [30] in the USA and the Low Frequency Array (LOFAR) [31] in Europe. Alternatively, dedicated interferometer hardware could be developed. Recent studies into proliferation of electromagnetic pollution originating in power lines via Sporadic E-layer ionization [23] have shown the potential of such research. More knowledge on this mechanism would enable prevention and mitigation of the spread of unwanted electromagnetic noise over a large area via ionospheric reflection.

It is also noted that despite intended worldwide coverage of the empirical ambient noise model described in Recommendation ITU-R P.372, the validation data for these models originates in very few countries. Therefore, the design cost-effective measurement systems for ambient noise is desired, as is worldwide deployment of such equipment for a more widespread collection of noise data. A prerequisite, however, remains that this equipment has sufficient accuracy and dynamic range, proper calibration and sufficient sensitivity to also observe remote rural background noise.