3.1 Basic principles

At the heart of acoustic analysis with soundgen is the short-time Fourier transform (STFT): we look at one short segment of sound at a time (one STFT frame), analyze its spectrum using Fast Fourier Transform (FFT), and then move on to the next - perhaps overlapping - frame. To analyze a sound with default settings and plot its spectrogram, all we need to specify is its sampling rate (the default in soundgen is 16000 Hz):

a1 = analyze(s1, samplingRate = 16000, plot = TRUE, ylim = c(0, 4))
# a1$detailed  # many acoustic predictors measured for each STFT frame
median(a1$detailed$pitch, na.rm = TRUE)  # our estimate of median pitch

## [1] 685.2083

# Pitch postprocessing is stochastic (see below), so the contour may vary.

There are several key parameters that control the behavior of STFT and affect all extracted acoustic variables. The same parameters serve as arguments to spectrogram. As a result, you can immediately see what frame-by-frame input you have fed into the algorithm for acoustic analysis by visually inspecting the produced spectrogram. If you can hear f0, but can’t see individual harmonics in the spectrogram, the pitch tracker probably will not see them, either, and will therefore fail to detect f0 correctly. The first remedy is thus to adjust STFT settings, using the spectrogram for visual feedback:

• windowLength: the length of sliding STFT window. Longer windows (e.g., 40 - 50 ms) improve frequency resolution at the expense of time resolution, so they are good for detecting relatively low, slowly changing f0. Shorter windows (e.g., 5 - 10 ms) improve time resolution at the expense of frequency resolution, so they are good for visualizing formants or for tracking high-frequency, rapidly changing f0 as in bird chirps or dolphin whistles.
• step: the step of sliding STFT window. For example, if windowLength = 50 and step = 25, each time we move the analysis frame, there is a 50% overlap with the previous frame. Short steps improve time resolution while maintaining relatively high frequency resolution at the price of processing time. For noisy recordings, pitch contour may actually be more accurate with relatively large steps and more smoothing.
• wn: the type of windowing function used to taper the analysis frame during STFT. In practice the windowing function doesn’t seem to have a major effect on the result, as long as you choose something reasonable like gaussian, hanning, or bartlett.
• zp: zero-padding. You can use a short STFT window and improve its frequency resolution by padding each frame with zeroes. This is a computational trick for improving frequency resolution while maintaining relatively high time resolution.
• silence: frames with root mean square (RMS) amplitude below silence threshold are not analyzed at all. Quiet frames are harder to analyze, because their signal-to-noise ratio is lower. As a result, we want to strike a good balance. Setting silence too low (close to 0) produces a lot of garbage, as the algorithm tries to analyze frames that are essentially just background noise without any signal. Setting silence too high (close to 1) excludes too many perfectly good frames, misrepresenting the signal. In soundgen silence is dynamically updated: it can never be lower than specified, but it may be raised to the minimum root mean square amplitude of all frames, if this minimum is higher than silence. This ensures that empty frames are not analyzed in recordings with unusually high levels of steady background noise (e.g., microphone hiss).

3.2 Spectral descriptives

Apart from pitch tracking, analyze calculates and returns several acoustic characteristics from each non-silent STFT frame:

• time: time of the middle of each frame (ms)
• duration: total duration (s)
• duration_noSilence: duration from the beginning of the first non-silent STFT frame to the end of the last non-silent STFT frame, s (NB: depends strongly on windowLength and silence settings)
• amEnvFreq, amEnvPurity, amEnvDep: frequency (Hz), purity (dB), and depth (0 to 1) of amplitude modulation estimated from a smoothed amplitude envelope
• amMsFreq, amMsPurity: the same as amEnvFreq and amEnvPurity, but estimated via modulationSpectrum, the arguments to which are passed in roughness = list())
• amMsFreq: frequency of amplitude modulation, Hz (estimated by modulationSpectrum, the arguments to which are passed in roughness = list())
• ampl: root mean square of amplitude per frame, calculated as sqrt(mean(frame ^ 2))
• dom: lowest dominant frequency band (Hz) (see “Pitch tracking methods / Dominant frequency”)
• entropy: Weiner entropy of the spectrum of the current frame. Close to 0: pure tone or tonal sound with nearly all energy in harmonics; close to 1: white noise
• f1_freq, f1_width, …: the frequency and bandwidth of the first nFormants formants per STFT frame, as calculated by phonTools::findformants with default settings
• flux: feature-based flux, the rate of change in acoustic features such as pitch, HNR, etc.; unitless (0 = none)
• harmEnergy: the amount of energy in upper harmonics, namely the ratio of total spectral energy above 1.25 x f0 to the total spectral energy below 1.25 x f0 (dB)
• harmHeight: how high harmonics reach in the spectrum, based on the best guess at pitch (or the manually provided pitch values): see soundgen:::harmHeight for details
• HNR: harmonics-to-noise ratio (dB), a measure of harmonicity returned by soundgen:::getPitchAutocor() (see “Pitch tracking methods / Autocorrelation”). If HNR = 0 dB, there is as much energy in harmonics as in noise
• loudness: subjective loudness in sone, assuming a certain sound pressure level (takes into account the energy in different frequency bands as well as the sensitivity of human ears to different frequencies); see getLoudness() and the section on Loudness below for details
• novelty: spectral novelty - a measure of how variable the spectrum is on a particular time scale, as estimated by ssm()
• peakFreq: the frequency with maximum spectral energy below cutFreq (Hz)
• roughness: the amount of spectrotemporal modulation in the “roughness” zone of frequencies (estimated by modulationSpectrum, the arguments to which are fpassed in roughness = list())
• quartile25, quartile50, quartile75: the 25th, 50th, and 75th quantiles of the spectrum below cutFreq (Hz) for VOICED frames
• specCentroid: the center of gravity of the frame’s spectrum below cutFreq, first spectral moment (Hz)
• specSlope: the slope of linear regression fit to the spectrum below cutFreq
• voiced: is the current STFT frame voiced? TRUE / FALSE

TIP: many of these descriptors are calculated separately for the entire sound and only for the voiced frames, resulting in extra output variables like “amplVoiced”. Output “voiced” gives the proportion of voiced frames out of the total sound duration, while “voiced_noSilence” corresponds to the proportion of voiced frames out of non-silent frames (ie with RMS amplitude above the “silence” threshold), which can exceed 1 if pitch is interpolated over putatively silent frames.

3.3 Custom spectral descriptives

The function soundgen::analyze returns a few spectral descriptives that make sense for nonverbal vocalizations, but additional predictors may be useful for other applications (bird songs, non-biological sounds, etc.). Many popular spectral descriptors are mathematically trivial to derive - all you need is the spectrum for each STFT frame, or perhaps even the average spectrum of the entire sound. Here is how you can get these spectra.

For the average spectrum of an entire sound, go no further than seewave::spec or seewave::meanspec:

spec = seewave::spec(s1, f = 16000, plot = FALSE)  # FFT of the entire sound
avSpec = seewave::meanspec(s1, f = 16000, plot = FALSE)  # STFT followed by averaging
# either way, you get a dataframe with two columns: frequencies and their strength
head(avSpec)

##            x            y
## [1,] 0.00000 6.336654e-05
## [2,] 0.03125 1.084848e-04
## [3,] 0.06250 2.052486e-04
## [4,] 0.09375 7.973666e-04
## [5,] 0.12500 2.243507e-03
## [6,] 0.15625 3.116417e-03

If you are interested in how the spectrum changes over time, extract frame-by-frame spectra - for example, with spectrogram(..., output = 'original'):

spgm = spectrogram(s1, samplingRate = 16000, output = 'original', plot = FALSE)
# rownames give you frequencies in KHz, colnames are time stamps in ms
str(spgm)

##  num [1:400, 1:78] 6.82e-05 5.28e-05 2.64e-05 1.62e-05 1.08e-05 ...
##  - attr(*, "dimnames")=List of 2
##   ..$ : chr [1:400] "0" "0.02" "0.04" "0.06" ...
##   ..$ : chr [1:78] "0" "15" "30" "45" ...

Let’s say you are working with frame-by-frame spectra and want to calculate skewness, the 66.6th percentile, and the ratio of energy above/below 500 Hz. Before you go hunting for a piece of software that returns exactly those descriptors, consider this. Once you have normalized the spectrum to add up to 1, it basically becomes a probability density function (pdf), so you can summarize it in the same way as you would any other distribution of a random variable. Look up the formulas you need and just do the raw math:

# Transform spectrum to pdf (all columns should sum to 1):
spgm_norm = apply(spgm, 2, function(x) x / sum(x))
# Set up a dataframe to store the output
out = data.frame(skew = rep(NA, ncol(spgm)),
                 quantile66 = NA,
                 ratio500 = NA)
# Process each STFT frame
for (i in 1:ncol(spgm_norm)) {
  # Absolute spectrum for this frame
  df = data.frame(
    freq = as.numeric(rownames(spgm_norm)),  # frequency (kHz)
    d = spgm_norm[, i]                       # density
  )
  # plot(df, type = 'l')
  
  # Skewness (see https://en.wikipedia.org/wiki/Central_moment)
  m = sum(df$freq * df$d)  # spectral centroid, kHz
  out$skew[i] = sum((df$freq - m)^3 * df$d)
  
  # 66.6th percentile (2/3 of density below this frequency)
  out$quantile66[i] = df$freq[min(which(cumsum(df$d) >= 2/3))]  # in kHz
  
  # Energy above/below 500 Hz
  out$ratio500[i] = sum(df$d[df$freq >= .5]) / sum(df$d[df$freq < .5])
}

## Warning in min(which(cumsum(df$d) >= 2/3)): no non-missing arguments to min;
## returning Inf

summary(out)

##       skew            quantile66       ratio500        
##  Min.   : 0.01562   Min.   :0.020   Min.   :  0.00146  
##  1st Qu.: 2.62994   1st Qu.:1.160   1st Qu.: 19.29105  
##  Median : 2.90867   Median :1.400   Median : 36.73054  
##  Mean   : 3.27167   Mean   :1.205   Mean   : 76.56794  
##  3rd Qu.: 3.12537   3rd Qu.:1.440   3rd Qu.:109.99833  
##  Max.   :13.70376   Max.   :2.840   Max.   :305.10464  
##  NA's   :1          NA's   :1       NA's   :1

If you need to do this analysis repeatedly, just wrap the code into your own function that takes a wav file as input and returns all these spectral descriptives. You can also save the actual spectra of different sound files and add them up to obtain an average spectrum across multiple sound files, work with cochleograms instead of raw spectra (check out tuneR::melfcc), etc. Be your own boss!

3.4 Pitch tracking

Fundamental frequency (f0) or its perceptual equivalent - pitch - is both highly salient to listeners and notoriously difficult to measure accurately. The approach followed by soundgen’s pitch tracker is to use several different estimates of f0, each of which is better suited to certain types of sounds. You can use any pitch tracker individually, but their output is also automatically integrated and postprocessed so as to generate the best overall estimate of frame-by-frame pitch. Autocorrelation is needed to calculate the harmonics-to-noise ratio (HNR) of an STFT frame, and then this information is used to adjust the expectations of the cepstral and spectral algorithms. In particular, if autocorrelation suggests that the pitch is high, confidence in cepstral estimates is attenuated; and if autocorrelation suggests that HNR is low, thresholds for spectral peak detection are raised, making spectral pitch estimates more conservative.

The plot below shows a spectrogram of the sound with overlaid pitch candidates generated by five different methods (listed in pitchMethods), with a very vague prior - that is, with no specific expectations regarding the true range of pitch values. The size of each point shows the certainty of estimation: smaller points are calculated with lower certainty and have less weight when all candidates are integrated into the final pitch contour (blue line).

a = analyze(s1, samplingRate = 16000, priorSD = 24, 
            pitchMethods = c('autocor', 'cep', 'dom', 'spec', 'hps', 'zc'),
            plot = TRUE, ylim = c(0, 4))

Different pitch tracking methods have their own pros and cons. Cepstrum is helpful for speech but pretty useless for high-frequency whistles or screams, harmonic product spectrum (hps) is easily misled by subharmonics (as in this example), lowest dominant frequency band (dom) can’t handle low-frequency wind noise or missing fundamental, etc. The default is to use “dom” and “autocor” as the most generally applicable, but you can experiment with all methods and check which ones perform best with the specific type of audio that you are analyzing. Each method can also be fine-tuned (see below), but first it is worth considering the general pitch-related settings.

par(mfrow = c(1, 2))
# default prior in soundgen
getPrior(priorMean = 300, priorSD = 6)
# narrow peak at 2 kHz
getPrior(priorMean = 2000, priorSD = 1)
par(mfrow = c(1, 1))

a = analyze(s1, samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'autocor',
            pitchAutocor = list(autocorThres = .45, 
                                # + plot pars if needed
                                col = 'green'),
            nCands = 3)

a = analyze(s1,  samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'dom',
            pitchDom = list(domThres = .1, domSmooth = 500, cex = 1.5))

a = analyze(s1, samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'cep',
            pitchCep = list(cepThres = .6),
            nCands = 3)

a = analyze(s1, samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'spec',
            pitchSpec = list())

a = analyze(s1, samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'hps',
            pitchHps = list(hpsNum = 2, # try 8 or so to measure subharmonics
                            hpsThres = .2))

a = analyze(s1, samplingRate = 16000, 
            plot = TRUE, ylim = c(0, 2), osc = FALSE,
            priorMean = NA, priorAdapt = FALSE,
            pitchMethods = 'zc',
            pitchCeiling = 600,  # higher pitch values blend with F1
            pitchZc = list(zcWin = 3, zcThres = 0))