Computer Science
The Computer Science sub-team work on whistle analysis and machine learning.
ARTwarp
After whistles have been extracted and measured in Raven and PAMGuard, their frequency contours (Figure 1) are categorised by similarity using ARTwarp.
Figure 1: Frequency contours plotted in MATLAB (Frequency in [Hz] on y-axis, time in [ms] on x-axis), each representing a whistle. These contours are generated from PAMGuard’s measurements of manual tracings of spectrogram selections from Raven.
ARTwarp is a MATLAB-based program which uses a combination of Dynamic Time Warping and a custom ART2 neural network to categorise tonal animal sounds. ART2 is an unsupervised neural network, which means that it generates categories of similar contours without first needing to be trained to ‘recognise’ what a category should look like. A category is represented by a reference contour, a bit like the ‘average’ contour of all the contours in the category (Figure 2). The aim of ARTwarp is to mimimise the difference between contours in each category, and maximise the difference between reference contours for each category.
Figure 2: A reference contour (displayed in red), representing all the input whistles which were placed in an ARTwarp-generated category.
The first category is generated at random, and is an exact copy of one of the input contours. Then, ARTwarp iterates through all the rest of the input contours, and checks if they match the reference contour of a particular category to a particular vigilance (or similarity). At this stage, ARTwarp’s Dynamic Time Warping algorithm is used to locally stretch or compress the input contour’s time-axis, to allow broader similarities between the input contour and the reference contour to be captured (Figure 3).
Figure 3: MATLAB plot of an input contour (shown in green) being compared to a reference contour for a particular category (shown in blue), with Frequency in [Hz] on the y-axis and Time in [ms] on the x-axis. The input contour is dynamically time-warped (shown in red) to give maximum frequency overlap. The warping factor (the maximum local compresion or expansion factor allowed of the time-axis of the input contour) can be changed by the researcher according to their requirements, and is set to 3 here.
If the input contour matches the reference contour to the given vigilance (for example, 96%), then the input contour is added to that category, and the reference contour for the category is adjusted to incorporate the new contour. If not, the input contour creates its own new category. In a randomised order, each input contour is checked against each category, with the reference contours for each category being continually updated as the category absorbs new contours. Then, the whole process repeats until no input contours are reclassified (moved category since the last iteration). ARTwarp’s user interface allows the user to visualise the categorisation process in real time (Figure 4).
Figure 4: ARTwarp user interface during categorisation. The current whistle contour being analysed is shown in the top left box (in red). The current category, or neuron, it’s being compared to is higlighted in red (Neuron 9 here).
The final result is a number of reference contours, each representing a category (Figure 5), as well as a lot of associated data (i.e how many contours were assigned which category, what their percentage match was, etc.). So, by running ARTwarp using whistles from the same species in different locations, we can compare differences in whistle repertoires between different locations.
Figure 5: reference contours, representing different categories, shown in red on the top row. The input contours assigned to each category are shown in black beneath the reference contour for that category.
This VIP is currently focussed on maintaining, documenting, and adding new functionality to ARTwarp, as well as integrating it to OCEAN to increase the automation of data flow in the project. It has been a great opportunity for students to learn good software development practises, and to improve their communication skills between disciplines.
Follow this link for more information and access to the software used: