logo

How to tag thermal videos

June 13, 2019 Browse Portal 4 minutes, 15 seconds

To streamline the training process for our artificial intelligence (AI) animal classifier, we are now applying tags to the individual areas of interest (called "tracks") in each video. Previously tags were being applied to each recording as a whole. This made it difficult to use the tags for training the classifier because it was never clear which part of the recording each tag applied to.

This new way of tagging means there's been some significant changes to our web-based browse interface.

Tracks

Tracks represent areas of interest in a video. They are generated by looking for warm areas of movement in the thermal video.

A video may have multiple tracks:

  • There may be multiple animals which typically results in one track per animal.
  • A single animal may move in and out of view. For example, a possum walks into view of the camera (Track 1), walks out of view, and then returns (starting Track 2).
  • Sometimes different track can be created of separate parts of a single animal (e.g. the body and the tail).
  • Other objects can sometimes trigger a track, for example, e.g. a tree trunk moving in the wind.

Tagging tracks

When tagging tracks it is important to only tag the subject of the currently selected track. The subject of the current track is highlighted in the video by a rectangle which is the same colour as the track icon on the right and is marked with the track title (e.g. "Track 1").

Only tag the animal/recording contained in the current track

As shown in this example there may be other tracks in the recording. These can be tagged when that track is selected.

Note that it is OK for different people to tag a track. These tags might agree (to confirm the identification) or not (e.g. difference of opinion).

The following steps describe how to tag the tracks in a recording.

Step 1: Tag track

Track details

To tag the track, click one of the animal buttons, or for more options select the other.. button and a popup will appear with more options.

Borders around these animal buttons indicate that the track has already been tagged as that animal. The colour of the border indicates the type of tag:

  • orange - automatically tagged by AI
  • green - tagged by a human user

Note that the AI tags just give an indication of what the current generation of the AI classifier thought the track was. These tags are not used for training. Human generated tags are what we really need to improve the classifier.

If the track is not clear but you can identify the animal from the whole video then tag it with the animal tag. Do this even if you can't tell from just the current track what the animal is. We hope our machine learning algorithm will be able to pick up more details than we can and learn to identify these animals even from relatively unclear footage.

Non-animal tags

In addition to the animal tags there are some other tags available for specific situations:

  • unknown - Use this if there is something of interest moving around but it is not possible to tell what it is.
  • nothing - Use this if the track is not of anything of interest (i.e. a false positive)
  • part of an animal - Use this if the track only represents a small part of the animal (e.g. the tail)
  • poor tracking - Use this if the track's rectangle doesn't follow the animal well
  • unidentified but not a bird - Use this if the animal in track is too unclear to identify but looks like it is one of our pest species. Do not use it if the animal hops around like a bird, or looks like an insect.

Step 2: Move to the next track

Either click on the arrow on the title of the track, or on another track title to move to another track. When you select a different track, the video will jump to where that track starts making it easy to locate the track in the video.

Continue

Repeat steps 1 and 2 until all tracks for the video are done.

Note that on small screens you will only see one track at a time and will need to use the arrow buttons to navigate between tracks. In this case the track title, e.g Track 1/3 will show you how many tracks there are in total.

Tagging recordings

Although we are mainly interested in tags for tracks, there are some tags that best apply to the whole recording. You may wish to add these as well.

Recording tags include:

  • missed track - Recording has an animal with no rectangle around it (no associated track). Use this even if you can't recognise the type of animal.
  • multiple animals - Multiple animals are present in the recording.
  • animal in trap - Animal has been caught in a trap.
  • cool video - Use this to mark a particularly interesting video so that that we can find it again and share with others.

To add a recording tag, click on the add tag in the recording section as shown below.

How to label recordings