BirdData: Multithread Python Wrapper for Xeno-canto api 2.0

Repo

realzza/birdData

Introduction

BirdData is an open-source python wrapper for Xeno-canto 2.0 API. It customizes the query with different parameters, then retrieves metadata and bird recordings in a multi-threaded manner.

What is Xeno-canto

Readers new to this API may be curious about what is xeno-canto. According to their website, xeno-canto is a website dedicated to sharing bird sounds from all over the world. It is powered by the Xeno-canto Foundation and Naturalis Biodiversity Center. Xeno-canto works with bird lovers, researchers, and mostly recordists globally to upload bird recordings from all over the world. It offers an ever-expanding database of bird recordings, which is good news for computer scientists interested in creating human-aiding bird sound detection and identification algorithms. Competitions regarding bird classification, such as the DCASE competition, are supported by data from xeno-canto.

Why am I building BirdData

Since my sophomore year, I have led a project on Bird Sound Detection and Classification based on Deep Learning Methods. Since then, I have been working on the research on bird sound classification. We mainly worked on the bird sound detection and classification task in Guangdong, China. As an extension of this research interest, I converted this task into my signature work to provide a bird sound classifier for common birds in the general China Region. Because I am dealing with many times more bird species than in Guangdong province only, I desperately need an efficient tool to help collect bird data in the China Region. But most of the tools I can find online are useless either because of lack of maintenance or low efficiency. I need a tool capable of handling the vast and efficient need for bird data and meta information. That’s the story about how I came to build BirdData.

Tutorial

This section is a tutorial on using birdData to crawl metadata and audio recordings.

Download metadata

Metadata is a simple configuration for each recording. Typically, metadata files contain information like recordist, recoding time, country, location, latitude, longitude, altitude, and recording length. Below is an example of a metadata file.

{
    "id": "426350", 
    "gen": "Abroscopus", 
    "sp": "superciliaris", 
    "ssp": "", 
    "en": "Yellow-bellied Warbler", 
    "rec": "Peter Boesman", 
    "cnt": "India", 
    "loc": "Eagle Nest, Sessni area and lower, Arrunachal Pradesh",
    "lat": "27.0223", 
    "lng": "92.4139",
    "alt": "", 
    "type": "song", 
    "url": "//xeno-canto.org/426350", 
    "file": "https://xeno-canto.org/426350/download"
}

Every recording has a unique metadata file; it is like an ID for bird recordings. We can apply Data Analytics to metadata files to gain insight into the ornithology conditions in certain regions, which greatly favors the ornithology research community.

To download metadata files from xeno-canto API, we need to formulate a query in the first place. A query is a command that communicates with the API about the information that the clients would like to retrieve. According to the official documentation of xeno-canto API, the birdData metadata downloader supports various definitive arguments. See download-meta.py

optional arguments:
  -h, --help       show this help message and exit
  --gen GEN        genus
  --ssp SSP        subspecies
  --cnt CNT        country
  --type TYPE      type
  --rmk RMK        remark
  --lat LAT        latitude
  --lon LON        longtitude
  --loc LOC        location
  --box BOX        box:LAT_MIN,LON_MIN,LAT_MAX,LON_MAX
  --area AREA      Continent
  --since SINCE    e.g. since:2012-11-09
  --year YEAR      year
  --month MONTH    month
  --output OUTPUT  directory to output directory. default: `dataset/metadata/`

A detailed explanation of the above arguments can be found here.
LLet me make an example to give you some intuition on using this tool. To download all the bird recording metadata files in Shanghai, China, since 2022, the query you can customize should look like this,

python download-meta.py --cnt China --loc Shanghai --since 2022-01-01 --output test/

Then you will receive all the metadata files that meet your query’s criteria.

Retrieving metadata...  
retrieving metadata: 100%|█████████████████████████████████| 1/1 [00:03<00:00,  3.51s/it]

You will find your requested metadata files in the assigned directory test/.

Download Recordings

For computer scientists who are more interested in identifying bird species via bird speech, the following recording downloading script will do you a lot of favor.

The download script added multithreading features, which means it supports downloading multiple recordings simultaneously. See how to use this download-mult.py

usage: download-mult.py [-h] --name NAME [--time-limit TIME_LIMIT] [--process-meta PROCESS_META] [--process-ratio PROCESS_RATIO] [--output OUTPUT] [--attempts ATTEMPTS]

download bird audios

optional arguments:
  -h, --help            show this help message and exit
  --name NAME           [1] name of one bird species; [2] file of bird species spaced by '\n'
  --time-limit TIME_LIMIT
                        length of downloaded audio of each birds
  --process-meta PROCESS_META
                        number of process to download metadata
  --process-ratio PROCESS_RATIO
                        float[0~1], define cpu utilities in downloading audios [default: 0.8]
  --output OUTPUT       path of output directory
  --attempts ATTEMPTS   number of retry download times

Remember that the argument --name accepts both strings and a file path to multiple bird names. A qualified bird name file should look like this:

birdname1
birdname2
birdname3

Lines should be separated by \n.

Convert .mp3 to .wav

If you happen to be a computer scientist that has the same purpose of me to train a deep learning model to recognize bird songs, you should be facing the same problem in audio format conversion. librosa is the most commonly used acoustic feature extractor tool, however, it does not support mp3 codec format as well.

Therefore, we need to convert mp3 to wav before we can extract features. Of course, you can search Online Free mp3 to wav converter on google, but you don’t need to be reminded to realize how inefficient it is if compared with using the alignDataset-mult.py

usage: alignDataset-mult.py [-h] [--dataDir DATADIR] [--outDir OUTDIR] [--process PROCESS]

align smaplerate of dataset

optional arguments:
  -h, --help         show this help message and exit
  --dataDir DATADIR  path to the input dir
  --outDir OUTDIR    path to the output dir
  --process PROCESS  number of process running

You can customize the Transformer object from SoX tool inside the script in line-51.

tfm = sox.Transformer()
tfm.set_output_format(file_type='wav', rate=16000, bits=16, channels=1)

A pre-requisite of running this script is that you need to have SoX installed. Please refer to my previous blog for the easiest way to install.

Summary

We have talked about the story behind BirdData and how to use the easy scripts within. If you wish to join me in developing more efficient scripts to retrieve data from xeno-canto, don’t hesitate to make pull requests or contact me.

Reference

• Xeno-canto API 2.0
• Naturalis Biodiversity Center Website
• Librosa Project
• Librosa documentation
• Blog: Install Sox Locally