Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.

There is a newer version of the record available.

Published September 25, 2019 | Version v1
Dataset Open

Identifying galaxies, quasars and stars with machine learning: a new catalogue of classifications for 111 million SDSS sources without spectra

  • 1. Jodrell Bank Centre for Astrophysics

Description

This is version 1, you should use the updated version 2 that was accepted for publication: 10.5281/zenodo.3768398

The Paper: https://arxiv.org/abs/1909.10963

We use 2.4 million spectroscopically labelled sources from the SDSS catalogue to train an optimized random forest classifier using photometry from the Sloan Digital Sky Survey (SDSS) and Widefield Infrared Survey Explorer (WISE). We apply this machine learning model to 111 million previously unlabelled sources from the SDSS photometric catalogue without existing spectroscopic observations. Our new catalogue contains 49.7 million galaxies, 2.4 million quasars, and 59.2 million stars. We provide individual classification probabilities for each source, with 6.4 million galaxies (13%), 0.35 million quasars (14%) and 44.3 million stars (75%) having classification probabilities greater than 0.99, and 34.8 million galaxies (70%), 0.77 million quasars (32%) and 55.3 million stars (93\%) having classification probabilities greater than 0.9. We determine Precision, Recall and F1 score as a function of feature selection, including scenarios with additional external variables. We investigate the effect of class imbalance on our machine learning model and discuss the implications of transfer learning for populations of sources at fainter magnitudes than the training set. We use a non-linear dimension reduction technique (Uniform Manifold Approximation and Projection: UMAP) in unsupervised, semi-supervised and fully-supervised schemes to visualise the separation of galaxies, quasars and stars in a two dimensional space using the 2.4 million spectroscopically labelled training sources. We confirm that when the same algorithm is applied to the 111 million sources without spectra it is in strong agreement with the class labels applied by our random forest model.

When using this dataset, please cite our paper (https://arxiv.org/abs/1909.10963) and this DOI (10.5281/zenodo.3459294). If you make use of our scripts please cite our Github repository DOI ().

Files

Files (29.8 GB)

Name Size Download all
md5:15bdb102eb4a380245410deb76f5bb8c
14.5 GB Download
md5:dc0cda7ebdb557d5200cf26dc0d39807
6.9 GB Download
md5:e4f700c626be7ca60931fc09d53ab429
333.9 MB Download
md5:e730b7956d9950729167e418fe70d408
8.2 GB Download