Our Motivation

Throughout the years, NASA’s telescopes have collected various interesting and informative data of our universe. Whilst open for public use, these data are really hard to understand and apply. Furthermore, data representation for the visually impaired community is still currently limited to NASA’s several sonifications. In order to provide explorative platform options, we developed our own sonification projects, consisting of 3D representation of the galaxy using the abundant data provided by NASA’s telescopes.

Development Overview

To provide our target audience with a user-friendly audio and visual representation, we designed an interactive 3D display and auditory dimension, skillfully clustering stars based on their right ascension, declination, and flux captured by ZTF, 2MASS, WISE and IRAS missions.

Our data was meticulously sourced from NASA's IRSA website, comes in various formats, calibrations, and levels of precision. We address these issues by applying rigorous filtering to extract only the essential information, which we then translate into a visual and audio representation. Users can personally explore the 3D cube representation by zooming, targeting, or selecting focus on different bands. Furthermore, to aid users in understanding the auditory representation, we have incorporated an audio spectrum visualization. This visual feedback enhances the educational and exploratory aspects of the platform.

By using our platform, we hope to lay the foundations for an innovative and educational approach, useful for students, and visually impaired users to explore celestial objects, and foster a deeper connection to the astronomical data.

Notable Features

Input of dataset: Users can input any dataset of their choice from IRSA’s database.
3D Navigation: 3D model can be navigated to show different aspects of the dataset (right ascension, declination, and flux).
Band Selection for Sonification: Users can choose specific bands to create their sonification.
Spatial Audio: Apparent direction of sound sources are in accordance with the direction of sonification.

User Interface

Our platform incorporates a simple and minimalist approach to the user interface. We break it down to three different exploratory stages, which are Data Selection, 3D visual representation, and Auditory Translation.

Our welcome page consists of a selection section of our various pre-stored dataset and an option to upload users own choosing dataset to be translated and further manipulated in the next stages.

The user interface is divided into three main sections. To the left, descriptions for each of the four bands are presented. The center features four distinct 3D cubes, each representing data from a specific band. On the right, the control panel offers tools for moving the data up, down, left, or right within each cube, zooming, move along the flux axis and filtering data points with color codes.

The interface displays the audio and visual representation of the selected data set. By clicking play, users can see the data in sequence and hear the corresponding output sonification. Users may pause and resume as well. The timeline on the top right and the frequency spectrum on the bottom left act as visual aids for the users to understand which frequency range contains the most data at each given moment.

TECHNICAL ASPECTS

Data Filtration

Data is retrieved from IRSA, which provides us with abundant data collected from various telescopes. These vast data volumes consist of multi-band datasets that need to be further filtered. Filtration is done with consideration of overall visualization and representation (as opposed to focusing on a single celestial object or system), and takes care to preserve other celestial articles such as dust around the target object.

Mission Databases

We chose the 4 following datasets that include catalogs containing our 4 target bands: optical, near-infrared, mid-infrared, and far-infrared.

ZTF (Zwicky Transient Facility)
2MASS (Two Micron All Sky Survey)
WISE (Wide-Field Infrared Survey Explorer)
IRAS (The Infrared Astronomical Satellite)

Filtration Criteria

Due to ample data source, we implemented high criteria in the filtration, extracting high quality, significant measurements.

Photo Quality (ph_qual) Highest quality (A) of excellent signal-to-noise ratio is extracted. Other data of lower quality (B, C, U) is filtered and omitted.
Signal to Noise Ratio (snr/refsnr) Data with high ratios where the signal is much stronger than the noise is extracted (> 5), utilizing only clear and reliable measurements.
Artifact Flags (cc_flags/cc_flg) Indicates if the flux source is affected by known artifacts. A value of '0000' means it's unaffected..
Saturation (sat) Reliable low saturated data of up to 10% of saturated pixels (< 0.1).
Number of (Good) Observations (nobs/ngoodobs) Repeatedly observed or detected in the mission more than 3 times.
Astrometric RMS (astrometricrms) To allow for more data representation, a slightly lower threshold of 0.3 is chosen.
Von Neumann Ratio (vonneumannratio) Here we chose data with a ratio between 0.7 to 1.3, which suggests that the variability in the light curve is consistent with white noise.
Photometric uncertainty (cmsig) Sources with errors less than 0.1 magnitudes.
Number of Reads (rd_flg) Sources with at least 6 individual 1.3-second "read" images.
Blend (bl_flg) To ensure representation of a broad swath while excluding problematic points, we kept sources with values from 0 to 7.
Positional Uncertainty (err_maj) Objects with positional uncertainties less than 0.08 arcseconds to obtain precise positional data.
Quality of the Position Agreement (q_pos) Highest quality (A) of position agreement between bands.
Overall Quality of the Source (q_merge) In the form of grades A, B, C, D, or Q, we extracted the highest overall quality in regards to source, combining position and flux information.

K-means Integration in Machine Learning: Combine Data in Personalized 3D Slices

To sonify our multidimensional dataset, we employed a technique that involves segmenting the 3D dataset into discrete slices. The direction in which these slices are cut is determined by user preference, offering a personalized approach to data exploration. Within each of these slices, we harnessed the power of the K-means Machine Learning algorithm. This algorithm enables us to effectively amalgamate nearby data points, ensuring a more cohesive and interpretable representation. Consequently, each consolidated cluster of data points within a slice is then transformed into a unique sonification clip. This process not only makes the data audibly comprehensible but also ensures that the sonification is a true reflection of the underlying patterns within the dataset.

Audio Representation of the 3 Data Dimensions

Right Ascension

Users can experience the change in right ascension as spatial audio panning between the two sides (left and right ear).

Declination

Declination of the objects are conveyed through the amplitude of the sounds, going from higher to lower, from the center value to outer values.

Flux

Flux values across the four bands are distinctively represented by different synthesizers. A synthesizer, an electronic musical instrument generating sound electronically, imparts unique timbres to each band. Consequently, even when producing the same tone, each instrument will resonate differently to the human ear due to these timbres.

As the frequency and wavelengths are inversely proportional, we take the values of the wavelength and convert it to frequency, and then map it to the frequency range of the note of the instrument. The wavelengths are then conveyed via the notes of the instrument in that frequency. Note that wavelengths that convert to frequencies that exceed the human hearing range (too high or too low) are ensured to be scaled up or down.

Optical -> Bass: Bass instruments are a family of instruments that produce lower-pitched sounds in music, and are known for their deep and texturized tones.
Near-Infrared -> Brass: Brass instruments produce sound by buzzing the lips into a cup-shaped mouthpiece, and are famous for their bright and brassy tones.
Mid-Infrared -> Pad: A pad is a synthesized musical sound characterized by its sustained, ambient, and ethereal quality.
Far-Infrared -> Box: A box is an electrical instrument that sounds almost like a marimba or xylophone. It is very light but sounds a little sharper than the rest of the instruments.

OUR VISION

Every year, an abundance of data is collected, processed, stored, and made open to the public. However, this information is often not comprehensible to the general population. We believe in the importance of the popularization of science and technology, and have built this project in hopes of creating new, interesting ways to showcase astronomy findings. Combining science and arts, and introducing new forms of art, we hope to open doors to bigger audiences as well as spark their interests in the exploration of our universe.

TECHNOLOGIES USED

In crafting our "SoniVerse" experience, we've interwoven diverse technologies, primarily based on the JavaScript language. Each technology has been judiciously chosen to cater to specific aspects of the user's journey, ensuring both visual delight and auditory engagement.

Visual Representation

The visual layer of our platform is brought to life using cutting-edge JavaScript frameworks and libraries:

Node.js: This platform acts as the foundational structure of our project, primarily handling frontend routing. It anchors data management, enables real-time communication, and ensures the scalability of our project.
Three.js: As the visual heart of our platform, this library constructs an engaging 3D environment. It offers a vivid display of celestial data, complemented by user-friendly controls for exploration.

Algorithm Processing

To make sense of the vast celestial data and to ensure meaningful audio translations, we employ:

k-means Machine Learning Algorithm: Essential to our data analysis, this algorithm clusters celestial datasets. Its integration ensures that our sonifications are deeply rooted in the patterns and structures within the data.

Sonification Production

Converting visual and algorithmically processed data into immersive soundscapes is achieved using:

p5.js: Operating atop the Web Audio API, p5.js is our sonic engine. It interprets and translates data into dynamic audio elements, letting users experience the cosmos not just visually but also aurally.

Project Evolution: Today's Achievements, Tomorrow's Possibilities

The current capabilities of our project allow users to delve into the auditory realm of celestial wonders by selecting pre-existing objects or by uploading data sourced from the IRSA website. However, the horizon for this web-based platform extends much further. In future iterations, we envision empowering users even more by allowing them to upload their own unique datasets. This adaptability not only amplifies the platform's versatility but also opens up avenues for diverse scientific and creative explorations, turning it into a potent tool for both researchers and enthusiasts alike.

OUR SPACE APP JOURNEY

Our Learning Experience

As with every space app challenge, we gain a new peek into the celestial aspect of our universe. This year, we obtained a basic understanding of how data obtained from telescopes is stored, how to access data from the catalogs of the IRSA viewer, how to collect different data from different missions, what types of data are collected by the different telescopes and more. Furthermore, with the challenge “Immersed in the Sounds of Space”, in addition to astronomical knowledge, we were also required to dive into the field of sound and audio.

Our Inspiration

Among all the challenges posed from this hackathon, “Immersed in the Sounds of Space” challenge opens the opportunity for us to reach the visually impaired community as our target audience. Moreover, one of our team members is working closely with musical technology and this challenge is a great opportunity to be able to implement this expertise in the project.

Our Approaches

Our approach in developing this project involved a breakdown of tasks into three key areas: data filtering, visual and audio implementation, and user experience. Each of these components was carefully crafted to ensure accuracy, interactivity, customization, and accessibility of the platform.

Our Challenges

Our main difficulties in this project is to handle the vast data volumes and data heterogeneities that we need to manage within the hackathon timelines. Getting just the right amount of data was a huge challenge. Hardware restrictions prevented us from extracting too much data, whereas too little would result in inadequate representation of data. To solve this, we implemented stricter filtering criteria for those with more data.

In regards to the workload, our team decided to divide and conquer, allowing each one of us to focus on certain aspects of the projects. Due to the time difference of the regions of each member, we implemented a routine communication to keep everyone updated on our progress in the respective fields.

2023 NASA Space Apps Challenge

Area Six