Background

We aim to broaden our perception of space data but also include accessibility for visually impaired individuals. The vastness of our universe can be daunting and difficult to comprehend, but by associating it with something familiar—our senses—we may develop a more comprehensive and profound understanding. We chose to implement our ideas through a website because of the increased level of accessibility: it can be used by anyone, be it an elementary school student, visually impaired citizen, or astrophysics enthusiast.

Where to start?

2D NASA sonifications work by scanning the image from one point to another, therefore a sonification of an image lasts a few seconds, but how do you implement the same method in a video that lasts multiple seconds and has multiple frames per second? That is why we decided to synthesize the properties of the pixels and use conventions to convert them into sound, in an intuitive manner that is easily understood by anyone.

Conventions used

Brightness -> Frequency

Hue -> The instrument

Objects in frame -> Amplitude

Location on screen -> Audio Channel

Red -> 165 - 15 -> violin

Orange + Yellow - 15 - 35 -> piano (also white- hue=0)

Green -> 35 - 85 -> harp

Cyan -> 85 - 100 -> guitar

Blue -> 100 - 135 -> flute

Mauve + Pink -> 135 - 165 -> double bass

Process

To transform the user's video into a telling sonification we process it using a combination of Python algorithms. Firstly, we subtract the individual frames from it using OpenCV and we iterate through them to analyse and convert each one into its own combination of notes. Each image is converted into an HSV (Hue Saturation Value) pixel array which we use to correlate different properties of the image to properties of sound (see conventions used section above). We use the power of stereo audio to map object sounds to their location: a pixel that is placed more to the left will have its corresponding note sound louder in the left side audio channel. After we process all the images, we will have created multiple MIDI (digital sheet music).files containing the notes for each instrument. These files are all then passed as arguments to a function that uses their corresponding instrument's sound fonts to convert them into a mixed WAV (Waveform Audio File Format, the output) file that is used as an overlay to the uploaded video.

Musicality

When you look at images captured in outer space you feel something, whether that is fear of the unknown, creativity, inspiration, or you're just left wondering about all the possibilities of the infinite, you feel an emotion. We wanted to convey this emotion as well, but that is impossible when all you hear is a bunch of different instruments and pitches mushed together! To add musicality and information conveyed by the track, we use a background sequence of notes that changes its tempo based on the sum tempo=a*L+b*S, where

L -> the average brightness of the whole image;

S -> the average saturation;

a and b -> multipliers we can use to tune the formula to its most intuitive value;

The result

In the end, we will get a sonification which can be easily interpreted, for example:

a loud, high-pitched sound that is accompanied by a slow beat could represent the image of empty space with lots of small but bright specks of light coming from far away stars;
a close-up of a supernova would sound like many loud instruments playing all at once (harmonically), accompanied by a quick beat;

More details

Check out the official nasa.lazar.lol website.

Tools used

-Image processing: OpenCV

-MIDI files: Midiutil

-MIDI processing to WAV: midi2audio (FluidSynth interface)

-WAV Processing: Pydub

-Testing: Matplotib

-Used for math: Numpy

-Webdev: Astro.js with Svelte Integration

-Realtime communication: Flask, Python-Socket.io

-CI/CD Pipeline && DevOps: Docker, Nginx, Portainer, Cloudflare, Github Actions, Oracle Cloud VMs.

-Google Fonts: Titillium Web

2023 NASA Space Apps Challenge

Kilonova