How does it work?

The app was designed for the school population from grade 1 to grade 12, yet accessible and beneficial to everyone. By placing emphasis on reaching out to school age population, using a novel method of sonification and a visually engaging website, images, and videos, it gives them a captivating way to navigate the sounds and images of universe and help them understand that space is not just million miles away but can be reached, accessed and understood in more ways than one, opening the way to not only educate future generation but encourage, motivate, create further engagement and participation in space-related pursuits and endeavors in our future.

Sonification:

Tools used:

Tested:

Astronify
Pyaudio

Implemented

numpy library, matplotlib
Pydub library
moviepy.editor library
PIL
Audiolazy, midiutil

Approach 1:

Loop through the columns of pixels within the image from left to right (with steps > 1 to optimise the run time of the image processing step).
Take the average RGB value of each column and turn them into a corresponding sound
Based on the difference in the average RGB values, the program will produce different sounds, thus creating a melodic tune as the program sweeps through the image

Method:

In Python, loop through each column of pixels within the image, then depending on whether the image is RGB/RGBA, or grayscale, we calculate the average R, G, B values of the column:
If the mode of the image is RGB or RGBA, we use PIL’s getpixel function to assign r, g, b to the corresponding RGB value
If the mode of the image is grayscale, we assign to r, g, b the same grayscale value.
Calculating the average of r, g, and b, then turning those averages into pitch
Using the equation pitch = (red + blue + green)/10, we derive a pitch from the average RGB value of each column.

The reason why we used this equation is that, through various experiments, the resultant pitch would produce the best, most soothing sound to the ears, while still maintaining the effect of the change in RGB on the audible sound.

With the pitch produced, we derive certain notes using: note = base_frequency * (semitone_ratio ** pitch) (Where: Semitone_ratio = 2 ** (1 / 12), Sase_frequency = 130.81 (C3 frequency in Hz)
Using the notes, we plug it into a few sine waves to complicate the waveforms in order to produce a more intriguing pattern of sound. From this, we produce an audio using Pydub.
Performing the same sweeping method from left to right with the animation using moviepy.editor, and syncing the sound up with the sweeping motion through Pydub.
Finally, add the existing audio into the video using AudioSegment of Pydub and export the product.

Approach 2:

Split space image into smaller sections and map the average brightness of each section to a certain pitch.
Add the notes and export as a midi file.

Problems & Solutions:

We did not know which approach/library/tool would world the best and so we were more or less at a loss as to how to approach the problem

=> Tried every single tool we have at our disposal: TwoTone, Astronify,...

Luckily, we found Pydub as the best candidate for the challenge

In our first few attempts, the notes were too monotonous and sounded incredibly boring

=> We employed a mixture of different Sine waveforms to complicate and sci-fied the sounds. Through trial and error, we found out that:

audio = (

Sine(frequency).to_audio_segment(duration=duration_ms) +

Sine(2 * frequency).to_audio_segment(duration=duration_ms) +

Sine(3 * frequency).to_audio_segment(duration=duration_ms)

)

Produced the best results

Web Application

For the web application, we used Astro (Typescript, Javascript) and Tailwind CSS to implement the frontend of the website and deployed the completed application to Vercel via GitHub. The purpose of this web app is to promote data exploration and space sonification, improve accessibility and educational resources for the general public, and to express art and creativity on this interesting scientific topic.

3D Flythrough

For the 3D Flythrough visualizations, we adopted a similar approach to 2D image sonification. First, we tried to manually generate a sonification for the 3D flythrough. To do so, we generated a clip for each frame in the video by using RGB values and brightness values by column. We decided not to take the average of the entire 2D image since the output pitch and volume would be similar for each frame. Then, we concatenated the clips together to form a 3D flythrough video. For future enhancements, we are looking to automate this process.

Future considerations

Listening to sonification data in the form of sound/music does not have to be a passive, impersonal experience. It can be much more which captures the user’s attention in a more compelling way. As we develop capacities to make our vision come true in the form of a more interactive app, we could enable users to take their own photos of a night sky, and once they upload the photo, can have it linked back to NASA data, which then can be processed for sonification so that they will be able to access sonification data that is more personal to them. It could be also possible for users to be provided with an app feature to mix and match their own photos of a night sky, where a different sequential order would lead to a different combination of sounds/music, so every single time they can create a different musical experience for themselves using their own photos.

We can create an app with better features to help ensure an experience just as enriching for visually impaired users, who may not have the ability to look at the photos nor the night sky themselves, and help them experience acoustically what they cannot visually, by providing extra support such as screen readers, keyboard navigation, high contrast modes, and so on. Other possibilities include gamification elements of their experience with features like quizzes, challenges, fun facts, etc., which could help further increase users’ engagement with space. Also, allowing users to share their sonification through collaborative forums could have added benefits of fostering greater learning communities about space. Lastly, if given the opportunity, collaboration with space agencies, educational institutions, and healthcare associations could promote greater interest in space science & space exploration for underrepresented communities as well.

2023 NASA Space Apps Challenge

SuperSonic