***Please take only 30 secs of the Project Demo video into consideration

Our project is a web app with its primary feature being the combining of the ‘sounds’ of preset three-dimensional images into a musical mashup. The web app has a library of images that 'display' sounds, the idea is that the user would choose two images in the mashup tool within the app and it would produce a mixture of the sounds of the chosen images. Its primary benefit comes in the fact that this web app is designed to aid those that are visually impaired or primarily reliant on their auditory senses the most. Furthermore, since sound waves cannot travel in space, this web app is an attempt at generating sounds & adding personalisation to the many areas of space. This web app could be instrumental in further exhibiting the beauty of space to the general public including those that may be at a disadvantage when it comes to accessing/viewing images of space. This web app could be the key in allowing these people to have an opportunity at perceiving the beauty of space in the form of sound, thus increasing the level of overall appreciation of our universe.

We utilised NASA’s catalog of RGB colour model images taken by the Wide-field Infrared Survey Explorer (WISE) that were available in TIFF format. We chose Python as the primary coding language to analyse and sonify these images as well as making use of a variety of libraries such as ‘Pandas’, ‘NumPy’, ‘Matplotlib’, and ‘MIDIUtil’. Furthermore, ‘Poetry’ and ‘Anaconda’ were used as dependencies for the litany of libraries that we have used as a team. Going back to MIDIUtil, it was used as the primary library to write the code behind the sounds which was then synthesised into the audible sounds via the software synthesiser called ‘FluidSynth’. When it comes to the graphical user interface, ‘Figma’ was used to create the pages that would be on the front-end of the web app.

Below are the steps we undertook to develop our web app demo:

Feature A - Convert lists of RGB pixels into lists of frequencies for a python audio package

Find image datasets from WISE that contain RGB data
Add NASA .tiff images to GitHub and download.
Unravel the .tiff images to access RGB values for each pixel.
Create data structures that describe the image data.
Datastructure1: Data structure that contains nested lists of chunks of averaged RGB pixels values. The data structure creates chunk segments of each row of pixels where the chunk summarises the average of the RGB values. The benefit of this is that the RGB values in each chunk can be mapped to 3 instances of arbitary instruments. The different RGB values of the chunks characterise different sounds (pitch, loudness etc,) of the instrument.
Datastructure2: The data structure uses a version of the RGB image that has been converted to grey scale. The grey scale metadata is classified as a shade and chunked. The greyscale shades are mapped to three backing tracks, where their volume is determined by their quantitative value.
Add loudness, pitch, track, duration and start time parameters to each instrument node.
Build MIDI file and output MIDI file.
Convert to .WAV file.

Feature B - Create a UI interface for the mash-up app

Use Figma to build a prototype to demo the graphic user interface of the web app including the mash-up tool and sonification library.
Add videos of combined visualization and sonification of the WISE images to UI.

Feature C - create a visualization of the sonificiation

Use python pillow to read .tiff files into IDE.
Create 3D render of image with colour as third dimension in matplotlib.
Create animated change in RGB values in matplotlib.
Create animation of sound progression through .tiff image.

Here is the link to our project Google Drive:

https://drive.google.com/drive/folders/1afCM2oGI0iTOVSatthurf14MCfNTU908?usp=sharing

2023 NASA Space Apps Challenge

Team Cassini