• Description
• Documentation
• Build
• Docker
• Usage
• Version history
• Future Work
• Feedback
• Related

This project develops an innovative interactive tool designed to offer a real-time audiovisual experience for users. The envisioned objective of IAV is to provide individuals with limited mobility or minimal musical background an accessible way to engage with positive musical stimuli, using simple actions like hand movements, to inspire and motivate them to participate in creative expression.

You can access the full documentation for the project in PDF format by clicking here.

If you'd like to generate the documentation in different formats using Doxygen, run the following command in your terminal:

cd interactive-audio-visualizer
doxygen files/docs/Doxyfile

You may then find the generated documentation in the files/docs/documentation/ folder. For example, you can open the files/docs/documentation/html/index.html file in your web browser to view the webpage-like documentation.

A UML class diagram that illustrates the structure of all the classes in the project is available for reference. You can view it by clicking the following link: UML Class Diagram Image or check it online using the draw.io online viewer.

Before building, the following dependencies must be installed on your system:

• jack audio
• opencv
• fftw3
• samplerate
• asound2
• libx11
• xrandr
• qt6
• sqlite3
• v4l

To compile the software, follow these steps:

1. Navigate to the project directory:

cd interactive-audio-visualizer

2. Create and enter a build directory:

mkdir build && cd build

3. Generate the build files:

cmake ..

4. Compile the project:

cmake --build .

You can customize the build process with the following CMake options:

• To build the tests:

cmake -DBUILD_TESTS=ON ..

• To run the tests:

./test/test_all

• To enable static analysis using cppcheck and clang-tidy:

cmake -DSTATIC_ANALYSIS=ON ..

Otherwise, to conduct analytic checks run:

bash scripts/genStaticAnalysisLogs.sh

to generate the checks/cppcheck.log and checks/clang-tidy.log files within the checks directory.

To build the Docker image, execute the following command in the project directory:

1. Build the Docker image:

cd interactive-audio-visualizer
docker build -t iav:latest .

2. Run the Docker container:

xhost +local:docker
docker run --rm --privileged --security-opt seccomp=unconfined -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --device /dev/snd -v /dev:/dev -v $(pwd)/data:/home/iav/interactive-audio-visualizer/data iav:latest
xhost -local:docker

Before running the Interactive Audio Visualizer application, ensure that the following hardware components are connected to your PC:

• a webcam
• an audio output device (e.g., speakers or headphones)
• a display screen (obviously!)

The application will automatically detect available hardware before opening the settings menu. The application will automatically detect available hardware components before launching the settings menu, upon running the compiled executable.

To launch the Interactive Audio Visualizer, run the compiled executable:

./interactive-audio-visualizer

Upon launch, you'll be presented with the settings interface window which allows you to configure various hardware-related settings:

• Configure the audio output device
• Adjust sample rate
• Set buffer size
• Modify bit quantization factor

• Select webcam device
• Adjust resolution
• Set frame rate

• Configure screen resolution
• Set display frame rate

• Frequency Range: Adjust the frequency range for audio generation.
• ROI (Region of Interest): Configure the area of interest for the video tracking.
• Trigger Mode (Currently only default option is available): Select the trigger mode for initiating the audio visualization (/ visual audiolization) pipeline.
• Tracking Algorithm: Choose a tracking algorithm.
• Accuracy / Economy Slider (Currently unavailable): Adjust performance settings

Once you have configured the settings, press the Start button to begin the interactive audio-visual experience.. Otherwise, pressing X button in the top-right corner of the interface will close the application.

To close the application, click the "q" button.

1. A 5-second countdown timer initiates the experience.
2. The webcam captures a frame to initialize the tracking algorithm.
3. The interactive experience begins:
   • The tracking algorithm continuously updates the audiolizer.
   • The audiolizer streams audio and shares audio data with the visualizer.
   • The visualizer updates in real-time, responding to both visual and audio inputs.
4. After 10 seconds, the cycle restarts from the beginning.
5. If the tracking algorithm fails, the application automatically restarts the cycle.'

2nd Feb 2025

• Refactor the code to improve readability and maintainability.
• Encapsulate jack audio server in a class, instead of using it through a system call.
• Use Qt for the graphical user interface (GUI).
• Use various libraries (libX11, libxrandr) to detect hardware components (webcam, audio output device, etc.).
• Use SQLite for storing and retrieving settings.
• Make config a thread-safe singleton pattern for configuring the application.
• Thread refactoring to improve performance and resource usage.
• Impletent testing
• Github actions for CI/CD
• Static analysis with cppcheck and clang-tidy
• Add Docker support
• Enrich visualization with spectrogram representation
• Enrich visualization with audio volume levels visualization
• fix bugs and optimizations
  • fix camera mirroring
  • upgrade compiling to CMake
  • comply to the rule of 5

8th Feb 2024

• instructions update and improvement in compilation method

20th Nov 2023

• demo app 0.9

In the future, the app could evolve to focus on more specific and impactful goals, such as aiding music therapy, assisting those with limited mobility, or supporting individuals with little musical background.

If exciting ideas pop up, like ways to help with music therapy, support people with limited mobility, or assist those with little musical background, I encourage you to fork the project and build upon the existing codebase. Additionaly, feel free to share your insights, improvements, or inspiration on Gitter, or contact me directly at [email protected]. Contributions and feedback are always welcome!

If you liked this project, you may also like:

• Eye Harp - Playing music with the eyes