ABC-ORA

Onchain music and more, produced with ORA's onchain AI oracle

Hello all, and welcome to our project for the Autonomous Agent hackathon, an event exploring the intersection between the technologies of blockchain and artificial intelligence.

Blog : https://medium.com/@udhaykumarbala/unleashing-creativity-on-the-blockchain-with-abc-ora-e0c2027d9a80

Introduction

Our project began with the observation that the large language model Llama-2 could output short musical pieces if prompted to give them in ABC format, a type of compact musical notation often used for digitizing folk songs.

In ABC format, the melody is given in text using the letters A-G for notes, with the characters ^, _ , and = designating sharp, flat, and natural, respectively. Duration of each note in number of beats is written next to the note: if no number is written, it's assumed to be one beat. For example, in 4/4 time, DF2D would denote a quarter note D (since quarter note gets the beat), followed by a half note F (2 beats), followed by a quarter note D again. The data necessary to play the piece is in the key signature, time signature, and melody fields, although ABC format comes with other fields for metadata about the piece, such as the author. For more information on ABC format, see this resource.

If we were creating on-chain music using a web2 approach, we could produce a Llama-2 output by contacting a server off-chain, and then post the results to the chain. However, we wanted to create on-chain music in a way that was completely decentralized. Due to the high cost of memory and computation on the blockchain, having a large language model (LLM) such as Llama-2 running on layer one is infeasible. ORA provides a solution to this problem with their Onchain AI Oracle (OAO), which uses optimistic machine learning (opML) to verify on-chain ML inferences which are performed off-chain by a decentralized network. This offloads the computational burden of ML inferences from the blockchain without sacrificing decentralization or adding new trust assumptions. For more information on OAO, you can check out the official ORA website.

If the goal were only to send a prompt to Llama-2 via the OAO and receive an output on-chain, that would be easy enough to accomplish. We decided to put an extra twist on the project. Instead of just creating music based on a prompt, we would use the two AI models currently supported by OAO (Llama-2 and Stable Diffusion) to create an on-chain fictional character. The character would have an associated music and an image, both verifiably AI generated.

After that, the two are combined into an NFT representing the character. Our NFT is a special kind, however, that uses the Artificial Intelligence-Generated Content Non-Fungible Token (AIGC-NFT) standard, described in detail in Ethereum Request for Comment (ERC) 7007. It extends the common NFT standard, so every ERC-7007 token is an ERC-721 as well. However, it includes metadata specific to verifiably AI-generated content, including information relating to the proof. For more information on the AIGC-NFT standard, see ERC-7007.

Project Summary

With ABC-ORA, users can:

Input a short description of a fictional character and the mood of a piece.
Generate theme music and an image of the character onchain, with verifiable inference provided by ORA's onchain AI oracle.
Play the character's theme music through a MIDI player on the frontend.
Register this character art as an AIGC-NFT.

This project is experimental in nature: it is only intended to show what is possible with verifiable onchain ML inference, and to have some lighthearted fun. All associated smart contracts are deployed on Sepolia testnet only.

How It Works

Frontend and User Prompt

Through our frontend, the user enters a simple prompt describing their character in a few words, along with the intended mood of the theme music. To give a few examples, the user could enter

character description: "old samurai"
mood: "pensive"

character description: "schoolboy"
mood: "happy"

After connecting a wallet with enough Sepolia testnet ETH, the user can submit their input to a transaction. This will be transformed into an extended prompt, as described in the next subsection, before being sent to the onchain AI oracle.

Prompt Engineering

The user-provided input is used to create a more detailed prompt. First, the short description is given as input to Llama-2, a large language model, to produce a longer, more detailed description. This is expanded further with specific, fixed instructions for designing appropriate theme music. For example, you can think of this extended prompt as saying something like "Design a theme song in ABC format for Haru, an old samurai who has been wandering the world for decades. The piece should be in 4/4 time and reflect the pensive mood of the character." This is just to give you an idea: actual examples of extended prompts are substantially longer and more precise about the instructions for music generation.

The extended prompt will then be sent as input to ORA's onchain AI oracle (OAO).

Verifiable Inference with OAO

The OAO reads the extended prompt which is sent to it as data in a transaction. The ML inference is performed by a node on the ORA network and sent back to the chain. The inference is secured by an optimistic protocol. During a challenge period of one week, a user may try to dispute the inference. If the inference was computed incorrectly, the dispute process will find the error and the node that submitted the incorrect inference will be slashed. The result may be finalized early if the dispute process verifies an inference before the one-week challenge period is over.

While the opML protocol can be used on any model in theory, at the moment the ORA network has nodes running Llama-2 and Stable Diffusion.

NFT Creation

Once the music and image is generated for the on-chain character, the two are combined and minted as an NFT. We use the AIGC-NFT standard for verifiably AI generated content. In our case, the proof is optimistic, so the proof doesn't need to be included in the metadata.