Traits in Rust

Here’s a simple example of how Rust deals with traits:

What happened there? I created 2 structs Foo and Bar, and for each of them I implemented a function X() that would print their a field. This works because both Foo and Bar have a field a. After, I also implemented a function Y() that would print the b field of Bar since only Bar has a field named b. If I tried the same macro trick to implement Y() for Foo, the compiler would yell at me that Foo doesn’t have a field b.

Let’s see how to do this in JavaScript

What happened here? Functions in JavaScript are passed a context argument when they’re being called. It’s automatically assigned when invoking functions as X() or Y(), but if you invoke them with .call() you can specify your own context. The context is available as this inside your function. This allows us to model a struct in Rust as a typical JavaScript object. Then implementing a function for a struct is simply passing in our JavaScript object as the context to the function.

Pros and Cons

Ultimately, it all boils down to safety vs. flexibility. Rust guarantees at compile time the struct supports the function being implemented. Using the earlier example, implementing Y() for Foo is impossible because Foo doesn’t contain the field b. In JavaScript this is totally possible, Y.call(foo); would simply result in a is 1 \n b is undefined. On the other hand, JavaScript doesn’t require a “solution” like a macro to implement the same function for multiple structs.

In a future post I will also expand on the similarities of function overriding between Rust and JavaScript.

Picking a dataset

We used Microsoft’s CustomVision to train our datasets and handle API calls for classifying images. Our first plan was to teach the AI how to understand the whole ASL alphabet. We found a dataset online with almost 2000 images, but they were all of the same person in the same setting. This made it very difficult to classify our signs because we were different people and the pictures were in a different setting. Additionally, we had a limit of 1000 images to train and with 26 different classes, we only had roughly 50 pictures per letter. We had an 8% prediction rate with this dataset.

Our 2nd approach was to restrict our domain to only a few words. This would allow us to have more training images for each class. Also, we would generate our own dataset using a variety of people (other hackers at the hackathon) in a variety of settings (against a white wall, green wall, people walking in the background, etc.) to build a more robust classifier. Our training set has 1000 pictures, here’s a few of them:

With our new dataset ready and trained we got an 80% prediction rate!

Sign language isn’t static, what about signs that have movement?

The time constraint didn’t allow for developing a context-aware solution to sign movement so we had to improvise. Our solution was to select “key frames” of a particular sign movement and train our AI on those.

Here’s how you can say no in ASL:

We chose 3 key frames for this sign:

This allowed us to capture any point during the motion of the sign and still correctly classify it.

Not a perfect solution

ASL (like any language) is much more complex than what you can capture in the span of 36 hours. Here are a few improvements we thought of:

• Languages are contextual and the meaning of a sign can change based on what was said earlier
• We had a very limited vocabulary
• People have different “accents” when they sign
• The signs we trained mostly “looked” different so we didn’t have to handle minuscule changes in movement that change the meaning
• Many signs also have facial expressions (like the man shaking his head for “no”) which we didn’t focus on for our training data

Communicating with the bot

We used Microsoft’s BotFramework for developing a conversational bot. After parsing the sign into text, the python server would prompt the bot API for a response. The bot was designed to hold up a simple conversation with you by answering questions and asking a few of its own.

A quick walkthrough

Your webcam captures you signing the word “hello”

The image is sent to CustomVision for classifying and returns a tag. That tag is sent to the bot. The bot responds accordingly.

Looking forward

Since no special hardware is required (just a webcam) and all of the hard work is done by Microsoft’s servers, we can easily extend this into a mobile phone app.

Finally, we would love to extend our dataset and train on a variety of people to improve our classifier. Some variables we would like to consider:

• Varying skill levels (all the images were of beginners)
• Different lighting/camera conditions
• Multiple people signing in one image

Rumsha7/signkit-learn

Special thanks: Jaison Loodu, Sophia Tao, Rumsha Siddiqui, Aman Adhav

How it got started

The idea for this project came while I was at a hackathon in Montreal, Canada with Justin Leger and @icefalcn. Our plan was to hide sensitive files in unassuming ones like vacation photos and PDFs. While we were successful in hiding the files in an easy to use system, the implementation was well… hacky. Our solution was to encrypt a sensitive file and append it to the end of an image. Looking at the image in a viewer wouldn’t give away the encrypted file added to the end, but looking at the file size or cating the image definitely would. With another hackathon come and gone, I set out to implement it right.

Why Rust

I chose Rust because it had good library support for image manipulation, it worked seamlessly on both my Chromebook and Windows boxes, and most importantly, I wanted to gain expertise with the language.

Getting my grips

The library started simply as reading and writing buffers to the alpha channel of an image. It grew as I added utility functions converting a String to an array of bytes and reading or writing that. Then I added converting whole files to byte arrays. And I realized something: my library was starting to become about encoding things as byte arrays to read/write into the alpha channel. It should have been about adding actual steganographic methods.

The Big Refactor AKA: 1.0.0 (stable)

I factored the program into 3 modules: Encoder, Decoder and Util. I moved all the “added functionality” of converting things to byte arrays into the Util module. The core of the library, the actual steganography, now exists in Encoder and Decoder. With the functionality of the library nicely separated, it was very easy to add some new features! Alongside encoding bytes into the alpha channel of images, I also added encoding bytes as an image.

If you look closely at the bottom of mspaint.exe, there’s some nice 🌈s.

Next steps

Short term, adding a command line interface is top priority.

It can be easy to spot fishiness when encoding bytes straight into the alpha channels. So I’d like to add some established steganographic methods like F5, JSteg and LSB.

The crate should be (and is!) easy to use. It’s how I was sidetracked into spending so much time at the beginning developing utility functions. I want to keep the focus on steganography, but I see a big benefit in adding some simple encryption functions to the Utility module.

Steganography isn’t limited to just images… The next big step will be adding methods to discreetly encode data in audio and video.

A performant binary which can run on many platforms that allows for advanced steganographic methods through a simple interface will have a profound impact on privacy. I imagine a DSLR camera that can hide pictures in other pictures on the fly, giving reporters more freedom to not fear reporting the truth.

steganography