Why Transactions Are Good for You

Thu, 27 Feb 2020 00:00:00 +0000

Databases are magical and amazing pieces of software. If you don’t agree with me, please let these awesome resources change your mind. In addition to being magical and amazing, they’re also mysterious and complex, using advanced data structures and algorithms to store data efficiently and handle concurrent access. I use databases all the time but am pretty much oblivious to how they work under the hood. Let’s fix that knowledge deficiency by getting down and dirty with a foundational concept of modern databases: transactions.

This is the first part of a multi-part series of articles. Stay tuned for Part 2, in which we’ll begin building a simple transactional database from scratch! Let’s get started.

What the hell are ACID transactions?

You might have heard that ACID in database-land stands for Atomicity, Consistency, Isolation, and Durability. Those terms are fine and dandy, but what do they actually mean? ACID refers to properties of database transactions, which are indivisible units of work that a database executes. We’ll be going over transactions, each of the ACID properties, and why they’re important.

What happens when you don’t have ACID

To better understand what transactions are, we need to first understand the problems they solve. Consider this Go function that withdraws money from a bank account. It does not use database transactions to accomplish this.

// withdraw decrements account balance by a specified amount.
func withdraw(account string, amount int) error {
    balance, err := db.Get(account)
    if err != nil {
        return fmt.Errorf("account %v not found: %w", account, err)
    }

    balance -= amount

    // Enforce constraint that balances cannot be negative.
    if balance < 0 {
        return fmt.Errorf("balance of %v for account %v is below $0",
            balance, account)
    }

    return db.Put(account, balance) // return error if Put() fails
}

This code works perfectly if only one thread is accessing the database at a time. However, we live in a world where concurrency abounds. Applications must handle requests concurrently to serve multiple users at the same time. What happens if we call withdraw() from two threads (or goroutines) at the same time?

func main() {
    // Account "brian" initially has $100.
    go withdraw("brian", 70)
    withdraw("brian", 80)
    // Now "brian" has $20???
}

This code may result in a database access pattern that looks like this:

In this example, Thread_1 and Thread_2 withdraw a total of $150 from brian’s original account balance of $100. Both calls to withdraw() succeed and the final state of the database says that brian has $20. Why?

Thread_1 and Thread_2 both read the original balance of brian and then Thread_1 commits its updated balance. Soon after, Thread_2 commits its own updated balance. Since Thread_1 and Thread_2 are unaware of each other’s actions, they both commit and the results of Thread_1 are overwritten by Thread_2. In fancier, more technical terms, this is called a lost update and is just one type of write anomaly¹ that you are vulnerable to if you don’t use transactions.

As soon as we introduce concurrency into our application, it becomes much harder to reason about database accesses. Each read or write to the database from different threads can be interleaved in any order, nondeterministically². The diagram above specifies just one of many ways those operations could have executed. If we’re lucky, both threads could run and work as expected, with one of the calls to withdraw() returning an error. This nondeterminism is what makes concurrency bugs so insidious and hard to catch.

How transactions help

Transactions solve this by being atomic. They allow you to execute multiple operations together, as a single unit of work. Transactions guarantee that their operations will never be interleaved with those of another transaction that might conflict with it.

Here’s what a transactional version of our withdraw() function might look like³:

// withdraw decrements account balance by a specified amount.
func withdraw(account string, amount int) error {

    // Execute database transaction to atomically withdraw from account.
    return db.Update(func (txn *db.Txn) error {
        balance, err := txn.Get(account)
        if err != nil {
            return fmt.Errorf("account %v not found: %w", account, err)
        }

        balance -= amount

        // Constraint: can't have a negative balance!
        if balance < 0 {
        return fmt.Errorf("balance of %v for account %v is below $0",
            balance, account)
    }

        return txn.Put(account, balance) // return error if Put() fails
    })
}

If we use transactions, withdraw()’s Get and Put operations will always execute together. Thus, the prior example of conflicting calls to withdraw() will always result in one of the calls failing. In other words, using transactions makes lost updates impossible! Hooray!

In addition to atomicity, transactions offer other guarantees like consistency, isolation, and durability. Here are their definitions:

Atomicity – all steps of a transaction will finish successfully or none of them will. There will never be an “in-between” state that other transactions can see.
Consistency – a transaction can only cause the database to move from one valid state to another valid state. All database invariants and constraints defined by the user must be honored at all times. Consistency guarantees that databases cannot be corrupted by illegal transactions⁴.
Isolation – transactions can run without interference from any other concurrently executing transactions. The isolation level of a database controls both when and what changes to the database become visible to transactions. Isolation is heavily influenced by the database’s mechanism for concurrency control.
Durability – when a transaction commits, it will immediately write changes to persistent storage. If the database or machine crashes, the database can recover all committed changes from disk.

All of these properties work together to help ensure that modern databases maintain data integrity.

Conclusion

Yay! We learned a lot about ACID transactions and the types of bugs that they help prevent. That’s pretty good for one blog post, if you ask for my totally unbiased opinion :wink:.

In Part 2 of this series, we’ll start building lsdb, an in-memory key-value store that supports ACID transactions. We’ll learn how to implement transactions in code and explore multiversion concurrency control, the concurrency control mechanism used in Postgres. Stay tuned!

References

Database Internals by Alex Petrov
Designing Data Intensive Applications by Martin Kleppmann

Other read and write anomalies that can occur include dirty reads, nonrepeatable reads, phantom reads, dirty writes, and write skew. You should look them up if you’re interested in them but they won’t be covered here for the sake of conciseness. ↩
The order is nondeterministic because thread execution is dictated by the operating system’s scheduler. If multiple threads are performing reads/writes at the same time, the order in which they execute is largely opaque to and uncontrollable by applications unless some sort of synchronization method is used. We’ll be covering concurrency control methods in Part 2. ↩
This syntax is inspired by BadgerDB, a database written in Go that supports ACID transactions. ↩
Note that consistency does not guarantee the correctness of the transactions that you write. Constraints here refer to things like UNIQUE or NOT NULL in Postgres, not application logic constraints. ↩

Counting People with Machine Learning

Thu, 07 Dec 2017 00:00:00 +0000

I am a member of HackCville, a collective of UVA students and Charlottesville community members who are passionate about entrepreneurship, technology, and getting stuff done. This semester, I was a member of the Node, HC’s data science program (I’ll be a node TA next semester! HC is open to everyone, apply!). In order to become a fully fledged HackCville member, every program member must complete and present a final project using what they learned in their program. For my project, I chose to build a program that could track the number of people, via camera, at a place over time.

[12/8/17] Note: Wow, this post really blew up on HackerNews! You can check out the discussion here.

Brainstorming

I frequent HackerNews, and as any HN reader knows, machine learning and the internet of things are The Future and if you aren’t working in those areas you might as well not be working (/s!). So, I set out to find a project that combined these two topics in an interesting way. While brainstorming, I remembered a website that I had stumbled upon a few months ago: a directory of unsecured, internet connected webcams from all over the world called Insecam. Although I was initially creeped out by Insecam, I was fascinated with the idea that I could peer into so many different corners of the world just by clicking on a couple of links. I also remembered a cool neural network architecture that I had read about that achieved state of the art object detection, YOLO (read the paper here), which is built using the Darknet deep learning library. I thought it would be pretty cool if I could scrape data from Insecam and then process it using YOLO to gain some sort of insight. With that, I set out to analyze the frequency of people at certain locations using Python, Jupyter Notebooks, YOLO, and data from Insecam.

Scraping Video

First thing’s first, I had to figure out how to scrape data from Insecam. I first went on a hunt for an Insecam API, which unsurprisingly (due to Insecam’s sketchiness), did not exist. I then investigated each individual video stream and noticed that right clicking on each video frame and opening the link in a new tab brought me to each webcam’s IP address. Great! Now I had something that I could scrape. I first tried to use wget to see if I could simply send a request and get a file. Unfortunately, wget seemed to continuously download the video stream without ever stopping. I learned that most of these IP webcams streamed videos in Motion JPEG and so I had to figure out a way to process it. Luckily, after some quick digging, I found that OpenCV has a quick and easy cv2.VideoCapture class that can pull image frames from mjpg video streams.

OpenCV to the rescue! A frame from an mJPG plotted in my Jupyter Notebook.

Python and C Interop

Next, I had to find out how to run Darknet, which is written in C, from Python. While some obvious choices would be to use FFI or to make a Python extension module, I didn’t want to get lost in the rabbit hole and spend days developing a Python extension, so I found a somewhat hacky solution: using the Python subprocess library, I would run the YOLO executable on an image pulled from a webcam and then parse the console output from that executable to obtain the results. Using this strategy, I could get predicted objects and confidence data from YOLO quite easily. By counting the number of output lines with person in them, I could get the number of people that YOLO thought were in the image frame at that instant in time. It was at this stage that I decided to analyze a small restaurant in Rotterdam.

Examples of YOLO running on images from the restaurant webcam.

cam.png: Predicted in 4.906712 seconds.
person: 61%
person: 56%

Example output from the Darknet executable

Collecting Data (Lots of it!)

Now that I had a means of obtaining the number of people at a given location based on webcam data, I wanted to collect a lot of data. Since I didn’t want to have to run YOLO on my laptop 24/7, I figured that the best way to collect data nonstop would be to write a quick and dirty Flask server that could be deployed on a cheap VPS or AWS instance. In order to do this, I wrote two Python scripts. The first Python script continuously pulled image frames from the webcam, processed the frames with YOLO, and then appended the UNIX timestamp and number of people to a CSV file on the disk. The second script was a simple Flask server that hosted the raw webcam image, the YOLO processed image, and the data log. Over the course of the week that I ran this experiment, I was able to collect over 55,000 data points, adding a new data point every 20 to 30 seconds.

Insights

After collection over 55,000 data points, I loaded the file into a Jupyter Notebook and used Pandas and matplotlib to try and gain some insights from the data. I binned the data into windows of 5 minutes and averaged all data points withing those 5 minutes to account for variations in the number of people that YOLO counted. I then plotted the data in two ways. One plot showed the average number of people over the course of a single data and the other showed the number of people over the course of the week that I ran the experiment. It’s pretty cool how the data turned out– there is noticeable periodicity in the week graph and you can clearly see the number of people vary with closing hours and dinner time in the day graph. Pretty neat, if you ask me.

Average number of people over the course of a day

Number of people over the course of a week

Conclusion

Overall, this little experiment turned out much better than expected. While the results themselves were unsurprising, I was surprised that YOLO was able to recognize people so well as to see noticeable periodicity over time that correlated with the closing hours and dinner spike of the restaurant. However, this project was far from perfect and there were some problems that could be improved with future work.

Problems & Future Work

One major problem was that YOLO sometimes failed to detect people. Whether it be because of poor camera quality, bad lighting conditions, or a problem with YOLO, this may have caused the number of people recorded in the data to be artificially low. In future work, a convolutional neural network specifically trained to detect people could be used instead of YOLO, which is a pretrained convnet. Furthermore, a better user interface could be developed and a better camera could be used to perhaps turn this into a product that restaurant or property owners could use to keep track of foot traffic.

Oops! YOLO missed that whole group of people.

Endnotes

If you want to get this up and running on your machine, all the code is in a Github repo. Just download and compile Darknet from the YOLO page, download the YOLO 2.0 weights and then run both prediction.py and server.py. Alternatively, you can look through the Jupyter notebook. Warning! This code is quite ugly, I will try to clean it up in the future.

Any questions? Just shoot me an email!