MongoDB Pub/Sub with Capped Collections

If you've been following this blog for any length of time, you know that my NoSQL database of choice is MongoDB. One thing that MongoDB isn't known for, however, is building a publish / subscribe system. Redis, on the other hand, is known for having a high-bandwith, low-latency pub/sub protocol. One thing I've always wondered is whether I can build a similar system atop MongoDB's capped collections, and if so, what the performance would be. Read on to find out how it turned out...

MongoDB Schema Design at Scale

I had the recent opportunity to present a talk at MongoDB Seattle on Schema Design at Scale. It's basically a short case study on what steps the MongoDB Monitoring Service (MMS) folks did to evolve their schema, along with some quantitative performance comparisons between different schemas. Given that one of my most widely read blog posts is still MongoDB's Write Lock, I decided that my blog readers would also be interested in the quantitative comparison as well.

Python and MongoDB Travel Plans

It's been a little longer than normal between articles, so I wanted to let everyone know what I've been up to lately, and hope that I'll be able to see some of you at some upcoming events. So here's a summary of the things I'll be involved with over the next couple of months:

MongoDB Seattle

I have the privilege of speaking at MongoDB Seattle tomorrow, September 14, 2012. My talk is on "Schema Design at Scale", and wll have some insights based on experience 10gen gained when developing the MongoDB Monitoring Service. I'll even have some cool graphs based on experiments I did to illustrate the schema design tips I'll be giving.

MongoDB for Developers Online Training

I'm offering an 4-week online training class starting this Monday at 8pm EDT. I'll be delivering the training in 2 parts each week: a "lecture" portion on Mondays and an unstructured "office hours" on Tuesdays where I'll be available for questions. If you're interested, you can sign up by clicking the following link: Register for MongoDB for Developers. Right now the class is quite small, so I'll be able to spend a lot of time answering questions and interacting with students.

Python Training in Beijing

I'll be in Beijing, China to deliver an onsite "Fast Track to Python" class from October 5-10. The actual training is only on the 8-10, so if anyone's interested in getting together before that, I'd love to hang out in the People's Republic.

Big Dive and MongoTorino

I'm also happy to announce that I'm one of the featured instructors for the Big Dive training event in Turin, Italy this October. I'll be teaching a 2-day class on MongoDB and a 1-day class on Gevent, so if you're in northern Italy, I'd love to see you there! I'll also be at MongoTorino on October 19th, but the webpage for that doesn't seem to be updated yet. I'll be in Turin from October 13-20 if you want to come to either event, or just hang out over a bottle of wine.

PyCarolinas

To cap off my world tour I'm happy to be able to speak on Gevent and Socket.io at PyCarolinas on October 21st, so I'd love to see any of you there. Sadly I'll have to miss the 1st day of PyCarolinas on my way back from Italy.

And that's about it (and that's enough!). I'd love to see any of you who are in any of the cities mentioned above when I'm passing through, so feel free to contact me on twitter (@rick446), email (rick at arborian.com), or in the comments below!

Python and MongoDB Online Training

One thing that I've been planning on offering as part of Arborian's services is online training. For the inaugural class, I've decided to offer a MongoDB and Python developer training class. Read on to hear more about this class...

Last Chance for Early Bird Atlanta MongoDB Training

As announced earlier, I'm offering some MongoDB training classes in the first week of August in Atlanta. So why am I cluttering up your inbox today? Mainly because the early bird price of $400 per day ends at 11:59 pm (EDT) tonight, after which the price goes up to $500.

Two MongoDB books (one now, one early 2013)

Today I'm happy to make two book-related announcements.

MongoDB with Python and Ming Book

First off, I've collected, edited, and expanded upon the the MongoDB and Python series I've been working on at Just a Little Python and released it as an eBook. MongoDB with Python and Ming is now available on Lulu as an epub and on Amazon (at MongoDB with Python and Ming). In the book, I cover the following topics:

• Using PyMongo, from installing & basic queries to the new aggregation framework
• Overview of tuning your application for the best performance under MongoDB including discussions of replication and sharding
• Using the Ming toolkit to enforce your schema in MongoDB
• Using Ming's object-document mapper (ODM) to raise the abstraction level of your MongoDB programming
• Various tips, tricks, and goodies with Ming including Mongo-in-Memory, extending the ODM, and schema migrations

MongoDB Applied Design Patterns

The second announcement is that I've agreed to write another book for O'Reilly, tentatively titled MongoDB Design Patterns. That one's not done yet, and it's still early in the authoring / publication process, but it should be available early 2013. Sadly I don't have a nice product page link to share with you yet, but I will leave you with a summary of the topics I plan to cover:

• Part 1: MongoDB Design Patterns
  • Embedding versus referencing documents
  • Using polymorphic schemas
  • Using complex atomic updates
  • Optimistic updates with compensation (in lieu of multi-document transactions)
• Part 2: Use Cases
  • Operational intelligence / real-time analytics
  • E-Commerce
  • Content management systems
  • Online advertising networks
  • Social Networking
  • Online Gaming

I hope that's enough to whet your appetite. Of course, the book's not finished, so if you have any other topics you'd love to see covered in such a book, I'd love to hear about it in the comments!

MongoDB Training Classes in Atlanta

As promised earlier, we have now officially launched MongoDB and Python training classes. The first batch of classes will be offered between August 7-10, 2012, with early bird registration closing on July 31. We'd love to have you attend one or more classes, so read on for more info!

Multi-Master Replication in MongoDB

Since starting my own consulting business, I've had the opportunity to work with lots of interesting technologies. Today I wanted to tell you about some interesting technology I got to develop for MongoDB: multi-master replication.

Relational Mapping with MongoDB, Ming, and Python

Continuing on in my series on MongoDB and Python, this article delves into the object-document mapper (ODM) included with the Python MongoDB toolkit Ming. If you're just getting started with MongoDB, or with Ming, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework
• Schema Maintenance with Ming and MongoDB
• Declarative Schemas for MongoDB in Python Using Ming

And now that you're all caught up, let's take a look at what makes up an object document mapper, anyway...

Declarative Schemas for MongoDB in Python using Ming

Continuing on in my series on MongoDB and Python, this article continues to explore the Python MongoDB toolkit Ming, this time looking at an alternative, declarative syntax for schema definition. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework
• Schema Maintenance with Ming and MongoDB

And now that you're all caught up, let's take a look at that declarative syntax....

Schema Maintenance with Ming and MongoDB

Continuing on in my series on MongoDB and Python, this article introduces the Python MongoDB toolkit Ming and what it can do to simplify your MongoDB code and ease maintenance. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework

And now that you're all caught up, let's jump right in with Ming....

Python and MongoDB Training Classes, Anyone?

Departing just a bit from my current series on how to effectively use MongoDB from Python, today I wanted to let you, my blog readers, know that as of this past February I left SourceForge to start a new python and mongodb consulting business, Arborian Consulting. Why is this (hopefully) relevant to you? At this past PyCon, I had the opportunity to lead a python and mongodb tutorial and had a great time, but realized that the three hour format is just not enough time. So I'm hoping to start some full- or multi-day training classes, but I need your input to know what and where to offer them.

Using MongoDB's New Aggregation Framework in Python (MongoDB Aggregation Part 2)

Continuing on in my series on MongoDB and Python, this article will explore the new aggregation framework introduced in MongoDB 2.1. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)

And now that you're all caught up, let's jump right in....

Aggregation in MongoDB (Part 1)

In some previous posts on mongodb and python, pymongo, and gridfs, I introduced the NoSQL database MongoDB how to use it from Python, and how to use it to store large (more than 16 MB) files in it. Here, I'll be showing you a few of the features that the current (2.0) version of MongoDB includes for performing aggregation. In a future post, I'll give you a peek into the new aggregation framework included in MongoDB version 2.1.

GridFS: The MongoDB Filesystem

In some previous posts on mongodb and python and pymongo, I introduced the NoSQL database MongoDB and how you can use it from Python. This post goes beyond the basics of MongoDB and pymongo to give you a taste for MongoDB's take on filesystems, GridFS.

Moving along with PyMongo

In a previous post, I introduced MongoDB and its Python driver pymongo. Here, I continue in the same vein, describing in more detail how you can become productive using pymongo.

Getting Started with MongoDB and Python

If you've been following this blog for a while, you've seen me mention MongoDB more than once. One exciting thing for me is that I'll be co-teaching a tutorial at PyCon this year on Python and MongoDB that will cover MongoDB, PyMongo, and Ming. So to hopefully whet your appetite for learning more at the tutorial, I thought I'd write a few posts covering MongoDB, PyMongo, and Ming from a beginner's perspective.

MongoDB's Write Lock

MongoDB, as some of you may know, has a process-wide write lock. This has caused some degree of ridicule from database purists when they discover such a primitive locking model. Now per-database and per-collection locking is on the roadmap for MongoDB, but it's not here yet. What was announced in MongoDB version 2.0 was locking-with-yield. I was curious about the performance impact of the write lock and the improvement of lock-with-yield, so I decided to do a little benchmark, MongoDB 1.8 versus MongoDB 2.0.

MongoDB Rocks My World

I am delighted to announce that I'll be speaking at several MongoDB-related events over the next couple of months. So if you're in or near Dallas, Seattle, or Silicon Valley, I'd love it if you could make it to one of my talks. 10gen (the makers of MongoDB) do a great job of putting on regional conferences that are extremely reasonably priced (typically $30 student and $50 early bird), and there's usually a nice after-party for each one, so there's no excuse not to go.

So anyway, here are the conferences I'll be speaking at:

• November 17: MongoDallas 
• December 1: MongoSeattle 
• December 9: MongoSV (Santa Clara, CA) 

I also wanted to say a few words on why I enjoy using MongoDB and why it is such a good fit for a lot of the problems I face...

MongoDB Atlanta User Group

So last night I had the opportunity to speak about Zarkov at the Atlanta MongoDB user group (not to be confused with the MongoATL conference). It was a good crowd, and we had a good time munching on pizza beforehand and talking about MongoDB afterward. Anyway, I did a little realtime demo combining some stuff I've been doing with Socket.io, Zarkov, and Flot and promised I'd make the code available. A mostly-functional version is available at http://sf.net/u/rick446/mongodb-atl/.

Also, I figured I'd use this post to collect all the presentation paraphernalia from last night, so below are the slides and somewhat poor quality video I recorded. Hope you manage to enjoy it anyway!