With a NoSQL DB like MongoDB, that may not be the case. While it is nice to have Django-like objects for your documents, such as MongoEngine provides, if write/update performance is critical, it may not be the best route, as my last experiments showed. But after deciding to use PyMongo directly (a reader pointed out that MongoKit allows the direct use of PyMongo, so check out if that framework suites your needs), and working with it a bit, I felt the need for at least some tiny abstraction layer. Why?

Interfacing with MongoDB is very easy. You want to insert a product in a collection called ‘products’?:

from pymongo import Connection

connection = Connection()
db = connection['mydb']

db.products.insert({'article_number': '100', 'name': 'Cool Shoes'})

It is even too easy. Let’s insert a new product:

db.productts.insert({'article_number': '101', 'name': 'Sunglasses Aviator'})

What’s wrong? A typo will cause your new product to be inserted to a wrong collection named “productts”. No, you have not previously defined any such collection, but PyMongo will create it on the spot, potentially leading to bad, nasty bugs. For this and other reasons, you need to abstract your MongoDB collections, which I do with some light-weight classes:

from pymongo import Connection

connection = Connection()
db = connection['mydb']
Product.ensure_indexes()

class Product(object):
    collection = db.products

    def __init__(self, data):
        self.data = data

    @staticmethod
    def ensure_indexes(self):
        Product.collection.ensure_index(
            ('article_number', ASCENDING), unique=True)

    def save(self):
        current_date = get_timestamp()
        o = self.collection.find_one()
        self.collection.update(
            {'article_number': self.data['article_number']},
            {**self.data}, upsert=True, safe=True)

Now the error described previously won’t ever happen again, because you will always use your (non-instanciated) class:

Product.collection.insert(
    {'article_number': '101', 'name': 'Sunglasses Aviator'})

or

Product.collection.find_one({'article_number': '101'})

And you can build on your class as you like to add validation, timestamps, indexes, or any nicety you may need. In my example you can save a new document in this familiar way:

p = Product({'article_number': '101', 'name': 'Sunglasses Aviator'})
p.save()

Of course, if you see that your wrapper classes grow in complexity and/or start impacting write speed, you should better use MongoEngine or MongoKit. But I think that this solution strikes a nice middle ground.

The new data import should be able to import or update a million products much quicker than the current 2 hours needed, and it should be able to keep scaling. MongoDB seems like a good candidate because it should be much faster than SQL. The new import will also be able to be executed as distributed jobs.

The new product store backend being written in Python, what abstraction should we use on top of MongoDB?. Working directly with PyMongo is an option, but usually an Object Document Mapper (like SQLAlchemy or Django ORM) is preferable.

As it was my first time with MongoDB, I researched a bit, and  MongoEngine looked best, as it has a similar syntax to Django and seems well supported. But will it be fast enough, or does it introduce too much overhead compared to bare PyMongo? Not being one that just assumes things, I went ahead and checked that out myself.

The system I’ve built reads rows from a CSV, validates fields (for example that a given field is a valid price), and saves them to an storage backend. It is very modular, as not only is that usually a good design decision, but it allows me to have pluggable storage backends:

• to_file backend. This just drops the validated row or product into a new file. We need this as a baseline to now the absolute maximum speed rows could be saved at.
• An SQL backend. This will serve as a comparison to a classical SQL system. I use the Django ORM here, and to not be unfair I enable the transactions.commit_on_success() decorator and use a sqlite in-memory DB.
• MongoEngine. The code is nearly the same as in the Django ORM case. You can find it here.
• PyMongo. This will directly use the MongoDB Python driver, using safe=True. The importing method looks like this.
• devnull. We all want to be web scale, so I went ahead and implemented a devnull backend.

The code is purposely a naive implementation, to test how fast each back end is without resorting to optimizations, hacks or tricks. There are probably ways of making it much faster. And even though the production code will be very different to this early experiment, it is not an evil, synthetic micro-benchmark: on the contrary, it is a real application!

Benchmark details

• Average of three runs
• Importing 2 small-medium sized product lists with 4000 and 8000 rows
• 100% updates (will be the most common case)
• Code flow:
  1. Set all existing entries to old=True
  2. Upsert all new entries and set old=False
  3. Delete all entries still marked old=True

Results

Some take aways

• MongoEngine is 7x faster than SQL. This is actually eerily similar to Peter’s results!
• PyMongo 5x faster than MongoEngine
• PyMongo 36x faster than SQL
• Incredibly, the devnull backend is even faster than writing to a file!
• PyMongo 40% as fast as writing to a file

That last bit of data is nothing short of amazing. Since then, I have added a bit of an abstraction to the PyMongo storage engine, more MongoDB operations, per product/per field timestamps and more, and it is still every bit as fast.

This experiment has settled the matter: we will be using PyMongo directly (I am tempted to use the super-cool devnull backend, but I have heard that it is somewhat difficult to later retrieve data).

In the next post, I will be explaining how I’ll go about abstracting PyMongo a little bit, without taking away one ounce of its incredible speed.

The Django ORM is really great. It allows you to design and code your web application in a very simple and readable way.

Some projects have special needs, or are very demanding on the DB, and in that case the ORM can became a bottleneck. Fortunately, Django has different ways to avoid getting in your way.

Inserting thousands of entries in the DB

After the Overmind 0.1 release, I have implemented the caching of provider images, so that the node creation form loads instantly. Once implemented though, a problem quickly became apparent: Amazon’s EC2 has nearly 3000 AMI images. Saving all of them using the Django ORM is extremely slow. It takes 10 minutes!

While Overmind will get a work queue and the user won’t have to wait after creating a new cloud provider, 10 minutes is way beyond reasonable. Then again, I really don’t want to start entering SQL code in this early stage of Overmind development, as we all know that “Premature optimization is the root of all evil”[1].

Fortunately, Django has a nifty and often overlooked feature: transactions.

When you look for a solution to bulk inserts with Django, most answers recommend SQL statements and when someone proposes using transactions, they answer that they are not real bulk inserts and they won’t bring very big performance improvements. Let us put that to the test.

Benchmarking transactions

The tests consists in creating an EC2 provider in Overmind, and meassure how long it takes to complete the action. In the process, the nearly 3000 EC2 images will be imported into the local Overmind SQLite DB.

The simple Image model:

class Image(models.Model):
    '''OS image model'''
    image_id = models.CharField(max_length=20)
    name     = models.CharField(max_length=30)
    provider = models.ForeignKey(Provider)
    favorite = models.BooleanField(default=False)

    def __unicode__(self):
        return self.name

    class Meta:
        unique_together  = ('provider', 'image_id')

The test has been run 3 times with and without transactions. To that end, the only change was adding a decorator to the Provider. import_images() method:

...
    @transaction.commit_on_success()
    def import_images(self):
...

Results

The results are outstanding. This is not a trivial improvement, it takes only 2.7% of the time it needed before!

If you want more performance, there are some custom solutions for real bulk saving, and if all else fails, Django even lets you enter custom raw SQL statements. django.db.transaction though, should be a beautifully simple and effective solution for a lot of projects.

References
[1]: Famous quote by Donald Knuth