Purging Varnish cache from Python web application

Varnish is a popular front end caching server. If you are using it at the front of your web site you have situations where you wish to force the cached content out of the cache E.g. site pages have been updated.

Below is an example how to create an action to purge selected pages from the Varnish cache. The example shows how to purge the whole cache, but you can use regular expressions to limit the effect of the purge.

This tutorial has been written for Ubuntu 8.04, Python 2.6, Varnish 2.x and Plone 4.x. However, it should be trivial to apply the instructions for other systems.

We use Requests library which is a great improvement over Python stdlib’s infamous urllib. After you try Request you never want to use urllib or curl again.

Configuring Varnish

First you need to allow HTTP PURGE request handler in default.vcl from localhost. We’ll create a special PURGE command which takes URLs to be purged out of the cache in a special header:

acl purge_hosts {
        "localhost";
        # XXX: Add your local computer public IP here if you
        # want to test the code against the production server
        # from the development instance
}

...

sub vcl_recv {

        ...

        # PURGE requests call purge() with a regular expression payload from
        if (req.request == "PURGE") {
                # Check IP for access control list
                if (!client.ip ~ purge_hosts) {
                        error 405 "Not allowed.";
                }
                # Purge for the current host using reg-ex from X-Purge-Regex header
                purge("req.http.host == " req.http.host " && req.url ~ " req.http.X-Purge-Regex);
                error 200 "Purged.";
        }
}

Creating a view

Then let’s create a Plone view which will make a request from Plone to Varnish (upstream localhost:80) and issue PURGE command. We do this using Requests Python lib.

This is useful when very fine tuned per page purgingn fails and the site admins simply want to make sure that everything from the front end cache will be gone.

This view is created so that it can be hooked in portal_actions menu and only the site admins have permission to call it.

Example view code:

import requests

from Products.CMFCore.interfaces import ISiteRoot
from five import grok

# XXX: The followig monkey-patch is unneeded soon
# as the author updated Requets library to handle this case

from requests.models import Request

if not "PURGE" in Request._METHODS:
    lst = list(Request._METHODS)
    lst.append("PURGE")
    Request._METHODS = tuple(lst)

class Purge(grok.CodeView):
    """
    Purge upstream cache from all entries.

    This is ideal to hook up for admins e.g. through portal_actions menu.

    You can access it as admin:: http://site.com/@@purge
    """

    grok.context(ISiteRoot)

    # Onlyl site admins can use this
    grok.require("cmf.ManagePortal")

    def render(self):
        """
        Call the parent cache using Requets Python library and issue PURGE command for all URLs.

        Pipe through the response as is.
        """

        site_url = "http://www.site.com/"

        # We are cleaning everything, but we could limit the effect of purging here
        headers = {
                   # Match all pages
                   "X-Purge-Regex" : ".*"
        }

        resp = requests.request("PURGE", site_url, headers=headers)

        self.request.response["Content-type"] = "text/plain"
        text = []

        text.append("HTTP " + str(resp.status_code))

        # Dump response headers as is to the Plone user,
        # so he/she can diagnose the problem
        for key, value in resp.headers.items():
            text.append(str(key) + ": " + str(value))

        # Add payload message from the server (if any)

        text.append(str(resp.body))

        return "\n".join(text)

Now if you are a Plone site admin and visit in URL:

http://site.com/@@purge

You’ll clear the front end cache.

More info

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Mobile browser cache sizes

iPhone can only hold 15 x 25 kb objects in cache is not true. At least anymore, I wonder if it ever was.

Android and iOS boast 2 MB – 4 MB cache for a session. This should easily include your Javascripts, no matter what kind of monster sizes you are serving. So the cache exhaustion should not be a problem in everyday usage.

This also means that mobilize.js should not, if using the cloud version which properly set expire headers, cause extra traffic after initial load.

Running 32-bit chroot on 64-bit Ubuntu server to reduce Python memory usage

Here are documented brief instructions how to run 32-bit chroot’ed environment on 64-bit Ubuntu server. chroot means that you run re-rooted and jailed system inside another system.

What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much  as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption)

32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB).

There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article.

Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support.

Installing

Basic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch.

apt-get install debootstrap
apt-get install schroot

# Old schroot uses global schroot.conf, new versions have
# chroot.d directory

# This is a heredoc, but use what ever editor you like
# to create the configuration
cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf
[hardy_i386]
description=Ubuntu 8.04 Hardy for i386
location=/srv/chroot/hardy_i386
personality=linux32
root-users=bob
run-setup-scripts=true
run-exec-scripts=true
type=directory
users=bob,john,alice,ploneuser
EOF

mkdir -p /srv/chroot/hardy_i386
debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \
http://archive.ubuntu.com/ubuntu/

# Check that the chroot is created and working
schroot -l

# Enter the chroot (logged in as bob)
schroot -c hardy_i386 -u root

# Once inside, install python2.4-dev and other needed tools
# Installing PIL with easy_install didn't work for some
# reason, so we use python-imaging package.
apt-get install python2.4-dev python-setuptools python-imaging
easy_install-2.4 ZopeSkel

And that’s about it.

Chroot’ed environment will have

  • It’s own application binaries (and userland bitness)

Chroot’ed environment will share with the host system

  • Ports
  • Processes
  • User accounts

Chroot’ed users

  • Can’t list filesystem outside chroot

…giving additional safety for shared hosting in the case of chroot’ed environment is compromised.

Entering chroot’ed environment as a specific user

Try

schroot -c hardy_i386 -u root

…or…

schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment

All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below.

Creating chroot sessions

Sessions enable running commands in chroot without the need to have it constantly open.

# Create a new schroot session
schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession \
--begin-session

# Run commands in the created session
schroot --chroot=plonesession --user=ploneuser --run-session \
/srv/plone/yourplonesite/bin/instance start

# Ending session
schroot --chroot=plonesession --user=ploneuser --end-session

Note that –-chroot parameter takes in both actual chroot installations and session ids.

Doing resets for chrooted environment

The session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent. chroot’ed environment is temporary unless you explicitly specify it not be Thus if you want to run daemonized services in chrooted environment you need to take care of session handling manually. Here is an example how do you construct a session (as a real root user) and then launch a shell script which will take care of launching applications inside the chroot.

# Terminate previous session if any
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session
#Start the session (again)
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session
# Run a start script inside the chroot'ed environment which will start Plone
# NGINX and other necessary 32-bit services
schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh

Running sessions at startup

You can add schroot bootstrap in real /etc/rc.local:

schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession --begin-session \
&& schroot --chroot=plonesession --run-session \
/srv/plone/inst/bin/instance start

Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this

adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser

Then to create the directory inside the chroot and set its ownership to the newly created user and group.

32-bit vs 64-bit memory consumption

Reason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010)

Memory figures from a development Zope/Plone 3.3.5 server

32-bit 64-bit
After startup 112 MiB RES 116 MiB VIRT 175 MiB RES 342 MiB VIRT
After normal usage 159 MiB RES 194 MiB VIRT 236 MiB RES 487 MiB VIRT

This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase.

More information

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

To gzip or not to gzip CSS and Javascript?

Dear Lazyweb,

Is it wise to gzip static resources (javascript and CSS) before sending them to client?

The opinion of Internet seems to be divided

  • GZip decompression takes too much time at the client end and thus it is not wise (latency at the clients end)
  • Bandwidth save is enough to counter the decompression latency

So if we put this to context of

  • Compressed and merged CSS and Javascript files of Plone
  • Assuming the users are using the state of the art browsers: Safari 4, Chrome, Firefox 3.6 and IE8
  • Connections are faster than 384 Kbit/s

…and forget…

  • Recompressing images (GIF, PNG and JPEG) using GZip as there is no notable save

should I enable GZip compression on the front-end server (Apache) with disk cache enabled?

Of course, I could do testing and timing myself, but I’ll simply ask for your experiences first before investing few hours for this. Also, hints how to measure how fast GZip decompression is, are welcome.

Thank you.

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+