WebAlchemy accelerates Django in 100 times

I host my Django blog at shared hosting environment on Dreamhost, where hundreds of sites are also running on the same server. In general I enjoy Dreamhost hosting (especially, taking into account their price for unlimited domains hosting plan) but sometimes I need more speed. When I wrote WebAlchemy and configured www.mysoftparade.com to use it, I achieved almost immediate responses of my web-site.

With WebAlchemy only pages involved in form processing are served directly by Django, the rest of the pages most of the time are served directly by Apache as static content with static content speed. In other words for Django-powered site it's possible to achieve speed about 2000 request/sec, against about 500 request/sec with memcached and about 20 request/sec for "typical" (10 fast SQL queries) page without caching at all. Actual performance results of course will vary from server/application/configuration.

As it was said: there are only two hard things in Computer Science: cache invalidation and naming things. WebAlchemy radically resolves the problem of stalled resources in cache that is natural for any caching layer based on memcached, squid or etc. With WebAlchemy pages are never become staled.

The magic is done inside concept.webalchemy.core.WebAlchemyMiddleware. When page request is passed via this middleware, it saves the response data as file inside Apache htdocs directory and add to .htaccess file in the same directory a special rewrite instruction. Then all next requests to the same url, Apache will treat as requests to static file saved at the previous step. When something that affects rewritten url will be changed in application, WebAlchemy will update corresponding .htacces file again and default behavior of the web-server will be restored. So WebAlchemy depending on situation will made the same url dynamic or static. When someone writes to url, it becomes dynamic. When someone reads url, it becomes static. Like yin and yang in the Chinese philosophy.

For example, let's analyze life-cycle of the current page http://www.mysoftparade.com/blog/webalchemy-django-apache/.

1. The blog entry was just wrote and stored in database.
2. The web-server receives request to the page /blog/webalchemy-django-apache/, request is passed to mod_python/FastCGI/etc and then to Django, that routes it to concept.webalchemy.core.WebAlchemyMiddleware.
3. The middleware finds view responsible for serving the url – concept.blog.views.entry_detail and realizes that this view in the list of binded views. WebAlchemyMiddleware saves response generated by entry_detail view in the file htdocs/blog/webalchemy-django-apache/index.html and populate file htdocs/blog/webalchemy-django-apache/.htaccess with following content:

   #webalchemy_begin
   <IfModule mod_rewrite.c>
   RewriteEngine On
   RewriteBase /blog/webalchemy-django-apache/
   RewriteRule ^$ /blog/webalchemy-django-apache/index.html [L]
   RewriteRule ^index\.html$ - [L]
   RewriteRule ^dispatch\.fcgi/.*$ - [L]
   RewriteRule ^(.*)$ /dispatch.fcgi/blog/hello-world/$1 [L]
   </IfModule>
   #webalchemy_end
   

4. Web-server receives new request to to the page /blog/webalchemy-django-apache/ but this request will not be passed to Django because created at the previous step .htaccess contains rewrite instruction for Apache that instruct it to send file /blog/webalchemy-django-apache/index.html back to client immediately. Apache do this and will continue to do this even if you will stop Django and MySQL for maintenance. Here is an important nuance that I want to highlight: WebAlchemy don't uses any timeouts, so when everyone forgot this article and it will not be changed/commented anymore, web-server will serve it as static file forever: days, weeks, months...
5. However if something that affects this page will occur (a comment to this entry will be posted, or entry is updated for example), WebAlchemy will immediately remove /blog/webalchemy-django-apache/index.html and htdocs/blog/webalchemy-django-apache/.htaccess and so next request to this page will again be passed to Django. So we return to the stage 2 of the page lifecycle.

WebAlchemy uses Django signal framework to monitor changes in models. Mapping between models and views are defined in file webalchemy_settings.py. Bellow is content of this file for the www.mysoftparade.com:

# -*- coding:utf-8 -*-

# $Id: webalchemy_settings.py 289 2007-11-18 17:03:13Z dogada $
# Copyright (c) 2007, Dima Dogadaylo (www.mysoftparade.com)

"""WebAlchemy configuration for the www.mysoftparade.com.
Mapping between models and views that models affect.
WebAlchemy uses this information to switch between
dinamic and static versions of an url.
"""
import sys
from os import path

from django.core.urlresolvers import reverse
from django.conf import settings

from concept.views import home, about,
     latest_feed, latest_feed_clone, tag_feed

from concept.tagging.models import TaggedItem, Tag
from concept.tagging.views import tag_list, tag_detail

from concept.blog.views import *
from concept.blog.models import Entry

from concept.comments.models import Comment
from concept.projects.models import Project

from concept.projects.views import *
from concept.webalchemy.core import Site

from concept.webalchemy.utils import content_object

class TagHandler(object):
    """Produce list of urls affected by a changed Tag,
    TaggedItem or  object with tags."""

        
    def get_affected_paths(self, obj):
        """Return list of paths affected by this object.
        It handles only pages depends on tag directly, content objects
        details and lists pages are handled when content_object saved.
        It also possible to rebuild here content object related pages,
        but this is not need for concept.
        """
        tags = []
        if type(obj) == TaggedItem:
            tags = [obj.tag.name]
        elif type(obj) == Tag:
            tags = [obj.name]
        else:                           # an tagged object

            tags = [tag.name for tag in Tag.objects.get_for_object(obj)]
        paths = [reverse(tag_list, None, (), {})]
        for tag_name in tags:
            paths.append(reverse(tag_detail, None, (),
                                 {'object_id': tag_name}))
            paths.append(reverse(tag_feed, None, (),
                                 {'url': 'tag/%s' % tag_name}))
        return paths

class EntryArchiveHandler(object):
    """Archive urls affected by the changed Entry."""

    def get_affected_paths(self, obj):
        paths = [reverse(entry_archive_year, None,
                         (str(obj.created.year),))]
        paths.append(reverse(entry_archive_month, None,
                        (str(obj.created.year),str(obj.created.month),)))
        return paths

    
site = Site(settings.WA_PUBLISHER)
site.bind(home, [Entry, Project])

site.bind_static(about)
site.bind([entry_detail, entry_queue], Entry, None, slug="slug")

site.bind([entry_detail, entry_queue], Comment,
          content_object(Entry), slug="slug")

site.bind([entry_list, latest_feed, latest_feed_clone], Entry)

site.bind(project_detail, Project, None, slug="slug")

site.bind(project_list, Project)

site.bind_custom([tag_feed, tag_detail, tag_list], [TaggedItem, Tag],
                 TagHandler())

# even if tags itself aren't changed we need to rebuild tag views because
# it may use changed parts of content objests (name, summary, etc.)
site.bind_custom([tag_feed, tag_detail, tag_list], [Entry], TagHandler())

site.bind_custom([entry_archive_year, entry_archive_month], [Entry],
                 EntryArchiveHandler())

To test WebAlchemy with your project you need to download webalchemy.tar.gz with following content:

webalchemy/core.py
webalchemy/htaccess.py
webalchemy/__init__.py
webalchemy/models.py
webalchemy/publishers.py
webalchemy/rewriters.py
webalchemy/utils.py
LICENSE.txt

$ ./manage.py shell
>>> import concept.webalchemy

WEBALCHEMY_SETTINGS = "concept.webalchemy_settings"

HTDOCS_ROOT = '/root/dogada/mysoftparade.com/'
from webalchemy.rewriters import FastCGIRewriter
from webalchemy.publishers import ApacheHtaccessPublisher

WA_PUBLISHER = ApacheHtaccessPublisher(FastCGIRewriter,
               HTDOCS_ROOT, '', delete_files = True)

WEBALCHEMY_SETTINGS = "concept.webalchemy_settings"
HTDOCS_ROOT = '/var/www/concept/public/'
from webalchemy.rewriters import ModPythonRewriter

from webalchemy.publishers import ApacheHtaccessPublisher
WA_PUBLISHER = ApacheHtaccessPublisher(ModPythonRewriter,
               HTDOCS_ROOT, '/cache', delete_files = True)

<Location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE concept.settings
    SetEnv PYTHON_EGG_CACHE /var/www/concept/.cache
</Location>

<Location "/cache">
    SetHandler None
</Location>

#webalchemy_begin
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /blog/webalchemy-django-apache/
RewriteRule ^$ /cache/blog/webalchemy-django-apache/index.html [L]
</IfModule>
#webalchemy_end

Also you need to add concept.webalchemy.core.WebAlchemyMiddleware to the end of settings.MIDDLEWARE_CLASSES and create file webalchemy_settings.py in the same directory with settings.py. My webalchemy_settings.py you already saw. For your site you can start with very simple webalchemy_settings that will map some models to the homepage view. It may look like:

from django.conf import settings
from concept.webalchemy.core import Site
from myproject.views import homepage
from myproject.blog.models import Entry

site = Site(settings.WA_PUBLISHER)
site.bind(homepage, Entry)

HTTP/1.1 200 OK
Date: Sun, 18 Nov 2007 17:33:26 GMT
Server: Apache/2.2.3 (Ubuntu) DAV/2 SVN/1.4.3 mod_python/3.2.10 Python/2.5.1
Last-Modified: Sun, 18 Nov 2007 16:58:22 GMT
Accept-Ranges: bytes
Content-Length: 17810
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

HTTP/1.1 200 OK
Date: Sun, 18 Nov 2007 17:36:22 GMT
Server: Apache/2.2.3 (Ubuntu) DAV/2 SVN/1.4.3 mod_python/3.2.10 Python/2.5.1
vary: Cookie
webalchemy: saved
Connection: close
Content-Type: text/html; charset=utf-8

from myproject.utils import MyRewriter
from webalchemy.publishers import ApacheHtaccessPublisher

WA_PUBLISHER = ApacheHtaccessPublisher(MyRewiter, HTDOCS_ROOT)

For example it's possible not to deal with file extensions at all (now extensions are used to provide original mime-type of url even after it becomes static file) and use [Type] directive of .htaccess file. Or use send-as-is Apache handler for sending content with custom headers. Also it's interesting to implement rewriter for RESTful web-site when only GET-requests will be redirected to static file, but POST, PUT, DELETE request always will be redirected to Django, even if GET-requests are served directly by Apache.

If you are so big that need to use several app servers, you will need a shared between all app servers htdocs filesystem to which all app servers will write. However for reading each app server can use local htdocs filesystem replicated from shared htdocs filesystem. From the other point of view, do you really need many app servers and deal with replication/sharding if with WebAlchemy you can decrease a lot the database load and accelerate "reader" pages in 100 times?

Behind the scene I'm leaving for now other interesting issues like setting correct content-type for urls, merging manual and autogenerated content of .htaccess files, creating non-conflict rewriting rules, etc.

Enjoy the code and have fun!