“Caching problem” no more: SVN Assets

Useless requests
There is a fundamental problem with static assets on the web pages, like images, CSS files and most importantly JavaScript files – they are requested from the server over and over again even if they were not modified between page requests. This slows down page rendering and in case of JavaScripts quite dramatically as JavaScript downloads block downloads for the rest of the assets by default.

Infinite Expiration
Setting infinite Expires headers is a great solution to the problem – this way everything downloaded from the server will be just kept in a browser’s cache! It is very effective and ranked 3rd on best practices lists provided by Yahoo! Exceptional performance team and even first on Google PageSpeed’s list.

You can clearly see the difference on this graph below:
Waterfall diagram
Scripts and CSS in the headers load from cache eliminating rendering delay and images load from cache as making document load event happening much sooner.

Caching problem
But there is a problem with this on the opposite side – when assets change during normal development, e.g. image gets updated or JavaScript or CSS modified to fix a bug or add a new feature, it must be pushed to user’s browser, but it is not requesting it anymore loading the ones it saved to cache. Developers are used to press Shift+Reload or Shift+F5 to force browser to refresh the page, but users don’t do that and that’s why many just don’t use infinite caching technique, they prefer peaceful development without “caching problem” paying the price with degraded user experience.

URL fingerprinting
To solve this problem, simple technique, sometimes called “fingerprinting” or “cache busting” (term usually indicates that techniques used for wrong reasons) can be used to replace Entity Tags and Conditional GET techniques both of which require server requests and will not work with infinite expiration.

The idea is simple — URL of the asset should be unique for each version of the asset, effectively changing every time you update you script or an image. This way browser will not find the asset in it’s cache and request new one from the server storing results in another cache entry and eventually pushing old one out.

Perceived as complex
The problem implementing this lies within current status quo with web servers that use file system having no idea about file versions. And to be fair, URL<->file one-to-one paradigm is just plain simple to understand helping web grow as fast as it did, but it affects knowledge for all developers and almost no systems are built with versioning in mind.

My experience with many teams shows that this technique is perceived as overly complex and people avoid it till the last moment while this quite obvious solution should be common functionality within the web servers.

Multi-layer problem
Another problem, I believe, is that this solution affects many “layers of the pie” – from asset publishing process and HTML modification by designers and front-end developers to software modifications done by backend developers to web server configuration done by system administrators. This means many groups are involved and more people involved, the harder it gets to push through and nobody wants to bring it up.

SVN Assets logoSolution: SVN Assets
Building HowDoable mostly alone, I don’t have “multi-layer” problem, more over, being a performance geek, I’m concerned with performance probably more then needed at this stage of development. So I spent some time early in the project to make sure my builds and upgrades don’t suffer from “caching problem” and still perform as good as they can with cached content.

What I did was simple – I used a basic and most obvious source of versioning info there can be – source control software, namely Subversion and made sure that my code does not insert a single asset URL into HTML without appending it’s Subversion version to it.

Since many project use Subversion and PHP for their content management, I thought I’d share it with the community and being a strong believer in open source I try to release as much infrastructure free as possible.

So, welcome SVN Assets, a set of tools to make your assets cache using Subversion and simple build process.

I’ve built it for PHP, but will be happy to see it used by developers on other platforms.

Usage is described in README file – just generate data file from Subversion and use assetURL('images/my.jpg')to insert your assets in the code. There is also a command line script that updates all CSS files to point to the proper versions of the files so you don’t have to maintain them by hand and still serve them directly from the file system.

Please, go check it out and let me know what you think about it!

You can also subscribe to a mailing list if you’d like to ask questions and discuss possible improvements:
http://groups.google.com/group/svn-assets

This entry was posted in SVN Assets, Web Performance. Bookmark the permalink.

5 Responses to “Caching problem” no more: SVN Assets

  1. Great suggestion. When I load this blog post I see assets with the version number appended in the querystring, eg:
    openid.css?ver=519
    jquery.js?ver=1.3.2
    comment-reply.js?ver=20090102

    Many proxies ignore the querystring when it comes to caching, so they will serve an outdated file when you change the version in the querystring. The fix for this is to embed the version in the filename, like this
    openid.519.css
    jquery.1.3.2.js
    comment-reply.20090102.js

    Thanks for all the work you do to promote web performance.

  2. Yeah, I actually read your post about that and SVN Assets is using the filename.123.ext schema for files.

    This blog is WordPress with some theme which have many ways to improve ;)

    More over, they have that “multi-layer” challenge I mentioned in the post where one of their goals is easy setup and ?ver=123 is easier to install everywhere while filename.123.ext requires additional “layer” of rewrite rules to be involved.

    I was planning to have some Open Source Speedup sessions as part of NY Web Performance Meetup aimed at improving tools like WordPress or Drupal or writing tools like SVN Assets, but didn’t have time to organize one yet.

  3. saul devitt says:

    Sounds great, I know you have been pushing this idea for a while. When I get the chance, I will give it a try with one of my wordpress mu clients.

  4. Specifying file version on directory level is even better than query string or file name.
    This is especially useful for CSS images.

    Let’s say we have a CSS file /static123/style.css with
    body {
    background: url(bg.jpg);
    }

    Path to image is /static123/bg.jpg

    Now if we do file versioning on directory level bg.jpg file will be reloaded.
    When we update CSS file new path to css is /static124/style.css and path to image is /static124/bg.jpg

    For images we don’t want to cachebust we can use absolute paths:
    body {
    background: url(/images/bg.jpg);
    }

    We can go even wurther with data: URIs.

  5. Actually, SVN Assets now uses filename rewriting and has drop-in .htaccess file that helps with that.

    Just uploading to a new folder is not a very good idea as not all files get changed at the same time and invalidating all files will re-load too many items. This is why version is attached directly to SVN version of specific file instead.

    data: URIs is another powerful tool and there are many other techniques that can be used, it’s just that this particular issue mapped directly to SVN and that’s why I created it as a separate library.

Leave a Reply

Your email address will not be published. Required fields are marked *