I'm experimenting with Tumblr and used it for my latest entry: Deploying Django Sites using RPM

Mark Nottingham recently released redbot, a modern replacement for the classic cacheability tester. I've been using it at work to audit website performance before releases since proper HTTP caching makes an enormous difference in perceived site performance.

redbot is a focused tool and provides a great deal of detail about at most one page and, optionally, its resources. I wanted to expand the scope to testing an entire site and performing content validation and with a little work came up with red_spider.py, which produces a consolidated report like this.

I have a few ideas for the future, which should involve splitting the code into a separate project rather than a fork of redbot as it acquires more validation capabilities such as borrowing from something like collective.validator.css to validate CSS, RSS/Atom, etc., using PIL to verify that images don't have things like wasteful embedded thumbnails, and borrowing from my wk-bench experiment to load pages using WebKit and report JavaScript errors.

I've extended the earlier VCS-friendly shell prompt to add support for Mercurial and you can now get my current .bash_profile from GitHub:

__has_parent_dir () {
    # Utility function so we can test for things like .git/.hg without firing
    # up a separate process
    test -d "$1" && return 0;

    current="."
    while [ ! "$current" -ef "$current/.." ]; do 
        if [ -d "$current/$1" ]; then
            return 0;
        fi
        current="$current/..";
    done

    return 1;
}

__vcs_name() {
  if [ -d .svn ]; then 
    echo " [svn]"; 
  elif [ -d RCS ];  then 
    echo " [RCS]";  
    elif __has_parent_dir ".git"; then
        echo "$(__git_ps1 ' [git %s]')";
    elif __has_parent_dir ".hg"; then
        echo " [hg $(hg branch)]"
  fi
}

PS1='\[\033]0;\u@\h:\w\007\]\u@\h:\w$(__vcs_name) $ '

Recently the topic of enhancing web pages came up at work. It's a lot easier than it used to be thanks to two trends: the rise of modern JavaScript libraries and public CDNs hosting those libraries. This makes a lot easier to enhance content which you can't easily alter (e.g. the forms used by various big companies with marginal web competency) or in situations where you're worried about compatibility with existing code (some squirrelly vertical apps in our case).

Updated 2009-04-03: Moved the template and example scripts to Gist for ease of copying/maintenance: bookmarklet-template.js, enable-autocomplete.js and resize-textareas.js

Updated 2008-10-14: there's a very similar jQuery-lovefest on Sam Ruby's weblog with plenty of useful tips.

To illustrate just how little code this can require, here's an example which uses jQuery to install a function which sanitizes input (we have a legacy app chokes on smart-quotes and people paste text in from Word), copies the submit buttons from the bottom of the form to the top and adds a graphical datepicker for every date field on the page:

jQuery(":text,textarea").bind("change", sanitizer);
jQuery("form").bind("submit",
    function() {
        jQuery(":text,textarea").each(sanitizer);
    }
);

var submit_buttons = jQuery('input[type="submit"]');
submit_buttons.parent().clone(true).prependTo(
    submit_buttons.parents().filter('form')
);

jQuery('input[id*="DATE"]').datepicker();

That's the complete, ready-to-go, “even works with crotchety old Internet Explorer” guts of the code (the take-home lesson is that jQuery is awesome for busy developers). The downside is that this requires a little but of work: you need to have jQuery (and possibly dependencies like the UI plugin I used above) available and you need to jump through some hoops to load jQuery into an existing page efficiently and without conflicts.

Didn't we used to pay for hosting?

One drawback to all of this is that you need somewhere to host your external libraries since you can't fit the core jQuery into a URL, much less UI components or the less svelte libraries. This meant setting up a server, getting an SSL certificate if you need to work on HTTPS sites, etc. Not that much work but it's now a lot easier and quite noticeably faster because Google makes it trivial to get the popular AJAX libraries from their CDN.

Developing with Bookmarklets

The deployment scenario for the major projects where I've used these techniques is a situation where you have some limited access to the page source: perhaps inserting a single script tag into a template or using something like MonkeyGrease or an Apache proxy with mod_substitute to rewrite the generated HTML as it passes through. This is great for making minimal changes but a bit cumbersome to develop and test with, particularly if you need to work on a production site or your instructions begin something like “Go change your browser's proxy settings…”

If I was only working in Firefox I could use GreaseMonkey but I need to test in Safari and Internet Explorer, too. The portable solution is a simple bookmarklet. I use a simple template (bookmarklet-template.js) which loads jQuery from the Google CDN and, after everything is ready to go, runs either a simple function or the external script of my choosing. This makes it easy to prepare an injector bookmarklet which can be used to pull my code into the current page, after which I can run and debug it using Firebug.

Useful Examples

This is also a useful technique for fixing other people's pages. Here are two bookmarklets and the commented source for tools which I use often:

1. Enable autocomplete - changes autocomplete="off" to on throughout the page (Source: enable-autocomplete.js)
2. Resizable Textareas - makes all textareas resizable (alá Safari 3) using jQuery UI Resizable (Source: resize-textareas.js)

I keep both of these in my Firefox & IE bookmark toolbar since they come in handy throughout the day and I've created more any time I find myself regularly needing to deal with a cranky legacy site. The process is simple: copy bookmarklet-template.js, add the code which does whatever fixups the target page needs, run the entire thing through JSLint and, finally paste it into Ted Mielczarek's very handy Bookmarklet Crunchinator.

Good Code Injection Practices

Use Anonymous functions

What's the difference between this bit of code and the first example above?

(function(){
    jQuery(":text,textarea").bind("change", sanitizer);
    jQuery("form").bind("submit",
        function() {
            jQuery(":text,textarea").each(sanitizer);
        }
    );

    var submit_buttons = jQuery('input[type="submit"]');
    submit_buttons.parent().clone(true).prependTo(
        submit_buttons.parents().filter('form')
    );

    jQuery('input[id*="DATE"]').datepicker();
})();

It looks almost identical but there's a key difference: this code is inside an anonymous function and that means that all of my variables are local to the function itself, which means that they won't be visible to other JavaScript on the page and I don't have to worry about conflicting variable or function names. Note that this is only true for variables declared using "var" - if you leave that out or do something like window.foo you can still touch the rest of the page if you need to - for example, replacing the broken validation logic on Comcast's forms.

Reliably detecting when external code has loaded

When jQuery has loaded, it's easy to say "Load this .js file and run this function when it's ready" - here's how the text-area resizer works:

jQuery.getScript(document.location.protocol + "//ajax.googleapis.com/ajax/libs/jqueryui/1.5.2/jquery-ui.js",
    function() {
        jQuery("textarea").resizable();
    }
); 

Loading jQuery itself requires you to do this the hard way: generate a script tag on the fly, insert it into the document and listen for the load events to tell when it's safe to run code which depends on the library you're loading. This is easy for Safari, Firefox, etc. which support the standard W3C DOM addEventListener: simply run your code after the script tag fires a "load" event. Unfortunately, it's not that simple for Internet Explorer: in theory attachEvent("onload") would be equivalent but unfortunately load events are quite unreliable for script tags with IE and so we need to use an onReadyStateChange handler as seen below and check for either of two events which may be fired:

var s = document.createElement('script');
s.type = "text/javascript";
s.setAttribute('src', document.location.protocol + '//ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js');

if (s.addEventListener) {
    s.addEventListener("load", loader, false);
} else if ("onreadystatechange" in s) {
    s.onreadystatechange = function() {
        if (this.readyState == 'complete' || this.readyState == 'loaded') {
            loader();
        }
    };
} else {
    // Chances are if your browser is this old jQuery won't even work but just in case:
    window.setTimeout(loader(), 2500);
}

document.getElementsByTagName('head')[0].appendChild(s);

It's conceivable that a buggy browser could fire the same event twice in an unusual scenario and if you have any sort of user-driven or timer-based code, you'll want to prevent your payload from being run multiple times using a guard like this which allows the function to check whether it has executed before without using the more common approach of relying on a global variable. Besides cleanliness, this also makes it easy if you might inject multiple things onto a page and don't want to have to rely only on a global variable naming convention to prevent chaos:

// Avoid executing this function twice:
if (arguments.callee._executed)  return;
arguments.callee._executed = true;

Avoid HTTP/HTTPS conflicts

If you're injecting code into pages which may or may not use SSL, you have a problem: if you hard-code a URL in your code and the protocol doesn't match you'll either incur the extra overhead of starting an SSL session (which isn't a major problem) by using https even when you don't need to or encounter Internet Explorer's popular mixed-mode security warning. This is easy to avoid by using the current page's protocol for your scripts as long as you're using a server which can handle either protocol (Google's CDN does; Yahoo's does not):

document.location.protocol + '//path.to.example.com/something.js'

Just a quick note to boost Google rank: if you use the mod_auth_cas Apache module for CAS single-signon and have noticed sporadic Apache segfaults (typically a blank page which reloads correctly), mod_auth_cas 1.0.9 includes my patch which traps crashes caused by corrupted XML ticket files. Please let me know if this fixes your problem as we're still trying to isolate the underlying failure.

I've started adding some more advanced ctypes wrappers for the OS X Keychain to PyMacAdmin. There's still a bunch of work to do but the upshot is that you can write code like this and expect it to work:

try:
    keychain = Keychain(options.keychain)
    item = keychain.find_generic_password(
        service_name=options.service,
        account_name=options.account
    )

    print "Removing %s" % item
    keychain.remove(item)
except KeyError, exc:
    print >>sys.stderr, exc.message
except RuntimeError, exc:
    print >>sys.stderr, "Unable to delete keychain item: %s" % exc

and get output like this:

chris@Enceladus:~/Development/pymacadmin [git master] $ ./bin/keychain-delete.py -a "acdha"
Removing GenericPassword(service_name='', account_name='acdha', label='Audioscrobbler: acdha')

There's a bunch of stuff going on behind the scenes now to make things easier than the sadly-unimproved state discussed at length by Wil Shipley back in 2006:

• PyMacAdmin.Security now defines many of the kSec* defines from SecKeychainItem.h, simplifying many calls which either required magic numbers or complicated struct.unpack() calls.
• PyMacAdmin.Security.Keychain now defines a few classes which represent some of the native typedefs: SecKeychainAttribute, SecKeychainAttributeList and SecKeychainAttributeInfo.
• Core PyMacAdmin improvements to make it a little easier to work with Carbon-style APIs:
  • mac_strerror() provides a way to lookup the error message for Carbon return codes
  • carbon_call() simplifies the process of calling a Carbon function by automatically checking its return code and throwing an exception any time the return code is negative.
  • carbon_errcheck() can be used as a ctypes errcheck function using the same logic as carbon_call()
  • load_carbon_framework() wraps the ctypes LoadLibrary() method to automatically use carbon_errcheck() for every function in the loaded library - this won't work for functions which use negative return codes for non-error results but the Carbon APIs are pretty consistent in that regard.
• All of the above combines to allow PyMacAdmin.Security.Keychain's find_generic_password() to call SecKeychainItemCopyAttributesAndData behind the scenes to collect the item's label and will allow arbitrary other attributes in the future.

If you have any interest in wrapping native Mac APIs with Python please join the discussion over on the PyMacAdmin group - any sort of Python-OS X integration discussion is welcome.

Since this post was originally written, I've been working on the PyMacAdmin project with Nigel Kersten. The information below is still correct but the kicker-replacement script has gained the ability to handle filesystem events and workspace notifications and been renamed to crankd.

In a perfect world software would gracefully network transitions. Unfortunately my users have encountered a fair number of things which don't always handle things like a laptop moving from ethernet to WiFi, a DHCP server taking awhile to respond, etc. While many programs have at least reached the point of eventually timing out and retrying it would be nice to automatically restart something as soon as the system network configuration changes. This is unfortunately system-specific and frequently required some hackish approach involving tail -f or equivalent to watch a log file, which is slow and tends to break on upgrades.

OS X has a nice way to query the current system configuration and receive event notifications when things change: the SystemConfiguration Framework (Technical Note TN1145: Living in a Dynamic TCP/IP Environment is also of interest). You can explore this using the scutil command-line tool - in the example below, I've looked at the list of available events and chosen to watch for power-state changes, receiving a notice when I unplugged the power cable from my laptop:

chris@Enceladus:~ $ scutil
> list
  subKey [0] = Plugin:IPConfiguration
  subKey [1] = Plugin:InterfaceNamer
  subKey [2] = Setup:
  subKey [3] = Setup:/
  subKey [4] = Setup:/Network/Global/IPv4
  subKey [5] = Setup:/Network/HostNames
…
  subKey [21] = State:/IOKit/PowerManagement/CurrentSettings
  subKey [22] = State:/IOKit/PowerSources/InternalBattery-0
…
> n.add State:/IOKit/PowerSources/InternalBattery-0
> n.watch
> notification callback (store address = 0x1036c0).
  changed key [0] = State:/IOKit/PowerSources/InternalBattery-0
notification callback (store address = 0x1036c0).
  changed key [0] = State:/IOKit/PowerSources/InternalBattery-0

This is pretty cool stuff but I'd like to do something smarter than scripting a copy of scutil. I could write an Objective-C application but OS X 10.5 included the very handy PyObjC 2.0 which allows access to most of the native APIs directly from within Python. James Reynolds posted a message to the MacEnterprise mailing list which prompted me to stop procrastinating and actually write some code. A little poking around later and I have a Python script which is ready for me to add whatever custom actions I want to take when the network state changes - the version below is abbreviated so you'll want to download the full watch-network-config.py for your own use:

from Cocoa import *
from SystemConfiguration import *

def handleNetworkConfigChange(store, changedKeys, info):
	print "Global network configuration changed: ", changedKeys
	# Kick a change-intolerant service in the head here

store = SCDynamicStoreCreate(None, "global-network-watcher", handleNetworkConfigChange, None)

SCDynamicStoreSetNotificationKeys(store, None, [ 'State:/Network/Global/IPv4', 'State:/Network/Global/IPv6' ])

CFRunLoopAddSource(CFRunLoopGetCurrent(), SCDynamicStoreCreateRunLoopSource(None, store, 0), kCFRunLoopCommonModes)
CFRunLoopRun()

Geoff Franks took the time to have the event handler use a dictionary so you can listen for multiple events and run a specific command for each one; I added a little syslog support and am releasing this version as a replacement for the widely-used Kicker which was removed in 10.5: Download kicker-replacement

Fun lessons from the trenches: in versions of OS X prior to 10.5 there were several nasty bugs due to lookupd and DirectoryService not having real timeouts: we have some rigs which use DHCP on our public network and static IPs on a private experiment network. When the system booted the private interface didn't need to wait for a DHCP lease and thus came up slightly faster than the public interface — this should have been harmless except that DirectoryService immediately attempted to connect to our LDAP server which isn't reachable on the private network and network timeout values aren't actually used in any version prior to 10.5, causing network accounts and NFS mounts to be unavailable until someone manually killed DirectoryService!

One of the fixes for the LDAP problems I wrote about last year was just integrated into pam_ldap. This would have taken a very brief amount of time except that an updated patch vanished at some point between my mail server and the upstream bugzilla — ah, software!