I've been looking at automated testing again, after having one of those situations where some test cases might have caught a bug earlier.
For testing how the site works in reality, it is rather code heavy, which is why I got interested in Selenium. Selenium allows one to create test cases by recording what one's doing in the browser and adding in asserts that the server is returning the right responses. Now all we need is a bit of integration.
There is a Drupal module, but it haven't been updated since 4.7 (I wonder why). I did find someone that's attempting a 5.x port, but aside from seeming dead, it's in my opinion going in the entirely wrong direction. It depends on CCK, computed_field, content_copy and views, and that's way too much extra crap to install. It's all good modules that we use a lot, but I don't want to depend on them only for testing purposes. I want a single self contained module.
So, I thought I'd try with the old version, and after fixing some 4.7 -> 5.x issues, it seems to be working fine. The neat thing is that the tests are contained in the site, and can be run in any compatible browser (which means most of them).
So, if anyone else is interested, here it is. I don't know where I'm going with this, I should probably offer it up to the maintainer, but as he haven't updated it already, I doubt he's very interested in maintaining it anymore. If anyone's using it, give me a shout.
Some improvements I could imaggine:
A block for quickly adding tests, perhaps a plugin to Selenium IDE to allow saving the tests directly from the window (if that's possible).
Not using nodes as storage. While it does give some advantages, I really don't see the tests as part of the site content.
Working together with SimpleTest. One button to test everything would ne nice.