Drupal Updates and Upgrades - What, Why, and How

The Drupal community is perpetually working on the next great major release of Drupal. At the time of this writing, Drupal 7 is nearing its first beta release, meaning that the number of major bugs identified by testers and developers is reaching a managable level, and that there are few if any "show stoppers" which would keep people from using the wealth of new functionality and the increased power and efficiency that it offers. Once the Drupal developer community is satisfied, the Drupal 7.x series will be officially released, and will become the new stable major version.

When a new major version of the Drupal core is released, it is released with a minor version number of 0, so for example, the release of the Drupal 6.x series started on 14 February 2008, with the release of Drupal 6.0. The Drupal 5.x series started when Drupal 5.0 came out on 16 January 2007.

Because of the substantial work involved, and the increasing difficulty of working with older and older web technologies, the Drupal community limits full security vulnerability and performance bug fixes support for two Drupal series at any given time - the current stable series and the previous "older" series. That means each time a new major version of Drupal is officially released, the previous "older" supported version drops into unsupported status. That means, for example, that Drupal 6.x series is the current stable major version and the Drupal 5.x series is the older supported version; when Drupal 7.0 is released Drupal 6.x will be the older supported version and Drupal 5.0 will no longer be officially supported by the Drupal community for security or bug fixes.

Historically, the period between major Drupal upgrades has has been between a year and two years. There are many good business reasons to keep your Drupal site tracking the current stable release, even though doing so can sometimes require a considerable investment. The improvements in both the underlying Drupal core and support for developers creating modules and themes targeting that core version mean that end users - namely Egressive's customers who upgrade existing sites, or commission new Drupal sites - get a broad range of substantially improved capabilities, plus the ability to benefit from new cutting-edge modules provided by other developers within the community, at very little additional cost.

It is important to note that it is neither important or recommended for you to upgrade a Drupal site to the new stable Drupal version when a new one is first released. There is normally a cost associated with such an upgrade, and the more customised and involved your site is, the more it is likely to cost. In addition, if your site depends on any community modules, it is likely to take a while following the initial Drupal core release for module developers to upgrade a sufficient number of the modules on which your site depends to make it worth upgrading. With the release of Drupal 6.x, for instance, Egressive did not start upgrading sites until at least 9 months after the release due to delays in releasing upgrades to critical modules like CCK and Views.

The Drupal Minor Version Update Process

From time to time, the Drupal community releases minor updates to either the stable Drupal release (currently the 6.x series) or the one prior (currently the 5.x series). These are denoted by the fractional suffix of the version number, known as the minor version number; so Drupal 5.21 would be the 21st update to the 5.x series, and Drupal 6.1 would be the first update to the 6.x series.

In general, minor version updates are provided either to address functional bugs or identified security vulnerabilities. In most cases they are simply changes to the Drupal source code (held in files) which Drupal uses to operate on your site data, stored in the database. As such, minor version updates are generally relatively straightforward. In most cases they can be performed while the site is live, and won't have any impact on your users. In some cases, however, we will put the site into maintenance mode while applying a minor version update, as the update process could result in the possibility of users seeing an error message, though not related to their activities, which might spook them.

Along with the rest of the Drupal community, Egressive has invested substantially in building tools to smooth these updates, and we provide pro-active minor Drupal core upgrades as part of all our new Drupal hosting packages.

For a history of Drupal minor releases, the Drupal community provides these:

the Drupal 5.x release history
the Drupal 6.x release history

The Drupal Major Version Upgrade Process

Between major versions of the Drupal core, however, not only do all the source code files representing the Drupal core change, but there's also a good chance that much of the database structure will have been refactored to reflect the wisdom of the Drupal core developers. The purpose of major upgrades is to engineer around any fundamental performance, functionality or scalability limitations in the previous major version. New major versions only see the light of day if the Drupal core developers determine that it offers better resources for the developer community (of which Egressive is proud to be a part) as well as better performance, and provides elegant means for consolidating ("refactoring" in developer-speak) functionality that was previously provided by more convoluted, fiddly means.

The Drupal community supports upgrades based on a module hierarchy:

Drupal Core Modules - both the required and optional modules - The community guarantees a well defined scripted upgrade process for data related to these modules from the previous major version. Here is a list of Drupal Core Modules for Drupal 5 and Drupal 6 for your reference.
Community Modules - also known as Third Party Modules - modules made available to the community by their developers - these are typically available through the Drupal community site: drupal.org. For modules which have equivalents for the current and previous major Drupal versions, the community encourages developers to provide a scripted upgrade process, or at least documentation explaining a manual upgrade process. Luckily, thanks to the benificence of the Drupal community and the open source model used by Drupal, nearly all widely used modules have scripted upgrade processes, although some are more convoluted and reliable than others. Some modules only exist only for a single major version of Drupal in which case no upgrade path is necessary or possible. In some cases complementary modules in the next Drupal version will exist, and these may provide an upgrade path for data created by one or more module from the previous major version.
Custom Modules - modules developed for a specific site - the upgrade process is completely dependent on the developer of the module, and the organisation for whom the module was developed has full responsibility for (and is likely to bear the cost of) developing an upgrade process. In some cases, however, functionality requiring a customised module in one Drupal version might be achieved using either Core or Community modules in the next major version. For example, Egressive developed many customised modules for Drupal 5, whose functionality can now be replicated (and improved upon in many cases) with the Drupal 6 versions of the CCK and Views modules.

How Egressive does Drupal major version upgrades

The process which Egressive uses to upgrade a Drupal site, in this case a Drupal 5.x site to Drupal 6.x, is described below. It involves a trial upgrade which occurs on Egressive's development infrastructure, and allows us to identify and resolve any problems with the upgrade process before we do it for real in the final upgrade.

Create a "snapshot" of the current live Drupal 5 site on Egressive's development infrastructure for the trial upgrade. The customer's live site functions unchanged throughout the trial upgrade.
On the dev snapshot (not visible to your site users), disable any custom themes and custom or community modules, leaving the default theme and only core modules enabled. Backup the dev database
Create a new default Durpal 6 site on the development infrastructure and import the Drupal 5 database backup.
Run a core upgrade on the database. This upgrades all core data in the database, but not data that might be associated with community or custom modules or themes.
Enable critical modules, particularly CCK and Views, one at a time, and run the update scripts for each in sequence, identifying any errors.
Identify any modules for whom no upgrade path exists, or which experienced errors in the process.
Apply all Egressive "base" functionality tweaks so that the upgraded site reflects Egressive's state-of-the-art for usability and functionality for that Drupal core series.
Develop an upgraded version of the previous theme or a new "refreshed" theme for the site depending on the customer's preference.
Fix all of the errors by debugging the upgrade process, identify upgrade paths for all modules possible and either identify alternatives or negotiate with the customer to either drop functionality that is no longer supported or develop customised functionality. Document all upgrade procedures applied. Where-ever possible, ensure that the upgraded functionality can be exported from the development snapshot (for import as part of the subsequent "final upgrade").
Present the upgraded snapshot - now Drupal 6 and providing equivalent functionality to the live Drupal 5 site except where agreed - to the customer for approval. Note: achieving the desired look and feel for the upgraded site will sometimes depend on the addition of specific content, which is held in the database. Adding this content during the trial upgrade will necessitate that it be re-added for the final upgrade. Depending on how much content is involved, this might be more costly than desired. In some cases Egressive will advise the customer to treat this content, and the look and feel elements depending on it, as post-upgrade enhancements to be applied on the live site as a separate step (and possibly a separate project) following the major version upgrade.
Once accepted, Egressive schedules a time with the customer when the live site can be put into maintenance mode for a several hours while the final upgrade is undertaken. Egressive also assists the customer to arrange for any Domain Name changes that the upgrade might require, e.g. if moving the site to Egressive's new hosting infrastructure.
Finally, Egressive initiates the final upgrade, putting the live site into maintenance mode at the time agreed. Egressive then runs through the documented upgrade process which could take as little as an hour or as many as 12 hours depending on the complexity of the upgrade process determined during the trial upgrade. The time frame is compressed considerably this time around because all of the upgrade problems encountered should already have been solved during the trial upgrade, but there might still be some manual steps in the process which take time. When the customer's site is brought out of maintenance mode it will have metamorphed into a beautiful new Drupal 6 site.
Following the completed upgrade, by agreement with the customer, Egressive might immediately embark on content enhancements (sometimes in conjunction with the customer) to bring the upgraded site into alignment with the new theme concepts.

Update

For recent Drupal 5 to 6 upgrades, we have begun using the excellent Features module (and some of the many other great import/export modules provided by Drupal community developers and companies like Development Seed and Lullabot) to help minimise the risk of the final upgrade. We've also begun labouriously scripting the upgrade process from start to finish to ensure the whole process it is repeatable. Without this repeatability, it's almost guaranteed that any Drupal site of typical complexity would experience some functional deficiencies after the final upgrade. We would strongly prefer to minimise that possibility.