Where Are We Using That Module Again?

Like many people, I'm often looking for ways to "make our lives easier." At CommonPlaces, we have many different Drupal projects and contributed modules in use in those projects. When a new advisory is released for a module -- often an important security advisory requiring immediate attention -- we would have to look through all the projects to find out which ones are using the module and what version of the module is in use, so we could determine if that version needs to be upgraded.

In response, we have implemented the following pair of scripts that automate the collection of this information for us. It greatly reduces the time we need to spend tracking all of this down.

As more and more Drupal developers are moving their projects into a source control system like Subversion, we thought we would offer these scripts here in the hopes that they would help others with the same problem.

An overview:

• A PHP script runs every six hours via cron on our development server where the Subversion repositories are stored. The script accesses the Drupal RSS feed for contributed module advisories and extracts the ones that have been issued in the last six hours, or since the last time the script was run.
• If a new advisory is found, the PHP script needs to determine what module it was for. This actually isn't in the RSS feed itself -- but we can and do obtain it from the drupal.org node which has the complete advisory.
• A bash script is called with the module name. The bash script, which can also be run by itself to check manually, looks in our repositories for the module and prints its location along with the version of the module from the .info file.
• The PHP script takes the output of the bash script, packages it with the relevant information from the RSS feed and emails it to a set address which in our case is an internal engineering mailing list.

Let's take a look at the code. First up is the PHP script, check-modules.php. Note that it is PHP5.

#!/usr/bin/env php
<?php
/**
 * Examines the Drupal contributed module RSS feed for latest security
 * alerts and sends email to notify of the alert.
 */

// Begin Configuration

// The URL to the RSS feed
$rss_url = "http://drupal.org/security/contrib/rss.xml";
// The number of hours between runs (must match cron for this to be effective)
$interval_hours = 6;
// email to_address, possibly a company mailing list
$to_address = "to-user@DOMAIN.com";
// email headers including from address
$headers = 'From: user@DOMAIN.com';

// End Configuration

// parse the rss feed
$doc = new DOMDocument();
$doc->load($rss_url);
foreach ($doc->getElementsByTagName('item') as $node) {

  $title = $node->getElementsByTagName('title')->item(0)->nodeValue;
  $desc = $node->getElementsByTagName('description')->item(0)->nodeValue;
  $link = $node->getElementsByTagName('link')->item(0)->nodeValue;
  $date = $node->getElementsByTagName('pubDate')->item(0)->nodeValue;
  // i.e.
<pubDate>Wed, 10 Jun 2009 21:07:08 +0000</pubDate>

  // Get the timestamp from the date and compare with current
  $ts = strtotime($date);
  $ts_now = strtotime("-".$interval_hours." hours");

  if ($ts + ($interval_hours*60*60) > $ts_now) {
    // This advisory was issued since the last run
    // Process it and send the alert email

    $subject = file_get_contents($link);

    // Extract the url for the project page and get the module name
    // The rss feed does not contain the url to the module project page,
    // so we have to access the vulnerability page and get the url, and
    // then the module name
    // i.e. "http://drupal.org/project/services"

    $num = preg_match_all("/http://drupal.org/project/([a-z_]+)/", $subject, $matches);
    $module = $matches[1][0];

    $output = "";
    $output .= "ATTENTION: The following Drupal contributed module advisory has been issued:nn";
    $output .= "$titlen";
    $output .= "$linkn";
    $output .= strip_tags($desc);
    $output .= "Date: ".date('D, d M Y h:i:s A',$ts)."n";
    //$output .= "Timestamp: $tsn";
    $output .= "Project page: ".$matches[0][0]."n";

    $output .= "nThis module was found in the following repositories. Click the "
      . "link at the top of this message and compare "
      . "the version of the module in the repository with the version affected by the "
      . "vulnerability to determine whether or not an upgrade is necessary. If no "
      . "repositories are listed, the module does not appear in any repositories.n";

    // Empty the output array
    $cmd_output = array();
    exec("/usr/local/bin/repo-module.sh $module", $cmd_output);
    foreach ($cmd_output as $key => $val) {
      $output .= $val . "n";
    }
    $output .= "nn";
    //print($output);

    // send mail
    if (!mail($to_address, $title, $output, $headers)) {
      // Message for testing
      //print("Error sending mailn");
    } else {
      //print("Sent mailn");
    }
  }
}
?>

Next is the bash script called by check-modules.php. This one is called repo-module.sh.

#!/bin/bash

# Takes a Drupal module name and determines what repositories it exists in,
# along with the version of the module according to the .info file.
#
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

for repos in $(ls /your-svn-dir)
do
  for dir in $(svn ls "file:///your-svn-dir/$repos/branches" 2> /dev/null)
  do
    if svn ls "file:///your-svn-dir/$repos/branches/$dir/all/modules" 2> /dev/null | egrep -q ^$1/$
    then
      echo ""
      echo "Found in $repos : branches/$dir"
      svn cat "file:///your-svn-dir/$repos/branches/$dir/all/modules/$1/$1.info" | grep version
    fi
  done

  for dir in "trunk/all/modules" "trunk/sites/all/modules"
  do
    if svn ls "file:///your-svn-dir/$repos/$dir" 2> /dev/null | egrep -q ^$1/$
    then
      echo ""
      echo "Found in $repos : $dir"
      svn cat "file:///your-svn-dir/$repos/$dir/$1/$1.info" | grep version
    fi
  done
done

I should explain that at CommonPlaces, we have chosen to store each Drupal project in its own repository, using a two-tiered system. We do this for a variety of reasons, probably too many to go into here. But the important things to note for purposes of this discussion are:

• Only the project sites directory is stored in a repository. Our two-tiered checkout grabs a copy of Drupal core from another repository, not referenced here.
• The structure of those project repositories is the basic Subversion trunk-branches-tags structure. We are interested in what is in branches and trunk.
• If your repository structure is different, or if you use another source control system such as cvs, you will need to modify the bash script accordingly. Also, be sure to replace "your-svn-dir" with the directory on your filesystem where you store your repositories.
• To use these scripts, save the first one as "check-modules.php" and the second one as "repo-module.sh", drop both into your /usr/local/bin, and make them executable. Also create a cron job like the following:

  • # Check drupal for module security advisories every 6 hrs and send mail
    
    0 0,6,12,18 * * * /usr/local/bin/check-modules.php > /dev/null 2>&1

  • If you decide to run the cron job at a different interval than six hours, be sure to edit check-modules.php and make $interval_hours match the cron timeframe.

Here's what a sample email looks like: