A couple ways to monitor a particular URL for updates

Because you never know exactly when a data dump is going to dump.

By Joe Murphy

First published

File this one under sysadmin tricks for journalists.

Tuesday the JFK files — a gaggle of PDFs (that's what a bunch of PDFs are called) — were supposed to land. Supposedly in the afternoon. And maybe they would land at https://www.archives.gov/research/jfk/jfkbulkdownload.

The goal: Get an alert when that URL updates.

The enemy here is refreshing a URL by hand until you see the change you wish to see on the page. And in this little article will show you how to leverage shell scripts to defeat that enemy and make your life easier. Note that this won't help you if you're on a Windows machine, I'm on a Macintosh, this procedure is aimed at that and maybe linux boxes.

There are a few ways to do this, and I'll go through two slightly different ways. In addition to assuming you’re on a Macintosh, it assumes you’re within earshot of the computer.

Method 1: Look for changes in the HTML document

This approach will alert you if any changes in an HTML document are seen. It's a little bit of a sledgehammer and can produce false positives, but this is the essence of it: Save a representation of the contents of the HTML document once, and then every ten seconds re-download the document and see if anything's changed.

How you do it

  1. Download the web page with curl
  2. Hash the output (i.e. algorithmically convert it) with md5, save that as your "base" copy.
  3. Then do that again and again and compare the md5 hashes, and if any subsequent hash you take doesn't match with the base then you know something on the web page has changed.
  4. When you ascertain something's changed, speak out an alert (in this case it’s 'oh boy oh boy oh boy').

The code you use to monitor a web page for changes

This code will do all the steps above, monitoring a web URL for changes and alerting you when something changes. This is good for situations where you're expecting a change in the near future and the alert only works if you're in the same room as your computer, though it could probably be adjusted for other situations.

To use this code, copy it and edit the URL so it pulls from the URL you want to monitor, then paste it into your terminal program and press return.

curl --silent https://www.archives.gov/research/jfk/jfkbulkdownload | md5 > base;

watch -n 10 'curl --silent https://www.archives.gov/research/jfk/jfkbulkdownload | md5 > new; if ! cmp base new > /dev/null; then say 'oh boy oh boy oh boy'; fi'

As they say, there is another way.

Method 2: Look for changes in the HTTP request headers

Every request on the web has two parts: The actual document, and the meta information (i.e. "headers") attached to that document.

For example, this is what the headers on the JFK files page look like:

HTTP/2 200
content-type: text/html; charset=utf-8
content-length: 34220
date: Wed, 19 Mar 2025 22:20:58 GMT
content-language: en
last-modified: Wed, 19 Mar 2025 19:48:23 GMT
x-content-type-options: nosniff
etag: W/"1742413703-0-gzip"
v-ttl: 77245
cache-control: public, max-age=60, s-maxage=86400
v-cache-ttl: 77245
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
accept-ranges: bytes
vary: Cookie,Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 801c4cdd177872a11b03f54a2b3b464e.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO53-C1
x-amz-cf-id: 2d4HH0RiR3c8Y2nd553ocDfimjWP-bJexT_6uYtKHZRgitblsRVeHw==

Most of that is gobbledegook. But there is one piece we care about: That last-modified timestamp. Not every website will reliably use this, but if your target is one that does, here's how to monitor and alert for changes in the last-modified timestamp.

The how-to is pretty much the same as above, and the code is similar but not identical.

curl --silent --head https://www.archives.gov/research/jfk/jfkbulkdownload | grep modified | md5 > base;

watch -n 10 'curl --silent --head https://www.archives.gov/research/jfk/jfkbulkdownload | grep modified | md5 > new; if ! cmp base new > /dev/null; then say 'oh boy oh boy oh boy'; fi'

You can email me at joe.murphy@gmail.com.