Healthcheck V2 — Hardening Script

v1 worked for the happy path and failed everything else — no timeout, no retries, no way to tell a connection failure apart from an unhealthy response, no logging beyond a bare echo, and no protection against spamming an alert on every single run during a real outage. This chapter fixes all of it, evolving the exact same script.

Adding A Timeout

Straight from last chapter’s preview, actually integrated this time:

max_time=5
response=$(curl -s -o /dev/null -w "%{http_code}" --max-time "$max_time" "$url")

A service that never actively refuses the connection, but never responds either, no longer hangs this script indefinitely — curl gives up after max_time seconds and reports failure instead.

Adding Retries

A single failed attempt shouldn’t immediately mean “the service is down” — a brief network blip is common and usually resolves itself within a few seconds:

max_attempts=3
attempt=1
response=""

while (( attempt <= max_attempts )); do
    if response=$(curl -s -o /dev/null -w "%{http_code}" --max-time "$max_time" "$url"); then
        [[ "$response" == "200" ]] && break
    fi
    (( attempt < max_attempts )) && sleep $(( attempt * 2 ))
    (( attempt++ ))
done

This is a close relative of the retry_with_backoff function from the Real-World Patterns chapter, hand-adapted here because this loop needs to log something different depending on which kind of failure occurred at each attempt — covered next.

Distinguishing Connection Failure From An Unhealthy Response

if response=$(curl ...); then is doing more work than it might look like. Note this is a plain assignment, not a local one — and that distinction matters directly: local var=$(cmd) masks the command substitution’s real exit status behind local’s own success. A plain assignment, without local, has no such problem — response=$(curl ...)’s own exit status genuinely is curl’s exit status, which is exactly what makes it usable directly as an if condition here.

A connection failure (DNS failure, refused connection, or the new timeout) makes curl itself exit nonzero, caught by this if; a successful connection returning an unhealthy status code still makes curl exit 0 (it successfully retrieved something), just with response holding something other than 200. These are genuinely different failure modes, and this pattern tells them apart cleanly.

Adding Structured Logging

Reusing the exact leveled logging function from the backup project:

log() {
    local level="$1"
    shift
    local ts
    ts=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$ts] [$level] $*" >&2
}
log_info()  { log "INFO"  "$@"; }
log_warn()  { log "WARN"  "$@"; }
log_error() { log "ERROR" "$@"; }

Rate-Limiting Alerts

A small state file records when the last alert actually fired, letting the script suppress repeat alerts for the same ongoing outage while still alerting immediately on a new one:

alert_state_file="/tmp/healthcheck-last-alert.state"
min_alert_interval=300

should_alert() {
    [[ ! -f "$alert_state_file" ]] && return 0
    local last_alert
    last_alert=$(cat "$alert_state_file")
    local now
    now=$(date +%s)
    (( now - last_alert >= min_alert_interval ))
}

record_alert() {
    date +%s > "$alert_state_file"
}

On a successful check, the state file is removed entirely — so the next real failure, whenever it happens, alerts immediately rather than being suppressed by a stale record of a long-resolved incident.

The Full v2 Script

#!/usr/bin/env bash
set -euo pipefail

readonly EXIT_INVALID_ARGS=2
readonly EXIT_UNHEALTHY=1

url="${1:-}"
max_time=5
max_attempts=3
alert_state_file="/tmp/healthcheck-last-alert.state"
min_alert_interval=300

log() {
    local level="$1"
    shift
    local ts
    ts=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$ts] [$level] $*" >&2
}
log_info()  { log "INFO"  "$@"; }
log_warn()  { log "WARN"  "$@"; }
log_error() { log "ERROR" "$@"; }

if [[ -z "$url" ]]; then
    log_error "Usage: $0 <url>"
    exit "$EXIT_INVALID_ARGS"
fi

should_alert() {
    [[ ! -f "$alert_state_file" ]] && return 0
    local last_alert
    last_alert=$(cat "$alert_state_file")
    local now
    now=$(date +%s)
    (( now - last_alert >= min_alert_interval ))
}
record_alert() {
    date +%s > "$alert_state_file"
}

attempt=1
response=""
while (( attempt <= max_attempts )); do
    if response=$(curl -s -o /dev/null -w "%{http_code}" --max-time "$max_time" "$url"); then
        if [[ "$response" == "200" ]]; then
            log_info "Attempt $attempt: OK ($response)"
            break
        else
            log_warn "Attempt $attempt: unhealthy status $response"
        fi
    else
        log_warn "Attempt $attempt: connection failed"
        response=""
    fi
    (( attempt < max_attempts )) && sleep $(( attempt * 2 ))
    (( attempt++ ))
done

if [[ "$response" == "200" ]]; then
    log_info "Service healthy: $url"
    rm -f "$alert_state_file"
    exit 0
fi

if [[ -z "$response" ]]; then
    log_error "Service unreachable after $max_attempts attempts: $url"
else
    log_error "Service unhealthy after $max_attempts attempts: $url (last status: $response)"
fi

if should_alert; then
    log_error "ALERT: $url is down"
    record_alert
else
    log_warn "Suppressing repeat alert (last alert within ${min_alert_interval}s)"
fi

exit "$EXIT_UNHEALTHY"

What Changed From v1

--max-time on every curl call, closing v1’s most dangerous gap.
Up to three attempts with increasing delay, absorbing brief transient failures instead of alerting on every one.
Connection failure and unhealthy response are now logged distinctly, using a plain (not local) assignment’s genuine exit status.
Leveled, timestamped logging throughout, replacing v1’s single bare echo.
Rate-limited alerting, with automatic recovery-triggered reset via the removed state file.

Best Practices

Always set an explicit timeout on any network call in a script — the default OS-level timeout, if there even is one, is rarely what you actually want.
Use a plain assignment (not local) when a command substitution’s own exit status needs to be checked directly — applied deliberately rather than accidentally.
Reset any rate-limiting state on recovery, not just on the next failure — otherwise a resolved incident’s suppression window can mask the start of a completely unrelated new one.

Shell-Safety Considerations

The rate-limiting logic above has a real gap this script hasn’t addressed at all: nothing prevents two overlapping runs from both deciding to alert at the same time.

The problem, demonstrated directly — two “runs” checking should_alert before either has recorded anything:

alert_state_file="/tmp/shell-course-demo-alert.state"
rm -f "$alert_state_file"

should_alert() {
    [[ ! -f "$alert_state_file" ]] && return 0
    local last_alert
    last_alert=$(cat "$alert_state_file")
    local now
    now=$(date +%s)
    (( now - last_alert >= 300 ))
}
record_alert() { date +%s > "$alert_state_file"; }

if should_alert; then echo "Run A: would alert"; fi
if should_alert; then echo "Run B: would alert"; fi
record_alert

Run A: would alert
Run B: would alert

Both runs saw no state file and both decided independently to alert — exactly what would happen if a scheduled run overlapped with a still-in-progress previous one (increasingly likely now that retries make a single run take several seconds longer than it used to). Whoever’s on the receiving end gets duplicate alerts for the same single incident.

The fix is the same lockfile pattern the backup project built in full — applied here to ensure only one instance of this script’s check-and-alert logic runs at a time:

lock_file="/tmp/shell-course-demo-healthcheck.lock"

if [[ -f "$lock_file" ]]; then
    existing_pid=$(cat "$lock_file")
    if kill -0 "$existing_pid" 2>/dev/null; then
        echo "Already running as PID $existing_pid — skipping this run"
        exit 0
    fi
fi
echo "$$" > "$lock_file"
trap 'rm -f "$lock_file"' EXIT

echo "Proceeding — no overlapping run detected"

Wrapping this around the health-check logic means a second, overlapping invocation exits immediately rather than racing the first one to should_alert.

Hands-On: Building And Testing v2

1. Set up a working directory.

mkdir -p ~/shell-course/healthcheck-project
cd ~/shell-course/healthcheck-project

2. Write v2 and test the happy path.

cat > healthcheck-v2.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
readonly EXIT_INVALID_ARGS=2
readonly EXIT_UNHEALTHY=1
url="${1:-}"
max_time=5
max_attempts=3
alert_state_file="/tmp/shell-course-healthcheck.state"
min_alert_interval=300

log() {
    local level="$1"
    shift
    local ts
    ts=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$ts] [$level] $*" >&2
}
log_info()  { log "INFO"  "$@"; }
log_warn()  { log "WARN"  "$@"; }
log_error() { log "ERROR" "$@"; }

if [[ -z "$url" ]]; then
    log_error "Usage: $0 <url>"
    exit "$EXIT_INVALID_ARGS"
fi

should_alert() {
    [[ ! -f "$alert_state_file" ]] && return 0
    local last_alert
    last_alert=$(cat "$alert_state_file")
    local now
    now=$(date +%s)
    (( now - last_alert >= min_alert_interval ))
}
record_alert() { date +%s > "$alert_state_file"; }

attempt=1
response=""
while (( attempt <= max_attempts )); do
    if response=$(curl -s -o /dev/null -w "%{http_code}" --max-time "$max_time" "$url"); then
        if [[ "$response" == "200" ]]; then
            log_info "Attempt $attempt: OK ($response)"
            break
        else
            log_warn "Attempt $attempt: unhealthy status $response"
        fi
    else
        log_warn "Attempt $attempt: connection failed"
        response=""
    fi
    (( attempt < max_attempts )) && sleep $(( attempt * 2 ))
    (( attempt++ ))
done

if [[ "$response" == "200" ]]; then
    log_info "Service healthy: $url"
    rm -f "$alert_state_file"
    exit 0
fi

if [[ -z "$response" ]]; then
    log_error "Service unreachable after $max_attempts attempts: $url"
else
    log_error "Service unhealthy after $max_attempts attempts: $url (last status: $response)"
fi

if should_alert; then
    log_error "ALERT: $url is down"
    record_alert
else
    log_warn "Suppressing repeat alert (last alert within ${min_alert_interval}s)"
fi
exit "$EXIT_UNHEALTHY"
EOF
chmod +x healthcheck-v2.sh
./healthcheck-v2.sh https://example.com

3. Trigger retries against an unreachable host.

./healthcheck-v2.sh https://this-domain-genuinely-does-not-exist.invalid

4. Confirm alert rate-limiting on a second consecutive failure, then simulate the interval elapsing.

rm -f /tmp/shell-course-healthcheck.state
./healthcheck-v2.sh https://this-domain-genuinely-does-not-exist.invalid
./healthcheck-v2.sh https://this-domain-genuinely-does-not-exist.invalid

echo $(( $(date +%s) - 400 )) > /tmp/shell-course-healthcheck.state
./healthcheck-v2.sh https://this-domain-genuinely-does-not-exist.invalid

Confirm the second run’s log shows the suppression message, while the third — after backdating the state file past min_alert_interval — alerts again.

5. Reproduce the concurrent-alert race, then fix it with a lockfile.

alert_state_file="/tmp/shell-course-demo-alert.state"
rm -f "$alert_state_file"

should_alert() {
    [[ ! -f "$alert_state_file" ]] && return 0
    local last_alert
    last_alert=$(cat "$alert_state_file")
    local now
    now=$(date +%s)
    (( now - last_alert >= 300 ))
}
record_alert() { date +%s > "$alert_state_file"; }

echo "--- broken: no locking ---"
if should_alert; then echo "Run A: would alert"; fi
if should_alert; then echo "Run B: would alert"; fi
record_alert

echo "--- fixed: with locking ---"
lock_file="/tmp/shell-course-demo-healthcheck.lock"
rm -f "$lock_file"

run_locked() {
    local label="$1"
    if [[ -f "$lock_file" ]]; then
        existing_pid=$(cat "$lock_file")
        if kill -0 "$existing_pid" 2>/dev/null; then
            echo "$label: already running — skipping"
            return
        fi
    fi
    echo "$$" > "$lock_file"
    echo "$label: proceeding"
    rm -f "$lock_file"
}
run_locked "Run A"
run_locked "Run B"

6. Clean up.

cd ~
rm -rf ~/shell-course/healthcheck-project
rm -f /tmp/shell-course-healthcheck.state /tmp/shell-course-demo-alert.state /tmp/shell-course-demo-healthcheck.lock

What’s Left For Readers

curl without -L flag does not follow redirects which means with this version of script, site like google.com is ALERTed as down. Either add -L flag or handle redirection codes such as: 301, 302.
getopts can be implemented along with shell parameters so support options like: max-attempt, wait-period, etc. directly from the command line.

Summary

You’ve now added a real timeout, retries with backoff, structured logging, rate-limited alerting with recovery-triggered reset, and — reaching back to the very first project’s locking pattern — closed a genuine concurrency gap this script would otherwise have carried into production. That completes both projects. The final chapter of this course moves to the capstone: a complete, original script built from a specification rather than evolved from an existing one, combining everything this course has covered from scratch.

Last updated on August 31, 2026

Healthcheck V1 — Core Logic