Debugging Failed Workflows Systematically

A run turns red. The only step with a red X is “Wait for app to become healthy” — but that step didn’t cause the problem, it just ran out of patience waiting for something that broke earlier. Finding the real cause means working backward from where things stopped, not forward from where you assume they started.

This chapter walks one real debugging session end to end. The pipeline we’re debugging is deliberately small: a stdlib-only Python app, “deployed” by starting it as a background process on the same runner that builds it. No cloud account, no waiting on external infrastructure — just enough of a real deploy sequence (build, stop the old version, start the new one, check it’s actually healthy) to practice the same skills you’d need against anything bigger.

The Pipeline

.
├── app.py
└── .github/workflows/deploy-app.yml

app.py — a minimal HTTP server with a health endpoint:

import http.server
import json
import os

PORT = int(os.environ.get("PORT", 8000))
VERSION = os.environ.get("APP_VERSION", "unknown")


class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            body = json.dumps({"status": "ok", "version": VERSION}).encode()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
        else:
            body = b"Hello, devstuffs!"
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body)


if __name__ == "__main__":
    server = http.server.HTTPServer(("0.0.0.0", PORT), Handler)
    print(f"Serving version {VERSION} on port {PORT}")
    server.serve_forever()

.github/workflows/deploy-app.yml:

name: Deploy App

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Build release
        id: build
        run: |
          mkdir -p release
          cp app.py release/
          echo "version=${{ github.sha }}" >> "$GITHUB_OUTPUT"

      - name: Stop previous instance
        run: |
          [ -f app.pid ] && kill "$(cat app.pid)" 2>/dev/null || true
          rm -f app.pid

      - name: Start app
        run: |
          cd release
          APP_VERSION=${{ steps.build.outputs.version }} PORT=8000 \
            nohup python3 app.py > app.log 2>&1 &
          echo $! > ../app.pid

      - name: Wait for app to become healthy
        run: |
          for i in $(seq 1 30); do
            curl -sf http://localhost:8000/health > /dev/null && exit 0
            sleep 2
          done
          echo "App never became healthy"
          exit 1

      - name: Verify deployed version
        run: |
          actual=$(curl -s http://localhost:8000/health | python3 -c "import sys,json;print(json.load(sys.stdin)['version'])")
          [ "$actual" = "${{ steps.build.outputs.version }}" ] && echo "Verified: $actual"

Setting Up The Exercise

Copy the workflow so you can break it safely, and point its trigger at a throwaway debug-exercise branch:

git checkout -b debug-exercise
cp .github/workflows/deploy-app.yml .github/workflows/deploy-app-debug.yml

 on:
   push:
-    branches: [main]
+    branches: [debug-exercise]

Injecting The Bug

In the copy, find app.py and change the line that reads the port:

- PORT = int(os.environ.get("PORT", 8000))
+ PORT = int(os.environ["APP_PORT"])

This is the kind of thing that slips in during a rename — someone standardized on APP_PORT elsewhere in the codebase and updated the app but not the workflow that starts it (or vice versa). The workflow still sets PORT=8000; the app now looks for APP_PORT, which doesn’t exist.

Commit to debug-exercise and push.

git add app.py .github/workflows/deploy-app-debug.yml
git commit -m 'Start Failure Exercise'
git push origin debug-exercise

Reading The Failed run

“Build release,” “Stop previous instance,” and “Start app” all report success — but the line nohup ... & returns immediately whether or not the command it launched actually keeps running, so backgrounding a process that crashes on its first line still looks like a successful step because backgrounding succeed (exit status 0) — the step reports no failure. The red X lands on “Wait for app to become healthy,” after the full 30 tries and roughly a minute of waiting:

App never became healthy
Error: Process completed with exit code 1.

It’s tempting to conclude the health check itself is broken. It isn’t — it’s the first step that actually verifies anything, so it’s the first place a failure from two steps earlier gets to surface.

Cross-referencing Against What Actually Happened

The workflow log won’t say why the app never came up — for that you need the app’s own output. Add a diagnostic step gated to only run when something upstream (the healthcheck step) failed:

      - name: Show app logs on failure
        if: failure()
        run: cat release/app.log || echo "no log found"

Push again with this step added. Now the failed run includes something similar to:

Traceback (most recent call last):
  File "app.py", line 5, in <module>
    PORT = int(os.environ["APP_PORT"])
KeyError: 'APP_PORT'

Now the failure has a name. The app crashed on its very first line, before it ever bound a socket — which is exactly why the health check never got a response and burned the full minute retrying against nothing.

The Diagnostic Process, Generalized

    flowchart TB
    A[Run shows red X] --> B["Open the failed step's log"]
    B --> C{"Did this step cause the failure, or just detect it?"}
    C -->|Detected it| D["Look at the step(s) before it"]
    C -->|Caused it| E["Read the actual error text"]
    D --> E
    E --> F["Cross-reference against the real system: logs, output files, if: failure() steps"]
    F --> G["Form a hypothesis, then fix it"]
  

Two habits make this repeatable: read the failed step’s raw log text instead of trusting the pass/fail summary, and don’t assume the last thing that ran is the thing that broke — a process that starts and immediately dies still counts as “started” to a step that only checks its own exit code.

git push origin --delete debug-exercise
rm .github/workflows/deploy-app-debug.yml