Testing Across Python Versions and OSes

Your pipeline runs on one Python version, on one operating system, every time. That’s fine for catching most bugs — but “works on Ubuntu with 3.13” and “works everywhere your users actually run this” are two different claims, and right now you can only make the first one. This chapter builds the second, using GitHub’s matrix strategy — and along the way, fixes two real mistakes that are easy to make the first time you write one.

Why Test Across Versions And Operating Systems

Our project (sending and receiving files) isn’t a web app calling a well-behaved HTTP library — send_file.py and receive_file.py talk directly to the OS’s socket layer. That’s exactly the kind of code where “works on my machine” can quietly mean “works on Linux” and nothing more. Socket behavior, default buffer sizes, and how a platform handles a closed connection can all differ between Linux, macOS, and Windows. Testing this project across multiple operating systems isn’t a formality — it’s the one part of your CI setup that’s actually likely to catch something real.

The Python version axis matters for a different reason: you want to know your code works on the versions your users might run, not just whichever one happens to be on your machine.

Meet `strategy.matrix`

A matrix tells GitHub Actions: “run this same job once for every combination of these variables.” Define two lists — say, three Python versions and three operating systems — and you get nine jobs, run in parallel, each with a different combination.

strategy:
  matrix:
    python-version: ["3.12", "3.13", "3.14"]
    os: [ubuntu-latest, windows-latest, macos-latest]

Inside the job, matrix.python-version and matrix.os become variables you can reference anywhere an expression is allowed — most importantly, in runs-on and in the inputs to setup-python.

The Rough Draft

Here’s a first attempt at a dedicated matrix workflow for the project, testing pytest across all nine combinations. It looks reasonable. It has two real bugs in it:

name: Matrix Tests

on: [push, workflow_dispatch, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12", "3.13", "3.14"]
        os: [windows-latest, ubuntu-latest, masos-latest]

    steps:
      - name: Checkout repo
        uses: actions/checkout@v7

      - name: Setup Python
        uses: actions/setup-python@v7
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install pytest
        run: python -m pip install --upgrade pip pytest

      - name: Run Tests
        run: python -m pytest

If you’d pushed this as-is, the Actions tab would show nine jobs running, and most of them going green. It would look like a working cross-platform matrix. It isn’t.

Bug 1: The Runner Label Typo

Look closely at the os list: masos-latest. That’s not a valid GitHub-hosted runner label — the real one is macos-latest. This is the kind of typo that’s genuinely easy to miss; the two labels differ by one transposed letter, and nothing in the YAML editor will flag it as wrong syntax, because it isn’t — masos-latest is a perfectly valid string, just not one that matches any runner GitHub actually has.

Warning

This doesn’t fail loudly. A job requesting a runner label that doesn’t exist doesn’t error out — it just sits there waiting for a runner that will never show up. In the Actions UI, that job shows as queued, indefinitely, while its siblings finish normally around it. If you don’t notice a job stuck in “Queued” and go cancel it yourself, it’ll eventually time out — but nothing points you at the typo directly. You have to go check the label yourself.

The fix is simple once you’ve spotted it: macos-latest, spelled correctly.

Bug 2: The Matrix Variable Nobody Used

This one is more important, and much easier to miss, because it produces no error at all — every job finishes and goes green.

Look at runs-on: ubuntu-latest at the top of the job. It’s hardcoded. The os list inside matrix is defined — three values, including the (now-fixed) macos-latest and windows-latest — but nothing in the job actually reads matrix.os. runs-on never references it.

Warning

Defining a matrix variable doesn’t automatically wire it into anything. matrix.os existing in your strategy.matrix block has no effect unless something in the job — almost always runs-on — actually references it with ${{ matrix.os }}. Without that, every “OS leg” of your matrix runs on the exact same runner your runs-on line names, three times over. You get three green checkmarks that look like Windows, macOS, and Linux coverage, and you’ve actually tested Ubuntu three times.

This is the trap: the workflow looks correct. It has a matrix, the matrix has an os axis, the job runs nine times, everything passes. Nothing in the YAML is invalid. The only way to catch it is to actually check what runs-on resolves to — and remember that a matrix variable is inert until something references it.

The fix:

runs-on: ${{ matrix.os }}

The Fixed Workflow

Putting both fixes together:

name: Matrix Tests

on: [push, workflow_dispatch, pull_request]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        python-version: ["3.12", "3.13", "3.14"]
        os: [windows-latest, ubuntu-latest, macos-latest]

    steps:
      - name: Checkout repo
        uses: actions/checkout@v7

      - name: Setup Python
        uses: actions/setup-python@v7
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install pytest
        run: python -m pip install --upgrade pip pytest

      - name: Run Tests
        run: python -m pytest

Note

This workflow is deliberately separate from your main ci.yml. Linting doesn’t need to run nine times — the code either follows the style rules or it doesn’t, regardless of OS or Python version. Running it once, on one platform, in ci.yml is enough. The matrix is reserved for the one check that’s genuinely platform- and version-sensitive: does the test suite actually pass everywhere. And you can remove:

- name: Run Tests
  run: python -m pytest

Exactly this step from ci.yml and put it on matrix-tests.yml.

You’ll notice continue-on-error and fail-fast are absent here for now — they’re both real matrix settings you’ll want, but they deserve their own explanation rather than being copied in blind. That’s the next chapter.

Hands-On: Running It For Real

Add the fixed workflow:

cat > .github/workflows/matrix-tests.yml << 'EOF'
name: Matrix Tests

on: [push, workflow_dispatch]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        python-version: ["3.12", "3.13", "3.14"]
        os: [windows-latest, ubuntu-latest, macos-latest]

    steps:
      - name: Checkout repo
        uses: actions/checkout@v7

      - name: Setup Python
        uses: actions/setup-python@v7
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install pytest
        run: python -m pip install --upgrade pip pytest

      - name: Run Tests
        run: python -m pytest
EOF

And push the changes:

git add .github/workflows/ci.yml .github/workflows/matrix-tests.yml
git commit -m "Add cross-platform matrix test workflow"
git push

Open the Actions tab. You should see nine jobs listed under “Matrix Tests,” each labeled with its Python version and OS combination, all running in parallel. Give them a few minutes — Windows and macOS runners typically take longer to spin up than Linux — and confirm all nine finish green.

That’s real coverage now: the same test suite, proven to pass on three operating systems and three Python versions, on every push, with no one having to remember to check it manually on a Mac or a Windows machine by hand.

Nine parallel jobs also raises a new question, though: what should happen when one of them fails? Right now, the answer is whatever GitHub’s defaults do — and those defaults are more aggressive than you might expect. That’s exactly what the next chapter covers.

Last updated on September 14, 2026

Running Tests in CI fail-fast vs continue-on-error in Matrix