Project Introduction

Even before you starting writing first line in your CI/CD workflow, it is always to have general idea about the project which will better shape you to write better pipeline that has less chances to do things wrong. Everything you do after this chapter depends on this one.

The project is intentionally simple because practice gets much easier when the project itself is small enough that you can understand every moving part. Before we start building CI/CD for project, let’s first make sure we know exactly what the pipeline will be testing.

This is a small Python client-server project, made available at my own repo. The application is intentionally simple: one program waits for a connection and sends a file, while another connects to it and receives that file. That gives us just enough code to demonstrate testing and CI/CD without getting buried in application complexity.

Fork and Clone The Project

The Zeroth towards writing any workflow is making the project available as your local repository. So, fork and then clone (maybe give a star too) the devstuffs-building-real-ci-pipelines repository.

Kudos to the OG creators: This project is replicated from The Algorithms/Python/file_transfer. The main idea behind choosing this project is to choose random and easy project from the internet to get started and being luckly we don’t have to write tests by hand — it’s already there.

The Project Structure

The fork looks like this:

.
├── .github
│   ├── actions
│   │   └── setup-python-env
│   │       └── action.yml
│   └── workflows
│       ├── ci.yml
│       └── matrix-tests.yml
├── .gitignore
├── README.md
├── __init__.py
├── mytext.txt
├── receive_file.py
├── requirements.txt
├── scripts
│   └── summarize_report.py
├── send_file.py
└── tests
    ├── __init__.py
    └── test_send_file.py

Clone the repository and intentionally delete the following because we will be learning these all in this section of Building Real CI Pipelines:

git clone https://github.com/mragochar/devstuffs-building-real-ci-pipelines.git
cd devstuffs-building-real-ci-pipelines/
rm -r .github scripts requirements.txt

tree

Now you are ready to start:

.
├── README.md
├── __init__.py
├── mytext.txt
├── receive_file.py
├── send_file.py
└── tests
    ├── __init__.py
    └── test_send_file.py

There are really only three pieces we care about right now:

• send_file.py — the server
• receive_file.py — the client
• tests/test_send_file.py — the automated test

mytext.txt is the file transferred between the two programs.

The Small Change In The Fork

The fork makes one functional change to the test file.

Originally, the tests/test_send_file.py imported the function like this:

from file_transfer.send_file import send_file

It was changed to:

from send_file import send_file

Why?

We are going to run the test from the root of this project, where send_file.py is directly available as a module. Python includes this path (working directory) already in its module search path (sys.path), so the test can import send_file directly.

This also makes the example convenient for the CI pipeline we are about to build: the repository root is the working directory, and the test command can run against the tests/ directory from there.

Understanding The Application

Before automating anything, let’s run the application manually. CI/CD will eventually do some of this work automatically, so you should first see what the application actually does.

Client-Server Model

Think of the two Python files as two ends of a network connection:

    flowchart LR
    A["receive_file.py<br/>Client"] -->|"connects to TCP port 12312"| B["send_file.py<br/>Server"]
    B -->|"sends mytext.txt"| A
    A -->|"writes received data"| C["Received_file"]
  

send_file.py creates a socket, listens on TCP port 12312, and waits for clients. When a client connects, it sends the contents of mytext.txt.

receive_file.py acts as the client. It connects to port 12312, receives the data, and writes it to a file named Received_file.

There is one detail worth making clear because the words server and client can otherwise make the behavior sound more complicated than it is.

The server stays alive and keeps listening for connections. It doesn’t shut down after serving one client.

The individual client connection does not stay open forever. Once the server has finished sending the file, that connection is closed. The client also closes its side after successfully receiving the file.

So the lifecycle looks like this:

    sequenceDiagram
    participant S as send_file.py
    participant C as receive_file.py

    S->>S: Listen on port 12312
    C->>S: Connect
    S->>C: Send mytext.txt
    S-->>C: Close client connection
    C->>C: Save as Received_file
    C-->>S: Client connection closed
    Note over S: Server continues listening
  

That distinction matters: the server keeps serving, while each client connection is temporary.

See It In Action

Let’s run the application manually before worrying about automation.

1. Start The Server

Open one terminal in the project directory and start the server:

python3 send_file.py

You should see:

Server listening....

At this point, the server is waiting for a client.

2. Start The Client

Open another terminal in the same project directory and run:

python3 receive_file.py

The client should show something similar to:

file.py
File opened
Receiving data...
Successfully received the file
Connection closed

Meanwhile, the server will report the connection and the data it sent:

Got connection from ('127.0.0.1', 35282)
Server received: data = b'Hello server!'
Sent b'Hello\nThis is sample data\n\xc2\xabk\xc3\xbc\xc3\x9f\xc3\xae\xc2\xbb\n\xe2\x80\x9c\xd0\x8c\xcf\x8d\xd0\x91\xd0\x87\xe2\x80\x9d\n\xf0\x9f\x98\x80\xf0\x9f\x98\x89\n\xf0\x9f\x98\x8b\n'
Done sending

The exact client port, such as 35282, can be different on every run. It is the temporary source port chosen for that client connection.

The important part is the sequence:

1. The server starts listening.
2. The client connects.
3. The server sends mytext.txt.
4. The server finishes that client connection.
5. The client writes the received data to Received_file.
6. The server remains available for another client.

3. Verify The Result

The simplest way to inspect the received file is:

cat Received_file

You should get the same contents as mytext.txt:

Hello
This is sample data
«küßî»
“ЌύБЇ”
😀😉
😋

We can verify that this isn’t merely visually the same file.

Use diff:

diff mytext.txt Received_file

If the files are identical, diff produces no output.

That gives us a useful definition of success for this application:

The client successfully receives the file when Received_file is identical to mytext.txt.

This is exactly the sort of behavior we want automated tests to verify. Instead of manually starting programs and looking at their output every time someone changes the code, we can let a test check the important behavior for us.

How Testing Looks

The project already contains a test under tests/.

We use pytest to run it. If you don’t have it installed yet:

pip3 install pytest

Then run the tests from the root of the project:

python3 -m pytest tests/

A successful run looks like this:

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/devstuffs/building-real-ci-pipelines
collected 1 item

tests/test_send_file.py .                                                  [100%]

============================== 1 passed in 0.05s ===============================

The important line is:

1 passed

There is currently only one test, so the test suite is deliberately tiny. That’s fine. We aren’t trying to build a sophisticated Python application here.

The application gives us something much more useful for learning CI/CD: a small, reproducible project with an automated test that can tell us whether a change still passes.

That is the foundation our pipeline will use.

Why Start Here?

CI/CD is not magic that somehow makes an application reliable. A pipeline simply automates steps that you could otherwise perform manually.

For this project, we already have the essential ingredients:

    flowchart LR
    A["Developer changes code"] --> B["Run automated test"]
    B --> C{"Test passes?"}
    C -->|"Yes"| D["Safe to continue pipeline"]
    C -->|"No"| E["Stop and report failure"]
  

Right now, you have to remember to run:

python3 -m pytest tests/

After we build the CI/CD pipeline, the automation will run that command for us when a change reaches the repository.

That is the point of this exercise. The application is intentionally simple so that we can concentrate on the pipeline itself rather than spending the lesson debugging a complicated application.