Union/Overlay Filesystems: Docker Building Block

I knew from the beginning that union or overlay filesystem is one of the building block of Docker which makes image layers read only with tiny writable layer on top of them but never knew WTH is OverlayFS. I said “enough is enough” and finally learnt. Here’s everything you need to know — Docker is not only the use case tho, there is more OverlayFS offers.

OverlayFS can look strange at first: you mount several directories together, yet you work with what looks like one normal directory. Where did a file come from? Where does a new file go? What happens when you change something that came from the lower layer? What happens on deletion?

Once you understand those rules, the whole thing becomes much simpler. You can think of OverlayFS as a normal-looking workspace built from a read-only base plus a writable layer. This is not Docker-centric but when you get the general idea about OverlayFS, you can directly relate that with the concept of Docker image — tiny writable layer on top of read-only layers.

What Is OverlayFS?

A union filesystem is a general idea: make multiple filesystem trees appear as one.

OverlayFS is Linux’s implementation of that idea.

For this lesson, we will work with three important directories:

lower/     → existing base files
upper/     → writable changes
merged/    → the combined view you actually use

There is also a work/ directory that OverlayFS needs for a writable mount. You normally don’t put your files there.

The basic picture is:

    flowchart TB
    L["lowerdir<br/>Read-only base"]
    U["upperdir<br/>Writable changes"]
    M["merged<br/>Your filesystem view"]

    L --> M
    U --> M
  

The important thing is that merged/ is not a third copy of everything.

It is a view created by OverlayFS.

Why Have Two Layers?

Imagine you have a base filesystem that you don’t want to modify:

lower/
├── app.conf
├── common.txt
└── old.txt

You want to make changes without touching that base.

OverlayFS lets you put those changes somewhere else:

upper/
├── app.conf
└── new.txt

Then merged/ makes them look like one filesystem:

merged/
├── app.conf
├── common.txt
├── new.txt
└── old.txt

This is the basic idea:

             lower + upper
                   ↓
                 merged

How Does OverlayFS Merge Files?

The rules are easier than they first appear.

What If a File Exists in Only One Layer?

If a file exists only in lower, it appears in merged.

If a file exists only in upper, it also appears in merged.

For example:

lower/
└── lower.txt

upper/
└── upper.txt

The merged view contains:

merged/
├── lower.txt
└── upper.txt

What If the Same File Exists in Both?

The upper version wins.

For example:

lower/conflict.txt
    → "I am lower"

upper/conflict.txt
    → "I am upper"

Then:

cat merged/conflict.txt

shows:

I am upper

The lower file hasn’t disappeared. It is simply hidden by the upper file with the same path.

So remember:

Same path in both layers? Upper wins.

What If a Directory Exists in Both?

Directories are different from conflicting regular files.

Suppose:

lower/
└── config/
    ├── app.conf
    └── database.conf

upper/
└── config/
    └── debug.conf

The merged directory contains all three:

merged/
└── config/
    ├── app.conf
    ├── database.conf
    └── debug.conf

The contents of the two directories are combined.

So the simple rule is:

Same file path
    → upper hides lower

Same directory
    → contents are combined

What Happens When You Use the Merged Directory?

This is the most important part.

After the OverlayFS mount, you normally work inside merged/.

You don’t start writing your application directly inside upper/.

Think of the directories like this:

lower/   → base, don't modify
upper/   → OverlayFS stores your changes here
work/    → OverlayFS uses this internally
merged/  → the place you actually work

If you are writing code, editing configuration, creating files, or running an application, you normally use:

merged/

OverlayFS decides what needs to happen underneath.

For example:

cd merged
touch new.py
mkdir src
echo "hello" > src/app.py

You don’t have to manually decide: “Should this go into upper?”

OverlayFS handles that.

A useful way to remember it is:

merged is your workspace. upper is where OverlayFS stores your changes.

File Lookup: Where Does a Read Come From?

Suppose:

lower/
└── hello.txt

upper/

You run:

cat merged/hello.txt

Conceptually, OverlayFS checks the upper layer first:

Look for hello.txt
       ↓
   upper/hello.txt?
       ↓
      no
       ↓
   lower/hello.txt?
       ↓
      yes
       ↓
     return it

If the file exists in both:

Look for hello.txt
       ↓
   upper/hello.txt?
       ↓
      yes
       ↓
 use upper version

So:

For a conflicting file, OverlayFS uses the upper version.

Directory Lookup: What Does ls Show?

Suppose:

lower/
└── src/
    ├── main.py
    └── config.py

upper/
└── src/
    └── test.py

Then:

ls merged/src

shows:

config.py
main.py
test.py

The directory is presented as one combined directory. You don’t need to know which layer each file came from to use it.

Where Do Writes Go?

This is where OverlayFS becomes especially interesting.

Creating a Completely New File

Suppose there is no new.txt anywhere:

lower/
upper/

Now:

echo "hello" > merged/new.txt

The new file is created in the upper layer:

upper/
└── new.txt

The lower layer is untouched.

The merged view now shows:

merged/
└── new.txt

So:

Create new file in merged
          ↓
      upper layer

Creating a File Inside a Lower Directory

Now consider:

lower/
└── src/
    └── main.py

upper/

You create:

touch merged/src/test.py

You are working inside merged, but the change belongs to the writable layer.

Conceptually:

merged/src/test.py
        ↓
upper/src/test.py

OverlayFS creates the necessary upper-layer directory structure.

You do not need to manually create upper/src.

Modifying a Lower File

This is the part that introduces copy-up.

Suppose:

lower/
└── app.conf

and there is no upper/app.conf.

You run:

echo "changed" > merged/app.conf

OverlayFS cannot modify the lower file directly because the lower layer is read-only.

Instead, it copies the file into the upper layer and applies the change there.

Conceptually:

lower/app.conf
      │
      │ copy-up
      ▼
upper/app.conf
      │
      ▼
modified version

Afterward:

lower/
└── app.conf        ← original remains unchanged

upper/
└── app.conf        ← modified copy

merged/
└── app.conf        ← upper version is visible

This is called copy-up.

The important thing is not the name. Remember the behavior:

If you modify a lower-layer file, OverlayFS creates an upper-layer copy and modifies that copy.

Modifying a File That Is Already in Upper

If the file already exists in upper, there is nothing to copy.

For example:

upper/
└── app.conf

Then:

echo "another change" >> merged/app.conf

simply modifies the upper copy.

So there are three useful write cases:

New file
    → create in upper

Modify lower file
    → copy-up → modify upper

Modify upper file
    → modify upper directly

What Happens When You Delete a Lower File?

Suppose:

lower/
└── old.txt

You run:

rm merged/old.txt

OverlayFS cannot physically remove lower/old.txt.

Instead, it records the deletion in the upper layer so that the file is hidden from the merged view.

This is called a whiteout. You don’t normally create whiteouts yourself. OverlayFS handles it. See ls -l upper/old.txt after whiteout is marked:

c--------- 2 root root 0, 0 Aug 16 19:44 old.txt

The result is:

lower/
└── old.txt          ← still physically there

upper/
└── deletion marker  ← hides old.txt

merged/
                    ← old.txt is gone

When Is OverlayFS Useful?

OverlayFS is not only a Docker trick.

The general use case is:

Keep a base filesystem unchanged while allowing another layer to contain changes.

Temporary or Disposable Environments

You may have a filesystem that you want to experiment with without modifying the original.

Read-only base
      +
Temporary changes
      ↓
Working filesystem

When you’re finished, the writable layer can be discarded.

The original base remains unchanged.

Read-Only System Images

An embedded system or appliance may want most of its base filesystem to remain read-only while still allowing runtime changes.

The same model works:

Stable base
    +
Writable runtime changes
    ↓
One filesystem

Testing

OverlayFS can also be useful when you want to experiment with a filesystem tree without changing the original data.

You can mount a writable layer over the base, perform changes through merged, and later throw away the upper layer.

Containers

Containers are one of the most familiar uses of this general idea.

Docker can use layered filesystems so that image content remains in read-only layers while a container gets its own writable layer.

Conceptually:

Docker image layers
        ↓
   read-only layers
        +
container changes
        ↓
   writable layer
        ↓
      merged
        ↓
container filesystem

The exact storage implementation depends on Docker’s storage driver and configuration, but the OverlayFS model you’ve just learned is the important foundation.

Let’s Build One Ourselves

Now let’s stop talking about the model and prove it.

Create the Directory Structure

Create a small playground:

mkdir -p overlay-demo/{lower,upper,work,merged}
cd overlay-demo

We’ll create files that make the merge rules easy to see.

Put these in the lower layer:

printf 'I am from lower\n' > lower/lower-only.txt
printf 'I am the lower version\n' > lower/conflict.txt
mkdir lower/src
printf 'I am main.py\n' > lower/src/main.py

Put these in the upper layer:

printf 'I am from upper\n' > upper/upper-only.txt
printf 'I am the upper version\n' > upper/conflict.txt
mkdir upper/src
printf 'I am config.py\n' > upper/src/config.py

At this point:

lower/
├── conflict.txt
├── lower-only.txt
└── src/
    └── main.py

upper/
├── conflict.txt
├── src/
│   └── config.py
└── upper-only.txt

Mount the Overlay

Now mount the two layers:

sudo mount -t overlay overlay \
  -o lowerdir="$PWD/lower",upperdir="$PWD/upper",workdir="$PWD/work" \
  "$PWD/merged"

The merged directory is now the filesystem view produced by OverlayFS.

Check it:

find merged -type f -print

You should see files from both layers.

Prove That Upper Wins

Read the conflicting file:

cat merged/conflict.txt

You should get:

I am the upper version

Even though the lower layer also contains conflict.txt.

Now check the lower version directly:

cat lower/conflict.txt

It still says:

I am the lower version

The lower file was not replaced.

The upper file simply hides it from the merged view.

Prove That Directories Are Combined

Check:

find merged/src -type f -print

You should see:

merged/src/main.py
merged/src/config.py

The two src directories became one view.

Create a New File Through merged

Now work where you normally would:

printf 'Created through merged\n' > merged/new.txt

Check:

cat merged/new.txt

Then inspect the upper layer:

cat upper/new.txt

The file is there.

You created it through merged, but OverlayFS stored it in upper.

Prove Copy-Up

Now modify the lower-only file:

printf 'I changed this through merged\n' > merged/lower-only.txt

Read it through the merged view:

cat merged/lower-only.txt

You should see:

I changed this through merged

Now inspect the lower layer:

cat lower/lower-only.txt

It should still contain:

I am from lower

Finally:

cat upper/lower-only.txt

You should see the changed version.

That proves copy-up happened:

lower/lower-only.txt
        ↓
      copy-up
        ↓
upper/lower-only.txt
        ↓
     modified

Delete a Lower File

Finally:

rm merged/lower-only.txt

Now:

ls merged

The file is gone from the merged view.

But:

ls lower

can still show the original lower-layer file.

The lower file wasn’t physically deleted. OverlayFS recorded the deletion in the upper layer so that the merged view hides it.

The Rules to Remember

After working through the example, the behavior can be summarized like this:

The most useful mental model is:

                         merged/
                    YOUR WORKSPACE
                           │
            ┌──────────────┼──────────────┐
            │              │              │
           read           create        modify
            │              │              │
            ▼              ▼              ▼
      upper first        upper       upper if present
      then lower                      copy-up if lower

And the three directories have very different jobs:

lower/   → base content
upper/   → where changes are stored
merged/  → where you actually work
work/    → OverlayFS working area

Clean Up

Unmount the filesystem first:

cd ..
sudo umount overlay-demo/merged

Then remove the exercise:

rm -rf overlay-demo

Unmounting matters because merged is currently a mounted filesystem view. Removing the directory without unmounting first is not the right cleanup procedure.

The Mental Model to Keep

When you think about OverlayFS, remember this:

                 MERGED
             what you use
                  │
          ┌───────┴───────┐
          │               │
      UPPERDIR         LOWERDIR
      writable          read-only
          │               │
          └───────┬───────┘
                  │
             combined view

Then remember the rules:

Read:
    upper first → lower if not there

Same file in both:
    upper wins

New file:
    upper

Modify lower file:
    copy-up → upper

Delete lower file:
    hide it from merged view

The most important distinction is:

You work in merged; OverlayFS stores your changes in upper.

Once that makes sense, Docker’s layered filesystem becomes much easier to understand: the image can provide read-only lower layers, while the container gets a writable layer for its own changes.