Skip to content

Debugging With Proc Directory

You’ve seen /proc as a place to inspect process information and kernel state. The useful part is that /proc is not just a reference you browse when learning Linux — it is a practical troubleshooting interface you can use when a process is actually running.

The /proc/<PID>/ directory gives you a view into a specific process. By combining a few of its files and directories, you can answer questions such as:

  • Which files does this process currently have open?
  • What environment did it start with?
  • What memory regions does it have mapped?

Finding A Process In /proc

Every running process gets a directory under /proc named after its PID.

For example, if a process has PID 4821:

/proc/4821/

You can inspect it directly:

ls /proc/4821

You will see entries such as:

cmdline
cwd
environ
exe
fd
maps
status

These are not ordinary files stored on disk. They are interfaces provided by procfs, and the kernel generates their contents from the current state of the process.

A useful way to remember this is:

/proc/4821/
      │
      ├── cmdline  → how it was started
      ├── environ  → its environment
      ├── fd/      → its open file descriptors
      ├── maps     → its memory mappings
      └── ...

The exact contents of /proc/<PID>/ are broader than these examples, but these three are particularly useful for troubleshooting.

Inspecting The Command

Start with:

cat /proc/4821/cmdline

You may notice that the arguments are not displayed like a normal shell command. The arguments are separated by NUL characters rather than ordinary spaces.

For a more readable version:

sudo cat /proc/4821/environ | tr '\0' '\n'

For example:

python3 /home/alice/server.py --port 8080

This is useful when you want to know exactly how a running process was started rather than relying only on its short process name.

You can get similar information with:

ps -p 4821 -o pid,user,cmd

The difference is that /proc/<PID>/cmdline exposes the process’s command-line data directly through the kernel’s process interface.

Inspecting The Process Environment

Every process has an environment containing variables such as:

PATH
HOME
USER
LANG

You can inspect it through:

sudo cat /proc/4821/environ

Again, the values are separated by NUL characters, so this is easier to read with:

sudo cat /proc/4821/environ | tr '\0' '\n'

You might see:

HOME=/home/alice
USER=alice
PATH=/usr/local/bin:/usr/bin:/bin
LANG=en_US.UTF-8

This can be extremely useful when a running service behaves differently from the command you run manually.

For example, suppose a program works when you run:

python3 server.py

but fails when started as a service.

One thing worth checking is the environment of the running service:

sudo cat /proc/4821/environ | tr '\0' '\n'

You may discover that variables such as PATH, configuration locations, or application-specific settings differ.

Why Can’t I Always Read environ?

Process information can contain sensitive data.

The environment may contain things such as:

API_KEY=...
DATABASE_PASSWORD=...
TOKEN=...

Therefore, access to another user’s process information can be restricted.

You may need:

sudo

to inspect another user’s environment, depending on the system’s permissions and security settings.

This is also why dumping /proc/<PID>/environ into logs or sharing it casually can expose secrets.

Inspecting Open File Descriptors

One of the most useful directories is:

/proc/<PID>/fd/

For example:

ls -l /proc/4821/fd

You might see:

0 -> /dev/null
1 -> /var/log/server.log
2 -> /var/log/server.log
3 -> socket:[12345]
4 -> /home/alice/data.db

These entries represent the process’s open file descriptors.

The numbers are the descriptor numbers:

0
    standard input

1
    standard output

2
    standard error

3, 4, ...
    additional descriptors opened by the application

The targets show what those descriptors currently refer to.

So:

/proc/4821/fd/4
        │
        ▼
/home/alice/data.db

means that file descriptor 4 of PID 4821 refers to that file.

Why Is /proc/<PID>/fd Useful?

Suppose someone deletes a large log file:

rm /var/log/server.log

but disk space doesn’t come back.

That can happen because a running process still has the deleted file open.

You can investigate with:

sudo ls -l /proc/4821/fd

and may find:

5 -> /var/log/server.log (deleted)

The directory entry has been removed from the filesystem, but the process still has an open file descriptor referring to the underlying file.

The file’s data can therefore remain allocated until that descriptor is closed.

This is a classic Linux troubleshooting case.

The workflow becomes:

Disk space unexpectedly missing
          ↓
Find suspicious process
          ↓
Inspect /proc/<PID>/fd
          ↓
Find "(deleted)" file
          ↓
Process still has it open

This is also something lsof can expose, but /proc lets you see the underlying process interface directly.

Inspecting Memory Mappings

The other useful file is:

/proc/<PID>/maps

Read it with:

cat /proc/4821/maps

The output can be long:

55c1a2b00000-55c1a2b01000 r--p 00000000 08:01 123456 /usr/bin/python3
55c1a2b01000-55c1a2b02000 r-xp 00001000 08:01 123456 /usr/bin/python3
7f21a4000000-7f21a4200000 rw-p 00000000 00:00 0
7f21b0000000-7f21b2000000 r--p 00000000 08:01 654321 /usr/lib/libc.so.6
...

Each line represents a memory region mapped into the process.

You don’t need to become a memory-management expert to get useful information from this file.

The final column often tells you what a mapped region corresponds to:

/usr/bin/python3
/usr/lib/libc.so.6

Other regions may be anonymous memory with no associated file.

What Does A Mapping Mean?

A process does not simply have one continuous block of memory.

Its virtual address space is divided into regions with different purposes and permissions.

A simplified picture is:

Process virtual address space

┌──────────────────────────┐
│ Program / executable     │
├──────────────────────────┤
│ Shared libraries         │
├──────────────────────────┤
│ Heap                     │
├──────────────────────────┤
│ Anonymous mappings       │
├──────────────────────────┤
│ Shared memory / mmap()   │
├──────────────────────────┤
│ Stack                    │
└──────────────────────────┘

/proc/<PID>/maps gives you the kernel’s view of those mappings.

The permission field is particularly useful:

r
    readable

w
    writable

x
    executable

p
    private

s
    shared

So:

r-xp

means a region is readable and executable, privately mapped.

You don’t need to interpret every mapping manually during ordinary troubleshooting. The useful skill is knowing that /proc/<PID>/maps exists when you need to investigate a process’s address space.

A Small Practical Investigation

Let’s put the pieces together with a simple Python process.

Create a small program:

nano /tmp/proc-demo.py

Put this in it:

import os
import time

print(f"PID: {os.getpid()}")
print("Running...")
time.sleep(300)

Run it:

python3 /tmp/proc-demo.py

Suppose it prints:

PID: 4821
Running...

From another terminal, inspect it through /proc.

First:

sudo cat /proc/4821/cmdline | tr '\0' '\n'
echo

Then:

sudo cat /proc/4821/environ | tr '\0' '\n'

Inspect its file descriptors:

ls -l /proc/4821/fd

And finally:

cat /proc/4821/maps

You have now investigated a live process without using a specialized process-inspection command for each piece of information.

The kernel is exposing the information through:

/proc/4821/

/proc Is A Live View

There is one important property to keep in mind: /proc represents current kernel state.

For example:

cat /proc/4821/status

can show information about the process at that moment.

If the process changes state, the information you read later may be different.

If the process exits:

ls /proc/4821

will no longer find that process directory.

This is why /proc should be thought of as a live interface rather than a collection of stored reports.

/proc And The Tools You’ve Learned

At this point, you can see how several Linux tools are often different ways of accessing the same underlying system information.

For example:

ps
 │
 └── convenient process information

/proc/<PID>/status
 │
 └── direct kernel process interface


lsof
 │
 └── convenient open-resource investigation

/proc/<PID>/fd/
 │
 └── direct view of a process's open descriptors

You don’t need to replace ps or lsof with manual /proc inspection.

The point is to understand what those higher-level tools are ultimately helping you inspect.

When a tool doesn’t quite answer your question, knowing the underlying /proc interface often gives you another way to investigate.

Cleaning Up

The Python process from the example is still running if you left it alone.

Find its PID if necessary:

pgrep -af proc-demo.py

Then stop it gracefully:

kill -TERM <PID>

Verify:

pgrep -af proc-demo.py

There should be no matching process left.

Remove the temporary script:

rm /tmp/proc-demo.py

What You Should Remember

The most useful /proc/<PID>/ entries from this chapter are:

/proc/<PID>/cmdline
    → command-line arguments

/proc/<PID>/environ
    → process environment

/proc/<PID>/fd/
    → open file descriptors

/proc/<PID>/maps
    → memory mappings

These are live kernel interfaces, not ordinary files containing information saved on disk.

That gives /proc a particularly useful role in troubleshooting:

Something about a running process is strange
              ↓
        identify its PID
              ↓
       inspect /proc/<PID>/
              ↓
     see what the kernel sees

You don’t need to memorize the entire /proc hierarchy. The important skill is knowing that it exists and knowing where to look when a process needs deeper investigation.

What’s Next

This completes the Virtual & Kernel Filesystems section. The next section returns to the core Linux administration workflow with storage and disks: mounting filesystems, understanding /etc/fstab, checking disk usage, and working with a second disk.

Last updated on
VFS — tmpfs devtmpfs cgroupfs