Tracing Syscalls with eBPF in Docker: A Practical Example

TL;DR: In this post, I'll walk you through an example of combining a FastAPI service with an eBPF tracer to monitor syscalls. Along the way, I'll share some of the pitfalls I encountered while trying to develop on macOS, how I ended up containerizing everything, and how I finally managed to see the syscalls I was interested in. It was a fun refresher of kernel + eBPF knowledge!

Jump to the complete code on GitHub.

Why eBPF?

eBPF (extended Berkeley Packet Filter) is a powerful technology in the Linux kernel that lets you run sandboxed programs in the kernel space without having to write and load kernel modules. This enables you to hook into kernel-level events, like syscalls, network packets, and many more, all in a safe manner.

In this project, I wanted to track syscalls for a certain process (by PID) so that I could observe all the calls that process makes to the kernel in real time. To do so, I used a tracepoint (raw_syscalls:sys_enter) that the Linux kernel exposes for hooking into the entry point of a syscall.

Project Overview

The repository is divided into two main parts:

These two services are tied together by a docker-compose.yml, which spins up:

High-Level Structure

.
├── docker-compose.yml
├── fastapi-server/
│   ├── main.py
│   ├── models.py
│   ├── schemas.py
│   ├── Dockerfile
│   └── ...
└── tracer/
    ├── cmd/
    │   └── tracer/
    │       └── main.go
    ├── pkg/bpf/
    │   ├── tracer.c
    │   └── tracer.h
    ├── Dockerfile
    └── ...

Note: The tracer.c code is compiled into eBPF bytecode, and the Go code (main.go) is responsible for loading it into the kernel, attaching it, and reading out the events.

The Roadblocks (and Lessons Learned)

1. Developing on macOS

I originally tried to compile and run eBPF code directly on macOS. Turns out, eBPF is a Linux kernel feature and does not work on macOS. While macOS is UNIX-like, it's definitely not a "standard Linux environment," so the compilation toolchain for eBPF programs and the kernel debug interfaces are missing.

Lesson: If you want to do eBPF development, you need a Linux kernel. The simplest route is to use Docker with a Linux-based image or run on a native Linux machine or VM.

2. Attempting to Run the Tracer in a Container (on macOS) to Capture Host Syscalls

I tried something like this:

docker run --privileged \
  --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /sys/kernel/tracing:/sys/kernel/tracing \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -e TARGET_PID=79696 \
  ebpf-tracer

I was hoping to see syscalls for my host's PID 79696. However, I got no data from my macOS host processes. Why? Because even though I used --privileged and shared the various sysfs directories, Docker Desktop for Mac runs inside a Linux VM; it won't magically show you processes on the macOS host. In short, there's a mismatch between the host environment (macOS) and the container environment (Linux).

Lesson: If you're on macOS, the best approach is to treat Docker Desktop's Linux VM as the actual "host" kernel. You won't see macOS processes (like PID 79696 from the macOS standpoint). You will see processes inside the Docker VM.

3. Hooking the Correct Calls

Initially, I tried hooking into sched_process_exec (a tracepoint under sched). That gave me some insight into processes being executed, but I specifically needed syscalls. So I switched to:

SEC("tracepoint/raw_syscalls/sys_enter")
int handle_sys_enter(struct syscall_enter_args *ctx)
{
    // ...
}

This is the correct tracepoint for hooking entering syscalls. Each time a process triggers a syscall, I get a new handle_sys_enter call. From there, I can extract the PID, the timestamp, the syscall ID, the command name, etc.

Lesson: eBPF has a wide range of hooks (kprobes, uprobes, tracepoints, etc.). Use the right hook for the data you want.

4. Containerizing Everything and Linking the Tracer to the FastAPI Service

Finally, I containerized both components:

With the right privileges and mounts (/sys/kernel/debug, /sys/kernel/tracing, etc.), I can see the syscalls made by processes inside the FastAPI container. This means that any user hitting my FastAPI endpoints can indirectly trigger syscalls (e.g., open file, read file, write file, etc.), and the eBPF tracer picks that up.

Lesson: Putting your monitored application and eBPF tracer in the same Docker Compose environment (or similar) helps you see the syscalls of interest.

Example: The Tracer Code

Here's a quick snippet of my main.go (in tracer/cmd/tracer/main.go) showing how we load the eBPF objects and attach them to the tracepoint:

tp, err := link.Tracepoint("raw_syscalls", "sys_enter", objs.HandleSysEnter, nil)
if err != nil {
    log.Fatalf("Opening tracepoint: %v", err)
}
defer tp.Close()

rd, err := perf.NewReader(objs.Events, os.Getpagesize()*16)
if err != nil {
    log.Fatalf("Creating perf event reader: %v", err)
}
defer rd.Close()

go func() {
    for {
        record, err := rd.Read()
        // ...
        // parse record and print the results
    }
}()

// Wait for Ctrl-C
<-sig
fmt.Println("\nReceived signal, exiting...")

We attach to raw_syscalls:sys_enter, create a perf buffer reader, and then go into an infinite loop reading events (syscall logs). If the user is in the same container (or we share PID + mount namespaces) and triggers a syscall, we'll see the record in real time.

Example: The FastAPI Server

The FastAPI side is a standard CRUD for users plus file upload/download. When you do something like:

curl -F file=@myfile.txt http://localhost:8000/upload/

The Python code in main.py calls open(...) under the hood to write the file to disk. That triggers a syscall event, which our tracer sees. Similarly, for downloads or CRUD operations on the users.db (SQLite), you'll see calls to open, read, write, fstat, etc.

@app.post("/upload/")
def upload_file(file: UploadFile = File(...)):
    file_location = os.path.join(UPLOAD_DIR, file.filename)
    with open(file_location, "wb") as f:
        f.write(file.file.read())
    return {"info": f"File '{file.filename}' saved at '{file_location}'"}

Running it All with Docker Compose

Here's what the docker-compose.yml might look like in simplified form:

services:
  fastapi:
    build:
      context: ./fastapi-server
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    volumes:
      - ./fastapi-server:/app
    command: uvicorn main:app --host 0.0.0.0 --port 8000 --reload

  tracer:
    build:
      context: ./tracer
      dockerfile: Dockerfile
    privileged: true
    pid: "host"
    network_mode: "service:fastapi"
    volumes:
      - /sys/kernel/debug:/sys/kernel/debug
      - /sys/kernel/tracing:/sys/kernel/tracing
      - /sys/fs/bpf:/sys/fs/bpf
    depends_on:
      - fastapi

Wrap-Up

In summary, the key takeaways for building an eBPF-based tracer in a containerized environment are:

I hope this example helps you understand how to tie together a minimal eBPF tracer with a Python-based FastAPI application. It was an excellent exercise to refresh my kernel + eBPF knowledge and taught me the importance of properly configuring Docker and hooking into the right kernel tracepoints.

Happy tracing! Check out the complete code on GitHub.