DEV Community

Cover image for 🔬 Internals of PostgreSQL WAL Streaming with pgrwl: A Cloud-Native Receiver Built in Go
alexey.zh
alexey.zh

Posted on

🔬 Internals of PostgreSQL WAL Streaming with pgrwl: A Cloud-Native Receiver Built in Go

#postgres #go #devops #backup

PostgreSQL's Write-Ahead Logging (WAL) underpins everything from crash recovery to PITR and streaming replication. But streaming WALs safely and reliably, especially into cloud or container-native environments, can be fragile.

Enter pgrwl: a modern WAL receiver written in Go that gives you:

  • 📆 WAL streaming to local/S3/SFTP
  • 🔐 On-the-fly encryption + compression
  • 🐳 Kubernetes-native workflows
  • 📊 Prometheus metrics + Grafana dashboards
  • 🧹 Automatic retention + cleanup
  • ⚙️ Safe WAL handling, with proper fsync guarantees

Let’s explore how it works under the hood.

🧪 Note: pgrwl is a research project, inspired by PostgreSQL’s built-in tools like pg_receivewal, and is
based on careful reading of their official source code and streaming behaviors.

pgrwl serves as a starting point to explore more cloud-native, container-friendly, and extensible
implementations of these time-tested concepts.

📆 The Problem:

For a cloud-native WAL streamer, these features are expected:

  • Cloud uploads (S3, SFTP)
  • Streaming compression
  • Encryption
  • Metrics and observability
  • Retention and cleanup
  • Pod/container integration

Most of all, it should be integrated with modern DevOps pipelines.

⚙️ How pgrwl Works

The architecture captures the streaming and archival lifecycle:

Image description

🔍 Component Breakdown

📅 WAL Receiver

  • Connects to PostgreSQL replication slot
  • Receives XLogData messages via pglogrepl
  • Buffers segment into a .partial file

  • Once the 16 MiB boundary is reached:

    • fsync() the file
    • Atomically rename to final segment

🧠 Archive Supervisor

  • Periodically triggers:

    • WAL uploader
    • WAL retention sweeper

☁️ WAL Uploader

  • Scans for complete segments
  • Compresses + encrypts
  • Streams directly to remote storage
  • Deletes local copy on success

🧹 WAL Retainer

  • Lists remote storage
  • Applies time-based retention
  • Deletes expired segments

🧪 Integrity Testing: Byte-for-Byte Fidelity

pgrwl is tested to ensure:

  • Received WALs match PostgreSQL originals
  • .partial files are never uploaded
  • Segment naming and metadata are preserved

Related articles:

🔄 Use Cases

  • 🛡️ WAL archiving for disaster recovery (PITR)
  • 🔐 Encrypted WAL pipelines
  • 🐳 Kubernetes integration

🤝 Contribute & Collaborate

You can help by:

  • Suggest features
  • File issues
  • Improving CLI and UX

Start here: Contributing

📂 Links

  • GitHub: hashmap-kz/pgrwl
  • Docker: ghcr.io/hashmap-kz/pgrwl
  • Docs: In-repo README & examples
  • License: MIT

Stream safely, backup smart. 🌊

Top comments (0)