2ndQuadrant is now part of EDB

Bringing together some of the world's top PostgreSQL experts.

2ndQuadrant | PostgreSQL
Mission Critical Databases
  • Contact us
  • EN
    • FR
    • IT
    • ES
    • DE
    • PT
  • Support & Services
  • Products
  • Downloads
    • Installers
      • Postgres Installer
      • 2UDA – Unified Data Analytics
    • Whitepapers
      • Business Case for PostgreSQL Support
      • Security Best Practices for PostgreSQL
    • Case Studies
      • Performance Tuning
        • BenchPrep
        • tastyworks
      • Distributed Clusters
        • ClickUp
        • European Space Agency (ESA)
        • Telefónica del Sur
        • Animal Logic
      • Database Administration
        • Agilis Systems
      • Professional Training
        • Met Office
        • London & Partners
      • Database Upgrades
        • Alfred Wegener Institute (AWI)
      • Database Migration
        • International Game Technology (IGT)
        • Healthcare Software Solutions (HSS)
        • Navionics
  • Postgres Learning Center
    • Webinars
      • Upcoming Webinars
      • Webinar Library
    • Whitepapers
      • Business Case for PostgreSQL Support
      • Security Best Practices for PostgreSQL
    • Blog
    • Training
      • Course Catalogue
    • Case Studies
      • Performance Tuning
        • BenchPrep
        • tastyworks
      • Distributed Clusters
        • ClickUp
        • European Space Agency (ESA)
        • Telefónica del Sur
        • Animal Logic
      • Database Administration
        • Agilis Systems
      • Professional Training
        • Met Office
        • London & Partners
      • Database Upgrades
        • Alfred Wegener Institute (AWI)
      • Database Migration
        • International Game Technology (IGT)
        • Healthcare Software Solutions (HSS)
        • Navionics
    • Books
      • PostgreSQL 11 Administration Cookbook
      • PostgreSQL 10 Administration Cookbook
      • PostgreSQL High Availability Cookbook – 2nd Edition
      • PostgreSQL 9 Administration Cookbook – 3rd Edition
      • PostgreSQL Server Programming Cookbook – 2nd Edition
      • PostgreSQL 9 Cookbook – Chinese Edition
    • Videos
    • Events
    • PostgreSQL
      • PostgreSQL – History
      • Who uses PostgreSQL?
      • PostgreSQL FAQ
      • PostgreSQL vs MySQL
      • The Business Case for PostgreSQL
      • Security Information
      • Documentation
  • About Us
    • About 2ndQuadrant
    • 2ndQuadrant’s Passion for PostgreSQL
    • News
    • Careers
    • Team Profile
  • Blog
  • Menu Menu
You are here: Home1 / Blog2 / Gabriele's PlanetPostgreSQL3 / Management of the WAL archive in Barman
Gabriele Bartolini

Management of the WAL archive in Barman

August 22, 2012/4 Comments/in Gabriele's PlanetPostgreSQL /by Gabriele Bartolini

Barman, backup and recovery manager for PostgreSQL, is designed to manage the archive of WAL files separately from periodical backups (in Postgres terms, base backups).

You can see this archive as a “continuous” stream of files from the first available backup to the last shipped file (backup available history for a server).

In this article you will see how Barman manages WAL compression and archival, as well as how a particular WAL file is associated to a base backup in the history, granting database administrator the privilege to immediately know the exact physical size of a periodical backup (WAL files included) through the catalogue.

A quick recap:

  • Backup is performed on a per server/instance basis (which means that single database backup is not supported)
  • Backup servers in Barman are defined in the configuration file (each server has a section in the INI file, as described in the documentation)

The WAL archiver of the Postgres server asynchronously sends the newly generated WAL file containing binary transaction information to the backup server. WAL files must be deposited in the incoming_wals_directory for that server, as showed by the “barman show-server” command (by default, the convention is to place these files in the SERVER_BACKUP_DIR/incoming directory).

Usually, the best approach (especially in a local network), is to let Barman compress these 16MB files on the backup server end (you can use the “compression” option for this purpose). This operation is performed jointly with archival by the “barman cron” command, which goes through every file in the incoming directory, performs compression (where applicable) and permanently moves the WAL file in the WAL archive for that server. Bear in mind that this process is asynchronous and does not cause any delay for the normal archiving procedures of the server.

According to Barman design, every WAL file is automatically associated to the closest previous base backup available in the server’s backup history.

This definition implies that, in case no base backup has yet been taken, the cron command discards the incoming WAL files from the server’s archive.

The same definition however allows Barman to give DBAs more accurate information about the overall size of a periodical backup, including the size of the base backup (with tablespace data) and the number and size of the compressed WAL files.

The following screenshot for the “barman show-server” command gives you an idea (the test example does not take advantage of WAL compression – see the 48MBs occupied by the three WAL files?).

An example of output from the "barman show-backup" command

Another command that has an impact on the WAL archive is the “backup delete” command.

In case the first available backup is deleted, all the associated WAL files are removed from the archive as well.

Things get more interesting if you delete an intermediate backup. Consider this scenario: your server’s backup history shows three backups, taken at three subsequent times (t1, t2 and t3 with t3 > t2 > t1). If you get rid of the “t2” backup, the WAL files that were originally associated to this backup are not removed, rather  immediately assigned to the previous available backup (“t1” in this example).

Among other things, I will be covering this topic as well at PostgreSQL Sessions in Paris on October 4th and Postgres Open in Chicago on September 19th. Hopefully I will be speaking about this at the next PostgreSQL European Conference in Prague, where my proposed talk on Disaster recovery with Barman is in the reserve list (if not, I will be there anyway, ready to share ideas about Barman and its future with you).

You can download Barman 1.0 from SourceForge.net, or you can get more information by visiting the website or joining our new #barman IRC channel on freenode. Ciao!

Tags: backup, Barman, disaster recovery, postgres, PostgreSQL, WAL archive
Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on LinkedIn
4 replies
  1. pg_lizard
    pg_lizard says:
    August 23, 2012 at 10:40 pm

    Nice tool that fits my needs perfectly. Thanks for this nice article as well as the wonderful PDF documentation(available on barman website). I have just two concerns:

    -> What if I started the backup and an hour later decided to kill it. Do I need to cleanup the “bad” backup myself or barman can handle it.

    -> “INCOMING_WAL_DIRECTORY”- can the location of this be configured. Currently my server
    archives its WALS to a specific location on a backup machine. Can I have barman
    use this location.

    Thanks to the entire development team behind this amazing tool!

    Reply
    • gabriele.bartolini
      gabriele.bartolini says:
      August 24, 2012 at 6:39 am

      Hello,

      first I want to thank for the nice words which I happily extend to the rest of the team.

      As far as your question about canceling a backup is concerned, by default Barman does not delete an incomplete backup. However, it is labelled as ‘FAILED‘. You can then remove it at a later time and get rid of the partial files. Bear in mind that we’ve just committed in the HEAD of the project in the GIT repository the support for hook scripts to be invoked before and after a backup. You can write your own script that after a backup check if everything went fine and if not, deletes the backup.

      In the future we might even add a configuration option for this.

      As far as the incoming_wals_directory is regarded, you are completely free to change it. You can do it on a per-server basis. We designed Barman to follow the “convention over configuration” paradigm, which does not force you to specify every single parameter, while leaving you the possibility to do so. You just need to set that option to that directory in your server configuration section.

      Cheers,
      Gabriele

      Reply
  2. Fernando
    Fernando says:
    October 12, 2018 at 2:56 pm

    Many thanks for the article of great use, but I have the following problem and I do not know if barman does it: when barman generates a backup in my home server it leaves a baseline, but it does not clean the old wal and accumulate until complete the disk, the point is if barman cleans the old wal from my main server? or should I manually apply pg_archivecleanup? Thank you

    Reply

Trackbacks & Pingbacks

  1. Retention of backups with Barman | 2ndQuadrant says:
    April 15, 2013 at 1:47 pm

    […] the same topic, you may wish to read one of my previous blog posts about the backup catalogue and the WAL archive in […]

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

Get in touch with us!

Recent Posts

  • Random Data December 3, 2020
  • Webinar: COMMIT Without Fear – The Beauty of CAMO [Follow Up] November 13, 2020
  • Full-text search since PostgreSQL 8.3 November 5, 2020
  • Random numbers November 3, 2020
  • Webinar: Best Practices for Bulk Data Loading in PostgreSQL [Follow Up] November 2, 2020

Featured External Blogs

Tomas Vondra's Blog

Our Bloggers

  • Simon Riggs
  • Alvaro Herrera
  • Andrew Dunstan
  • Craig Ringer
  • Francesco Canovai
  • Gabriele Bartolini
  • Giulio Calacoci
  • Ian Barwick
  • Marco Nenciarini
  • Mark Wong
  • Pavan Deolasee
  • Petr Jelinek
  • Shaun Thomas
  • Tomas Vondra
  • Umair Shahid

PostgreSQL Cloud

2QLovesPG 2UDA 9.6 backup Barman BDR Business Continuity community conference database DBA development devops disaster recovery greenplum Hot Standby JSON JSONB logical replication monitoring OmniDB open source Orange performance PG12 pgbarman pglogical PG Phriday postgres Postgres-BDR postgres-xl PostgreSQL PostgreSQL 9.6 PostgreSQL10 PostgreSQL11 PostgreSQL 11 PostgreSQL 11 New Features postgresql repmgr Recovery replication security sql wal webinar webinars

Support & Services

24/7 Production Support

Developer Support

Remote DBA for PostgreSQL

PostgreSQL Database Monitoring

PostgreSQL Health Check

PostgreSQL Performance Tuning

Database Security Audit

Upgrade PostgreSQL

PostgreSQL Migration Assessment

Migrate from Oracle to PostgreSQL

Products

HA Postgres Clusters

Postgres-BDR®

2ndQPostgres

pglogical

repmgr

Barman

Postgres Cloud Manager

SQL Firewall

Postgres-XL

OmniDB

Postgres Installer

2UDA

Postgres Learning Center

Introducing Postgres

Blog

Webinars

Books

Videos

Training

Case Studies

Events

About Us

About 2ndQuadrant

What does 2ndQuadrant Mean?

News

Careers 

Team Profile

© 2ndQuadrant Ltd. All rights reserved. | Privacy Policy
  • Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Mail
How to test Greenplum Community Edition 4.2.1 on VirtualBox Submit your talk for PGDay.IT
Scroll to top
×