2ndQuadrant is now part of EDB

Bringing together some of the world's top PostgreSQL experts.

2ndQuadrant | PostgreSQL
Mission Critical Databases
  • Contact us
  • EN
    • FR
    • IT
    • ES
    • DE
    • PT
  • Support & Services
    • Support
      • 24/7 PostgreSQL Support
      • Developer Support
    • DBA Services
      • Remote DBA
      • Database Monitoring
    • Consulting Services
      • Health Check
      • Performance Tuning
      • Database Security Audit
      • PostgreSQL Upgrade
    • Migration Services
      • Migrate to PostgreSQL
      • Migration Assessment
  • Products
    • Postgres-BDR ®
    • PostgreSQL High Availability
    • Kubernetes Operators for BDR & PostgreSQL
    • Managed PostgreSQL in the Cloud
    • Installers
      • Postgres Installer
      • 2UDA
    • 2ndQPostgres
    • pglogical
    • Barman
    • repmgr
    • OmniDB
    • SQL Firewall
    • Postgres-XL
  • Downloads
    • Installers
      • Postgres Installer
      • 2UDA – Unified Data Analytics
    • Whitepapers
      • Business Case for PostgreSQL Support
      • Security Best Practices for PostgreSQL
    • Case Studies
      • Performance Tuning
        • BenchPrep
        • tastyworks
      • Distributed Clusters
        • ClickUp
        • European Space Agency (ESA)
        • Telefónica del Sur
        • Animal Logic
      • Database Administration
        • Agilis Systems
      • Professional Training
        • Met Office
        • London & Partners
      • Database Upgrades
        • Alfred Wegener Institute (AWI)
      • Database Migration
        • International Game Technology (IGT)
        • Healthcare Software Solutions (HSS)
        • Navionics
  • Postgres Learning Center
    • Webinars
      • Upcoming Webinars
      • Webinar Library
    • Whitepapers
      • Business Case for PostgreSQL Support
      • Security Best Practices for PostgreSQL
    • Blog
    • Training
      • Course Catalogue
    • Case Studies
      • Performance Tuning
        • BenchPrep
        • tastyworks
      • Distributed Clusters
        • ClickUp
        • European Space Agency (ESA)
        • Telefónica del Sur
        • Animal Logic
      • Database Administration
        • Agilis Systems
      • Professional Training
        • Met Office
        • London & Partners
      • Database Upgrades
        • Alfred Wegener Institute (AWI)
      • Database Migration
        • International Game Technology (IGT)
        • Healthcare Software Solutions (HSS)
        • Navionics
    • Books
      • PostgreSQL 11 Administration Cookbook
      • PostgreSQL 10 Administration Cookbook
      • PostgreSQL High Availability Cookbook – 2nd Edition
      • PostgreSQL 9 Administration Cookbook – 3rd Edition
      • PostgreSQL Server Programming Cookbook – 2nd Edition
      • PostgreSQL 9 Cookbook – Chinese Edition
    • Videos
    • Events
    • PostgreSQL
      • PostgreSQL – History
      • Who uses PostgreSQL?
      • PostgreSQL FAQ
      • PostgreSQL vs MySQL
      • The Business Case for PostgreSQL
      • Security Information
      • Documentation
  • About Us
    • About 2ndQuadrant
    • 2ndQuadrant’s Passion for PostgreSQL
    • News
    • Careers
    • Team Profile
  • Blog
  • Menu Menu
You are here: Home1 / Blog2 / Craig's PlanetPostgreSQL3 / Don’t set fsync=off if you want to keep your data
craig.ringer

Don’t set fsync=off if you want to keep your data

April 28, 2016/3 Comments/in Craig's PlanetPostgreSQL /by craig.ringer

There are a lot of amazing features coming in PostgreSQL 9.6, but I’m personally very happy about a really small, simple one that helps close a long-standing user foot-gun.

commit a31212b429cd3397fb3147b1a584ae33224454a6
Author: Robert Haas 
Date:   Wed Apr 27 13:46:26 2016 -0400

    Change postgresql.conf.sample to say that fsync=off will corrupt data.

    Discussion: [email protected]
    
    Per a suggestion from Craig Ringer.  This wording from Tom Lane,
    following discussion.

There’s a bit of terrible advice floating around that turning fsync=off will make PostgreSQL run faster. Which is true, as far as it goes, but neglects that little risk massive data corruption on crash part. So users get bitten. I’ve tried to scrub as much of that advice from the Internet as I can or get it qualified with warnings, but I still see people advised to do it on random forums sometimes.

The docs do a good job of explaining that setting fsync=off is a bad idea, but the sample config file made it seem pretty innocuous:

#fsync = on                             # turns forced synchronization on or off

so users keep turning it off. Then teams like 2ndQuadrant land up doing expensive forensic data recovery on their database clusters once they crash weeks, months or years later and experience severe corruption… or notice the corruption they experienced a while ago. Or users just write off their data and start again because it’s all too hard and expensive.

To make turning fsync=off a little less attractive the sample now reads:

#fsync = on                            # flush data to disk for crash safety
                                       # (turning this off can cause
                                       # unrecoverable disk corruption)

It won’t stop someone using ALTER SYSTEM SET without realising, but there’s only so much you can do.

I’m really happy about this. It’s nice to knock off such minor improvements that have a disproportionate impact on usability and UX.

If you are ever tempted to set fsync=off, pretend it’s called eat_my_data_if_you_feel_like_it=on and see if you still want to set it. synchronous_commit=off is probably a better choice. Read the manual.

Tags: corruption, crash safety, fsync, PostgreSQL
Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on LinkedIn
3 replies
  1. Dianne Skoll
    Dianne Skoll says:
    April 29, 2016 at 5:01 pm

    I have one use-case for fsync=off. If I am restoring a dump into a brand-new PostgreSQL instance, I do the restore with fsync=off. Then I stop PostgreSQL; run the Unix command “sync” and turn fsync back on. Then I restart PostgreSQL.

    Other than that… yeah, not valid reason for turning it off unless you don’t care about your data.

    Reply
    • craig.ringer
      craig.ringer says:
      May 2, 2016 at 6:24 am

      That’s very reasonable, and one of the non-test/development reasons the “off” setting exists at all.

      It’s only safe if the whole instance is new; if you turn fsync off to restore a DB to an instance with other running DBs you’ll still risk a giant mess on crash since the DB you’re doing the data load into shares xlogs, clog, shared catalog tables etc with the other DBs on the instance. You probably know that, but I’m being explicit for other readers.

      Another reasonable reason to turn fsync off can be if you intend to rely entirely on synchronous replication for durability. In particular, if you’re running on something like an AWS instance using local instance storage not EBS, the storage isn’t durable anyway. On crash it might just go away permanently and unrecoverably. It certainly will if you stop and start the instance. In this case fsync doesn’t do you any good-the data’s gone, so crash safety is irrelevant. However, you must make sure that if the instance crashes but the data is *not* lost you still throw it away, since it could be corrupted and if you start running with it you might then sync corrupt data to your replicas. Fencing and STONITH is important.

      For most other purposes it’s better to just set synchronous_commit = off if you don’t mind the risk of losing recent data but don’t want to risk possible corruption.

      Reply
  2. Daniël van Eeden
    Daniël van Eeden says:
    May 2, 2016 at 8:14 am

    Would it be possible to let ALTER SYSTEM SET return a warning in this case?

    I can also recommend this for those who like fsync=off: https://www.flamingspork.com/projects/libeatmydata/

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

Get in touch with us!

Recent Posts

  • Random Data December 3, 2020
  • Webinar: COMMIT Without Fear – The Beauty of CAMO [Follow Up] November 13, 2020
  • Full-text search since PostgreSQL 8.3 November 5, 2020
  • Random numbers November 3, 2020
  • Webinar: Best Practices for Bulk Data Loading in PostgreSQL [Follow Up] November 2, 2020

Featured External Blogs

Tomas Vondra's Blog

Our Bloggers

  • Simon Riggs
  • Alvaro Herrera
  • Andrew Dunstan
  • Craig Ringer
  • Francesco Canovai
  • Gabriele Bartolini
  • Giulio Calacoci
  • Ian Barwick
  • Marco Nenciarini
  • Mark Wong
  • Pavan Deolasee
  • Petr Jelinek
  • Shaun Thomas
  • Tomas Vondra
  • Umair Shahid

PostgreSQL Cloud

2QLovesPG 2UDA 9.6 backup Barman BDR Business Continuity community conference database DBA development devops disaster recovery greenplum Hot Standby JSON JSONB logical replication monitoring OmniDB open source Orange performance PG12 pgbarman pglogical PG Phriday postgres Postgres-BDR postgres-xl PostgreSQL PostgreSQL 9.6 PostgreSQL10 PostgreSQL11 PostgreSQL 11 PostgreSQL 11 New Features postgresql repmgr Recovery replication security sql wal webinar webinars

Support & Services

24/7 Production Support

Developer Support

Remote DBA for PostgreSQL

PostgreSQL Database Monitoring

PostgreSQL Health Check

PostgreSQL Performance Tuning

Database Security Audit

Upgrade PostgreSQL

PostgreSQL Migration Assessment

Migrate from Oracle to PostgreSQL

Products

HA Postgres Clusters

Postgres-BDR®

2ndQPostgres

pglogical

repmgr

Barman

Postgres Cloud Manager

SQL Firewall

Postgres-XL

OmniDB

Postgres Installer

2UDA

Postgres Learning Center

Introducing Postgres

Blog

Webinars

Books

Videos

Training

Case Studies

Events

About Us

About 2ndQuadrant

What does 2ndQuadrant Mean?

News

Careers 

Team Profile

© 2ndQuadrant Ltd. All rights reserved. | Privacy Policy
  • Twitter
  • LinkedIn
  • Facebook
  • Youtube
  • Mail
Column Store Plans PGLogical 1.1 released with sequence support and more
Scroll to top
×