{"id":108,"date":"2026-07-12T00:00:00","date_gmt":"2026-07-12T00:00:00","guid":{"rendered":"http:\/\/\/var\/www\/html\/?p=108"},"modified":"2026-07-21T01:02:35","modified_gmt":"2026-07-21T01:02:35","slug":"postgresql-pglog-disk-space-exhausted-database-lock-lockfile-on-debian-12-bookworm","status":"publish","type":"post","link":"https:\/\/butitworkedlocal.com\/postgresql-pglog-disk-space-exhausted-database-lock-lockfile-on-debian-12-bookworm\/","title":{"rendered":"Resolving PostgreSQL pg_log Disk Space Exhaustion & Database Lock on Debian 12 Bookworm"},"content":{"rendered":"\n

Fix PostgreSQL database locks and service failures on Debian 12 caused by full pg_log disk space. Learn to clear logs, free space, and prevent recurrence.<\/strong><\/p>\n\n\n

When your PostgreSQL database suddenly becomes unresponsive, applications fail to connect, or the entire web application ecosystem grinds to a halt, one of the most insidious culprits can be a full disk. Specifically, an overgrowth of PostgreSQL's transaction logs or pg_log<\/code> files can consume all available space, leading to critical database failures, service unresponsiveness, and "lockfile" errors as PostgreSQL struggles to write essential operational data. This guide provides a highly technical, step-by-step resolution for this common issue on Debian 12 (Bookworm) systems.<\/p>\n

Symptom & Error Signature<\/h3>\n

Users typically experience one or more of the following symptoms:<\/p>\n

    \n
  • Web applications reporting database connection errors (e.g., "could not connect to server: Connection refused", "FATAL: remaining connection slots are reserved for non-replication superuser connections").<\/li>\n
  • The PostgreSQL service (often postgresql@15-main.service<\/code> or similar) is reported as failed<\/code> or inactive<\/code> by systemctl<\/code>.<\/li>\n
  • Attempts to start or restart PostgreSQL fail immediately.<\/li>\n
  • System logs (journalctl<\/code>) show errors related to disk I\/O or inability to write.<\/li>\n<\/ul>\n

    Here are typical error messages you might encounter:<\/p>\n

    From journalctl -u postgresql<\/code> (or journalctl -u postgresql@15-main.service<\/code> for specific versions):<\/strong><\/p>\n

    systemd[1]: postgresql@15-main.service: Start request repeated too quickly.\nsystemd[1]: postgresql@15-main.service: Failed with result 'exit-code'.\nsystemd[1]: Failed to start PostgreSQL Cluster 15-main.\n...\npostgresql[12345]: 2024-06-25 10:30:00.123 UTC [12345] LOG: could not write lock file "\/var\/run\/postgresql\/.s.PGSQL.5432.lock": No space left on device\npostgresql[12345]: 2024-06-25 10:30:00.123 UTC [12345] FATAL: could not create lock file "\/var\/run\/postgresql\/.s.PGSQL.5432.lock": No space left on device\npostgresql[12345]: 2024-06-25 10:30:00.123 UTC [12345] LOG: database system is shut down\n<\/code><\/pre>\n
    From dmesg<\/code> (kernel messages, indicating disk I\/O issues):<\/strong><\/p>\n
    [123456.789012] EXT4-fs (sda1): write access unavailable, fs is full\n[123456.789013] EXT4-fs (sda1): Remounting filesystem read-only\n<\/code><\/pre>\n
    From df -h<\/code> (showing disk usage):<\/strong><\/p>\n
    Filesystem      Size  Used Avail Use% Mounted on\n\/dev\/sda1        50G   50G     0 100% \/\nudev             10M    0   10M   0% \/dev\ntmpfs           7.8G  1.2M  7.8G   1% \/run\n<\/code><\/pre>\n
    Root Cause Analysis<\/h3>\n
    The root cause of this issue is typically a filesystem reaching 100% capacity on the partition where PostgreSQL stores its data and\/or log files. This commonly happens on the root partition (\/<\/code>) if \/var<\/code> is not a separate mount, or on a dedicated \/var<\/code> partition.<\/p>\n
    Here's a breakdown of why this leads to the observed symptoms:<\/p>\n
      \n
    1. Excessive Log Growth:<\/strong> PostgreSQL, by default, writes detailed logs to files within a directory like \/var\/log\/postgresql\/<\/code> or within the data directory itself (\/var\/lib\/postgresql\/<version>\/main\/log\/<\/code>). Without proper log rotation (e.g., via logrotate<\/code>), these log files can grow indefinitely, consuming gigabytes or even terabytes of disk space over time.<\/li>\n
    2. Lack of Free Space:<\/strong> Once the filesystem reaches 100% utilization, the operating system can no longer write any new data to that disk.<\/li>\n
    3. PostgreSQL's Inability to Operate:<\/strong> PostgreSQL requires free disk space for several critical operations:
        \n
      • Writing Log Files:<\/strong> It cannot log new events, leading to internal errors.<\/li>\n
      • Writing Transaction Logs (WAL):<\/strong> Crucial for data integrity and recovery. While WAL segments are usually pre-allocated, operational temporary files and checkpoints might require space.<\/li>\n
      • Creating Lock Files:<\/strong> When PostgreSQL starts, it creates a lock file (e.g., \/var\/run\/postgresql\/.s.PGSQL.5432.lock<\/code>) to prevent multiple instances from running on the same port. If it cannot create this file, it fails to start.<\/li>\n
      • Temporary Files and Caching:<\/strong> Queries that require large sorts or joins might create temporary files on disk. If these cannot be written, queries will fail.<\/li>\n<\/ul>\n<\/li>\n
      • Service Failure:<\/strong> The inability to write lock files or perform other critical disk operations prevents the PostgreSQL server from starting or causes it to shut down gracefully (or crash) to prevent data corruption. systemd<\/code> will then report the service as failed.<\/li>\n<\/ol>\n
        Step-by-Step Resolution<\/h3>\n
        Follow these steps carefully to recover your PostgreSQL service and prevent future occurrences.<\/p>\n
        1. Verify Disk Space and Identify Culprit<\/h4>\n
        First, confirm that your disk is indeed full and identify which directory is consuming the most space.<\/p>\n
          \n
        1. Check overall disk usage:<\/strong><\/p>\n
          df -h\n<\/code><\/pre>\n
          Look for a partition (often \/<\/code> or \/var<\/code>) showing 100% Use%<\/code>.<\/p>\n<\/li>\n
        2. Identify large directories:<\/strong>\nNavigate to the suspected full partition and use du<\/code> to find the largest directories. Start broad and narrow down.<\/p>\n
          sudo du -h \/var\/log | sort -rh | head -n 10\nsudo du -h \/var\/lib\/postgresql | sort -rh | head -n 10\n<\/code><\/pre>\n
          You're likely to see \/var\/log\/postgresql\/<\/code> or a subdirectory within \/var\/lib\/postgresql\/<\/code> consuming significant space. For example:<\/p>\n
          50G     \/var\/log\/postgresql\n48G     \/var\/log\/postgresql\/postgresql-15-main.log.1\n<\/code><\/pre>\n<\/li>\n<\/ol>\n
          2. Stop PostgreSQL Service<\/h4>\n
          Before manipulating PostgreSQL's log files or data directory, it is critical<\/strong> to stop the service to prevent data corruption.<\/p>\n
          Do NOT proceed to delete files if PostgreSQL is still running, especially not within its data directory (\/var\/lib\/postgresql<\/code>). This can lead to irreversible data loss or corruption.<\/p>\n<\/blockquote>\n
          sudo systemctl stop postgresql\n# Or for a specific version:\nsudo systemctl stop postgresql@15-main.service\n<\/code><\/pre>\n
          Verify the service is stopped:<\/p>\n
          sudo systemctl status postgresql\n<\/code><\/pre>\n
          It should show Active: inactive (dead)<\/code>.<\/p>\n
          3. Clear Excessive pg_log<\/code> Files<\/h4>\n
          Once the service is stopped, you can safely remove old log files.<\/p>\n
            \n
          1. Navigate to the PostgreSQL log directory:<\/strong>\nFor Debian, this is typically \/var\/log\/postgresql\/<\/code>.<\/p>\n
            cd \/var\/log\/postgresql\/\n<\/code><\/pre>\n<\/li>\n
          2. List files and confirm their size:<\/strong><\/p>\n
            ls -lh\n<\/code><\/pre>\n
            This will show you the massive log files, usually named postgresql-<version>-main.log<\/code> or similar, possibly with suffixes like .1<\/code>, .gz<\/code>, etc.<\/p>\n<\/li>\n
          3. Delete old log files:<\/strong>\nStart by deleting the oldest<\/em> and largest<\/em> log files to free up space. You might need to delete iteratively until you have enough free space to restart the database.<\/p>\n
            Do not delete the current<\/em> log file (postgresql-<version>-main.log<\/code>) or files still being written to (though this is less likely if the service is stopped). Focus on .1<\/code>, .2<\/code>, .gz<\/code> files.<\/p>\n
            Be cautious with rm -rf<\/code>. Always verify the directory and files before execution.<\/p>\n<\/blockquote>\n
            To delete log files older than, say, 7 days:<\/p>\n
            sudo find . -name "postgresql-*-main.log.*" -mtime +7 -exec rm {} ;\n<\/code><\/pre>\n
            To delete all but the most recent 2-3 log files (be careful with this, adjust head -n -3<\/code> to suit):<\/p>\n
            # List files sorted by modification time (oldest first)\nls -t postgresql-*-main.log.* | tail -n +4 | xargs sudo rm -v\n# The above command keeps the 3 newest logs. Adjust `+4` to keep `N-1` logs.\n# E.g., `tail -n +2` keeps 1 newest log.\n# To delete all but the current one and the most recent rotated one:\n# sudo rm postgresql-*-main.log.2* postgresql-*-main.log.3*\n<\/code><\/pre>\n
            If you're desperate for space and can afford to lose older logs for recovery:<\/p>\n
            sudo rm postgresql-*-main.log.*\n<\/code><\/pre>\n<\/li>\n
          4. Verify free space:<\/strong><\/p>\n
            df -h\n<\/code><\/pre>\n
            You should now see some available space. Aim for at least 1-2GB free to allow the database to start and operate normally.<\/p>\n<\/li>\n<\/ol>\n
            4. Restart PostgreSQL Service<\/h4>\n
            With disk space freed, attempt to restart the database service.<\/p>\n
            sudo systemctl start postgresql\n# Or for a specific version:\nsudo systemctl start postgresql@15-main.service\n<\/code><\/pre>\n
            5. Verify Database Operation<\/h4>\n
            Check the status of the service and confirm your applications can connect.<\/p>\n
              \n
            1. Check service status:<\/strong><\/p>\n
              sudo systemctl status postgresql\n<\/code><\/pre>\n
              It should now show Active: active (running)<\/code>.<\/p>\n<\/li>\n
            2. Check logs for errors:<\/strong><\/p>\n
              sudo journalctl -u postgresql -f\n<\/code><\/pre>\n
              Look for LOG: database system is ready to accept connections<\/code>.<\/p>\n<\/li>\n
            3. Test application connectivity:<\/strong>\nAccess your web application or connect via psql<\/code>:<\/p>\n
              sudo -u postgres psql\n<\/code><\/pre>\n
              If you get a psql<\/code> prompt, the database is running. Type q<\/code> to exit.<\/p>\n<\/li>\n<\/ol>\n
              6. Implement Log Rotation (Long-Term Fix)<\/h4>\n
              To prevent this issue from recurring, configure logrotate<\/code> for PostgreSQL. Debian's postgresql-common<\/code> package usually sets up a basic logrotate<\/code> configuration, but it might be overridden or misconfigured.<\/p>\n
                \n
              1. Inspect existing logrotate configuration:<\/strong>\nCheck \/etc\/logrotate.d\/postgresql-common<\/code> or \/etc\/logrotate.d\/postgresql<\/code>.<\/p>\n
                sudo cat \/etc\/logrotate.d\/postgresql-common\n<\/code><\/pre>\n
                A typical configuration might look like this:<\/p>\n
                \/var\/log\/postgresql\/*.log {\n        daily\n        missingok\n        rotate 7\n        compress\n        delaycompress\n        notifempty\n        create 0640 postgres adm\n        su postgres adm\n        postrotate\n                # Find the PIDs of running postgresql clusters and send them a SIGHUP\n                if [ -e \/etc\/postgresql\/15\/main\/postgresql.conf ] ; then\n                        pg_ctlcluster 15 main reload > \/dev\/null || true\n                fi\n        endscript\n}\n<\/code><\/pre>\n<\/li>\n
              2. Adjust logrotate<\/code> settings if necessary:<\/strong><\/p>\n
                  \n
                • rotate 7<\/code>: Keeps 7 rotated logs. Adjust this number (e.g., rotate 14<\/code> for two weeks) based on your disk space and retention needs.<\/li>\n
                • daily<\/code>, weekly<\/code>, monthly<\/code>: How often logs are rotated. daily<\/code> is generally good for busy systems.<\/li>\n
                • compress<\/code>: Compresses old log files. Highly recommended.<\/li>\n
                • size 100M<\/code>: You can add size <bytes><\/code> (e.g., size 100M<\/code>) to force rotation when a log file reaches a certain size, regardless of the daily<\/code>\/weekly<\/code> setting. This is very effective for high-volume logs.<\/li>\n<\/ul>\n
                  If you make changes, ensure they are compatible with existing logrotate options. A common issue is a missing postrotate<\/code> script to signal PostgreSQL to reopen log files. Without it, PostgreSQL might continue writing to the old (renamed) log file. The pg_ctlcluster ... reload<\/code> command correctly handles this.<\/p>\n<\/blockquote>\n<\/li>\n
                • Manually test logrotate<\/code>:<\/strong><\/p>\n
                  sudo logrotate -f \/etc\/logrotate.d\/postgresql-common\n<\/code><\/pre>\n
                  This forces a rotation immediately. Check \/var\/log\/postgresql\/<\/code> afterwards to confirm new log files are created and old ones are rotated\/compressed.<\/p>\n<\/li>\n<\/ol>\n
                  7. Configure PostgreSQL Logging (Proactive)<\/h4>\n
                  Fine-tuning PostgreSQL's logging behavior in postgresql.conf<\/code> can also help manage disk space.<\/p>\n
                    \n
                  1. Edit postgresql.conf<\/code>:<\/strong><\/p>\n
                    sudo nano \/etc\/postgresql\/15\/main\/postgresql.conf\n<\/code><\/pre>\n<\/li>\n
                  2. Review and adjust these parameters:<\/strong><\/p>\n
                      \n
                    • log_destination<\/code>: Set to stderr<\/code> or csvlog<\/code> (if you want structured logs). stderr<\/code> is standard for syslog\/journal.<\/li>\n
                    • logging_collector = on<\/code>: (Default) This sends log messages to files managed by PostgreSQL itself. If off<\/code>, logs go to syslog, which then needs syslog's own rotation. For \/var\/log\/postgresql<\/code> management, keep it on<\/code>.<\/li>\n
                    • log_directory = 'pg_log'<\/code>: (Relative to data directory) or \/var\/log\/postgresql<\/code> (absolute path).<\/li>\n
                    • log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'<\/code>: This creates unique log files per start time, which logrotate<\/code> can then manage.<\/li>\n
                    • log_rotation_age = 1d<\/code>: Rotates logs daily. This works in conjunction with logrotate<\/code>.<\/li>\n
                    • log_rotation_size = 0<\/code>: Disables size-based rotation by PostgreSQL itself<\/em> if logrotate<\/code> is handling it. Set a value (e.g., 100MB<\/code>) if you want PostgreSQL to rotate before<\/em> logrotate. It's often better to let logrotate<\/code> handle it consistently.<\/li>\n
                    • log_min_duration_statement = 1000<\/code>: Logs all statements taking longer than 1 second (1000ms). Adjust this value to 0<\/code> to log all statements (VERY VERBOSE, use with caution on production), or increase it to 5000<\/code> (5 seconds) to reduce log volume.<\/li>\n
                    • log_statement = 'none'<\/code>: Logs no statements. Can be ddl<\/code>, mod<\/code>, all<\/code>. none<\/code> is recommended for most production environments unless debugging.<\/li>\n
                    • log_checkpoints = on<\/code>: Logs each checkpoint. Useful for performance monitoring.<\/li>\n
                    • log_connections = on<\/code>: Logs successful client connections.<\/li>\n
                    • log_disconnections = on<\/code>: Logs end of client sessions.<\/li>\n<\/ul>\n
                      Setting log_rotation_age<\/code> or log_rotation_size<\/code> directly in postgresql.conf<\/code> provides a fallback or additional rotation layer. However, logrotate<\/code> is generally preferred for system-wide log management. Ensure your postgresql.conf<\/code> settings don't conflict or create redundant files. If logrotate<\/code> is robustly configured, you can set log_rotation_age = 0<\/code> and log_rotation_size = 0<\/code> in postgresql.conf<\/code> to let logrotate<\/code> handle it exclusively.<\/p>\n<\/blockquote>\n<\/li>\n
                    • Restart PostgreSQL after postgresql.conf<\/code> changes:<\/strong><\/p>\n
                      sudo systemctl restart postgresql\n<\/code><\/pre>\n<\/li>\n<\/ol>\n
                      8. Consider Dedicated Disk\/Partition (Advanced)<\/h4>\n
                      For high-traffic production databases, it's often prudent to separate PostgreSQL's data and log directories onto dedicated partitions or even separate physical disks. This isolates I\/O and prevents a full log partition from impacting the entire system or vice-versa.<\/p>\n
                        \n
                      1. Mount a new disk\/partition:<\/strong>\nCreate a new filesystem and mount it, e.g., to \/mnt\/pg_data<\/code> or \/mnt\/pg_logs<\/code>.\nEdit \/etc\/fstab<\/code> to ensure it mounts automatically on boot.<\/p>\n<\/li>\n
                      2. Move PostgreSQL data or logs:<\/strong><\/p>\n
                          \n
                        • Moving logs:<\/strong> Change log_directory<\/code> in postgresql.conf<\/code> to point to the new location (e.g., \/mnt\/pg_logs<\/code>). Remember to create the directory and set correct permissions (chown postgres:postgres \/mnt\/pg_logs<\/code>).<\/li>\n
                        • Moving data directory:<\/strong> This is more involved. You would stop PostgreSQL, copy \/var\/lib\/postgresql\/<\/code> to the new location (e.g., \/mnt\/pg_data<\/code>), update data_directory<\/code> in postgresql.conf<\/code>, and potentially update the systemd service file or pg_ctlcluster<\/code> configuration.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
                          This comprehensive approach will not only resolve the immediate disk space and lockfile issues but also establish robust logging and maintenance practices to prevent recurrence.<\/p>\n","protected":false},"excerpt":{"rendered":"
                          Fix PostgreSQL database locks and service failures on Debian 12 caused by full pg_log disk space. Learn to clear logs, free space, and prevent recurrence.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[187],"tags":[100,282,76,40,124,281,150,56,6],"class_list":["post-108","post","type-post","status-publish","format-standard","hentry","category-database","tag-bookworm","tag-database-lock","tag-debian-12","tag-devops","tag-disk-space","tag-pg_log","tag-postgresql","tag-systemd","tag-troubleshooting"],"_links":{"self":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/comments?post=108"}],"version-history":[{"count":2,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/108\/revisions"}],"predecessor-version":[{"id":356,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/108\/revisions\/356"}],"wp:attachment":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/media?parent=108"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/categories?post=108"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/tags?post=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}