Resolve critical PostgreSQL startup failures on Ubuntu 20.04 due to misconfigured shared_buffers exceeding system or kernel shared memory limits.
When managing high-performance PostgreSQL databases, optimizing memory usage through parameters like shared_buffers is crucial. However, misconfiguring this setting can lead to a critical PostgreSQL service crash upon startup, presenting as a "Cannot allocate memory" error specifically related to shared memory segments. This guide will walk you through diagnosing and resolving this issue on Ubuntu 20.04 LTS systems.
Symptom & Error Signature
The most common symptom is that your PostgreSQL service fails to start, leading to applications being unable to connect to the database. When checking the service status or logs, you'll encounter FATAL errors indicating a failure to allocate shared memory.
You can observe this by checking your PostgreSQL logs or journalctl:
sudo systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2023-10-27 10:00:05 UTC; 5s ago
Process: 12345 ExecStart=/usr/lib/postgresql/12/bin/pg_ctlcluster --skip-systemctl-redirect 12 main start (code=exited, status=1/FAILURE)
Oct 27 10:00:05 webserver systemd[1]: Starting PostgreSQL RDBMS…
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: FATAL: could not create shared memory segment: Cannot allocate memory
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: DETAIL: Failed system call was shmget(key=5432001, size=16777216000, 0x1FF).
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits.
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: The PostgreSQL process would require 16777216000 bytes of shared memory.
Oct 27 10:00:05 webserver pgctlcluster[12345]: Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing sharedbuffers or max_connections), or check system limitations.
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: LOG: database system is shut down
Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Failed with result 'exit-code'.
Oct 27 10:00:05 webserver systemd[1]: Failed to start PostgreSQL RDBMS.
`
Or by examining the PostgreSQL log file directly (e.g., for PostgreSQL 12):
sudo tail -n 50 /var/log/postgresql/postgresql-12-main.log
2023-10-27 10:00:05.123 UTC [12345] LOG: starting PostgreSQL 12.16 (Ubuntu 12.16-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
2023-10-27 10:00:05.123 UTC [12345] LOG: listening on IPv4 address "127.0.0.1", port 5432
2023-10-27 10:00:05.123 UTC [12345] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-10-27 10:00:05.124 UTC [12345] FATAL: could not create shared memory segment: Cannot allocate memory
2023-10-27 10:00:05.124 UTC [12345] DETAIL: Failed system call was shmget(key=5432001, size=16777216000, 0x1FF).
2023-10-27 10:00:05.124 UTC [12345] HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits. The PostgreSQL process would require 16777216000 bytes of shared memory. Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing shared_buffers or max_connections), or check system limitations.
2023-10-27 10:00:05.124 UTC [12345] LOG: database system is shut down
Root Cause Analysis
The FATAL: could not create shared memory segment: Cannot allocate memory error, specifically with the shmget system call failure, indicates that PostgreSQL attempted to allocate a shared memory segment that either exceeded the physical memory available on the system or, more commonly, violated the Linux kernel's System V IPC (Interprocess Communication) shared memory limits.
PostgreSQL uses a significant portion of its memory for shared_buffers, which acts as a cache for database pages. This memory is allocated from the kernel's shared memory pool. The Linux kernel has two primary parameters that govern System V shared memory:
-
kernel.shmmax: This parameter defines the maximum size (in bytes) of a single shared memory segment that the kernel will allow. If PostgreSQL'sshared_buffers(which is typically one large shared memory segment) configured value exceedskernel.shmmax, the allocation will fail. -
kernel.shmall: This parameter defines the total amount of shared memory (in pages) that the system can use at once. If the sum of all shared memory segments requested by all processes (including PostgreSQL) exceedskernel.shmallmultiplied by the system'sPAGESIZE, allocation will fail. On x86-64 systems,PAGESIZEis typically 4096 bytes (4KB).
The crash typically occurs when:
* You've set shared_buffers in postgresql.conf to a value too high for your system's physical RAM.
* The shared_buffers value, even if reasonable for RAM, exceeds the current kernel.shmmax limit.
* The overall shared memory footprint requested (primarily shared_buffers) exceeds the total allowed by kernel.shmall.
Step-by-Step Resolution
Follow these steps to diagnose and resolve the PostgreSQL shared buffers memory allocation crash.
1. Verify PostgreSQL Service Status and Examine Logs
First, confirm that the PostgreSQL service is indeed failing and check the logs for the exact error message.
# Check service status
View recent PostgreSQL specific logs (PostgreSQL 12) sudo journalctl -u postgresql –no-pager | grep -i "fatal"
Alternatively, check the main PostgreSQL log file
sudo tail -n 100 /var/log/postgresql/postgresql-12-main.log
`
Confirm the presence of the FATAL: could not create shared memory segment: Cannot allocate memory error. Note the size mentioned in the DETAIL line, as this is the amount PostgreSQL tried to allocate.
2. Identify Current shared_buffers Configuration
Locate your postgresql.conf file and check the configured shared_buffers value. For PostgreSQL 12 on Ubuntu 20.04, it's typically located at /etc/postgresql/12/main/postgresql.conf.
# Display the shared_buffers line from your config
grep -E '^shared_buffers' /etc/postgresql/12/main/postgresql.conf
3. Determine Current Kernel Shared Memory Limits
Check the current values of kernel.shmmax and kernel.shmall on your system.
# View current kernel shared memory maximum segment size
View current kernel shared memory total pages sudo sysctl kernel.shmall
Get system page size (usually 4096 bytes or 4KB)
getconf PAGESIZE
`
[!IMPORTANT] The
HINTin the PostgreSQL error message will often provide the exact byte size that PostgreSQL attempted to allocate. This is your target value forkernel.shmmax. Forkernel.shmall, you'll need to divide that byte size by your system'sPAGE_SIZE(e.g., 4096).
4. Option A: Reduce shared_buffers (Recommended First Approach)
If your shared_buffers value is excessively high (e.g., more than 25-30% of your total system RAM on a dedicated DB server, or even less if running other services), reducing it is often the simplest and safest fix.
- Edit
postgresql.conf: -
`bash - sudo nano /etc/postgresql/12/main/postgresql.conf
-
` - Adjust
shared_buffers: - Find the
shared_buffersline and set it to a more conservative value. A common recommendation is 25% of your system's total RAM. For example, if you have 8GB of RAM,2GBwould be a good starting point. If you have 4GB RAM,1GBis appropriate.
# Example: Reduce shared_buffers to 2GB
shared_buffers = 2GB
```
> [!WARNING]
- Save and Exit:
- Press
Ctrl+X, thenYto save, andEnterto confirm the filename.
- Restart PostgreSQL:
-
`bash - sudo systemctl restart postgresql
-
`
- Verify Status:
-
`bash - sudo systemctl status postgresql
-
` - If successful, the service should now be
active (running).
5. Option B: Increase Kernel Shared Memory Limits (If Reduction is Not Desirable)
If you have ample physical RAM and genuinely require a larger shared_buffers value (e.g., 4GB or more on systems with 16GB+ RAM), you can increase the kernel's shared memory limits. This requires calculating appropriate shmmax and shmall values.
- Determine Required Values:
- Let's assume you want to set
shared_buffers = 4GB.
-
kernel.shmmax: This should be at least the value ofshared_buffersin bytes. -
4 GB = 4 1024 1024 * 1024 = 4294967296 bytes - So,
kernel.shmmax = 4294967296
-
kernel.shmall: This is calculated as the desired total shared memory in bytes, divided by the system's page size, then rounded up. AssumingPAGE_SIZE = 4096bytes: -
4294967296 bytes / 4096 bytes/page = 1048576 pages - So,
kernel.shmall = 1048576
- Create/Edit a
sysctl.dConfiguration File: - It's best practice to create a new
.conffile in/etc/sysctl.d/rather than modifying/etc/sysctl.confdirectly.
sudo nano /etc/sysctl.d/99-postgresql-shm.conf
- Add/Modify Parameters:
- Add the calculated values to the file.
# PostgreSQL shared memory settings
kernel.shmmax = 4294967296
kernel.shmall = 1048576
- Save and Exit:
- Press
Ctrl+X, thenYto save, andEnterto confirm the filename.
- Apply New Kernel Parameters:
-
`bash - sudo sysctl -p /etc/sysctl.d/99-postgresql-shm.conf
-
` - You should see output confirming the new values:
-
` - kernel.shmmax = 4294967296
- kernel.shmall = 1048576
-
` - These changes are persistent across reboots due to being in
/etc/sysctl.d/.
- Verify New Kernel Parameters:
-
`bash - sudo sysctl kernel.shmmax kernel.shmall
-
`
- Adjust
shared_buffersinpostgresql.conf(if not already done): - If you increased kernel limits to support a larger
shared_buffersvalue, ensurepostgresql.confreflects this.
sudo nano /etc/postgresql/12/main/postgresql.conf
```
- Restart PostgreSQL:
-
`bash - sudo systemctl restart postgresql
-
`
- Verify Status:
-
`bash - sudo systemctl status postgresql
-
` - The service should now be
active (running).
[!IMPORTANT] Always ensure that the sum of
sharedbuffersand other memory-intensive PostgreSQL settings (likeworkmem,maintenanceworkmem,walbuffers, and memory formaxconnections) does not exceed your system's physical RAM to prevent excessive swapping and performance degradation. As a general rule forshared_buffers, start with 25% of system RAM and never exceed 50% without very specific reasons and careful monitoring.