Staging ButItWorkedLocal

Tag: oom

  • Redis OOM Error: ‘command not allowed when used memory limit reached’ on Alpine Linux

    Troubleshoot Redis OOM errors on Alpine Linux, common in containerized environments. Learn to analyze memory, adjust limits, and optimize Redis for production stability.

    Introduction

    As an experienced Systems Administrator, encountering Redis Out-Of-Memory (OOM) errors is a critical incident that can bring applications to a halt. The specific error message "OOM command not allowed when used memory limit reached" indicates that your Redis instance has hit its configured maxmemory threshold and, crucially, is configured with a noeviction policy, preventing it from automatically removing keys to free up space. This is particularly common in resource-constrained environments like Docker containers running Alpine Linux, where memory limits are often tightly controlled. When this occurs, your application will experience command failures, timeouts, and data unavailability, leading to a degraded user experience or complete service outage.

    This guide will provide a highly technical, accurate, and step-by-step approach to diagnose, understand, and resolve this critical Redis OOM issue on Alpine Linux environments.

    Symptom & Error Signature

    When Redis reaches its maxmemory limit with a noeviction policy, all write commands (and some read commands depending on specific configuration) will be blocked. You will typically observe the following:

    1. Application Logs: Your application (e.g., PHP, Python, Node.js) will receive errors from its Redis client library:

    # Example PHP client error
[2026-07-18 10:30:05] app.ERROR: RedisException: OOM command not allowed when used memory > 'maxmemory' in /app/vendor/phpredis/phpredis/src/Redis.php:123
Stack trace:
#0 /app/src/Service/CacheService.php(45): Redis->setEx()

    Example Python client error Traceback (most recent call last): File "myapp.py", line 25, in <module> r.set("user:123", "data") File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1914, in set return self.execute_command('SET', args, *kwargs) File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1021, in execute_command return self.executecommand(conn, command_name, args, *kwargs) File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1047, in executecommand response = conn.read_response() File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 828, in read_response raise self.read_error() redis.exceptions.ResponseError: OOM command not allowed when used memory > 'maxmemory' (bytes) `

    2. Redis Server Logs: The Redis server itself will log warnings indicating the OOM condition:

    1:M 18 Jul 2026 10:30:00.123 # WARNING: OOM command not allowed when used memory > 'maxmemory' (1073741824 bytes)

    3. redis-cli Output: Direct attempts to write data via redis-cli will also fail:

    redis-cli SET mykey myvalue
(error) OOM command not allowed when used memory > 'maxmemory' (1073741824 bytes)

    Root Cause Analysis

    The "OOM command not allowed" error on Alpine Linux, or any other OS, points to a fundamental conflict between available memory, configured limits, and the Redis eviction policy.

    1. maxmemory Limit Reached: This is the primary trigger. Redis is configured with a maxmemory directive in its redis.conf (or via CONFIG SET) which sets an explicit upper bound on the amount of memory it will use for data. When the used_memory metric (reported by INFO memory) exceeds this threshold, Redis enters an OOM state.
    1. noeviction Policy: The choice of maxmemory-policy dictates Redis's behavior when maxmemory is reached.
    2. * If maxmemory-policy is set to noeviction (which is often the default or chosen for critical data stores), Redis will return OOM errors for all write commands (e.g., SET, LPUSH, HSET) and some read commands that might potentially create new keys or increase memory usage (e.g., GETSET, SADD if a set key doesn't exist).
    3. * Other policies (e.g., allkeys-lru, volatile-ttl) would attempt to evict keys to free up space, preventing the OOM error but potentially losing data.
    1. Actual Memory Consumption: The memory usage within Redis can grow due to several factors:
    2. * Data Growth: Natural increase in the number of keys or the size of values stored.
    3. * Temporary Keys: Keys without TTLs that accumulate over time.
    4. * Memory Fragmentation: The memfragmentationratio (from INFO memory) indicates how efficiently Redis is using physical memory. A high ratio (e.g., > 1.5) means Redis is consuming significantly more physical RAM than its reported used_memory due to the underlying memory allocator.
    5. * Client Output Buffers: Large or numerous client connections can consume significant memory for output buffers.
    6. * Replication Buffers: Master-replica replication links maintain buffers for changes, which can grow large, especially if replicas fall behind.
    7. * RDB/AOF Background Saves: During a background save operation (BGSAVE or BGREWRITEAOF), Redis uses a copy-on-write mechanism. If data changes significantly during a save, the memory footprint can temporarily double.
    8. * Lua Scripting: Complex or long-running Lua scripts can hold onto memory.
    1. Alpine Linux & Docker Specifics:
    2. Container Resource Limits: In Docker or Kubernetes, the container itself has memory limits (e.g., --memory flag for docker run, resources.limits.memory in Kubernetes). It's crucial that the Redis maxmemory setting is less than the container's memory limit. If Redis's usedmemoryrss (Resident Set Size) approaches or exceeds the container's limit, the OS's OOM killer will terminate the entire container*, which is much more disruptive than Redis simply refusing commands.
    3. * Memory Allocator: Alpine Linux by default uses musl libc for its C standard library, which includes a simple malloc implementation. Official Redis builds typically link against jemalloc on Linux, which is highly optimized for Redis's allocation patterns and generally results in lower fragmentation. If your Redis build on Alpine is using musl's malloc, you might experience higher memory fragmentation and less efficient memory utilization compared to a jemalloc build. You can check mem_allocator in INFO memory.

    Step-by-Step Resolution

    Resolving this OOM issue requires a systematic approach, combining immediate mitigation with long-term optimization and scaling strategies.

    1. Analyze Current Redis Configuration and Memory Usage

    First, gather critical information about your Redis instance. If running in Docker, you'll need to execute commands inside the container.

    # Example: Accessing a Dockerized Redis instance

    — OR —

    Example: Accessing a directly installed Redis instance redis-cli `

    Once connected to redis-cli:

    # Check current maxmemory limit and eviction policy
CONFIG GET maxmemory

    Get detailed memory statistics INFO memory

    Expected important output from INFO memory: # used_memory: The total number of bytes allocated by Redis using its allocator (usually jemalloc) # usedmemoryrss: The number of bytes that Redis allocated as reported by the operating system. # usedmemorypeak: The maximum memory consumed by Redis (in bytes) since server startup. # memfragmentationratio: usedmemoryrss / used_memory. A ratio > 1 means fragmentation. # maxmemory_human: The configured maxmemory limit in human-readable format. # maxmemory_policy: The configured eviction policy. # mem_allocator: The memory allocator used (e.g., "jemalloc-5.1.0", "libc")

    Check connected clients (can consume memory for output buffers) INFO clients

    Check persistence information (RDB/AOF background saves consume temporary memory) INFO persistence `

    [!IMPORTANT] A memfragmentationratio significantly above 1.0 (e.g., 1.5+) indicates memory fragmentation. If memallocator is libc, especially on Alpine, this could be a contributing factor. If usedmemory_rss is close to your Docker container's memory limit, you risk container OOM kills.

    2. Identify Memory Consumers

    Pinpoint which keys or operations are consuming the most memory.

    # Identify large keys (requires Redis 4.0+)
# This is a good starting point, but can be slow on very large databases
# Replace <prefix>* with actual key patterns if known

    For more detailed memory breakdown (Redis 4.0+) redis-cli MEMORY STATS

    Monitor commands in real-time (use cautiously in production due to overhead) # This can help identify commands that might be creating large objects or many keys rapidly. redis-cli MONITOR `

    If running in Docker, monitor the container's resource usage:

    docker stats <redis-container-id-or-name>

    Look for trends in MEM USAGE / LIMIT.

    3. Adjust Redis maxmemory and maxmemory-policy (Short-term / Mitigation)

    This step provides immediate relief but might not be a long-term solution without further optimization.

    A. Increase maxmemory (Temporary Relief or Resource Upgrade)

    If you have available RAM on your host or can increase your container's memory limit, raising maxmemory can alleviate the immediate pressure.

    [!WARNING] Increasing maxmemory without addressing underlying data growth is merely a band-aid. Also, ensure your new maxmemory value is comfortably below your Docker container's memory limit to prevent container OOM kills. A good rule of thumb is maxmemory < containermemorylimit * 0.8 to account for overhead.

    Method 1: Via redis-cli (Runtime – temporary until restart)

    # Example: Set maxmemory to 2GB (2147483648 bytes)
CONFIG SET maxmemory 2gb

    Method 2: Via redis.conf (Persistent)

    Edit your redis.conf file (typically located at /etc/redis/redis.conf or in your Docker volume mount) and modify the maxmemory directive.

    # /etc/redis/redis.conf or mounted config file
maxmemory 2gb

    After modifying redis.conf, you must restart the Redis service or container:

    # For a non-containerized Redis instance (e.g., Systemd on Ubuntu)

    For a Docker container docker restart <redis-container-id-or-name> `

    B. Adjust maxmemory-policy (Data Loss Risk vs. Availability)

    Changing the eviction policy can prevent the OOM error by allowing Redis to automatically delete keys. This is suitable for Redis instances primarily used as a cache.

    [!IMPORTANT] Carefully consider the implications of changing maxmemory-policy. Policies other than noeviction will result in data loss when the maxmemory limit is hit. Choose a policy that aligns with your application's data criticality.

    Common eviction policies: * noeviction: (Current problematic state) Returns errors on write operations when maxmemory is reached. Use for critical data. allkeys-lru: Evicts keys least recently used (LRU) from all* keys until maxmemory is respected. Ideal for general-purpose caching. volatile-lru: Evicts LRU keys only among those with an expire set*. Useful if you mix persistent and cache data. allkeys-random: Evicts random keys from all* keys. volatile-random: Evicts random keys only among those with an expire set*. allkeys-ttl: Evicts keys with the shortest time to live (TTL) from all* keys. volatile-ttl: Evicts keys with the shortest time to live (TTL) only among those with an expire set*.

    To change the policy:

    Method 1: Via redis-cli (Runtime – temporary until restart)

    # Example: Set policy to allkeys-lru for a cache
CONFIG SET maxmemory-policy allkeys-lru

    Method 2: Via redis.conf (Persistent)

    Edit your redis.conf file:

    # /etc/redis/redis.conf or mounted config file
maxmemory-policy allkeys-lru

    Then restart Redis as described above.

    4. Optimize Data Structures and Application Usage

    This is a crucial long-term strategy to reduce Redis's memory footprint.

    • Set TTLs for Transient Data: Ensure that session data, temporary caches, and other non-persistent data automatically expire.
    • `bash
    • SET mytempkey "some data" EX 3600 # Expires in 1 hour
    • `
    • Use Efficient Data Structures:
    • * Hashes for Related Fields: Instead of SET user:1:name "Alice", SET user:1:email "[email protected]", use a single hash: HSET user:1 name "Alice" email "[email protected]". Hashes (especially "ziplist" encoded ones for small numbers of fields/values) are more memory-efficient than many individual string keys.
    • * Bitmaps for Booleans/Flags: For many on/off flags, bitmaps are extremely memory-efficient.
    • * HyperLogLogs for Unique Counts: For approximate unique counts (e.g., unique visitors), HyperLogLogs (PFADD, PFCOUNT) use fixed, small amounts of memory regardless of the number of unique items.
    • Avoid Large Keys/Values: Very large lists, sets, hashes, or string values consume significant memory. Consider breaking them down or offloading large binary data to object storage.
    • Connection Pooling: Ensure your application uses connection pooling to manage client connections efficiently, reducing the number of active clients and their associated output buffers.

    5. Scale Redis Resources (Long-term / Scaling)

    When optimization isn't enough, scaling is necessary.

    • Vertical Scaling: Increase the RAM allocated to the VM or Docker container running Redis. This is often the simplest first step if you have available host resources.
    • `bash
    • # Example for Docker compose
    • services:
    • redis:
    • image: redis:7-alpine
    • deploy:
    • resources:
    • limits:
    • memory: 2G # Increase to 2GB
    • `
    • Remember to also adjust Redis's maxmemory accordingly to utilize the new resources.
    • Horizontal Scaling (Redis Cluster): For very large datasets or high throughput, consider a Redis Cluster. This shards your data across multiple Redis nodes, distributing memory and CPU load. This requires significant application-level changes and operational complexity.
    • Read Replicas: If your workload is read-heavy, offload read queries to Redis replicas, reducing the load (and potentially memory usage if client buffers are an issue) on the primary instance.

    6. Tune Alpine/Docker Memory Allocator (Advanced)

    If you suspect memory fragmentation is a major issue (high memfragmentationratio) and your INFO memory shows mem_allocator: libc on Alpine, you might benefit from using jemalloc.

    # Check current allocator
redis-cli INFO memory | grep mem_allocator
    • Official Redis Docker Images: The official redis:<version>-alpine Docker images typically ship with Redis compiled to use jemalloc. If you're using a custom Dockerfile or a non-official image, you might not have jemalloc.
    • Building with jemalloc: If you're building Redis from source on Alpine, ensure jemalloc is included. This usually involves installing jemalloc-dev and configuring Redis with MALLOC=jemalloc.
    • `dockerfile
    • # Example Dockerfile snippet for building Redis with jemalloc on Alpine
    • FROM alpine:3.18
    • RUN apk add –no-cache gcc make musl-dev jemalloc-dev tcl
    • WORKDIR /tmp/redis
    • RUN wget http://download.redis.io/releases/redis-7.0.11.tar.gz &&
    • tar xvzf redis-7.0.11.tar.gz &&
    • cd redis-7.0.11 &&
    • make MALLOC=jemalloc &&
    • make install

    … rest of your Redis setup … ` > [!IMPORTANT] > Switching memory allocators requires rebuilding Redis and is a non-trivial change. It should be thoroughly tested. For most users, simply using the official redis:alpine Docker images is sufficient as they are pre-optimized.

    7. Implement Robust Monitoring and Alerting

    Proactive monitoring is key to preventing future OOM issues.

    • Key Metrics to Monitor:
    • * used_memory: Absolute memory usage.
    • * usedmemoryrss: Physical memory consumed.
    • * memfragmentationratio: Detects memory fragmentation.
    • * keys: Number of keys.
    • * expires: Number of keys with TTLs.
    • * evicted_keys: (If using an eviction policy) indicates when Redis is actively deleting keys.
    • * blocked_clients: Number of clients blocked by commands (can indicate issues).
    • Alerting: Set up alerts for when usedmemory (or usedmemory_rss) exceeds a predefined threshold (e.g., 70-80% of maxmemory or container limit) to provide early warning. Also, alert on OOM command not allowed errors in logs.
    • Tools: Utilize monitoring solutions like Prometheus + Grafana, Datadog, New Relic, or custom scripts to collect and visualize Redis metrics.

    By systematically applying these steps, you can effectively troubleshoot and resolve the "Redis OOM command not allowed when used memory limit reached" error, ensuring the stability and performance of your applications.

  • Troubleshooting Docker Container Exited with Code 137: OOM Killed on Alpine Linux

    Resolve Docker containers exiting with code 137 due to OOM kills on Alpine Linux. Learn to identify, diagnose, and fix out-of-memory issues effectively.

    When a Docker container abruptly stops functioning, exhibiting an "Exited (137)" status, it's a strong indicator that the Linux kernel's Out-Of-Memory (OOM) killer has terminated the process. This guide provides a comprehensive, expert-level approach to diagnose and resolve these critical memory-related issues, specifically in environments utilizing Alpine Linux base images for their Docker containers.

    Symptom & Error Signature

    The most prominent symptom is a container that repeatedly crashes, fails to start, or becomes unresponsive, leading to service disruption. Your application, whether it's a web server like Nginx, a database, or a custom microservice, will cease to function.

    You'll typically observe the following:

    1. Container Status:
    2. `bash
    3. $ docker ps -a
    4. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    5. a1b2c3d4e5f6 my-alpine-app:latest "node /app/index.js" 2 minutes ago Exited (137) 5 seconds ago web_app
    6. `
    1. Docker Daemon Logs (on the host system):
    2. These logs often provide the first explicit mention of an OOM event.
    3. `bash
    4. $ sudo journalctl -u docker.service | grep -i "oom|killed process"
    5. Jul 15 10:30:45 hostname dockerd[1234]: containerd/task/v2/sender/pipe.go:39: "OOM killer killed containerida1b2c3d4e5f6"
    6. Jul 15 10:30:45 hostname dockerd[1234]: shimexit: containerid_a1b2c3d4e5f6 exit status 137
    7. `
    1. Kernel Logs (dmesg on the host system):
    2. The definitive evidence comes from the kernel itself, detailing the OOM event.
    3. `bash
    4. $ sudo dmesg -T | grep -i "oom-killer|killed process"
    5. [Wed Jul 15 10:30:44 2026] node invoked oom-killer: gfpmask=0x100cca(GFPHIGHUSERMOVABLE|GFPCOMP), order=0, oomscoreadj=0
    6. [Wed Jul 15 10:30:44 2026] oom-kill:constraint=CONTAINER,nodemask=(null),cpuset=docker/a1b2c3d4e5f6,memsallowed=0,globaloom,task_memcg=/docker/a1b2c3d4e5f6,task=node,pid=5678,uid=0
    7. [Wed Jul 15 10:30:44 2026] Memory cgroup out of memory: Killed process 5678 (node) total-vm:450000kB, anon-rss:400000kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:800kB oomscoreadj:0
    8. [Wed Jul 15 10:30:44 2026] Tasks state (memory and swap accounts):
    9. [Wed Jul 15 10:30:44 2026] [ pid ] uid tgid totalvm rss nrptes nrpmds swapents oomscore_adj name
    10. [Wed Jul 15 10:30:44 2026] [ 5678] 0 5678 112500 100000 200 0 0 0 node
    11. [Wed Jul 10:30:44 2026] oom-killer: Kill process 5678 (node) score 1000 or sacrifice child
    12. [Wed Jul 10:30:44 2026] Killed process 5678 (node) total-vm:450000kB, anon-rss:400000kB, file-rss:0kB, shmem-rss:0kB
    13. `
    14. This output clearly shows oom-killer being invoked and targeting process 5678 (node) within the specified Docker cgroup.

    Root Cause Analysis

    The "Exited with code 137" error specifically means the container process received a SIGKILL signal (signal 9). When this is combined with "OOM killed," it unambiguously indicates that the Linux kernel's Out-Of-Memory (OOM) killer decided to terminate the container's main process.

    Here's a breakdown of the underlying reasons:

    • Linux OOM Killer: This is a critical kernel mechanism designed to maintain system stability. When the system's memory (RAM + swap) is exhausted, the OOM killer steps in to free up resources by terminating one or more processes. It uses an internal heuristic (oom_score) to decide which processes are "least important" to kill.
    • Docker Memory Limits (cgroups): Docker leverages Linux Control Groups (cgroups) to isolate and limit resource usage for containers.
    • * mem_limit (or -m flag): This sets the maximum amount of RAM a container can use.
    • * memswap_limit (or --memory-swap flag): This defines the combined limit for RAM and swap space.
    • If a container's processes attempt to allocate memory beyond its memlimit, the OOM killer is triggered within that container's cgroup*, effectively killing the container without affecting other processes on the host or in other containers (unless the host itself is completely out of memory). If memswaplimit is hit, and there's no more swap available, the OOM killer will also be triggered.
    • Application Memory Profile:
    • * Memory Leak: The application has a bug that causes it to continuously consume more memory over time without releasing it.
    • * Memory Spike: The application might have a legitimate, but unexpected, memory spike during certain operations (e.g., loading large datasets, complex computations, garbage collection cycles), especially during startup or peak load.
    • * Incorrect Configuration: Application-specific memory settings (e.g., JVM heap size -Xmx, Node.js --max-old-space-size, PHP memory_limit) are misconfigured or too high for the allocated container limits.
    • * Unoptimized Code/Libraries: Using inefficient data structures or libraries that are memory-hungry.
    • Host System Memory Exhaustion: While less common when cgroup limits are in place, if the host system itself runs out of memory (e.g., too many containers, other demanding host processes, insufficient swap), the OOM killer might target container processes even if they haven't explicitly hit their Docker limits, or if limits are unset/very high.
    • Alpine Linux Specifics: While Alpine's small base image size (often just a few MB) reduces the baseline memory footprint, it doesn't fundamentally change how the OOM killer or cgroups operate. The application inside the Alpine container is still subject to the same memory demands. Often, developers choose Alpine for its small size and then forget to properly account for the application's actual memory needs, leading to hitting tighter limits more quickly.

    Step-by-Step Resolution

    Resolving OOM issues requires a systematic approach, combining host-level diagnostics, container-level inspection, and application-specific tuning.

    1. Confirm OOM Kill and Identify the Victim Process

    Before making any changes, confirm the container was indeed OOM killed and identify the exact process within the container that was targeted.

    • Check dmesg output on the host:
    • `bash
    • sudo dmesg -T | grep -E "oom-killer|killed process"
    • `
    • Look for entries similar to the "Symptom & Error Signature" section, specifically mentioning Memory cgroup out of memory and the container's cgroup path (e.g., /docker/a1b2c3d4e5f6). Note the pid and name of the killed process.
    • Check Docker daemon logs:
    • `bash
    • sudo journalctl -u docker.service | grep "OOM killer killed container"
    • `
    • This confirms Docker's daemon saw the OOM event.

    2. Inspect Container Memory Limits and Usage

    Understand what memory limits, if any, were applied to the container, and review historical usage if possible.

    • Check configured memory limits:
    • `bash
    • docker inspect <containeridor_name> | grep -E "Memory|Swap"
    • `
    • Look for Memory and MemorySwap under HostConfig. Values of 0 typically mean unlimited, but this is relative to the host's physical memory.
    • Example output:
    • `json
    • "Memory": 0,
    • "MemorySwap": 0,
    • "MemoryReservation": 0,
    • "KernelMemory": 0,
    • "OomKillDisable": false,
    • "OomScoreAdj": 0,
    • `
    • If Memory is 0, the container can consume all available host memory, making the host's overall memory the limiting factor. If it's a specific value (e.g., 536870912 for 512MB), that's your hard limit.
    • Monitor live memory usage (if the container starts but crashes later):
    • `bash
    • docker stats <containeridor_name>
    • `
    • This command provides real-time statistics, including memory usage, against the configured limits. Run this frequently to observe memory trends before a crash.

    3. Analyze Application Memory Footprint and Configuration

    This is often the most critical step. You need to understand how much memory your application genuinely needs.

    • Profile the application:
    • * Run with generous limits: Temporarily remove or significantly increase memory limits (-m 2g --memory-swap -1) on a test environment to see how much memory the application consumes at peak load and steady state.
    • * Use in-container tools: If your Alpine image has tools like ps or top (you might need to install procps), exec into a running container:
    • `bash
    • docker exec -it <container_id> sh
    • / # apk add procps # if not already installed
    • / # ps aux
    • / # top
    • `
    • Monitor the VSZ (virtual size) and RSS (resident set size) columns.
    • * Language-specific profiling:
    • * Java: Tune JAVA_OPTS with -Xmx (max heap size) and -Xms (initial heap size). Remember that JVM also uses off-heap memory.
    • * Node.js: Check for memory leaks using built-in profiling tools or Chrome DevTools. Adjust --max-old-space-size if needed via NODE_OPTIONS.
    • * PHP: Adjust memorylimit in php.ini or via iniset().
    • * Python: Libraries like memory_profiler can help.
    • * Go: Go applications are often very efficient, but Goroutine leaks can lead to memory growth.
    • * Review application logs: Look for errors or warnings that might indicate resource exhaustion, large data operations, or unhandled exceptions that could trigger memory spikes.

    4. Adjust Docker Container Memory Limits

    Based on your analysis, you will likely need to increase the memory allocated to your container.

    • Increase mem_limit: This directly increases the available RAM.
    • * Docker CLI:
    • `bash
    • docker run -d –name my-app –restart always -m 768m my-alpine-app:latest
    • `
    • This sets the RAM limit to 768MB.
    • * Docker Compose: This is the recommended approach for defining services.
    • `yaml
    • # docker-compose.yml
    • version: '3.8'
    • services:
    • my_app:
    • image: my-alpine-app:latest
    • container_name: my-app
    • restart: always
    • mem_limit: 768m # Max RAM usage
    • mem_reservation: 512m # Soft limit, Docker tries to keep usage below this
    • memswap_limit: 768m # Total RAM + SWAP. Set to -1 for unlimited swap
    • `
    • > [!IMPORTANT]
    • > If memswaplimit is set to the same value as memlimit, it means the container has no swap available. If the application hits its memlimit, it will immediately be OOM killed. Setting memswaplimit to -1 allows the container to use unlimited swap on the host if its memlimit is reached. While this prevents OOM kills, it can lead to severe performance degradation if the application frequently swaps. A better approach is to set memswaplimit higher than mem_limit (e.g., 1g for 512m RAM limit) to allow for some controlled swapping.
    • Consider memreservation: This is a "soft limit." Docker will try to keep the container's memory usage below this value, but will allow it to exceed it up to memlimit if the host has spare memory. This is useful for resource scheduling.

    5. Optimize the Application and Dockerfile

    Reduce your application's memory footprint or ensure it's configured to respect container limits.

    • Application-level tuning:
    • * JVM: Reduce -Xmx if it's too high for the container's memlimit. Ensure the container's memlimit is greater than -Xmx to account for off-heap memory.
    • * Node.js: NODE_OPTIONS=--max-old-space-size=512 (for 512MB) can prevent Node from attempting to use too much memory.
    • * PHP: Set memorylimit in php.ini to a value below your Docker container's memlimit.
    • * Caching: Implement efficient caching strategies to reduce repetitive memory-intensive operations.
    • * Data Structures: Review code for inefficient data structures or algorithms that cause excessive memory allocation.
    • * Garbage Collection: Tune garbage collection parameters for languages like Java or Node.js.
    • Dockerfile optimization (especially for Alpine):
    • * Multi-stage builds: Use multi-stage builds to create extremely lean final images that only contain the necessary runtime dependencies, removing build-time tools and intermediate files.
    • `dockerfile
    • # Example Multi-stage Dockerfile for Node.js on Alpine
    • # Stage 1: Build dependencies
    • FROM node:18-alpine AS builder
    • WORKDIR /app
    • COPY package*.json ./
    • RUN npm ci –production –frozen-lockfile
    • COPY . .
    • RUN npm run build # if you have a build step

    Stage 2: Runtime image FROM node:18-alpine WORKDIR /app COPY –from=builder /app/nodemodules ./nodemodules COPY –from=builder /app/dist ./dist # Assuming build output is in dist COPY –from=builder /app/package.json ./package.json COPY –from=builder /app/index.js ./index.js # Your main app file EXPOSE 3000 CMD ["node", "index.js"] ` * Minimalist packages: When installing packages with apk add, only install what's absolutely necessary. Use apk add --no-cache to prevent package caches from increasing image size.

    6. Increase Host System Memory or Swap (If Necessary)

    If multiple containers are consistently hitting OOM issues, or the host itself is frequently low on memory, it might be time to provision more RAM for your server.

    • Add more RAM: The most straightforward solution for overall system capacity.
    • Increase host swap space: While not a substitute for RAM, sufficient swap space can prevent the host's global OOM killer from being invoked during temporary memory spikes.
    • `bash
    • > [!IMPORTANT]
    • > Relying heavily on swap can severely degrade application performance due to disk I/O. Use it as a last resort or for minor overflow, not as a primary memory extension.

    Example: Add a 4GB swap file on Ubuntu/Debian sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile

    Make swap persistent across reboots by adding to /etc/fstab: echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab `

    7. Adjust OOM Score (Advanced – Use with Extreme Caution)

    For highly critical containers that must stay alive at all costs (e.g., a monitoring agent), you can adjust their oomscoreadj. A lower score makes the process less likely to be chosen by the OOM killer.

    • Docker CLI:
    • `bash
    • docker run -d –name critical-service –oom-score-adj -500 my-alpine-app:latest
    • `
    • Values range from -1000 (least likely to be killed) to 1000 (most likely).

    [!WARNING] > Lowering the OOM score for a memory-hungry process can lead to the OOM killer targeting other, potentially more critical, system processes or even the Docker daemon itself, potentially destabilizing the host system. Use this only for processes that are truly essential and where all other alternatives for memory optimization and provisioning have been exhausted.

    8. Implement Robust Monitoring and Alerting

    Proactive monitoring can help identify memory pressure before it leads to a full OOM kill.

    • Memory Usage: Monitor container memory usage (RSS, cache, swap) using tools like Prometheus, Grafana, cAdvisor, or specialized APM solutions.
    • Container Status: Set up alerts for containers exiting with a 137 status.
    • Host System Metrics: Monitor overall host memory usage, swap usage, and dmesg output for OOM events.
    • Application Metrics: If your application exposes memory metrics, integrate them into your monitoring stack.

    By following these steps, you can effectively diagnose and resolve Docker container OOM issues, ensuring the stability and reliability of your containerized applications on Alpine Linux.

  • Troubleshooting Node.js JavaScript Heap Out of Memory on Debian 12 Bookworm

    Resolve Node.js 'JavaScript heap out of memory' errors on Debian 12. Learn to diagnose, increase V8 heap limits, and optimize your application for stability.

    Node.js applications, while powerful, can sometimes hit memory limitations, particularly when handling large datasets, complex computations, or suffering from memory leaks. One of the most common critical errors encountered by Node.js developers and systems administrators is the "JavaScript heap out of memory" error. This guide provides a comprehensive, expert-level approach to diagnosing and resolving this issue on a Debian 12 Bookworm environment, ensuring your Node.js services remain robust and performant.

    Symptom & Error Signature

    When your Node.js application exhausts its allocated memory for the JavaScript heap, it typically crashes with a FATAL ERROR. Users might experience unresponsive web applications, HTTP 500 errors if proxied by Nginx, or the application service failing to start or restarting repeatedly.

    You will observe logs similar to these in your application's output, journalctl, or PM2 logs:

    [2766:0x55dc5c46e3d0] 89006 ms: Mark-sweep (reduce) 808.6 (825.9) -> 808.6 (819.9) MB, 50.8 / 0.0 ms (average mu = 0.176, where mu = makeprogresssincelastgcstart + topofstackgc_latency) allocation failure; GC in old space requested [2766:0x55dc5c46e3d0] 89045 ms: Mark-sweep (reduce) 808.6 (819.9) -> 808.6 (819.9) MB, 39.4 / 0.0 ms (average mu = 0.176, where mu = makeprogresssincelastgcstart + topofstackgc_latency) allocation failure; GC in old space requested

    <— JS stacktrace —>

    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed – JavaScript heap out of memory 1: 0x7a3000b0c60f node::OOMErrorHandler::OOMErrorHandler() 2: 0x7a3000b0d4ff node::OOMErrorHandler::~OOMErrorHandler() 3: 0x7a3000b380bf v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) 4: 0x7a3000b38206 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) 5: 0x7a3000f36815 v8::internal::Heap::FatalProcessOutOfMemory(char const*) 6: 0x7a3000f3933c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) 7: 0x7a3000f3c05e v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) 8: 0x7a3000f074d2 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType) 9: 0x7a3000f076c8 v8::internal::Factory::NewHeapNumberFromDouble(double) 10: 0x7a30010837f4 v8::internal::Object::AllocateHeapNumber(v8::internal::Isolate*, double) 11: 0x7a3000c01777 v8::internal::Builtin_NumberConstructor(v8::internal::Builtin, v8::internal::Isolate*, v8::internal::Builtin::CallInfo const&) 12: 0x7a3000c02111 v8::internal::BuiltinImplNumberConstructor(v8::internal::Builtin, v8::internal::Isolate*, v8::internal::Builtin::CallInfo const&) 13: 0x7a3000f898e1 v8::internal::Builtin_HandleApiCall(v8::internal::Builtin, v8::internal::Isolate*, v8::internal::Builtin::CallInfo const&) `

    Root Cause Analysis

    The "JavaScript heap out of memory" error indicates that the V8 JavaScript engine, which powers Node.js, has tried to allocate more memory than its configured heap limit. This can stem from several underlying issues:

    1. Default V8 Heap Limit: By default, Node.js on a 64-bit system allocates approximately 1.4 GB (for Node.js 12 and above) to 4 GB of memory to its JavaScript heap. While seemingly generous, complex applications or those processing large data objects can quickly exceed this.
    2. Memory Leaks: This is the most common culprit. A memory leak occurs when your application continuously allocates memory but fails to release it when no longer needed, leading to a gradual increase in memory consumption over time. Common sources include:
    3. * Unclosed database connections or file handles.
    4. * Improperly managed event listeners.
    5. * Caching without eviction policies.
    6. * Global variables holding large objects that are never de-referenced.
    7. * Retained closures.
    8. Large Data Processing:
    9. * Loading entire large files (e.g., CSV, JSON, images) into memory at once.
    10. * Extensive string manipulations or regular expressions on very long strings.
    11. * Performing complex computations that generate large intermediate data structures.
    12. * Handling a high volume of concurrent requests, each requiring significant memory.
    13. Insufficient System Memory: While the Node.js process might be the primary consumer, the server itself might not have enough RAM to accommodate the Node.js application, its dependencies, the operating system, and other running services.
    14. Inefficient Algorithms: Algorithms with high space complexity (e.g., O(n^2) or O(2^n)) can quickly consume memory, especially with larger inputs.
    15. Dependency Bloat: Third-party libraries, particularly those that are not well-optimized, can contribute to overall memory usage.

    Step-by-Step Resolution

    Addressing this error typically involves a combination of increasing the V8 heap limit as a temporary measure and, crucially, optimizing your application code and server resources for long-term stability.

    1. Analyze Current Memory Usage and Server Resources

    Before making changes, understand your current memory profile.

    # Check overall system memory

    Monitor real-time process usage (install htop if not present: sudo apt install htop) htop

    Check memory usage of your Node.js process using its PID (e.g., 12345) # Use 'ps aux | grep node' to find the PID cat /proc/12345/status | grep VmRSS `

    For deeper Node.js specific analysis: * Node.js built-in profiling: Run your application with --trace-gc or --heap-prof to get detailed garbage collection logs and heap profiles.

        node --trace-gc --heap-prof your_app.js
    ```
        // In your Node.js application
    console.log('Heap usage:', process.memoryUsage().heapUsed / 1024 / 1024, 'MB');
    ```
        npm install heapdump
    ```
    ```javascript
    // In your Node.js application (e.g., triggered by a signal or specific route)
    const heapdump = require('heapdump');
    process.on('SIGUSR2', () => {
      heapdump.writeSnapshot((err, filename) => {
        if (err) console.error('Error writing heap snapshot:', err);
        else console.log('Heap snapshot written to', filename);
      });
    });
    // Then, send SIGUSR2 to your Node.js process: kill -SIGUSR2 <PID>

    2. Increase Node.js V8 Heap Memory Limit

    This is often the quickest way to mitigate the immediate crash, but it does not solve underlying memory leaks. It buys you time to implement proper code optimizations.

    The V8 heap memory limit can be adjusted using the --max-old-space-size flag for the Node.js process. The value is specified in megabytes (MB).

    [!WARNING] While increasing the heap limit can prevent immediate crashes, arbitrarily large values can mask memory leaks, lead to longer garbage collection pauses, and consume excessive system RAM, potentially causing system-wide performance degradation or out-of-memory kills by the kernel's OOM killer. Use this as a carefully considered adjustment, not a blanket solution.

    a. For applications run directly or with npm start:

    node --max-old-space-size=4096 your_app.js
# Or via environment variable (preferred)
export NODE_OPTIONS="--max-old-space-size=4096"
node your_app.js

    b. For systemd managed services:

    Modify your service unit file (e.g., /etc/systemd/system/your_app.service).

    sudo systemctl edit --full your_app.service

    Locate the [Service] section and add or modify the Environment variable:

    [Unit]
Description=Your Node.js Application

    [Service] User=your_user Group=your_group WorkingDirectory=/path/to/your/app Environment="NODE_ENV=production" Environment="NODE_OPTIONS=–max-old-space-size=4096" # <— Add this line ExecStart=/usr/bin/node /path/to/your/app/index.js Restart=always

    [Install] WantedBy=multi-user.target `

    After modifying, reload systemd and restart your service:

    sudo systemctl daemon-reload
sudo systemctl restart your_app.service

    c. For PM2 managed applications:

    Modify your ecosystem.config.js file.

    module.exports = {
  apps : [{
    name: "my-app",
    script: "./index.js",
    instances: "max",
    exec_mode: "cluster",
    // Options to pass to Node.js executable
    node_args: "--max-old-space-size=4096", // <--- Add this line
    env: {
      NODE_ENV: "production",
    }
  }]
};

    Then reload or restart your PM2 application:

    pm2 reload ecosystem.config.js --env production

    3. Optimize Node.js Application Code

    This is the most critical long-term solution.

    a. Identify and Fix Memory Leaks: * Profile your application: Use the heap snapshots generated in step 1 and analyze them with Chrome DevTools. Look for objects that are growing in count or size over time without being released. * Event Listeners: Ensure event listeners are properly removed when no longer needed (e.g., eventEmitter.removeListener('event', handler)). * Caches: Implement a cache with a size limit and eviction policy (e.g., LRU cache). * Global Variables: Avoid storing large objects in global scopes or closures that are never garbage collected. * Streams: Use Node.js streams for reading/writing large files or network data instead of loading everything into memory.

    b. Efficient Data Handling: * Pagination: Retrieve data from databases or APIs in smaller chunks using pagination. * Data Transformation: Process large data arrays iteratively or in batches rather than creating new large arrays for intermediate results. * Avoid Cloning Large Objects: Pass objects by reference when possible instead of deep cloning. * JSON Processing: For extremely large JSON files, consider streaming parsers (e.g., jsonstream).

    c. Reduce Object Creation: * Object Pooling: For frequently created/destroyed objects, consider an object pool pattern. * Memoization: Cache results of expensive function calls to avoid recalculating. * String Manipulation: Be mindful of repeated string concatenations, which can create many intermediate strings. Use array join() for many small strings.

    d. Update Dependencies: * Ensure all your Node.js package dependencies are up-to-date. Newer versions often include performance improvements and memory optimizations. * Audit dependencies for known memory issues.

    4. System Resource Allocation

    Ensure your server has adequate physical RAM for your Node.js application and the entire software stack.

    a. Increase Server RAM: If code optimization and heap limit adjustments aren't enough, your application might genuinely require more physical memory. Consider upgrading your server's RAM or migrating to a larger instance if running in a cloud environment.

    b. Configure Swap Space: While not a replacement for RAM, having sufficient swap space can prevent the OOM killer from terminating your process prematurely by allowing the OS to offload less frequently accessed memory pages to disk.

    # Check current swap usage
swapon --show

    Example: Create and enable a 4GB swap file if none exists or is insufficient # (Adjust count to desired size in blocks of 1M) sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile

    Make swap persistent across reboots echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

    [!NOTE] Excessive swapping indicates a severe memory shortage and will significantly degrade performance due to slow disk I/O. It's a stopgap, not a solution for insufficient RAM. `

    5. Containerized Environments (Docker/Kubernetes)

    If your Node.js application runs in Docker containers or Kubernetes, you must configure resource limits at the container level.

    a. Docker Compose:

    version: '3.8'
services:
  app:
    image: your_nodejs_app_image
    environment:
      NODE_ENV: production
      NODE_OPTIONS: "--max-old-space-size=4096" # Set V8 heap limit
    deploy:
      resources:
        limits:
          memory: 6G # Limit container memory to 6GB
          cpus: '2'
        reservations:
          memory: 2G # Reserve 2GB memory for the container

    b. Kubernetes:

    In your deployment manifest (deployment.yaml):

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: your-nodejs-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: your_nodejs_app_image
        env:
          - name: NODE_ENV
            value: "production"
          - name: NODE_OPTIONS
            value: "--max-old-space-size=4096" # Set V8 heap limit
        resources:
          limits:
            memory: "6Gi" # Limit container memory to 6Gi
            cpu: "2"
          requests:
            memory: "2Gi" # Request 2Gi memory for scheduling
            cpu: "500m"

    [!IMPORTANT] When setting container memory limits, ensure NODE_OPTIONS=--max-old-space-size is less than or equal to the container's memory limit. If the Node.js process tries to allocate more than the container's limit, the container will be OOMKilled by the kernel or Kubernetes, regardless of the V8 heap limit. Factor in memory usage from native modules, non-heap memory, and other processes in the container.

    6. Monitor and Alert

    Implement robust monitoring to track Node.js memory usage over time. * Prometheus/Grafana: Collect Node.js process metrics (RSS, heap usage, GC stats) and visualize trends. * APM Tools: Solutions like New Relic, Datadog, AppDynamics provide deep insights into application memory profiles, garbage collection events, and potential leaks. * Custom Scripting: Periodically log process.memoryUsage() metrics to a file or a logging service for long-term analysis.

    By combining careful analysis, judicious heap limit adjustments, thorough code optimization, and proper resource allocation, you can effectively resolve Node.js "JavaScript heap out of memory" issues and maintain a stable, high-performance application on Debian 12 Bookworm.

  • Troubleshooting Docker Container Exited with Code 137: OOM Killed Error

    Diagnose and resolve Docker containers exiting with code 137 due to Out-Of-Memory (OOM) killer terminations. Optimize resource allocation and prevent service downtime.

    When a Docker container unexpectedly stops with an Exited (137) status, especially when combined with messages indicating an "OOM killed" event, it signifies a critical resource allocation failure. This guide delves into the technical intricacies of why this happens and provides a systematic, expert-level approach to diagnose, resolve, and prevent such occurrences in your Dockerized environments. Understanding and mitigating Out-Of-Memory (OOM) kills is crucial for maintaining stable and performant containerized applications.

    Symptom & Error Signature

    The primary symptom is an application or service running within a Docker container becoming unresponsive and then stopping abruptly. When inspecting the container's status or logs, you will typically encounter the following signatures:

    1. docker ps -a Output:

    root@server:~# docker ps -a
CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                         PORTS     NAMES
a1b2c3d4e5f6   my-app:latest         "node server.js"         5 minutes ago   Exited (137) 3 minutes ago               my-app-container

    The Exited (137) status is a strong indicator. Code 137 specifically means the container received a SIGKILL (signal 9), which is typically issued by the Linux kernel's Out-Of-Memory (OOM) killer.

    2. docker logs Output:

    Reviewing the container's logs might show an abrupt halt, or sometimes the last few entries before the termination, but rarely a graceful shutdown message related to memory:

    root@server:~# docker logs a1b2c3d4e5f6
[INFO] Application started on port 3000
[INFO] Processing request /api/data
[WARN] High memory usage detected...
<--- The logs abruptly stop here without any further application output or error handling --->

    3. Kernel dmesg Output:

    The most definitive evidence of an OOM kill comes from the kernel's message buffer. This is where the OOM killer announces its actions:

    root@server:~# dmesg -T | grep -i 'oom|killed process' | tail -n 5
[Fri May 17 08:30:15 2026] node invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Fri May 17 08:30:15 2026] CPU: 3 PID: 12345 Comm: node Not tainted 5.15.0-89-generic #99-Ubuntu
[Fri May 17 08:30:15 2026] Mem-Info:
[Fri May 17 08:30:16 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_cgroup=docker/a1b2c3d4e5f6...,task_memcg=/docker/a1b2c3d4e5f6.../...,pgsk_memcg=/docker/a1b2c3d4e5f6.../...
[Fri May 17 08:30:16 2026] Killed process 12345 (node) total-vm:4194304kB, anon-rss:2097152kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8192kB oom_score_adj:0

    The oom-killer invocation and the line Killed process <PID> (<processname>) confirm that the kernel terminated the process due to memory exhaustion. The memscgroup line is particularly useful as it often points directly to the Docker container's cgroup.

    Root Cause Analysis

    The "Docker container exited with code 137 OOM killed" error is a direct consequence of the Linux kernel's Out-Of-Memory (OOM) killer terminating a process (or an entire cgroup, like a Docker container) because the system or the cgroup has exhausted its available memory. Understanding the underlying mechanisms is key:

    1. Insufficient Host System Memory: The most straightforward cause. The entire Docker host (VM or bare metal) simply doesn't have enough physical RAM to accommodate all running containers and host processes. When total memory is scarce, the OOM killer steps in to reclaim memory by terminating the largest or highest-scoring process.
    1. Insufficient Docker Container Memory Limits: Docker uses Linux control groups (cgroups) to enforce resource limits on containers. If a specific container is configured with a memory limit (e.g., --memory flag in docker run or memory in docker-compose.yml) that is lower than its actual memory requirements, the OOM killer will terminate that container once it hits its cgroup-defined memory ceiling. This can happen even if the host machine has plenty of free RAM. The kernel prioritizes killing processes within the cgroup that exceeded its limits.
    1. Application Memory Leak or Inefficient Usage: The application running inside the container might have a memory leak, where it continuously consumes more and more RAM over time without releasing it. Alternatively, the application might be poorly optimized, using excessive memory for specific tasks, processing large datasets, or loading entire libraries into memory unnecessarily. A sudden spike in legitimate memory usage (e.g., handling a large number of concurrent requests, processing a massive file upload) can also trigger the OOM killer if the allocated memory limits are too tight.
    1. Swap Space Depletion (or lack thereof): While not directly causing an OOM kill, insufficient or non-existent swap space can exacerbate memory pressure. When physical RAM is exhausted, the kernel typically moves less-used pages to swap. If swap is also full or unavailable, the OOM killer is invoked sooner and more aggressively.
    1. Cgroup V1 vs V2 Behavior: Modern Linux kernels (especially with Ubuntu 22.04 and newer) often use cgroup v2. While the core concept of memory limits and OOM killing remains, the exact reporting and interaction might differ slightly from older v1 systems. Docker transparently handles this, but it's a detail to be aware of for advanced debugging.

    Step-by-Step Resolution

    Resolving OOM kills requires a methodical approach, starting with identifying the immediate cause and then moving towards long-term optimizations.

    1. Confirm the OOM Event and Identify the Culprit

    First, re-verify that an OOM event indeed occurred and pinpoint which process was killed.

    # Check the kernel message buffer for OOM events
dmesg -T | grep -i 'oom|killed process' | less

    Examine the output for lines like oom-kill:constraint=CONSTRAINTMEMCG or Killed process <PID> (<processname>). The mems_cgroup entry often directly names the Docker container's cgroup, confirming it was the target. Note the total-vm and anon-rss values to understand how much memory the process was attempting to use.

    2. Monitor Container and Host Memory Usage

    Before making changes, gather data on current memory consumption.

    a. Real-time Docker Container Stats:

    Use docker stats to see live memory usage, limits, and percentage.

    docker stats --no-stream my-app-container
    CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O     BLOCK I/O   PIDS
a1b2c3d4e5f6   my-app-container   0.50%     1.897GiB / 2GiB       94.85%    1.5MB / 0B  0B / 0B     12

    This output immediately shows if the container is consistently hitting its defined limit.

    b. Host System Memory:

    Check the overall memory usage of the Docker host.

    free -h
                   total        used        free      shared  buff/cache   available
Mem:            7.8Gi       6.5Gi       235Mi       1.2Gi       1.1Gi       100Mi
Swap:           2.0Gi       1.5Gi       500Mi

    If available memory is consistently low, the host itself is under memory pressure.

    c. Advanced Monitoring (e.g., Prometheus/Grafana, cAdvisor):

    For long-term trends and historical data, integrate monitoring tools like cAdvisor, Prometheus, and Grafana. These tools provide invaluable insights into memory usage patterns over time, helping you identify peak loads or slow memory leaks.

    3. Analyze Application Memory Footprint (Inside the Container)

    If the container runs long enough, you can inspect its internal memory usage.

    # Execute a shell inside the running container (if it's not immediately OOM killed)

    Inside the container: apt update && apt install -y procps # Install ps and free if not present ps aux –sort -rss | head -n 5 # Top processes by Resident Set Size (RSS) free -h # Container's view of memory (constrained by cgroups) `

    For language-specific applications: * Node.js: Use process.memoryUsage() or tools like heapdump. * Python: memory_profiler or objgraph. * Java: jstat, jmap (requires JDK inside container or attach from host if compatible). * PHP: memorygetusage().

    4. Increase Docker Container Memory Limits

    This is often the quickest fix if the application legitimately needs more memory than allocated.

    a. For docker run:

    > [!IMPORTANT]

    Stop and remove the old container docker stop my-app-container && docker rm my-app-container

    Run with increased memory (e.g., 2GB physical RAM, no swap) docker run -d –name my-app-container –memory="2g" –memory-swap="2g" my-app:latest `

    b. For docker-compose.yml:

    version: '3.8'
services:
  my-app:
    image: my-app:latest
    ports:
      - "80:3000"
    deploy:
      resources:
        limits:
          memory: 2G # Hard limit for the container
        reservations:
          memory: 1G # Guaranteed memory for the container
    restart: always
    • limits.memory: The maximum amount of memory the container can use. If it exceeds this, the OOM killer will step in.
    • reservations.memory: The amount of memory reserved for the container. This helps with scheduling and ensures the container gets at least this much.

    5. Increase Host System Memory or Swap Space

    If all containers combined are pressuring the host, you need more resources.

    a. Upgrade Physical RAM: The most effective long-term solution. Increase the RAM of your VM or physical server.

    b. Increase Swap Space (Use with Caution): While not a substitute for RAM, swap can prevent hard OOM kills by providing an overflow mechanism. However, relying heavily on swap will degrade performance significantly.

    > [!WARNING]

    Check current swap status sudo swapon –show sudo free -h

    Create a new swap file (e.g., 4GB) sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile

    Make swap persistent across reboots by adding to /etc/fstab echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

    Adjust swappiness (optional, lower value means kernel tries to keep more data in RAM) # Current swappiness (default is 60 on Ubuntu) cat /proc/sys/vm/swappiness # Set swappiness to 10 (less aggressive swapping) sudo sysctl vm.swappiness=10 echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf `

    6. Optimize Application Memory Usage

    This is a fundamental solution for long-term stability and efficiency.

    a. Identify and Fix Memory Leaks: * Use language-specific profiling tools (as mentioned in Step 3) to identify objects that are not being garbage collected or released. * Review application code, especially in loops, data structures, and resource handling (file descriptors, database connections).

    b. Efficient Data Handling: * Process large files or datasets in chunks instead of loading them entirely into memory. * Use streaming APIs where possible. * Optimize database queries to fetch only necessary data.

    c. Configure Application-Specific Memory Settings: * Java: Adjust JVM heap size (-Xmx, -Xms) within the container. * PHP-FPM: Configure phpmemorylimit in php.ini and worker memory limits in FPM pool configurations. * Node.js: Node.js has a default memory limit (often around 1.5-2GB for 64-bit systems) which can be increased using --max-old-space-size.

    # Example for Node.js in Dockerfile or startup script
CMD ["node", "--max-old-space-size=3072", "server.js"] # Allows Node.js to use up to 3GB

    7. Adjust OOM Score (Advanced, Use with Extreme Caution)

    The OOM killer uses an oom_score to decide which process to kill. Lower scores mean lower likelihood of being killed. You can adjust this for containers.

    > [!WARNING]

    For docker run docker run -d –name my-app-container –memory="2g" –memory-swap="2g" –oom-score-adj=500 my-app:latest

    For docker-compose.yml version: '3.8' services: my-app: image: my-app:latest oomscoreadj: 500 # Range from -1000 (least likely) to 1000 (most likely) `

    A value of 0 is default. Negative values make it less likely to be killed, positive values make it more likely. Setting it to -1000 essentially exempts it from OOM killing (unless it's the only process left).

    8. Review Docker Daemon Configuration

    Ensure the Docker daemon itself isn't operating under resource constraints, although this is less common for OOM kills within containers.

    systemctl status docker

    Check /etc/docker/daemon.json for any global default limits or experimental features that might impact memory management. For example, default-ulimits might affect file descriptor limits, indirectly impacting memory use.

    By systematically applying these steps, you can effectively diagnose, resolve, and prevent Docker containers from exiting with the dreaded code 137 OOM killed error, ensuring the stability and reliability of your containerized services.

  • Troubleshooting Redis OOM: ‘Maximum Memory Limit Reached OOM Command Not Allowed’

    Diagnose and resolve the 'Redis maximum memory limit reached OOM command not allowed' error, preventing data writes and service interruptions. Optimize Redis memory usage and configuration for stability.

    Introduction

    Encountering the "Redis maximum memory limit reached OOM command not allowed" error indicates a critical issue where your Redis server has consumed all its allocated memory and is refusing write operations. This typically manifests as sudden application failures, inability to save new data to Redis (e.g., user sessions, cache entries, queue items), and a general degradation of services relying on Redis. While read operations might still function, any attempt to modify data will be met with an error, effectively crippling your application's dynamic functionality. This guide will walk you through diagnosing the root causes and implementing robust solutions to restore stability and prevent future occurrences.

    Symptom & Error Signature

    When Redis hits its configured maxmemory limit and is unable to evict keys according to its maxmemory-policy, it enters an Out-Of-Memory (OOM) state. In this state, by default or with a noeviction policy, it will reject write commands.

    You will typically observe this error in:

    • Application Logs: Errors reported by your application framework (PHP, Node.js, Python, Ruby on Rails) when trying to write to Redis.
    • Redis Server Logs: Direct OOM warnings from the Redis instance itself.
    • Redis CLI: Attempts to execute write commands manually will fail.

    Example Application Log Output (PHP/Predis):

    [2023-10-27 14:35:12] app.ERROR: PredisClientException: OOM command not allowed when used_memory > 'maxmemory'. in /var/www/html/vendor/predis/predis/src/Client.php:370
Stack trace:
#0 /var/www/html/vendor/predis/predis/src/Client.php(303): PredisClient->onErrorResponse()
#1 /var/www/html/vendor/predis/predis/src/Client.php(329): PredisClient->executeCommand()
#2 /var/www/html/app/Services/CacheService.php(80): PredisClient->__call()
#3 /var/www/html/app/Http/Controllers/UserController.php(45): AppServicesCacheService->set()
...

    Example Redis Server Log Output:

    12345:M 27 Oct 2023 14:35:12.678 # OOM command not allowed when used_memory is greater than 'maxmemory' (configured as 512mb)

    Example Redis CLI Interaction:

    redis-cli
127.0.0.1:6379> SET mykey "hello"
(error) OOM command not allowed when used_memory > 'maxmemory'.

    Root Cause Analysis

    The "Redis maximum memory limit reached OOM command not allowed" error stems from a fundamental imbalance between the data stored in Redis and the memory allocated to it. The primary underlying reasons include:

    1. Insufficient maxmemory Configuration: The most common cause is that the maxmemory directive in your redis.conf is set too low for your application's actual data storage needs. As data accumulates, Redis hits this ceiling.
    2. Ineffective maxmemory-policy: Redis's maxmemory-policy dictates how it behaves when the maxmemory limit is reached.
    3. * If set to noeviction (the default for Redis without a specific policy), Redis will stop accepting write commands and return an OOM error.
    4. * If set to an eviction policy (e.g., allkeys-lru, volatile-lru), but there are no keys eligible for eviction, or the eviction process cannot keep up with the incoming writes, the OOM error can still occur.
    5. Application Memory Leaks or Inefficient Usage:
    6. * Unbounded Data Structures: Lists, Sets, or Hashes that continuously grow without proper trimming or expiration.
    7. * Large Keys: Storing extremely large strings, objects, or serialized data.
    8. * Missing Key Expirations: Data that should be temporary (e.g., cache entries, session tokens) is not given a TTL (Time-To-Live) and persists indefinitely.
    9. * High Churn Rates: Rapid creation of many temporary keys without efficient eviction, leading to memory pressure.
    10. Traffic Spikes: A sudden, unexpected surge in application traffic or data writes can quickly push Redis beyond its memory limits if not properly provisioned.
    11. Persistence Overhead: When Redis performs an RDB snapshot or AOF rewrite, it forks a child process. This child process shares memory with the parent via copy-on-write. If maxmemory is set very close to the total system RAM, this forking can momentarily double the memory footprint, potentially triggering system-level OOM killer or making Redis unable to persist. While this usually results in a system OOM, it can exacerbate Redis's memory pressure.

    Step-by-Step Resolution

    Addressing this error requires a methodical approach, starting with diagnosis and moving towards configuration adjustments and application-level optimizations.

    1. Understand Current Redis Memory Usage and Configuration

    Begin by inspecting the current state of your Redis instance.

    • Connect to Redis CLI:
    • `bash
    • redis-cli
    • `
    • Get Memory Information:
    • `bash
    • 127.0.0.1:6379> INFO memory
    • `
    • Pay close attention to:
    • * usedmemoryhuman: Current memory consumption by Redis.
    • * maxmemory_human: The configured maximum memory limit.
    • * maxmemory_policy: The current eviction policy.
    • * memfragmentationratio: Ratio of usedmemoryrss to used_memory. A high ratio (>1.5) indicates fragmentation, wasting memory.
    • Check maxmemory and maxmemory-policy specifically:
    • `bash
    • 127.0.0.1:6379> CONFIG GET maxmemory
    • 127.0.0.1:6379> CONFIG GET maxmemory-policy
    • `
    • Examine Key Space:
    • `bash
    • 127.0.0.1:6379> INFO keyspace
    • `
    • This shows the number of keys and keys with expires per database.

    2. Analyze Redis Key Usage and Data Structures

    Identify which keys or data structures are consuming the most memory.

    • Find Big Keys: The redis-cli --bigkeys command can help identify the largest keys in your instance.
    • `bash
    • redis-cli –bigkeys
    • `
    • This command iterates over all keys and reports the largest keys of each data type (string, list, hash, set, zset).
    • Review Application Code: This is crucial. Examine how your application interacts with Redis:
    • * Are lists or hashes growing indefinitely without trimming or expiration?
    • * Are large objects being stored as single string keys?
    • * Are temporary cache items or sessions being created without EXPIRE times?

    3. Adjust maxmemory Limit (Increase Cautiously)

    If used_memory is consistently at or near maxmemory, and your system has available RAM, increasing the maxmemory limit is often the quickest temporary fix.

    • Identify Available System RAM:
    • `bash
    • free -h
    • `
    • Ensure your server has sufficient free RAM. Do not allocate all available RAM to Redis.
    • Temporarily (Runtime) Increase maxmemory:
    • `bash
    • redis-cli CONFIG SET maxmemory 512mb # Example: Increase to 512MB
    • `
    • > [!WARNING]
    • > This change is not persistent. It will revert to the value in redis.conf upon Redis restart. Use this only for immediate relief while you implement a permanent solution.
    • Permanently Increase maxmemory:
    • 1. Edit Redis Configuration File:
    • `bash
    • sudo nano /etc/redis/redis.conf
    • # Or for a specific instance, e.g., /etc/redis/6379.conf
    • `
    • 2. Locate and Modify maxmemory:
    • Find the maxmemory directive. If it's commented out, uncomment it. Adjust the value based on your server's available RAM and Redis's role.
    • `
    • # Old: # maxmemory <bytes>
    • # New:
    • maxmemory 512mb
    • # Or 1gb, 2gb, etc.
    • `
    • > [!IMPORTANT]
    • > Memory Allocation Best Practices:
    • > * Do not set maxmemory to 100% of your system's RAM. Leave adequate memory for the operating system, other system processes, and especially for Redis's persistence mechanisms (RDB/AOF). When Redis forks a child process for persistence, memory usage can temporarily spike, potentially doubling for RDB snapshots.
    • > * A good starting point is often 50-70% of the total dedicated RAM for the Redis instance.
    • 3. Restart Redis Service:
    • `bash
    • sudo systemctl restart redis-server
    • `
    • For Dockerized Redis:
    • `bash
    • docker restart <rediscontainernameorid>
    • `
    • Ensure your Docker Compose or docker run command for Redis includes the -maxmemory flag or links to a custom redis.conf.

    4. Configure an Effective maxmemory-policy

    If your maxmemory-policy is noeviction, Redis will not automatically free up space. Changing this policy allows Redis to evict keys to make room for new data.

    • Edit Redis Configuration File:
    • `bash
    • sudo nano /etc/redis/redis.conf
    • `
    • Locate and Modify maxmemory-policy:
    • Choose a policy that fits your use case.
    • `
    • # Old: maxmemory-policy noeviction
    • # New (example for general caching):
    • maxmemory-policy allkeys-lru
    • `
    • Common maxmemory-policy options:
    • * noeviction: (Default) Returns errors on write commands when memory limit is reached.
    • allkeys-lru: Evicts the least recently used (LRU) keys among all* keys. Ideal for general-purpose caching.
    • volatile-lru: Evicts LRU keys only among those that have an expire set*. Useful if some keys are essential and should never be evicted.
    • allkeys-lfu: Evicts the least frequently used (LFU) keys among all* keys. (Available since Redis 4.0)
    • volatile-lfu: Evicts LFU keys only among those that have an expire set*.
    • allkeys-random: Evicts random keys among all* keys.
    • volatile-random: Evicts random keys only among those that have an expire set*.
    • volatile-ttl: Evicts keys only among those that have an expire set*, preferring those with the shortest remaining TTL.
    • > [!IMPORTANT]
    • > Carefully consider your maxmemory-policy. If Redis is used for critical data that should not be lost (rather than just a cache), then noeviction might be appropriate, and the real solution lies in increasing maxmemory or optimizing data structures. If used as a cache, an LRU or LFU policy is usually preferred.
    • Restart Redis Service:
    • `bash
    • sudo systemctl restart redis-server
    • `

    5. Optimize Application-Level Redis Usage

    This is often the most sustainable long-term solution.

    • Implement Key Expiration (TTL): For cache entries, sessions, or temporary data, always set a TTL using EXPIRE or SETEX.
    • `python
    • # Example in Python using redis-py
    • redisclient.setex("user:session:123", 3600, "sessiondata_here") # Expires in 1 hour
    • `
    • Trim Growing Lists: If using lists as queues or activity streams, use LTRIM to keep them bounded.
    • `bash
    • 127.0.0.1:6379> RPUSH mylist item1 item2 item3
    • 127.0.0.1:6379> LTRIM mylist 0 99 # Keep only the last 100 items
    • `
    • Use Hashes for Related Data: Instead of storing many small individual keys, group related attributes into a single Hash data type. This reduces key overhead.
    • `bash
    • # Instead of:
    • # SET user:1:name "Alice"
    • # SET user:1:email "[email protected]"
    • # Use:
    • HMSET user:1 name "Alice" email "[email protected]"
    • `
    • Avoid Storing Large Blobs: If you need to store very large objects, consider if Redis is the right tool, or if they can be stored in a file system/object storage and Redis only stores the pointer/metadata.
    • Code Review for Anti-Patterns: Look for patterns that fetch all keys (KEYS * – avoid in production!), or add data without any expiration or trimming logic.
    • Sharding or Clustering: For very high-scale applications, consider distributing your data across multiple Redis instances (sharding) or using Redis Cluster.

    6. Monitor and Alert

    Proactive monitoring can prevent future OOM issues.

    • Monitor Key Metrics: Use monitoring tools (e.g., Prometheus/Grafana, Datadog, New Relic) to track:
    • * used_memory
    • * maxmemory
    • * memfragmentationratio
    • * evicted_keys (indicates your eviction policy is working)
    • * keyspacehits and keyspacemisses
    • * System-level RAM usage.
    • Set Up Alerts: Configure alerts for high memory usage (e.g., when used_memory exceeds 80-90% of maxmemory) to get notified before an OOM condition occurs.

    7. Consider Redis Persistence Settings

    If using RDB snapshots or AOF persistence, ensure these are configured thoughtfully regarding memory.

    • RDB and AOF Rewrite: As mentioned in Root Cause Analysis, these operations involve forking. Ensure maxmemory is not so close to total system RAM that a fork causes a system OOM.
    • vm.overcommitmemory: On Linux, ensure vm.overcommitmemory is set to 1 (the default in many systems) to allow the kernel to overcommit memory. This is crucial for Redis's copy-on-write mechanism during persistence. You can check with sysctl vm.overcommit_memory and set it persistently in /etc/sysctl.conf.
    • `bash
    • # Check current value
    • sysctl vm.overcommit_memory

    If not 1, set it (temporary) sudo sysctl vm.overcommit_memory=1

    To make it persistent, edit /etc/sysctl.conf sudo nano /etc/sysctl.conf # Add or modify the line: vm.overcommit_memory = 1 # Then apply changes sudo sysctl -p ` > [!WARNING] > While vm.overcommit_memory=1 is generally recommended for Redis persistence, understand that it tells the kernel to allow processes to allocate more memory than physically available. This relies on processes not actually using all that memory simultaneously. A misbehaving process could still trigger system OOM.

    By systematically applying these steps, you can effectively resolve the "Redis maximum memory limit reached OOM command not allowed" error and establish a more robust, performant, and stable Redis environment.