import sys
import heapq

def main() -> None:
lower: list[int] = [] # Max-heap represented by negated values
upper: list[int] = [] # Min-heap

for raw_line in sys.stdin:
    line = raw_line.rstrip("\r\n")

    if not line.strip():
        continue

    try:
        value = int(line)
    except ValueError:
        print(f"INVALID: {line}", file=sys.stderr)
        continue

    if not lower or value <= -lower[0]:
        heapq.heappush(lower, -value)
    else:
        heapq.heappush(upper, value)

    if len(lower) > len(upper) + 1:
        heapq.heappush(upper, -heapq.heappop(lower))
    elif len(upper) > len(lower):
        heapq.heappush(lower, -heapq.heappop(upper))

    if len(lower) > len(upper):
        print(-lower[0])
    else:
        left = -lower[0]
        right = upper[0]
        print(f"{(left + right) / 2:.1f}")

if name == "main":
main()

!/usr/bin/env python3

import sys
import heapq

def print_even_average(a, b):
total = a + b
sign = "-" if total < 0 else ""
total_abs = abs(total)
integer_part = total_abs // 2
fractional_part = "5" if total_abs % 2 else "0"
print(f"{sign}{integer_part}.{fractional_part}")

def main():
lower = [] # max-heap via negative values
upper = [] # min-heap

for line in sys.stdin:
    original = line.rstrip("\n")

    if original.strip() == "":
        continue

    try:
        value = int(original.strip())
    except ValueError:
        print(f"INVALID: {original}", file=sys.stderr)
        continue

    if not lower or value <= -lower[0]:
        heapq.heappush(lower, -value)
    else:
        heapq.heappush(upper, value)

    if len(lower) > len(upper) + 1:
        heapq.heappush(upper, -heapq.heappop(lower))
    elif len(upper) > len(lower):
        heapq.heappush(lower, -heapq.heappop(upper))

    if (len(lower) + len(upper)) % 2 == 1:
        print(-lower[0])
    else:
        print_even_average(-lower[0], upper[0])

if name == "main":
main()

1. Bugs

Primary bug: updating an existing key does not unlink its node first

In put, the existing node is inserted at the front while it is still linked at its old position:

if key in self.map:
    node = self.map[key]
    node.value = value
    self._insert_front(node)  # node is still in the list
    return

A linked-list node cannot occupy two positions simultaneously. _insert_front overwrites node.prev and node.next, but the node’s former neighbors still point to it. This creates cycles, stale links, or disconnected portions of the list.

The fix is to call _remove(node) before _insert_front(node).

Secondary edge-case bug: capacity zero

For capacity == 0, this block attempts to evict self.tail.prev, which is the head sentinel:

lru = self.tail.prev  # self.head
self._remove(lru)

The head sentinel has prev = None, so _remove fails. If zero capacity is allowed, put should be a no-op.

2. Why the observed sequence fails

Using H and T for the head and tail sentinels:

cache.put(1, 1)

List:

H <-> 1 <-> T

After:

cache.put(2, 2)

List:

H <-> 2 <-> 1 <-> T

After:

cache.get(1)

Key 1 is moved to the front:

H <-> 1 <-> 2 <-> T

Now:

cache.put(2, 20)

The implementation calls _insert_front(2) without first removing node 2.

Before insertion, relevant pointers are:

H.next = 1
1.prev = H
1.next = 2
2.prev = 1
2.next = T
T.prev = 2

_insert_front(2) performs:

2.next = H.next      # 2.next = 1
2.prev = H
H.next.prev = 2      # 1.prev = 2
H.next = 2

The resulting forward pointers include a cycle:

H -> 2 -> 1 -> 2 -> 1 -> ...

Meanwhile:

T.prev = 2

still claims that 2 is the least-recently-used node. The list is no longer internally consistent because 2.next is now 1, not T.

Then:

cache.put(3, 3)

uses:

lru = self.tail.prev

which incorrectly selects key 2. It removes key 2 even though updating key 2 should have made it the most recently used key.

The correct recency before inserting key 3 is:

MRU: 2, 1 :LRU

Therefore, inserting key 3 should evict key 1, leaving keys 2 and 3.

The corruption can also cause later evictions to follow stale links or attempt to delete a key that is no longer in map.

3. Minimal patch

from dataclasses import dataclass

@dataclass
class Node:
    key: int
    value: int
    prev: 'Node' = None
    next: 'Node' = None


class LRUCache:
    def __init__(self, capacity: int):
        self.capacity = capacity
        self.map = {}
        self.head = Node(0, 0)
        self.tail = Node(0, 0)
        self.head.next = self.tail
        self.tail.prev = self.head

    def _remove(self, node: Node):
        node.prev.next = node.next
        node.next.prev = node.prev

    def _insert_front(self, node: Node):
        node.next = self.head.next
        node.prev = self.head
        self.head.next.prev = node
        self.head.next = node

    def get(self, key: int) -> int:
        if key not in self.map:
            return -1

        node = self.map[key]
        self._remove(node)
        self._insert_front(node)
        return node.value

    def put(self, key: int, value: int) -> None:
        if self.capacity <= 0:
            return

        if key in self.map:
            node = self.map[key]
            node.value = value
            self._remove(node)          # Required fix
            self._insert_front(node)
            return

        if len(self.map) == self.capacity:
            lru = self.tail.prev
            self._remove(lru)
            del self.map[lru.key]

        node = Node(key, value)
        self.map[key] = node
        self._insert_front(node)

The essential fix is the added _remove(node). The capacity check handles the separate zero-capacity edge case.

4. Three targeted tests

Examples using pytest:

def test_update_existing_key_changes_recency():
    cache = LRUCache(2)

    cache.put(1, 1)
    cache.put(2, 2)
    cache.put(1, 10)  # Key 1 becomes MRU.
    cache.put(3, 3)   # Key 2 should be evicted.

    assert cache.get(2) == -1
    assert cache.get(1) == 10
    assert cache.get(3) == 3

This catches both failure to move an updated key and failure to preserve its new value.

def test_update_non_front_node_after_get():
    cache = LRUCache(2)

    cache.put(1, 1)
    cache.put(2, 2)
    assert cache.get(1) == 1       # Order: 1, 2

    cache.put(2, 20)               # Correct order: 2, 1
    cache.put(3, 3)                # Must evict 1

    assert cache.get(1) == -1
    assert cache.get(2) == 20
    assert cache.get(3) == 3

This directly covers the reported sequence. Updating key 2 makes it MRU, so key 1, not key 2, must be evicted.

def test_zero_capacity_never_stores_or_evicts_sentinel():
    cache = LRUCache(0)

    cache.put(1, 1)
    cache.put(2, 2)

    assert cache.get(1) == -1
    assert cache.get(2) == -1
    assert cache.map == {}

This catches the sentinel-removal failure for zero-capacity caches.

1. Precise bug

The bug is in put() when updating an existing key:

if key in self.map:
    node = self.map[key]
    node.value = value
    self._insert_front(node)
    return

The node is inserted at the front without first being removed from its current position.

That means the same Node object becomes linked into the list twice, or worse, creates a cycle. After that, tail.prev may no longer point to the true least-recently-used node, so eviction can delete the wrong key. The linked list and map go out of sync.

There is also an edge-case bug for capacity == 0: the cache will try to evict head, the sentinel node.

2. Why the failure occurs

Initial state:

head <-> tail

After:

cache.put(1, 1)
cache.put(2, 2)

List is:

head <-> 2 <-> 1 <-> tail

2 is most recently used, 1 is least recently used.

After:

cache.get(1)

1 is removed and moved to front:

head <-> 1 <-> 2 <-> tail

Now 1 is MRU, 2 is LRU.

Then:

cache.put(2, 20)

Because key 2 already exists, the code updates the value and calls:

self._insert_front(node)

But node is still currently at the end of the list.

Before update:

head <-> 1 <-> 2 <-> tail

The buggy _insert_front(2) rewires 2 as if it were detached. It sets:

node.next = self.head.next   # 2.next = 1
node.prev = self.head        # 2.prev = head
self.head.next.prev = node   # 1.prev = 2
self.head.next = node        # head.next = 2

Now the list becomes corrupted:

head -> 2 -> 1 -> 2 -> 1 -> ...

There is a cycle between 2 and 1.

Also, tail.prev still points to 2, even though the logical order is now invalid.

Then:

cache.put(3, 3)

The cache thinks it is full, so it evicts:

lru = self.tail.prev

But tail.prev is stale/corrupted. It may point to the wrong node. In this sequence it points to 2, even though after updating 2, the intended LRU should be 1.

So the cache evicts the wrong key and leaves the linked list further corrupted.

3. Minimal patch

from dataclasses import dataclass

@dataclass
class Node:
    key: int
    value: int
    prev: 'Node' = None
    next: 'Node' = None

class LRUCache:
    def __init__(self, capacity: int):
        self.capacity = capacity
        self.map = {}
        self.head = Node(0, 0)
        self.tail = Node(0, 0)
        self.head.next = self.tail
        self.tail.prev = self.head

    def _remove(self, node: Node):
        node.prev.next = node.next
        node.next.prev = node.prev

    def _insert_front(self, node: Node):
        node.next = self.head.next
        node.prev = self.head
        self.head.next.prev = node
        self.head.next = node

    def get(self, key: int) -> int:
        if key not in self.map:
            return -1

        node = self.map[key]
        self._remove(node)
        self._insert_front(node)
        return node.value

    def put(self, key: int, value: int) -> None:
        if self.capacity <= 0:
            return

        if key in self.map:
            node = self.map[key]
            node.value = value
            self._remove(node)          # Fix: detach before moving to front
            self._insert_front(node)
            return

        if len(self.map) == self.capacity:
            lru = self.tail.prev
            self._remove(lru)
            del self.map[lru.key]

        node = Node(key, value)
        self.map[key] = node
        self._insert_front(node)

The essential fix is:

self._remove(node)
self._insert_front(node)

when updating an existing key.

4. Three targeted test cases

Test 1: Updating an existing key should move it to MRU and evict the other key

def test_update_existing_moves_to_front():
    cache = LRUCache(2)

    cache.put(1, 1)
    cache.put(2, 2)
    cache.get(1)
    cache.put(2, 20)
    cache.put(3, 3)

    assert cache.get(1) == -1
    assert cache.get(2) == 20
    assert cache.get(3) == 3

This catches the exact bug from the failing sequence. Since put(2, 20) refreshes key 2, key 1 should be evicted when inserting 3.

Test 2: Updating the current LRU should prevent it from being evicted

def test_update_lru_makes_it_mru():
    cache = LRUCache(2)

    cache.put(1, 1)
    cache.put(2, 2)

    # Current order: 2 MRU, 1 LRU
    cache.put(1, 10)

    # Now order should be: 1 MRU, 2 LRU
    cache.put(3, 3)

    assert cache.get(2) == -1
    assert cache.get(1) == 10
    assert cache.get(3) == 3

With the buggy implementation, put(1, 10) corrupts the list because 1 is reinserted without removal.

Test 3: Repeated updates should not corrupt the list or create stale values

def test_repeated_updates_existing_key():
    cache = LRUCache(2)

    cache.put(1, 1)
    cache.put(2, 2)
    cache.put(2, 20)
    cache.put(2, 200)

    assert cache.get(2) == 200

    cache.put(3, 3)

    assert cache.get(1) == -1
    assert cache.get(2) == 200
    assert cache.get(3) == 3

This catches cycles/self-linking from repeatedly inserting an already-linked node at the front.

Optionally, you can also add an invariant helper to catch linked-list corruption directly:

def assert_cache_integrity(cache):
    seen_nodes = set()
    seen_keys = []

    node = cache.head.next
    while node is not cache.tail:
        assert id(node) not in seen_nodes, "Cycle detected in linked list"
        seen_nodes.add(id(node))
        seen_keys.append(node.key)

        assert node.key in cache.map
        assert cache.map[node.key] is node

        node = node.next

    assert set(seen_keys) == set(cache.map.keys())
    assert len(seen_keys) == len(cache.map)

Then call assert_cache_integrity(cache) after each operation in the tests.

Incident assessment

Treat this as an ingest write-path outage introduced by the deploy 25 minutes ago.

The service is still accepting and buffering messages, but no completed batches have reached object storage for 23 minutes. Memory growth is consistent with batches being retained after a failed flush. The immediate goals are:

1. Restore durable writes.
2. Stop sending new traffic to unhealthy instances.
3. Preserve buffered data where safely possible.
4. Verify downstream ETL freshness recovers.
5. Determine whether the missing 23-minute interval can be replayed.

Prioritized hypotheses

H1 — Gzip writer pooling returns a closed writer, causing flush failure

Priority: Critical / highest confidence

Evidence:

DEBUG flush started batch=8812
ERROR write failed batch=8812 err="gzip: writer closed"

The recent deployment added gzip writer pooling. A gzip.Writer that has been closed must be reset correctly before reuse. Likely failure modes include:

• The writer is returned to the pool after Close() and reused without calling Reset(destination).
• The writer is reset to an invalid or previous destination.
• The same writer is returned to the pool before the current flush is finished.
• A defer writer.Close() runs after the writer has already been returned to the pool.
• The pool contains a wrapper that permanently records a closed state.

This directly explains why object-store PUT errors remain zero: gzip generation fails before the object-store request starts.

Confirmation evidence

• Stack trace or source inspection shows the error originates during gzip Write, Flush, or Close, before object upload.
• Reproduction with two consecutive batches shows the first succeeds and the second gets gzip: writer closed.
• Disabling pooling or always creating a fresh gzip.NewWriter(...) restores writes.
• Code returns a closed writer to sync.Pool without a valid Reset.

Falsification evidence

• The error is generated by a distinct non-pooled writer.
• Every checkout from the pool demonstrably calls Reset with a live destination before use.
• A fresh, non-pooled gzip writer fails identically.
• Object-store upload begins and fails elsewhere despite the logged gzip error.

H2 — A failed flush leaves the batch permanently marked already flushing

Priority: Critical / high confidence

Evidence:

ERROR write failed batch=8812 err="gzip: writer closed"
...
DEBUG flush skipped batch=8812 reason="already flushing"

The flush state is likely not cleared on the error path. The same failed batch remains active, retains its messages, and rejects subsequent flush attempts. This explains:

• ingest_messages_written_total being flat.
• Batch size remaining around 50,000 and continuing to grow.
• Process memory rising from 400 MB to 1.9 GB.
• No object-store PUT errors.
• The service continuing to accept messages while making no durable progress.

Confirmation evidence

• Source inspection finds flushing = false only on the success path.
• Goroutine dump shows no active flush goroutine despite the batch reporting already flushing.
• Instrumentation shows flush_in_progress=1 indefinitely after write failed.
• A unit test injecting a gzip error leaves the batch in flushing state and prevents retry.

Falsification evidence

• The flag is unconditionally cleared with defer under the same lock.
• A flush goroutine is still actively running or blocked.
• Later flush attempts actually start on the same batch after the error.

H3 — Removing the timer made flushes dependent on new messages and introduced a threshold boundary bug

Priority: High

Evidence:

count=50000
flush skipped ... reason="threshold not met"
count=50001
flush started

The condition appears to be count > 50000, not count >= 50000. More importantly, the 60-second ticker was removed and flush checks now occur only in onMessage(). Consequently:

• A low-volume or idle batch may never flush after 60 seconds.
• Data cannot meet the intended 60-second object creation cadence without a periodic trigger.
• The exact count boundary is off by one.
• If a flush error occurs and no robust retry scheduler exists, there is no independent mechanism to retry.

The boundary bug alone does not explain the 23-minute outage because message 50,001 starts a flush. The architectural removal of the timer makes the failure less recoverable and creates a separate freshness bug.

Confirmation evidence

• Code contains a condition similar to if count > maxBatch || age > maxAge exclusively within onMessage.
• There is no background timer, scheduler, or idle-batch flush.
• A batch receiving no further messages remains open beyond 60 seconds.
• A test at exactly 50,000 messages does not flush.

Falsification evidence

• Another timer or scheduler still flushes idle batches.
• The logged “threshold not met” is caused by a different threshold configuration.
• Exactly 50,000 is intentionally below the configured limit.

H4 — A race permits gzip writer reuse while a prior flush is still using it

Priority: Medium

This is a variant or contributing cause of H1. If flushes are asynchronous, a pooled writer may be put back too early and acquired by another batch. The apparent closed state could be caused by concurrent ownership rather than simply missing Reset.

Confirmation evidence

• Race detector reports access to the same gzip writer from multiple goroutines.
• Pool Put occurs before all writes and Close operations complete.
• Logs augmented with writer identity show two active flushes using the same writer.
• The problem is intermittent under concurrent load but absent in serial tests.

Falsification evidence

• Writer checkout/use/close/reset/return is fully scoped to one synchronous flush.
• Pool return only happens after upload completion, with no references retained.
• The issue reproduces deterministically in a single-threaded two-batch test.

Execution plan

1. Declare and stabilize the incident

1. Mark the latest deployment as suspect and freeze further unrelated deploys.
2. Record:
   - Deployment version and timestamp.
   - First timestamp where ingest_messages_written_total stopped.
   - Affected instance IDs.
   - Last successfully written object key and timestamp.
3. Page or notify the ingest and ETL owners.
4. Start an incident timeline.

Example commands:

kubectl -n logs get deploy ingest -o wide
kubectl -n logs rollout history deploy/ingest
kubectl -n logs get pods -l app=ingest -o wide
git log -n 5 --oneline
git show --stat HEAD

Why: This prevents compounding changes and establishes whether the outage aligns exactly with the deploy.

2. Immediately stop exposing the broken version to new traffic

The fastest safe mitigation is to bring up the last known-good version, verify it can write, and then shift traffic to it.

Do not begin by killing all current instances simultaneously. They may contain the only in-memory copy of the last 23 minutes.

Preferred sequence

1. Scale up or create a canary instance running the previous image.
2. Send a controlled test stream to that instance.
3. Verify a gzip object is successfully created and readable.
4. Shift producer traffic to known-good instances.
5. Remove broken instances from service discovery/readiness so they receive no additional messages.
6. Keep broken instances alive temporarily for evidence collection and possible buffer recovery, subject to memory limits.

Example:

# Capture current deployment before changing it.
kubectl -n logs get deploy ingest -o yaml > /tmp/ingest-broken-deploy.yaml

# Identify previous ReplicaSet/image.
kubectl -n logs rollout history deploy/ingest
kubectl -n logs get rs -l app=ingest \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image,REPLICAS:.status.replicas

# Start the previous version as a separate recovery/canary deployment if possible.
kubectl -n logs create deployment ingest-rollback-canary \
  --image=<previous-known-good-image>
kubectl -n logs scale deployment ingest-rollback-canary --replicas=1

If traffic routing only supports the primary deployment, perform:

kubectl -n logs rollout undo deployment/ingest --to-revision=<previous-revision>
kubectl -n logs rollout status deployment/ingest --timeout=5m

Set rollout strategy to avoid taking all instances down at once:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Why: The service is currently acknowledging or accepting messages without durable writes. Stopping further accumulation takes precedence over preserving every broken process indefinitely.

3. Protect against memory exhaustion

The broken instances have risen to 1.9 GB and may be approaching OOM. Once known-good capacity is receiving new traffic:

1. Remove broken pods from the load balancer.
2. Capture diagnostics quickly.
3. Do not leave them running indefinitely if memory continues to grow.
4. If a broken instance is close to its memory limit, prioritize process diagnostics and then terminate it in a controlled way.

Capture:

kubectl -n logs top pods -l app=ingest
kubectl -n logs logs <broken-pod> --since=30m > /tmp/<pod>-logs.txt
kubectl -n logs get pod <broken-pod> -o yaml > /tmp/<pod>-state.yaml
kubectl -n logs exec <broken-pod> -- kill -QUIT 1

For a Go service, SIGQUIT commonly emits goroutine stacks to stderr. Retrieve them immediately:

kubectl -n logs logs <broken-pod> --since=5m > /tmp/<pod>-goroutines.txt

If profiling is enabled:

curl -sS http://<pod>:<admin-port>/debug/pprof/goroutine?debug=2 \
  > /tmp/goroutines.txt
curl -sS http://<pod>:<admin-port>/debug/pprof/heap \
  > /tmp/heap.pb.gz
go tool pprof -top /tmp/heap.pb.gz

Why: This distinguishes a stuck logical state from an active blocked flush and preserves evidence before rollback destroys it.

4. Determine whether buffered messages can be recovered

Before terminating a broken pod, inspect the implementation and available admin endpoints for any supported operation that:

• Dumps the active batch to disk.
• Retries a failed flush.
• Rotates the current batch.
• Clears a stuck flush state.
• Gracefully drains and flushes.

Commands:

git grep -nE 'flush|drain|rotate|admin|debug|signal|shutdown' -- .
git grep -nE 'SIGUSR|SIGHUP|SIGTERM' -- .

Only use an existing, tested recovery mechanism. Do not mutate process memory, attach an unsafe debugger, or manually clear a flag in production.

If graceful shutdown invokes the same broken gzip path, it is unlikely to save the batch. Test it on one noncritical/canary instance before applying it broadly.

Document the likely loss window:

• Start: timestamp of the first message after the last successful object.
• End: timestamp when traffic reaches the rolled-back version.
• Scope: messages accepted only into volatile memory on terminated broken instances.
• Recovery possibility: producer retry, upstream replay, or any existing source retention. Do not assume replay exists.

Investigation actions

5. Correlate the outage with the deployment and individual pods

Query metrics by version and instance:

sum by (version) (rate(ingest_messages_received_total[5m]))
sum by (version) (rate(ingest_messages_written_total[5m]))
sum by (instance, version) (process_memory_mb)
sum by (instance, version) (ingest_batch_size_current)
increase(object_store_put_errors_total[30m])

Also calculate the gap:

sum(rate(ingest_messages_received_total[5m]))
-
sum(rate(ingest_messages_written_total[5m]))

Inspect logs:

kubectl -n logs logs -l app=ingest --since=30m --prefix \
  | grep -E 'flush|gzip|write failed|opened new batch|already flushing'

Expected result: Only the new version has flat writes, rising memory, and gzip: writer closed errors.

6. Inspect the exact diff

Use the deployment revisions rather than relying only on the summary:

git diff <previous-commit>..<deployed-commit> -- \
  '*/batch*' '*/flush*' '*/gzip*' '*/ingest*'
git show <deployed-commit>

Focus on:

• Flush threshold operators: > versus >=.
• Age comparison and time units.
• Removal of ticker lifecycle.
• Whether checks happen only in onMessage.
• How flushing is set and cleared.
• Whether cleanup is deferred before any error can occur.
• Whether failed batches are retained, retried, dropped, or replaced.
• sync.Pool.Get() and Put() ordering.
• Calls to gzip.Writer.Reset.
• Calls to Close and whether Close occurs more than once.
• Whether a writer or output buffer escapes to an async goroutine.
• Whether the output buffer is reused before object upload finishes.

Useful searches:

git grep -n 'sync.Pool'
git grep -n 'gzip.NewWriter\|gzip.NewWriterLevel\|\.Reset('
git grep -n 'flushing'
git grep -n '50000\|maxBatch\|flushInterval\|maxAge'

7. Reproduce locally with focused tests

Run or add tests that isolate each failure.

Test A: pooled writer reuse

1. Acquire writer.
2. Compress batch A.
3. Close and return it.
4. Acquire it again.
5. Reset it onto a new destination.
6. Compress batch B.
7. Decompress both outputs and verify contents.

go test ./... -run 'TestGzipPoolReuse' -count=100

Test B: flush failure resets state

Inject a compressor that returns gzip: writer closed or another write error.

Assertions:

• flushing becomes false after the failed attempt.
• The batch remains available for retry.
• A subsequent flush can start.
• Messages are not counted as written.
• No duplicate object is committed.

go test ./... -run 'TestFlushErrorClearsState|TestFlushRetry' -count=100

Test C: exact threshold

go test ./... -run 'TestFlushAtMaxBatchSize'

Verify flush starts at count == 50000, not only at 50,001.

Test D: idle time flush

1. Add one message.
2. Advance a fake clock beyond 60 seconds.
3. Add no more messages.
4. Verify the batch flushes.

go test ./... -run 'TestIdleBatchFlushesByAge'

Test E: concurrency and ownership

go test -race ./... -run 'TestConcurrentFlush|TestGzipPool' -count=50

Likely root cause

The most likely root cause is a compound regression:

1. The gzip pooling implementation reuses a gzip.Writer after it has been closed without correctly resetting it, or returns it to the pool before its lifecycle is complete.
2. The resulting gzip: writer closed error occurs before any object-store PUT, explaining why PUT errors remain zero.
3. The flush error path fails to clear the batch’s flushing state.
4. All later attempts are skipped as already flushing, so the active batch continues retaining messages and memory grows.
5. Removing the periodic ticker eliminates independent time-based flush and retry opportunities.
6. The threshold check also appears to be off by one because 50,000 is reported as “threshold not met” while 50,001 starts flushing.

The gzip pooling defect is the initiating failure; the stuck flush state turns it into a sustained outage; removal of the ticker and the boundary bug reduce resilience and violate freshness behavior.

Safe rollback strategy

Preferred immediate action: rollback the full deploy

Rollback both changes together rather than trying to selectively configure around the failure:

• Restore the previous timer-based flush logic.
• Remove/disable gzip writer pooling.
• Restore the previous known-good error handling.

A rolling rollback is lower risk than an emergency untested hotfix.

kubectl -n logs rollout undo deployment/ingest --to-revision=<known-good>
kubectl -n logs rollout status deployment/ingest --timeout=5m

Requirements:

• maxUnavailable: 0
• At least one old-version pod verified healthy before shifting all traffic.
• Readiness should require that the service is able to initialize its write path, not merely listen on TCP.
• Watch memory and write counters continuously.

Important data-preservation caveat

Restarting broken processes may discard their in-memory batches. However, leaving them in service causes continued accumulation and risks a larger loss through OOM. Therefore:

1. Bring up known-good capacity first.
2. Route new traffic away from broken instances.
3. Attempt only supported recovery/drain operations.
4. Capture diagnostics.
5. Terminate unrecoverable broken instances before OOM.

Hotfix strategy

Develop the hotfix after rollback has stabilized production. Keep it minimal and independently reviewable.

Hotfix contents

1. Temporarily remove gzip pooling
   go zw := gzip.NewWriter(dst)
   Favor correctness over allocation reduction until lifecycle tests are comprehensive.

2. Clear flush state on every exit path
   Use a narrow ownership model and unconditional cleanup, generally with defer, while avoiding holding a mutex during network I/O.

Conceptually:

```go
if batch.flushing {
return
}
batch.flushing = true

defer func() {
lock.Lock()
batch.flushing = false
lock.Unlock()
}()
```

Exact locking must match the existing design. The key invariant is that success, compression errors, upload errors, cancellation, and panic-safe recovery cannot leave the state permanently set.

3. Retain failed batches for bounded retry
   - Retry with backoff.
   - Do not accept unbounded memory growth.
   - Expose retry/error metrics.
   - Avoid duplicate object commits through deterministic object keys or explicit commit semantics.

4. Restore an independent periodic flush trigger
   - Flush at 60 seconds even when no new message arrives.
   - Stop the ticker cleanly on shutdown.
   - Ensure timer-triggered and size-triggered flushes share the same synchronized tryFlush path.

5. Fix the threshold
   go count >= maxBatchMessages
   Confirm whether byte threshold should also use >=.

6. Add observability
   - ingest_flush_attempts_total
   - ingest_flush_failures_total{stage="gzip|upload"}
   - ingest_flush_in_progress
   - ingest_oldest_buffered_message_age_seconds
   - ingest_buffered_messages
   - ingest_buffered_bytes
   - ingest_last_successful_flush_timestamp_seconds

7. Add a backpressure or fail-safe policy
   Once buffered bytes or oldest-message age exceeds a safe limit, the service should stop claiming readiness or stop accepting unbounded traffic rather than silently buffering until OOM. The exact behavior must match producer retry semantics.

Post-mitigation validation

Immediate service validation

Within one flush interval after rollback:

1. ingest_messages_written_total resumes increasing.
2. Receive and write rates converge.
3. New object-store files appear.
4. Files are nonempty, valid gzip, and contain valid newline-delimited JSON.
5. Batch count and memory stop growing.
6. No gzip: writer closed or repeated already flushing messages appear.

Commands:

# Metrics
curl -sS http://<metrics-endpoint>/metrics \
  | grep -E 'ingest_messages_(received|written)_total|ingest_batch_size|process_memory'

# Object listing
object-store-cli ls <bucket>/<prefix>/ --sort-by-time | tail -20

# Validate a new object
object-store-cli cp <new-object-uri> /tmp/new-batch.json.gz
gzip -t /tmp/new-batch.json.gz
gzip -dc /tmp/new-batch.json.gz | head
gzip -dc /tmp/new-batch.json.gz | jq -c . >/dev/null

Check that each decompressed line parses independently as JSON, rather than parsing the entire NDJSON document as one JSON value:

gzip -dc /tmp/new-batch.json.gz \
  | while IFS= read -r line; do printf '%s\n' "$line" | jq -e . >/dev/null || exit 1; done

Metrics criteria

Over at least two flush intervals:

• Written rate is approximately equal to received rate.
• Buffered message count trends downward or remains bounded.
• Memory stabilizes near the previous baseline.
• Successful object creation occurs at least every 60 seconds under traffic.
• Idle low-volume input is still flushed within approximately 60 seconds.
• Flush failure count remains zero.
• No pod restarts or OOM kills occur.

ETL validation

Because ETL runs every five minutes:

1. Verify the new files are visible to the next ETL run.
2. Watch ETL discovery and processing logs.
3. Confirm the input freshness metric drops below 20 minutes.
4. Continue monitoring until it returns to the normal range, ideally under 5–10 minutes.
5. Check whether ETL encountered corrupt or partial gzip objects.
6. Identify the exact missing interval and whether files arrive out of order.

Example checks:

etl-cli runs list --since 30m
etl-cli logs <latest-run-id>

Do not close the incident solely because ingest writes resume. Require one successful downstream ETL cycle and a recovered freshness metric.

Hypothesis evidence matrix

Exit criteria

The incident can be considered mitigated when all of the following hold:

• Known-good or fixed ingest instances handle all traffic.
• Write counters increase normally for at least 10 minutes.
• At least two consecutive flushes produce valid gzip NDJSON objects.
• Memory and buffered batch size are stable.
• ETL completes a run consuming the new files.
• ETL input freshness is below the alert threshold and trending normally.
• The potential data-loss window is documented.
• Buffered data on old instances has either been safely recovered or explicitly declared unrecoverable.
• Regression tests for gzip reuse, flush-error cleanup, threshold behavior, idle flushing, and concurrency are committed before reintroducing the optimization.

0) Situation assessment / severity

The alert is consistent with an ingest-side write stall:

• Producers are connected normally.
• ingest_messages_received_total is increasing normally.
• ingest_messages_written_total has been flat for 23 minutes.
• Object store reports no PUT errors.
• Batch size remains around 48k–52k.
• Memory is rising rapidly.
• Logs show the active batch failed to flush with gzip: writer closed, then subsequent flushes are skipped as already flushing.

This strongly suggests the ETL freshness issue is caused by the ingest service no longer producing object-store files, not by the ETL job or object storage.

Primary objective: restore file production quickly and stop unbounded memory growth.

1) Prioritized hypotheses

Hypothesis 1 — Most likely: flush state is stuck after gzip writer error

Theory

The new batch-flush logic starts a flush, encounters:

ERROR write failed batch=8812 err="gzip: writer closed"

Then the batch remains marked as flushing:

DEBUG flush skipped batch=8812 reason="already flushing"

So the service keeps accepting messages into the same or blocked batch but refuses future flush attempts. This explains:

• ingest_messages_written_total flat.
• Memory rising.
• Batch count continuing above threshold.
• No object store PUT errors, because the write fails before/during gzip serialization, before PUT.
• Recent deploy touched both flush logic and gzip writer pooling.

Evidence that confirms

• Logs repeatedly show reason="already flushing" for the same or newer batches after a gzip: writer closed error.
• No later flush completed / batch closed / object put success logs.
• Code path has something like:

batch.flushing = true
err := flush(batch)
if err != nil {
    return err // missing defer/reset
}
batch.flushing = false

or equivalent.

Evidence that falsifies

• The flushing flag is always reset on error.
• New batches are being created and flushed successfully.
• ingest_messages_written_total starts increasing while the same errors continue.

Hypothesis 2 — Very likely: gzip writer pooling reuses a closed or invalid gzip writer

Theory

The deploy added gzip writer pooling. A gzip writer may be returned to the pool after Close() and then reused without Reset(), or the same writer may be used concurrently by multiple flushes.

The Go gzip writer pattern requires something like:

zw := gzipPool.Get().(*gzip.Writer)
zw.Reset(dst)
defer func() {
    zw.Close()
    gzipPool.Put(zw)
}()

If a previously closed gzip writer is used without Reset, writes can fail with:

gzip: writer closed

This would cause the flush to fail before object store upload.

Evidence that confirms

• Code gets a gzip writer from a pool but does not call Reset() before reuse.
• Code calls Close() before all writes are complete.
• Code returns gzip writer to pool before the flush goroutine finishes.
• Race/concurrent use of one pooled writer is possible.
• Error rate began immediately after deploy introducing the pool.

Evidence that falsifies

• The pool always uses fresh gzip writers or resets correctly.
• The gzip: writer closed is coming from a different code path unrelated to the pool.
• Disabling the pool does not resolve write failures.

Hypothesis 3 — Likely: conditional flush logic has incorrect threshold/age predicate

Theory

The deploy removed a periodic ticker and moved flushing into onMessage(). Logs show:

count=50000 age_ms=61005
flush skipped reason="threshold not met"

At count=50000 and age_ms=61005, a sane condition should flush if either count threshold or age threshold is met.

This suggests a bad predicate, for example:

if count > maxCount && age > maxAge { flush() }

instead of:

if count >= maxCount || age >= maxAge { flush() }

Or a strict > instead of >= combined with ordering/timing.

However, the log also shows the next message at count=50001 does start a flush, so this may be a secondary bug or an off-by-one issue rather than the main stall.

Evidence that confirms

• Code uses incorrect && condition instead of ||.
• Code uses > where the configured threshold semantics expect >=.
• Code relies on new messages to trigger flushes, so low-traffic periods can fail to flush by age.
• Tests are missing for exact-boundary thresholds and age-only flush.

Evidence that falsifies

• Predicate is correct.
• The threshold not met log is misleading or logged before final count/age update.
• Flush starts correctly for count and age thresholds in unit tests.

Hypothesis 4 — Possible: flush goroutine deadlock or panic leaves batch locked

Theory

The service may set an already flushing flag, launch a goroutine, then the goroutine deadlocks, panics, or blocks while holding state. The visible error may be a symptom, and the actual reason subsequent flushes skip is that the goroutine never completed cleanup.

Evidence that confirms

• Goroutine dumps show flush goroutines stuck.
• Logs show flush started with no terminal log.
• Mutex/channel operations around the flush path can block forever.
• Panic logs are present around the flush path.

Evidence that falsifies

• Flush goroutine exits but simply fails to clear state.
• No blocked goroutines in flush code.
• State cleanup is missing rather than blocked.

2) Immediate investigative actions

I would run these in parallel, prioritizing customer impact.

A. Confirm active impact and blast radius

Action

Check freshness and write metrics for all ingest instances/pods.

Example commands/tools:

# Metrics dashboard / Prometheus-style queries
rate(ingest_messages_received_total[5m])
rate(ingest_messages_written_total[5m])
ingest_batch_size_current
process_memory_mb
rate(object_store_put_errors_total[5m])
tcp_connections_open

Break down by instance, version, availability zone, and tenant if labels exist.

Why

Determine whether all instances on the new version are wedged or only a subset. This controls mitigation:

• All instances affected: rollback immediately.
• Some instances affected: drain/restart/rollback affected ones first.
• Only one tenant/partition affected: targeted restart may buy time.

Expected evidence

If the new deploy is the cause, instances running the new build should show:

• Normal receive rate.
• Zero write rate.
• Increasing memory.
• Repeated gzip: writer closed / already flushing.

B. Inspect recent deploy state

Action

Identify deployed version and rollout timing.

git log --oneline -5
git show --stat <new_sha>
git show <new_sha> -- path/to/ingest/flush path/to/gzip/pool

Check deployment platform:

kubectl get deploy ingest -o wide
kubectl rollout history deploy/ingest
kubectl get pods -l app=ingest -o wide
kubectl describe deploy ingest

Why

Verify that the alert aligns with the deploy and identify the previous good revision for rollback.

Expected evidence

Deploy occurred 25 minutes ago and all/most affected pods are on the new revision.

C. Query service logs around first failure

Action

Search logs for flush lifecycle events.

# Conceptual log queries
service=ingest level>=ERROR since=30m
service=ingest "gzip: writer closed" since=30m
service=ingest "flush started" since=30m
service=ingest "flush completed" since=30m
service=ingest "flush skipped" "already flushing" since=30m
service=ingest "opened new batch" since=30m

Correlate by batch id.

Why

Need to know whether batches ever recover after a failed flush.

Expected evidence

For affected batch IDs:

opened new batch
queued...
flush started
ERROR write failed err="gzip: writer closed"
flush skipped reason="already flushing"
flush skipped reason="already flushing"
...

No terminal cleanup.

D. Inspect code for flush state cleanup

Action

Review the new flush path.

Likely files:

git diff <previous_good_sha>..<new_sha> -- path/to/ingest
grep -R "already flushing\|flushing\|flush skipped\|flush started" -n .
grep -R "gzip.*Pool\|sync.Pool\|gzip.NewWriter\|Reset\|Close" -n .

Look for:

• batch.flushing = true
• Missing defer to reset state.
• Early return on error.
• Goroutine launching without cleanup.
• Incorrect locking around state changes.
• Reuse of closed gzip writer.
• Pool Put() before Close() or before async work finishes.

Why

This identifies whether a rollback is sufficient and what the hotfix should target.

Evidence to collect

Code snippets showing:

b.flushing = true
err := writeGzip(...)
if err != nil {
    return err
}
b.flushing = false

or:

zw := gzipPool.Get().(*gzip.Writer)
// missing zw.Reset(buf)
...
zw.Close()
gzipPool.Put(zw)

E. Inspect runtime health on affected pods

Action

Check memory and logs per pod.

kubectl top pods -l app=ingest
kubectl logs <pod> --since=30m | grep -E "gzip: writer closed|already flushing|flush started|flush completed"

If available, capture goroutine/profile endpoints:

curl http://<pod>:<debug-port>/debug/pprof/goroutine?debug=2
curl http://<pod>:<debug-port>/debug/pprof/heap

Why

Confirm whether pods are at risk of OOM and whether a stuck goroutine/deadlock exists.

Evidence

• Memory rising.
• Flush goroutines stuck or no flush goroutines active but batch state stuck.
• Large in-memory batch objects dominating heap.

3) Immediate mitigation steps to reduce customer impact

Mitigation priority

1. Stop the bad version from accumulating more unflushed memory.
2. Restore object-store writes.
3. Preserve as much buffered data as possible, if feasible.
4. Avoid making ETL consume corrupt or partial gzip files.

Step 1 — Freeze the rollout

Action

Immediately pause any ongoing rollout or autoscaling to the bad version.

kubectl rollout pause deploy/ingest

If using a deployment pipeline, stop promotion of the current build.

Why

Prevents more bad instances from replacing healthy ones.

Step 2 — Roll back to the previous known-good version

Action

Rollback deployment.

kubectl rollout undo deploy/ingest
kubectl rollout status deploy/ingest

Or explicitly deploy previous image:

kubectl set image deploy/ingest ingest=<previous_good_image>
kubectl rollout status deploy/ingest

Why

The recent deploy changed exactly the two areas implicated by the metrics/logs. Rollback is the fastest safe mitigation.

Important operational note

Do not mass-kill all bad pods simultaneously if they hold large in-memory buffers and producer clients reconnect aggressively. Prefer a controlled rolling rollback if capacity allows.

Recommended rollout settings:

• maxUnavailable: 0
• maxSurge: 25% or higher if cluster capacity allows.
• Confirm new/old-good pods begin writing before terminating all bad pods.

If memory is near OOM on bad pods, prioritize replacing the worst offenders first.

Step 3 — Route traffic away from wedged pods once good pods are available

Action

After old-version pods are healthy, drain bad pods:

kubectl cordon/drain-equivalent-for-service <bad-pod>
# or remove from service endpoints using readiness
kubectl delete pod <bad-pod>

If the service supports readiness toggling:

curl -X POST http://<bad-pod>:<admin-port>/admin/readiness/disable

Then wait for TCP clients to reconnect to healthy pods.

Why

Wedged pods are accepting data but not writing it. Keeping them in rotation increases data freshness delay and memory pressure.

Step 4 — Consider a controlled restart of wedged pods

Action

If rollback deploy does not terminate bad pods quickly enough, restart affected pods one by one:

kubectl delete pod <bad-pod>

Tradeoff

This may lose in-memory buffered messages unless producers retry or the service persists local spool data. However, the current process is not flushing and memory is rising; eventually it may OOM and lose buffers anyway.

Given the alert is about freshness and the batch appears unrecoverable due to stuck state, controlled restart after old-good capacity is restored is acceptable.

Step 5 — Scale out known-good version temporarily

Action

Increase replicas on the known-good version during recovery.

kubectl scale deploy/ingest --replicas=<higher_count>

Why

Once producers reconnect, old-good pods may need to absorb backlog and current traffic. More capacity reduces time to freshness recovery.

4) Likely root cause

The likely root cause is a combination of two bugs introduced in the recent deploy:

1. Gzip writer pooling incorrectly reuses a closed gzip writer, causing flush attempts to fail with:

text gzip: writer closed

Probable implementation issue:
- Missing gzip.Writer.Reset(destination) after taking writer from sync.Pool.
- Returning the writer to the pool after Close() and reusing it without reset.
- Returning the writer to the pool while an async flush still uses it.
- Concurrent reuse of the same writer.

2. Flush state cleanup is not exception/error safe, so after the failed write the batch remains marked already flushing.

Probable implementation issue:
- flushing flag is set before writing.
- On error path, the function returns before clearing the flag.
- Subsequent onMessage() calls see already flushing and skip forever.

A secondary issue is likely present in the new conditional flush logic:

flush skipped reason="threshold not met" count=50000 age_ms=61005

At 50k messages and 61 seconds, that should normally meet count and/or age thresholds. The threshold predicate may be off-by-one or using the wrong boolean operator. This may not be the direct cause of the total stall, because count=50001 did trigger flush, but it is still a correctness bug.

5) Safe rollback strategy

Rollback approach

Use the previous known-good release, because it had:

• Time-based flush ticker.
• No new gzip writer pooling behavior.
• Presumably stable ingest_messages_written_total.

Steps

1. Identify previous good revision/image:

bash kubectl rollout history deploy/ingest git log --oneline

2. Pause current rollout:

bash kubectl rollout pause deploy/ingest

3. Roll back:

bash kubectl rollout undo deploy/ingest

Or:

bash kubectl set image deploy/ingest ingest=<previous_good_image>

4. Monitor rollout:

bash kubectl rollout status deploy/ingest kubectl get pods -l app=ingest -o wide

5. Confirm old-good pods are receiving and writing.

6. Drain/delete remaining bad pods.

Rollback safety concerns

• In-memory batches on bad pods may be lost on termination.
• However, continuing to run wedged pods likely causes more memory growth and eventual OOM.
• If producer retry/ack semantics exist, confirm whether messages are acknowledged before or after object-store persistence.
• If ack-after-persist: producers may retry, low data loss.
• If ack-on-receive: bad pods may have accepted data that can only be recovered from memory, but no evidence suggests they can flush now.

Given no durable local spool is mentioned, rollback is still the least harmful option to restore service.

6) Hotfix strategy if rollback is unavailable or insufficient

If rollback cannot be performed, ship a minimal hotfix. Avoid broad refactors.

Hotfix 1 — Make flush state cleanup error-safe

Ensure flushing is always reset.

Example pattern:

func (b *Batch) Flush(ctx context.Context) error {
    b.mu.Lock()
    if b.flushing {
        b.mu.Unlock()
        return nil
    }
    b.flushing = true
    b.mu.Unlock()

    defer func() {
        b.mu.Lock()
        b.flushing = false
        b.mu.Unlock()
    }()

    if err := b.write(ctx); err != nil {
        return err
    }

    return nil
}

If a failed batch cannot be safely retried, mark it failed and rotate to a new batch rather than leaving the active batch blocked:

if err != nil {
    markBatchFailed(b, err)
    rotateToNewBatch()
    return err
}

Hotfix 2 — Fix gzip writer pooling

Correct pattern:

var gzipPool = sync.Pool{
    New: func() any {
        return gzip.NewWriter(io.Discard)
    },
}

func writeGzip(dst io.Writer, messages [][]byte) error {
    zw := gzipPool.Get().(*gzip.Writer)
    zw.Reset(dst)

    defer gzipPool.Put(zw)

    for _, msg := range messages {
        if _, err := zw.Write(msg); err != nil {
            zw.Close()
            return err
        }
        if _, err := zw.Write([]byte("\n")); err != nil {
            zw.Close()
            return err
        }
    }

    if err := zw.Close(); err != nil {
        return err
    }

    return nil
}

Important details:

• Always call Reset(dst) after Get().
• Do not use a gzip writer concurrently.
• Do not return to pool until all writes and Close() complete.
• Do not use writer after Close() unless reset.
• Consider not pooling gzip writers at all in the emergency hotfix. Simpler is safer:

zw := gzip.NewWriter(dst)
...
return zw.Close()

For an emergency patch, I would remove gzip pooling entirely unless profiling proves it is required.

Hotfix 3 — Restore a periodic flush ticker

The prior architecture flushed every 60 seconds. Reintroduce it or add a background timer as defense-in-depth.

ticker := time.NewTicker(60 * time.Second)
defer ticker.Stop()

for {
    select {
    case <-ticker.C:
        flushEligibleBatches()
    case <-ctx.Done():
        return
    }
}

The onMessage() conditional can remain, but age-based flushing should not depend solely on receiving a new message.

Hotfix 4 — Fix threshold predicate

Expected logic:

shouldFlush := count >= maxBatchCount ||
               bytes >= maxBatchBytes ||
               age >= maxBatchAge

Not:

count > maxBatchCount && age > maxBatchAge

Add logging that reports threshold values:

flush decision count=50000 max_count=50000 age_ms=61005 max_age_ms=60000 should_flush=true

7) Validation steps after mitigation

A. Metrics validation

Within 1–2 minutes after rollback/hotfix:

rate(ingest_messages_written_total[1m]) > 0

Expected:

• ingest_messages_written_total starts increasing.
• ingest_batch_size_current drops periodically after flush.
• process_memory_mb stabilizes or decreases.
• object_store_put_errors_total remains near zero.
• tcp_connections_open remains normal.

Within 5–10 minutes:

• New gzipped files appear in object storage.
• ETL begins consuming new inputs.
• ETL input freshness trends downward from >20m.

Within 20–30 minutes:

• Alert clears.
• Freshness returns below threshold with margin.

B. Log validation

Search for healthy flush lifecycle:

INFO opened new batch id=...
DEBUG flush started batch=...
INFO flush completed batch=... count=... bytes=... object_key=...
INFO opened new batch id=...

Ensure absence or sharp reduction of:

gzip: writer closed
flush skipped reason="already flushing"

Some already flushing logs may be normal if concurrent triggers race, but they must not persist indefinitely for the same batch.

C. Object storage validation

Check that new files are being written at the expected cadence.

Expected:

• Roughly one object per partition/instance per 60 seconds, depending on batching.
• Gzip files are non-empty.
• Gzip files are readable by ETL.

Conceptual checks:

object-store ls <bucket>/<prefix> --since 15m
object-store stat <new-object>
object-store cat <new-object> | gunzip | head

D. ETL validation

Check downstream job input lag and processing.

Metrics/logs:

etl_input_freshness_seconds
etl_files_discovered_total
etl_files_processed_total
etl_last_successful_run_timestamp
etl_parse_errors_total

Expected:

• ETL discovers newly written files on the next 5-minute cycle.
• No spike in parse/gzip errors.
• Freshness decreases.

E. Runtime validation

Check pods:

kubectl top pods -l app=ingest
kubectl get pods -l app=ingest

Expected:

• No crash loops.
• Memory no longer grows unbounded.
• CPU may temporarily rise due to backlog but should normalize.

8) Evidence matrix

9) Tests to add before re-deploying fixed version

Flush state cleanup tests

• Flush success clears flushing.
• Flush gzip error clears flushing.
• Object store error clears flushing.
• Panic in flush path recovers or at least clears readiness/restarts safely.
• A failed batch is retried or rotated according to explicit policy.

Gzip pooling tests

• Writer from pool can be reused for multiple destinations.
• Every pooled writer is reset before use.
• No writes occur after close.
• No concurrent use of a pooled writer.
• Run with race detector in CI:

go test -race ./...

Threshold tests

Test each trigger independently:

• count == maxCount flushes.
• count > maxCount flushes.
• age == maxAge flushes.
• age > maxAge flushes.
• bytes == maxBytes flushes.
• Low traffic still flushes by age due to ticker.

Integration test

Simulate TCP producers sending newline-delimited JSON and assert:

• Files appear in object storage mock every 60 seconds.
• Files are valid gzip.
• Message count written equals message count received.
• Flush errors do not permanently wedge ingestion.

10) Final recommended execution order

1. Declare incident / page owning team if not already done.
2. Pause rollout.
3. Confirm affected pods are on new revision and show flat writes/rising memory.
4. Roll back to previous known-good version.
5. Scale out old-good version if needed.
6. Drain/delete wedged new-version pods after healthy capacity exists.
7. Validate:
   - writes resume,
   - memory stabilizes,
   - object files appear,
   - ETL freshness decreases.
8. Review code diff and confirm root cause.
9. Prepare hotfix:
   - remove or fix gzip writer pooling,
   - guarantee flush cleanup with defer,
   - restore periodic flush ticker,
   - fix threshold predicate.
10. Add regression tests.
11. Deploy hotfix to staging/canary.
12. Canary production with tight metrics gates:
    - nonzero writes,
    - no gzip: writer closed,
    - no persistent already flushing,
    - stable memory,
    - freshness healthy.
13. Resume normal rollout only after canary remains healthy for at least several flush/ETL cycles.

Overall assessment

The endpoints allow unauthenticated callers to write and read files using attacker-controlled paths. The most serious issues are arbitrary file write/read through path traversal, spoofable user identity, symlink attacks, resource-exhaustion risks, and non-atomic file publication.

Findings

1. Arbitrary file write through path traversal

Severity: Critical

user and fileName are inserted into a filesystem path without validation:

path.join('/var/app/uploads', user, fileName)

Values such as:

X-User: ../../somewhere
name: target-file

or:

{"name":"../../../tmp/payload","data":"..."}

can normalize outside /var/app/uploads. Depending on the service account's permissions, an attacker could overwrite application files, configuration, scripts, SSH keys, cron-related files, or files consumed by privileged jobs.

Fix:

• Do not map raw user input directly to filesystem paths.
• Derive the user directory from authenticated server-side identity.
• Generate server-side storage IDs rather than trusting filenames.
• If filenames must be retained, use a strict allowlist and verify containment with path.resolve/path.relative.

Example:

const uploadRoot = path.resolve('/var/app/uploads');

function safeComponent(value, label) {
  if (typeof value !== 'string' || !/^[A-Za-z0-9._-]{1,128}$/.test(value)) {
    throw new Error(`Invalid ${label}`);
  }
  if (value === '.' || value === '..') {
    throw new Error(`Invalid ${label}`);
  }
  return value;
}

function containedPath(root, ...components) {
  const result = path.resolve(root, ...components);
  const relative = path.relative(root, result);

  if (relative === '' || relative.startsWith(`..${path.sep}`) || relative === '..' ||
      path.isAbsolute(relative)) {
    throw new Error('Path escapes storage root');
  }

  return result;
}

Containment checks alone do not prevent symlink attacks; those require separate handling.

2. Arbitrary file read through path traversal

Severity: Critical

The download endpoint has the same issue:

const p = path.join('/var/app/uploads', req.query.user, req.query.name);
res.sendFile(p);

An attacker may request paths containing .. and retrieve files outside the upload root, subject to process permissions. This can expose secrets, environment/configuration files, source code, service credentials, or other users' files.

Fix:

• Authenticate the request.
• Authorize access to a server-side file record.
• Resolve files by opaque database/object-storage ID.
• If filesystem lookup is unavoidable, validate components, enforce canonical containment, and reject symlinks.
• Run the process with minimal filesystem read permissions.

3. X-User is not authentication

Severity: Critical

Any caller can claim any identity:

const user = req.header('X-User') || 'anonymous';

Internal reachability does not make the header trustworthy. An internal user, compromised workload, SSRF primitive, or misconfigured proxy could upload as another user. The download endpoint similarly accepts a user directly from the query string.

Fix:

• Authenticate using a verified session, JWT, mTLS identity, or proxy-authenticated identity.
• If the reverse proxy injects identity headers, configure it to strip incoming copies before setting them.
• Restrict direct access so clients cannot bypass the trusted proxy.
• Use only the verified identity populated by authentication middleware:

const userId = req.auth.subject;

Do not silently fall back to anonymous.

4. Missing authorization / insecure direct object reference

Severity: High

Even if X-User were trustworthy on upload, /download lets the requester specify any user:

/download?user=another-user&name=private-file

There is no check that the authenticated requester owns or is permitted to read the file.

Fix:

Store file metadata with ownership and authorize each operation:

const file = await files.findById(req.params.fileId);

if (!file || file.ownerId !== req.auth.subject) {
  return res.sendStatus(404);
}

Use 404 rather than revealing whether another user's file exists where appropriate.

5. Symlink and filesystem-link attacks

Severity: Critical or High, depending on filesystem permissions

Even after rejecting .., an attacker or another process able to create entries under the upload tree could create a symlink such as:

/var/app/uploads/alice/report -> /etc/sensitive-file

writeFileSync would follow the symlink and overwrite its target. sendFile can likewise follow symlinks and disclose files outside the intended tree. There is also a time-of-check/time-of-use race if code checks a path and opens it later.

Fix:

• Ensure only the service account can modify storage directories.
• Do not share the upload tree with untrusted users or jobs.
• Open files with no-follow/exclusive semantics where supported, such as O_NOFOLLOW and O_EXCL.
• Inspect opened descriptors with fstat, rather than trusting an earlier pathname check.
• Consider storing uploads in object storage instead of a shared filesystem.
• Run upload processing in an isolated, unprivileged environment.

Example conceptually:

const flags =
  fs.constants.O_WRONLY |
  fs.constants.O_CREAT |
  fs.constants.O_EXCL |
  (fs.constants.O_NOFOLLOW || 0);

Platform behavior must be tested; safely traversing user-controlled directory trees can require descriptor-relative OS APIs not conveniently exposed by Node.

6. Synchronous filesystem operations block the event loop

Severity: High

These operations are synchronous:

fs.mkdirSync(...)
fs.writeFileSync(...)

Writing a large file can block the entire Node.js event loop. During a slow disk, full disk, network filesystem stall, or large upload, unrelated requests cannot be served. Multiple concurrent uploads become serialized and can cause severe latency or health-check failures.

Fix:

Use asynchronous or streaming filesystem APIs:

await fs.promises.mkdir(dir, { recursive: true });
await fs.promises.writeFile(path, content);

Prefer streaming uploads to a temporary file or object store so the full payload is not buffered in memory.

7. Excessive memory usage and denial of service

Severity: High

The service permits a 50 MB JSON body:

express.json({ limit: '50mb' })

The actual peak memory per request can be much larger because it may include:

• Raw request bytes.
• The parsed JavaScript string.
• JSON object overhead.
• A second decoded Buffer.
• Temporary allocations during base64 decoding.
• Multiple concurrent requests.

Base64 also adds approximately 33% transport overhead. A small number of concurrent requests can exhaust the Node heap or trigger long garbage-collection pauses.

Fix:

• Reduce the request limit to the actual business requirement.
• Use streaming multipart or raw binary uploads.
• Enforce per-user and global concurrency limits.
• Add request rate limits and timeouts.
• Reject requests early based on authenticated quotas.
• Consider direct uploads to object storage using short-lived signed URLs.

8. No storage quota or free-space protection

Severity: High

A caller can repeatedly upload files until the filesystem is full. A full disk can affect not only uploads but logs, databases, temporary files, deployments, and the host itself.

No per-user, file-count, total-size, or global quota is enforced.

Fix:

• Enforce maximum file size, file count, and total bytes per tenant.
• Reserve free-space thresholds and reject writes before reaching them.
• Use a dedicated filesystem or object-storage bucket with quotas.
• Add retention and deletion policies.
• Monitor disk usage, inode usage, and upload rejection rates.
• Rate-limit uploads.

Quotas must be updated atomically to avoid concurrent requests bypassing checks.

9. Existing files are silently overwritten

Severity: High

writeFileSync(outPath, content) truncates and replaces an existing file. An attacker can destroy another upload if they can predict or control its path. Accidental duplicate names also cause silent data loss.

Fix:

• Generate unique opaque file IDs.
• Use exclusive creation (flag: 'wx') when replacement is not intended.
• If versioning is required, store immutable versions and update metadata transactionally.
• Require an explicit, authorized replace operation rather than implicit overwrite.

await fs.promises.writeFile(tempPath, content, { flag: 'wx', mode: 0o600 });

10. Uploads are not published atomically

Severity: High

A processor or download request can see the destination file while it is being created or truncated. It may read a partial file and treat it as complete. This is particularly dangerous because the assumptions state that other jobs process uploaded files.

Fix:

Use a staging workflow:

1. Write to a unique temporary file in the same filesystem.
2. Flush/close it.
3. Validate size, hash, and content as needed.
4. Atomically rename it to its final immutable location.
5. Only then create a database record or enqueue processing.

await fs.promises.writeFile(tempPath, content, {
  flag: 'wx',
  mode: 0o600
});
await fs.promises.rename(tempPath, finalPath);

Use the same filesystem for temporary and final paths so rename remains atomic. Handle cleanup of abandoned temporary files.

11. Concurrent requests have race conditions

Severity: High

Multiple workers, processes, or service instances can upload the same path concurrently. Possible outcomes include:

• Last-writer-wins data loss.
• A processing job observing the wrong version.
• Quota-check races.
• Temporary-file name collisions.
• Metadata and filesystem state disagreeing.

Although synchronous calls serialize work within one Node event loop, they do not coordinate separate Node processes, containers, hosts, or external jobs.

Fix:

• Use immutable, server-generated IDs.
• Use exclusive file creation.
• Keep authoritative metadata in a transactional database.
• Use atomic state transitions such as uploading -> ready -> processing.
• Enqueue jobs only after successful atomic publication.
• Use distributed coordination only where truly necessary; immutable naming usually avoids most locking.

12. Invalid or missing input can trigger unhandled failures

Severity: Medium to High

The handler assumes all fields exist and have the correct type:

Buffer.from(req.body.data, 'base64')
path.join(..., fileName)

Examples:

• Missing data.
• data is an object or array.
• Missing name.
• Query parameters represented as arrays/objects.
• Invalid pathname values.
• Filesystem permission errors.
• ENOSPC, EACCES, EMFILE, or path-length errors.

These can produce exceptions. Because the route is async, behavior also depends on the Express version: Express 4 does not automatically handle rejected async handlers in the same way as Express 5. This can cause unhandled rejections, hung requests, process instability, or generic 500 responses.

Fix:

• Validate request shape before using it.
• Wrap async handlers or use Express 5.
• Install centralized error middleware.
• Map expected errors to stable HTTP responses.
• Never expose stack traces in production.

Example:

function asyncHandler(fn) {
  return (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next);
}

13. Base64 input is not strictly validated

Severity: Medium

Buffer.from(value, 'base64') is permissive and may ignore whitespace or tolerate malformed input. The service does not verify that:

• data is a string.
• The encoding is canonical base64.
• The decoded size matches a declared size.
• Decoding did not produce an unexpectedly empty or truncated result.
• The content matches a supplied checksum.

Clients may believe a file was uploaded successfully when the stored bytes differ from what they intended.

Fix:

• Prefer binary streaming rather than base64 in JSON.
• If base64 is required, validate type, length, alphabet, and padding.
• Calculate the maximum decoded size before allocation.
• Require and verify a cryptographic checksum such as SHA-256.
• Return the server-computed size and checksum.

Be careful that a simple decode-and-reencode comparison accounts for canonical padding rules.

14. No filename, path-depth, or length restrictions

Severity: Medium

Attackers can submit extremely long names, deeply nested paths, unusual Unicode, control characters, platform-specific reserved names, or names that collide after normalization. Consequences include:

• ENAMETOOLONG and repeated 500 errors.
• Log injection or unreadable audit records.
• Ambiguous names due to Unicode normalization.
• Operational tooling failures.
• Large directory trees and inode exhaustion.
• Unexpected hidden files such as .something.

Fix:

• Prefer server-generated IDs instead of caller-selected paths.
• If retaining an original display name, store it as metadata, not as the physical path.
• Enforce a maximum UTF-8 byte length.
• Reject path separators, control characters, . and ...
• Normalize Unicode consistently if names are compared.
• Limit directory depth to a fixed server-defined layout.

15. Uploaded active content may execute in users' browsers

Severity: High if downloads share an authenticated web origin

res.sendFile determines content type from the filename and may display content inline. An attacker could upload HTML, SVG, or another active format and have it served from the application's origin. If another user opens it, the file might execute script in a trusted origin, enabling stored XSS or credential/action abuse depending on browser and header behavior.

Fix:

• Serve untrusted files from a separate origin with no authentication cookies.
• Force download disposition:

res.download(filePath, safeDisplayName);

• Add:

X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox

• Use a conservative content type such as application/octet-stream where appropriate.
• Sanitize only if inline display is an explicit supported feature.

16. Uploaded files can attack downstream processors

Severity: High

The code accepts arbitrary bytes and the assumptions state that other jobs process them. Crafted files can exploit parsers, image libraries, archive tools, office-document converters, media codecs, or antivirus engines. Archives can also cause decompression bombs, path traversal during extraction, excessive CPU usage, or huge file expansion.

Fix:

• Treat all uploads as hostile.
• Quarantine files before processing.
• Run processors in sandboxed containers/VMs with:
• No unnecessary network access.
• Read-only root filesystems.
• Low privileges.
• CPU, memory, file-size, process-count, and time limits.
• Pin and patch parser dependencies.
• Validate actual file format using magic bytes, not just extensions.
• For archives, reject absolute paths, .., symlinks, device entries, and excessive expansion ratios.
• Scan where appropriate, while not treating malware scanning as a complete sandbox substitute.

17. Filesystem permissions are unspecified and may be too broad

Severity: High

mkdirSync and writeFileSync use permissions influenced by the process umask. Files may be readable by other users on the host, and directories may allow access beyond what was intended. If the process runs as root, traversal and symlink issues become substantially more damaging.

Fix:

• Run as a dedicated non-root account.
• Use a restrictive umask, commonly 0o077.
• Explicitly create directories and files with restrictive modes such as 0o700 and 0o600.
• Mount the upload volume with appropriate controls such as noexec, nodev, and nosuid where applicable.
• Do not grant the service write access outside its dedicated storage location.

18. Sensitive server filesystem paths are disclosed

Severity: Low to Medium

The upload response returns:

res.json({ ok: true, path: outPath });

This reveals internal directory layout and embeds the user-controlled path in the response. Internal path disclosure assists reconnaissance and unnecessarily couples API consumers to implementation details.

Fix:

Return an opaque file identifier and API URL instead:

{
  "ok": true,
  "fileId": "01HV...",
  "downloadUrl": "/files/01HV..."
}

Keep physical paths internal.

19. Missing rate limiting and abuse controls

Severity: High

Any reachable client can repeatedly send expensive 50 MB JSON requests. Even when disk quotas exist, request parsing, base64 decoding, authentication, and failed writes consume CPU, memory, bandwidth, and event-loop time.

Fix:

Apply:

• Per-identity and per-source rate limits.
• Concurrent-upload limits.
• Request-body and request-duration limits at the proxy and application.
• Authentication before accepting large bodies where the architecture permits.
• Circuit breakers or load shedding under memory/disk pressure.

Distributed deployments need a shared limiter or proxy-level enforcement rather than only in-process counters.

20. Missing request and socket timeout strategy

Severity: Medium to High

The code does not configure application-level timeout behavior. Slow clients may hold connections and resources for long periods. Proxy timeouts help but should not be the only control, especially if internal clients can reach the service directly.

Fix:

Configure and coordinate:

• Header timeout.
• Request/body timeout.
• Keep-alive timeout.
• Proxy upload timeout.
• Minimum transfer rate where supported.
• Maximum concurrent connections.

Ensure timeout values are compatible across proxy and Node to avoid retry storms or prematurely terminated valid uploads.

21. Download errors and partial responses are not explicitly handled

Severity: Medium

res.sendFile(p) can fail due to missing files, permissions, directory paths, filesystem errors, or a client disconnect. Without explicit handling, behavior may be inconsistent, errors may be logged poorly, and internal details may leak through default handlers.

Fix:

Use a callback or route errors through centralized middleware:

res.sendFile(filePath, err => {
  if (!err) return;
  if (res.headersSent) return next(err);

  if (err.code === 'ENOENT') {
    return res.sendStatus(404);
  }

  next(err);
});

Do not return raw filesystem errors to clients.

22. No auditability or lifecycle metadata

Severity: Medium

The filesystem alone does not reliably record:

• Authenticated uploader.
• Original filename.
• Size and checksum.
• Upload completion state.
• Processing status.
• Retention/deletion state.
• Access history.
• Rejected attempts.

This makes incident response, ownership enforcement, cleanup, deduplication, and safe job coordination difficult.

Fix:

Maintain a metadata record with an immutable ID, owner, size, hash, storage key, timestamps, status, and processing result. Emit structured security/audit events without logging file content or credentials.

23. Service binds to all interfaces by default

Severity: Medium

app.listen(3000);

By default, Node generally listens on unspecified addresses, potentially exposing the service beyond the intended reverse proxy path. The assumption explicitly says internal users can reach it, which permits bypass of proxy authentication, rate limits, TLS, or identity-header sanitation.

Fix:

• Bind to loopback or a dedicated private interface where deployment permits:

app.listen(3000, '127.0.0.1');

• Enforce firewall/security-group/network-policy restrictions.
• If direct internal access is required, implement authentication and TLS at the service as well.
• Do not rely solely on network location as authorization.

Recommended design

A safer flow would be:

1. Authenticate the requester before reading a large body.
2. Authorize upload permission and check quota.
3. Allocate an opaque, server-generated file ID.
4. Stream binary data to a uniquely named quarantine file or object-store key.
5. Enforce byte limits while streaming.
6. Create with exclusive/no-follow behavior and restrictive permissions.
7. Compute size and SHA-256 during the stream.
8. Close, validate, and atomically publish the file.
9. Commit metadata with owner and ready status.
10. Enqueue downstream processing only after publication.
11. Process in an isolated sandbox.
12. Download by opaque ID after an ownership/authorization check.
13. Serve untrusted content as an attachment from a separate origin.

The physical storage path should never contain raw identity headers, query parameters, or original filenames.

Below are concrete security, reliability, and operational findings. Many of these interact; the biggest themes are: untrusted path components, unauthenticated identity, synchronous filesystem I/O, unsafe upload validation, and exposing stored files directly.

1. Path traversal on upload filename

Severity: Critical

Issue:

const fileName = req.body.name;
const outPath = path.join('/var/app/uploads', user, fileName);

fileName is fully attacker-controlled. Values like:

../../../../etc/cron.d/pwn
../otheruser/file.txt
subdir/../../../app/config.json

can escape the intended upload directory.

Exploit / failure mode:

An attacker may overwrite arbitrary files that the Node process can write to, including application files, logs, configuration, or files later consumed by other jobs.

path.join() does not prevent traversal. It merely normalizes the path.

Fix:

Resolve the final path and verify it stays under a fixed base directory.

const UPLOAD_ROOT = '/var/app/uploads';

function safeResolve(base, ...parts) {
  const resolved = path.resolve(base, ...parts);
  const baseResolved = path.resolve(base) + path.sep;

  if (!resolved.startsWith(baseResolved)) {
    throw new Error('Invalid path');
  }

  return resolved;
}

Use it:

const outPath = safeResolve(UPLOAD_ROOT, safeUser, safeFileName);

Also restrict filenames to a safe allowlist.

if (!/^[a-zA-Z0-9._-]{1,128}$/.test(fileName)) {
  return res.status(400).json({ error: 'Invalid filename' });
}

2. Path traversal via X-User header

Severity: Critical

Issue:

const user = req.header('X-User') || 'anonymous';
const outPath = path.join('/var/app/uploads', user, fileName);

The user value is trusted from a request header. An attacker can send:

X-User: ../../../../tmp

or:

X-User: victim

Exploit / failure mode:

This allows writing outside the upload root or writing into another user’s upload directory.

Fix:

Do not trust X-User directly. Use authenticated identity from verified middleware, such as a session, JWT, mTLS identity, or proxy-authenticated header that is stripped and re-added by the reverse proxy.

Also validate user identifiers:

const user = req.auth.userId;

if (!/^[a-zA-Z0-9_-]{1,64}$/.test(user)) {
  return res.status(400).json({ error: 'Invalid user' });
}

At the proxy, strip inbound X-User from clients and only inject it after authentication.

3. Path traversal on download

Severity: Critical

Issue:

const p = path.join('/var/app/uploads', req.query.user, req.query.name);
res.sendFile(p);

Both req.query.user and req.query.name are attacker-controlled.

Exploit / failure mode:

A request such as:

GET /download?user=../../../../etc&name=passwd

may resolve to /etc/passwd, depending on normalization and filesystem permissions.

res.sendFile() will send arbitrary readable files if given a valid absolute path.

Fix:

Use path.resolve() containment checks and safe identifiers. Prefer Express’s root option with validated relative paths.

const user = validateUser(req.query.user);
const name = validateFileName(req.query.name);
const filePath = safeResolve(UPLOAD_ROOT, user, name);

res.sendFile(filePath, err => {
  if (err) {
    if (!res.headersSent) res.status(err.statusCode || 404).end();
  }
});

Do not allow arbitrary path separators in name unless subdirectories are explicitly supported and safely handled.

4. Authentication and authorization are missing

Severity: Critical

Issue:

Anyone who can reach the service can upload and download files for any user.

const user = req.header('X-User') || 'anonymous';

GET /download?user=someoneElse&name=secret.pdf

Exploit / failure mode:

Internal users or compromised internal hosts can impersonate any user, overwrite files, or download other users’ content.

Fix:

Require authentication middleware.

app.use(requireAuth);

Authorize access based on the authenticated principal:

app.post('/upload', (req, res) => {
  const user = req.user.id;
  // no user supplied by client
});

app.get('/download', (req, res) => {
  const user = req.user.id;
  // only download own files unless explicit ACL check passes
});

If admin or cross-user access is required, implement explicit authorization checks.

5. Unbounded memory amplification from JSON plus base64 decoding

Severity: High

Issue:

app.use(express.json({limit: '50mb'}));
const content = Buffer.from(req.body.data, 'base64');

A 50 MB JSON body containing base64 may decode into tens of megabytes of binary data. Express must buffer and parse the whole JSON body in memory. Then Buffer.from() creates another large buffer.

Exploit / failure mode:

Concurrent uploads can exhaust memory and crash the process. Base64 also adds roughly 33% overhead, and JSON parsing adds more overhead.

Example: many clients send near-50 MB JSON bodies concurrently.

Fix:

Avoid JSON/base64 for file uploads. Use streaming multipart upload handling with size limits, such as busboy, multer, or direct object storage pre-signed uploads.

Example with streaming controls:

app.post('/upload', requireAuth, upload.single('file'), async (req, res) => {
  // enforce fileSize limit in multer config
});

Or stream directly:

req.pipe(fs.createWriteStream(tempPath));

Set tighter body limits if JSON is unavoidable.

6. Synchronous filesystem operations block the event loop

Severity: High

Issue:

fs.mkdirSync(path.dirname(outPath), { recursive: true });
fs.writeFileSync(outPath, content);

These operations block the Node.js event loop.

Exploit / failure mode:

Large or slow disk writes make the server unresponsive to all other requests. Concurrent uploads serialize at the event loop level and can cause timeouts, proxy failures, and cascading outages.

Fix:

Use asynchronous filesystem APIs or streams.

await fs.promises.mkdir(path.dirname(outPath), { recursive: true });
await fs.promises.writeFile(outPath, content, { flag: 'wx' });

For large files, prefer streaming:

const stream = fs.createWriteStream(tempPath, { flags: 'wx' });

7. Existing files can be overwritten

Severity: High

Issue:

fs.writeFileSync(outPath, content);

By default, writeFileSync overwrites existing files.

Exploit / failure mode:

An attacker can replace another file, corrupt data, poison downstream jobs, or race with legitimate uploads.

Fix:

Use unique server-generated object names, or fail if the file already exists.

await fs.promises.writeFile(outPath, content, { flag: 'wx', mode: 0o600 });

Better:

const id = crypto.randomUUID();
const outPath = safeResolve(UPLOAD_ROOT, user, id);

Store original filename as metadata, not as the filesystem name.

8. Symlink and hardlink attacks

Severity: High

Issue:

If an attacker can create symlinks under /var/app/uploads or influence the filesystem state, this code will follow symlinks.

fs.writeFileSync(outPath, content);
res.sendFile(p);

Exploit / failure mode:

An attacker could make:

/var/app/uploads/attacker/profile.txt -> /etc/app/config

Then upload to profile.txt and overwrite the symlink target, or download through a symlink to read files outside the upload area.

Fix:

Use a directory with strict ownership and permissions, not writable by other OS users.

Use safer open flags and checks:

const fh = await fs.promises.open(outPath, fs.constants.O_CREAT | fs.constants.O_EXCL | fs.constants.O_WRONLY | fs.constants.O_NOFOLLOW, 0o600);

For downloads, use lstat to reject symlinks and verify the real path remains inside the upload root:

const real = await fs.promises.realpath(filePath);
const root = await fs.promises.realpath(UPLOAD_ROOT);

if (!real.startsWith(root + path.sep)) {
  return res.status(403).end();
}

9. Lack of validation for request body fields

Severity: Medium to High

Issue:

const fileName = req.body.name;
const content = Buffer.from(req.body.data, 'base64');

The code assumes req.body.name and req.body.data exist and are strings.

Exploit / failure mode:

Malformed requests can trigger exceptions:

• req.body is null
• name is missing
• data is not a string
• data is extremely large
• name contains null bytes, slashes, Unicode control characters, or reserved names

This can produce 500 responses, noisy logs, or process termination depending on error handling.

Fix:

Validate schema before use.

if (
  !req.body ||
  typeof req.body.name !== 'string' ||
  typeof req.body.data !== 'string'
) {
  return res.status(400).json({ error: 'Invalid request' });
}

Use a schema validator such as zod, joi, or ajv.

10. Base64 decoding is permissive

Severity: Medium

Issue:

Buffer.from(req.body.data, 'base64');

Node’s base64 decoder is permissive and may ignore whitespace or accept malformed input.

Exploit / failure mode:

Invalid input may still produce bytes. Downstream processors may treat the result as a valid file. Attackers can also send unexpected encodings or malformed content that bypasses superficial checks.

Fix:

Strictly validate base64 before decoding.

function isValidBase64(s) {
  if (typeof s !== 'string') return false;
  if (s.length % 4 !== 0) return false;
  return /^[A-Za-z0-9+/]*={0,2}$/.test(s);
}

Also check decoded size.

const maxBytes = 10 * 1024 * 1024;
const estimatedBytes = Math.floor(req.body.data.length * 3 / 4);

if (estimatedBytes > maxBytes) {
  return res.status(413).json({ error: 'File too large' });
}

11. No upload size limit after decoding

Severity: High

Issue:

The JSON body limit is 50 MB, but there is no explicit decoded file size limit.

Exploit / failure mode:

An attacker can upload many large files, fill disk, consume memory, or overload downstream processing jobs.

Fix:

Set explicit per-file, per-user, and global limits.

const MAX_FILE_BYTES = 10 * 1024 * 1024;

if (content.length > MAX_FILE_BYTES) {
  return res.status(413).json({ error: 'File too large' });
}

Also enforce quotas at storage level.

12. No rate limiting or abuse protection

Severity: High

Issue:

The API accepts uploads from any reachable internal client without rate limiting.

Exploit / failure mode:

An internal user or compromised host can repeatedly upload large files, causing:

• disk exhaustion
• memory exhaustion
• high CPU from JSON/base64 parsing
• downstream processing backlogs