Longer public pressure test

Memory measurement correction: the Fly samples below read PID 1, which was Fly init. They do not establish native-server RSS. Functional query results remain valid; the public native-memory gate is reopened. The replacement sampler resolves the server executable and records PID, start time, boot ID, UID and high-water RSS.

The first 40-wave run did not pass. It used the unchanged reserved-search candidate (322b1cde…) and the original 88-origin proxy (f0f4c5aa…), on fresh Fly hostnames. The full report and executed sources are in docs/results/public-steady-pressure-v1/.

Of 160 queries over 177 seconds, 130 met the existing gate: all eight honest results, including four newcomers, in less than four seconds. Six requests failed before HTTP with a client TCP connection timeout. Twenty-three returned HTTP 200 with missing honest results; these started at wave 27. One complete response exceeded four seconds. The final fixture still had 211 seconds of validity, so expiry does not explain these losses.

Fly-init RSS samples stayed between 5,300 and 5,312 KiB. All four temporary apps were removed; the standing operator trial was left running. These memory samples do not establish indefinite stability.

Investigation

The shared test proxy originally allowed four seconds to connect to a paused loopback backend. A full backend listen queue can therefore occupy a shared proxy slot after the discovery client's request deadline. This is a possible fixture-level source of late losses, not yet a confirmed explanation of this public run.

The revised proxy bounds that connection attempt to 400 ms and preserves its four-second response timeout and frontend-disconnect handling. A regression fills a real listen queue and checks the connection fails within 1.5 seconds:

python3 scripts/test_http_pool_connect_bound.py

The next run keeps the native candidate, workload, and success criteria fixed. It records buffered proxy-log counts and socket samples at each tenth wave to help distinguish proxy contention from network or runtime failures. Client TCP failures remain a separate unresolved observation; they are not excused by the proxy change.

Local validation of the revised proxy

The new test image (4d3a8531…) completed 40 Linux withholding waves: all 160 queries returned eight honest results and four newcomers, with the slowest at 2,248.47 ms. Recovery passed and the container was removed. The container memory sample was 557.9 MiB, including all 88 origin processes and the test directories; it is not a per-node measurement. Evidence and executed proxy source are in docs/results/pressure-connect-bound-v1/.

Public repeat with bounded backend connections

The repeat retains a failed overall result: 153 of 160 queries pass. All 153 HTTP responses contain eight honest results and four newcomers, within the four-second limit (maximum 3,907.13 ms). Seven attempts fail before TCP connects. No HTTP-level honest-result loss recurs in this run. This supports keeping the proxy fix, but does not prove that it explains every earlier failure.

Fly-init RSS ranges from 5,140 to 5,152 KiB. Pool socket samples grow from 3,129 to 10,381 TCP sockets in use as stopped origins accumulate unaccepted connections. Buffered proxy snapshots show no slot-exhaustion messages; the final snapshot contains 95 proxy errors. Snapshot counts are not complete event totals. The run exits before directory-loss, recovery and cold-cache phases because the unchanged withholding gate fails. All four apps are independently confirmed removed. Reports and executed sources are in docs/results/public-steady-pressure-v2/.

The next diagnostic probes the standing trial endpoint without the discovery pressure fixture. It uses the same client connection deadline and four-request concurrency, retaining address attempts and failures without retrying them.

Completed cloud-client pressure phases

The IPv4 cloud-client run completes 160 of 160 withholding queries within the unchanged gate: eight honest results, four newcomers, at most 128 results and 192 attempts, in less than four seconds. The slowest remote wall-time upper bound is 3,170 ms. The phase spans 302 seconds including control and sampling time. Directory loss passes in 2,080 ms; recovery passes in 960 ms. Fly-init RSS samples range from 5,064 to 5,076 KiB.

The combined report remains failed: 70 seconds of fixture validity remained, below the 80-second prerequisite for the 61-second cold-cache wait. That phase did not run. A separate renewed multi-scope run must close it. The five temporary apps were removed and their absence independently checked. Full evidence and a derived phase summary are in docs/results/public-steady-pressure-cloud-v1/.

Correct native-memory repeat

The corrected run passes all 160 queries, directory loss and recovery. Its withholding phase spans 263.7 seconds; the slowest query is 3,240 ms. The sampler identifies the native executable as PID 646, UID 10001, and verifies unchanged start ticks and boot ID throughout. Native high-water RSS reaches 15,760 KiB (15.4 MiB), below the 192 MiB gate. All five temporary apps are removed and their absence independently checked. Evidence is in docs/results/public-native-memory-v1/. The separate passing multi-scope run supplies cold-cache coverage. Earlier init samples remain invalid and are not used in the replacement prerequisite manifest.