Missing trailing slash in nginx proxy_pass — and why live config quietly drifts from the repo

Two distinct failure modes that show up together in practice: a proxy_pass URI that silently mis-forwards paths under a location block, and a live nginx config that has quietly diverged from whatever is checked into the repo.

The signal that gives it away

Bug #1 — trailing slash. The tell is asymmetric breakage: the HTML page behind a proxied path loads fine, but *every* static asset under that path (.js, .css, images, favicon) 404s — and the response Content-Type is text/html, not the expected MIME type. That combination (page OK, all sub-resources 404 with an HTML error page instead of the real asset) is the signature of a backend app's own 404 page being served, not nginx's. Browser console shows a wall of "Failed to load resource: 404" plus "Refused to apply style … MIME type text/html" errors — dozens of them, one per asset, all under the same path prefix.

Bug #2 — config drift. The tell is different: the fix "works" when you edit the live file directly, and keeps working — until someone runs a deploy/restore step that copies "the canonical config" from the repo back onto the server, and the bug (or several previously-fixed bugs) reappears. If a bug you're sure was fixed comes back after a routine redeploy with no code change involved, suspect drift before suspecting a regression in the code.

How to exploit it — i.e. how to fix it

Trailing slash mechanics. proxy_pass behaves differently depending on whether its target has a URI component (a path, even just /) after the host:port:

```nginx # BROKEN — proxy_pass has no URI part → forwards the FULL request URI, # including the matched location prefix location /app/ { proxy_pass http://127.0.0.1:9000; } # request /app/js/main.js → forwarded as /app/js/main.js → backend 404 # (nothing exists at /app/js/main.js on the backend's own webroot)

# FIXED — proxy_pass has a URI part (the trailing slash) → nginx strips # the matched location prefix before forwarding location /app/ { proxy_pass http://127.0.0.1:9000/; } # request /app/js/main.js → forwarded as /js/main.js → backend 200 ```

The rule, confirmed against the official module docs: if proxy_pass has a URI (even a bare /), nginx replaces the part of the request URI matching the location prefix with that URI before forwarding; if proxy_pass has no URI, the full original request URI is passed through unchanged. A backend that serves assets from its own webroot root (common for classic PHP MVC frameworks, but not limited to them) will only find js/main.js, never app/js/main.js — hence the 404-with-HTML-error-page instead of the asset.

Quick verification, in two steps — first bypass nginx entirely to confirm the backend is fine, then go back through the vhost:

```bash # 1. Hit the backend directly, skip nginx curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9000/js/main.js # expected: 200 (file exists in the backend's own webroot)

# 2. Hit it through the public vhost curl -sS -o /dev/null -w "%{http_code} (%{content_type})\n" https://example.com/app/js/main.js # expected: 200 text/javascript # if you get 404 text/html → trailing-slash bug on this location block ```

Fix, verify, then close the loop with an E2E check for zero console errors (not just a 200 on the one file you tested — every asset under the path):

``bash sudo cp /etc/nginx/sites-enabled/example-prod \ /etc/nginx/site-backups/example-prod.bak.$(date +%Y%m%d-%H%M%S) sudo sed -i '/location \/app\//,/^}/ s|proxy_pass http://127.0.0.1:9000;|proxy_pass http://127.0.0.1:9000/;|' \ /etc/nginx/sites-enabled/example-prod sudo nginx -t && sudo systemctl reload nginx ``

A bug like this can be latent for a long time. If the affected path isn't on the app's main navigation flow (e.g. it's a secondary tool nobody opens daily), the missing slash can ship broken from day one and only get noticed weeks later when someone finally clicks into that path — don't assume "it broke recently" just because it was "discovered recently"; check history/backups for when it actually appeared.

Config drift. The underlying cause: live edits made directly on the server (an emergency SSL fix, a CDN proxy tweak, an added header) never get copied back into the repo. The fix that looks safest — "restore the known-good version from the repo" — is exactly the operation that reintroduces every live-only fix made since the repo copy was last updated.

Operational rule that actually prevents recurrence: any edit to a live vhost file gets backported to the repo in the same turn/commit, no exceptions. Add a periodic (weekly, or pre-deploy) drift check:

``bash for f in app-prod billing-prod admin-prod; do diff -q /etc/nginx/sites-enabled/$f ./nginx-configs/$f done # expected: no output (all identical). Any diff → drift, backport now. ``

Collateral gotcha: backups left inside the active directory. nginx's include /etc/nginx/sites-enabled/*; is a bare glob — it loads *everything* in that directory, including timestamped backup files (name.bak.20260101-0930) left there after an edit. Symptom: nginx -t runs clean functionally but spits out one [warn] conflicting server name "..." on 0.0.0.0:443, ignored per duplicated server_name across all those backups. It doesn't break anything today — the real vhost usually wins by alphabetical ordering ahead of its own .bak.* suffix — but it's fragile (a rename could flip which file wins) and the warning noise buries other, real warnings. Fix: move backups to a directory *outside* the include glob (e.g. /etc/nginx/site-backups/) as a matter of routine, never leave them in sites-enabled/. Restricting the include to sites-enabled/*.conf (rather than a bare *) is the more permanent structural fix, if you're willing to rename all vhost files to match.

What does NOT work

- "Just restore the config from git, it's the source of truth." It's the source of truth for what was *committed*, not for what's actually serving traffic. If live-only fixes were never backported, restoring from the repo is a regression, not a fix — always diff live vs. repo *before* overwriting, not after something breaks. - Assuming a 404 that returns HTML is nginx's own 404 page. It usually isn't — it's the *backend's* 404/error page, proxied through faithfully. Chasing an nginx-side error_page directive here is a dead end; the actual bug is upstream of nginx, in what URI got forwarded. - Trusting that "config passed `nginx -t`" means the config is clean. nginx -t will pass (with warnings) even with duplicate server_name blocks from stray backup files — a clean *test* is not the same as a clean *load*. Read the warning count, don't just check the exit code. - Fixing the trailing slash and stopping there. Without the backport step, the fix survives only until the next "restore canonical config" operation. The trailing-slash fix and the drift-prevention rule are two separate fixes for two separate root causes that happen to surface through the same broken page.

Related: Pre-clustering an nginx upstream to one live server lets you scale out in minutes when you actually need to — same vhost family, a different kind of low-risk nginx change (pre-wiring an upstream block) that also depends on live-vs-repo config staying in sync.