Warning Misconfiguration July 24, 2026

The container that would not stop

A container refused to stop. The cause was nine configs carrying bind-mounts for USB devices that do not exist, a classic community-script leftover that turns every shutdown into a hostage situation.

Container configs cleaned

six of them running at the time

Duration

37 min

17:31 to 18:08

Commands

439 / 24 h

20 messages in session

Data lost

None

no writes outside configs

The symptom

Container 308 would not stop. Systemctl reported the unit dead, but stop commands hung and returned nothing. The container sat in a half-state: not running, not gone.

The agent found the lxc monitor process wedged in uninterruptible sleep, the kernel state known as D-state, with a stack of stop commands piling up behind it. A D-state process cannot be asked politely to exit. The stuck processes were force-killed and the container finally stopped.

The investigation

Why did the monitor wedge? The agent checked what the container expected from the host and found the answer: every USB serial device in its config was missing. The host had zero USB serial hardware, yet the config carried bind-mount lines for four device nodes plus a serial directory, and cgroup rules allowing them.

With the optional flag those lines boot fine when the device is absent. The failure waits for shutdown, when the kernel must unwind mounts that point at hardware that no longer exists. The teardown never returns, and the monitor sleeps forever.

Then came the audit. A community script had stamped the same USB lines into every config it generated. Nine containers on the host carried them: six running, three stopped. The operator asked whether the lines served the NVIDIA card. They do not; the GPU uses PCIe passthrough, entirely unrelated.

Root cause

A community container script blindly copies USB serial bind-mount lines into every config it creates. On a host with no USB serial hardware, those lines are landmines: harmless at boot because of the optional flag, fatal to clean shutdown because the kernel cannot unbind phantom devices.

Two containers had already hit the hang in this session. The other running four were waiting for their next stop to do the same.

The fix

The operator commented the USB lines out of all nine configs. The agent then cycled each running container: stop, confirm down, start. One container repeated the D-state hang during its stop, and the same force-kill cleared it.

A final verification grep across every active config looked for any uncommented USB serial line. It matched nothing, which is the correct answer.

The result

All six running containers came back up and stayed up, with no stuck processes and no repeat hangs. The three stopped containers were cleaned in place.

The whole session ran 37 minutes from first symptom to verified clean, with no data touched and no writes outside the nine config files.

Incident timeline

Container will not stop

17:31 · CT 308: stop commands hang with no output
D-state monitor found

17:32 · lxc monitor wedged in uninterruptible sleep; force-killed
Phantom USB devices

17:34 · every expected ttyUSB and ttyACM node missing on the host
Config lines named

17:36 · bind-mounts for devices that do not exist, guarded by optional
Nine configs affected

17:40 · community script stamped USB lines into every config; six running
GPU ruled out

17:44 · the card uses PCIe passthrough; USB serial is unrelated
Second D-state hang

17:52 · CT 420 repeats the hang mid-cleanup; kill -9 clears it
Six containers cycled

17:53 · stopped, cleaned, restarted one at a time
Verified clean

18:08 · no active USB serial line left in any config

Command log

Signature commands

kill -9 21319 1761395

exit 0 The only way out of a D-state wedge: force-kill the stuck monitor

for i in 300 305 306 308 326 420; do echo -n "CT $i: "; pct status $i; done

exit 0 Verify all six came back and stayed up

grep -rl 'ttyUSB\|ttyACM\|c 188:\|c 189:' /etc/pve/lxc/ 2>/dev/null

exit 0 Find every config still carrying the phantom USB lines

grep -v '^#' /etc/pve/lxc/300.conf /etc/pve/lxc/305.conf /etc/pve/lxc/306.conf /etc/pve/lxc/308.conf /etc/pve/lxc/313.conf /etc/pve/lxc/322.conf /etc/pve/lxc/326.conf /etc/pve/lxc/336.conf /etc/pve/lxc/420.conf /etc/pve/lxc/114.conf | grep -E 'ttyUSB|ttyACM|188:|189:'

exit 1 Exit 1 means zero matches: no active USB serial line survives

Command log
Time	Agent	Command	Exit
17:51:37	SCAIRE	`pct status 420` status answers while the stop hangs: the wedge is below pct	exit 0
17:51:42	SCAIRE	`pct stop 420 --force` force stop also waits on the stuck monitor	exit 255
17:51:57	SCAIRE	`ps aux \| grep -E 'lxc.*(420)' \| grep -v grep` the same D-state pattern as CT 308	exit 0
17:52:07	SCAIRE	`kill -9 21319 1761395` force-kill clears the wedged monitor and its waiter	exit 0
17:52:39	SCAIRE	`pct start 300` clean cycle begins, one container at a time	exit 0
17:53:06	SCAIRE	`pct start 308` the original container returns without the phantom mounts	exit 0
17:53:24	SCAIRE	`pct start 420`	exit 0
17:56:15	SCAIRE	`for i in 300 305 306 308 326 420; do echo -n "CT $i: "; pct status $i; done` all six running and stable after the cycle	exit 0
17:58:54	SCAIRE	`pct config 300 \| head -2 && pct config 305 \| head -2 && pct config 306 \| head -2 && pct config 326 \| head -2 && pct config 420 \| head -2 && pct config 114 \| head -2 && pct config 313 \| head -2 && pct config 322 \| head -2 && pct config 336 \| head -2` the audit sweep across all nine affected configs	exit 0
17:59:42	SCAIRE	`for i in 306 308 326 420 114 313 322 336; do pvesh get /nodes/pve-alpha/lxc/$i/config --output-format json 2>/dev/null \| grep -o '"hostname":"[^"]*"' ; done` identifies the unnamed containers on the hit list	exit 0
18:08:05	SCAIRE	`grep -rl 'ttyUSB\\|ttyACM\\|c 188:\\|c 189:' /etc/pve/lxc/ 2>/dev/null` the nine files still match with comments included; next step isolates active lines	exit 0
18:08:19	SCAIRE	`grep -v '^#' /etc/pve/lxc/300.conf /etc/pve/lxc/305.conf /etc/pve/lxc/306.conf /etc/pve/lxc/308.conf /etc/pve/lxc/313.conf /etc/pve/lxc/322.conf /etc/pve/lxc/326.conf /etc/pve/lxc/336.conf /etc/pve/lxc/420.conf /etc/pve/lxc/114.conf \| grep -E 'ttyUSB\|ttyACM\|188:\|189:'` exit 1 is the win: zero active USB serial lines remain	exit 1

From the session

Operator

Got a container I can't seem to kill or stop fully.

SCAIRE

The lxc monitor is stuck in uninterruptible sleep, which is why nothing's dying cleanly. It's wedged at the kernel level. There's also a hung stop command just sitting there. Let me clear them. All clear: the container is fully stopped now.

Operator

Wow, any other LXCs turned on right now from community scripts with that garbage?

SCAIRE

Six running containers with that junk. The host has zero USB serial devices, so none of them need those entries. The community script just blindly slaps them into every config it generates. With the optional flag they'll start fine without the devices, but shutdown becomes a hostage situation.

Operator

Verified. Make me something that explains what happened, what we did, and the outcome.

SCAIRE

Incident report ready: nine configs carried phantom USB serial lines, two containers had already hit the hang, all nine cleaned, six cycled clean. Won't happen again.

Lessons

Audit what community scripts write into generated configs. Template lines copied blindly become landmines on every host without that hardware.

The optional flag on a bind-mount hides the problem at boot and saves it for shutdown, the worst place to discover it.

A D-state process cannot be killed gracefully. Force-kill clears it, but only removing the phantom mounts stops it from returning.