The crash that printed nothing
A feed reader crash-looped with exit 1 and zero output. The trail led through empty journals, silent JVMs, and finally bytecode, to an app compiled for a newer Java than the runtime.
The symptom
The CommaFeed service on container 319 was down. Systemd had restarted it five times and given up. Every start ended the same way: the Java process died in about one second with exit code 1, and printed nothing. Not to the journal, not to a log file, not anywhere.
Resources were fine. Memory, disk, and CPU all held normal values, and the container itself stayed up. The crash was inside the application, and it was refusing to explain itself.
The investigation
The agent worked an elimination path. Run the jar manually: silent. Swap the log manager: silent. Capture through a pseudo-terminal: silent. Ask the JVM itself with -Xlog:exceptions: nothing thrown at all. Each dead end narrowed the shape of the fault: the process was exiting deliberately, before any logging layer could speak.
Two clues turned the case. The package log showed an apt upgrade the day before had updated the installed Java 17 and added a Java 25 toolchain. And class-loading traces showed the JVM dying a fraction of a second after loading the bootstrap classes, with no exception ever raised.
To find who was swallowing the error, the agent read the application's own bytecode with javap. The bootstrap entry point routes errors through a delayed log handler and exits. System.exit runs before the handler flushes, so the stack trace is buffered and lost. The error existed the whole time; the app was holding it.
Root cause
UnsupportedClassVersionError. The updated application was compiled for class file version 69, which is Java 25. The container's default runtime was Java 17, which understands up to version 61. The first class the app loaded was too new for the JVM, and the version check failed before logging initialized.
The upgrade that broke it never touched the app's config. The old release ran on Java 17; the new release silently raised its compile target, and nothing in the update path warned that the runtime needed to move with it.
The fix
Java 25 was already installed on the box from the day-before upgrade, so the fix was one switch: update-alternatives pointing the default java at the Java 25 toolchain, then restart the service.
Before proposing it, the agent verified the path was sound by running the application directly under the new JVM. It booted clean, banner and all. The switch was then applied, and the service came back.
The result
The service returned after roughly two hours of downtime, with the database untouched and no data lost. Root cause was isolated in about forty minutes, without a single guess applied to the system.
The whole diagnosis was then written into the homelab wiki as a playbook, including a class-file-version reference table so the next person can skip most of the investigation.
Incident timeline
Command log
Signature commands
pct exec 319 -- systemctl status commafeedpct exec 319 -- bash -c "cd /opt/commafeed && java -verbose:class -jar quarkus-run.jar 2>&1 | tail -40"pct exec 319 -- javap -c -p -cp /opt/commafeed/lib/boot/io.quarkus.quarkus-bootstrap-runner-3.37.2.jar io.quarkus.bootstrap.runner.QuarkusEntryPointpct exec 319 -- bash -c "/usr/lib/jvm/temurin-25-jdk-amd64/bin/java -jar /opt/commafeed/quarkus-run.jar | head -30"| Time | Agent | Command | Exit |
|---|---|---|---|
| 13:39:48 | SCAIRE |
pct exec 319 -- systemctl status commafeed
failed state: five start attempts, then systemd quit
|
exit 3 |
| 13:39:49 | SCAIRE |
pct exec 319 -- journalctl -u commafeed --no-pager -n 80
journal holds lifecycle only; the error never lands
|
exit 0 |
| 13:34:49 | SCAIRE |
pct exec 319 -- grep -i "upgrade\|install" /var/log/dpkg.log | tail -30
finds the Jul 30 Java change that set the trap
|
exit 0 |
| 13:35:37 | SCAIRE |
pct exec 319 -- systemd-run --pipe --wait --working-directory=/opt/commafeed java -jar quarkus-run.jar
piped stdout and stderr: still nothing
|
exit 1 |
| 13:36:30 | SCAIRE |
pct exec 319 -- java -Dorg.jboss.logging.provider=jul -jar /opt/commafeed/quarkus-run.jar | head -30
alternate logging stack: silent as well
|
exit 0 |
| 13:43:57 | SCAIRE |
pct exec 319 -- bash -c "cd /opt/commafeed && java -verbose:class -jar quarkus-run.jar 2>&1 | tail -40"
class loading stops at bootstrap; nothing after
|
exit 0 |
| 13:45:39 | SCAIRE |
pct exec 319 -- unzip -l /opt/commafeed/quarkus-run.jar
thin jar: manifest only, normal for Quarkus
|
exit 0 |
| 13:46:55 | SCAIRE |
pct exec 319 -- javap -c -p -cp /opt/commafeed/lib/boot/io.quarkus.quarkus-bootstrap-runner-3.37.2.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint
reads the entry point bytecode for the error path
|
exit 0 |
| 13:47:27 | SCAIRE |
pct exec 319 -- javap -c -p -cp /opt/commafeed/lib/boot/io.quarkus.quarkus-bootstrap-runner-3.37.2.jar io.quarkus.bootstrap.runner.QuarkusEntryPoint | grep -A 200 "private static void doRun"
the doRun body: rethrow is real, the flush is the leak
|
exit 0 |
| 13:48:14 | SCAIRE |
pct exec 319 -- script -qec "java -jar /opt/commafeed/quarkus-run.jar" /dev/null
pseudo-terminal capture: still zero output
|
exit 1 |
| 13:49:26 | SCAIRE |
pct exec 319 -- bash -c "java -Xlog:exceptions=info -jar /opt/commafeed/quarkus-run.jar | head -20"
JVM-level exception log: nothing thrown
|
exit 0 |
| 13:49:43 | SCAIRE |
pct exec 319 -- update-alternatives --list java
Java 25 is already installed from the Jul 30 upgrade
|
exit 0 |
| 13:49:58 | SCAIRE |
pct exec 319 -- /usr/lib/jvm/temurin-25-jdk-amd64/bin/java -version
Java 25.0.4 LTS verified working
|
exit 0 |
| 13:51:04 | SCAIRE |
pct exec 319 -- bash -c "/usr/lib/jvm/temurin-25-jdk-amd64/bin/java -jar /opt/commafeed/quarkus-run.jar | head -30"
the app boots clean under Java 25: root cause proven
|
exit 0 |