How to Trace Why a Systemd Service Exits with Status 203/EXEC

When a custom systemd service refuses to start, the output from systemctl status can feel unhelpfully curt. You run the service, watch it instantly flip to failed, and see a single line stating Main process exited, code=exited, status=203/EXEC. The service did not crash five minutes in, nor did it write a stack trace to its application logs. It never executed at all. In systemd's exit code taxonomy, 203/EXEC corresponds to EXIT_EXEC. When systemd handles a service unit, it forks a child process, sets up the requested process environment (such as cgroups, user credentials, namespaces, and file descriptors), and finally calls execve() to replace the child image with the target binary defined in ExecStart. If that execve() system call fails for any reason, the child process exits immediately with return code 203. Understanding that distinction changes how you troubleshoot: you are not debugging your application's logic, you are debugging the kernel's ability to locate, load, or run the file.

The four common culprits behind status 203

Because execve() handles everything between the unit definition and the first CPU instruction of your program, multiple unrelated misconfigurations collapse into this same exit code. The most frequent issue is a path mismatch. Systemd does not look up commands in a standard user $PATH unless explicitly told where to look; absolute paths are mandatory in ExecStart for traditional unit files. If a deploy script dropped a binary into /usr/local/bin/app but the service file specifies /usr/bin/app, execve() returns ENOENT (No such file or directory), which systemd translates directly to 203. The second culprit is permissions. If the binary file exists but lacks an executable bit (chmod +x), or if the directory hierarchy leading down to the executable cannot be traversed by the unprivileged User= specified in the service file, execution aborts with EACCES (Permission denied). The third culprit catches engineers deploying interpreted scripts, such as Python, Bash, or Node.js scripts. If your ExecStart points to /opt/scripts/backup.sh, the kernel reads the very first line of that file (the shebang) to determine which interpreter to run. If that shebang points to #!/usr/bin/python on a modern system where only #!/usr/bin/python3 exists, the error returned by the kernel is still ENOENT. To someone skimming logs, it looks like /opt/scripts/backup.sh is missing, even when ls -l confirms it sits right there. A related nuance is Windows-style CRLF line endings: if the script was edited on Windows, the shebang might read #!/bin/bash\r, pointing to a non-existent binary named bash\r. The fourth culprit involves systemd's built-in isolation directives. Modern unit files often include security hardening such as ProtectSystem=strict, ProtectHome=true, or TemporaryFileSystem=. If an executable resides in /home/deploy/ but the unit defines ProtectHome=true or ProtectHome=read-only, systemd mounts an inaccessible directory over /home inside the service's mount namespace before calling execve(). From the perspective of the child process, the target file simply vanished.

Tracing the failure systematically

Instead of making random edits to the unit file, you can isolate whether the problem lives in the filesystem, the interpreter, or systemd's sandboxing within a few commands. First, inspect the unit's journal output with systemd-level metadata included. Running journalctl with the -u flag reveals whether systemd printed an explanatory error string right before reporting code 203:

BASH

journalctl -u my-service.service -n 20 --no-pager

In modern systemd versions, the journal often prints an explicit diagnostic message alongside the 203 code, such as Failed at step EXEC spawning /usr/bin/myapp: No such file or directory or Failed at step EXEC spawning /usr/bin/myapp: Permission denied. That single string tells you whether you are fighting a missing path or an access block. If the journal states No such file or directory for a script you can see on disk, test the shebang and the dynamic linker directly:

BASH

# Inspect the shebang line and line endings
head -n 1 /opt/scripts/run-app.sh | cat -A

# Check if the interpreter specified exists and can execute
/bin/bash -n /opt/scripts/run-app.sh

# If it is a compiled ELF binary, check dynamic library and loader requirements
file /usr/local/bin/myapp
ldd /usr/local/bin/myapp

The cat -A command is handy because it renders carriage returns as ^M. If you see #!/usr/bin/env bash^M$, the kernel failed to find the interpreter because of the trailing hidden return character. For compiled binaries, running ldd verifies whether all required shared libraries and the ELF dynamic linker (like /lib64/ld-linux-x86-64.so.2) exist. If an executable was compiled for an architecture or glibc version different from the host, the kernel fails during dynamic linking with ENOENT, presenting identically to a missing file.

Practice safely on a test instance

Practicing these edge cases on a non-production virtual machine gives you muscle memory for reading ambiguous errors. You can deliberately create three dummy services to watch how systemd behaves:

Point ExecStart to /usr/bin/test-missing and inspect the resulting journal message.
Point ExecStart to a real text file that has permissions 644 (no execute bit) and observe Failed at step EXEC ... Permission denied.
Create a bash script with #!/bin/nonexistent as line one, make the script executable (chmod +x), point a service at it, and observe how systemd reports No such file or directory on the script itself, rather than naming the missing interpreter.

When a service exits with 203/EXEC, resist the temptation to tweak your application configuration. Treat 203 as an operating system loader problem: check whether the binary exists at that exact absolute path, ensure execution bits and directory permissions allow the targeted user to reach it, verify the shebang interpreter, and check whether systemd sandbox directives have hidden the path from the process namespace.