Service Fails at Boot With Network Unreachable: network.target or network-online.target?

A service runs fine when you start it with systemctl start, but after every reboot it sits in a failed state. The journal says Network is unreachable, or the app could not resolve a hostname it needs at startup. Start it again by hand and it works. This is almost always an ordering problem. The unit started before the network was usable, and the fix comes down to two systemd targets that sound alike but promise very different things. The rest of this piece follows the shape of a post-mortem: impact, investigation, the discovery, the stopgap, and the permanent fix. It is a generic walkthrough of a common pattern, not a specific incident.

What the impact looks like

After a reboot, whether planned maintenance, a kernel update, or a host-side restart of a VPS, one service stays down while everything else comes back. It is usually something that connects outward early: an app that opens a database connection at startup, a backup agent, a monitoring exporter, a tunnel client, or a worker that fetches config from a remote URL. The confusing part for site owners is that nothing looks broken once the server is up. SSH works, ping works, and systemctl restart fixes it instantly. So it gets written off as a fluke until the next reboot does the same thing. In support, this often arrives as 'my app dies every time the VPS restarts', and the first useful question is whether it fails at boot only or also when started by hand.

How the investigation narrows it down

First rule out the app itself. If a manual start always works, the binary, config, and credentials are fine. What differs is the environment at the moment systemd launched it. Look at the boot where it failed and at how the unit is wired:

BASH

# Logs for this unit from the current boot (use -b -1 for the previous boot)
journalctl -b -u myapp.service

# The unit as systemd sees it, including any drop-in overrides
systemctl cat myapp.service

# What the unit is ordered after, and what it pulls into the boot
systemctl show -p After -p Wants myapp.service

# Which units this one waited on during boot, with timings
systemd-analyze critical-chain myapp.service

# Is a wait-online service present and enabled? One of these may report not found
systemctl is-enabled systemd-networkd-wait-online.service NetworkManager-wait-online.service

The journal typically shows a short burst like this, a second or two after the kernel brought the interface up but before an address or route was in place: Illustrative output format:

TEXT

myapp[612]: lookup db.example.internal: Temporary failure in name resolution
myapp[612]: dial tcp 10.0.0.20:5432: connect: network is unreachable
systemd[1]: myapp.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: myapp.service: Failed with result 'exit-code'.

The two errors mean slightly different things. network is unreachable is the kernel saying there is no route to that destination yet, usually because DHCP or static configuration has not finished. Temporary failure in name resolution means the resolver had nothing usable to ask, often because the DNS server list had not been written yet. Both point the same way: the process ran too early. Then check the unit. Very often systemctl cat shows After=network.target and nothing else network related. It looks correct, which is exactly why this problem survives so long.

The discovery: network.target only means networking has started

network.target does not mean the network is up. It is a synchronization point meaning the network management service has been started. That service may still be waiting on DHCP, bringing up a bond or VLAN, or applying addresses. The systemd documentation itself says network.target has very little meaning during startup. Where it does matter is shutdown. Because systemd stops units in reverse order, a service ordered After=network.target is stopped before the network is torn down. That is useful for something like a web server that should finish serving clients before the interface disappears. It does nothing to make sure an IP and a route exist when the service starts. network-online.target is the one that waits. It is reached once a wait-online helper decides the network is configured. With systemd-networkd that helper is systemd-networkd-wait-online.service. With NetworkManager it is NetworkManager-wait-online.service. Other stacks have their own equivalents, so what 'online' means depends on which one the server uses. By default the networkd helper waits until the links it manages are configured, which normally means addresses assigned and routes in place. There is a second detail people miss. network-online.target is passive. It only becomes part of the boot when some unit asks for it. That is why the fix needs two lines, not one:

After=network-online.target controls ordering. It says 'if that target is part of this boot, start me after it'. On its own it does not pull the target in.
Wants=network-online.target pulls the target into the boot transaction. On its own it does not order anything, so both could start in parallel.

With only After=, the ordering can quietly do nothing if no other unit wanted the target. With only Wants=, the target runs but your service does not wait for it. You need both.

The stopgap: let systemd retry

If the service has to come back right now and you cannot fully test a unit change yet, letting systemd retry is a reasonable temporary measure. Setting Restart=on-failure with a RestartSec= of a few seconds means a failed early start gets another attempt once the network has caught up. The delay matters. Without it, fast repeated failures can hit systemd's start rate limit, and the unit gives up for good. This hides the race rather than removing it, so treat it as a mitigation. A common alternative hack is ExecStartPre=/bin/sleep 10. Avoid it. It slows every start, including manual restarts, and still fails on a slow DHCP server or a busy host.

The proper fix in the unit file

Add the ordering through a drop-in instead of editing the packaged unit file, so package updates do not overwrite it. systemctl edit myapp.service opens an override file and reloads systemd when you save:

INI

[Unit]
Wants=network-online.target
After=network-online.target

[Service]
# Safety net for brief upstream problems, not a substitute for ordering
Restart=on-failure
RestartSec=5s

Reboot and confirm with the same commands as before. systemctl show -p After -p Wants should now list network-online.target. systemd-analyze critical-chain myapp.service should show the wait-online service ahead of your unit. A few things are worth knowing before you add this to everything:

Only use it for services that genuinely need a working network when they start. A local web server that binds to all addresses does not, and holding it back just makes boot slower.
Wait-online can delay boot if an interface never comes up, such as a configured NIC with no cable or an unused secondary interface. The helper then waits until its timeout. On netplan-based systems, an interface can be marked optional: true so it does not block. With networkd directly, the helper accepts options such as --any or --interface= through an override. Check the man page for your version before changing it.
'Online' is a local judgment. It means addresses and routes are configured, not that a remote database answers or that your upstream DNS resolver is reachable. If the service depends on a remote host, the app should still retry its connection rather than exit on the first failure. That is the actual fix for flaky dependencies, and Restart= covers whatever is left.
If the error is Cannot assign requested address rather than unreachable, the service is trying to bind to a specific IP that does not exist yet. The same ordering fix usually helps, but that is a separate case.

The practical takeaway: when a service only fails after reboot and the journal mentions an unreachable network or a failed lookup, open the unit and look for a lone After=network.target. That line protects shutdown, not startup. Add Wants= and After= on network-online.target for services that truly need the network at start, make sure the wait-online service matches the network stack actually in use, and keep a short Restart=on-failure as a safety net rather than the main fix.