1. What systemd is, and what a unit is

systemd is the init system and service manager that runs as PID 1 on most modern Linux distributions. You drive it with systemctl to control units, and you read its logs with journalctl. Everything else in this post is a closer look at those two commands and the unit files they operate on.

The unit of management is, fittingly, the unit. A unit is a small text file that tells systemd how to manage some piece of the system: a service to run, a mountpoint to set up, a socket to listen on, a timer to fire. systemd reads the unit file, builds a dependency graph from all of them, and brings the system up by walking that graph.

You’ll see several unit types in the wild. The ones worth knowing by name:

.service: a process to supervise. This is where you’ll spend most of your time.
.target: a group of other units. A way to say “bring up everything that should be running by the time we’re at a multi-user login prompt”.
.socket: a listening socket that systemd opens on behalf of a service, so the service can start lazily on first connection.
.timer: a scheduled trigger for another unit. The systemd replacement for cron.

The rest of this post focuses on .service: how to control existing ones with systemctl, how to read their output with journalctl, and how to write your own.

1.1. Units, targets, and the dependency graph

Targets are how systemd composes the boot. multi-user.target is the standard “fully booted, multi-user, no GUI” state. graphical.target adds the display manager on top. They aren’t magic states; they’re units that declare dependencies on other units, the same way services do.

The two main dependency directives you’ll see in a unit file are Wants= and Requires=. Wants=foo.service says “I’d like foo to be running, but don’t fail me if it isn’t”. Requires=foo.service says “if foo fails, I fail too”. Combined with After=, which sets ordering, that’s most of what you need to read a real-world unit.

When the box boots, systemd reads the unit named by the default target (almost always multi-user.target on a server or graphical.target on a desktop) and pulls in everything that target wants, transitively. Two commands let you see this for yourself:

systemctl get-default     # which target is the boot target
systemctl list-units      # every unit currently loaded

Run them once on a machine you administer; it’s the clearest way to develop a feel for what systemd is actually doing.

2. Controlling units with systemctl

systemctl is the one command you use to control units. Most of its subcommands fall into four buckets: inspecting state, changing runtime state, changing boot-time state, and telling systemd that the unit files on disk have changed.

For inspection:

systemctl status nginx              # current state + tail of logs
systemctl list-units --type=service # everything loaded right now
systemctl list-unit-files           # everything installed, enabled or not
systemctl cat nginx                 # print the unit file (and any overrides)
systemctl is-active nginx           # exit 0 if active, non-zero otherwise
systemctl is-enabled nginx          # exit 0 if enabled, non-zero otherwise

is-active and is-enabled are worth remembering specifically because they’re scriptable. They print state and set an exit code, so you can use them inside shell pipelines and CI checks without parsing prose output.

For runtime control:

sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
sudo systemctl reload nginx         # if the service supports SIGHUP-style reload

For boot-time control:

sudo systemctl enable nginx         # add to the default target's dependency graph
sudo systemctl disable nginx        # remove it
sudo systemctl enable --now nginx   # enable AND start in one step

And when you’ve edited a unit file on disk:

sudo systemctl daemon-reload        # re-read units from disk

If you edit a .service file and systemctl restart it without daemon-reload first, systemd restarts the old unit definition; it hasn’t noticed your edit yet. This is by far the most common “but I just changed it!” failure.

2.1. Reading systemctl status

Every field in systemctl status is load-bearing: Loaded tells you where the unit file lives, Active tells you the runtime state, Main PID and CGroup tell you which processes belong to it, and the trailing block is the unit’s journal tail. Here’s a real-shaped example:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-05-25 10:14:03 UTC; 2h 11min ago
       Docs: man:nginx(8)
   Main PID: 1421 (nginx)
      Tasks: 3 (limit: 4612)
     Memory: 8.2M
        CPU: 412ms
     CGroup: /system.slice/nginx.service
             ├─1421 "nginx: master process /usr/sbin/nginx -g daemon on;"
             ├─1422 "nginx: worker process"
             └─1423 "nginx: worker process"

May 25 10:14:03 host systemd[1]: Starting A high performance web server...
May 25 10:14:03 host systemd[1]: Started A high performance web server.

Reading it top-down:

Loaded tells you where the unit file lives and whether it’s currently enabled (will start at boot). preset is the distro’s default for this unit.
Active is the current runtime state. active (running) is the happy path. Others you’ll meet: inactive (dead), failed, activating, deactivating.
Main PID is the supervisor’s view of the service’s main process. Useful for strace, gdb or lsof, anything else you need to point at the running process.
CGroup lists every process systemd thinks belongs to this service, grouped under a control group named after the unit. If your service spawns children, they show up here unless something went badly wrong.
The last block is the tail of the unit’s journal, the same lines you’d get from journalctl -u nginx.

When a service is broken, almost everything you need is in this output. The Active line tells you the state, the journal tail usually tells you why, and the CGroup tells you whether the process is even alive.

2.2. Enable vs start: why your service doesn’t come back after reboot

systemctl start runs a unit right now. systemctl enable wires it into the dependency graph so it starts automatically at boot. They are independent operations, and forgetting that is the most common systemd mistake.

Run only systemctl start myservice and your service runs until the next reboot, then doesn’t come back. Run only systemctl enable myservice and your service is configured to start at boot, but it isn’t running yet. The shortcut you almost always want is:

sudo systemctl enable --now myservice

which does both. disable --now does the inverse (disable + stop) and is equally useful when retiring a service.

There’s a third state worth knowing: mask. systemctl mask foo symlinks foo.service to /dev/null, which makes it un-startable, and even other units can’t pull it in as a dependency. It’s the strongest “no, really, do not run this” you have. Reverse it with unmask.

3. Writing a .service unit file

A .service unit is a small INI-style text file with three sections: [Unit], [Service], and [Install]. Here’s a complete, minimal unit for a Python HTTP server, the kind of thing you might want supervised across reboots:

[Unit]
Description=Static file server for /srv/public
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/srv/public
ExecStart=/usr/bin/python3 -m http.server 8080
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Save it as /etc/systemd/system/static-server.service, then:

sudo systemctl daemon-reload
sudo systemctl enable --now static-server
systemctl status static-server

Walking through the directives:

Description= is human-readable text. It’s what shows up next to the dot in systemctl status.
After=network-online.target says “don’t start me until the network is up”. Wants=network-online.target is the same Wants= from section 1: it pulls in network-online.target so it actually gets activated. After= on its own only specifies order; if nothing else pulls the target in, you’ll order yourself after something that never happens. The pair is the canonical idiom.
Type=simple means ExecStart= is the long-running foreground process, and systemd treats the service as “started” the moment the process is execed. Most modern programs that stay in the foreground (Python servers, Node apps, Go binaries) want this.
User=www-data runs the process as a non-root user. If you omit User=, the service runs as root, which is almost never what you want for a network-facing service.
WorkingDirectory= sets cwd for the process. Useful when your code uses relative paths.
ExecStart= is the command line. Use an absolute path to the binary.
Restart=on-failure tells systemd to restart the process if it exits with a non-zero status, dies on a signal, or otherwise fails. RestartSec=5s waits 5 seconds between restarts.
[Install] is only consulted when you run systemctl enable. WantedBy=multi-user.target says “when this unit is enabled, add it to multi-user.target’s wants”. That’s how you get pulled in at boot.

One ExecStart= gotcha worth pulling out: systemd does not invoke a shell to run the command. Pipes (|), boolean operators (&&), tilde expansion (~), and environment-variable expansion don’t work the way you’d expect from bash. If you genuinely need shell features, set ExecStart=/bin/sh -c '...' explicitly and accept that you’ve added a shell to the supervision chain.

A few file-location rules worth knowing. Your own units go in /etc/systemd/system/. Distro-packaged units live in /lib/systemd/system/ (or /usr/lib/systemd/system/) and you should not edit them in place; package updates will overwrite your changes. If you need to tweak a packaged unit, use systemctl edit <unit>, which creates an override file in /etc/systemd/system/<unit>.d/override.conf that systemd layers on top of the original.

3.1. Choosing a Type= value

The Type= directive tells systemd how to decide that your service is “started”. Getting this wrong is the second most common new-unit bug (the first being forgetting daemon-reload).

Type=simple: the default. ExecStart= runs in the foreground. systemd considers the unit started as soon as it has execed the binary. It does not wait for the program to actually be ready to serve requests. Use this for modern apps that stay in the foreground.
Type=forking: for old-school daemons that fork into the background and exit the parent. systemd waits for the parent to exit and treats the surviving child as the service. You usually pair this with PIDFile= so systemd knows which child to track. Avoid unless you’re packaging something that genuinely behaves this way.
Type=notify: the program is expected to call sd_notify(READY=1) once it’s actually ready. systemd waits for that notification before declaring the unit started. This is the right choice for anything with a non-trivial startup (loading a large dataset, opening many sockets) that other units depend on with After=.
Type=oneshot: for a unit that runs to completion and exits. Useful for one-off setup steps you want sequenced into the boot. Pair with RemainAfterExit=yes if you want the unit to count as active after the command finishes.

If in doubt, start with simple. It’s right more often than anything else.

3.2. Restart policies

Restart= controls when systemd brings the process back after it exits. The three values you’ll actually use:

Restart=no (default): never restart for any reason.
Restart=on-failure: restart on non-zero exit, signal, or watchdog timeout. Clean shutdowns (exit 0, SIGTERM during stop) leave the service stopped. This is what you want for almost every long-running service.
Restart=always: restart on any exit, including clean ones. Useful when any exit is treated as a bug.

RestartSec= sets the delay between restarts. The default is 100ms, which is fast enough to spin if your service crashes in a tight loop. Setting it to a few seconds is sensible.

Two related knobs prevent restart storms: StartLimitBurst= (default 5) and StartLimitIntervalSec= (default 10s). If the service tries to start more than StartLimitBurst times within StartLimitIntervalSec, systemd gives up and marks it failed. That’s the behavior you want; a service that can’t stay up shouldn’t be hammering its dependencies forever.

4. Logs with journalctl

journalctl reads the system journal, the structured, indexed log store where systemd captures stdout and stderr for every service it supervises. The mental shift from tail -f /var/log/app.log is that journal entries aren’t just lines of text. Each one carries fields (timestamp, unit, PID, priority, hostname, the message itself), and the filters operate on those fields.

The handful of filters that cover almost everything:

journalctl -u nginx                       # only nginx's logs
journalctl -u nginx -f                    # follow (like tail -f)
journalctl -u nginx -e                    # jump to the most recent entries
journalctl -u nginx -b                    # only this boot
journalctl -u nginx -b -1                 # only the previous boot
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2026-05-25 09:00" --until "2026-05-25 10:00"
journalctl -u nginx -p err                # only errors and worse
journalctl -u nginx -p warning..emerg     # a priority range

The -u flag is the one you’ll use most often. It scopes the output to a single unit, which on a busy machine is the difference between five lines of relevant context and ten thousand lines of everything else.

A practical “my service is broken” loop usually looks like:

systemctl status myservice                # state + last few log lines
journalctl -u myservice -b -e             # everything since boot, jumped to end
journalctl -u myservice -f                # if the service is restarting in a loop

For machine-readable output, --output=json gives you one JSON object per entry. Useful for piping into jq to extract specific fields, or for ingesting into a log aggregator.

One detail that bites people coming from plain log files: priorities. journald assigns each entry a priority from emerg (0) to debug (7), and many programs log everything at info by default. journalctl -p err only shows priority err (3) and lower-numbered, more severe levels.

If you don’t see your warning messages with -p warning, check whether the program is actually emitting them at warning level. Filter shows you nothing when the source produced nothing.

4.1. Making the journal survive reboots

By default on many distros, the journal lives in /run/log/journal/, which is tmpfs. That means logs survive only until the next reboot. If you want logs that persist across reboots, and on any server you do, the journal needs a directory at /var/log/journal/.

The control is Storage= in /etc/systemd/journald.conf:

Storage=volatile: /run/log/journal/ only. Logs lost at reboot.
Storage=persistent: /var/log/journal/. Logs survive reboots; the directory is created if needed.
Storage=auto (the default on most distros): persistent if /var/log/journal/ already exists, volatile otherwise. On a fresh install the directory often doesn’t exist, so auto silently behaves as volatile.

The fix is either flipping Storage=persistent or simply creating the directory:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Once persistent storage is on, you’ll want bounds. SystemMaxUse= caps the total disk used by the journal; SystemMaxFileSize= caps individual files. Sensible starting values on a small box:

[Journal]
Storage=persistent
SystemMaxUse=500M
SystemMaxFileSize=50M

journalctl --disk-usage shows you what the journal is currently consuming. If a noisy service has filled your disk, journalctl --vacuum-size=100M will trim it down without waiting for the bounded values to take effect.

5. Troubleshooting a service that won’t start

When a service won’t start, or won’t stay started, run these four commands in this order. They cover the vast majority of failures.

systemctl status myservice: the current state, the most recent journal lines, and (often) the specific error.
systemctl cat myservice: print the unit file, including any overrides. Confirms that the unit on disk actually says what you think it says. After every edit, this is the sanity check.
journalctl -u myservice -b: the full logs since boot. If systemctl status showed a generic “process exited with status N”, the journal will usually have the specific reason a few lines above.
systemd-analyze verify /etc/systemd/system/myservice.service: a static check of the unit file’s syntax and references. Catches typos in directive names, missing dependencies, and a number of subtler issues before you waste time wondering why your edit didn’t take effect.

If step 2 reveals you forgot to daemon-reload after editing, that’s the answer. If step 3 shows the process exec’ing and immediately exiting the bug is in the program, not the unit. If step 4 reports a parse error, fix the unit first; there’s no point reading logs for a unit systemd couldn’t parse.

systemd has plenty more, including timers, sockets, slices, and sandboxing, but the path from systemctl restart nginx to authoring and debugging your own services runs through these four commands and the unit-file fundamentals above.