Understanding Systemd Services

The init system is the very first process that starts on the system and its job is to schedule and manage all other processes on the system. It is also referred to as PID 1 short for process id 1. When a process such as Apache or Nginx is started on a Linux server, the request to start that process is sent to the init system which then starts it up.

systemd is a newer init system with a more modern approach to process management. It can enable the administrator to schedule tasks to run when specific criteria are met and also to ensure that a process is always running. It allows you to configure logging, how DNS is resolved, and much more.

Units in systemd are resources that systemd can manage for you. These include services, timers, mounts, automounts, and more. systemd runs in the background and is completely self-sufficient.

How `systemd` is Structured

systemd looks at unit files to understand how to interact with the processes. Service files are just text files that contain instructions that tell systemd how it needs to manage that particular service. Service files can be found in any of several directories and they will have a file extension of .service . Service unit files are typically located in:

Location	Description
`/etc/systemd/system`	This directory is for custom or overridden service files. Anything inside this directory will have the ultimate priority. If we want a configuration or custom unit file to take effect, then we should make sure that the changes are added to this directory with a higher priority.
`/run/systemd/system`	This directory is for runtime-generated files. It stores runtime `systemd` units and it is the least looked directory.
`/lib/systemd/system`	This directory is for default service files provided by installed packages. When we install a package, a service file of that package will be installed for us and it will go inside the 3rd directory with the lowest priority. The reason is that the user has an opportunity to override the settings. It is not recommended to store custom unit files inside the 3rd directory, because if a package is updated, the custom unit files will be overwritten.

When the Linux system starts, it will start the systemd and systemd will go into these directories looking for unit files. If it finds any, it is going to load those into memory.

A service unit file structure is divided into 3 sections ([Unit], [Service], [Install]), each containing key-value pairs that configure the behavior of the service. Each section name must be started with a capital letter since the file is case-sensitive. For example:

[Unit] section: Describes the unit and its relationships to other units. Common options include Description, After, Requires, and Wants.
[Service] section: Specifies how the service should be started, the command to use, and its behavior. Options include Type, ExecStart, Restart, and User.
[Install] section: Defines how the service should be enabled or started as part of the system startup sequence. The most common option is WantedBy.

Dependencies between services are managed with directives like After=, Before=, Wants=, and Requires=. These directives help ensure services start in the correct order and that prerequisite services are running.

As an example, let's look at the apache2.service unit file which can be found in /lib/systemd/system directory:

Editing a systemd service unit file allows you to customize how a service starts, behaves, or stops on your system. To safely edit a service unit file, run:

sudo systemctl edit SERVICE_NAME.service

This command opens the service's override file in the default text editor. If the override file doesn’t exist, systemctl will create a new one. The override file allows you to specify only the changes or additions you want to apply to the service unit, without needing to copy and edit the entire original service file. It is stored in a directory named after the service within /etc/systemd/system/example.service.d/override.conf. This directory can contain multiple .conf files for granular overrides.

When systemd loads the service's configuration, it reads the original service file first and then applies the settings from the override file. If there are conflicting directives, the settings in the override file take precedence over those in the original service file. This merge happens in memory, leaving the original service file unchanged on disk.

To edit the entire service file, run:

sudo systemctl edit --full SERVICE_NAME.service

This command makes a copy of the entire service file and save it in /etc/systemd/system directory. This copy file will be in priority than the original service file.

Hands-on Exercise Overview

systemctl is the primary command-line utility for interacting with systemd. While it's used for managing various aspects of systemd, it's particularly focused on service management. This hands-on exercise shows how to use the six most important commands related to systemctl and systemd in general. It shows how to start, stop, restart, check the status, and enable, and disable the process using systemctl command. These are the bare minimum we should know when it comes to systemd.

Hands-on Exercise

Let's install a web server service for serving a web page.
```
 sudo apt install apache2
```
To check the status of the web service, run:
```
 systemctl status apache2.service
```
To confirm that Apache2 is serving its default page by accessing it from within the Vagrant box, run curl http://localhost in the command line. This should show the HTML content of the default Apache2 Ubuntu default page.

or, enter http://localhost:8080/ on the web browser of your host machine:

💡

Since this hands-on exercise is run via Vagrant Ubuntu 20.04, we need to modify the Vagrantfile to configure port forwarding for forwarding requests made to port 8080 on the host machine to port 80 on the Vagrant guest machine (where Apache2 listens by default).

To stop the web service with systemd immediately, run:
```
 sudo systemctl stop apache2.service
```
To start the service immediately:
```
 sudo systemctl start apache2.service
```
To restart the service, run:
```
 sudo systemctl restart apache2.service
```
This command stops the service and then starts it again. This is useful for applying changes that require a complete stop and start to take effect. Restarting is more disruptive but ensures that the service runs with the latest configuration and code.
To reload the service, run:
```
 sudo systemctl reload apache2.service
```
This command tells a service to reload its configuration files without interrupting ongoing operations. It is useful for applying non-disruptive changes. Not all services support reloading; it depends on whether the service can dynamically read new configurations without needing to stop and start again.
To enable a service to start at boot:
```
 sudo systemctl enable apache2.service
 sudo systemctl enable --now apache2.service
```
--now option allows to start the service immediately and enable it in one shot.
To disable a service from starting at boot:
```
 sudo systemctl disable apache2.service
```
preset: enabled means that the distribution has configured systemd such that new services will be enabled by default on boot. The vendor preset only indicates the initial recommended state. Debian and Ubuntu typically enable a newly started service for the user whereas Red Hat-based distributions such as Fedora, and CentOS disable it.
To list unit files that are of type service, run:
```
systemctl list-unit-files --type=service
```