Course: Linux System Administration

1. Process Management

1.1 Systemd

systemd is the init system used by most modern Linux distributions. It manages services, mounts, sockets, timers, and more via unit files.

Common systemd commands:

systemctl start|stop|restart|status <service>
systemctl enable|disable <service>
systemctl daemon-reload
systemctl list-units --type=service
journalctl -u <service> -f

Unit file example (/etc/systemd/system/myapp.service):

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=appuser
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

1.2 Cgroups v2

Control groups (cgroups) limit and account for resource usage (CPU, memory, I/O) of process groups.

Key concepts:

• cgroups v2 unified hierarchy (/sys/fs/cgroup/)
• Controllers: cpu, memory, io, pids, cpuset
• Systemd integrates directly with cgroups v2

# View cgroup of a process
cat /proc/<PID>/cgroup
# Check memory limit for a container
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max

1.3 Namespaces

Namespaces isolate global system resources per process. They are the foundation of containerization.

# List namespaces
lsns
# Enter a process's namespace
nsenter -t <PID> -n bash

2. Filesystem

2.1 FHS (Filesystem Hierarchy Standard)

/bin     - Essential user binaries
/sbin    - System binaries
/etc     - Configuration files
/var     - Variable data (logs, databases)
/tmp     - Temporary files
/usr     - User utilities and applications
/proc    - Virtual filesystem for process info
/sys     - Kernel and device information
/dev     - Device files

2.2 Filesystem Types

• ext4: Most common Linux filesystem, journaled, backward-compatible.
• XFS: High-performance, good for large files, online defragmentation.
• btrfs: Copy-on-write, snapshots, compression, RAID support.
• ZFS: Advanced COW filesystem, checksumming, volume manager.

# Filesystem info
df -hT
lsblk -f
blkid /dev/sda1

3. Networking

3.1 Iptables (Legacy)

# List rules
iptables -L -n -v
# Allow SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
# Masquerade (NAT)
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

3.2 Nftables (Modern Replacement)

# List ruleset
nft list ruleset
# Simple firewall
nft add table inet filter
nft add chain inet filter input { type filter hook input priority 0 \; }
nft add rule inet filter input tcp dport 22 accept
nft add rule inet filter input tcp dport 443 accept
nft add rule inet filter input drop

3.3 Network Configuration

# View interfaces
ip addr show
ip route show
ss -tulpn

# Bonding, VLANs, bridges
ip link add br0 type bridge
ip link set eth0 master br0

4. Bash Scripting

4.1 Production-Grade Script Template

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

# Logging
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; }
error() { echo "[ERROR] $*" >&2; exit 1; }

# Trap errors
trap 'error "Script failed on line $LINENO"' ERR

# Main
main() {
    local config_file="${1:-/etc/default/app.conf}"
    [[ -f "$config_file" ]] || error "Config not found: $config_file"
    log "Starting with config: $config_file"
    # ... logic ...
}

main "$@"

4.2 Best Practices

• set -euo pipefail — exit on error, undefined vars, pipe failure
• Use [[ ]] instead of [ ] for conditional expressions
• Prefer $(cmd) over backticks
• Use local for variables inside functions
• Validate all inputs
• Handle signals with trap

5. Performance Analysis

5.1 Perf (Linux Profiler)

# Real-time CPU sampling
perf top
# Record and report
perf record -a -g -- sleep 10
perf report -g graph --stdio
# Static tracing
perf stat -e cycles,instructions,cache-misses ./myapp

5.2 Strace (System Call Tracing)

# Trace system calls
strace -c -p <PID>          # Count syscalls
strace -e openat,read -p <PID>  # Filter syscalls
strace -T -p <PID>          # Show time spent in syscalls

5.3 Top/htop/atop

top -o %MEM          # Sort by memory
htop -p <PID>        # Monitor specific PID
atop -d 5            # Disk-focused every 5s
iotop               # Per-process I/O

5.4 BPFtrace

# Count syscalls per process
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[comm] = count(); }'
# Files opened by process
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s: %s\n", comm, str(args->filename)); }'

6. Package Management

# Debian/Ubuntu
apt update && apt upgrade -y
apt install -y nginx
dpkg -i package.deb

# RHEL/Fedora
dnf install -y nginx
rpm -ivh package.rpm

7. Linux Security

7.1 SELinux (Security-Enhanced Linux)

Mandatory Access Control (MAC) system embedded in the Linux kernel.

# Check status
getenforce        # Enforcing / Permissive / Disabled
sestatus

# Context management
ls -Z /var/www/html/index.html
chcon -t httpd_sys_content_t /var/www/html/index.html
semanage fcontext -a -t httpd_sys_content_t '/web(/.*)?'
restorecon -Rv /web

# Troubleshooting
ausearch -m avc -ts recent
sealert -a /var/log/audit/audit.log

7.2 AppArmor

Alternative MAC system using path-based profiles.

# Status
aa-status
# Enforce/Complain
aa-enforce /usr/bin/myapp
aa-complain /usr/bin/myapp
# Generate profile
aa-genprof /usr/bin/myapp
# Log
aa-logprof

7.3 Capabilities

Capabilities break root privileges into fine-grained units.

# Run with specific capabilities
setcap cap_net_bind_service=+ep /usr/bin/myapp
# View capabilities
getcap /usr/bin/ping
# In Docker
docker run --cap-drop ALL --cap-add NET_BIND_SERVICE ...

7.4 Seccomp (Secure Computing Mode)

Seccomp restricts system calls available to a process.

# Simple seccomp profile (allow list)
echo '{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64"],
  "syscalls": [
    {"names": ["read","write","open","close","exit","exit_group"], "action": "SCMP_ACT_ALLOW"}
  ]
}' > seccomp-profile.json

# With Docker
docker run --security-opt seccomp=seccomp-profile.json ...

Summary

Linux sysadmin is the foundation of DevOps engineering. Mastering process management (systemd, cgroups, namespaces), performance debugging (perf, strace, bpftrace), scripting, and security (SELinux, AppArmor, capabilities) enables you to build, debug, and secure modern infrastructure.