A practical, step-by-step guide for investigating a compromised Linux server. Real commands, real output examples, and what to look for at each step.
Most compromised servers show subtle symptoms that are easy to miss if you're not looking. Here are the most common indicators:
If you're seeing any of these, proceed through the steps below to investigate. Even if you're not sure, running these checks takes 15-20 minutes and will give you a clear picture. The commands below work on Ubuntu, Debian, and all RHEL-based distributions.
Start by looking at what's running on your server. Cryptominers typically consume 100% CPU and run from temporary directories. Reverse shells maintain outbound connections to attacker-controlled servers. After cleanup, harden your server using our VPS security checklist to prevent reinfection.
# Check for high-CPU processes (cryptominers often pin all cores)
$ ps aux --sort=-%cpu | head -20
# Look for processes running from /tmp, /dev/shm, or hidden directories
$ ls -la /tmp /dev/shm /var/tmp
$ find /tmp /dev/shm /var/tmp -type f -executable 2>/dev/null
# Check for recently modified executables
$ find /tmp /dev/shm -type f -mtime -7 2>/dev/null
# Check last logins for unauthorized access
$ last -a | head -20
$ lastlog | grep -v "Never logged in"
# Check for unauthorized cron jobs (persistence mechanism)
$ for user in $(cut -f1 -d: /etc/passwd); do crontab -l -u $user 2>/dev/null | grep -v "^#" | grep -v "^$" && echo " ^ $user"; done
# Check for new SSH authorized_keys entries
$ find / -name authorized_keys -exec ls -la {} \; 2>/dev/null
$ find / -name authorized_keys -exec wc -l {} \; 2>/dev/null
Tip: If you suspect a rootkit, the output of ps and ls may be compromised. Compare the process count from ps aux | wc -l with ls /proc | grep -E '^[0-9]+$' | wc -l. A significant difference suggests hidden processes.
Rootkits hide malicious processes, files, and network connections from standard tools. Two free scanners — rkhunter and chkrootkit — check for the most common rootkit signatures.
# Install rkhunter (Debian/Ubuntu)
$ sudo apt update && sudo apt install rkhunter -y
$ sudo rkhunter --update
$ sudo rkhunter --check --sk
# Install chkrootkit (Debian/Ubuntu)
$ sudo apt install chkrootkit -y
$ sudo chkrootkit
# Check ld.so.preload (library injection — should be empty or not exist)
$ cat /etc/ld.so.preload 2>/dev/null
# Compare ps output with /proc to detect hidden processes
$ ps aux | wc -l
$ ls /proc | grep -E '^[0-9]+$' | wc -l
# Check for processes with deleted binaries (common after rootkit install)
$ ls -la /proc/*/exe 2>/dev/null | grep deleted
Important: If /etc/ld.so.preload contains any entries, you likely have an active rootkit. The file should either not exist or be empty. Any shared library listed there is loaded into every process on the system.
ClamAV is the most widely used open-source antivirus for Linux. It's free and available in every major distro's package manager. It works, but has significant limitations for server malware.
# Install (Debian/Ubuntu)
$ sudo apt update && sudo apt install clamav clamav-daemon -y
# Update virus definitions
$ sudo systemctl stop clamav-freshclam
$ sudo freshclam
$ sudo systemctl start clamav-freshclam
# Scan web directories and temp folders
$ sudo clamscan -ri /var/www /tmp /home --log=/var/log/clamav-scan.log
# Check results
$ grep FOUND /var/log/clamav-scan.log
ClamAV limitations: ClamAV is designed for email gateway scanning, not web malware. It misses many PHP web shells, obfuscated backdoors, and modern server-targeted malware. Scan times on large directories can take hours. It's a useful first pass, but not comprehensive for server security.
Package managers maintain checksums of installed files. If a system binary has been modified (e.g., a rootkit replacing /usr/bin/ps), these tools will flag it.
# Debian/Ubuntu — check installed package files
$ sudo apt install debsums -y
$ sudo debsums -c 2>/dev/null
# Alternative for Debian/Ubuntu
$ dpkg -V
# RHEL/CentOS — verify all packages
$ rpm -Va | grep -v "^\.\.\.\.\.\.\.\.T"
Any output with 5 (checksum mismatch) in the results indicates a modified binary. Focus on executables in /usr/bin, /usr/sbin, and /bin. Configuration files and documentation changes are usually benign.
Web shells are PHP (or sometimes Python/Perl) scripts that give attackers remote command execution through a web browser. They're the most common malware on web servers.
# PHP files in upload directories (almost always malicious)
$ find /var/www -path "*/uploads/*.php" -type f 2>/dev/null
# Recently modified PHP files (last 7 days)
$ find /var/www -name "*.php" -mtime -7 -type f 2>/dev/null
# PHP files with suspicious function names
$ grep -rl "eval(base64_decode\|passthru\|shell_exec\|system(" /var/www --include="*.php" 2>/dev/null
# Files with unusually high entropy (obfuscated code)
$ find /var/www -name "*.php" -size +50k -type f 2>/dev/null | head -20
Common web shell names to look for: c99.php, r57.php, wso.php, alfa.php, b374k.php, and files with random-looking names like xkj82.php. Modern web shells often use heavy obfuscation — a single PHP file with 500KB of base64-encoded content is almost certainly malicious.
Once you've identified the malware, clean it up methodically. Missing any persistence mechanism means the attacker comes back.
Kill malicious processes — identify PIDs with ps aux and kill them: kill -9 <PID>
Remove malware files — delete identified malicious scripts, executables, and web shells
Remove unauthorized cron jobs — check all user crontabs and /etc/cron.d/ directory
Clean /etc/ld.so.preload — remove any entries and run ldconfig
Revoke all SSH keys — remove unauthorized entries from all authorized_keys files
Change all passwords — root, all users, database passwords, application secrets, API keys
Reinstall modified packages — apt install --reinstall <package> for any flagged by debsums/rpm -Va
Check for persistence in systemd — look for unknown service files in /etc/systemd/system/
Reboot and verify — restart the server and confirm the malware doesn't come back
Critical: If you found a rootkit, do not trust any commands on the compromised system. The safest approach is to provision a new server, restore data from a known-clean backup, and harden before reconnecting to the network.
The steps above take 15-20 minutes per server and require manual interpretation of results. Defensia automates all of it with a single command install.
$ curl -fsSL https://defensia.cloud/install.sh | sudo bash
Results appear in a real-time dashboard with expandable findings, severity ratings, and a "Not malware" button for false positives. The scanner runs on schedule (configurable) and alerts you via Slack, email, Discord, or webhook when new malware is found.
Removing malware is only half the job. If you don't close the entry point, the attacker will come back. Here's how to prevent reinfection:
Run apt upgrade / yum update regularly. Automate with unattended-upgrades (Debian/Ubuntu) or dnf-automatic (RHEL). Most malware exploits known vulnerabilities with available patches.
Manual checks are reactive. Defensia runs scheduled scans and monitors upload directories in real time, catching new malware within seconds of it appearing on disk.
A WAF blocks the exploit attempts that lead to malware installation. Defensia's WAF detects SQL injection, XSS, path traversal, and RCE from nginx/Apache logs with zero configuration.
SSH and wp-login.php brute force attacks are the top entry points. Automated blocking (Defensia or fail2ban at minimum) prevents attackers from gaining credentials in the first place.
Defensia watches upload directories for new PHP files and flags them immediately. This catches web shells within seconds of upload, before the attacker can use them.
The most common signs are unexplained high CPU usage (cryptominers), unknown processes running from /tmp or /dev/shm, unauthorized SSH keys in authorized_keys, unexpected cron jobs, and PHP files in web upload directories. Run the process, rootkit, and file integrity checks described in Steps 1-4 above for a thorough investigation.
ClamAV is the most popular free option but misses many PHP web shells and modern server malware. For web servers, Defensia provides better coverage with 64,000+ hash signatures, 684 dynamic patterns, rootkit detection, and WordPress database scanning. Unlike ClamAV, it also includes real-time monitoring of upload directories.
No. ClamAV was designed for email gateway scanning. It misses many PHP web shells, obfuscated backdoors, cryptominers with custom builds, and modern server-targeted malware. Its database is updated by the community and may lag behind new threats. Use it as a first pass, but don't rely on it as your only scanner.
First identify the process: ps aux --sort=-%cpu will show it using 100% CPU. Kill it with kill -9 <PID>. Then find the binary: ls -la /proc/<PID>/exe (before killing) to see the file path. Remove the binary, check all crontabs for persistence entries that re-download it, check systemd services, and check /etc/rc.local. Change all passwords and SSH keys afterward.
When Defensia detects malware, it moves the file to /var/lib/defensia/quarantine/ with permissions set to 000 (no read, write, or execute). The original file path and SHA256 hash are recorded in the dashboard. You can review quarantined files and restore them if they're false positives. The quarantine action is logged in the audit trail.
Defensia malware scanner: tested on 9 production servers, 64K+ hash signatures from MalwareBazaar (2025-2026)
ClamAV documentation: https://docs.clamav.net/
rkhunter documentation: http://rkhunter.sourceforge.net/
NIST SP 800-83: Guide to Malware Incident Prevention and Handling — https://csrc.nist.gov/publications/detail/sp/800-83/rev-1/final
SANS Incident Response: https://www.sans.org/white-papers/33901/
MalwareBazaar (abuse.ch): https://bazaar.abuse.ch/
64K+ hash signatures. 684 detection patterns. Scheduled scans. Real-time dashboard.
Free for 1 server. No credit card required.