How to Detect Intruders in Your VPS Server: Complete Security Guide

Detecting intruders in your VPS server requires monitoring login attempts, checking system logs, identifying modified files, and analyzing running processes for suspicious activity. Every access to your server leaves traces you can find and analyze.

Professional system administrators use multiple detection methods working together. They check authentication logs for failed logins, review user databases for unauthorized accounts, monitor network ports for suspicious listeners, and deploy automated tools that alert when files change unexpectedly.

Server security follows layered defense principles where each protection measure assumes all others have failed. This creates multiple barriers between attackers and your data.

Understanding layered security approach

Security should never rely on a single protection mechanism. Layered security assumes that every defense will eventually fail and designs each layer as if it were the last line of protection.

This approach separates defenses so they operate independently without depending on each other. Comprehensive security protects multiple levels:

• Data protection: encryption, hiding, backups
• Application protection: updates, monitoring, logs
• Service protection: databases, load balancers
• Endpoint protection: firewalls, users, authorizations
• Connection protection: WiFi, VLANs, network segmentation
• Human element: training, policies, procedures

The zero-trust security model embodies this philosophy by requiring every system component to verify every other component. While perfect zero-trust implementation remains impossible, the principle guides security architecture decisions.

Each security layer must function effectively even when all other layers have been bypassed. This redundancy ensures attackers must defeat multiple independent defenses rather than exploiting a single weakness.

Signs someone accessed your server

Every access to a computer leaves traces somewhere in the system. Attackers can gain entry through several methods:

• Standard authentication methods like SSH
• Exploits that bypass security controls
• Backdoors installed during previous intrusions
• Physical access by stealing hard drives

Understanding all these access vectors helps you know where to look for evidence. Start by examining logs for signs of intrusions, backdoors, or anything unusual.

Checking current and past logins

Standard authentication methods get recorded in system logs. Non-standard methods like backdoors and exploits bypass normal authentication and may not appear in standard logs.

Check who is currently logged in:

w

This displays active user sessions with login times and current activities. Review this output for unexpected users, unusual login times outside business hours, or connections from unfamiliar IP addresses.

Review historical login records:

last

This command shows all past login sessions including:

• Remote SSH logins
• Local console access
• System reboots
• Shutdown events

The command reads binary log files that track authentication:

• /var/log/utmp: Current system status and active users
• /var/log/wtmp: Historical records of all login sessions
• /var/log/btmp: Failed login attempts

Check failed login attempts:

lastb

High volumes of failed attempts from the same IP address indicate automated attack tools scanning for weak passwords.

See when each user last logged in:

lastlog

This helps identify dormant accounts that suddenly become active or accounts that have never been used legitimately.

Examining user and password databases

User account information lives in two critical files:

• /etc/passwd: User details (readable by everyone)
• /etc/shadow: Password hashes (readable only by root)

Inspect the shadow file:

cat /etc/shadow

The password field uses a format: algorithm-ID.cost-factor.salt.hash

Modern systems use strong hashing algorithms:

• y = yescrypt (recommended)
• 6 = SHA-512
• 5 = SHA-256
• 1 = MD5 (outdated, insecure)

Important password field symbols:

• * = No password assigned, account cannot authenticate
• ! = Account temporarily disabled
• Invalid format = Login prevented

Review /etc/shadow for unauthorized user accounts, especially those with UID 0 granting root privileges, or accounts with suspicious password values.

Identifying suspicious processes

Attackers execute malicious code that runs as processes in system memory. These processes continue running even if the original binary file gets deleted from disk.

Show all running processes:

ps afxu

This displays user ownership, CPU usage, memory consumption, and command lines.

Use interactive monitoring tools:

htop
# or
btop

What to look for:

• Unfamiliar process names
• Processes owned by unexpected users
• Processes running from /tmp or /var/tmp directories
• Legitimate processes typically run from /usr/bin, /usr/sbin, or /bin

Terminate suspicious processes:

kill -9 <process_id>

The -9 signal forces immediate termination that processes cannot intercept or ignore.

Kill all processes of a specific user:

killall -u <username>

Analyzing authentication logs

All Linux system logs concentrate in /var/log/ directory. Authentication information appears in specific log files you should check regularly.

Check authentication logs:

grep user /var/log/auth.log | grep -vi cron | less

Find failed password attempts:

grep -i "Failed password" /var/log/auth.log

Check system-wide logs:

cat /var/log/syslog | less

For compressed rotated logs:

zgrep user /var/log/auth.log.2.gz | grep -vi cron | less

What to look for in logs:

• Unusual login times (middle of night, weekends)
• Geographic patterns inconsistent with legitimate usage
• Sudden increases in failed authentication attempts
• Successful logins immediately following many failures

Finding recently modified files

Knowing which files changed after an attack proves critical for damage assessment. The find command searches for files based on modification time.

Files modified in the last 24 hours:

find / -mtime 0 -ls | grep -v "proc\|sys" | less

Files modified within last 48 hours:

find / -mtime -2 -ls | grep -v "proc\|sys" | less

Files modified in the last week:

find / -mtime -7 -ls | grep -v "proc\|sys" | less

Understanding -mtime syntax:

• -mtime 0: Exactly the last 24 hours
• -mtime -N: Less than N days ago
• -mtime +N: More than N days ago

Check detailed file timestamps:

stat /etc/passwd

This shows access time, modify time, change time, and birth time.

Critical directories to monitor:

• /etc: Configuration files
• /bin and /usr/bin: Executables
• /root and /home: User data
• Web server document roots

Checking for backdoors

Backdoors allow attackers to regain access after their initial entry point gets discovered and closed. Check multiple locations where backdoors commonly hide.

Check SSH authorized keys:

cat ~/.ssh/authorized_keys

Unauthorized keys in this file grant silent access bypassing password requirements.

Check startup scripts:

less /etc/rc.local
less /etc/profile
ls /etc/profile.d/*
cat ~/.bashrc
cat ~/.zshrc

Review cron jobs:

# User crontabs
crontab -l

# System crontabs
cat /etc/crontab
ls /etc/cron.d/
ls /etc/cron.daily/
ls /etc/cron.hourly/
ls /etc/cron.weekly/
ls /etc/cron.monthly/

Check systemd services:

systemctl list-unit-files --type=service | grep enabled

Check open network ports:

ss -anpult

What to look for:

• LISTEN processes in TCP
• Any UDP processes
• Unexpected ports (especially high-numbered ports)
• Processes without clear legitimate purpose

Hardening SSH access

SSH represents the primary remote access method for VPS servers, making SSH security critical. Configure SSH through /etc/ssh/sshd_config with settings that restrict access and enforce strong authentication.

Generate SSH keys on your local computer:

ssh-keygen

Copy public key to server:

ssh-copy-id -p 22 root@your-server.com

Edit SSH configuration:

vi /etc/ssh/sshd_config

Critical SSH security settings:

• PasswordAuthentication no - Force public key authentication
• PermitRootLogin no - Prevent direct root login
• AllowUsers username - Restrict to specific users
• DenyUsers username - Explicitly block certain users
• AllowGroups sshusers - Allow only specific groups

Reload SSH (not restart!):

/etc/init.d/ssh reload

Important: Always test with a backup connection open! If new settings prevent connection, you can revert through your backup session.

Enforce strong passwords:

vi /etc/pam.d/common-password

Add this line:

password requisite pam_pwquality.so retry=3 minlen=20 difok=3 dcredit=3 ucredit=3 lcredit=3

Parameters explained:

• retry=3: Attempt limit
• minlen=20: Minimum length
• difok=3: Maximum reused characters from old password
• dcredit=3: Minimum numerals required
• ucredit=3: Minimum uppercase characters
• lcredit=3: Minimum lowercase characters

Configuring file permissions

Linux file permissions control read, write, and execute access separately for the file owner, group members, and all other users.

Check permissions:

ls -alh /etc

Permission types:

• r (read): View file contents or list directory
• w (write): Modify file or create/delete files in directory
• x (execute): Run executable or enter directory

Change permissions:

chmod u+x filename      # Add execute for user
chmod go-w filename     # Remove write from group and others
chmod 644 filename      # Set specific permissions (rw-r--r--)

Critical file permission recommendations:

• /etc/shadow: 600 (root only)
• /etc/passwd: 644 (readable by all)
• Home directories: 700 or 750
• Web files: 644 for files, 755 for directories

Understanding umask

Umask sets default permissions for newly created files and directories system-wide.

Check current umask:

umask
# or
umask -S  # Symbolic notation

Set new umask:

umask 026

How umask works:

• Files default to 666 permissions
• Directories default to 777 permissions
• Umask subtracts from these defaults
• Example: 666 - 026 = 640 (rw-r-----)

File attributes and extended attributes

File attributes provide metadata controlling what operations are allowed beyond standard permissions.

View attributes:

lsattr filename

Useful attributes:

• a: Append-only (file can only be appended)
• i: Immutable (cannot be modified or deleted)
• A: No atime update (improves performance)
• s: Secure deletion (overwrite on delete)

Change attributes:

chattr +i filename      # Make immutable
chattr -i filename      # Remove immutable

Extended attributes (user space):

setfattr -n user.note -v "this is mine" file.txt
getfattr -n user.note file.txt
getfattr -d file.txt    # Show all attributes

Access Control Lists (ACL)

ACLs provide more specific permission control than traditional Unix permissions, allowing multiple users and groups with different access levels.

View ACLs:

getfacl filename

Set ACL for specific user:

setfacl -m u:username:rwx /path/to/directory

Remove ACL entry:

setfacl -x u:username /path/to/directory

Default ACLs for directories:

setfacl -d -m u:username:rwx /path/to/directory

The + symbol after permissions in ls -l indicates ACL entries exist.

Automated intrusion detection with AIDE

AIDE (Advanced Intrusion Detection Environment) automates file integrity checking by creating snapshot databases of file properties. It detects unauthorized additions, deletions, and modifications.

Install AIDE:

apt install aide

Initialize database:

aide --init --config=/etc/aide/aide.conf

This can take hours on large systems. Be patient!

Activate the database:

cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Run checks:

aide --check --config=/etc/aide/aide.conf

AIDE output indicators:

• + = Added files
• - = Removed files
• Changed attributes: p=permissions, u=user, g=group, m=modification time, c=change time, C=checksums

Update database after legitimate changes:

aide --update --config=/etc/aide/aide.conf
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Schedule daily checks with cron:

crontab -e

Add:

0 2 * * * /usr/bin/aide --check --config=/etc/aide/aide.conf

Important: Store baseline databases offline! Attackers with root access can modify local AIDE databases to hide their tracks.

Automated security auditing with Lynis

Lynis performs comprehensive automated security audits scanning for vulnerabilities, misconfigurations, and security weaknesses.

Install Lynis:

git clone https://github.com/CISOfy/lynis.git
cd lynis

Run full audit:

./lynis audit system

What Lynis checks:

• Password policies
• Firewall configurations
• Running services
• Kernel parameters
• Logging setup
• Installed security tools

Lynis generates a hardening index score reflecting overall security posture. Higher scores indicate better configurations.

Benefits of Lynis:

• Runs safely without modifying settings
• Generates detailed reports with severity ratings
• Provides specific hardening recommendations
• Safe for production environments

Schedule Lynis audits monthly or after major system changes to track security posture over time.

Log analysis with Logwatch

Logwatch automatically analyzes system logs generating human-readable reports summarizing important events.

Install Logwatch:

apt install logwatch

Run comprehensive analysis:

logwatch --service all --detail High --range All

Detail levels:

• Low: Critical events only
• Medium: Important warnings
• High: All noteworthy entries

Time ranges:

• Today: Last 24 hours
• Yesterday: Previous day
• All: Complete historical analysis

What Logwatch reports highlight:

• Failed login attempts
• Service errors
• Disk space issues
• Unusual system events
• Security-relevant activities

Configure Logwatch to email daily summaries to administrators for routine security awareness without requiring manual log review.

How ENGINYRING VPS supports security

ENGINYRING VPS plans provide full root access enabling complete control over security configurations. You can install and configure any security tools required for your environment.

Security advantages:

• Complete control: SSH hardening, firewall rules, intrusion detection
• Isolated environment: Your security measures protect only your resources
• DDoS protection: Network-level filtering before traffic reaches your VPS
• Automated backups: Recovery points for restoration after incidents
• Dedicated resources: Guaranteed CPU, memory, disk throughput
• Technical support: Installation assistance and incident response guidance

The combination of full administrative control and robust infrastructure creates optimal conditions for implementing enterprise-grade security practices.

Maintaining ongoing security vigilance

Security is not a one-time configuration but an ongoing process requiring constant vigilance and adaptation.

Regular security schedule:

• Daily: AIDE checks, log review with Logwatch
• Weekly: Manual log analysis, process review
• Monthly: Lynis security audits, permission audits
• As needed: Software updates, critical security patches

Document your security baseline:

• Authorized users and their access levels
• Required services and listening ports
• Legitimate remote access sources (IP addresses)
• Normal traffic patterns and usage times

This documented baseline makes anomaly detection significantly easier. When something unusual appears, compare against your baseline to quickly determine if it's legitimate or suspicious.

Practice incident response:

• Run tabletop exercises simulating attack scenarios
• Test backup restoration before needing it during incidents
• Verify monitoring alerts reach appropriate personnel
• Document response procedures and update regularly

Effective security combines technical controls with procedural discipline and continuous learning about evolving threats and defensive techniques.