Post-Mortem: WAF Evasion and Zero-Trust Failures During a Targeted SSH Brute-Force

A few weeks ago, our incident response team was alerted to an unusual spike in failed SSH login attempts across several production servers. While typical, the sustained nature and the distribution of source IPs suggested something more sophisticated than a standard opportunistic scan. Our initial assumption of a simple 'Fail2ban will handle it' quickly evaporated as the attack persisted, indicating a WAF bypass and a failure in our layered defense.

The Incident: Evasion Techniques and Initial Compromise Attempts

The attack vector was a targeted SSH brute-force, but the attackers employed several techniques to evade our existing protections:

Distributed Source IPs: The attempts originated from a large, rotating pool of IP addresses, making rate-limiting by single IP less effective for Fail2ban's default configurations.
Slow and Low: Unlike typical brute-force attacks that hammer a single target, this attack maintained a low rate of attempts per IP, spread over a longer duration, staying under common ModSecurity/Coraza 'request rate' thresholds.
Obfuscated Usernames: Instead of common usernames like 'root' or 'admin', they used a dictionary of less common but still prevalent system users, along with permutations of known application users on our systems.
WAF Bypass for SSH: This was the critical component. While our ModSecurity/Coraza WAF was configured to protect HTTP/S endpoints, its integration with SSH traffic was non-existent. This sounds obvious, but the incident highlighted a common oversight: assuming WAFs protect all ingress, rather than just HTTP/S.

Initial Indicators and Misinterpretations

Our initial alerts were from Fail2ban, showing bans for individual IPs. However, the sheer volume of unique IPs being banned, coupled with the persistence, raised flags. We then observed an increased load on authentication services, which pointed to the SSH daemon itself taking a hit.

BASH

# Example of unusual log entries
$ tail -f /var/log/auth.log | grep 'Failed password'
Jul 10 08:31:01 server sshd[12345]: Failed password for invalid user user1 from 192.0.2.1 port 54321 ssh2
Jul 10 08:31:02 server sshd[12346]: Failed password for invalid user user2 from 192.0.2.2 port 54322 ssh2
Jul 10 08:31:03 server sshd[12347]: Failed password for invalid user user3 from 192.0.2.3 port 54323 ssh2
# ... many more from different IPs

Remediation & Hardening Steps

This incident forced a re-evaluation of our SSH hardening and zero-trust policies, moving beyond simple Fail2ban and focusing on a more integrated security posture.

1. Enhancing Fail2ban with CrowdSec Integration

While Fail2ban is effective, its default configuration often falls short against distributed attacks. We integrated CrowdSec, a collaborative security engine, to leverage its global IP reputation database. First, install CrowdSec:

BASH

curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt update
sudo apt install crowdsec crowdsec-firewall-bouncer-nftables

Then, ensure Fail2ban is configured to send logs to CrowdSec's agent, and CrowdSec's bouncer (e.g., cs-firewall-bouncer) is active. CrowdSec's 'ssh' scenario detects brute-force attempts and can ban IPs globally based on community intelligence. We also tuned Fail2ban's SSH jail to be more aggressive locally, but critically, we allowed CrowdSec to manage the global blocking:

INI

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
logpath = %(sshd_log)s
maxretry = 3
findtime = 300
bantime = 86400 # Ban for a longer period (24 hours)
action = %(action_mwl)

2. Strengthening SSH Daemon Configuration

We reviewed and tightened our sshd_config considerably.

INI

# /etc/ssh/sshd_config

# Disable password authentication entirely if not strictly needed.
# Otherwise, use strong password policies and rate limiting.
PasswordAuthentication no

# Enforce key-based authentication
PubkeyAuthentication yes

# Permit specific users only (whitelist approach)
AllowUsers user1 user2 admin_user

# Disable root login
PermitRootLogin no

# Limit authentication attempts per connection
MaxAuthTries 3

# Configure client alive intervals to detect dead sessions
ClientAliveInterval 300
ClientAliveCountMax 0

# Use a strong KexAlgorithms and Ciphers
KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256
Ciphers [email protected],[email protected]

# Disable X11 forwarding if not required
X11Forwarding no

# Restart SSH service after changes
sudo systemctl restart sshd

3. Implementing Zero-Trust for SSH Access

The most significant change was the move towards a more robust zero-trust model for SSH. This involved:

Bastion Host / Jump Server: All SSH access to production servers now goes through a hardened bastion host. This host is the only server directly exposed to the internet on port 22, and it enforces strict IP whitelisting and multi-factor authentication (MFA).
VPN Requirement: All access to the bastion host itself requires a VPN connection from an approved IP range, further reducing the attack surface.
SSH Certificates: Instead of managing individual SSH keys, we implemented SSH certificates, allowing for centralized control over access, expiry, and revocation. This is a game-changer for managing access at scale.

4. ModSecurity/Coraza WAF for HTTP/S (Clarification and Scope)

While ModSecurity/Coraza wasn't directly involved in the SSH brute-force, the incident highlighted the importance of clearly defining its scope. Our WAF is critical for protecting web applications, but it does not, by default, protect non-HTTP/S services. We use ModSecurity (or its modern successor, Coraza) with the OWASP Core Rule Set (CRS). A typical Nginx configuration for a protected application looks like this:

NGINX

# Example Nginx configuration with ModSecurity/Coraza
server {
    listen 80;
    listen 443 ssl;
    server_name example.com;

    # SSL configuration...

    # ModSecurity/Coraza Integration
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        # Your application specific configuration
        proxy_pass http://backend_app;
        # ...
    }
}

Our main.conf includes the OWASP CRS:

APACHE

# /etc/nginx/modsec/main.conf
Include /etc/modsecurity/owasp-crs/crs-setup.conf
Include /etc/modsecurity/owasp-crs/rules/*.conf

This setup effectively protects web applications from common attacks like SQLi and XSS, but it is crucial to remember its limited scope.

Lessons Learned and Future Enhancements

This incident underscored that security is not a 'set it and forget it' task. Layered defenses are only as strong as their weakest link, and a zero-trust model must extend to all access vectors, not just web applications.

Feature/Tool	Pre-Incident State	Post-Incident Enhancement	Impact	Cost (Effort/Resources)
Fail2ban	Basic config, local bans only	Aggressive local bans, integrated with CrowdSec for global intelligence	Reduced attack surface, faster response to distributed attacks	Medium
CrowdSec	Not deployed	Deployed, active bouncers, integrated with logs	Global threat intelligence, proactive blocking, community defense	Medium
ModSecurity/Coraza	Active for HTTP/S	Scope clarified, focus remained on web applications	No direct impact on SSH, but reinforced WAF importance for web	Low
SSH Config	Password auth enabled, less strict 'AllowUsers'	Password auth disabled, 'AllowUsers' whitelisted, stronger ciphers	Significantly reduced SSH attack surface, harder to brute-force	Low
Zero-Trust (SSH)	Implicit trust, direct SSH from internet	Bastion host, VPN, SSH certificates, MFA	Eliminated direct internet exposure of production SSH, centralized access control	High

Moving forward, we are exploring more advanced zero-trust networking solutions like service meshes with strong authentication and authorization policies for inter-service communication, and re-evaluating our identity and access management (IAM) solutions for all infrastructure access. The goal is to enforce 'never trust, always verify' at every layer of our stack.