Troubleshooting

Do not regenerate your install token. A 429 means a request budget was spent, not that your token is wrong. Generating a new one spends another attempt and leaves you in the same place — and if you delete the server to get a new token, you have thrown away a token that was working.

The response body tells you which of the two 429s you have. Look at the reason field:

rate_limited — a capacity budget is full. Registration is bounded three ways, and the response names which one:

• Per install token: 15 per minute, 40 per hour
• Per organization API key (Kubernetes): 60 per minute, 300 per hour
• Per source address: 60 per minute, 300 per hour

Normally you reach these only when an install is looping — a service restarting on a host that never finishes registering, or a DaemonSet rolling across many nodes at once. Stop the loop first, then wait out the window shown in retry_after (also sent as the Retry-After header) and run the same command again.

invalid_credential_rate_limited — too many rejected credentials from your address. This is the one case where a fresh token is the answer: 10 rejected attempts per hour per IP, and only attempts whose credential the platform actually refused count toward it. A request carrying a valid token never draws on this budget, so your own earlier typos cannot lock out a correct token.

Check what the agent saw:

sudo journalctl -u defensia-agent -n 50 --no-pager | grep -i "regist\|429\|retry"

Agent 1.4.78 and newer handles this properly: it honours Retry-After when the wait is a minute or less, and otherwise stops and prints how long to wait instead of hammering the endpoint. It also caps a single install run at three attempts, so a failed run always leaves you a real retry.

Self-hosted panels: an operator can drop a stuck budget immediately rather than waiting out the hour:

php artisan agent:clear-register-limit --ip=1.2.3.4
# or, for one credential:
php artisan agent:clear-register-limit --token=

A 422 is a validation failure: the panel refused a field in the request. By far the most common cause is a truncated or padded install token.

The token is exactly 64 characters. The panel accepts nothing shorter or longer. A token that lost characters to a line wrap in your terminal, or picked up a trailing newline from a paste, fails here — and the failure looks identical to a server-side problem, which is why it gets misdiagnosed as a dead token. Check the length before anything else:

echo -n '' | wc -c   # must print exactly 64

Re-copy the token before you regenerate it. A 422 does not mean the token is spent or revoked — it means what reached the panel was not the token. Copy it again from the dashboard, paste it into a plain editor to confirm it is one unbroken 64-character string, and retry. Deleting the server to get a new token revokes a token that was almost certainly fine.

Agent 1.4.78 and newer checks the length locally and refuses to send, so a mis-pasted token now produces a clear message instead of a 422 — and costs you no rate-limit budget.

If the length is right and it still fails, read the response: since 1.4.78 the agent prints the HTTP status, the panel's own message, and each rejected field with its reason.

sudo journalctl -u defensia-agent -n 50 --no-pager | grep -i "regist"

Every other required field now has a fallback inside the agent — a missing OS version reports as unknown, a missing hostname falls back to the server name — so on a current agent a 422 is nearly always the token.

This is expected on the Free plan, and it is enforced by the platform rather than by a setting you can change: every Free server runs in monitor mode. Attacks are detected, scored and reported to your dashboard, but the agent never writes a firewall rule.

So on Free you will see SSH brute force, port scans and mail/FTP/database auth failures appear as events — with no bans behind them. The monitor-mode toggle on the server page only takes effect on Pro; on Free it is held on.

Two related Free limits explain other "missing data" reports:

• No Defensia WAF. The engine that reads your access logs for SQL injection, XSS, path traversal, web shells, plus bot policies, threat feeds and geoblocking, is Pro only. The agent is told to disable it, so it generates no events.
• 2,000 visible events per month and 7 days of history. Beyond the allowance, older events of the current month stop being shown.

If you run Apache with ModSecurity, expect the opposite surprise. The agent installs its own ModSecurity rule set on every plan, Free included, and it is not affected by monitor mode — no code path gates it. Those rules carry an inline deny,status:403, so on a Free server you may well see web attacks being intercepted and reported while your ban list stays empty. Both are correct: the 403 comes from your web server, not from a Defensia firewall rule. The event type will name a ModSecurity rule ID in the 9910xxx range.

The allowance applies to exporting too, because exporting is a way of viewing. A CSV cut short by it is never silently short: the file carries a NOTICE: row stating the limit and how many further events were not exported.

Upgrading to Pro lifts all three at once: blocking is enabled, the WAF starts reporting, and the event history opens up. Nothing you have already collected is lost.

This is a deliberate change in agent 1.4.76 – 1.4.78, not a regression. If you run ModSecurity, expect both the type and the severity of existing event classes to have moved.

What changed:

• Probes are now rated by outcome, not by how gravely the rule set rated them. An /.env or config probe the server refused is warning; one answered with a 2xx means the file was actually served and stays critical. This is where most of the drop comes from.
• Protocol-enforcement rules are capped at info. CRS ships "method not allowed" at CRITICAL, so malformed Host headers used to raise critical alerts.
• Comodo cWAF rules are now classified. Comodo ships by default on cPanel and DirectAdmin and uses its own reserved rule-ID range, which previously matched nothing — so every Comodo detection arrived as an uncategorised critical. Those now carry a real attack type.
• Classification keys off the rule-ID family rather than the first matching tag, so the most severe real finding in an audit-log entry wins. This surfaced attacks that protocol noise had been shadowing — notably LFI that never counted as path traversal.

If you have an alert rule on critical events, review it. Your critical volume is now a much better signal, but it is a smaller number than it was — a threshold tuned against the old counts will under-fire.

Lower severity here does not mean less detection: the events are still recorded, still searchable, and still counted. Only the severity they are filed under changed.

This error means your server's SSL certificate bundle is too old to recognize Let's Encrypt's certificates. It is common on CentOS 7 / RHEL 7 and any system whose ca-certificates package has not been updated since 2024. On CentOS 7, curl is compiled against NSS (not OpenSSL), so update-ca-trust alone has no effect on it.

One-time fix — run these three commands, then retry the install:

# 1. Download the Let's Encrypt root CA (using -k since curl can't verify LE yet)
curl -sk https://letsencrypt.org/certs/isrgrootx1.pem -o /tmp/isrg.pem

# 2. Tell curl to use it for this session
export CURL_CA_BUNDLE=/tmp/isrg.pem

# 3. Re-run the installer normally
curl -fsSL https://defensia.cloud/install.sh | sudo bash -s -- --token 

Once the agent is installed, the installer automatically bootstraps this fix on subsequent runs — you only need to do this manually for the first download.

Permanent fix (optional — survives reboots):

# Add ISRG Root X1 to system trust store
curl -sk https://letsencrypt.org/certs/isrgrootx1.pem \
  -o /etc/pki/ca-trust/source/anchors/isrg-root-x1.pem
update-ca-trust force-enable && update-ca-trust extract

# Also update the NSS database (required for curl on CentOS 7):
certutil -d /etc/pki/nssdb -A -t "CT,," -n "ISRG Root X1" \
  -i /etc/pki/ca-trust/source/anchors/isrg-root-x1.pem

Status 203/EXEC means systemd could not execute the agent binary. The most common causes are a missing binary, a corrupted binary (partial download), or incorrect file permissions.

1. Diagnose what's on disk:

ls -la /usr/local/bin/defensia-agent*

Expected: defensia-agent around 6 MB, permissions -rwxr-xr-x. If it is missing, near 0 bytes, or not executable — proceed to recovery.

2. Restore from backup (fastest path):

cd /usr/local/bin
cp defensia-agent.bak defensia-agent
chmod 755 defensia-agent
systemctl reset-failed defensia-agent
systemctl start defensia-agent

A .bak file is created automatically every time the agent self-updates. Once restored, the agent will auto-update to the current version within 60 seconds.

3. If no .bak exists — download a fresh binary:

ARCH=$(uname -m | sed "s/x86_64/amd64/;s/aarch64/arm64/")
curl -fsSL \
  "https://github.com/defensia/agent/releases/latest/download/defensia-agent-linux-${ARCH}" \
  -o /usr/local/bin/defensia-agent
chmod 755 /usr/local/bin/defensia-agent
systemctl reset-failed defensia-agent && systemctl start defensia-agent

4. If systemd refuses to start due to start-limit-hit:

grep -q StartLimitIntervalSec /etc/systemd/system/defensia-agent.service || \
  sed -i '/^\[Unit\]/a StartLimitIntervalSec=0' /etc/systemd/system/defensia-agent.service
systemctl daemon-reload
systemctl reset-failed defensia-agent
systemctl start defensia-agent

5. Verify the agent is running:

systemctl status defensia-agent
journalctl -u defensia-agent -n 20 --no-pager

The agent cannot reach your Defensia panel. Check the following:

1. Verify network connectivity:

curl -v https://your-defensia-server.com/api/v1/agent/heartbeat

2. Check firewall rules: Ensure outbound HTTPS (port 443) is allowed.

sudo iptables -L OUTPUT -n | grep 443

3. Verify the server URL in config:

grep -o '"server_url":"[^"]*"' /etc/defensia/config.json

4. Check DNS resolution:

dig your-defensia-server.com

If the server is behind a reverse proxy, make sure it correctly forwards the Authorization header.

The agent is installed but appears offline in the dashboard. This means heartbeats are not being received.

1. Check the agent service status:

sudo systemctl status defensia-agent

2. Restart the agent:

sudo systemctl restart defensia-agent

3. Verify the agent has credentials: they live in /etc/defensia/config.json. Check the file is present and complete without printing the token:

test -s /etc/defensia/config.json && grep -c agent_token /etc/defensia/config.json

A result of 1 means the token is there. A missing or empty file means registration never completed — reinstall with a fresh token.

4. Check agent logs for errors:

sudo journalctl -u defensia-agent --since "10 minutes ago" --no-pager

If the token has been revoked or the server was removed from the dashboard, you will need to reinstall the agent with a new token.

The agent is online but security events are not showing up in the dashboard.

1. Verify the auth log path is correct: the installer bakes it into the service unit as AUTH_LOG_PATH:

grep AUTH_LOG_PATH /etc/systemd/system/defensia-agent.service

The default paths are:

• Debian/Ubuntu: /var/log/auth.log
• RHEL/CentOS/Rocky/Alma: /var/log/secure

2. Check if the log file exists and has content:

ls -la /var/log/auth.log && tail -5 /var/log/auth.log

3. Check agent logs for parsing errors:

sudo journalctl -u defensia-agent | grep -i "error\|parse\|event" | tail -20

4. Verify the agent has read permissions on the log file:

sudo -u root test -r /var/log/auth.log && echo "readable" || echo "not readable"

The agent detects threats but IP bans are not being applied on the server.

1. Check if iptables is installed and running:

which iptables && sudo iptables -L -n | head -20

2. Install iptables if missing:

# Debian/Ubuntu
sudo apt-get install -y iptables

# RHEL/CentOS
sudo yum install -y iptables

3. Check if the agent has permission to modify iptables: The agent must run as root. Verify the service file:

grep "User=" /etc/systemd/system/defensia-agent.service

If it is set to a non-root user, remove that line and restart:

sudo systemctl daemon-reload
sudo systemctl restart defensia-agent

4. Check the current ban list:

sudo iptables -L INPUT -n | grep DROP

The agent cannot establish a WebSocket connection to Reverb for real-time updates.

1. Check the WebSocket endpoint the agent was given: it is returned at registration and stored in /etc/defensia/config.json:

grep -o '"reverb_url":"[^"]*"' /etc/defensia/config.json

2. Verify the WebSocket port is accessible:

curl -v ws://your-server:6001/app/your-app-key

3. Check if Reverb is running on the server side:

# On the Defensia panel server
php artisan reverb:start --debug

4. If behind a reverse proxy (Nginx): Make sure WebSocket upgrade headers are configured:

# Nginx config for /app/ location
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;

The agent will fall back to polling the /agent/sync endpoint if WebSocket is unavailable, but real-time updates will be delayed.

The agent logs to systemd journal. Use these commands to view logs:

Follow logs in real-time:

sudo journalctl -u defensia-agent -f

View logs from the last hour:

sudo journalctl -u defensia-agent --since "1 hour ago" --no-pager

View only error-level logs:

sudo journalctl -u defensia-agent -p err --no-pager

View logs from today:

sudo journalctl -u defensia-agent --since today --no-pager

Export logs to a file for support:

sudo journalctl -u defensia-agent --since "24 hours ago" --no-pager > defensia-logs.txt

The agent should use minimal resources. If you notice high CPU or memory usage:

1. Check current resource usage:

systemctl status defensia-agent
# or
top -p $(pgrep defensia-agent)

2. Check the auth log file size: Very large log files can cause increased parsing overhead.

ls -lh /var/log/auth.log

3. Check log rotation: Ensure logrotate is configured for the auth log.

cat /etc/logrotate.d/rsyslog

4. Narrow what the agent watches: the heartbeat runs on a fixed 60-second interval and is not a meaningful cost. Parsing is, so if one noisy access log dominates, pin the WAF to the logs you care about with WEB_LOG_PATH (comma-separated — it overrides all auto-detection):

# Edit /etc/systemd/system/defensia-agent.service
[Service]
Environment="WEB_LOG_PATH=/var/log/nginx/access.log"

5. Reload and restart after changing the unit:

sudo systemctl daemon-reload && sudo systemctl restart defensia-agent

Normal resource usage should be under 1% CPU and under 30MB RAM. If usage remains high after these steps, contact support with your logs.

To perform a clean reinstall of the Defensia agent:

1. Stop and remove the current installation:

sudo systemctl stop defensia-agent
sudo systemctl disable defensia-agent
sudo rm /usr/local/bin/defensia-agent
sudo rm /etc/systemd/system/defensia-agent.service
sudo rm -rf /etc/defensia
sudo systemctl daemon-reload

2. Remove the server from your dashboard: Go to the server detail page and click "Delete Server". This will revoke the old token.

3. Generate a new install token: Click "Add Server" in the dashboard to get a fresh token.

4. Run the installer again:

curl -fsSL https://defensia.cloud/install.sh | sudo bash -s -- --token 

5. Verify the new installation:

sudo systemctl status defensia-agent

The agent will re-register as a new server with a fresh token. Previous events and bans for the old server entry will remain in the dashboard history.

The agent is online but no WAF events (SQL injection, XSS, env probes, etc.) appear in the dashboard. The most common cause is that the agent cannot find your web server access logs.

1. Check which logs the agent is monitoring:

sudo journalctl -u defensia-agent | grep "webwatcher"

You should see lines like:

[webwatcher] watching /var/log/nginx/access.log (yourdomain.com)
[webwatcher] monitoring 2 log file(s) covering 3 domain(s)

If instead you see "no web server logs found — WAF disabled", the agent cannot access any access logs. Continue below.

2. If your web server runs natively (not Docker): Check that nginx or apache is installed and logs exist at the standard path:

ls -la /var/log/nginx/
ls -la /var/log/apache2/

If your logs are in a custom location, set WEB_LOG_PATH in the agent service:

# Edit /etc/systemd/system/defensia-agent.service
[Service]
Environment="WEB_LOG_PATH=/custom/path/access.log"

# Then reload and restart:
sudo systemctl daemon-reload && sudo systemctl restart defensia-agent

3. If your web server runs inside Docker: The agent cannot read container-internal logs unless you bind-mount the log directory to the host. Add a volume to your compose file:

# docker-compose.yml
services:
  nginx:
    image: nginx
    volumes:
      - /var/log/nginx:/var/log/nginx   # ← add this line

Then recreate the container and restart the agent:

docker compose up -d nginx
sudo systemctl restart defensia-agent

The agent automatically detects Docker containers running nginx/apache/caddy/httpd, reads their config, and maps log paths to host paths via the bind-mount. No further configuration is needed.

4. Verify the log format is Combined Log Format: The agent parses the standard Combined format. Custom log formats may cause parse failures (the agent will warn you in the logs):

sudo journalctl -u defensia-agent | grep "high parse failure"

If you see this warning, switch to combined format in your nginx config: access_log /var/log/nginx/access.log combined;

You have configured blocked countries but traffic from those countries is still getting through.

1. Verify the feature is enabled: Geoblocking requires a Pro subscription. Check your plan in the Billing page.

2. Check the agent received the config: The agent fetches blocked countries from the /agent/sync endpoint. Restart the agent to force a fresh sync:

sudo systemctl restart defensia-agent

3. Note there is no local on/off switch: blocked countries are delivered over the sync endpoint, so the dashboard is the only place to change them. GEOIP_DB_PATH only points at an alternative database file.

4. Check agent logs for GeoIP errors:

sudo journalctl -u defensia-agent | grep -i "geoip\|country\|geo" | tail -20

Geoblocking relies on GeoIP databases to map IP addresses to countries. The agent updates this database automatically. If issues persist, ensure your server has outbound internet access for database updates.

You can change your account password from the Security settings page.

1. Go to Settings → Security in the dashboard sidebar.

2. Scroll down to the Change Password section.

3. Enter your current password, then your new password twice for confirmation.

4. Click Update Password.

Your new password must be at least 8 characters. For best security, use a unique password that you don't reuse on other sites. If you've forgotten your current password, log out and use the "Forgot Password" link on the login page.

You can permanently delete your Defensia account and all associated data from the Security settings page.

1. Go to Settings → Security in the dashboard sidebar.

2. Scroll to the Delete Account section at the bottom.

3. Click Delete My Account and enter your password in the confirmation modal.

Warning: This action is irreversible. The following data will be permanently deleted:

• All your servers and their security events, bans, rules, and whitelists
• All your alert channels and notification rules
• Your account credentials and profile

If you have an active Pro subscription, make sure to cancel it from the Billing page before deleting your account to avoid future charges.

When running the agent as a Docker container, it needs access to host log files via volume mounts. If events are not appearing:

1. Verify log volume mounts:

docker inspect defensia-agent | grep -A5 "Mounts"

You should see /var/log mounted as read-only. If missing, recreate the container with the correct mounts:

docker run -d \
  --name defensia-agent \
  --network host --pid host \
  -v /var/log:/var/log:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /etc/defensia:/etc/defensia \
  -e DEFENSIA_TOKEN= \
  ghcr.io/defensia/agent:latest

2. If web server logs are inside another container: You must bind-mount the log directory to the host first:

# In your web server docker-compose.yml:
services:
  nginx:
    volumes:
      - /var/log/nginx:/var/log/nginx

The Defensia agent reads host-level paths. Container-internal logs are not accessible unless bind-mounted to the host filesystem.

The Defensia agent pod is restarting repeatedly after Helm install. Common causes:

1. Check pod logs:

kubectl logs -n defensia-system -l app.kubernetes.io/name=defensia-agent --tail=50

2. Verify the Organization API Key: the chart stores it in a Secret, not a ConfigMap:

kubectl get secret -n defensia-system defensia-agent -o jsonpath="{.data.api-key}" | base64 -d | wc -c

A non-zero length means the key was delivered. The Helm value is apiKey; generate the key in the dashboard from Add server → Kubernetes / Swarm → Generate Key.

3. Check the server URL is reachable from inside the cluster:

kubectl run test-curl --rm -it --image=curlimages/curl -- \
  curl -v https://defensia.cloud/api/v1/agent/heartbeat

4. Check resource limits: If the pod is OOMKilled, increase memory limits in your values.yaml:

resources:
  limits:
    memory: 128Mi  # increase if needed
  requests:
    memory: 64Mi

5. Re-install with updated values:

helm upgrade defensia-agent \
  oci://ghcr.io/defensia/charts/defensia-agent \
  -f values.yaml \
  -n defensia-system

The bot detection system may flag legitimate crawlers (Googlebot, Bingbot, etc.) if their user-agent matches a broad pattern. Here's how to fix it:

1. Check which bot policy is blocking the traffic: Go to the Bots page in the dashboard and look for the fingerprint that matched.

2. Change the policy to "Allow" or "Log": Click on the bot fingerprint and set its policy to Allow (passes through silently) or Log (records the event but does not ban).

3. If the bot was auto-suggested by the agent: Go to Bots → Suggestions and dismiss the suggestion. This prevents it from being re-suggested.

4. Whitelist the bot's IP range: For known crawlers with fixed IP ranges (e.g., Googlebot), add their CIDR to the server whitelist:

# Example: Googlebot IP ranges
66.249.64.0/19
66.249.96.0/19

Whitelisted IPs are still detected and logged, but they are never banned — even if they trigger bot detection or WAF rules.

The agent has a built-in crash loop detection mechanism. If an update causes the agent to crash repeatedly, it automatically rolls back to the previous version.

1. Check if a rollback occurred:

sudo journalctl -u defensia-agent | grep -i "rollback\|backup\|restored"

2. Check the current running version:

/usr/local/bin/defensia-agent --version

3. If the agent is stuck on an old version: The systemd service has an ExecStartPre directive that automatically restores from .bak if the main binary is corrupted or empty.

4. Force a manual update:

ARCH=$(uname -m | sed "s/x86_64/amd64/;s/aarch64/arm64/")
sudo systemctl stop defensia-agent
sudo rm /usr/local/bin/defensia-agent
curl -fsSL \
  "https://github.com/defensia/agent/releases/latest/download/defensia-agent-linux-$ARCH" \
  -o /usr/local/bin/defensia-agent
sudo chmod 755 /usr/local/bin/defensia-agent
sudo systemctl start defensia-agent

After manual update, the agent will resume auto-updates from the next heartbeat cycle.

Defensia uses a progressive ban escalation system. Each time an IP is banned again on the same server, the duration increases automatically:

• 1st ban: 24 hours
• 2nd ban: 7 days
• 3rd ban: 30 days
• 4th+ ban: Permanent (no expiry)

To check an IP's ban history:

1. Go to the Bans page and enable the "Show expired" toggle.

2. Search for the IP address — you'll see all previous bans including expired ones.

3. Click the IP to open the IP Detail Modal, which shows the total times banned and threat score.

To override a ban:

Use the "Unban" button to remove the ban immediately. If the IP is legitimate, add it to the server's Whitelist to prevent future bans.

Escalation is tracked per server — an IP banned on Server A does not affect its escalation level on Server B.