The manual way is to SSH into each one, run df -h, free -m, ps and ss -ltn, and copy the results into a spreadsheet. It is slow, you will skip steps on server 37, and nobody does it twice. What I wanted instead was one command, one report, the same checks every time, with the worst servers at the top.

So I built that with Ansible, and tested it against a fleet of 50 simulated servers with five problems hidden in them. This post explains how it works, shows the code that matters, and covers the things that went wrong, because those taught me the most.

The ideas behind it

A few Ansible concepts carry the whole project.

Agentless. Ansible needs only SSH and Python on the servers, which nearly every Linux machine already has. Nothing to install on 50 machines first. It runs from one control machine and reaches out.

Inventory. The list of servers and how they are grouped. A static file goes stale, so I wrote a small Python script that asks Docker which containers are running and hands Ansible the list. For cloud servers the same idea works with a cloud API, and Ansible ships ready-made plugins for that. (Docs: dynamic inventory)

Playbooks, plays, tasks. A playbook is a YAML file with one or more plays. Each play says "on these hosts, run these tasks", and each task calls a module such as ping, command or template.

Roles. A role is a folder with a fixed layout (tasks, defaults, templates, meta) that packages one job, so a playbook can simply say "apply this role". The audit is two roles, which keeps each piece small and reusable. (Docs: roles)

Read-only by design. The audit only reads: /proc, df, ps, ss. It changes nothing, so it is safe on production. Every command task sets changed_when: false, which keeps Ansible's output honest: a command ran, but nothing changed.

An unreachable server is a result, not a crash. With ignore_unreachable: true, a dead server does not stop the run. It becomes a row in the report, at the top, marked UNREACHABLE. That is exactly what you want from an audit: the server that does not answer is the first thing you need to know about. (Docs: error handling)

The design

I do not have 50 servers, so I built 50 lookalikes: Docker containers running a real SSH server and real processes. One image plays every role. An environment variable picks the service: nginx for web, a small Python server on 8080 for app, and a listener on 5432 for db. Each container carries labels (fleet.audit=true, fleet.role=web), and a script generates the Docker Compose file so nobody types 50 service blocks.

The fleet has 20 web, 20 app and 10 db hosts, with five problems seeded on purpose:

If the audit finds exactly these five and calls everything else OK, I know it works.

The playbook has two plays, and each one is a role (audit_host and audit_report), a pattern I call collect, then aggregate:

1. Collect. Runs on every host, 25 at a time. Each host reads its own numbers, compares them with thresholds and builds one small record: status, disk, memory, ports and a list of issues.

2. Aggregate. Runs once on my own machine. It reads every host's record, adds a placeholder for hosts that returned nothing, sorts worst first and writes the HTML, CSV and JSON reports.

Keeping the two apart means the hosts never need to know about each other, and the report logic lives in one place.

The code that matters

Survive a dead host. The first task of the audit_host role pings, and everything else is skipped for hosts that did not answer:

- name: Check the host is reachable
  ansible.builtin.ping:
  register: audit_host_reach
  ignore_unreachable: true

- name: Audit the host
  when: not (audit_host_reach.unreachable | default(false))
  block:
    # ...all the checks...

Collect raw numbers. Each check is a read-only command, saved with register:

- name: Read filesystem usage (mount point and use%)
  ansible.builtin.shell: |
    set -o pipefail
    df -P -x squashfs -x iso9660 -x erofs | awk 'NR > 1 && $6 !~ /^\/(proc|sys|dev|run|etc)(\/|$)/ {print $6, $5}'
  args:
    executable: /bin/bash
  register: audit_host_disk_out
  changed_when: false

Turn text into numbers. A set_fact task with Jinja filters parses the output:

audit_host_disk_pcts: "{{ audit_host_disk_out.stdout_lines | map('regex_search', '[0-9]+(?=%$)') | select | map('int') | list }}"

Keep the policy out of the logic. Thresholds and what each role should run live in group_vars, so web servers are expected to run nginx on port 80 and db servers are not. The role ships sensible defaults, and group_vars overrides them:

# playbooks/group_vars/all.yml
audit_host_disk_warn_pct: 80
audit_host_disk_crit_pct: 90

# playbooks/group_vars/web.yml
audit_host_expected_processes: [nginx]
audit_host_expected_ports: [80]

Decide. Each check adds a message to a list, or an empty list. Joined together, they give the host's status:

audit_host_crit: >-
  {{
    (['disk ' ~ audit_host_disk_max ~ '% full (critical at ' ~ audit_host_disk_crit_pct ~ '%)']
      if (audit_host_disk_max | int) >= (audit_host_disk_crit_pct | int)
      else [])
    + (['process not running: ' ~ (audit_host_missing_procs | join(', '))]
      if (audit_host_missing_procs | length) > 0
      else [])
    + (['port not listening: ' ~ (audit_host_missing_ports | join(', '))]
      if (audit_host_missing_ports | length) > 0
      else [])
  }}

The status is CRIT if that list has anything in it, otherwise WARN if the warning list does, otherwise OK. The second play then collects those records through Ansible's hostvars, sorts by severity and fills the report templates. Both plays live in roles, so audit.yml itself is only a few lines.

What went wrong

The first real run did not match my expectations, and these are the problems worth sharing.

1. Every host was CRIT, with all numbers at zero. The audit said 49 of 49 reachable hosts were critical, with 0% disk and open ports [0]. But running df and ss by hand on one host showed perfectly normal output. So the hosts were fine, and my parsing was wrong. I had used regex_replace with a '\\1' backreference to pull out the numbers, and the backreference was not being substituted, so every value turned into 0. I printed the issues, disks and ports fields from the JSON report, compared them with the raw command output, and rewrote the parsing with regex_search and lookahead and lookbehind, which need no backreference. The lesson: look at the raw data before blaming the system, and a test that only checks the report's shape (right columns, right host count) will happily pass when every number is wrong.

2. A healthy machine reported a full disk. My local test host came out CRIT at 100% disk. The culprit was a read-only image mount that is always 100% full (iso9660 here; snap packages show up as squashfs on Ubuntu). The fix was df -x squashfs -x iso9660 -x erofs. An audit has to know what counts as a real filesystem.

3. The playbook worked, but ansible ping did not. The ad hoc ping failed with "Permission denied (publickey)" because it tried to log in as my own user. I had put remote_user and the key path in playbooks/group_vars, and variables in a folder beside the playbook are loaded only when you run that playbook, not for ad hoc commands. Moving them into ansible.cfg fixed both.

4. Every container looked busy. Containers share the host's kernel, so /proc/loadavg shows my PC's load on all 50 hosts. That is a limit of simulating servers with containers. I raised the threshold for the simulation, wrote it down in the README, and noted that real servers should use about 1.0 load per CPU.

Smaller ones: a callback plugin (timer) that lives in a separate collection and could not load, ansible-lint rejecting lines over 160 characters, and Python refusing pip install --user on a modern system (a virtual environment solves it).

Forks: a result that surprised me

forks is how many hosts Ansible works on at the same time. The default is 5, so 50 hosts take 10 rounds. I expected 25 forks to be several times faster. I timed two runs of each:

Only about 8% faster. The reason is in the user and sys times: Ansible used between 1.5 and 2 CPU cores the whole run. My laptop was running all 50 containers, their SSH servers and Ansible's worker processes at once, so extra forks just queued for CPU. With real servers, most of each task is network waiting, the control machine sits idle, and more forks pay off much more.

The lesson: parallelism helps when you are waiting, not when you are computing. And a simulation can flatter or hide a setting, so say what it cannot show. The same run also does about 700 task executions (50 hosts times 14 tasks), each a separate SSH exchange. Collecting everything with one script per host would cut that sharply, and is the first thing I would try next.

What a reviewer suggested

After I shared the project, a friend with cloud experience gave me two pieces of feedback. Both were fair, and both changed the project in v0.2.0.

Use roles. My playbook was one long file holding every check and all the report logic. Now it is two roles, and playbooks/audit.yml only calls them:

playbooks/
  audit.yml                 two plays, each one calls a role
  group_vars/               thresholds and per-role expectations
  roles/
    audit_host/             collects and judges one host (runs on every host)
    audit_report/           builds the CSV, HTML and JSON (runs once)

I treated this as a refactor: same behaviour, new structure. The acceptance test was the same as before. The audit still had to find exactly the five seeded faults, which means 50 hosts with 1 UNREACHABLE, 3 CRIT, 1 WARN and 45 OK, and it did. One detail worth knowing: ansible-lint wants role variables prefixed with the role name (audit_host_disk_warn_pct, not disk_warn_pct), so two roles can never overwrite each other's settings.

Never let Ansible hold cloud keys. The second suggestion was to reach AWS, Azure and GCP through roles and short-lived identity instead of secret keys. The reasoning is sound: a stored access key can leak from a laptop, a repository or a CI log, and nobody remembers to rotate it. An assumed role, a managed identity or an attached service account gives Ansible credentials that expire within about an hour, so there is nothing long-lived to steal. I recorded the decision as ADR 0004 and added example inventories for the AWS, Azure and GCP inventory plugins.

I want to be clear about what is and is not done. Those example inventories have not been run yet, because the project has no cloud accounts attached, and the repo says so. Real runs will come in stages, each as its own release: AWS first, then Azure, then GCP.

Engineering around it

The automation is only half of the project. The rest is what makes it safe to change:

• Tests. Ten unit tests for the inventory script (using fake Docker data), ansible-lint, a syntax check, and an end-to-end run of the real playbook against the local machine as a one-host fleet.

• CI. GitHub Actions runs three jobs on every pull request: lint, a secret scan over the full git history, and the Ansible tests.

• A protected main. A GitHub ruleset blocks direct pushes and requires a pull request and all three checks. Merges are squashed, so history reads as one line per change.

• Signed commits. Each commit is signed with an SSH key, so GitHub shows Verified.

• Decision records and a runbook. Short ADRs explain why the fleet is simulated with containers, why the inventory is a script and why the design is collect-then-aggregate, and why cloud access should use short-lived roles instead of stored keys. The runbook says what to do about each finding.

What I would do differently next time:

• Run it against real VMs, starting with AWS, and compare the numbers with the simulation.

• Add service_facts to check systemd services properly.

• Collect everything in one script per host to cut SSH round trips.

• Keep reports over time to see trends, and run make audit-strict on a schedule so a CRIT or UNREACHABLE host fails the job and alerts someone.

• Turn host key checking back on. I disabled it only because containers get new keys on every rebuild.

Try it yourself

The whole project is on GitHub: OpeyemiAdeniji/ansible-fleet-audit. You need WSL2 or Linux, Docker and Python. Then:

git clone git@github.com:OpeyemiAdeniji/ansible-fleet-audit.git
cd ansible-fleet-audit
python3 -m venv .venv && source .venv/bin/activate
make setup
make fleet-up     # builds and starts the 50 containers
make audit        # writes reports/latest.html

The repo has a runbook for acting on each finding, and decision records for the main design choices.

If you take one thing from this post, let it be this: when your tool reports that everything is broken, check the raw data before you blame the system. Mine was a regex. Yours will be something else, and the habit is the same.

This is the first of a series of DevOps and SRE projects I am building and writing up. Next: a golden machine image built with Packer and hardened to a CIS benchmark.

Further reading:

• Ansible dynamic inventory

• Ansible error handling

• Ansible forks setting

• ansible-lint

• Ansible roles

• Conventional Commits