Open-source alternatives to Datadog and New Relic

Datadog and New Relic are both excellent, and both have a way of producing bills that make you sit down. Datadog's infrastructure monitoring starts at $15 per host a month on an annual plan, about £11. Ten servers is $1,800 a year, about £1,364, before you add logs, which are charged by the gigabyte and by the million events. New Relic's model is different: 100 GB of data a month free, then $0.40 a GB, plus $99 for each extra full user on the Standard plan.

For a big engineering team, that money buys a lot. For a small firm that wants to know when a server is running out of disk, it's overkill. The open-source tools below handle that job well, with no charge per host.

Zabbix

Zabbix is the all-in-one choice. You install it once and it does the monitoring, the alerting and the dashboards.

• Monitors nearly anything. A native agent for all the major operating systems, plus SNMP for switches, routers and printers.
• Finds things for you. Network discovery scans your IP ranges and can bring new devices in automatically.
• Templates out of the box, so you don't have to work out what's worth watching.
• Alerts where you'll see them: email, SMS, Slack, Teams, Telegram or webhooks, with escalations if nobody responds.
• Dashboards and scheduled PDF reports, which is handy for showing the boss.
• 100% free software under AGPLv3, with paid support if you want it.

It takes some setting up. You need a server and a database (MySQL, MariaDB or PostgreSQL), and the docs' example for a small install is 2 CPU cores and 8 GB of memory. The current release is 7.4, with 8.0 in development. Zabbix's interface has a reputation for having a lot in it, and in my view that's fair: it's a big product and it takes a while to find your way around.

Grafana

Grafana is the dashboard. It doesn't collect anything by itself. You point it at your data, and it draws you excellent charts and can send alerts.

It's light. The docs suggest 2 cores and 2 to 4 GB of memory for production, on Debian, Ubuntu, RHEL and others. It uses SQLite by default, but the docs say to use MySQL or PostgreSQL in production.

It pairs nicely with Loki, also from Grafana Labs, for logs. Loki indexes only labels rather than the full text, keeping storage cheap. It's AGPL too, and it's the usual open-source answer to Datadog Logs.

The licence is worth a note. Grafana moved to AGPL-3.0 some years ago. For a business that simply uses it, that causes no problems. It only matters if you change the code and offer it to other people.

Prometheus

Prometheus is the thing that collects the numbers. It's the standard for metrics in the container world.

• It pulls data. It reads metrics from your servers and apps over HTTP, on a schedule.
• Exporters for many common services, which translate their stats into something Prometheus can read.
• PromQL, a powerful query language, used for dashboards and alert rules.
• Alertmanager, a separate component, handles sending the alerts.
• Simple storage. Each server stores its data locally and stands alone, which keeps it reliable.

It's not for everything. It's metrics only, so logs need another tool. And the project itself says it isn't suited to anything needing 100% accuracy, such as per-request billing. PromQL takes a while to learn, and a full setup means running Prometheus, exporters, Alertmanager and Grafana. So you're assembling a kit rather than installing a product.

Which one?

My pick is Zabbix. For a small firm with a few servers and some network kit, one system covers monitoring, alerts and dashboards, and the templates give you a head start. The main drawback is the learning curve: it's a big product with a lot to get your head round.

• A handful of servers, network kit, one system: Zabbix.
• Containers, your own apps, someone technical on hand: Prometheus and Grafana, with Loki for logs.

The UK angle

• Mostly not personal data. CPU charts and disk space are nothing to worry about, and a US-hosted tool like Datadog or New Relic is fine for most small firms. Both are big, established suppliers.
• Logs are where to look. Application logs often hold email addresses, IP addresses and sometimes worse. Whichever tool you use, don't log what you don't need, and set a sensible retention period.
• Check the costs of EU storage. New Relic charges an extra $0.05 per GB a month to keep data in its EU data centre. If that's something you need, it's worth including in your sums.
• You keep your history. With self-hosted tools, retention is just a question of disk space, rather than a line on an invoice.

Honest drawbacks

• No application tracing out of the box. Datadog and New Relic show you exactly which line of code slowed a page down. Zabbix doesn't do that, and with Grafana you'd be adding further pieces.
• Your monitoring needs monitoring. If the box goes down, the alerts go with it. Something outside it should check that it's alive.
• Time to set up. The paid tools are quicker to get going. Here, expect to spend time tuning alerts so they don't cry wolf.
• Learning curve. PromQL and Zabbix's interface both take a while to learn.
• Someone has to own it. Monitoring nobody looks at is worse than none, because it gives you a false sense of security.

If you need custom dashboards, or monitoring built into your own software, the team at Dev Partners can help with that.

Sources

• Zabbix features
• Zabbix requirements
• Grafana installation requirements
• Grafana Loki documentation
• Grafana Loki licence
• Prometheus overview
• Datadog pricing
• New Relic pricing