Weekly and Monthly Alert Insights
1.15.20
SRE and Security teams rely heavily on alerts to know whether their systems are experiencing issues and to prevent any future outages. At Mezmo, formerly known as LogDNA, customers can set alerts that trigger when specific logs match (presence alerts) or set an alert to go off if there are expected lines that haven’t come through (absence alerts). These alerts can be set up with various channels so you can be alerted in the product of your choice (Slack, Email, PagerDuty, etc). You can learn more about Mezmo alerts here.
As a Security Officer, you can leverage Mezmo alerts as a form of threat detection. With the right alerts set, you can be notified of unusual activity in your system. Alert Insights is a summary of alerts that have been triggered in the last week or month for you to review to ensure you haven’t missed any important alerts, such as failed logins, attempted logins, root logins, etc. If you see many attempted or failed logins, it is important to know so you can investigate.
An important aspect of alert insights is knowing the change from week to week, this is why we have provided a delta section in addition to the count of how many times an alert was triggered. The delta shows the percentage of increase or decrease activity of a specific alert so you know whether what you’re reviewing is considered ‘normal’.
We understand that teams change, credentials might get rotated, and the email or Slack channel that was set up to receive alerts may change. This will usually result in alerts that are missed because Mezmo is unable to deliver the alerts due to expired/outdated credentials.
To tackle this problem, we’ve added a section called “Failed Alerts”. We provided a summary of all alerts that have been set up, triggered, but failed to deliver. Failed Alerts could occur due to various reasons, but in general, it means failure to get a response from the webhook. A few common reasons include, URLs being invalid, server not responding, webhook returned non 2xx OK code, etc. This is important because your team may be missing important alerts that are triggering but failed to make it to the channel that you wanted to be notified in. Armed with this list of “Failed Alerts”, you know exactly which ones need to be updated so your team can receive important alerts again.
Main use cases:
- Insights into unusual activities such as spikes in fatal error alerts, failed login alerts, etc
- Trends in alert activity (increased or decreased) for specific alerts
- A list of active alerts that have been triggered have failed at delivery
- Insight into alerts that are inactive and should be considered for deprecation
- Insight into alerts that have been extremely noisy and should be considered for refinement
You can find this feature under: /manage/ingestion in the Email Digest section
Check off the boxes to receive Weekly or Monthly Email digest. That’s it! No configuration necessary, we will automatically include the Alert Insights in your email digest.