Nobody "likes" getting alerts. In the best-case scenario, an alert is received because something went (or is about to go) wrong. In those cases, recipients can at least be grateful they found out before things became worse. Far more often, however, people hate getting alerts for the exact opposite reason: They alerts are meaningless, trivial, or just plain wrong - a source of constant interruptions, false alarms, unplanned work, and "noise."
While many are convinced that this is the inherent nature of alerts (and monitoring in general) the truth is that it can be so much better. Well-crafted alerts based on insightful monitoring are a benefit to the business and a downright gift to the recipient, saving hours of investigation and thousands of dollars.
The reality is that whether your organization views alerts (and the monitoring behind them) as a curse or a blessing depends largely on the design and implementation of those alerts, more so than any specific monitoring tool or technique. The good news is that, like most things in technology, good design can be taught and learned.
In this talk, we'll give a brief tour of the alerting hall of horrors, and then provide real-world, vendor-agnostic techniques to make alerts meaningful, effective, valuable, and actionable (and, as a bonus, we'll show how to make them manageable, too!). By breaking a few bad habits; understanding how and why vendors put their tools together in particular ways; and learning a few new concepts, you'll have people emailing you to say "thank goodness I got that alert!".
Now there's something you probably don't hear every day.
Leon Adato: In my sordid career, I have been an actor, bug exterminator and wild-animal remover (nothing crazy like pumas or wildebeests. Just skunks, snakes, and raccoons.), electrician, carpenter, stage-combat instructor, ASL interpreter, and Sunday school teacher. Oh, yeah, I've also worked with computers. While my first keyboard was an IBM Selectric, and my first digital experience was on an Atari 400, my professional work in tech started in 1989 (when you got Windows 286 for free on twelve 5¼” when you bought Excel 1.0). Since then I've worked as a classroom instructor, courseware designer, helpdesk operator, desktop support staff, sysadmin, network engineer, and software distribution technician. Then, about 25 years ago, I got involved with monitoring. I've worked with a wide range of tools: Tivoli, BMC, OpenView, janky perl scripts, Nagios, SolarWinds, DOS batch files, Zabbix, Grafana, New Relic, and other assorted nightmare fuel. I've designed solutions for companies that were modest (~10 systems), significant (5,000 systems), and ludicrous (250,000 systems). In that time, I've learned a lot about monitoring and observability in all its many and splendid forms.
Speaker: Leon Adato - Kentik