OpenResty XRay Abnormal Events

Abnormal Events Triggering

OpenResty XRay captures application monitoring metrics such as CPU usage, memory usage, and disk write IO. And if the monitoring metrics fluctuate by a threshold, it generates an abnormal event and triggers the analyzer to execute.

We refer to the 3σ rule for the determination of abnormal events, and the specific rules are as follows.

  1. the absolute value of the difference between the current value and the mean value within 60 seconds must be greater than the set minimum variation
  2. the absolute value of the difference between the current value and the mean value within 60 seconds is greater than the set n times the standard deviation
  3. the current application does not have a running analyzer task

When these conditions are met, an abnormal event is generated to trigger the analyzer to run.

We have configured some common analyzer triggering rules by default.

We can add rules for custom analyzers to run.

The threshold for event triggering can also be adjusted.

In order not to miss abnormal events, the analyzer will be triggered to run the acquisition immediately after the event occurs. But this will generate a lot of invalid acquisition tasks triggered by the metric burrs. We will judge again four seconds after the event is generated, and mark the event as valid if all the values within these four seconds match the conditions.

Invalid events are not displayed in the event list by default, and all caught abnormal events can be displayed with this switch.

Abnormal Events Notification

We provide two types of notifications, email and Webhook. After configuring event notifications, users will be notified in time when an abnormal event occurs.