OpenResty XRay Standard Analyzers

Detailed documentation for the standard analyzers of OpenResty XRay

OpenResty XRay provides standard analyzers for the following application types or stack stacks (the list is still growing fast).

C/C++

OpenResty XRay provides the following tools for analyzing C/C++ applications:

CPU

c-off-cpu: C-Land off-CPU Flame Graph
c-on-cpu: C-Land CPU Flame Graph
cpu-hogs: cpu hogs of the system
epoll-level-event-fgraph: Epoll Level Trigger Event Flame Graph
epoll-wait-ret-distr: Return Value Distribution of the epoll_wait
epoll-wait-timers: Wait timers for epoll
epoll-wait-timers-fgraph: Wait timers flamegraph for epoll
kernel-on-cpu: Kernel-Land on-CPU Flame Graph
pagefault-fgraph: Page Fault Flame Graph
process-ctx-sw-stats: Process Context Switches Statistics

Disk I/O

bench-io-create-files: benchmark of creating and writing into the files
diskstat: Virtual File System Read Write By Disk Name
disktop-by-proc: Virtual File System Read Write N Process
file-system-fgraph: File System Flame Graph
vfs-read-latency-top-files: Virtual File System Read Latency Top N files(including sendfile)
vfs-read-top-files: Virtual File System Read Top N files(including sendfile)
vfs-read-write-fgraph: Virtual File System Read/Write/Sendfile Flame Graph
vfs-read-write-latency-fgraph: Virtual File System Read/Write/Sendfile Latency Flame Graph
vfs-read-write-volume-fgraph: Virtual File System Read/Write/Sendfile Volume Flame Graph
vfs-write-latency-top-files: Virtual File System Write Latency Top N files
vfs-write-top-files: Virtual File System Write Top N files

Latency

epoll-loop-blocking-distr: Epoll Loop Blocking Latency Distribution
epoll-sched-latency-distr: Schedule Latency Distribution
func-latency-distr: Function Latency Distribution
socket-send-recv-latency-fgraph: Socket Send/Recv Message Latency Flame Graph

Memory

c-alloc-fgraph: C Memory Allocation Flame Graph
c-count-alloc-free: C Application-Level Memory API Calls
c-memory: C Application-Level Memory Usage Breakdown
c-memory-leak-fgraph: C-Land Memory Leak Flame Graphs
glibc-chunks: Distributions of Memory Chunk Sizes in Glibc Allocator
kmalloc-leak-fgraph: kmalloc leakage Flame Graph
libc-chunks: Distributions of Memory Chunk Sizes of the Memory Allocator
libc-mallinfo: Information about memory allocations performed by malloc and related functions
mmap-leaks: Mmap Leak Backtrace Sampling Tools for Leak Flame Graphs
process-memory: Virtual Memory Usage Breakdown

Miscellaneous

generic-core-dump-analysis: Generic Core Dump Analysis
ngx-signals: Nginx Sent or Received signals
syscall-execve: fork and execute commands
syscall-write-stat: Statistic of errno for write/Writev/send syscall

Network I/O

kernel-dropwatch: Packet Drop Flame Graph
socket-leak-fgraph: Socket Leakage Flame Graph
socket-send-recv-message: Socket Send/Recv Message Flame Graph
socket-send-recv-volume-fgraph: Socket Send/Recv Message Volume Flame Graph
tcp-reset-fgraph: TCP Reset Flame Graph

Nginx

OpenResty XRay provides the following tools for analyzing Nginx applications:

CPU

ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-req-latency: Nginx Request Latency
req-latency-distr: HTTP Request Latency Distribution

Memory

ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory

Miscellaneous

ngx-config: Configurations for Nginx
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-worker-id: Nginx Worker ID

Network I/O

ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration
upstream-error-packets: Capture the upstream packets that have specific error code

OpenResty

OpenResty XRay provides the following tools for analyzing OpenResty applications:

CPU

lj-add-timer-lua-fgraph: ngx.timer Add Timer Flamegraph
lj-c-off-cpu: C-Land off-CPU Flame Graph for LuaJIT
lj-c-on-cpu: C-Land CPU Flame Graph for LuaJIT
lj-excep-lua-fgraph: Lua Exception Flame Graph
lj-gc-step-calls: Number of LuaJIT GC function calls per second
lj-jit-state: Check The LuaJIT JIT State: on/off
lj-lua-err-msg: Lua Exception Error Message
lj-lua-full-bt: Dump Lua Full Backtrace including the variables in the stack
lj-lua-newthread: LuaJIT Thread Objects Allocation Flame Graph
lj-lua-off-cpu: Lua-Land off-CPU Flame Graph
lj-lua-on-cpu: Lua-Land CPU Flame Graph
lj-no-pcre-jit-lua-fgraph: Lua Land Flame Graph for function path that PCRE jit is disabled
lj-running-threads: dump the bts of the running Lua threads
lj-slowest-ngx-re: Slowest find/match/split Regular Expression of ngx.re
lj-slowest-str-gmatch: Slowest Lua string.gmatch
lj-slowest-str-gsub: Slowest String Global Substitution For Lua’s string.gsub
lj-slowest-str-match-find: Slowest String Match/Find
lj-str-find-no-plain-fgraph: Lua string.find Searching Plain String Without Plain Mode Set
lj-trace-abort-events-fgraph: Lua Land Trace Abort Events Flame Graph
lj-trace-stats: LuaJIT VM Trace Event Statistics
lj-vm-states: LuaJIT VM States Distribution
ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-cpu-hottest-hosts: Showing the top 10 hostnames consuming the most CPU resources
ngx-cpu-hottest-uris: Showing the top 10 URIs order by request count
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-lua-code-cache: Nginx Lua Code Cache Requests
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-pcre-compile-lua-fgraph: Openresty New PCRE Compilation Lua Land Flame Graph
ngx-shdict-hit-rate: Nginx Shared Dict Hit Rate
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config
ngx-timer-delay-distr: Timeout Value Distribution of Nginx Timer
openresty-compiled-regex-cache: OpenResty Compiled Regex Cache Statistics
openresty-new-timer-fgraph: Openresty New Timer Flamegraph
openresty-running-timers-fgraph: Openresty Running Timer Flamegraph

Disk I/O

lj-read-write-fgraph: Lua-Land Virtual File System Read/Write/Sendfile Flame Graph
lj-read-write-latency-fgraph: Lua-Land Virtual File System Read/Write/Sendfile Latency Flame Graph
lj-read-write-volume-fgraph: Lua-Land Virtual File System Read/Write/Sendfile Volume Flame Graph

Error

lj-lua-ngx-print-error: ngx.print and ngx.say error Flame Graph

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-http-access-time: Nginx HTTP Access Phase Latency
ngx-http-rewrite-time: Nginx HTTP Rewrite Phase Latency
ngx-http-upstream-time: Nginx HTTP Upstream Latency
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-read-req-body-time: Nginx Request Body Reading Time Distribution
ngx-req-latency: Nginx Request Latency
ngx-req-latency-us: Nginx Request Latency
ngx-req-lj-yield-count: Nginx Request LuaJIT Yield Times Flame Graph
ngx-req-lj-yield-latency: Nginx Request LuaJIT Yield Latency Flame Graph
ngx-req-lj-yield-time: Nginx Request LuaJIT Yield Latency
ngx-tls-handshake-latency: Nginx SSL/TLS handshake Latency
req-latency-distr: HTTP Request Latency Distribution
sema-wait-fgraph: Semaphore wait Calls

Memory

lj-alloc-stats: Statistics for LuaJIT’s builtin Allocator
lj-c-memory-leak-fgraph: Lua-Land Memory Leak Flame Graphs
lj-dump-loaded-mods: Showing the loaded modules of the LuaJIT VM
lj-err-mem: Luajit Memory Exception
lj-free-stats: Statistics of free chunks for LuaJIT’s builtin Allocator
lj-gco-ref: GC Object Reference Flame Graph
lj-gco-stat: Statistics for LuaJIT GC Objects
lj-lua-newcdata: Lua-Land New CData Creation Flame Graph Sampling Tool for LuaJIT Applications.
lj-lua-newfunc: Lua-Land New Function Creation Flame Graph sampling Tool for LuaJIT Applications.
lj-lua-newgco: LuaJIT GC Object Allocation Flame Graph
lj-lua-newgco-size: LuaJIT GC Object Allocation Size Flame Graph
lj-lua-newstr: LuaJIT String Objects Allocation Flame Graph
lj-lua-newtab: LuaJIT Table Objects Allocation Flame Graph
lj-lua-tab-resize: LuaJIT Table resize Flame Graph
lj-mem-realloc-distr: Lua Memory Realloc Size Distribution
lj-mem-realloc-fgraph: Lua Memory Realloc Size Flamegraphs
lj-tab-chain-loops: Find loops in LuaJIT table node chains.
lj-tab-unreachable-node: Find unreachable nodes in the lua tables.
ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shdict-stores: Nginx Shdict Stores Statistics
ngx-shm-lru: Nginx Share Memory LRU Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory
ngx-shm-tree-height: Compute the RBTree Height of Nginx Shared Memory Zones.
resty-memory: Application-Level Memory Usage Breakdown

Miscellaneous

collect-luajit-ffnames: Collect LuaJIT ffnames
lj-config: Configurations for LuaJIT
lj-lua-new-timer-errors: Nginx Lua Tracks Timer Creation Errors
ngx-config: Configurations for Nginx
ngx-config-servers: Dump The Name of The Http Servers
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-lua-sleep: Nginx Lua Sleep Calls Distribution
ngx-worker-id: Nginx Worker ID
openresty-core-dump-analysis: OpenResty Core Dump Analysis

Network I/O

lj-lua-ngx-exit-fgraph: Lua-Land Nginx Request Status Code Flame Graph
ngx-downstream-keepalive-stats: HTTP Downstream Request Connection Keepalive Statistics
ngx-dump-cur-req: Dump Nginx Request
ngx-dump-req: Dump Nginx Request
ngx-keepalive-statistics: Request/Upstream Connection Keep-alive Statistics
ngx-proxy-cache-status: Proxy Cache Status Distribution
ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-count: Count Nginx Subrequests and Main Requests
ngx-req-counts-by-hosts: Showing the top 10 hostnames order by request counts
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-req-size-by-hosts: Showing the top 10 hostnames order by request size
ngx-req-status-code: Request Status Code Per Second
ngx-rps: Nginx RPS Statistics
ngx-ssl-server-conf: Sample the Server Configurations by Requests
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
ngx-tls-handshake-info: SSL/TLS Session handshake information Statistics
ngx-tls-session-resumption: Downstream SSL/TLS Session Resumption Statistics
openresty-cosocket-connect: OpenResty Cosocket Connect Operation Flame Graph
openresty-cosocket-err-fgraph: OpenResty Cosocket Error Flame Graph
openresty-cosocket-err-packets: IP Packets of OpenResty Cosocket Which has Errors
openresty-cosocket-no-keepalive: OpenResty Cosocket Without Keepalive Flame Graph
openresty-cosocket-pool: OpenResty Cosocket Pool Statistics
openresty-cosocket-timeout: OpenResty Cosocket Timeout Distribution
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration
upstream-error-packets: Capture the upstream packets that have specific error code