OpenResty XRay Standard Analyzers

Detailed documentation for the standard analyzers of OpenResty XRay

OpenResty XRay provides standard analyzers for the following application types or stack stacks (the list is still growing fast).

APISIX

OpenResty XRay provides the following tools for analyzing APISIX applications:

CPU

lj-add-timer-lua-fgraph: ngx.timer Add Timer Flamegraph
lj-c-on-cpu: C-Land CPU Flame Graph for LuaJIT
lj-dump-traces: Dump LuaJIT Traces
lj-excep-lua-fgraph: Lua Exception Flame Graph
lj-func-events: Trace LuaJIT function call entry events
lj-gc-step-calls: Number of LuaJIT GC function calls per second
lj-jit-state: Check The LuaJIT JIT State: on/off
lj-live-coroutines: dump the bts of the living Lua coroutines
lj-lua-full-bt: Dump Lua Full Backtrace including the variables in the stack
lj-lua-newcdata: Lua-Land New CData Creation Flame Graph Sampling Tool for LuaJIT Applications.
lj-lua-newfunc: Lua-Land New Function Creation Flame Graph sampling Tool for LuaJIT Applications.
lj-lua-newgco: LuaJIT GC Object Allocation Flame Graph
lj-lua-newstr: LuaJIT String Objects Allocation Flame Graph
lj-lua-newtab: LuaJIT Table Objects Allocation Flame Graph
lj-lua-newthread: LuaJIT Thread Objects Allocation Flame Graph
lj-lua-on-cpu: Lua-Land CPU Flame Graph
lj-lua-tab-resize: LuaJIT Table resize Flame Graph
lj-no-pcre-jit-lua-fgraph: Lua Land Flame Graph for function path that PCRE jit is disabled
lj-slowest-ngx-re: Slowest find/match/split Regular Expression of ngx.re
lj-slowest-str-gmatch: Slowest Lua string.gmatch
lj-slowest-str-gsub: Slowest String Global Substitution For Lua’s string.gsub
lj-slowest-str-match-find: Slowest String Match/Find
lj-str-find-no-plain-fgraph: Lua string.find Searching Plain String Without Plain Mode Set
lj-trace-abort-events-fgraph: Lua Land Trace Abort Events Flame Graph
lj-trace-stats: LuaJIT VM Trace Event Statistics
lj-vm-states: LuaJIT VM States Distribution
ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-cpu-hottest-hosts: Showing the top 10 hostnames consuming the most CPU resources
ngx-cpu-hottest-uris: Showing the top 10 URIs order by request count
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-lua-code-cache: Nginx Lua Code Cache Requests
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-pcre-compile-lua-fgraph: Openresty New PCRE Compilation Lua Land Flame Graph
ngx-shdict-hit-rate: Nginx Shared Dict Hit Rate
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config
ngx-timer-delay-distr: Timeout Value Distribution of Nginx Timer
openresty-compiled-regex-cache: OpenResty Compiled Regex Cache Statistics
openresty-new-timer-fgraph: Openresty New Timer Flamegraph
openresty-running-timers-fgraph: Openresty Running Timer Flamegraph

Disk I/O

lj-read-fgraph: Lua-Land Virtual File System Read/Sendfile Flame Graph
lj-read-latency-fgraph: Lua-Land Virtual File System Read/Sendfile Latency Flame Graph
lj-read-volume-fgraph: Lua-Land Virtual File System Read/Sendfile Volume Flame Graph
lj-write-fgraph: Lua-Land Virtual File System Write Flame Graph
lj-write-latency-fgraph: Lua-Land Virtual File System Write Latency Flame Graph
lj-write-volume-fgraph: Lua-Land Virtual File System Write Volume Flame Graph

Error

lj-lua-ngx-print-error: ngx.print and ngx.say error Flame Graph

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-http-access-time: Nginx HTTP Access Phase Latency
ngx-http-rewrite-time: Nginx HTTP Rewrite Phase Latency
ngx-http-upstream-time: Nginx HTTP Upstream Latency
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-read-req-body-time: Nginx Request Body Reading Time Distribution
ngx-req-latency: Nginx Request Latency
ngx-req-latency-advance: Nginx Request Latency
ngx-req-latency-distr: HTTP Request Latency Distribution
ngx-req-lj-yield-count: Nginx Request LuaJIT Yield Times Flame Graph
ngx-req-lj-yield-latency: Nginx Request LuaJIT Yield Latency Flame Graph
ngx-req-lj-yield-time: Nginx Request LuaJIT Yield Latency
ngx-tls-handshake-latency: Nginx SSL/TLS handshake Latency
sema-wait-fgraph: Semaphore wait Calls

Memory

lj-alloc-stats: Statistics for LuaJIT’s builtin Allocator (Directly allocated memory blocks larger than 128K will not be counted)
lj-c-memory-leak-fgraph: Lua-Land Memory Leak Flame Graphs
lj-dump-loaded-mods: Showing the loaded modules of the LuaJIT VM
lj-err-mem: Luajit Memory Exception
lj-free-stats: Statistics of free chunks for LuaJIT’s builtin Allocator
lj-gco-cdata-fin: CData Object Statistics by Finalizer
lj-gco-cdata-stats: CData Object Statistics by CData Type
lj-gco-ref: GC Object Reference Flame Graph
lj-gco-stat: Statistics for LuaJIT GC Objects
lj-lua-newgco-size: LuaJIT GC Object Allocation Size Flame Graph
lj-mem-realloc-distr: Lua Memory Realloc Size Distribution
lj-mem-realloc-fgraph: Lua Memory Realloc Size Flamegraphs
lj-tab-chain-loops: Find loops in LuaJIT table node chains.
lj-tab-unreachable-node: Find unreachable nodes in the lua tables.
ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shdict-stores: Nginx Shdict Stores Statistics
ngx-shm-lru: Nginx Share Memory LRU Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory
ngx-shm-tree-height: Compute the RBTree Height of Nginx Shared Memory Zones.
resty-memory: Application-Level Memory Usage Breakdown

Miscellaneous

collect-luajit-ffnames: Collect LuaJIT ffnames
lj-config: Configurations for LuaJIT
lj-lua-new-timer-errors: Nginx Lua Tracks Timer Creation Errors
ngx-config: Configurations for Nginx
ngx-config-servers: Dump The Name of The Http Servers
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-lua-sleep: Nginx Lua Sleep Calls Distribution
ngx-worker-id: Nginx Worker ID
openresty-core-dump-analysis: OpenResty Core Dump Analysis

Network I/O

lj-lua-ngx-exit-fgraph: Lua-Land Nginx Request Status Code Flame Graph
lj-lua-socket-close-fgraph: Lua-Land close socket Flame Graph
ngx-downstream-keepalive-stats: HTTP Downstream Request Connection Keepalive Statistics
ngx-dump-cur-req: Dump Nginx Request
ngx-dump-dns: Dump Nginx DNS configuration and domain caches
ngx-dump-req: Dump Nginx Request
ngx-keepalive-statistics: Request/Upstream Connection Keep-alive Statistics
ngx-proxy-cache-status: Proxy Cache Status Distribution
ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-count: Count Nginx Subrequests and Main Requests
ngx-req-counts-by-hosts: Showing the top 10 hostnames order by request counts
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-req-size-by-hosts: Showing the top 10 hostnames order by request size
ngx-req-status-code: Request Status Code Per Second
ngx-rps: Nginx RPS Statistics
ngx-ssl-server-conf: Sample the Server Configurations by Requests
ngx-ssl-session-ticket-key: Nginx SSL Session Ticket Key Configuration
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
ngx-tls-handshake-info: SSL/TLS Session handshake information Statistics
ngx-tls-session-resumption: Downstream SSL/TLS Session Resumption Statistics
ngx-top-latency-reqs: IP packets of top latency HTTP requests
ngx-upstream-err-packets: Capture the upstream packets that have specific error code
openresty-cosocket-connect: OpenResty Cosocket Connect Operation Flame Graph
openresty-cosocket-err-fgraph: OpenResty Cosocket Error Flame Graph
openresty-cosocket-err-packets: IP Packets of OpenResty Cosocket Which has Errors
openresty-cosocket-no-keepalive: OpenResty Cosocket Without Keepalive Flame Graph
openresty-cosocket-pool: OpenResty Cosocket Pool Statistics
openresty-cosocket-timeout: OpenResty Cosocket Timeout Distribution
socket-close-fgraph: close socket Flame Graph
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration

off-CPU

lj-c-off-cpu: C-Land off-CPU Flame Graph for LuaJIT
lj-lua-off-cpu: Lua-Land off-CPU Flame Graph

C/C++

OpenResty XRay provides the following tools for analyzing C/C++ applications:

CPU

c-on-cpu: C-Land CPU Flame Graph
cpu-hogs: cpu hogs of the system
epoll-level-event-fgraph: Epoll Level Trigger Event Flame Graph
epoll-loop-blocking-distr: Epoll Loop Blocking Latency Distribution
epoll-wait-ret-distr: Return Value Distribution of the epoll_wait
epoll-wait-timers: Wait timers for epoll
epoll-wait-timers-fgraph: Wait timers flamegraph for epoll
kernel-on-cpu: Kernel-Land on-CPU Flame Graph
page-fault-by-pid: Page fault order by process ID
pagefault-fgraph: Page Fault Flame Graph
process-ctx-sw-stats: Process Context Switches Statistics
sys-load-avg-by-app: System Load Average By Application

Disk I/O

bench-io-create-files: benchmark of creating and writing into the files
diskstat: Virtual File System Read Write By Disk Name
disktop-by-proc: Virtual File System Read Write N Process
file-system-fgraph: File System Flame Graph
vfs-read-fgraph: Virtual File System Read/Sendfile Flame Graph
vfs-read-latency-fgraph: Virtual File System Read/Sendfile Latency Flame Graph
vfs-read-latency-top-files: Virtual File System Read Latency Top N files (including sendfile)
vfs-read-top-files: Virtual File System Read Top N files (including sendfile)
vfs-read-volume-fgraph: Virtual File System Read/Sendfile Volume Flame Graph
vfs-write-fgraph: Virtual File System Write Flame Graph
vfs-write-latency-fgraph: Virtual File System Write Latency Flame Graph
vfs-write-latency-top-files: Virtual File System Write Latency Top N files
vfs-write-top-files: Virtual File System Write Top N files
vfs-write-volume-fgraph: Virtual File System Write Volume Flame Graph

Latency

epoll-wait-overrun-time-distr: Schedule Latency Distribution
func-latency-distr: Function Latency Distribution
socket-send-recv-latency-fgraph: Socket Send/Recv Message Latency Flame Graph

Memory

c-alloc-fgraph: C Memory Allocation Flame Graph
c-count-alloc-free: C Application-Level Memory API Calls
c-memory: C Application-Level Memory Usage Breakdown
c-memory-leak-fgraph: C-Land Memory Leak Flame Graphs
kmalloc-leak-fgraph: kmalloc leakage Flame Graph
libc-chunks: Distributions of Memory Chunk Sizes of the Memory Allocator
libc-mallinfo: Information about memory allocations performed by malloc and related functions
mmap-leaks: Mmap Leak Backtrace Sampling Tools for Leak Flame Graphs
process-memory: Virtual Memory Usage Breakdown
thread-stack-loaded: Loaded stack memory used by all the threads

Miscellaneous

gen-core-dump: Generate the core dump file via gcore
generic-core-dump-analysis: Generic Core Dump Analysis
proc-reap: Reap the snapshot process
proc-snap: Create a snapshot of the process (only for expert)
signal-watcher: Sent or Received signals
syscall-execve: fork and execute commands
syscall-write-stat: Statistic of errno for write/Writev/send syscall

Network I/O

kernel-dropwatch: Packet Drop Flame Graph
socket-leak-fgraph: Socket Leakage Flame Graph
socket-send-recv-message: Socket Send/Recv Message Flame Graph
socket-send-recv-volume-fgraph: Socket Send/Recv Message Volume Flame Graph
tcp-reset-fgraph: TCP Reset Flame Graph

off-CPU

c-off-cpu: C-Land off-CPU Flame Graph

Envoy

OpenResty XRay provides the following tools for analyzing Envoy applications:

CPU

c-on-cpu: C-Land CPU Flame Graph

off-CPU

c-off-cpu: C-Land off-CPU Flame Graph

Erlang

OpenResty XRay provides the following tools for analyzing Erlang applications:

CPU

erl-on-cpu: Erlang-Land on-CPU Flame Graph

off-CPU

erl-off-cpu: Erlang-Land off-CPU Flame Graph

Go

OpenResty XRay provides the following tools for analyzing Go applications:

CPU

go-alloc-fgraph: Go Memory Allocation Flame Graph
go-excep-fgraph: Go-Land Exception Flame Graph
go-live-goroutines: dump the bts of the living goroutines
go-on-cpu: Go-Land on-CPU Flame Graph

Latency

go-net-http-req-latency: Go net/http Request Latency

Memory

go-gco-ref: Go-Land GC Object Reference Flame Graph
go-memory: Go-Land Memory Breakdown

Miscellaneous

go-core-dump-analysis: Go Application Core Dump Analysis

Network I/O

krakend-req-canceled: Krakend actively canceled requests

off-CPU

go-off-cpu: Go-Land off-CPU Flame Graph

Java

OpenResty XRay provides the following tools for analyzing Java applications:

CPU

java-find-method-entry: Find Java method entries
java-ftrace: Trace Java function call entry events
java-on-cpu: Java-Land on-CPU Flame Graph
pre-java-ftrace: Preliminary Analyzer for tracing Java function call entry events

Disk I/O

java-read-fgraph: Java-Land Virtual File System Read/Sendfile Flame Graph
java-read-latency-fgraph: java-Land Virtual File System Read/Sendfile Latency Flame Graph
java-read-volume-fgraph: Java-Land Virtual File System Read/Sendfile Volume Flame Graph
java-write-fgraph: Java-Land Virtual File System Write Flame Graph
java-write-latency-fgraph: Java-Land Virtual File System Write Latency Flame Graph
java-write-volume-fgraph: Java-Land Virtual File System Write Volume Flame Graph

Memory

java-gco-ref: Java-Land GC Object Reference Flame Graph

Network I/O

java-kafka-req-latency: Java Kafka Request Latency
java-tomcat-req-latency: Java Tomcat Request Latency
java-tomcat-top-latency-reqs: IP packets of top latency HTTP requests
pre-java-kafka-req-latency: Preliminary Analyzer for Java Kafka Request Latency
pre-java-tomcat-req-latency: Preliminary Analyzer for Java Tomcat Request Latency

off-CPU

java-off-cpu: Java-Land off-CPU Flame Graph

Kong

OpenResty XRay provides the following tools for analyzing Kong applications:

CPU

lj-add-timer-lua-fgraph: ngx.timer Add Timer Flamegraph
lj-c-on-cpu: C-Land CPU Flame Graph for LuaJIT
lj-dump-traces: Dump LuaJIT Traces
lj-excep-lua-fgraph: Lua Exception Flame Graph
lj-func-events: Trace LuaJIT function call entry events
lj-gc-step-calls: Number of LuaJIT GC function calls per second
lj-jit-state: Check The LuaJIT JIT State: on/off
lj-live-coroutines: dump the bts of the living Lua coroutines
lj-lua-full-bt: Dump Lua Full Backtrace including the variables in the stack
lj-lua-newcdata: Lua-Land New CData Creation Flame Graph Sampling Tool for LuaJIT Applications.
lj-lua-newfunc: Lua-Land New Function Creation Flame Graph sampling Tool for LuaJIT Applications.
lj-lua-newgco: LuaJIT GC Object Allocation Flame Graph
lj-lua-newstr: LuaJIT String Objects Allocation Flame Graph
lj-lua-newtab: LuaJIT Table Objects Allocation Flame Graph
lj-lua-newthread: LuaJIT Thread Objects Allocation Flame Graph
lj-lua-on-cpu: Lua-Land CPU Flame Graph
lj-lua-tab-resize: LuaJIT Table resize Flame Graph
lj-no-pcre-jit-lua-fgraph: Lua Land Flame Graph for function path that PCRE jit is disabled
lj-slowest-ngx-re: Slowest find/match/split Regular Expression of ngx.re
lj-slowest-str-gmatch: Slowest Lua string.gmatch
lj-slowest-str-gsub: Slowest String Global Substitution For Lua’s string.gsub
lj-slowest-str-match-find: Slowest String Match/Find
lj-str-find-no-plain-fgraph: Lua string.find Searching Plain String Without Plain Mode Set
lj-trace-abort-events-fgraph: Lua Land Trace Abort Events Flame Graph
lj-trace-stats: LuaJIT VM Trace Event Statistics
lj-vm-states: LuaJIT VM States Distribution
ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-cpu-hottest-hosts: Showing the top 10 hostnames consuming the most CPU resources
ngx-cpu-hottest-uris: Showing the top 10 URIs order by request count
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-lua-code-cache: Nginx Lua Code Cache Requests
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-pcre-compile-lua-fgraph: Openresty New PCRE Compilation Lua Land Flame Graph
ngx-shdict-hit-rate: Nginx Shared Dict Hit Rate
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config
ngx-timer-delay-distr: Timeout Value Distribution of Nginx Timer
openresty-compiled-regex-cache: OpenResty Compiled Regex Cache Statistics
openresty-new-timer-fgraph: Openresty New Timer Flamegraph
openresty-running-timers-fgraph: Openresty Running Timer Flamegraph

Disk I/O

lj-read-fgraph: Lua-Land Virtual File System Read/Sendfile Flame Graph
lj-read-latency-fgraph: Lua-Land Virtual File System Read/Sendfile Latency Flame Graph
lj-read-volume-fgraph: Lua-Land Virtual File System Read/Sendfile Volume Flame Graph
lj-write-fgraph: Lua-Land Virtual File System Write Flame Graph
lj-write-latency-fgraph: Lua-Land Virtual File System Write Latency Flame Graph
lj-write-volume-fgraph: Lua-Land Virtual File System Write Volume Flame Graph

Error

lj-lua-ngx-print-error: ngx.print and ngx.say error Flame Graph

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-http-access-time: Nginx HTTP Access Phase Latency
ngx-http-rewrite-time: Nginx HTTP Rewrite Phase Latency
ngx-http-upstream-time: Nginx HTTP Upstream Latency
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-read-req-body-time: Nginx Request Body Reading Time Distribution
ngx-req-latency: Nginx Request Latency
ngx-req-latency-advance: Nginx Request Latency
ngx-req-latency-distr: HTTP Request Latency Distribution
ngx-req-lj-yield-count: Nginx Request LuaJIT Yield Times Flame Graph
ngx-req-lj-yield-latency: Nginx Request LuaJIT Yield Latency Flame Graph
ngx-req-lj-yield-time: Nginx Request LuaJIT Yield Latency
ngx-tls-handshake-latency: Nginx SSL/TLS handshake Latency
sema-wait-fgraph: Semaphore wait Calls

Memory

lj-alloc-stats: Statistics for LuaJIT’s builtin Allocator (Directly allocated memory blocks larger than 128K will not be counted)
lj-c-memory-leak-fgraph: Lua-Land Memory Leak Flame Graphs
lj-dump-loaded-mods: Showing the loaded modules of the LuaJIT VM
lj-err-mem: Luajit Memory Exception
lj-free-stats: Statistics of free chunks for LuaJIT’s builtin Allocator
lj-gco-cdata-fin: CData Object Statistics by Finalizer
lj-gco-cdata-stats: CData Object Statistics by CData Type
lj-gco-ref: GC Object Reference Flame Graph
lj-gco-stat: Statistics for LuaJIT GC Objects
lj-lua-newgco-size: LuaJIT GC Object Allocation Size Flame Graph
lj-mem-realloc-distr: Lua Memory Realloc Size Distribution
lj-mem-realloc-fgraph: Lua Memory Realloc Size Flamegraphs
lj-tab-chain-loops: Find loops in LuaJIT table node chains.
lj-tab-unreachable-node: Find unreachable nodes in the lua tables.
ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shdict-stores: Nginx Shdict Stores Statistics
ngx-shm-lru: Nginx Share Memory LRU Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory
ngx-shm-tree-height: Compute the RBTree Height of Nginx Shared Memory Zones.
resty-memory: Application-Level Memory Usage Breakdown

Miscellaneous

collect-luajit-ffnames: Collect LuaJIT ffnames
lj-config: Configurations for LuaJIT
lj-lua-new-timer-errors: Nginx Lua Tracks Timer Creation Errors
ngx-config: Configurations for Nginx
ngx-config-servers: Dump The Name of The Http Servers
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-lua-sleep: Nginx Lua Sleep Calls Distribution
ngx-worker-id: Nginx Worker ID
openresty-core-dump-analysis: OpenResty Core Dump Analysis

Network I/O

lj-lua-ngx-exit-fgraph: Lua-Land Nginx Request Status Code Flame Graph
lj-lua-socket-close-fgraph: Lua-Land close socket Flame Graph
ngx-downstream-keepalive-stats: HTTP Downstream Request Connection Keepalive Statistics
ngx-dump-cur-req: Dump Nginx Request
ngx-dump-dns: Dump Nginx DNS configuration and domain caches
ngx-dump-req: Dump Nginx Request
ngx-keepalive-statistics: Request/Upstream Connection Keep-alive Statistics
ngx-proxy-cache-status: Proxy Cache Status Distribution
ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-count: Count Nginx Subrequests and Main Requests
ngx-req-counts-by-hosts: Showing the top 10 hostnames order by request counts
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-req-size-by-hosts: Showing the top 10 hostnames order by request size
ngx-req-status-code: Request Status Code Per Second
ngx-rps: Nginx RPS Statistics
ngx-ssl-server-conf: Sample the Server Configurations by Requests
ngx-ssl-session-ticket-key: Nginx SSL Session Ticket Key Configuration
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
ngx-tls-handshake-info: SSL/TLS Session handshake information Statistics
ngx-tls-session-resumption: Downstream SSL/TLS Session Resumption Statistics
ngx-top-latency-reqs: IP packets of top latency HTTP requests
ngx-upstream-err-packets: Capture the upstream packets that have specific error code
openresty-cosocket-connect: OpenResty Cosocket Connect Operation Flame Graph
openresty-cosocket-err-fgraph: OpenResty Cosocket Error Flame Graph
openresty-cosocket-err-packets: IP Packets of OpenResty Cosocket Which has Errors
openresty-cosocket-no-keepalive: OpenResty Cosocket Without Keepalive Flame Graph
openresty-cosocket-pool: OpenResty Cosocket Pool Statistics
openresty-cosocket-timeout: OpenResty Cosocket Timeout Distribution
socket-close-fgraph: close socket Flame Graph
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration

off-CPU

lj-c-off-cpu: C-Land off-CPU Flame Graph for LuaJIT
lj-lua-off-cpu: Lua-Land off-CPU Flame Graph

Lua

OpenResty XRay provides the following tools for analyzing Lua applications:

CPU

lua-excep-fgraph: Lua-Land Exception Flame Graph for standard Lua
lua-on-cpu: Lua-Land CPU Flame Graph for standard Lua

Memory

lua-gco-ref: GC Object Reference Flame Graph for Stand Lua

off-CPU

lua-off-cpu: Lua-Land off-CPU Flame Graph for standard Lua

Nginx

OpenResty XRay provides the following tools for analyzing Nginx applications:

CPU

ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-req-latency: Nginx Request Latency
ngx-req-latency-advance: Nginx Request Latency
ngx-req-latency-distr: HTTP Request Latency Distribution

Memory

ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory

Miscellaneous

ngx-config: Configurations for Nginx
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-worker-id: Nginx Worker ID

Network I/O

ngx-dump-all-reqs: Dump all Nginx HTTP Requests
ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-ssl-session-ticket-key: Nginx SSL Session Ticket Key Configuration
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
ngx-top-latency-reqs: IP packets of top latency HTTP requests
ngx-upstream-err-packets: Capture the upstream packets that have specific error code
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration

OpenResty

OpenResty XRay provides the following tools for analyzing OpenResty applications:

CPU

lj-add-timer-lua-fgraph: ngx.timer Add Timer Flamegraph
lj-c-on-cpu: C-Land CPU Flame Graph for LuaJIT
lj-coro-c-on-cpu: C-Land CPU Flame Graph for LuaJIT and Coro
lj-dump-traces: Dump LuaJIT Traces
lj-excep-lua-fgraph: Lua Exception Flame Graph
lj-func-events: Trace LuaJIT function call entry events
lj-gc-step-calls: Number of LuaJIT GC function calls per second
lj-jit-state: Check The LuaJIT JIT State: on/off
lj-live-coroutines: dump the bts of the living Lua coroutines
lj-lua-full-bt: Dump Lua Full Backtrace including the variables in the stack
lj-lua-newcdata: Lua-Land New CData Creation Flame Graph Sampling Tool for LuaJIT Applications.
lj-lua-newfunc: Lua-Land New Function Creation Flame Graph sampling Tool for LuaJIT Applications.
lj-lua-newgco: LuaJIT GC Object Allocation Flame Graph
lj-lua-newstr: LuaJIT String Objects Allocation Flame Graph
lj-lua-newtab: LuaJIT Table Objects Allocation Flame Graph
lj-lua-newthread: LuaJIT Thread Objects Allocation Flame Graph
lj-lua-on-cpu: Lua-Land CPU Flame Graph
lj-lua-tab-resize: LuaJIT Table resize Flame Graph
lj-no-pcre-jit-lua-fgraph: Lua Land Flame Graph for function path that PCRE jit is disabled
lj-slowest-ngx-re: Slowest find/match/split Regular Expression of ngx.re
lj-slowest-str-gmatch: Slowest Lua string.gmatch
lj-slowest-str-gsub: Slowest String Global Substitution For Lua’s string.gsub
lj-slowest-str-match-find: Slowest String Match/Find
lj-str-find-no-plain-fgraph: Lua string.find Searching Plain String Without Plain Mode Set
lj-trace-abort-events-fgraph: Lua Land Trace Abort Events Flame Graph
lj-trace-stats: LuaJIT VM Trace Event Statistics
lj-vm-states: LuaJIT VM States Distribution
ngx-access-log-buffer-size: Nginx Access Log Buffer Size Configuration
ngx-cpu-hottest-hosts: Showing the top 10 hostnames consuming the most CPU resources
ngx-cpu-hottest-uris: Showing the top 10 URIs order by request count
ngx-epoll-event-stats: Statistics of Epoll Events Received
ngx-epoll-wait-timers: Tracking the Nginx-Land Epoll Wait Routines
ngx-gzip-compression-level: Nginx GZIP Compression Level Statistics
ngx-gzip-compression-types: Nginx GZIP Compression Types Statistics
ngx-lua-code-cache: Nginx Lua Code Cache Requests
ngx-open-file-cache-conf: Nginx Open File Cache Configuration
ngx-pcre-compile-lua-fgraph: Openresty New PCRE Compilation Lua Land Flame Graph
ngx-shdict-hit-rate: Nginx Shared Dict Hit Rate
ngx-slowest-config-re: Slowest Regular Expression in Nginx Rewrite or Find Config
ngx-timer-delay-distr: Timeout Value Distribution of Nginx Timer
openresty-compiled-regex-cache: OpenResty Compiled Regex Cache Statistics
openresty-new-timer-fgraph: Openresty New Timer Flamegraph
openresty-running-timers-fgraph: Openresty Running Timer Flamegraph

Disk I/O

lj-read-fgraph: Lua-Land Virtual File System Read/Sendfile Flame Graph
lj-read-latency-fgraph: Lua-Land Virtual File System Read/Sendfile Latency Flame Graph
lj-read-volume-fgraph: Lua-Land Virtual File System Read/Sendfile Volume Flame Graph
lj-write-fgraph: Lua-Land Virtual File System Write Flame Graph
lj-write-latency-fgraph: Lua-Land Virtual File System Write Latency Flame Graph
lj-write-volume-fgraph: Lua-Land Virtual File System Write Volume Flame Graph

Error

lj-lua-ngx-print-error: ngx.print and ngx.say error Flame Graph

Latency

ngx-err-log-files: Nginx Error Log Files
ngx-err-log-lvl-distr: Nginx Error Log Level Distribution
ngx-http-access-time: Nginx HTTP Access Phase Latency
ngx-http-rewrite-time: Nginx HTTP Rewrite Phase Latency
ngx-http-upstream-time: Nginx HTTP Upstream Latency
ngx-https-handshake-time: Nginx HTTPs handshake - useless
ngx-read-req-body-time: Nginx Request Body Reading Time Distribution
ngx-req-latency: Nginx Request Latency
ngx-req-latency-advance: Nginx Request Latency
ngx-req-latency-distr: HTTP Request Latency Distribution
ngx-req-lj-yield-count: Nginx Request LuaJIT Yield Times Flame Graph
ngx-req-lj-yield-latency: Nginx Request LuaJIT Yield Latency Flame Graph
ngx-req-lj-yield-time: Nginx Request LuaJIT Yield Latency
ngx-tls-handshake-latency: Nginx SSL/TLS handshake Latency
sema-wait-fgraph: Semaphore wait Calls

Memory

lj-alloc-stats: Statistics for LuaJIT’s builtin Allocator (Directly allocated memory blocks larger than 128K will not be counted)
lj-c-memory-leak-fgraph: Lua-Land Memory Leak Flame Graphs
lj-dump-loaded-mods: Showing the loaded modules of the LuaJIT VM
lj-err-mem: Luajit Memory Exception
lj-free-stats: Statistics of free chunks for LuaJIT’s builtin Allocator
lj-gco-cdata-fin: CData Object Statistics by Finalizer
lj-gco-cdata-stats: CData Object Statistics by CData Type
lj-gco-ref: GC Object Reference Flame Graph
lj-gco-stat: Statistics for LuaJIT GC Objects
lj-lua-newgco-size: LuaJIT GC Object Allocation Size Flame Graph
lj-mem-realloc-distr: Lua Memory Realloc Size Distribution
lj-mem-realloc-fgraph: Lua Memory Realloc Size Flamegraphs
lj-tab-chain-loops: Find loops in LuaJIT table node chains.
lj-tab-unreachable-node: Find unreachable nodes in the lua tables.
ngx-mem-pools: Nginx Memory Pool Statistics
ngx-shdict-stores: Nginx Shdict Stores Statistics
ngx-shm-lru: Nginx Share Memory LRU Statistics
ngx-shm-slabs: Slab Memory Usage in Nginx Shared Memory
ngx-shm-tree-height: Compute the RBTree Height of Nginx Shared Memory Zones.
resty-memory: Application-Level Memory Usage Breakdown

Miscellaneous

collect-luajit-ffnames: Collect LuaJIT ffnames
lj-config: Configurations for LuaJIT
lj-lua-new-timer-errors: Nginx Lua Tracks Timer Creation Errors
ngx-config: Configurations for Nginx
ngx-config-servers: Dump The Name of The Http Servers
ngx-dump-timers: Dump nginx timers in the rbtree
ngx-graceful-exits: Nginx Graceful Exit Event
ngx-lua-sleep: Nginx Lua Sleep Calls Distribution
ngx-worker-id: Nginx Worker ID
openresty-core-dump-analysis: OpenResty Core Dump Analysis

Network I/O

lj-lua-ngx-exit-fgraph: Lua-Land Nginx Request Status Code Flame Graph
lj-lua-socket-close-fgraph: Lua-Land close socket Flame Graph
ngx-downstream-keepalive-stats: HTTP Downstream Request Connection Keepalive Statistics
ngx-dump-all-reqs: Dump all Nginx HTTP Requests
ngx-dump-cur-req: Dump Nginx Request
ngx-dump-dns: Dump Nginx DNS configuration and domain caches
ngx-dump-req: Dump Nginx Request
ngx-keepalive-statistics: Request/Upstream Connection Keep-alive Statistics
ngx-proxy-cache-status: Proxy Cache Status Distribution
ngx-read-req-body-packets: Capture the packets of the HTTP request that the read body time is greater than the threshold
ngx-req-count: Count Nginx Subrequests and Main Requests
ngx-req-counts-by-hosts: Showing the top 10 hostnames order by request counts
ngx-req-distribution: Nginx Traffic Distribution By Calls of Finalize Request
ngx-req-err-packets: IP Packets of Specified HTTP Errors
ngx-req-size-by-hosts: Showing the top 10 hostnames order by request size
ngx-req-status-code: Request Status Code Per Second
ngx-rps: Nginx RPS Statistics
ngx-ssl-server-conf: Sample the Server Configurations by Requests
ngx-ssl-session-ticket-key: Nginx SSL Session Ticket Key Configuration
ngx-status-code-fgraph: Nginx Request Status Code Flame Graph
ngx-tls-handshake-info: SSL/TLS Session handshake information Statistics
ngx-tls-session-resumption: Downstream SSL/TLS Session Resumption Statistics
ngx-top-latency-reqs: IP packets of top latency HTTP requests
ngx-upstream-err-packets: Capture the upstream packets that have specific error code
openresty-cosocket-connect: OpenResty Cosocket Connect Operation Flame Graph
openresty-cosocket-err-fgraph: OpenResty Cosocket Error Flame Graph
openresty-cosocket-err-packets: IP Packets of OpenResty Cosocket Which has Errors
openresty-cosocket-no-keepalive: OpenResty Cosocket Without Keepalive Flame Graph
openresty-cosocket-pool: OpenResty Cosocket Pool Statistics
openresty-cosocket-timeout: OpenResty Cosocket Timeout Distribution
socket-close-fgraph: close socket Flame Graph
tcpdump-upstream-no-keepalive: capture the packets that have no HTTP upstream ‘keepalive’ configuration

off-CPU

lj-c-off-cpu: C-Land off-CPU Flame Graph for LuaJIT
lj-lua-off-cpu: Lua-Land off-CPU Flame Graph

Perl

OpenResty XRay provides the following tools for analyzing Perl applications:

CPU

perl-excep-fgraph: Perl-Land Exception Flame Graph
perl-func-events: Trace perl function call entry events
perl-on-cpu: Perl-Land on-CPU Flame Graph

Disk I/O

perl-read-fgraph: Perl-Land Virtual File System Read/Sendfile Flame Graph
perl-read-latency-fgraph: Perl-Land Virtual File System Read/Sendfile Latency Flame Graph
perl-read-volume-fgraph: Perl-Land Virtual File System Read/Sendfile Volume Flame Graph
perl-write-fgraph: Perl-Land Virtual File System Write Flame Graph
perl-write-latency-fgraph: Perl-Land Virtual File System Write Latency Flame Graph
perl-write-volume-fgraph: Perl-Land Virtual File System Write Volume Flame Graph

Memory

perl-gco-ref: Perl GC Object Reference Flame Graph

off-CPU

perl-off-cpu: Perl-Land off-CPU Flame Graph

PHP

OpenResty XRay provides the following tools for analyzing PHP applications:

CPU

php-excep-fgraph: PHP-Land Exception Flame Graph
php-func-events: Trace PHP function call entry events
php-on-cpu: PHP-Land CPU Flame Graph

Disk I/O

php-read-fgraph: PHP-Land Virtual File System Read/Sendfile Flame Graph
php-read-latency-fgraph: PHP-Land Virtual File System Read/Sendfile Latency Flame Graph
php-read-volume-fgraph: PHP-Land Virtual File System Read/Sendfile Volume Flame Graph
php-write-fgraph: PHP-Land Virtual File System Write Flame Graph
php-write-latency-fgraph: PHP-Land Virtual File System Write Latency Flame Graph
php-write-volume-fgraph: PHP-Land Virtual File System Write Volume Flame Graph

Memory

php-bin-distr: Distributions of Memory Chunk Sizes in PHP Zend Allocator
php-gco-ref: PHP GC Object Reference Flame Graph
php-gco-stat: Statistics for PHP GC Objects

off-CPU

php-off-cpu: PHP-Land off-CPU Flame Graph

PostgreSQL

OpenResty XRay provides the following tools for analyzing PostgreSQL applications:

Memory

pg-mem-pool: PostgreSQL Memory Pool Size

Python

OpenResty XRay provides the following tools for analyzing Python applications:

CPU

py-excep-fgraph: Python-Land Exception Flame Graph
py-func-events: Trace python function call entry events
py-on-cpu: Python CPU Flame Graph
pyftrace: Trace python function call entry/exit events

Disk I/O

py-read-fgraph: Python-Land Virtual File System Read/Sendfile Flame Graph
py-read-latency-fgraph: Python-Land Virtual File System Read/Sendfile Latency Flame Graph
py-read-volume-fgraph: Python-Land Virtual File System Read/Sendfile Volume Flame Graph
py-write-fgraph: Python-Land Virtual File System Write Flame Graph
py-write-latency-fgraph: Python-Land Virtual File System Write Latency Flame Graph
py-write-volume-fgraph: Python-Land Virtual File System Write Volume Flame Graph