OpenResty Edge™ Log Collection

We recommend using Filebeat for Openresty Edge error log and access log collection.

# Filebeat Installation

Install Filebeat software on all Openresty Edge Node machines. Please refer to Filebeat installation.

# Filebeat Configuration

The following configuration is a simple example of collecting error and access logs on the Openresty Edge Node and sending them to Kafka:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/oredge-node/logs/access.log*
    - /usr/local/oredge-node/logs/error.log*

output.kafka:
  enabled: true
  # fill in the kafka broker address, this can also be multiple broker addresses
  hosts: ["127.0.0.1:9092"]
  topic: edge_node_logs

logging.level: error
# fill in the local host name or IP
name: 127.0.0.1

Save the above to /etc/filebeat/filebeat.yml and start the Filebeat service, which will send the logs to Kafka once successfully started.

sudo systemctl start filebeat

# Validation

We can use the command line tool kafka-console-consumer.sh provided by Kafka to check if the logs were sent successfully.

On a machine with Kafka installed, execute the following command (replace the Kafka installation directory as appropriate), and a log output indicates that Filebeat is working properly.

$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edge_node_logs --from-beginning --max-messages 1
{"@timestamp":"2022-12-27T07:06:53.045Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.3"},"message":"127.0.0.1 - - [14/Dec/2022:16:02:36 +0800] test.com \"GET http://test.com/ HTTP/1.1\" 200 22 0.000 \"-\" \"curl/7.61.1\" - - -","input":{"type":"log"},"ecs":{"version":"8.0.0"},"host":{"name":"127.0.0.1"},"agent":{"version":"8.5.3","ephemeral_id":"fd94a28e-91fe-4365-9584-fe6beb258751","id":"e84c2374-3272-430e-bfe6-84ad2b24b845","name":"127.0.0.1","type":"filebeat"},"log":{"offset":0,"file":{"path":"/usr/local/oredge-node/logs/access.log_20221214.160000"}}}