OpenResty Edge™ Log Collection

We recommend using Filebeat for Openresty Edge error log and access log collection.

Filebeat Installation

Install Filebeat software on all Openresty Edge Node machines. Please refer to Filebeat installation.

Filebeat Configuration

The following configuration is a simple example of collecting error and access logs on the Openresty Edge Node and sending them to Kafka:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/oredge-node/logs/access.log*
    - /usr/local/oredge-node/logs/error.log*

output.kafka:
  enabled: true
  # fill in the kafka broker address, this can also be multiple broker addresses
  hosts: ["127.0.0.1:9092"]
  topic: edge_node_logs

logging.level: error
# fill in the local host name or IP
name: 127.0.0.1

Save the above to /etc/filebeat/filebeat.yml and start the Filebeat service, which will send the logs to Kafka once successfully started.

sudo systemctl start filebeat

Validation

We can use the command line tool kafka-console-consumer.sh provided by Kafka to check if the logs were sent successfully.

On a machine with Kafka installed, execute the following command (replace the Kafka installation directory as appropriate), and a log output indicates that Filebeat is working properly.

$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edge_node_logs --from-beginning --max-messages 1
{"@timestamp":"2022-12-27T07:06:53.045Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.3"},"message":"127.0.0.1 - - [14/Dec/2022:16:02:36 +0800] test.com \"GET http://test.com/ HTTP/1.1\" 200 22 0.000 \"-\" \"curl/7.61.1\" - - -","input":{"type":"log"},"ecs":{"version":"8.0.0"},"host":{"name":"127.0.0.1"},"agent":{"version":"8.5.3","ephemeral_id":"fd94a28e-91fe-4365-9584-fe6beb258751","id":"e84c2374-3272-430e-bfe6-84ad2b24b845","name":"127.0.0.1","type":"filebeat"},"log":{"offset":0,"file":{"path":"/usr/local/oredge-node/logs/access.log_20221214.160000"}}}