Collect OpenResty Edge Logs with Vector and ClickHouse

This guide describes how to collect logs from OpenResty Edge Nodes with Vector and store them centrally in ClickHouse. ClickHouse runs on a dedicated database server through Docker Compose, while Vector runs as a systemd service on each Edge Node.

Edge Node log files -> local Vector -> HTTPS -> ClickHouse

# Collection scope

After deployment, Vector writes logs to five tables in the app database:

All four access log types must use the JSON formats below that match the deployment packages and include app_id. SNI proxies use the stream log file and table. Vector parses NGINX error logs in their default format. Logs that cannot be parsed are retained with parsed_ok set to 0, and their original contents are stored in raw_message.

Error log collection is optional. If you do not want to write error logs to ClickHouse, remove the following three complete configuration blocks from edge-vector-deployment/vector.yaml before deploying Vector:

• sources.nginx_error.
• transforms.normalize_error.
• sinks.clickhouse_error.

Remove only the Vector collection configuration. Do not delete the original error log files from the Edge Node. You can leave the unused nginx_error_logs table in ClickHouse; it remains empty and consumes no log data storage.

# Deployment prerequisites

Prepare one ClickHouse server and at least one Edge Node that has been added to Edge Admin.

This guide uses the following example values. Replace them as appropriate for your environment:

The delivery contains two packages with matching versions. The following examples use version 1.1.0:

# Extract the ClickHouse package

Download edge-clickhouse-deployment-1.1.0.tar.gz to /opt on the ClickHouse server and extract it:

cd /opt
sudo curl -O https://openresty.com/client/oredge/edge-clickhouse-deployment-1.1.0.tar.gz
sudo tar -xzf edge-clickhouse-deployment-1.1.0.tar.gz

The extracted directory has the following structure:

/opt/
└── edge-clickhouse-deployment/
    ├── .env.example
    ├── auto-deploy.sh
    ├── certs/
    ├── config.d/
    ├── docker-compose.yml
    ├── initdb/
    └── users.d/

# Extract the Vector package

Download edge-vector-deployment-1.1.0.tar.gz to /opt on every Edge Node and extract it:

cd /opt
sudo curl -O https://openresty.com/client/oredge/edge-vector-deployment-1.1.0.tar.gz
sudo tar -xzf edge-vector-deployment-1.1.0.tar.gz

The extracted directory has the following structure:

/opt/
└── edge-vector-deployment/
    ├── auto-deploy.sh
    ├── systemd/
    ├── vector.env.example
    └── vector.yaml

The extracted deployment directories do not contain version numbers. Always use /opt/edge-clickhouse-deployment and /opt/edge-vector-deployment. The two package versions must match. Stable directory names prevent version changes from affecting paths in deployment and upgrade scripts.

Verify that the time and time zone settings are correct on every host. ClickHouse stores table timestamps in UTC; query clients can convert timestamps to a display time zone as needed. Prepare a server certificate for clickhouse.example.com that is issued by a CA trusted by the clients. The certificate Subject Alternative Name must contain this hostname. Use a publicly trusted CA for a public service. If you use an internal organizational CA, install its CA certificate on every Edge Node.

# Configure Edge access logs

First ensure that both Edge Admin and Edge Node support $app_id in stream, HTTP forward proxy, and SOCKS5 logs. Use matching deployment packages that include the new proxy log tables and Vector pipelines. For an existing installation, follow Upgrade an existing deployment below before publishing formats that use the new variables. Use version 1.1.0 or later of both deployment packages; older 1.0.0 archives may lack the new proxy tables and pipelines.

Upgrading does not modify existing custom log formats. Add "app_id":"$app_id" explicitly. Republish HTTP forward proxy and SOCKS5 application configurations so that the application IDs are sent to the nodes.

# Configure the HTTP access log format

1. Sign in to Edge Admin and go to Global Config > General > Log.

2. Configure the HTTP access log format in either of the following ways:

   • Modify the default main format.
   • Add a format named vector-json and make it the default access log format.

3. Replace the selected format with the following content:

   {
   "app_id": "$app_id",
   "timestamp":"$time_iso8601",
   "remote_addr":"$remote_addr",
   "remote_user":"$remote_user",
   "http_host":"$http_host",
   "request":"$request",
   "status":$status,
   "body_bytes_sent":$body_bytes_sent,
   "request_time":$request_time,
   "http_referer":"$http_referer",
   "http_user_agent":"$http_user_agent",
   "upstream_addr":"$upstream_addr",
   "upstream_status":"$upstream_status",
   "upstream_connect_time":"$upstream_connect_time",
   "upstream_header_time":"$upstream_header_time",
   "upstream_response_time":"$upstream_response_time",
   "upstream_cache_status":"$upstream_cache_status",
   "pid":$pid,
   "req_id":"$req_id",
   "request_length":$request_length,
   "invalid_referer":"$invalid_referer",
   "internal_request":"$internal_request"
   }
   

4. Set Escape to json.

5. Check the HTTP applications whose logs you want to collect. Applications that use the default format automatically use this configuration. If an application explicitly selects another format, change it to main or vector-json.

6. Save and publish the configuration.

This format includes the application ID, request, status code, request latency, upstream latency, and other fields. For descriptions of the available variables, see OpenResty Edge access log variables.

Edge Admin automatically removes line breaks from a log format when it publishes the configuration. Although the JSON above is displayed across multiple lines for readability, each log entry written by the Edge Node occupies one line. The resulting file uses JSON Lines (JSONL).

# Configure the stream access log format

This format applies to TCP, UDP, SSL, and SNI proxy applications.

1. Locate the stream access log format under Global Config > General > Log.

2. Replace the original space-delimited format with the following content:

   {
   "app_id":"$app_id",
   "timestamp":"$time_iso8601",
   "remote_addr":"$remote_addr",
   "protocol":"$protocol",
   "status":$status,
   "bytes_sent":$bytes_sent,
   "bytes_received":$bytes_received,
   "session_time":$session_time,
   "upstream_addr":"$upstream_addr",
   "upstream_bytes_sent":"$upstream_bytes_sent",
   "upstream_bytes_received":"$upstream_bytes_received",
   "upstream_connect_time":"$upstream_connect_time",
   "server_addr":"$server_addr",
   "server_port":$server_port
   }
   

3. Set Escape to json, then save and publish the configuration.

The JSON format generates timestamp from $time_iso8601, and Vector writes it to the ClickHouse event_time field. The remaining fields map directly to the connection status, traffic, and upstream information from the original stream access log. Edge Admin also removes line breaks from this format, so each stream log record occupies one JSONL line.

If you do not need to collect stream logs, skip this step and leave the corresponding Vector collection configuration in place. Vector produces no stream log data when no matching file exists.

# Configure the HTTP forward proxy access log format

Open the HTTP forward proxy application in Edge Admin, enable access logging, set Escape to json, and use the following format:

{
"app_id":"$app_id",
"timestamp":"$time",
"client_addr":"$client_addr",
"proxy_port":$proxy_port,
"upstream_host":"$upstream_host",
"upstream_port":"$upstream_port",
"up_bytes":$up_bytes,
"down_bytes":$down_bytes,
"method":"$method",
"status":$status,
"failure":"$failure",
"account_key":"$account_key",
"account":"$account",
"url":"$url"
}

The example uses multiple lines for readability. Join them into a single line before saving, then save and publish the application configuration. The default log file is http_proxy_access.log.

# Configure the SOCKS5 access log format

Open the SOCKS5 application in Edge Admin, enable access logging, set Escape to json, and use the following format:

{
"app_id":"$app_id",
"timestamp":"$time",
"client_addr":"$client_addr",
"proxy_port":$proxy_port,
"upstream_host":"$upstream_host",
"upstream_port":"$upstream_port",
"up_bytes":$up_bytes,
"down_bytes":$down_bytes,
"username":"$username",
"status":$status,
"bandwidth":"$bandwidth",
"failure":"$failure",
"request_time":$request_time
}

Join the format into a single line before saving, then save and publish the application configuration. The default log file is socks5_access.log.

Both proxy formats use $time, which produces YYYY-MM-DD HH:MM:SS without a timezone offset. Vector parses this field in its local timezone, so the Vector and Edge Node processes must use the same timezone. These transforms also accept RFC 3339 timestamps. HTTP and stream logs continue to use $time_iso8601, which includes a timezone offset.

# Understand application IDs

All four access log tables have an app_id UInt64 column. Select both the correct log table and the application ID when querying, because IDs can overlap between application types. TCP, UDP, SSL, and SNI share the stream table and its application ID namespace. The error log table has no app_id column.

Keep $app_id quoted in all four JSON log formats. An unmatched stream connection or a proxy configuration without an application ID can emit an empty string. With "app_id":"$app_id", the JSON remains valid and Vector converts the empty ID to app_id = 0 while preserving other fields and parsed_ok = 1. Numeric strings are converted to integers. Republish proxy configurations after upgrading so that matched applications supply their IDs.

The old unquoted format can emit invalid JSON such as "app_id":, when the variable is empty. Vector retains that line in raw_message, but structured fields are lost, parsed_ok = 0, and the event time falls back to ingestion time. Quoting prevents this failure; it does not repair historical records or add support for an unknown variable on unsupported node versions.

parsed_ok = 1 means only that JSON parsing succeeded. It does not guarantee that the record has a valid application ID. When verifying the configuration, also check that new records contain the expected application ID.

# Keep the log formats consistent

The access log formats must match the table schema in the ClickHouse package of the same version. The following configurations in the two delivery packages are designed to work together:

• The HTTP, stream, HTTP forward proxy, and SOCKS5 JSON formats in this guide define the fields emitted by Edge.
• normalize_http_access, normalize_stream_access, normalize_http_proxy_access, and normalize_socks5_access in edge-vector-deployment/vector.yaml define field parsing and conversion.
• edge-clickhouse-deployment/initdb/001-nginx-logs.sql defines the ClickHouse column names and types.

Do not add or remove fields, or change field types, only in Edge Admin. To customize a log format, update the Vector transforms and ClickHouse schema at the same time, then verify all three configurations before publishing them.

# Match the Edge log filenames

OpenResty Edge uses stream_access.log by default and names rotated files with an underscore suffix, such as access.log_YYYYMMDD.HHMMSS. Before deploying Vector, edit edge-vector-deployment/vector.yaml and verify that the include setting of all five file sources covers both current and uncompressed rotated logs:

sources:
  nginx_http_access:
    include:
      - ${NGINX_LOG_DIR}/access.log
      - ${NGINX_LOG_DIR}/access.log_*

  nginx_stream_access:
    include:
      - ${NGINX_LOG_DIR}/stream_access.log
      - ${NGINX_LOG_DIR}/stream_access.log_*

  nginx_http_proxy_access:
    include:
      - ${NGINX_LOG_DIR}/http_proxy_access.log
      - ${NGINX_LOG_DIR}/http_proxy_access.log_*

  nginx_socks5_access:
    include:
      - ${NGINX_LOG_DIR}/socks5_access.log
      - ${NGINX_LOG_DIR}/socks5_access.log_*

  nginx_error:
    include:
      - ${NGINX_LOG_DIR}/*error*.log
      - ${NGINX_LOG_DIR}/*error*.log_*
      - ${NGINX_LOG_DIR}/**/*error*.log
      - ${NGINX_LOG_DIR}/**/*error*.log_*

If an application uses a custom log path, add that path and its rotated files to the matching source. Keep files for different log types separate.

Keep the rotated-log patterns in production. While Vector is running, it continues reading rotated files that are already open. If Vector is stopped or restarted across a rotation, patterns such as access.log_* let it use its checkpoints to resume collecting uncompressed older files. Matching only the current access.log is suitable only for real-time collection where a small amount of log loss is acceptable. It is not recommended for complete log retention.

Keep the *.gz exclusion in every source, and ensure that the collected logs do not use Gzip compression. This allows Vector to catch up on rotated files after a failure. For the default paths and rotation behavior, see Log file paths and Log rotation.

After generating test traffic on an Edge Node, verify that an HTTP access log entry is a single line of JSON:

sudo tail -n 1 /usr/local/oredge-node/logs/access.log

# Deploy ClickHouse

# Prepare the server certificate

Create certs in the ClickHouse deployment directory and place the certificate chain and private key in it:

edge-clickhouse-deployment/
└── certs/
    ├── server.crt
    └── server.key

server.crt must contain the full certificate chain. server.key must not be encrypted so that ClickHouse can start unattended. Restrict read access to the private key while ensuring that the ClickHouse process in the container can read it. Do not commit the certificate private key to version control.

# Enable ClickHouse HTTPS

The package configures edge-clickhouse-deployment/config.d/custom.xml as follows:

<clickhouse>
    <timezone>UTC</timezone>
    <max_connections>4096</max_connections>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <https_port>443</https_port>
    <tcp_port>9000</tcp_port>
    <openSSL>
        <server>
            <certificateFile>/etc/clickhouse-server/certs/server.crt</certificateFile>
            <privateKeyFile>/etc/clickhouse-server/certs/server.key</privateKeyFile>
            <loadDefaultCAFile>true</loadDefaultCAFile>
            <cacheSessions>true</cacheSessions>
            <disableProtocols>sslv2,sslv3,tlsv1,tlsv1_1</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
        </server>
    </openSSL>
</clickhouse>

The edge-clickhouse-deployment/docker-compose.yml file mounts the certificate but does not publish host ports by default. auto-deploy.sh generates docker-compose.ports.yml from the ports supplied by the user. For example, --https-port 443 produces a configuration equivalent to:

services:
  clickhouse:
    ports:
      - "${CLICKHOUSE_BIND_ADDRESS:-0.0.0.0}:443:443"

The left side of the mapping is the user-specified host port, and the right side is the fixed container port. ClickHouse always listens on HTTPS port 443 inside the container. With the option above, the host-to-container mapping is 443:443.

Do not remove the health check from the Compose file. It checks the service through HTTP port 8123 inside the container. clickhouse-init similarly uses Native TCP port 9000 within the Compose network for initialization. These ports are published to the host only when their corresponding deployment options are explicitly supplied.

# Configure the environment variables

Run the following commands on the ClickHouse server:

cd /opt/edge-clickhouse-deployment
sudo cp .env.example .env
sudo chmod 600 .env

Edit .env with sudoedit and change at least the following settings:

CLICKHOUSE_TAG=26.3.17.56
CLICKHOUSE_DB=app
CLICKHOUSE_USER=app
CLICKHOUSE_PASSWORD=replace-with-a-strong-admin-password
CLICKHOUSE_VECTOR_PASSWORD=replace-with-a-strong-writer-password
CLICKHOUSE_OREDGE_READER_PASSWORD=replace-with-a-strong-reader-password
CLICKHOUSE_RETENTION_DAYS=365
CLICKHOUSE_BIND_ADDRESS=0.0.0.0

Use a different strong password for each account. You can generate each one with openssl rand -hex 32. The .env file contains plaintext credentials. Allow only administrators to read it, and do not commit it to version control. Public ports are no longer configured in .env; specify them as command-line options the first time you run auto-deploy.sh.

# Run the automatic deployment script

Display the supported deployment options:

cd /opt/edge-clickhouse-deployment
sudo ./auto-deploy.sh --help

The following port options are available:

If you do not specify a port option, the script publishes no ports to the host. This guide requires only HTTPS and recommends mapping the standard host HTTPS port 443 to container port 443.

Deploy with Docker:

cd /opt/edge-clickhouse-deployment
sudo ./auto-deploy.sh --https-port 443

Deploy with Podman:

cd /opt/edge-clickhouse-deployment
sudo ./auto-deploy.sh --runtime podman --https-port 443

The remaining examples use Docker. Replace docker with podman when using Podman.

Specify the port options during the initial deployment. If ClickHouse containers already exist, the script exits without changing the containers or docker-compose.ports.yml.

The script creates the clickhouse_data and clickhouse_logs external volumes, starts ClickHouse, and uses the one-shot clickhouse-init container to perform the following tasks:

• Create the five log tables and configure data retention.
• Create the vector account, which can write only to the log tables.
• Create the oredge-reader account, which can query only the log tables.

Check the container status:

sudo docker compose \
    --project-name clickhouse-deployment \
    --file docker-compose.yml \
    --file docker-compose.ports.yml \
    ps --all

An Exited (0) status for clickhouse-init is expected. The ClickHouse service must be running and healthy. If initialization fails, inspect its logs:

sudo docker compose \
    --project-name clickhouse-deployment \
    --file docker-compose.yml \
    --file docker-compose.ports.yml \
    logs clickhouse-init

# Configure network access

Allow the TCP port specified by --https-port through the firewall or security group. This guide uses 443. Do not expose the plaintext ClickHouse HTTP or Native TCP ports to the public internet. If Edge Nodes use fixed egress addresses, restrict the HTTPS port to those source addresses where possible. When source restrictions are not possible, use random strong passwords and configure connection rate limiting and anomalous-access monitoring at the server perimeter.

Verify that clickhouse.example.com resolves to the ClickHouse server, then test HTTPS connectivity from every Edge Node. The command prompts for the vector account password:

curl --user vector --data-binary 'SELECT 1' \
    https://clickhouse.example.com/

A response of 1 confirms that DNS, TLS, the port, and the credentials work. For an internal organizational CA, specify the CA certificate with --cacert /path/to/organization-ca.crt. Do not use --insecure or -k to bypass certificate verification. If --https-port is not 443, specify the same port explicitly in the URL, for example https://clickhouse.example.com:8443/.

# Deploy Vector on an Edge Node

Perform the following steps on every Edge Node whose logs you want to collect.

# Configure Vector HTTPS

The edge-vector-deployment/vector.yaml file in the package enables certificate and hostname verification for all five ClickHouse sinks:

tls:
  verify_certificate: true
  verify_hostname: true

Run the automatic deployment script and pass the hostname covered by the server certificate:

cd /opt/edge-vector-deployment
sudo ./auto-deploy.sh --host clickhouse.example.com

The script uses HTTPS port 443 by default, prompts securely for the vector account password, and automatically performs the following tasks:

• Install the openresty-vector package if it is not already installed.
• Create an unprivileged vector system account.
• Install the Vector configuration, environment file, and vector-edge.service.
• Validate the configuration and start the service.

For a non-interactive deployment, put the password on the first line of a file that only root can read:

sudo ./auto-deploy.sh --host clickhouse.example.com \
    --password-file /root/vector-password

Add --force to replace an existing managed Vector configuration:

sudo ./auto-deploy.sh --force --host clickhouse.example.com

If you specified a non-443 port with --https-port when deploying ClickHouse, pass the same port through --port when deploying Vector. For example, if ClickHouse uses --https-port 8443, run:

sudo ./auto-deploy.sh --host clickhouse.example.com --port 8443

With the recommended configuration in this guide, Vector connects to https://clickhouse.example.com:443. Do not set --host to an IP address unless the server certificate Subject Alternative Name also contains that IP. When using an internal organizational CA, add its CA certificate to the Edge Node system trust store first. Do not bypass TLS errors by disabling verification.

The primary files and data directories are:

# Grant log read permissions

Use the Edge log group or ACLs to grant access. Do not run Vector as root. The following ACL example grants access to existing logs and causes files created after log rotation to inherit the same permissions:

sudo setfacl -m u:vector:x /usr/local/oredge-node
sudo setfacl -R -m u:vector:rX /usr/local/oredge-node/logs
sudo setfacl -d -m u:vector:rX /usr/local/oredge-node/logs

If setfacl is unavailable, install the ACL utilities or grant equivalent read and directory traversal permissions through the group that owns the Edge logs.

# Check the Vector service

Reload the systemd configuration and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now vector-edge.service

On its first start, Vector reads every matching uncompressed file from the beginning. On subsequent starts, it resumes from its checkpoints. If the node already contains a large volume of historical logs, estimate the initial import first or temporarily narrow the include patterns. The HTTP log sink has a 4 GiB disk buffer. The stream, HTTP forward proxy, SOCKS5, and error log sinks each have a 512 MiB buffer. Reserve sufficient space for /var/lib/vector-edge, and preserve this directory during restarts and upgrades.

Check the service status and follow its logs:

sudo systemctl status vector-edge.service
sudo journalctl -u vector-edge.service -f

After changing the configuration, reinstall the configuration file and restart the service. Do not delete the existing data directory:

cd /opt/edge-vector-deployment
sudo install -o root -g root -m 0644 vector.yaml \
    /etc/vector-edge/vector-edge.yaml
sudo systemctl restart vector-edge.service

# Check for Vector reporting errors

An active status for vector-edge.service means only that the process is running. It does not prove that logs are being written successfully to ClickHouse. Start by reviewing the service logs from the last 10 minutes:

sudo journalctl -u vector-edge.service --since "10 minutes ago" --no-pager

You can also filter for common reporting error keywords:

sudo journalctl -u vector-edge.service --since "10 minutes ago" --no-pager \
    | grep -Ei 'error|warn|failed|timeout|refused|tls|certificate|401|403|429|5[0-9]{2}|buffer'

Pay particular attention to the following messages:

The package enables a Vector API that listens only on 127.0.0.1:8686. Use vector top to observe throughput and error counts for the five ClickHouse sinks:

sudo /usr/local/openresty-vector/bin/vector top \
    --components 'clickhouse_*'

After generating test traffic, reporting has failed if ClickHouse sink error counts continue increasing, or input events continue increasing while sent events stop. A brief retry does not necessarily lose logs. As long as the disk buffer is not full, Vector resumes sending after ClickHouse recovers. Press Ctrl+C to exit vector top.

Finally, use the ClickHouse queries in the next section to verify that last_ingested_at continues to advance. The reporting path is healthy only when Vector sinks have no persistent errors, sent-event counts continue to increase, and the latest ClickHouse ingestion time remains current.

# Verify log collection

Generate several test requests through the configured HTTP, stream/SNI, HTTP forward proxy, or SOCKS5 application, then open the read-only client on the ClickHouse server:

cd /opt/edge-clickhouse-deployment
sudo docker compose \
    --project-name clickhouse-deployment \
    --file docker-compose.yml \
    --file docker-compose.ports.yml \
    exec clickhouse clickhouse-client \
    --user oredge-reader --password --database app

Enter CLICKHOUSE_OREDGE_READER_PASSWORD from .env, then run the following queries.

Check the row count and last ingestion time for each log type:

SELECT 'http' AS type, count() AS rows, max(ingested_at) AS last_ingested_at
FROM nginx_http_access_logs
UNION ALL
SELECT 'stream', count(), max(ingested_at)
FROM nginx_stream_access_logs
UNION ALL
SELECT 'http_proxy', count(), max(ingested_at)
FROM nginx_http_proxy_access_logs
UNION ALL
SELECT 'socks5', count(), max(ingested_at)
FROM nginx_socks5_access_logs
UNION ALL
SELECT 'error', count(), max(ingested_at)
FROM nginx_error_logs;

View the latest HTTP requests:

SELECT
    event_time,
    app_id,
    nginx_host,
    remote_addr,
    http_host,
    request,
    status,
    request_time,
    upstream_addr,
    upstream_status
FROM nginx_http_access_logs
ORDER BY event_time DESC
LIMIT 20;

View stream/SNI, HTTP forward proxy, and SOCKS5 logs for an application from the last hour. Replace 123, 456, and 789 with the actual application IDs for the corresponding types:

SELECT event_time, app_id, remote_addr, protocol, status,
       bytes_sent, bytes_received, upstream_addr
FROM nginx_stream_access_logs
WHERE app_id = 123 AND event_time >= now() - INTERVAL 1 HOUR
ORDER BY event_time DESC
LIMIT 20;

SELECT event_time, app_id, client_addr, method, status, url,
       up_bytes, down_bytes, failure
FROM nginx_http_proxy_access_logs
WHERE app_id = 456 AND event_time >= now() - INTERVAL 1 HOUR
ORDER BY event_time DESC
LIMIT 20;

SELECT event_time, app_id, client_addr, username, status,
       up_bytes, down_bytes, request_time, failure
FROM nginx_socks5_access_logs
WHERE app_id = 789 AND event_time >= now() - INTERVAL 1 HOUR
ORDER BY event_time DESC
LIMIT 20;

Check parsing quality using the HTTP table below. The same checks also work with the other access log tables:

SELECT
    nginx_host,
    parsed_ok,
    count() AS rows,
    max(ingested_at) AS last_ingested_at
FROM nginx_http_access_logs
GROUP BY nginx_host, parsed_ok
ORDER BY nginx_host, parsed_ok;

If any records have parsed_ok = 0, inspect the original logs that could not be parsed:

SELECT ingested_at, source_file, raw_message
FROM nginx_http_access_logs
WHERE parsed_ok = 0
ORDER BY ingested_at DESC
LIMIT 20;

# Capacity planning

Log length, field contents, and repetition all affect ClickHouse compression. For example, long URLs, User-Agent values, upstream addresses, and error messages increase the storage used by each log entry. Base your estimate on compressed data from representative production logs rather than only on raw log file sizes.

# Capacity formulas

Calculate capacity separately for HTTP, stream, HTTP forward proxy, SOCKS5, and optional error logs, then add the results:

Base data capacity = logs per day × retention days × average compressed bytes per log
Planned data disk capacity = base data capacity × 2 × (1 + expected growth rate)

The recommended operational safety factor of 2 reserves space for background merges, delayed TTL deletion, data-volume fluctuations, and at least 30% free space. This factor does not include backups or replicas:

• For one full backup on the same server, add at least one more base data capacity. Store backups on a separate disk or in object storage where possible.
• For multiple ClickHouse replicas, multiply total cluster storage by the number of replicas.
• If traffic is expected to grow by 30% within one year, use 0.3 as the expected growth rate in the formula.

# Capacity example for one million logs

Without measured data, you can initially estimate 500 compressed bytes per log. One million logs then occupy approximately 0.5 GB of active ClickHouse data. After applying the operational safety factor, plan approximately 1 GB of data disk capacity for those logs.

The following estimates show how log contents affect capacity. The table uses decimal units, where 1 GB = 1,000,000,000 bytes:

These values represent only the space required by the new log data. They are not minimum disk specifications for the ClickHouse server. Reserve additional space for the operating system, container images, ClickHouse service logs, and other operational files.

# Capacity examples for one-year retention

The following examples assume that the daily total already includes all four access log types and optional error logs. They use an average compressed size of 500 bytes per log, a retention period of 365 days, and an operational safety factor of 2. Additional growth, backups, and replicas are not included:

For example, if a customer generates one million logs per day and retains them for one year, start with a 365 GB data disk plan. If log volume is expected to grow by 30% within the year, adjust the plan to:

365 GB × 1.3 = 474.5 GB

Round up when purchasing storage, and use SSDs that can sustain continuous writes, background merges, and the expected query workload.

# Measure the actual size per log

Import at least one million representative production logs and wait for ClickHouse to complete the major background merges before querying system.parts. bytes_on_disk includes compressed columns, indexes, and metadata for active data parts, making it suitable for estimating actual disk usage:

SELECT
    table,
    sum(rows) AS rows,
    formatReadableSize(sum(bytes_on_disk)) AS size_on_disk,
    round(sum(bytes_on_disk) / nullIf(sum(rows), 0), 2) AS bytes_per_row
FROM system.parts
WHERE database = 'app'
  AND active
  AND table IN (
      'nginx_http_access_logs',
      'nginx_stream_access_logs',
      'nginx_http_proxy_access_logs',
      'nginx_socks5_access_logs',
      'nginx_error_logs'
  )
GROUP BY table
ORDER BY table;

Calculate capacity from each table’s bytes_per_row value for greater accuracy. For example, if you do not collect error logs, exclude nginx_error_logs from the calculation.

Count each type of log from the last 24 hours:

SELECT 'http' AS type, count() AS rows
FROM nginx_http_access_logs
WHERE ingested_at >= now() - INTERVAL 1 DAY
UNION ALL
SELECT 'stream', count()
FROM nginx_stream_access_logs
WHERE ingested_at >= now() - INTERVAL 1 DAY
UNION ALL
SELECT 'http_proxy', count()
FROM nginx_http_proxy_access_logs
WHERE ingested_at >= now() - INTERVAL 1 DAY
UNION ALL
SELECT 'socks5', count()
FROM nginx_socks5_access_logs
WHERE ingested_at >= now() - INTERVAL 1 DAY
UNION ALL
SELECT 'error', count()
FROM nginx_error_logs
WHERE ingested_at >= now() - INTERVAL 1 DAY;

Do not measure only a low-traffic day. Measure peak business days and account for promotions, attack traffic, and future growth when determining the daily log count. Re-measure regularly after the system has been running and revise the capacity forecast.

# Check disk space

Use the following query to view the total and remaining ClickHouse disk space:

SELECT
    name,
    path,
    formatReadableSize(total_space) AS total_space,
    formatReadableSize(free_space) AS free_space,
    round(free_space / total_space * 100, 2) AS free_percent
FROM system.disks;

Alert and expand capacity before free space drops below 30%. This prevents background merges or TTL cleanup from failing because of insufficient space. Monitor the clickhouse_logs volume and backup storage separately; they are not included in table capacity reported by system.parts.bytes_on_disk.

Every Edge Node also needs capacity for Vector disk buffers. When all five log types are collected, the configured buffers have a combined limit of 6 GiB. Reserve at least 8 GiB for /var/lib/vector-edge. If error log collection is disabled, you can subtract its 512 MiB buffer, but still account for a backlog while ClickHouse is temporarily unavailable.

# Upgrade an existing deployment

For an existing three-table deployment, upgrade the ClickHouse schema before enabling the new Vector pipelines:

1. Obtain matching packages that support stream app_id, HTTP forward proxy, and SOCKS5 logs, and update the deployment files on the respective hosts. Preserve the existing .env, certificates, port configuration, data volumes, and each node’s Vector environment file, checkpoints, and disk buffers.

2. On the ClickHouse server, rerun the initialization service with the existing account passwords and retention settings:

   cd /opt/edge-clickhouse-deployment
   sudo docker compose \
       --project-name clickhouse-deployment \
       --file docker-compose.yml \
       --file docker-compose.ports.yml \
       run --rm clickhouse-init
   

   The initializer adds an app_id UInt64 column to the existing stream table, creates both proxy log tables, and configures TTL and permissions for vector and oredge-reader. It is safe to repeat. Old stream records read as app_id = 0. The ClickHouse auto-deploy.sh exits when containers already exist, so it cannot replace this initialization step.

3. Upgrade Edge Admin and Edge Node, then update the log formats as described above. Republish HTTP forward proxy and SOCKS5 application configurations so that the nodes receive their application IDs.

4. Redeploy the updated Vector configuration on every Edge Node with the existing hostname, port, and CA settings. Use --force as described in the deployment steps when replacing managed configuration. Preserve /var/lib/vector-edge and verify that custom log paths remain included in their corresponding sources.

5. Generate test traffic for each enabled access log type. Query its table and confirm that new records contain the expected app_id and that last_ingested_at continues to advance.

# Change the log retention period

CLICKHOUSE_RETENTION_DAYS must be a positive integer. After changing .env on the ClickHouse server, run the initialization service again:

cd /opt/edge-clickhouse-deployment
sudo docker compose \
    --project-name clickhouse-deployment \
    --file docker-compose.yml \
    --file docker-compose.ports.yml \
    run --rm clickhouse-init

This operation updates the TTL for both new and existing log tables. ClickHouse deletes expired data during background merges, so the change does not release all space immediately.

# Troubleshooting

# Vector service does not start

View the latest errors:

sudo journalctl -u vector-edge.service -n 100 --no-pager

Check the ClickHouse hostname, HTTPS port, certificate chain, vector password, system time, and permissions on /etc/vector-edge/vector-edge.env. The environment file must be owned by root and have mode 0600.

If the logs contain certificate verify failed or unknown issuer, verify that the server sends the full certificate chain and that the Edge Node trusts the issuing CA. Do not disable verify_certificate or verify_hostname.

# Vector reports Permission denied

Verify that the vector user can traverse /usr/local/oredge-node, read the log directory and existing logs, and inherit an appropriate default ACL for files created after rotation.

# ClickHouse contains no access logs

Check the following in order:

1. HTTP applications use the configured main or default vector-json format. Stream/SNI and proxy applications use their corresponding JSON formats, with logging enabled and the configuration published.
2. The current log file on the Edge Node contains new entries, with one JSON record per line.
3. Filenames in edge-vector-deployment/vector.yaml match the actual filenames on the Edge Node.
4. Vector logs contain no connection, authentication, parsing, or buffer errors.
5. The Edge Node can access the ClickHouse hostname and port through HTTPS and certificate verification succeeds.

# Only current logs are collected, not rotated logs

Verify that each Vector include setting covers both the current log and rotated logs with underscore suffixes, and that rotated files are not Gzip compressed. Vector uses checkpoints in /var/lib/vector-edge to avoid reading the same data again after a normal restart.

# Duplicate records appear after redeployment

Do not delete /var/lib/vector-edge. Doing so removes Vector’s read checkpoints and can cause it to import historical logs again. The ClickHouse tables use ReplacingMergeTree; records with the same sorting key are deduplicated only during background merges. This is not a replacement for Vector checkpoints.

# Recommendations for ongoing use

• Use oredge-reader, or a separate read-only account for each analytics tool, for routine queries. Do not use the vector writer account.
• Use the clickhouse-client included in the container for routine operations. Use DBeaver for graphical browsing, or the Grafana ClickHouse data source for continuous monitoring and alerting.
• Add a time range to historical log queries to avoid scanning every partition unintentionally.
• Back up the data before changing the ClickHouse image version, and preserve the clickhouse_data external volume.
• Monitor the server certificate expiration date. After renewing and replacing the certificate files, restart the ClickHouse container and test HTTPS connectivity again.

For more TLS settings, see Configure ClickHouse TLS and the Vector ClickHouse sink configuration.