OpenResty XRay™ On-Premise Deployment on Azure Kubernetes Service

# Name

Installation and deployment guide for On-Premise (Azure Kubernetes Service) version of OpenResty XRay.

# Table of Content

• Overview
• Installation
  • Console Server
  • Agent
• Basic Ops

# Overview

OpenResty XRay (XRay) On-Premises edition consist of two parts: Console Server and Agent.

The Console Server runs in docker container, managed by kubernetes, and the Agent runs on the target machine.

XRay Console Server consist of two parts:

• OpenResty XRay Console:
  • xray-console: web service of console, provide a web UI based dashboard.
  • xray-console-pg: postgresql database for xray-console.
• OpenResty XRay BuildBox:
  • build-box-[dist]: the backend service for building xray tools, where dist represents the corresponding Linux distro.
  • xray-tenant-db: provide query API for xray-console and build-box-[dist], will query first local tenant-db service and then remote pkg-db The service will only request queries to our remote public pkg-db, and no data will be uploaded to our public pkg-db
  • xray-tenant-db-pl: provide API for private package info.
  • xray-tenang-db-pg: postgresql database for xray-tenant-db-pl.

In the following deployment sample, we will use Azure Disk as the persistence volume.

# Installation

# Console Server

# 1. Hardware requirements

• A basic Kubernetes platform
• 1Ti + 500Gi hard disk for the xray-console pod
• 100Gi hard disk for the build-box container of the tenant-db pod
• 2 x 10Gi hard disk for the tenant-db container of the tenant-db pod
• Do not use NFS for storage, there will be permission issues
• At least 256MB of free memory for each container
• console pod can connect to the Internet

Note: Based on our experience, the data sampled by a single agent node is about 10G for 30 days.

# 2. access private registry

First, create the namespace and the secret for kubernetes to access the openresty image repository using the username and password provided in the email:

# create namespace here
kubectl create ns openresty-xray

# create secret
kubectl create secret docker-registry openresty-xray \
  --docker-server=registry.openresty.com \
  --docker-username=<docker-registry username> \
  --docker-password=<docker-registry password> \
  -n openresty-xray

# 3. create secret for certificate

If you need to use a custom certificate, you need to perform this step, or the Console Server will use the built-in certificate (please ignore browser warnings when accessing the dashboard)

Copy XRay server site certificate files into cert/ssl.crt and cert/ssl.key, and then:

kubectl create secret generic cert-secret --from-file=./cert/ -n openresty-xray

Find the following lines in aks-kubernetes.yml and remove the comments to use the custom certificate:

#- mountPath: /usr/local/orxray-console/conf/cert
#  name: cert
#  readOnly: true

# ...
# ...

volumes:
#- name: cert
#  secret:
#    secretName: cert-secret

PS: the environment variable in kubernets.yml should be consistent with the domain name in certificate file:

        - name: CONSOLE_HOST
          value: xxx.xx.com

The usage of these certificate files are:

• Access XRay Console Dashboard through https protocol.
• Security communication between Agent and Console Server.

# 4. Use Azure Disk as a persistence volume

Deploying a console server side requires persistent data volumes, here is a persistent data volume solution using Azure Disk.

4.1 create Azure Disk

For more information: https://learn.microsoft.com/en-us/azure/aks/azure-disk-volume https://docs.microsoft.com/en-US/cli/azure/disk#az_disk_create

AKS_NODE_RESOURCE_GROUP=$(az aks show \
    --resource-group $AKS_CLUSTER_RESOURCE_GROUP \
    --name $AKS_NAME --query nodeResourceGroup -o tsv)

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_console_disk \
  --size-gb $ORXRAY_CONSOLE_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_console_pg_disk \
  --size-gb $ORXRAY_CONSOLE_PG_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_buildbox_disk \
  --size-gb $ORXRAY_BUILDBOX_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_tenant_db_disk \
  --size-gb $ORXRAY_TENANT_DB_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_tenant_db_pg_disk \
  --size-gb $ORXRAY_TENANT_DB_PG_DISK_SIZE \
  --query id --output tsv

• $AKS_CLUSTER_RESOURCE_GROUP is resource group of the kubernetes cluster
• $AKS_NAME is name of the AKS
• $ORXRAY_CONSOLE_DISK_SIZE is the disk to be used for console service, recommended size is 500GB and above
• $ORXRAY_CONSOLE_PG_DISK_SIZE is the disk to be used for console-pg service, recommended size is 1024GB and above
• $ORXRAY_BUILDBOX_DISK_SIZE is the disk to be used for build-box service, recommended size is 100GB and above
• $ORXRAY_TENANT_DB_DISK_SIZE is the disk to be used for tenant-db service, recommended size is 50GB and above
• $ORXRAY_TENANT_DB_PG_DISK_SIZE is the disk to be used for tenant-db-pg service, recommended size is 50GB and above

4.2 use the aks-kubernetes-pv.yml.sample file from email

The SUBSCRIPTION_ID and AKS_NODE_RESOURCE_GROUP need to be replaced with the currently used values. You can also modify the size of the corresponding persistent volume based on the size of the disk when it was created.

kubectl apply -f aks-kubernetes-pv.yml.sample

# 5. deployment and startup

use the aks-kubernetes.yml file from email

kubectl apply -f aks-kubernetes.yml

# 6. access the Console Server

Use ingress to expose the https service on port 443 of the xray-console to access the service.

# Agent

Agent side runs on the target machine, after installation, the agent side will communicate with the Console Server deployed in previous steps, the agent’s installation itself is also cooperated with server side:

Access your dashboard in your browser, login with the initial username and password from the email (you may change the password after first login). You’ll be greeted with installation guide after login, follow the installation guide to install the agent on the target machine.

# Basic Ops

# Check Console Server status

$ kubectl get pods -n openresty-xray

NAME                                  READY   STATUS    RESTARTS   AGE
build-box-centos-7-75c954d87d-h6fkz   1/1     Running   0          18h
xray-console-5d457bb49d-q49f8         1/1     Running   0          17h
xray-console-pg-64df4cbd86-tm44n      1/1     Running   0          18h
xray-tenant-db-69d9cbd77d-nccnb       1/1     Running   0          18h
xray-tenant-db-pg-845d5cb674-rrjz8    1/1     Running   0          18h
xray-tenant-db-pl-859c5c8dfd-xk7sg    1/1     Running   0          18h

# View Console Server logs

$ kubectl logs --tail=10 -n openresty-xray -l app=xray-console -c xray-console

# Component Upgrade

Stop the old pods first, then enable the new pods, and wait for the new pods to start successfully, then the upgrade will be successful! The aks-kubernetes-pods.yml file is provided in the upgrade email attachment.

$ kubectl delete -f aks-kubernetes-pods.yml
$ kubectl apply -f aks-kubernetes-pods.yml

There is no need to delete the old PVCs or the corresponding PVs. Because the PV is RWO (Read-Write-Once) attributed, if the PVC is deleted, the PV will also need to be deleted and then re-built (this does not affect data that has already been persisted).