OpenResty XRay™ On-Premise Deployment on Azure Kubernetes Service

# Name

Installation and deployment guide for On-Premise (Azure Kubernetes Service) version of OpenResty XRay.

# Table of Content

• Overview
• Installation
  • Console Server
  • Agent
• Basic Ops

# Overview

OpenResty XRay (XRay) On-Premises edition consist of two parts: Console Server and Agent.

The Console Server runs in docker container, managed by kubernetes, and the Agent runs on the target machine.

XRay Console Server consist of two parts:

• OpenResty XRay Console:
  • xray-console: web service of console, provide a web UI based dashboard.
  • xray-console-pg: postgresql database for xray-console.
• OpenResty XRay BuildBox:
  • build-box-[dist]: the backend service for building xray tools, where dist represents the corresponding Linux distro.
  • xray-tenant-db: provide query API for xray-console and build-box-[dist], will query first local tenant-db service and then remote pkg-db The service will only request queries to our remote public pkg-db, and no data will be uploaded to our public pkg-db
  • xray-tenant-db-pl: provide API for private package info.
  • xray-tenang-db-pg: postgresql database for xray-tenant-db-pl.

In the following deployment sample, we will use Azure Disk as the persistence volume.

# Installation

# Console Server

# 1. access private registry

First, create the namespace and the secret for kubernetes to access the openresty image repository using the username and password provided in the email:

# create namespace here
kubectl create ns openresty-xray

# create secret
kubectl create secret docker-registry openresty-xray \
  --docker-server=registry.openresty.com \
  --docker-username=<docker-registry username> \
  --docker-password=<docker-registry password> \
  -n openresty-xray

# 2. create secret for certificate

If you need to use a custom certificate, you need to perform this step, or the Console Server will use the built-in certificate (please ignore browser warnings when accessing the dashboard)

Copy XRay server site certificate files into cert/ssl.crt and cert/ssl.key, and then:

kubectl create secret generic cert-secret --from-file=./cert/ -n openresty-xray

Find the following lines in aks-kubernetes.yml and remove the comments to use the custom certificate:

#- mountPath: /usr/local/orxray-console/conf/cert
#  name: cert
#  readOnly: true

# ...
# ...

volumes:
#- name: cert
#  secret:
#    secretName: cert-secret

PS: the environment variable in kubernets.yml should be consistent with the domain name in certificate file:

        - name: CONSOLE_HOST
          value: xxx.xx.com

The usage of these certificate files are:

• Access XRay Console Dashboard through https protocol.
• Security communication between Agent and Console Server.

# 3. Use Azure Disk as a persistence volume

Deploying a console server side requires persistent data volumes, here is a persistent data volume solution using Azure Disk.

3.1 create Azure Disk

For more information: https://learn.microsoft.com/en-us/azure/aks/azure-disk-volume https://docs.microsoft.com/en-US/cli/azure/disk#az_disk_create

AKS_NODE_RESOURCE_GROUP=$(az aks show \
    --resource-group $AKS_CLUSTER_RESOURCE_GROUP \
    --name $AKS_NAME --query nodeResourceGroup -o tsv)

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_console_disk \
  --size-gb $ORXRAY_CONSOLE_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_console_pg_disk \
  --size-gb $ORXRAY_CONSOLE_PG_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_buildbox_disk \
  --size-gb $ORXRAY_BUILDBOX_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_tenant_db_disk \
  --size-gb $ORXRAY_TENANT_DB_DISK_SIZE \
  --query id --output tsv

az disk create \
  --resource-group $AKS_NODE_RESOURCE_GROUP \
  --name orxray_tenant_db_pg_disk \
  --size-gb $ORXRAY_TENANT_DB_PG_DISK_SIZE \
  --query id --output tsv

• $AKS_CLUSTER_RESOURCE_GROUP is resource group of the kubernetes cluster
• $AKS_NAME is name of the AKS
• $ORXRAY_CONSOLE_DISK_SIZE is the disk to be used for console service, recommended size is 500GB and above
• $ORXRAY_CONSOLE_PG_DISK_SIZE is the disk to be used for console-pg service, recommended size is 1024GB and above
• $ORXRAY_BUILDBOX_DISK_SIZE is the disk to be used for build-box service, recommended size is 100GB and above
• $ORXRAY_TENANT_DB_DISK_SIZE is the disk to be used for tenant-db service, recommended size is 50GB and above
• $ORXRAY_TENANT_DB_PG_DISK_SIZE is the disk to be used for tenant-db-pg service, recommended size is 50GB and above

3.2 use the aks-kubernetes-pv.yml.sample file from email

The SUBSCRIPTION_ID and AKS_NODE_RESOURCE_GROUP need to be replaced with the currently used values. You can also modify the size of the corresponding persistent volume based on the size of the disk when it was created.

kubectl apply -f aks-kubernetes-pv.yml.sample

# 4. deployment and startup

use the aks-kubernetes.yml file from email

kubectl apply -f aks-kubernetes.yml

# 5. access the Console Server

Use ingress to expose the https service on port 443 of the xray-console to access the service.

# Agent

Agent side runs on the target machine, after installation, the agent side will communicate with the Console Server deployed in previous steps, the agent’s installation itself is also cooperated with server side:

Access your dashboard in your browser, login with the initial username and password from the email (you may change the password after first login). You’ll be greeted with installation guide after login, follow the installation guide to install the agent on the target machine.

# Basic Ops

# Check Console Server status

$ kubectl get pods -n openresty-xray 

NAME                                  READY   STATUS    RESTARTS   AGE
build-box-centos-7-75c954d87d-h6fkz   1/1     Running   0          18h
xray-console-5d457bb49d-q49f8         1/1     Running   0          17h
xray-console-pg-64df4cbd86-tm44n      1/1     Running   0          18h
xray-tenant-db-69d9cbd77d-nccnb       1/1     Running   0          18h
xray-tenant-db-pg-845d5cb674-rrjz8    1/1     Running   0          18h
xray-tenant-db-pl-859c5c8dfd-xk7sg    1/1     Running   0          18h

# View Console Server logs

$ kubectl log --tail=10 xray-console-5d457bb49d-q49f8 -n openresty-xray 

# Error: ’tenant does not exist'

If you access the console with a browser and it says “This tenant does not exist”, check if the access domain or IP matches the value of the CONSOLE_HOST variable in aks-kubernetes.yml.

ex: If the access address is http://xray.example.com and the value of CONSOLE_HOST is not xray.example.com, you will be prompted with this message.