Azure Gateway Deployment¶

Deploying MazeVault Gateway in Azure for Hybrid On-Premise + Cloud Architecture

Document Version: 1.0.0
Last Updated: 2026-03-19
License Tier: Enterprise

1. Architecture Overview¶

MazeVault Gateway extends your on-premise MazeVault installation into Azure, providing secure secret management and certificate lifecycle automation for cloud workloads. The gateway communicates with your on-premise Primary backend via encrypted mTLS + E2EE channel.

flowchart TB
    subgraph OnPremise["🏢 On-Premise (Primary)"]
        FE["🖥️ MazeVault Frontend"]
        BE["⚙️ MazeVault Backend<br/>(Orchestrator Mode)"]
        DB_OP["🗄️ PostgreSQL"]
        RD_OP["⚡ Redis"]
    end

    subgraph Azure["☁️ Azure Cloud"]
        subgraph GWC["🌍 Primary Region (e.g. Germany West Central)"]
            subgraph AKS_GWC["AKS Cluster"]
                GW["🌐 MazeVault Gateway<br/>(1 replica)"]
                AGENT["🤖 MazeVault Agent<br/>(DaemonSet)"]
            end
            PG["🗃️ Azure Database for<br/>PostgreSQL Flexible Server"]
            Redis["⚡ Azure Cache<br/>for Redis"]
            KV_GW["🔐 Key Vault<br/>(Gateway Secrets)"]
            KV_PRJ["🔑 Key Vault<br/>(Project Secrets)"]
            MI["🆔 Managed Identity"]
        end

        subgraph WE["🛡️ DR Region (e.g. West Europe) — Cold Standby"]
            AKS_DR["⏸️ AKS — Stopped"]
            PG_DR["⏸️ PostgreSQL — Stopped"]
            KV_DR["✅ Key Vault — Active"]
        end

        EA["🆔 Entra ID<br/>App Registration"]
        ACR["📦 Container Registry"]
    end

    FE --> BE
    GW -->|"mTLS + E2EE<br/>Port 8443"| BE
    GW --> PG
    GW --> Redis
    GW --> KV_GW
    GW --> KV_PRJ
    MI -.-> KV_GW
    MI -.-> KV_PRJ
    AGENT -.-> GW
    AKS_GWC -.->|Pull| ACR
    KV_GW -.->|Geo-replication| KV_DR

    classDef onprem fill:#E8EAF6,stroke:#3F51B5,stroke-width:2px,color:#283593
    classDef azure fill:#EBF5FB,stroke:#2196F3,stroke-width:2px,color:#1565C0
    classDef data fill:#E8F5E9,stroke:#4CAF50,stroke-width:2px,color:#2E7D32
    classDef security fill:#FFF8E1,stroke:#FF9800,stroke-width:2px,color:#E65100
    classDef dr fill:#F5F5F5,stroke:#9E9E9E,stroke-width:2px,color:#424242
    classDef identity fill:#FCE4EC,stroke:#E91E63,stroke-width:2px,color:#880E4F

    class FE,BE,DB_OP,RD_OP onprem
    class GW,AGENT,PG,Redis azure
    class KV_GW,KV_PRJ security
    class AKS_DR,PG_DR,KV_DR dr
    class MI,EA,ACR identity

Key Components¶

Component	Purpose
MazeVault Gateway	Backend running in `gateway` mode — syncs with on-premise Primary
Azure Key Vault (Gateway)	Stores operational secrets (master key, JWT, DB credentials)
Azure Key Vault (Project)	Stores project secrets managed by MazeVault (Orchestrator Mode)
Managed Identity	System-level access to Key Vaults (read gateway secrets, read/write project secrets)
Entra ID App Registration	User-level access via OBO flow — each Key Vault has its own RBAC
MazeVault Agent	Distributes secrets/certificates to application servers

Agent Binaries¶

Agents connected to the Gateway download installers and updates from the Gateway. The Gateway serves the agent version bundled in its image and fetches any other version from the Primary over its existing connection, so agent versions do not depend on the Gateway version. The Gateway checks every binary's signature itself. Versions the Primary does not bundle need the Primary's license and access to the MazeVault registry; the Gateway itself needs neither.

2. Prerequisites¶

2.1 Azure Requirements¶

[ ] Azure Subscription with Contributor + User Access Administrator roles
[ ] Azure CLI (az) 2.50+ installed
[ ] kubectl installed
[ ] Helm 3.12+ installed
[ ] Terraform 1.5+ installed

2.2 On-Premise Requirements¶

[ ] MazeVault Primary backend running in Orchestrator Mode
[ ] Primary backend accessible from Azure (VPN/ExpressRoute or public endpoint with IP whitelist)
[ ] Admin access to MazeVault Primary (for generating bootstrap token)

2.3 Entra ID Requirements¶

[ ] Global Administrator or Application Administrator role in Entra ID
[ ] Ability to grant admin consent for API permissions

3. Network Requirements¶

3.1 Network Topology¶

flowchart LR
    subgraph OnPrem["🏢 On-Premise"]
        FW["🛡️ Firewall"]
        PRIMARY["⚙️ Primary Backend<br/>:8443"]
    end

    subgraph Azure["☁️ Azure"]
        NSG["🛡️ NSG"]
        AKS["🌐 AKS Gateway"]
    end

    subgraph Entra["🆔 Microsoft Services"]
        LOGIN["login.microsoftonline.com"]
        GRAPH["graph.microsoft.com"]
        MGMT["management.azure.com"]
        KVEP["*.vault.azure.net"]
    end

    AKS -->|"HTTPS :8443"| FW --> PRIMARY
    AKS -->|"HTTPS :443"| LOGIN
    AKS -->|"HTTPS :443"| GRAPH
    AKS -->|"HTTPS :443"| MGMT
    AKS -->|"HTTPS :443"| KVEP

    classDef onprem fill:#E8EAF6,stroke:#3F51B5,stroke-width:2px,color:#283593
    classDef azure fill:#EBF5FB,stroke:#2196F3,stroke-width:2px,color:#1565C0
    classDef entra fill:#FFF8E1,stroke:#FF9800,stroke-width:2px,color:#E65100

    class FW,PRIMARY onprem
    class NSG,AKS azure
    class LOGIN,GRAPH,MGMT,KVEP entra

3.2 Firewall Rules — On-Premise¶

Direction	Source	Destination	Port	Protocol	Purpose
Inbound	Azure AKS Subnet	Primary Backend	TCP 8443	HTTPS/mTLS	Gateway heartbeat, registration, CSR signing, write queue reconciliation

3.3 NSG Rules — Azure (Outbound from AKS)¶

Direction	Source	Destination	Port	Purpose
Outbound	AKS Subnet	On-Premise Primary IP	TCP 8443	Gateway → Primary communication
Outbound	AKS Subnet	`login.microsoftonline.com`	TCP 443	Entra ID SSO login
Outbound	AKS Subnet	`graph.microsoft.com`	TCP 443	Graph API (groups, RBAC)
Outbound	AKS Subnet	`management.azure.com`	TCP 443	Azure Management API (KV RBAC discovery)
Outbound	AKS Subnet	`*.vault.azure.net`	TCP 443	Key Vault access
Outbound	AKS Subnet	PostgreSQL Private Endpoint	TCP 5432	Database
Outbound	AKS Subnet	Redis Private Endpoint	TCP 6380	Cache (TLS)

4. Entra ID App Registration¶

Critical Step

This step must be completed before deploying the gateway. Without proper Entra ID configuration, SSO login and Key Vault OBO access will not work.

4.1 Create App Registration¶

Navigate to Azure Portal → Entra ID → App registrations → New registration
Fill in:

Field	Value
Name	`APP_{PROJECT}_{ZONE}_{ENV}_01` (e.g., `APP_MZV_INT_PRO_01`)
Supported account types	Single tenant (this organization directory only)
Redirect URI (Web)	`https://{your-api-domain}/api/v1/auth/sso/entra/callback`

4.2 Configure API Permissions¶

Navigate to API permissions → Add a permission:

API	Permission	Type	Purpose
Microsoft Graph	`openid`	Delegated	SSO login
Microsoft Graph	`profile`	Delegated	User profile info
Microsoft Graph	`email`	Delegated	User email
Microsoft Graph	`User.Read`	Delegated	Read user profile
Microsoft Graph	`GroupMember.Read.All`	Delegated	Read groups for RBAC mapping
Azure Key Vault	`user_impersonation`	Delegated	OBO access to Key Vault on behalf of user
Azure Service Management	`user_impersonation`	Delegated	KV RBAC discovery via Management API

Grant Admin Consent

After adding all permissions, click Grant admin consent for {tenant}. Without this, the OBO flow will fail with AADSTS65001.

4.3 Create Client Secret¶

Navigate to Certificates & secrets → New client secret
Description: MazeVault Gateway Secret
Expires: 24 months (recommended)
Copy the Value immediately — it will not be shown again

4.4 Configure App Roles (Recommended for RBAC Mapping)¶

App Roles are the recommended mechanism for mapping Entra ID users to MazeVault roles. When assigned, they appear automatically as the roles claim in the ID token.

Navigate to App registrations → your SSO app → App roles → Create app role
Set Allowed member types to Users/Groups
Set Value to the identifier that will be used in MazeVault mappings (e.g. APP_MZV_PRO_ADMIN)
Navigate to Enterprise Applications → find the corresponding Enterprise App → Users and groups → Add user/group → Assign users to the created App Roles

In MazeVault, create mappings under Access Control → Group/App Role Mappings where the External ID matches the App Role value.

No optional claim needed

App Role values appear in the roles claim automatically. No optional claim configuration is needed for this mechanism.

4.5 Configure Token Claims (Optional)¶

Navigate to Token configuration → Add optional claim:

Claim	Token Type	Purpose
`email`	ID Token	User email address (recommended)
`groups`	ID Token	Security Group membership for RBAC mapping (only needed if using Security Groups instead of App Roles)

Do NOT enable emit_as_roles

When adding the groups claim, do NOT check the emit_as_roles option. It converts group Object IDs into the roles claim, conflicting with App Role values and breaking RBAC mapping.

4.6 Record Configuration Values¶

You will need these values for Terraform deployment:

Value	Where to Find	Environment Variable
Tenant ID	Entra ID → Overview	`ENTRA_TENANT_ID`
Client ID	App Registration → Overview → Application (client) ID	`ENTRA_CLIENT_ID`
Client Secret	App Registration → Certificates & secrets → Value	`ENTRA_CLIENT_SECRET`
Redirect URI	App Registration → Authentication → Redirect URIs	`ENTRA_REDIRECT_URI`

5. Resource Naming Convention¶

MazeVault uses a configurable naming convention for Azure resources. The default pattern is:

5.1 Production Environments (PRO, PRE, DRN, MIG, EOY, HOT)¶

{company}{environment}{zone}{project}{region}{type}{number}

5.2 Non-Production Environments (DEV, ITG, TST)¶

{company}npr{environment}{zone}{project}{region}{type}{number}

5.3 Examples¶

Resource	Pattern	Example
Key Vault	`{co}{env}{zone}{proj}{region}kv{nn}`	`crdprointmzvgwckv01`
PostgreSQL	`{co}-{env}-{zone}-{proj}-{region}-psql{nn}`	`crd-pro-int-mzv-gwc-psql01`
Redis	`{co}-{env}-{zone}-{proj}-{region}-redis{nn}`	`crd-pro-int-mzv-gwc-redis01`
AKS	`{co}-{env}-{zone}-{proj}-{region}-aks{nn}`	`crd-pro-int-mzv-gwc-aks01`
Enterprise App	`APP_{PROJ}_{ZONE}_{ENV}_{nn}`	`APP_MZV_INT_PRO_01`
Resource Group	`rg-{co}-{env}-{zone}-{proj}-{region}`	`rg-crd-pro-int-mzv-gwc`

Customizable

The naming convention can be customized per customer in MazeVault UI under Organization Settings → Config Templates → Naming Convention.

6. Deployment Steps¶

Important: local build vs Azure deployment

docker compose up updates only a local Compose stack; it does not change AKS workloads. The gateway in AKS runs the signed images from the MazeVault Harbor and is updated with helm upgrade (Step 5).

Step 1: Generate Bootstrap Token on Primary¶

On your on-premise MazeVault Primary server:

# Login as admin
TOKEN=$(curl -sk -X POST https://PRIMARY_URL/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@mazevault.com","password":"YOUR_PASSWORD"}' | jq -r '.token')

# Generate bootstrap package
curl -sk -X POST https://PRIMARY_URL/api/v1/gateways/bootstrap \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"azure-gateway-gwc-01"}'

Save the returned token value — you will need it as gateway_bootstrap_token.

Step 2: Configure Terraform Variables¶

Create or update terraform.tfvars:

# Naming Convention
company_code  = "crd"       # Your 3-letter company code
project_code  = "mzv"       # MazeVault project code
environment   = "pro"       # Environment (pro, pre, dev, etc.)
is_nonprod    = false        # true for DEV/ITG/TST
location_zone = "int"       # Network zone (int or dmz)

# Regions
primary_region      = "germanywestcentral"
primary_region_code = "gwc"
dr_region           = "westeurope"
dr_region_code      = "we"
dr_enabled          = false

# Infrastructure
kubernetes_version  = "1.32"
aks_node_count      = 2
aks_vm_size         = "Standard_D2s_v3"
enable_redis        = true

Set sensitive values via environment variables:

export TF_VAR_primary_backend_url="https://your-primary.example.com:8443"
export TF_VAR_gateway_bootstrap_token="token-from-step-1"
export TF_VAR_entra_tenant_id="your-tenant-id"
export TF_VAR_gateway_frontend_url="https://mazevault.your-domain.internal"

Step 3: Deploy Infrastructure with Terraform¶

cd deploy/terraform/azure-gateway

# Initialize
terraform init \
  -backend-config="resource_group_name=rg-terraform-state" \
  -backend-config="storage_account_name=stterraformstate" \
  -backend-config="container_name=tfstate" \
  -backend-config="key=mazevault-gateway.tfstate"

# Plan
terraform plan -var-file=terraform.tfvars -out=tfplan

# Apply
terraform apply tfplan

Step 4: Get the Chart and Images from the MazeVault Harbor¶

MazeVault publishes signed images and the customer chart mazevault-customer to the MazeVault Harbor (registry.mazevault.com/mazevault); release vX.Y.Z has images tagged vX.Y.Z and chart version X.Y.Z. Use the robot account MazeVault issued to your company:

helm registry login registry.mazevault.com -u 'robot$mazevault+yourcompany'

# Download the chart; it contains examples/values-gateway-azure.yaml
helm pull oci://registry.mazevault.com/mazevault/mazevault-customer --version X.Y.Z --untar
cp mazevault-customer/examples/values-gateway-azure.yaml values-gateway.yaml

Choose how AKS pulls the gateway image:

Directly from Harbor. Create a pull secret in the namespace from Step 5 (the example values reference it as mazevault-registry):

kubectl create secret docker-registry mazevault-registry -n mazevault \
  --docker-server=registry.mazevault.com \
  --docker-username='robot$mazevault+yourcompany' \
  --docker-password='<robot-secret>'

From the ACR created in Step 3 (AKS can already pull from it; use this when AKS has no route to Harbor). Run mirror-to-registry.sh from the Compose kit (On-Premise Deployment) on a machine that reaches both registries, then set backend.image.repository to <acr-login-server>/mazevault/mazevault-backend and remove global.imagePullSecrets in values-gateway.yaml:
```
docker login registry.mazevault.com -u 'robot$mazevault+yourcompany'
az acr login --name $(terraform output -raw acr_name)
bash mirror-to-registry.sh --version vX.Y.Z --target $(terraform output -raw acr_login_server)/mazevault
```

Step 5: Deploy Gateway to AKS¶

The gateway is the mazevault-customer chart with only the backend enabled, running in gateway mode. Its runtime secrets come from the gateway Key Vault through External Secrets Operator and Workload Identity.

# Get AKS credentials
az aks get-credentials \
  --resource-group $(terraform output -raw gwc_resource_group) \
  --name $(terraform output -raw gwc_aks_cluster_name)

# External Secrets Operator (skip if the cluster already has it)
helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

# Create namespace
kubectl create namespace mazevault

KV=$(terraform output -json helm_values_summary | jq -r '.kv_gateway_name')

# Session secret (Terraform stores the master key, JWT key and connection strings, not this one)
az keyvault secret set --vault-name "$KV" --name mazevault-session-secret \
  --value "$(openssl rand -base64 32 | tr -d '\n/+=')"

# Bootstrap token from Step 1, which Terraform stored in the gateway Key Vault
kubectl create secret generic mazevault-gateway-bootstrap -n mazevault \
  --from-literal=token="$(az keyvault secret show --vault-name "$KV" --name gateway-bootstrap-token --query value -o tsv)"

Fill in values-gateway.yaml:

Value	Source
`global.workloadIdentity.clientId`, `backend.config.AZURE_MANAGED_IDENTITY_CLIENT_ID`	`terraform output -raw gwc_managed_identity_client_id`
`global.keyVault.secretStore.vaultUrl`	`https://<kv_gateway_name>.vault.azure.net/` (`terraform output -json helm_values_summary`)
`global.keyVault.secretStore.tenantId`, `backend.config.AZURE_TENANT_ID`	`tenant_id` from `helm_values_summary`
`backend.image.tag`, `backend.config.MAZEVAULT_VERSION`	the release you deploy (`vX.Y.Z` / `X.Y.Z`)
`backend.config.PRIMARY_BACKEND_URL`	URL of your Primary backend (`TF_VAR_primary_backend_url`)
`backend.config.GATEWAY_NAME`	the name used for the bootstrap token in Step 1
`backend.config.MAZEVAULT_GATEWAY_ENVIRONMENT` (or `…_ENVIRONMENTS`)	lowercase environment slug(s) this gateway serves, e.g. `pro`
`backend.config.MAZEVAULT_GATEWAY_ROLE`	`primary`, or `dr-standby` for a DR gateway
`backend.config.MAZEVAULT_CUSTOMER_*`, `MAZEVAULT_COMPANY_ID`	same values as on the Primary

The example keeps the service account name mazevault-gateway, which the managed identity's federated credential from Step 3 expects. Then deploy:

# Dry run first
helm template mazevault-gateway oci://registry.mazevault.com/mazevault/mazevault-customer \
  --version X.Y.Z -n mazevault -f values-gateway.yaml > /dev/null

helm upgrade --install mazevault-gateway oci://registry.mazevault.com/mazevault/mazevault-customer \
  --version X.Y.Z \
  -n mazevault \
  -f values-gateway.yaml \
  --wait --timeout 10m

To update the gateway later, change the version in --version, backend.image.tag and MAZEVAULT_VERSION and run the same helm upgrade.

Step 6: Verify Deployment¶

# Check pod status
kubectl get pods -n mazevault

# Check logs
kubectl logs -f deployment/mazevault-gateway-backend -n mazevault

# Expected log output:
# Starting MazeVault in gateway mode
# Registering gateway with primary...
# Setting up mTLS...
# Gateway registered successfully: <UUID>
# Heartbeat loop started (60s interval)

# Health check
kubectl port-forward svc/mazevault-gateway-backend 8080:8080 -n mazevault
curl -s http://localhost:8080/api/v1/health
curl -s http://localhost:8080/  # Returns: {"mode":"gateway"}

7. Post-Deployment Configuration¶

7.1 Add Key Vault Integration in MazeVault UI¶

On your Primary MazeVault frontend:

Navigate to Settings → Integrations → Add Integration
Select Azure Key Vault
Configure:
Vault URL: https://{kv_project_name}.vault.azure.net/
Auth Method: on_behalf_of
Tenant ID: Your Entra ID Tenant ID
Client ID: App Registration Client ID
Client Secret: App Registration Client Secret
Click Test Connection to verify OBO flow

7.2 Configure Sync Rules¶

Navigate to Integrations → Select your Key Vault integration
Click Sync Rules → Add Rule
Configure sync direction (Bidirectional, Push, or Pull)
Set scope (per project/environment)

7.3 Deploy MazeVault Agent¶

On target application servers, create /etc/mazevault/agent.yaml:

server_url: "https://gateway-url:8443"
agent_name: "app-server-01"
project_id: "<PROJECT_UUID>"
bootstrap_token: "<AGENT_BOOTSTRAP_TOKEN>"
cert_store_path: "/etc/mazevault/certs"
state_dir: "/var/lib/mazevault"
sync_interval_seconds: 3600
orchestrator_mode: true

gateway_discovery:
  enabled: true
  mode: "static"
  static_urls:
    - "https://gateway-url:8443"
  prefer_local: true

8. Key Vault RBAC Model¶

Each Key Vault Has Its Own RBAC

MazeVault respects Azure Key Vault RBAC. Users can only access secrets in Key Vaults where they have been granted appropriate roles.

8.1 System Access (Managed Identity)¶

Key Vault	Role	Purpose
Gateway KV	Key Vault Secrets User (Read)	Read operational secrets
Project KV	Key Vault Secrets Officer (Read/Write)	Sync and rotate project secrets

8.2 User Access (OBO Token)¶

When a user accesses secrets through MazeVault UI, the system uses the On-Behalf-Of (OBO) flow:

User logs in via Entra ID SSO
MazeVault exchanges the user's token for a Key Vault-scoped token
The Key Vault checks the user's RBAC role assignments
Access is granted or denied based on the user's individual permissions

To grant a user access to a Key Vault:

# Grant read access
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee user@company.com \
  --scope /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.KeyVault/vaults/{kv-name}

# Grant read/write access
az role assignment create \
  --role "Key Vault Secrets Officer" \
  --assignee user@company.com \
  --scope /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.KeyVault/vaults/{kv-name}

9. Disaster Recovery¶

9.1 DR Architecture¶

The DR region (West Europe) contains all resources in stopped/deallocated state:

AKS cluster — stopped
PostgreSQL — stopped
Redis — stopped
Key Vault — active (for geo-replication of secrets)

9.2 DR Activation Procedure¶

Start AKS cluster in DR region
Start PostgreSQL (restore from backup or geo-replica)
Start Redis
Generate new bootstrap token on Primary
Update bootstrap token in DR Key Vault
Gateway will automatically register with Primary

# Start DR AKS
az aks start --resource-group rg-{co}-{env}-{zone}-{proj}-we \
  --name {co}-{env}-{zone}-{proj}-we-aks01

# Start DR PostgreSQL
az postgres flexible-server start \
  --resource-group rg-{co}-{env}-{zone}-{proj}-we \
  --name {co}-{env}-{zone}-{proj}-we-psql01

10. Monitoring¶

10.1 Health Endpoints¶

Endpoint	Expected Response
`GET /api/v1/health`	`200 OK` with system status
`GET /`	`{"mode":"gateway"}`

10.2 Key Metrics¶

Metric	Description	Alert Threshold
Heartbeat failures	Consecutive failed heartbeats to Primary	> 3
Write queue size	Pending writes during Primary outage	> 1000
Circuit breaker state	Primary connectivity status	`open`
KV access denied	403 responses from Key Vault	Any
mTLS cert expiry	Days until gateway certificate expires	< 30 days

10.3 Log Analytics¶

All gateway logs are sent to Azure Log Analytics. Query example:

ContainerLog
| where ContainerName == "backend"
| where LogEntry contains "heartbeat"
| summarize count() by bin(TimeGenerated, 5m)

11. Troubleshooting¶

Gateway Not Registering¶

# Check PRIMARY_BACKEND_URL is reachable from AKS
kubectl exec -n mazevault deploy/mazevault-gateway-backend -- \
  curl -sk https://PRIMARY_URL/api/v1/health

# Check bootstrap token
kubectl exec -n mazevault deploy/mazevault-gateway-backend -- \
  env | grep GATEWAY_BOOTSTRAP_TOKEN

Key Vault Access Denied¶

# Verify Managed Identity has correct role
az role assignment list \
  --assignee $(terraform output -raw gwc_managed_identity_client_id) \
  --scope $(terraform output -raw gwc_key_vault_gateway_id)

# Check if admin consent was granted
# Azure Portal → Entra ID → App registrations → API permissions

Heartbeat Failures¶

# Check gateway logs
kubectl logs -f deployment/mazevault-gateway-backend -n mazevault | grep heartbeat

# Check network connectivity
kubectl exec -n mazevault deploy/mazevault-gateway-backend -- \
  curl -sk -o /dev/null -w "%{http_code}" https://PRIMARY_URL/api/v1/health

Multi-Environment Deployment (NPR + PRO)¶

Overview¶

For enterprise deployments, MazeVault supports a multi-environment gateway architecture with separate Non-Production (NPR) and Production (PRO) environments, each with primary and disaster-recovery (DR) standby gateways.

┌─────────────────────────────────────────────────────┐
│              MazeVault Central Server                │
│                                                     │
│  ┌─────────────┐          ┌─────────────┐          │
│  │ NPR Routing │          │ PRO Routing │          │
│  └──────┬──────┘          └──────┬──────┘          │
└─────────┼────────────────────────┼──────────────────┘
          │                        │
    ┌─────┴─────┐            ┌─────┴─────┐
    │           │            │           │
┌───┴───┐ ┌────┴──┐    ┌───┴───┐ ┌────┴──┐
│NPR-PRI│ │NPR-DR │    │PRO-PRI│ │PRO-DR │
│Active │ │Standby│    │Active │ │Standby│
└───────┘ └───────┘    └───────┘ └───────┘

Gateway Instances¶

Gateway	Environment	Role	Description
`gw-npr-primary`	NPR	primary	Handles dev, staging, QA workloads
`gw-npr-dr`	NPR	dr-standby	DR standby for NPR
`gw-pro-primary`	PRO	primary	Handles production workloads
`gw-pro-dr`	PRO	dr-standby	DR standby for PRO

Environment Variables¶

Each gateway instance requires these additional environment variables:

# In your Helm values or AKS deployment:
env:
  - name: MAZEVAULT_GATEWAY_ENVIRONMENT
    value: "NPR"          # or "PRO"
  - name: MAZEVAULT_GATEWAY_ROLE
    value: "primary"       # or "dr-standby"
  - name: MAZEVAULT_VERSION
    value: "1.5.0"         # your MazeVault version
  - name: AZURE_MANAGED_IDENTITY_CLIENT_ID
    value: "<mi-client-id>" # environment-specific MI

Terraform Configuration¶

Deploy each environment with separate Terraform workspaces:

# NPR Primary
cd deploy/terraform/azure
terraform workspace select npr
terraform apply \
  -var="gateway_environment=NPR" \
  -var="gateway_role=primary" \
  -var="managed_identity_name=mi-mazevault-npr"

# PRO Primary
terraform workspace select pro
terraform apply \
  -var="gateway_environment=PRO" \
  -var="gateway_role=primary" \
  -var="managed_identity_name=mi-mazevault-pro"

DR Configuration¶

DR standby gateways are deployed in a secondary region and remain in standby mode until activated:

# NPR DR
terraform workspace select npr-dr
terraform apply \
  -var="gateway_environment=NPR" \
  -var="gateway_role=dr-standby" \
  -var="managed_identity_name=mi-mazevault-npr"

# PRO DR
terraform workspace select pro-dr
terraform apply \
  -var="gateway_environment=PRO" \
  -var="gateway_role=dr-standby" \
  -var="managed_identity_name=mi-mazevault-pro"

DR standby gateways:

Send heartbeats to the central server but do not poll for tasks.
Share the same Managed Identity as their primary counterpart.
Are activated only during failover events.

Failover Procedures¶

Activate DR Gateway¶

When the primary gateway is unhealthy or unreachable:

# Check current gateway health
curl -s https://CENTRAL_SERVER/api/v1/admin/gateway-health | jq

# Check failover status for an environment
curl -s https://CENTRAL_SERVER/api/v1/admin/failover/NPR | jq

# Activate DR for NPR
curl -X POST https://CENTRAL_SERVER/api/v1/admin/failover/NPR/activate-dr \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Activate DR for PRO
curl -X POST https://CENTRAL_SERVER/api/v1/admin/failover/PRO/activate-dr \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Failback to Primary¶

Once the primary gateway is healthy again:

# Failback NPR to primary
curl -X POST https://CENTRAL_SERVER/api/v1/admin/failover/NPR/failback \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Failback PRO to primary
curl -X POST https://CENTRAL_SERVER/api/v1/admin/failover/PRO/failback \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Managed Identity Isolation¶

Each environment uses a dedicated Azure Managed Identity:

Environment	Managed Identity	Key Vault Access
NPR	`mi-mazevault-npr`	`kv-mazevault-npr`
PRO	`mi-mazevault-pro`	`kv-mazevault-pro`

This ensures production secrets are never accessible from non-production gateways, providing blast-radius isolation and compliance separation.

Monitoring¶

Monitor multi-gateway health via Prometheus metrics:

Metric	Type	Description
`mazevault_gateway_health_score`	Gauge	Health score per gateway (0.0–1.0)
`mazevault_gateway_heartbeat_age_seconds`	Gauge	Seconds since last heartbeat
`mazevault_gateway_failover_active`	Gauge	1 if DR is active for environment
`mazevault_gateway_tasks_total`	Counter	Total tasks by environment and status
`mazevault_gateway_task_duration_seconds`	Histogram	Task execution duration