Skip to content

Agent Deployment Guide

Installation, Configuration, and Operation of MazeVault Agents

Document Version: 1.1.0
Last Updated: 2026-10-26

1. Overview

The MazeVault Agent is a lightweight Go binary deployed on infrastructure nodes. It provides:

  • Secret synchronization — Automatic delivery of secrets to local applications
  • Certificate discovery — Scanning infrastructure for certificates (filesystem, JKS, Windows cert store)
  • Certificate lifecycle — Automated renewal and deployment of managed certificates
  • Real-time updates — WebSocket connection for instant push notifications
  • Local proxy — Secure proxy for application access to secrets without direct API calls
  • Offline resilience — Local caching ensures operation during network interruptions

2. Supported Platforms

Platform Architecture Notes
Linux x86_64, ARM64 Ubuntu, RHEL, Rocky Linux, Debian, Alpine
Windows x86_64 Windows Server 2019+
Kubernetes — No agent container image; run the agent on nodes/VMs (see 3.3)

3. Installation

3.1 Linux — Binary Installation

# Download the agent binary
curl -sL https://vault.example.com/api/v1/agents/download/linux-amd64 -o mazevault-agent
chmod +x mazevault-agent
sudo mv mazevault-agent /usr/local/bin/

# Create configuration directory
sudo mkdir -p /etc/mazevault
sudo mkdir -p /var/lib/mazevault/certs

3.2 Linux — systemd Service

Create /etc/systemd/system/mazevault-agent.service:

[Unit]
Description=MazeVault Agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=/etc/mazevault/agent.env
ExecStart=/usr/local/bin/mazevault-agent
Restart=always
RestartSec=10
User=mazevault
Group=mazevault

[Install]
WantedBy=multi-user.target

# Create service user
sudo useradd -r -s /sbin/nologin mazevault

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable mazevault-agent
sudo systemctl start mazevault-agent

3.2.1 Linux — Layout Installed by the Deployment Script

The install script generated under Deployment → Deploy (and the rpm/deb package) uses this layout. It is designed for hardened hosts (CIS/STIG baselines) where /var and /tmp are mounted noexec and SELinux is enforcing.

Path Purpose Owner / mode
/opt/mazevault-agent/bin/mazevault-agent Agent binary. Outside /var (noexec), default SELinux label bin_t. The directory is writable by the agent user because self-update stages the new binary next to the running one and the remote-uninstall runner executes a copy from it mazevault-agent, 0750 dir / 0755 file
/etc/mazevault/agent.yaml Configuration and state (agent.crt, agent.key) mazevault-agent, 0640
/etc/mazevault/certs Deployed certificates (cert_store_path) mazevault-agent, 0750
/var/log/mazevault Logs mazevault-agent, 0750
/etc/systemd/system/mazevault-agent.service Unit: ProtectSystem=strict, ReadWritePaths=/opt/mazevault-agent/bin /var/lib/mazevault /var/log/mazevault /etc/mazevault root, 0644

The script is idempotent and safe to re-run: it verifies the host before enabling the service and migrates hosts installed with the previous layout (binary under /var/lib/mazevault/bin, ExecStart drop-in overrides).

Step Behaviour
Preflight Required tools present; /opt/mazevault-agent/bin not on a noexec mount; SELinux mode detected
Download Certificate validation is never disabled. With a MazeVault CA (MAZEVAULT_CA_FILE on the server) the download uses it and falls back to the system trust store; without one, the system trust store is used and no ca.crt is written (server_ca_file stays empty)
Verification ELF check and size check; the cosign signature the server sends (X-Signature, X-Signature-Key-Id) is verified against the signing keys embedded in the script and the X-SHA256 is compared with the download, so an unsigned, tampered or truncated transfer is rejected. A failed download (including a transfer that broke off after HTTP 200) prints the HTTP status and the server's response body
Exec probe The new binary is staged as mazevault-agent.new next to the final path and run as mazevault-agent (mazevault-agent version) before it replaces the installed binary and before the unit is written — a noexec mount, an SELinux denial or a binary built for a newer glibc stops the installer, leaves a previously installed agent untouched and never puts systemd into a restart loop
SELinux When enforcing and semanage is available, /opt/mazevault-agent/bin is labelled bin_t explicitly
Start systemctl enable + restart; the final status and the last journal lines are printed with tokens and UUIDs redacted

Exit codes: 2 preflight failed, 3 download or binary verification failed, 4 the service did not start.

Hardened hosts

  • Do not move the binary to a root-owned directory such as /usr/local/bin: self-update needs a directory writable by the agent user and listed in ReadWritePaths.
  • Agents before 1.0.61 were dynamically linked and required glibc ≥ 2.34; they do not run on RHEL/Rocky/AlmaLinux 8 (GLIBC_2.34 not found). From 1.0.61 the agent is statically linked.
  • Remote uninstall (Deployment → Agents) requires the agent to run as root; script-installed agents run as mazevault-agent, so use the manual steps in section 9, Agent Lifecycle.

3.3 Kubernetes

MazeVault does not publish an agent container image, and the sidecar agent mode is not supported. To manage certificates used by Kubernetes workloads, run the agent on the nodes or VMs that hold them (sections 3.1–3.2), or use the MazeVault integrations for Kubernetes secrets. The agent binary always comes from your own MazeVault backend (/api/v1/agents/download/...), which serves signed binaries from the MazeVault Harbor.

4. Configuration

4.1 Environment Variables

Create /etc/mazevault/agent.env:

# Required
MAZEVAULT_SERVER_URL=https://vault.example.com
MAZEVAULT_BOOTSTRAP_TOKEN=mvbt_xxxxxxxxxxxxx

# Optional — Agent Identity
MAZEVAULT_AGENT_NAME=app-server-01
MAZEVAULT_PROJECT_ID=proj_xxxxxxxxxxxxx

# Optional — Storage
MAZEVAULT_CERT_STORE_PATH=/var/lib/mazevault/certs
MAZEVAULT_STATE_DIR=/etc/mazevault

# Optional — Logging
MAZEVAULT_LOG_LEVEL=info        # debug, info, warn, error
MAZEVAULT_LOG_FORMAT=text       # text, json

# Optional — Sync
SYNC_INTERVAL_SECONDS=300       # Default: 300 (5 minutes)

4.2 Variable Reference

Variable Required Default Description
MAZEVAULT_SERVER_URL Yes — MazeVault backend URL
MAZEVAULT_BOOTSTRAP_TOKEN Yes ¹ — One-time registration token
MAZEVAULT_AGENT_NAME No hostname Display name in UI
MAZEVAULT_PROJECT_ID No — Project scope for secrets
MAZEVAULT_CERT_STORE_PATH No /var/lib/mazevault/certs Certificate storage directory
MAZEVAULT_STATE_DIR No /etc/mazevault Configuration and state directory
MAZEVAULT_LOG_LEVEL No info Log verbosity
MAZEVAULT_LOG_FORMAT No text Log format (text or json)
SYNC_INTERVAL_SECONDS No 300 Secret sync interval in seconds

¹ Required only for initial registration. After bootstrap, the agent uses mTLS certificates.

5. Bootstrap & Registration

The agent uses a two-phase authentication model:

Phase 1: Bootstrap (One-Time)

  1. Generate a bootstrap token in the MazeVault web UI: Settings → Deployment → Agents → Generate Bootstrap Token
  2. Configure the agent with MAZEVAULT_SERVER_URL and MAZEVAULT_BOOTSTRAP_TOKEN
  3. Start the agent — it registers automatically via POST /api/v1/agents/register

Phase 2: mTLS (Ongoing)

After successful registration, the agent receives:

  • Client certificate — For mutual TLS authentication
  • CA certificate — For verifying the server
  • Heartbeat interval — Recommended reporting frequency
  • Sync interval — Secret synchronization frequency

All subsequent communication uses mTLS. The bootstrap token is no longer needed and can be revoked.

┌──────────┐          ┌───────────────┐
│  Agent   │──────────│  MazeVault    │
│          │ Bootstrap│  Backend      │
│          │  Token   │               │
│          │◀─────────│               │
│          │ mTLS Cert│               │
│          │          │               │
│          │──────────│               │
│          │ mTLS     │               │
│          │ (ongoing)│               │
└──────────┘          └───────────────┘

6. Periodic Operations

The agent runs several background tasks automatically:

Operation Interval Description
Secret sync Configurable (default 5m) Synchronizes secrets from backend to local storage
Heartbeat 60 seconds Reports health, metrics (CPU, memory, disk), and version to backend
Certificate discovery 1 hour Scans configured paths for certificates (filesystem, JKS, Windows cert store)
Configuration discovery With the filesystem scan Scans policy-defined paths for configuration files (.env, YAML, JSON, INI, properties, TOML, XML), detects embedded secrets, and reports candidates — requires the config_management license feature; see Configuration Management
Update check 24 hours Checks for newer agent versions
License heartbeat 5 minutes Sends telemetry to license server (when configured)
K8s reconciliation 5 minutes Reconciles Kubernetes Secret certificates (K8s only)
WebSocket reconnect Exponential backoff Maintains real-time push connection (1s → 2s → 4s → ...)

All intervals are fixed except secret sync (SYNC_INTERVAL_SECONDS).

Updates and Rollback Floor

On start and at every update check the agent asks its backend which version the agent version policy offers it (see Environment Variables § Agent Binaries). It installs a newer version, or a lower one only when the backend marks it as a downgrade and it is not below the agent's rollback floor. Every binary is verified (hash, signature, reported version) before the swap. The floor is the highest of:

  • the floor built into the agent binary (raised by MazeVault releases),
  • the highest min_rollback_version the agent has accepted from its backend, persisted in <state_dir>/min-rollback-version (the backend can only raise it),
  • min_rollback_version in agent.yaml.
# /etc/mazevault/agent.yaml
min_rollback_version: v1.2.0   # never install a version below v1.2.0

A refused version is logged with the running version, the offered version and the floor. Lowering the floor is a local action only: lowering agent.yaml has no effect below the persisted value; to clear the value persisted from the backend run

mazevault-agent -config /etc/mazevault/agent.yaml reset-min-rollback-version

Certificate Deployment Requests

Besides the periodic tasks, the backend pushes certificate deployment work to the agent over the WebSocket connection when a rotation runs. The agent handles these request types:

Request What the agent does
certificate_sync Installs certificate + key into a keystore (jks, pkcs12, windows_my) and the issuer chain into the platform trust store. windows_root / windows_ca targets receive public certificates only — a private key is never placed in a trust store.
certificate_file_sync Writes certificate files according to the target's content mode (public_only, public_chain, full_chain_with_key, key_only): leaf, chain, key and an optional combined bundle (PEM or PKCS#12). Private material is written 0600; existing files are kept as .BAK.
agent_command Runs a command for a deployment step (service restart, WLST restart script, orapki wallet import, keytool trust-store import). Only allowlisted commands execute. Arguments the backend flags as sensitive (store and wallet passwords) are redacted in the agent's audit log, response and error messages.

Command allowlist

The agent ships with a built-in allowlist (systemctl restart|reload, nginx -s reload, apachectl graceful, iisreset, IIS app-pool recycle, orapki wallet import_pkcs12 of a MazeVault-staged PKCS#12 and its cleanup, keytool -importcert of a MazeVault-staged issuer PEM into a JKS/PKCS12 trust store and its cleanup). Anything else — including the WLST restart script of a WebLogic target — must be allowed explicitly as a regular expression in the agent configuration:

allowed_commands:
  - '^/opt/weblogic/scripts/restart-managed-server\.sh\s+[\w\-\./:= ]+$'

Patterns are matched against the full command line. Keep them narrow (absolute paths, no shell metacharacters) — a rejected command fails the deployment step with a clear message instead of running.

Agent version requirements

certificate_file_sync, agent_command and the public-only semantics of windows_root/windows_ca require this agent version or newer. An older agent answers such requests with an explicit unsupported message type error, and the deployment step fails fast instead of installing material in the wrong place.

7. Certificate Discovery

The agent discovers certificates across infrastructure and reports them to the backend:

Discovery Sources

Source Platform Description
Filesystem Linux/Windows Scans configured directories for PEM, DER, PFX files
Java KeyStore (JKS) All Parses .jks and .p12 files
Windows Certificate Store Windows Enumerates system and user certificate stores
Kubernetes Secrets K8s Watches kubernetes.io/tls secrets

Discovery Configuration

Discovery paths are configured via the MazeVault web UI under:

Project → Settings → Agent Discovery → Paths

Results appear in the Discovered Certificates view in the dashboard.

Configuration file discovery is driven by organization-level discovery policy templates rather than by the paths above; results appear under Enterprise Functions → Configuration Management → Discovered Configurations. See Configuration Management. Agents can also deliver externally-issued API tokens to configuration files as agent_file deployment targets; see Token Management.

8. Monitoring

Health Endpoint

The agent reports health via heartbeat. Monitor agent status in the web UI:

Deployment → Agents

Key Metrics Reported

  • cpu_percent — Agent CPU usage
  • memory_percent — Agent memory usage
  • secrets_synced — Number of secrets synchronized
  • certificates_managed — Number of managed certificates
  • last_sync_time — Timestamp of last successful sync
  • uptime_seconds — Agent uptime

Prometheus Metrics

The backend exposes agent metrics at /metrics:

mazevault_agents_online{environment="production"} 12
mazevault_agent_sync_operations_total{agent_id="agt_xyz"} 1842
mazevault_agent_discovery_findings_total{agent_id="agt_xyz"} 37

9. Troubleshooting

Agent Won't Start

# Check logs
journalctl -u mazevault-agent -f

# Verify connectivity
curl -sk https://vault.example.com/api/v1/health
Problem Cause Solution
connection refused Backend unreachable Verify MAZEVAULT_SERVER_URL and firewall rules
certificate expired Agent cert expired Re-register with new bootstrap token
bootstrap token invalid Token expired or used Generate a new bootstrap token in the web UI
TLS handshake error CA certificate mismatch Verify CA certificate chain
status=203/EXEC in systemctl status systemd cannot execute the binary: noexec mount (typically /var), SELinux label, or missing execute bit Re-run the install script (installs to /opt/mazevault-agent/bin, see 3.2.1); check findmnt -T <binary dir> and ls -Z
GLIBC_2.34' not found Agent older than 1.0.61 on a host with glibc < 2.34 (RHEL/Rocky/Alma 8) Install agent 1.0.61 or later (statically linked)
Installer exits with 2, 3 or 4 Preflight, download/verification, or service start failed Read the printed reason; 3 with HTTP 503 means the server has no agent binaries configured

Agent Lifecycle (Deployment → Agents)

Registered agents are managed from Deployment → Agents via the per-row actions menu. These operations act on the MazeVault control-plane identity of an installed agent and are distinct from deleting a generated install-script record under Deployment → Deploy.

Action What it does When to use
Unregister (Odregistrovat) Retires the agent, invalidates its mTLS identity and recovery tokens, deactivates project bindings, and blocks all future runtime use. History is preserved. Choose a security reason category (compromised / lost credential / stolen host) to record it as a revoke. Decommissioned, lost, offline, or manually removed hosts.
Remote uninstall (Odinstalovat vzdáleně) Asks a connected agent to uninstall itself from the host and reports the real outcome back before finalizing. Supported on Linux (systemd package install, agent running as root) and Windows (MSI service install; the MazeVaultAgent service is removed via msiexec). The agent must be currently connected. The agent is retired only after it confirms a successful removal; if the removal fails the agent stays active and the console shows Uninstall failed so you can retry or intervene. Cleanly remove the agent from a reachable Linux or Windows host.
Re-register / Renew identity (Přeregistrovat) Retires the old identity and issues a single-use replacement bootstrap token (valid 24 h) for re-enrolling the host. mTLS certificate expired or compromised; replacing a host.

An impact preview (managed certificates, project bindings, rotation targets, SSH keys, pending keytab work) is shown before unregistering. Unresolved work such as pending keytab deployments or actively managed certificates requires the Force option to proceed.

Unregister does not uninstall the host binary. For offline hosts, or where remote uninstall is unsupported (non-packaged/manual installs, macOS), unregister the agent and remove it locally.

Linux:

sudo systemctl stop mazevault-agent && sudo systemctl disable mazevault-agent
sudo apt remove mazevault-agent   # or: sudo rpm -e mazevault-agent
sudo rm -rf /etc/mazevault /var/lib/mazevault /var/log/mazevault /opt/mazevault-agent

Windows (PowerShell, elevated):

sc.exe stop MazeVaultAgent
$p = Get-WmiObject Win32_Product -Filter "Name='MazeVault Agent'"
if ($p) { $p.Uninstall() } else { sc.exe delete MazeVaultAgent }
Remove-Item -Recurse -Force "$env:ProgramData\MazeVault" -ErrorAction SilentlyContinue

Re-Registration

If the agent's mTLS certificate expires or is compromised:

  1. In Deployment → Agents, open the agent's actions menu and choose Re-register / Renew identity (or Unregister with a security reason for a compromised host).
  2. Copy the single-use replacement bootstrap token from the dialog.
  3. Remove local state on the host: rm -rf /etc/mazevault/agent-state.json
  4. Restart the agent with the new MAZEVAULT_BOOTSTRAP_TOKEN.

Debug Logging

Enable verbose logging for diagnostics:

MAZEVAULT_LOG_LEVEL=debug mazevault-agent

10. Security Considerations

  • Bootstrap tokens are single-use and time-limited (default: 24 hours)
  • mTLS certificates are scoped to the specific agent and project
  • Secret data is encrypted in transit (TLS 1.2+) and cached encrypted locally
  • Minimal permissions — agents only access secrets assigned to their project
  • No inbound ports — the agent initiates all connections (outbound only)