Maintenance¶
Upgrade Procedures, Key Rotation, and Scheduled Maintenance
Document Version: 1.1.0
Last Updated: 2026-04-07
1. Maintenance Schedule¶
| Task | Frequency | Downtime Required | Priority |
|---|---|---|---|
| Platform upgrade | As released | Yes (rolling for K8s) | High |
| Database backup verification | Weekly | No | High |
| TLS certificate renewal | Before expiry | No (reload) | Critical |
| Secret cipher key rotation | 90 days | No (background) | High |
| JWT signing key rotation | On demand | No | Medium |
| Master key rotation | Annual | Yes (planned) | Critical |
| Database vacuum/analyze | Weekly | No | Medium |
| Log rotation/cleanup | Daily | No | Low |
| DR test | Quarterly | No (test env) | High |
2. Platform Upgrade¶
Update Notifications¶
When a newer release exists, administrators see a banner in the console (GET /api/v1/system/updates/check, permission system:read). The backend asks the MazeVault license server (LICENSE_SERVER_URL) a minute after start and then daily (or on Re-check) and needs a registered license; the request carries only the license and instance identity and the current version. The banner links to the release's changelog in the MazeVault documentation, which requires a docs login. Development and pre-release builds (dev, staging-…) never show it. Without a license or license server connection the banner stays hidden.
Pre-Upgrade Checklist¶
- [ ] Review release notes for breaking changes
- [ ] Create full database backup
- [ ] Verify backup integrity
- [ ] Document current version and configuration
- [ ] Schedule maintenance window (if required)
- [ ] Notify affected users
Kubernetes Upgrade¶
# Update Helm repo
helm repo update
# Diff changes
helm diff upgrade mazevault mazevault/mazevault \
-n mazevault -f values-production.yaml
# Create backup
kubectl exec -n mazevault deploy/mazevault-backend -- \
pg_dump -h $DB_HOST -U $DB_USER $DB_NAME | gzip > backup-pre-upgrade.sql.gz
# Upgrade
helm upgrade mazevault mazevault/mazevault \
-n mazevault -f values-production.yaml \
--wait --timeout 10m
# Verify
kubectl get pods -n mazevault
curl -sk https://vault.example.com/api/v1/health | jq .
On-Premise Upgrade¶
cd /opt/mazevault
# Backup
./scripts/mazevault-backup.sh
# Pull new images (connected) or load from offline package (air-gapped)
docker compose pull
# OR
docker load < images/mazevault-backend-1.0.17.tar
docker load < images/mazevault-frontend-1.0.17.tar
docker load < images/mazevault-ocsp-1.0.17.tar
# Update image tags in docker-compose.yml
# Apply upgrade
docker compose up -d
# Verify
docker compose ps
curl -sk https://localhost/api/v1/health | jq .
Post-Upgrade Verification¶
- [ ] All containers/pods running
- [ ] Health endpoint returns
healthy - [ ] Database migrations applied successfully
- [ ] Secret access verified (read a test secret)
- [ ] Certificate operations verified
- [ ] Agent heartbeats resuming
- [ ] Audit log recording events
- [ ] License validation successful
Rollback Procedure¶
# Kubernetes
helm rollback mazevault -n mazevault
# On-Premise
docker compose down
# Restore previous image tags in docker-compose.yml
docker compose up -d
# Restore database if migration was applied
3. Database Maintenance¶
Vacuum and Analyze¶
# On-Premise
docker exec client-postgres psql -U mazevault -d mazevault -c "VACUUM ANALYZE;"
# Azure
# Automatic for Azure Database for PostgreSQL Flexible Server
Reindex (if needed)¶
Monitor Database Size¶
docker exec client-postgres psql -U mazevault -d mazevault -c "
SELECT pg_size_pretty(pg_database_size('mazevault')) as db_size;
"
4. Secret Rotation Schedule¶
| Key Type | Rotation | How |
|---|---|---|
| Secret cipher keys | Every 90 days | Automatic (background re-encryption) |
| JWT signing keys | On demand | API call or admin interface |
| Master encryption key | Annual | Planned maintenance window |
| API keys | Organization policy | Manual via admin interface |
JWT Key Rotation¶
Rotate JWT signing keys via the admin interface:
Settings → Security → JWT Signing Keys → Rotate
The system performs seamless key rollover — existing tokens remain valid until expiry, and new tokens are signed with the new key.
5. Certificate Maintenance¶
CA Key Rotation¶
Root CA keys should be rotated according to your organization's certificate policy. Typical schedules:
| CA Type | Recommended Lifetime |
|---|---|
| Root CA | 10-20 years |
| Intermediate CA | 5-10 years |
| Issuing CA | 2-5 years |
CRL Maintenance¶
CRLs are regenerated automatically. Verify CRL freshness:
curl -sk https://vault.example.com/api/v1/crl/pem | \
openssl crl -inform PEM -noout -lastupdate -nextupdate
6. Maintenance Window Communication¶
Template for maintenance notification:
Subject: MazeVault Scheduled Maintenance — [DATE]
Dear Team,
MazeVault will undergo scheduled maintenance on [DATE] from [TIME] to [TIME] (UTC).
During this window:
- The platform will be [briefly unavailable / in read-only mode]
- Affect: [describe impact]
Actions required:
- [Any pre-maintenance actions]
Post-maintenance:
- Service will be restored automatically
- Verify access at https://vault.example.com
For questions, contact: [support contact]
7. Automated System Jobs¶
MazeVault runs several background jobs automatically. These do not require manual intervention but are listed here for operational awareness, firewall planning, and troubleshooting.
Backend Jobs¶
Certificate Management¶
| # | Job | Interval | Description |
|---|---|---|---|
| 1 | Certificate lifecycle scheduler | 1h + 6h | Marks expired certificates, checks auto-renewal eligibility, auto-archives expired items. Syncs status from external CAs every 6 hours |
| 2 | Certificate renewal scheduler | 1h | Detects certificates nearing expiry, enqueues renewal tasks, processes renewal queue (5 concurrent workers, PostgreSQL advisory lock) |
| 3 | CRL regeneration | 168h full / 24h delta | Regenerates Certificate Revocation Lists per CA. Defaults: full CRL weekly, delta CRL daily (configurable per CA) |
| 4 | CA sync | 5 min | Discovers and imports certificates from connected external CA accounts |
| 5 | ACME server cleanup | 15 min / 30 min | Cleans up expired ACME nonces (15 min) and expired certificate orders (30 min) |
Secrets & Rotation¶
| # | Job | Interval | Description |
|---|---|---|---|
| 6 | Rotation scheduler | 1 min | Detects due secret and certificate rotations. Executes via 3-worker pool with database row-level locking and exponential backoff |
| 7 | Sync rules engine | 1 min | Evaluates secret synchronization rules and triggers sync to configured external targets |
| 8 | Key rotation | 24h | Rotates internal system keys (RSA signing, cipher, API keys) with configurable grace periods |
Notifications & Reporting¶
| # | Job | Interval | Description |
|---|---|---|---|
| 9 | Notification scheduler | 24h | Sends expiry alerts for certificates expiring within 30 days and rotation failure alerts via configured channels (Teams, Slack, Email, JIRA) |
| 10 | Weekly report | 1 min check | Generates weekly compliance and audit summary report. Triggers every Monday at 7:00 AM (Europe/Prague) |
Azure Entra ID¶
| # | Job | Interval | Description |
|---|---|---|---|
| 11 | Entra ID sync | 1h | Synchronizes Azure Entra ID app registrations and their credentials |
| 12 | Entra credential expiry monitor | 1h | Monitors Entra app credentials for upcoming expiry (30-day and 7-day thresholds), sends alerts |
Licensing¶
| # | Job | Interval | Description |
|---|---|---|---|
| 13 | License scheduler | 5 min / 1h / 24h | Heartbeat to license server (5 min), usage metrics reporting (1h), full license validation and status renewal (24h) |
Audit & Logging¶
| # | Job | Interval | Description |
|---|---|---|---|
| 14 | Audit log retention | 24h | Enforces configured audit log retention policy, removes expired entries |
| 15 | Audit archive | 5 min | Batches audit events and archives to configured long-term storage (S3, GCS, Azure Blob) |
| 16 | Log stream flush | 5 sec | Flushes buffered audit and event logs to configured SIEM endpoints (Splunk, Elasticsearch, Syslog) |
System & Infrastructure¶
| # | Job | Interval | Description |
|---|---|---|---|
| 17 | Archive cleanup | 1h | Hard-deletes archived secrets and certificates after retention period expires |
| 18 | TLS certificate reload | 30 sec | Hot-reloads server TLS certificate from disk without restart. Configurable via MAZEVAULT_TLS_RELOAD_INTERVAL |
| 19 | Gateway health monitor | 30 sec | Monitors gateway heartbeats, detects stale gateways (3 consecutive missed heartbeats), triggers DR failover if enabled |
| 20 | External change detector | 1h | Orchestrator mode only — detects drift between local secret metadata and values stored in external secrets manager |
Agent Jobs¶
| Job | Interval | Description |
|---|---|---|
| Secret synchronization | Configurable (default 5 min) | Syncs secrets from backend to local store |
| Heartbeat | 60 seconds | Reports agent status, metrics, and version |
| Certificate discovery | 1 hour | Scans infrastructure for certificates (filesystem, JKS, Windows cert store) |
| Update check | 24 hours | Checks for newer agent version availability |
Gateway Jobs¶
| Job | Interval | Description |
|---|---|---|
| Heartbeat | 60 seconds | Reports gateway status and connectivity |
| Task polling | 10 seconds | Polls backend for pending certificate operations |
| Queue reconciliation | 30 seconds | Reconciles local task queue with backend state |
Configurable intervals
Most intervals are fixed by design for optimal performance. User-configurable intervals:
- Agent secret sync —
SYNC_INTERVAL_SECONDSenvironment variable - TLS certificate reload —
MAZEVAULT_TLS_RELOAD_INTERVALenvironment variable (default: 30s) - CRL regeneration — configurable per CA in CA settings (full and delta intervals independently)
Related¶
- Backup & Restore — Backup procedures before maintenance
- Health Checks — Post-maintenance verification
- Scaling — Capacity planning