Best Practices for Architecting a Global AKS Disaster Recovery Strategy

Orchestrate application-consistent backups across dependencies

Velero hooks provide per-pod consistency by annotating commands to run before and after backup operations. A MongoDB replica set benefits from coordinated write flushing:

annotations:
  pre.hook.backup.velero.io/command: '["/usr/bin/mongo", "--eval", "db.fsyncLock()"]'
  post.hook.backup.velero.io/command: '["/usr/bin/mongo", "--eval", "db.fsyncUnlock()"]'
  pre.hook.backup.velero.io/timeout: 30s

This works well for single applications, but in production environments, distributed systems make the problem more complex. 

Consider an ecommerce platform with frontend pods, an API gateway, an inventory service, and a PostgreSQL database. When Velero backs up the database at 2:00 PM, it locks tables and creates a consistent snapshot, but the inventory service doesn’t know this is happening—it keeps processing orders and writing new data. By 2:05 PM, the database backup completes, and the database is unlocked. At 2:10 PM, Velero backs up the inventory service, which now contains five minutes of transactions that the database snapshot doesn’t have.

During a disaster recovery event weeks later, you restore both backups. The inventory service shows orders that don’t exist in the database. Customers received confirmation emails for purchases that the restored database does not have a record of. This happens because Velero executes each pod’s hooks independently without coordinating timing across related services.

Dependency-aware backup workflows address this situation through coordinated execution sequences, like the one provided below:

1. Signal the application tier to stop accepting new requests.
2. Drain in-flight requests from the API gateway.
3. Flush application caches to the database.
4. Execute database pre-backup hook (lock tables).
5. Snapshot database persistent volumes.
6. Execute database post-backup hook (unlock tables).
7. Execute application-tier backups.
8. Resume normal operations.

Enterprise backup solutions add workflow coordination to Velero’s capabilities. A backup plan defines which applications depend on others. The database backs up first, then the application services that use that database back up second. This sequencing prevents problems where restored applications look for data that isn’t in the restored database backup.

Helm-based applications introduce additional considerations. A Helm release deploys multiple resources simultaneously: deployments, services, ConfigMaps, Secrets, and persistent volume claims. Velero can capture these resources, but Helm release metadata (release history, chart values) requires explicit backup configuration. Organizations that use Helm extensively should verify that the backup scope includes the release metadata required for operational procedures that rely on helm upgrade or helm rollback commands. Trilio’s Cloud-Native Backup and Restore for Helm helps address those nuances out of the box.

Test disaster recovery procedures regularly and measure RTO

Theoretical RTO estimates assume perfect execution of documented procedures under ideal conditions. Reality introduces delays: Cluster provisioning takes longer than expected, backup data transfer saturates available network bandwidth, and manual configuration steps require multiple attempts due to typos or missed dependencies. 

Regular DR testing reveals these gaps between theory and practice. A good test workflow validates multiple aspects simultaneously. 

The diagram below structures this validation process into three sequential phases, each with concrete steps that move from controlled test setup through timed execution to documented analysis. The RTO breakdown at the bottom translates the workflow into measurable time ranges per recovery stage, giving you a baseline to compare against your theoretical estimates and track improvement across quarterly test cycles.

DR testing and RTO measurement workflow

Non-disruptive testing uses namespace isolation and network policies to prevent test workloads from impacting production:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dr-test-isolation
  namespace: dr-test-2024-q1
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: dr-test-2024-q1

This policy restricts test-namespace pods to communicating only within the test namespace, preventing accidental connections to production databases or external services. Test completion involves validating application functionality in the isolated environment, documenting the actual RTO and any manual steps required, and then deleting the test namespace.

Be sure to update DR documentation based on test findings. If a test revealed that storage class mapping required three attempts to get correct, document the correct mapping explicitly. If network policy CIDR ranges caused connectivity failures, add validation steps to verify ranges before executing a full restore. Documentation improvements compound over multiple test cycles, reducing errors during actual incidents.

Organizations often discover that continuous restoration claims of sub-15-minute RTO require specific conditions: applications with rapid startup characteristics, low data change rates enabling efficient incremental replication, and network connectivity between regions that supports replication bandwidth requirements. Testing validates whether your specific applications and infrastructure meet these conditions rather than assuming that vendor claims apply universally.

Choose DR solutions aligned with operational maturity and risk

DR tooling selection balances capabilities against operational complexity and risk tolerance during production incidents. Velero provides robust backup and restore functionality as a community-supported CNCF project. Organizations with Kubernetes expertise, flexible RTO requirements, and a tolerance for community-supported models may find Velero sufficient. Others require guaranteed support during regional outages or automation that eliminates manual steps during high-stress incident response.

The support model difference matters most during actual incidents. Your primary Azure region goes down at 2 AM on a Saturday. You start the restore process to your DR cluster, but the PostgreSQL backup fails with an error about incompatible storage class configurations. With community-supported Velero, you open a GitHub issue. It joins the several hundred already open and waits. Maybe someone responds in an hour, maybe Monday morning. There’s no one to call, no escalation for production emergencies, no engineer who knows your specific setup.

Enterprise solutions like Trilio provide vendor support with committed SLAs:

• 24/7 support with response times based on severity level
• Root-cause analysis for failed restore operations
• Configuration review before DR tests to identify potential issues
• Direct engineering escalation for production-blocking problems
• Hotfixes and patches without waiting for community release cycles

Organizations face a tradeoff: accept the uncertainty of community support or invest in guaranteed response times during production incidents. A team with deep Kubernetes knowledge and experience in troubleshooting Velero operations may operate effectively with community support. Teams with limited Kubernetes expertise or those managing AKS alongside other responsibilities benefit from vendor support that guides incidents.

The radar chart below compares Velero and enterprise solutions across the five factors that matter most when production recovery is on the line. Velero holds its own on DR test frequency and team Kubernetes expertise, where deep hands-on knowledge compensates for the lack of automated tooling. The gap widens on the remaining three dimensions—application complexity, cross-region failover, and production support tolerance—where enterprise solutions pull significantly ahead. That last dimension is how much uncertainty your team can absorb without guaranteed support escalation paths, is often the deciding factor.

Velero vs enterprise solutions: capability comparison

Organizations should test both approaches when possible. Deploy Velero via Azure Backup, run several DR tests, measure the actual RTO, including all manual steps, and evaluate the operational burden. If the results meet requirements and the team can handle troubleshooting effectively, Velero provides cost-effective DR protection. If testing reveals operational gaps, automation needs, or support concerns, evaluate enterprise solutions based on specific gaps identified through testing rather than theoretical capability comparisons.