Failed Vendor Recovery: The 12-Week Rescue Architecture

Week 1: The Triage Assessment

The first week is not about fixing anything. It is about understanding the full scope of what you inherited. Start with the codebase: run static analysis tools (SonarQube, Semgrep, or Snyk) against every repository. You are looking for hardcoded secrets, dependency vulnerability chains, and test coverage below 40% on core business logic. Pull the dependency manifest and cross-reference against the NIST NVD for CVEs rated CVSS 7.0 or higher. Any critical vulnerability in a data-handling service is a blocker, not a backlog item.

For compliance posture, request every document the outgoing vendor holds: BAA and DPA agreements, penetration test reports (look for the date — anything older than 12 months in a HIPAA or PCI environment is expired), SOC 2 reports, and any in-progress compliance certifications. If the vendor cannot produce a BAA for every subprocessor they used, you have a gap that must be addressed before the system touches production PHI or PCI data again. Check cloud provider configurations against the CIS Benchmarks for AWS, Azure, or GCP — particularly IAM policy scope and S3/Blob public access settings.

Data integrity requires running row counts and checksum validation across every primary entity table. Ask: does the record count in the database match what the application reports? Are there orphaned foreign keys? Are audit tables present, and do their timestamps align with application logs? For the deployment pipeline, look for whether CI/CD gates actually enforce test passage or whether they can be bypassed. Test coverage gaps are often worst in authentication flows, payment processing, and data export endpoints — exactly where regulators look first.

→Run SonarQube/Semgrep static analysis on all repositories on day 1
→Cross-reference dependencies against NIST NVD for CVSS ≥7.0 CVEs
→Request all BAAs, DPAs, and subprocessor agreements from outgoing vendor
→Validate database row counts and foreign key integrity
→Check CI/CD pipeline for bypassable test gates
→Pull CIS Benchmark scan results for cloud environments

Salvageable vs. Rebuild: The Decision Framework

The most expensive mistake in vendor recovery is rebuilding what could be salvaged, or salvaging what should be rebuilt. The first question is whether compliance debt is architectural or surface-level. Surface-level compliance debt — missing log fields, incorrect session timeout values, a BAA that needs updating — can be remediated in days. Architectural compliance debt — PHI stored in a schema with no field-level tagging, access control enforced at the application layer with no service-level equivalent, audit events written to mutable application logs — requires rework that touches every service. If the data model does not support the compliance requirements, you are rebuilding.

The second question is whether the data model is sound. A well-normalized schema with clear entity relationships and no PHI/PII scattered into JSON blobs or untyped text fields can usually be preserved even when the application layer above it needs replacement. A schema that has grown by ad hoc column additions over three years, with patient identifiers appearing in 23 tables under 6 different column names, is not salvageable as-is and will create ongoing compliance liability if you build on top of it.

The third question is team retention. If the outgoing vendor's engineers are available for a 4-week knowledge transfer, the salvage threshold shifts significantly — institutional knowledge about undocumented integrations and data migration quirks is worth more than it appears on paper. If the team is unavailable or hostile, factor in a 2-week archaeology period for any system you choose to retain. Document every decision in this framework — regulators and future auditors will ask why architectural decisions were made, and 'we assessed and chose to retain' is a defensible answer when documented.

→Architectural compliance debt (schema-level, service-layer) → rebuild
→Surface-level compliance debt (config, BAA, log fields) → remediate
→Unsound data model with scattered PII → rebuild data layer
→Well-normalized schema with clear entities → salvage and extend
→Team available for knowledge transfer → raise salvage threshold
→No knowledge transfer possible → add 2-week archaeology period per retained system

The Parallel Track Architecture

The cardinal rule of vendor recovery is that the existing system stays live until the replacement is validated and cut over. This is not optional — it is a compliance and business continuity requirement. The parallel track architecture runs the legacy system in read/write production mode while the replacement system is built on a shadow deployment. Shadow deployments receive a copy of production traffic (sanitized of live credentials and PHI outside the test environment) and process it independently, allowing output comparison before cutover.

Feature flag infrastructure is the mechanism that makes cutover reversible. Using a flag system like LaunchDarkly or a self-hosted equivalent, individual features or user segments can be migrated to the new system while the legacy system handles the remainder. This means a failed cutover of a billing module affects 5% of transactions, not 100%, and can be reversed in under a minute. Every feature migration should have a documented rollback flag that does not require a code deployment to activate.

The shadow deployment must match the production compliance configuration from day one. Running PHI through an environment without HIPAA-compliant logging and access controls — even temporarily, even in a 'test' capacity — is a potential breach event. Provision the shadow environment with the same encryption configuration (AES-256 at rest, TLS 1.2 minimum in transit), the same IAM policy structure, and the same audit log sinks as production. This is slower to set up and worth every hour it costs.

→Legacy system stays live in production until cutover is fully validated
→Shadow deployment receives sanitized traffic copy for output comparison
→Feature flags (LaunchDarkly or equivalent) enable per-feature, per-segment migration
→Every feature migration requires a no-deploy rollback flag
→Shadow environment must match production compliance configuration from day one
→AES-256 at rest, TLS 1.2+ in transit required in shadow environment

Compliance Gap Remediation

Compliance gap remediation runs in parallel with the rebuild, not after it. The gaps most commonly left by failed vendors fall into three categories: audit trail gaps, access control holes, and missing or incomplete legal agreements. Audit trail gaps are the most operationally complex: if the system has been processing PHI without generating complete audit logs, you cannot retroactively create compliant audit records. What you can do is document the gap formally, establish a cutoff date, and ensure the new system generates compliant audit events from that date forward. This documented remediation is more defensible than fabricated records.

Access control holes are usually faster to close but require careful mapping. Pull every IAM role and policy in the cloud environment and every application-level permission assignment. Cross-reference against the minimum necessary standard (HIPAA) or least privilege principle (general). Any role with broader access than its function requires is a finding. Service accounts with human-level permissions, shared credentials across services, and MFA exemptions for administrative accounts are the three patterns auditors find first.

Missing BAAs and DPAs require legal coordination alongside the technical work. Every cloud service, SaaS product, and third-party API that touches PHI or personal data requires a signed agreement. Build a subprocessor inventory as a living spreadsheet — service name, data categories accessed, agreement status, agreement expiry. This document becomes evidence in audit. For DPAs under GDPR or UK GDPR, ensure standard contractual clauses (SCCs) are in place for any transfer outside the EEA or UK.

→Document audit trail gaps formally with cutoff date — do not fabricate records
→Map all IAM roles against minimum necessary / least privilege standard
→Remove MFA exemptions for all administrative and PHI-access accounts
→Eliminate shared credentials across services
→Build subprocessor inventory: service, data categories, BAA/DPA status, expiry
→Verify SCCs are in place for any cross-border personal data transfer

Week-by-Week Timeline

Weeks 1-2 are triage and decision. By the end of week 2, you need: a complete compliance gap register, a salvage/rebuild decision for each system component, a subprocessor inventory with gap identification, shadow environment provisioned and compliant, and an engineering team (internal or augmented) committed and onboarded. Any ambiguity in team composition at week 2 means the week-12 deadline is at risk.

Weeks 3-6 are parallel build and gap remediation. The engineering team builds replacement components in priority order (highest compliance risk first, then business-critical path). Legal closes outstanding BAAs and DPAs. Infrastructure team hardens the cloud environment against the CIS Benchmarks. Feature flag infrastructure is deployed and tested. By week 6, the first replacement component should be in shadow production with output parity validated against the legacy system.

Weeks 7-10 are migration and validation. Components move from shadow to live traffic via feature flags, starting with lowest-risk user segments. Each migration gate requires: output parity validation, compliance configuration verification, load test at 150% of expected traffic, and rollback test confirming the flag works. Weeks 11-12 are final cutover and legacy decommission. Legacy system is taken to read-only on week 11, decommissioned on week 12 after a 7-day data integrity validation period. Compliance documentation package is finalized: architecture diagram, data flow diagram, BAA register, access control policy, audit log retention configuration.

→Week 1-2: Triage, decisions, shadow environment live, team committed
→Week 3-4: Compliance gap remediation begins, parallel build starts
→Week 5-6: First component in shadow production, BAAs/DPAs closed
→Week 7-8: Feature flag migration begins, lowest-risk segments first
→Week 9-10: Full traffic migration with per-gate validation
→Week 11: Legacy to read-only, 7-day integrity validation
→Week 12: Legacy decommission, compliance documentation package complete