Integration and Dev Environments

The sandbox is the most flattering place your AI will ever run. The data is clean, every response comes back instantly, and the one test account has every permission turned on. Then the AI meets a real system.

Picture a support agent that passed every test in development. In integration testing, the CRM sends back fields in a different shape than expected. Half the customer records are incomplete. The outside API starts rate-limiting. A webhook arrives twice, so the agent answers the same customer twice. A database call that took one second now takes three.

Nothing in the logic changed. The real environment exposed what the build had been assuming all along.

This is Part 9 of From AI Idea to Production. Part 8 put someone other than the builder between the build and the next stage. This part covers the environments on the way to users, and what to check when a build first connects to real systems.

Five environments, each with one job

Part 6 described environments by what each one holds, with separate data, credentials, and settings. Sorted by the job each one does, I like five environments, each with one job. Integration and user testing get their own step, because those are the two places a build most often meets something it was not built for.

Five environments, one job each. A build moves right only when the check between two steps passes.

What each environment is for, and what it should catch
Environment	Its one job	What it should catch
Development	Build safely with isolated data and mocked services	Logic errors, before anything real is touched
Integration	Connect to real APIs, databases, and tools under control	Shape changes, slow calls, rate limits, and duplicate events
Staging	Behave like production	Settings that differ from production, and deployment mistakes
UAT	Let the people who own the workflow try to break it	Steps and handoffs the build does not match
Production	Release with monitoring, a named owner, and a rollback plan	What the earlier stages could not reproduce

Smaller teams can combine some of these. They cannot skip the purpose of each one. If integration and staging share one environment, the build still needs one run with production settings before users see it. If there is no separate UAT environment, the people who own the workflow still get a fixed window to try to break the build before anyone calls it ready.

Staging earns its place only if it rejects what production rejects. Any setting that changes what the model sees or does has to match production, including the model and prompt version, the redaction rules, timeouts and retries, and the service account's access. A different model in staging means you tested a different tool.

A made-up week on real connections

Back to the made-up support assistant for an online retailer from Part 6. In development it talked to fakes that always answered quickly with clean records. In integration it talks to the sandboxes of the real order system, CRM, carrier feed, and payment provider. Here is what one week there might turn up.

One made-up week in integration, and the check that catches each surprise
What the real connection did	What the assistant did	The check that catches it
The CRM returned the customer's tier as a nested object, not the plain text the mock sent	Read the tier as empty and treated a long-standing customer as new	Contracts
The integration service account lacked a right the all-access test account had	The refund step failed with an access error nobody had seen before	Permissions
The carrier feed started rate-limiting at busy hours	Retried at once, hit the limit again, and gave up with no answer	Failure paths
The refund webhook arrived twice	Answered the same customer twice and logged two refunds	Failure paths
A database call that took one second in development took three	Ran past the chat window's timeout, and nobody could tell which call was slow	Failure paths and logging

None of this showed up on the developer's laptop, because every test was written against the fakes.

Tip

Every surprise from a real connection should leave something behind, such as a contract test, an eval case, or a rule in code. If it only leaves a note in a chat thread, it will happen again.

At the integration step, I check six things on purpose. The next six sections take them one at a time.

Check 1. Contracts

A contract is what each system expects, what it returns, and what it does when something fails. Most teams write down the first two. The third is the one that bites. Does the CRM answer a missing customer with an error or with an empty record? Does the payment provider say "accepted" before it says "confirmed"?

Ordinary software usually fails loudly when a contract breaks. An AI tool often fails quietly. The missing field becomes a gap in the prompt, the model fills the gap with something plausible, and the output looks fine to everyone who reads it. A contract test writes the agreement down and checks it against the sandbox on a schedule, so a renamed field shows up as a failed test the same day.

YAML

contract: order_lookup
source: order system sandbox
checked: nightly, and on every release
expects:
  order_id: string, required
  status: one of [placed, shipped, partially_shipped, delivered, cancelled]
  promised_by: date with time zone, required
  carrier_estimate: may be empty
on_failure:
  not_found: 404 with an error code, never an empty order
  rate_limited: 429 with a retry-after time
  slow: no answer in 2 seconds counts as a failed call
on_unknown_status: fail the check and alert the owner
owner: support assistant team

A made-up contract for the order lookup connection, including what failure looks like.

Watch out

A new value in a status field is the classic silent break. The code does not crash, the model sees a word it was never told about, and it guesses what the word means.

Check 2. Messy data

Real records are incomplete, stale, duplicated, and strange. Development data is usually none of those, because someone wrote it by hand to make a feature work. And an AI tool rarely owns its data. It reads from systems other teams run and change.

Each of those systems has its own idea of a client record. Integration is where you learn that a field you counted on is often empty, or that a nightly sync lands after your job has run. Neither is a model problem, but both change what the model sees.

Healthcare adds one more rule. Copying production data into a test environment is the quickest way to make it realistic, and the quickest way to put patient data somewhere with weaker controls. Lower environments get synthetic data, with the mess built in on purpose, like these records.

An order split across two shipments, one late and one on time.
The same customer twice, under two spellings of one name.
A customer name with an apostrophe, an accent, or a single word.
A free-text note where someone typed a card number even though the form said not to.
A refund requested on an order that was already refunded.

If every synthetic record is clean, your integration tests are measuring the demo again.

Check 3. Permissions

The one test account with every permission turned on is the most flattering part of the sandbox. Everything works, because nothing is ever refused. The account production uses is narrower, and the first time a build meets it is often the day something breaks.

So integration gets a service account with the same narrow access production will grant. I give each agent only the access its job requires, so a mistake stays small. Test both sides of that rule, what the tool may do and what it must be refused.

Permission tests for the support assistant's service account (a made-up example)
What the assistant tries	Expected result in integration
Read the order for the customer in this chat	Allowed
Read another customer's orders	Refused, and the refusal is logged
Issue a shipping refund under the limit	Allowed, through the refund tool only
Change a customer's address	Refused, because a person does this
Search the CRM notes of every customer at once	Refused, because it reads one customer at a time

Check 4. Secrets

Part 6 covered where secrets live. At the integration step the question is narrower. Does each environment have its own credentials, kept in one managed place, and does a key from one environment fail everywhere else?

Integration has its own key for every outside system, issued by that system's sandbox or test mode.
No production key can be read from integration, by people or by the pipeline.
A key copied from integration is refused in production, and someone has watched it be refused.
Sandbox keys are renewed in the same managed place, so nobody keeps a spare copy in a chat message.

Check 5. Failure paths

Timeouts, rate limits, partial answers, duplicate events, and services that are down will all happen in production. Integration is where you make them happen on purpose, while the cost of finding out is a failed test.

Failures to cause in integration, and what passing looks like
Failure	How to cause it	What passing looks like
Timeout	Slow the sandbox down, or point the call at a stand-in that never answers	One retry, then a clear "I can't see your order right now" message
Rate limit	Send a burst of requests at the sandbox	The tool waits as long as the provider asks and never retries in a tight loop
Partial answer	Return an order with no carrier estimate	The blank goes to a person, and the model never treats it as a fact
Duplicate event	Send the same webhook twice	One reply and one refund, with the second event logged and skipped
Service down	Switch the sandbox off	The assistant says what it cannot do right now and hands off to a person

The duplicate webhook is the one that costs money. Every step should be safe to run twice, which in practice means every event carries an ID and the step checks it before acting.

TEXT

on event: refund_requested
key: event_id
before acting, look up event_id in processed_events
  found: log "duplicate, skipped" and stop
  not found: issue the refund, then record event_id with the refund ID
           in the same transaction

A made-up rule that makes the refund step safe to run twice.

A call that answers in one second for one user can take three under load.

Some connections are too risky to point at anything real below production, such as refunds, payments, and outgoing email. Use the provider's test mode, an email sink, or a simulated step that returns the expected result without doing anything.

Check 6. Logging

Every outside call should be recorded, so any failure can be replayed later. Part 6 described the log for a whole request. This one is narrower. It is one line per call to another system, with enough detail to send the same call again once the fix is in.

JSON

{
  "request_id": "req-91d2",
  "env": "integration",
  "system": "crm",
  "operation": "get_customer",
  "attempt": 2,
  "status": 429,
  "retry_after_s": 4,
  "latency_ms": 3120,
  "contract": "crm_customer v3",
  "contract_check": "failed: tier is a nested object",
  "response": "stored, masked, kept 14 days"
}

A made-up log line for one call to the CRM.

That line says which system answered, how slowly, on which attempt, and whether the answer matched the contract. Keep the masked response as well. A failure you can replay against the fix is a failure you can prove is gone.

One pipeline through every environment

In a DevOps role, I replaced a slow manual deployment with an automated one. Beyond the time it saves, a pipeline changes what integration testing can tell you.

When every change reaches each environment the same way, a failure in integration points at the change, not at whoever deployed it by hand. For AI tools, the pipeline is also where the eval cases from Part 7 rerun against the real model provider, so a prompt edit gets the same scrutiny as a code edit.

What the integration step pays back

The payoff shows up in two kinds of value. Green dollars, the savings that hit the budget, come from less rework, fewer failed releases, lower support costs, and faster launches. Blue dollars, the capacity you get back, come from steadier operations, more confidence to scale, and fewer hours lost to firefighting. Parts 14 and 15 cover how to measure each one.

Score your integration environment

Score the six checks, plus the pipeline that carries a build through every environment, 0, 1, or 2 each. Anything under 2 is a surprise waiting for user testing to find it.

Before the build moves on to staging

Tick each box you can say yes to.

0 of 9 done

Every source system is connected through the same path production will use.
Each connection has a contract test that covers failures, runs on a schedule, and has an owner.
Synthetic data includes incomplete, stale, duplicated, and strange records on purpose.
Lower environments hold synthetic data only, with no real patient or personal data anywhere, including the logs.
The service account has production-level access, and tests confirm what it must be refused.
Timeouts, rate limits, partial answers, duplicate events, and outages have each been caused on purpose.
Every step that acts on another system is safe to run twice.
Every outside call is logged with enough detail to replay it.
The eval cases from Part 7 pass against the real model provider, deployed by the pipeline that will deploy to production.

From working connections to working people

Getting two systems to talk is the easy part. Keeping the workflow reliable when those systems misbehave is the real work. A build that passes the six checks can live with its neighbors. Whether the people who own the workflow can live with it is still open, and Part 10 puts the build in front of them.

Questions people ask about integration environments

What is the difference between an integration environment and staging?

Integration is where a build first connects to real APIs, databases, and tools under control, using their sandboxes and synthetic records. Its job is to expose shape changes, slow calls, rate limits, and duplicate events. Staging comes after and behaves like production, with production settings and realistic synthetic data, so it catches settings that differ from production and mistakes in the deployment itself.

How do you test an AI agent against real APIs without risking real data?

Connect it to each system's sandbox or test mode with synthetic records, and give it a service account with the same narrow access production will grant. For risky connections such as refunds, payments, and outgoing email, use the provider's test mode, an email sink that catches every message, or a simulated step that returns the expected result without doing anything.

What should a contract test check for an API an AI tool depends on?

It should check what the system expects, what it returns, and what it does when something fails, such as a missing record, a rate limit, or a slow answer. Run it against the sandbox on a schedule and on every release, with a named owner. A new status value or renamed field should fail the check before the model starts guessing what it means.

How do you stop a duplicate webhook from triggering an AI action twice?

Make every step that acts on another system safe to run twice. Give each event an ID, and have the step look up that ID before it acts. If the event was already processed, log it as a duplicate and stop. If not, take the action and record the ID with its result in the same transaction, so a refund is never issued twice.

Should test environments use copies of production data?

No, especially in healthcare. Copying production data is the quickest way to make a test environment realistic and the quickest way to put patient data somewhere with weaker controls. Use synthetic data with the awkward cases built in on purpose: split orders, duplicate customers, odd names, stale records, and a stray card number in a free-text note.