Field sales software has an interesting deployment constraint that most SaaS products don't: our customers' most critical usage happens during business hours, Monday through Friday, when reps are in the field and managers are watching dashboards. A deployment that causes a two-minute degradation at 10am on a Tuesday is genuinely disruptive in a way that a 10pm degradation isn't.
We ship multiple times a day and we have a two-engineer on-call rotation. This is how we've made that sustainable.
The foundation: feature flags over big deployments
The single most impactful change we made to our deployment culture was committing to feature flags for any change that touches user-visible behaviour. New features go out dark (deployed to production but not enabled for any users) and are turned on gradually: internal team first, then a small percentage of accounts, then broader rollout.
This decouples deployment from release. The deployment risk (does the code work in production?) is separated from the release risk (do users experience something unexpected?). A bad deployment with a feature flag active for one percent of users is annoying. The same deployment with full rollout is an incident.
We use LaunchDarkly for flag management. It's not cheap but the time it saves in incident response is significant. When something unexpected happens after a release, the first action is to kill the flag, not to roll back a deployment.
Blue-green deployments for zero downtime
Our production environment runs two identical sets of application servers. Blue and green. At any given time, one set is live and one is idle. When we deploy, we deploy to the idle set, run smoke tests against it, and then shift traffic over using our load balancer. If something goes wrong, the shift back to the previous environment takes about thirty seconds.
The tricky part is database migrations. A blue-green deployment only works if both versions of the application can run against the same database schema simultaneously. Which means you can't ship a migration that removes a column the old version still reads from. We follow a three-phase approach:
- Deploy the new code that handles both old and new schema
- Run the migration
- Deploy the cleanup that removes the backwards-compatibility layer
This adds a deployment to every schema change but removes the class of incidents where a migration and an application change interact badly in production.
Observability before deployment, not after
Our deployment pipeline blocks on health checks, but health checks only tell you the service is running. Not that it's running correctly. We instrument every meaningful operation with structured logs and we have dashboards that track our key metrics (visit log latency, API error rate, dashboard load time) in real time.
When a deployment goes out, whoever deployed it watches the dashboards for ten minutes. Not because we expect something to go wrong (most deployments are completely uneventful) but because ten minutes of human attention immediately post-deploy is the cheapest possible form of incident detection. Automated alerting has a lag. A person watching a graph notices a trend starting.
The on-call rotation and why it's sustainable
Two engineers on call in rotation sounds brutal. It's sustainable because we've invested heavily in making incidents rare and short. When an incident does happen, the runbook for the ten most common failure modes is on the first page of our internal wiki. Not because incidents are common, but because 3am is not the time to be figuring out how to fail over the database from scratch.
We also have a strict policy: any incident that requires more than thirty minutes to resolve gets a blameless post-mortem the following week, and the output of that post-mortem is always a change to the system (better alerting, a runbook entry, a guard rail in the deployment pipeline) not a change to a person's behaviour. Systems are auditable. "Be more careful" is not.
The result is an on-call rotation where the average week involves zero pages. Not because we don't deploy often, but because the investment in reliability tooling compounds over time.