Continuous Integration & Continuous Delivery (CI/CD) Workflow
The X web team delivers small application changes and bug fixes up to several times a day. The CI/CD workflow involves multiple steps to deliver a change all the way from pull request to being served to customers. Where possible, the steps are automated--freeing team members to spend their time building features rather than delivering them.
Delivering web application changes
Figure 1.0: CI/CD Workflow for an application change.
Deployment steps and roles
The X web team members each have roles and responsibilities while delivering the change to customers. Developers, Software Development Engineers in Test (SDET), Quality Assurance Analysts (QA), Designers, and Product Managers are all involved. DevOps build agents also perform a critical role triggering and automating tasks that can be done without human interaction.
The following information details the purpose, responsibility, and troubleshooting for each step in the workflow.
Develop
0 Monitor, analyze, and hypothesize
1 Submit pull request
2 Deploy to developer slot
3 Review
Deliver
4 Merge
5 Deploy
6 E2E tests
7 Swap
Note: The steps start with zero because, well, we're engineers after all.
Develop
Figure 1.1: Develop
Step 0: Monitor, analyze, and hypothesize
Create a change request for an improvement or a bug based on production data analysis.
| Role | Responsibilities |
|---|---|
 |
Monitor health metrics of live site and alert team of any issues |
| Developer | Monitor production for customer or application health issues |
| SDET | Monitor production for performance and errors |
| QA | Analyze overall application quality |
| Product & Design | Analyze customer usage patterns, consider improvements, and write stories |
Step 1: Submit pull request
A developer posts a proposed code change in a pull request using a feature branch.
| Role | Responsibilities |
|---|---|
|
Run unit tests and mocked E2E tests |
| Developer | Submit code changes in a pull request with description and purpose of changes |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| Tests fail | False positive | Check for the test in the main branch to determine if there is an issue with the test itself |
| Tests fail | Legitimate failure | Run test locally to identify and fix root cause |
Step 2: Deploy to developer slot
The feature is deployed to their developer slot for other team members to review and approve.
| Role | Responsibilities |
|---|---|
|
Upload to deployment slot |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| Automation job fails on build step | Issue with dependency update or missing file | Read output and troubleshoot |
| Automation job fails on deploy step | Build agent issues or issue with infrastructure | Check status of cloud provider for current outages |
| Dev slot returns a "500: Application error" | Breaking change introduced | Run the application locally in production mode to reproduce and troubleshoot |
Step 3: Review
Quality assurance engineers validate the changes, while other developers review the code changes.
| Role | Responsibilities |
|---|---|
| Developer | Review code, suggest improvements, and approve |
| SDET | Write/update E2E tests to accommodate change |
| QA | Validate change works as expected |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| Rejected by one or more reviewer | Changes are not determined to be quality, or introduce unwanted side effects | Address all team members' concerns until approved by all reviewers |
| Reviewers suggest a significantly different approach | Not all implementation options were considered | Close pull request and start new |
Deliver
Figure 1.2: Deliver
Step 4: Merge
Upon approval, the developer merges the feature branch into the main branch.
| Role | Responsibilities |
|---|---|
|
Generate build including changes |
| Developer | Merge pull request upon approval |
Step 5: Deploy
The main branch will automatically be built and deployed to the test and stage environments.
| Role | Responsibilities |
|---|---|
|
Deploy build to test and stage environments |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| Deployment job fails | Build agent issues or issue with infrastructure | Check status of cloud provider for current outages, and rerun job when outages is resolved |
Step 6: E2E tests
The end-to-end (E2E) integrations tests will run on the stage and test environments.
| Role | Responsibilities |
|---|---|
|
Run E2E tests--transactional tests in the test environment, and non-transactional tests in stage |
| SDET | Analyze test results |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| Test(s) fail | External or environmental issus | Run same test in production to determine if the issue is external |
Step 7: Swap
The stage slot is swapped with production. The changes are live for customers.
| Role | Responsibilities |
|---|---|
|
Swap build into production |
| Developer | Monitor measurable outcome of change |
What could possibly go wrong?!
| Problem | Possible cause | Solution |
|---|---|---|
| A team member discovers a major bug in production | Inadequate test coverage | Rollback build by running swap job again to restore the previous build |
| A team member discovers a minor bug in production | Unforeseen use case | Submit a bug report for fixing in the next deployment |
Success!
The change has been delivered to the production environment and is available to customers. Analyze the outcome and repeat the process with the next change.