6.5 Continuous Integration, Continuous Delivery (CI/CD) and Test Automation
CI/CD Overview
Continuous Integration and Continuous Delivery, also known as CI/CD, is an important capability in today's software development landscape. CI/CD provides the foundation for agile, efficient, and high-quality software delivery. It addresses the need for speed, collaboration, reliability, and scalability, empowering organizations to respond quickly to market demands and deliver software with confidence. Automated testing practices are a vital part of any CI/CD capability, which we will explore in this chapter.
Continuous Integration (CI)
Continuous Integration is a software development practice where developers regularly merge their code changes into a central repository, followed by automated builds and tests. The primary goal is to detect integration issues early in the development process.
Continuous Delivery (CD)
Continuous Delivery extends CI by automating the release process. Code changes are automatically prepared for a release but are not necessarily deployed to production automatically. Typically, a release is pushed to a staging environment for further evaluation of some kind before being promoted to a production environment.
Continuous Deployment (CD+)
Continuous Deployment is an extension of Continuous Delivery, and it takes the automation a step further by automatically deploying the release candidate to production without manual intervention.
The choice between Continuous Delivery and Continuous Deployment depends on organizational policies, risk tolerance, and the level of confidence in the automated testing processes.
A Typical CI/CD Pipeline
The typical CI/CD capability consists of a pipeline of steps that together ensure the proper building, testing and delivery of any code checked in. Let’s consider a typical CI/CD pipeline.
In this example the CI/CD pipeline executes the following steps.
-
Source Code Management (SCM):
- Connects to the version control system (for example, Git) to retrieve the latest code.
-
Build Stage:
- Compiles the code and generates executable artifacts.
-
Automated Testing Stages:
- Execute various automated tests, including unit tests, integration tests, and end-to-end tests.
-
Staging Area Deployment:
- Deploy the application to a staging environment for further testing.
-
Manual Testing (if required):
- Optionally, manual testing can be incorporated at this stage for additional validation.
-
Production Deployment:
- If all tests pass successfully, the application is deployed to the production environment.
-
Monitoring and Feedback:
- Continuous monitoring helps identify issues in production, providing feedback for further improvements. These improvements become part of the next check-in to source control.
Automated Testing in CI/CD Pipelines:
Automated testing is a critical component of CI/CD pipelines, providing rapid feedback on code changes and ensuring that new features or bug fixes do not introduce regressions. The integration of automated testing in CI/CD pipelines helps maintain a consistent and reliable software delivery process. A variety of tools and techniques may be applied to this stage of the pipeline.
Let’s discuss a few of these testing stages in more detail.
Unit Tests: Unit tests are automated and executed as part of the CI/CD pipeline, often in the early stages of the pipeline. Unit tests are designed to run quickly, allowing for fast feedback on code changes. CI/CD tools may include code coverage analysis, providing insights into how much of the code is covered by unit tests.
Integration Tests: Integration tests are automated and executed as part of the CI/CD pipeline, often following unit testing. Integration tests may be performed in an environment that simulates the production environment, ensuring realistic interactions. Inside a CI/CD platform, integration tests may also make extensive use of drivers, stubs or mocks to avoid environment setup complexities. To speed up testing, some integration tests may be executed in parallel.
End-To-End Tests: E2E tests are automated and executed as part of the CI/CD pipeline, typically after integration tests. Automated tools interact with the application's user interfaces, validating different scenarios. Like integration tests, E2E tests often run in a staging environment that closely resembles the production environment. This provides a more accurate simulation of real-world conditions and user interactions. E2E tests may be also executed after deployment to ensure that the deployed version functions as expected in the production environment. To expedite testing, E2E tests may be executed in parallel.
Security Testing: Security scans are integrated into the CI/CD pipeline, both before and after commits.Early detection of security issues allows developers to address vulnerabilities during the development phase, reducing the cost and impact of fixing issues later in the pipeline. Automated security checks may serve as gatekeepers in the pipeline. This prevents insecure code from progressing to later stages of the pipeline, ensuring that only secure code reaches production. CI/CD tools may provide workflows for handling and remediating security vulnerabilities.This streamlines the process of identifying, prioritizing, and fixing vulnerabilities, ensuring a coordinated response.
Performance Testing: Automated performance testing tools may be integrated into the CI/CD pipeline to simulate and measure an application's behavior under different workloads. This may include load tests, stress tests or soak tests which were described previously. Automated performance tests may be a gatekeeper, preventing poorly performing code from being released before remediation. Performance tests are often conducted in a staging environment that closely resembles the production environment. Some organizations may perform performance testing in production using techniques like A/B testing, where a small percentage of real users are exposed to the new version to monitor its performance.
Static Code Analysis: Automated tools analyze the source code without executing it, identifying issues related to coding standards, potential bugs, and security vulnerabilities. Static analysis tools may also identify “code smells”, anti-patterns, and potentially problematic constructs in the code. Static code analyzers can also identify common security vulnerabilities, such as SQL injection, cross-site scripting (XSS), or insecure cryptographic practices. These tools are integrated into the CI/CD pipeline and automatically run during code builds or pull requests. CI/CD pipelines may be configured to fail builds if critical issues are identified by static code analysis.
Using Headless Mode within CI/CD Pipelines
Headless mode refers to running a web browser without a graphical user interface (GUI). Instead of rendering the web page visually, the browser processes the page and executes JavaScript, but it doesn't display anything on the screen. This mode is particularly useful for automated testing, especially in Continuous Integration/Continuous Deployment (CI/CD) pipelines, where speed and efficiency are crucial.
In a CI/CD pipeline, headless mode can be utilized for tests that typically require a browser window. End-to-end tests can be run in headless mode to execute tests in a browser environment without GUI. While end-to-end tests traditionally involve interacting with the UI, headless browsers such as Headless Chrome or PhantomJS allow running these tests without launching a visible browser window. Selenium WebDriver has a headless mode which allows the unit tests created to be run without a visible browser.
Some tools like Postman allow you to create scripts to do integration testing, then export these tests to another framework like JUnit or Pytest. These exported integration tests are then able to be used directly in a CI/CD framework as well without requiring the tool to be present in the build environment.
Running end-to-end tests and integration tests in headless mode also speeds up the testing process and facilitates seamless integration into CI/CD pipelines, as they can be executed on headless servers or containers without requiring a graphical environment.
Testing in Production
Testing in production is a testing approach where software changes, updates, or new features are tested directly in the live or production environment rather than in isolated testing environments. This approach allows organizations to gather real-world insights, validate performance, and identify issues that might not be apparent in testing environments. In times past, testing in production was seen as a inadvisable or lazy approach to testing. Today, testing in production is seen as a viable alternative to spinning up expensive production-like environments while still managing risk appropriately.
The techniques of testing in production, including feature flags, canary deployment, and blue/green deployment, are closely related to the principles and practices of Continuous Integration and Continuous Delivery (CI/CD). These strategies enhance the CI/CD pipeline by providing mechanisms to validate changes, manage risk, and deliver software with more agility and confidence.
Feature Flags (Feature Toggles):
-
Definition:
- Feature flags are a software development technique that allows developers to toggle certain features on or off at runtime without changing the codebase. This enables the controlled release of features to different user groups.
-
How it Works:
- Developers embed conditional statements in the code to enable or disable specific features based on the state of a feature flag.
- Feature flags are configured at runtime, either through configuration files, environment variables, or a centralized feature flag management system.
- Features can be gradually rolled out to specific user segments, regions, or percentages of users.
-
Benefits:
- Enables a controlled and gradual release of features to mitigate risks.
- Supports A/B testing by enabling or disabling features for different user groups.
- Allows for quick rollback if issues are identified, as the feature can be toggled off centrally.
Canary Deployment:
-
Definition:
- Canary deployment is a deployment strategy where a new version of the software is gradually rolled out to a small subset of users or servers before being released to the entire user base.
-
How it Works:
- The new version is deployed to a limited number of users, often starting with internal or less critical users.
- Performance, stability, and user feedback are monitored before a wider release.
-
Benefits:
- Early identification of issues, minimal impact if problems arise, and the ability to halt the rollout if necessary.
Blue/Green Deployment:
-
Definition
- Blue/green deployment involves maintaining two separate environments (blue and green), with only one active at a time. The inactive environment can be updated, tested, and switched to production with minimal downtime.
-
How it Works
- The active environment (for example, blue) serves live traffic, while the inactive environment (for example, green) is updated with new changes. Once the green environment is ready, traffic is redirected to it.
-
Benefits
- Minimal downtime during deployments, easy rollback by switching back to the previous environment, and efficient testing in a production-like environment.
Testing in Production Precautions
- Robust monitoring and observability tools must be in place for detecting issues and performance metrics during live testing.
- Have well-defined rollback plans in case issues are identified, ensuring a quick return to a stable state.
- Communicate changes transparently to users, especially if testing involves visible features or changes to user experience.
- Ensure compliance with data privacy and security regulations, especially when testing involves real user data.
Useful Links: ←Unit 6.4 | Unit 6.6→ | Table of Contents | Canvas