Shifting Left On A Shoestring: Azure DevSecOps Pipelines

It’s generally accepted that DevOps is an effective means of delivering higher quality software products at a faster rate, compared to more traditional software development and IT infrastructure management approaches.

For some organisations, however, it can be difficult to see a clear, pain-free path out of an environment characterised by waterfall lifecycles and on-premise infrastructure which once served them well, but which has now become an encumbrance and an impediment to progress.

Table of Contents

Background

The underlying belief may be that the cost and upheaval incurred by a migration to DevOps outweighs any potential benefits to such a degree, that “carrying on as before” seems to be better option.

The problem becomes critical, however, when the issue of software application security is brought into the mix. Bad actors are becoming increasingly resourceful in finding new ways to read, steal, alter and destroy information of value, and although there are are multitude of ways that an organisations defences can be breached, software application flaws introduced by bad coding practices can increase the risk of an attack leading to a successful security breach.

As a result has become incumbent upon those of us who develop software to adopt practices which help, detect and correct such vulnerabilities before they reach production, and ideally prevent occurring in the first place.

Shifting Left

The danger is that security testing is viewed as a bolt-on after-thought, conducted outside of the main development process by staff with limited understanding of the software development process itself. Under these circumstances its easy to see how security and development teams might end up working antagonistically – not only failing to effectively manage security risk, but potentially even slowing down the delivery of story point functionality, reducing overall product quality and increasing costs.

The evolution of DevOps into its security-conscious descendent DevSecOps is an attempt to combat this. The result is in an approach which allows security requirements to be afforded the same level of importance as functional and other non-functional requirements (eg scalability, safety, portability, etc) by ensuring that security testing is firmly embedded as integral part of the CI/CD pipeline.

The UK Government’s National Cyber Security Centre (NCSC) advocates a security focused approach to software development which aligns with the principles promoted by DevSecOps. This support is echoed in the whitepaper “Mitigating the Risk of Software Vulnerabilities by Adopting a Secure Software Development Framework (SSDF)” published by the US National Institute of Standards and Technology (NIST), identifies a number of key DevSecOps practices which organisations are advised to adopt to ensure security is embedded early in entire development lifecycle:

– Ensure that security requirements for software development are known at all times so that they can be taken into account throughout the SDLC.

– Identify and evaluate the applicable security requirements for the software’s design; determine what security risks the software is likely to face during production operation and how those risks should be mitigated by the software’s design.

– Ensure that the software resulting from the SDLC meets the organization’s expectations by defining criteria for checking the software’s security during development.

Pentesting – No Silver Bullet

It’s not an uncommon approach for organisations who have yet to embrace CI/CD to commission a penetration test when development of a system is complete (sometimes even after it is been deployed to production). Although post-development penetration testing is probably better than no security testing at all, there are a number of reasons why this approach might be considered flawed:

  • Because penetration tests are typically conducted just before (or sometimes after) the system goes into production, the cost of fixing any issues found is likely to be considerably higher compared to fixing them earlier in the development cycle.
  • Test results represent a single snapshot of the system’s security status in time – a vulnerability introduced the day after the test is conducted could be entrenched for months before the next planned test.
  • Tests are often conducted by security consultants more familiar with IT infrastructure than software development, making the process of translating test findings into actionable remediation advice for developers problematic.
  • Tests can easily be mis-scoped if the development team doesn’t have a clear idea of what information they need from the test. It’s all too easy to end up with a test report which enumerates all the security misconfigurations within the host system’s infrastructure, but reveals very little about the types of vulnerabilities found through application security testing, such as broken authentication mechanisms, command or query injection, or unsafe object deserialisation.

Rather than discounting the value of penetration testing however, perhaps it makes sense to consider repurposing it, to serve more as a means of verifying the efficacy of a development process which actively seeks to “shift security left“.

To be truly effective however, this may mean a shift to a more black-box focused testing approach. Pen pen testers with access to development artefacts such source code, design documents, architecture analysis results, misuse and abuse cases, code review results and deployment configurations, coupled with an “attacker mindset” are more likely to identify the obscure edge case vulnerabilities that persistent, stealthy threat actors are becoming increasingly adept at finding and exploiting.


Objectives

The objective of the exercise detailed in this post then, is to attempt to answer the question: How easy is it to stand up a proof of concept demonstrator, to help interested but hesitant stake holders understand why and how adopting a cloud native, continuous integration/continuous delivery DevSecOps pipeline is a viable approach to reducing risk* and improving quality, without negatively impacting speed of delivery ?

*How do we define risk ?

  • IT organisations deal with information – i.e. facts and figures which have meaning in a specific context.
  • Information within an organisation is typically viewed as a collection of assets, where each asset has a value which reflects its importance to the organisation.
  • This value can be derived from many factors, but is generally proportional to the seriousness of sustaining a degradation to one of three properties used to express the security state an information asset, namely: Confidentiality, Integrity and Availability (or “CIA“).
  • These properties can be degraded through a variety of possible scenarios involving potential threats, each of which needs to be assessed to estimate the likelihood of it occurring, via a processes such threat modelling.
  • Information systems can contain flaws, or vulnerabilities, which can adversely affect the CIA of the information assets it manipulates.
  • The severity of risk is assessed by considering how likely it is that a threat will exploit a vulnerability and what the impact could be given the value of affected information asset.
  • Vulnerabilities present in software systems expose the organisation to risk through a variety of impact types, including:
    • Financial loss
    • Operational capacity reduction
    • Legal or regulatory penalties
    • Reputational damage
    • Harm or damage to persons and property.

For more information on the relationship between software vulnerabilities and business risk, please have a read of my post: “Software Vulnerabilities and Business Risk” which outlines the concept with a case study using Java insecure deserialisation as an example.

Viability

Approaching this from the point of view of stakeholders, and queries and concerns that might be expressed when considering a shift to a DevSecOps approach:

Bottleneck: will it slow the development process process down ?
Feedback: will it allow developers to find out about security issues soon after they’ve been introduced, when remediation is comparatively easier and cheaper ?
Constructive: will it provide information that can be understood by a development audience so it can be acted upon ?
Overhead: how much additional workload will it cause for the development team ?
Scalable: how easy is to change, adapt, grow / shrink ?
Experimentation: learning leads to innovation: developers are problem solvers – does it give developers the freedom try out different ways to fix security bugs, quickly, in an isolated environment without fear of breaking the release process, learning from mistakes without fear of retribution ?
Costs: What are the initial costs ? What does the ongoing spend profile looks like – is it easy to quantify and predict ?
Metrics: how easy is it to assess and report on the level of business risk exposure from the results generated from security testing in the pipeline ?
ROI: How can the organisation’s return on investment be measured using the results generated ?


Approach

Azure DevOps

Note: The decision to use Microsoft Azure as the cloud platform for this exercise is purely arbitrary. There is no intention to present an evaluation of/comparison between the services offered by each of the various cloud providers. The assumption is that it would be relatively straightforward to port this architecture to Amazon Web Services, Google Cloud Platform or some other provider.

Schematic

The following image provides a pictorial overview of the overall process, and illustrates the separation between the build and release pipelines. In a production environment, it’s likely that there would be multiple development, test and live deployments but to keep things simple we have just a single live deployment here.

The Azure DevOps web interface provides a dashboard which makes it easy to switch between views of the build and release pipelines, code repositories and build artefacts. There’s also support for Scrum style development in the form of sprint issue boards, backlogs, burndown and velocity charts, etc.

Each stage of the pipeline is outlined below.

Pipeline Components

Development

I put together a simple Java Spring Boot MVC web application to serve as the target for this exercise. It contains a single “User” data entity type stored in an in-memory H2 database and provides create, read, update and delete (CRUD) functionality via an HTTP JSON API.

The project includes a suite of JUnit tests, is built using the Apache Maven build system, and is packaged into a Docker image for ease of deployment.

Apache Maven is used to build the application and manage dependencies
For simplicity, the application contains just a single entity type

To keep things simple, the database is created at start up and populated with a initial set of test data:

Initial dataset
An H2 database is created at start up

The Controller class maps API endpoints to the CRUD methods using standard Spring decorators:

Spring MVC maps API endpoint mappings

With the application running locally, the API can tested via a web browser:

Local API testing – query by ID
Local API testing – query by username

Source Code Configuration

The project is configured in a Git repository, with the remote located on GitHub:

GitHub repo

To import the source code into the Azure environment, I forked the GitHub repository to an Azure Git Repo:

Azure Git Repo

Third Party Components

When the application docker image is built, a number of required dependencies are incorporated, including Java OpenJDK, and several libraries to support secure network communications, certificate management, source code management, file compression, etc on the target operating system.

The Maven build process parses the application’s Project Object Model (POM) file which defines the dependencies needed to build the application as a docker container.

Build Pipeline

The Azure build pipeline is defined using a YAML configuration file, which is configured in the project source code repository along with source for the project itself. The YAML file contains a sequence of jobs and tasks, defining the actions which are performed at each stage of the pipeline.

Azure DevOps provides support for generating the YAML file visually or manually – to try out both, I used the manual method for the build pipeline and the visual method for the release pipeline:

Build pipeline YAML fragment – global variables
Commit Hook

The Build Pipeline is automatically triggered when a change to a configured source file is pushed to the Git Repo:

Build run details included the hash of the commit which triggered it
Build pipeline mid execution
Build System (Maven)

The project utilises Apache Maven to manage the application’s build process and its dependencies. The POM file lists the applications dependencies – this is executed by a dedicated task in the Azure Build pipeline:

Build pipeline YAML fragment – Maven task
Unit Test

The application includes a suite of JUnit unit tests, which the Maven build process executes when the build pipeline runs:

JUnit tests
Containerisation

The application POM file references a plugin which handles the packaging of the the application into a docker container, and which is again also executed by the Maven process when the build pipeline runs:

POM file docker container build plugin
Static Application Security Testing (SAST)

Static Application Security Testing (SAST) is performed using a pre-packaged Azure Task, which exposes functionality provided by SAST-Scan” – a free, open-source source code scanner, maintained by ShiftLeft Inc. This utility is capable of analysing Java class files for evidence of known software security weaknesses, including those classified within the OWASP Top Ten and CWE Top 25.

The scanner is packaged as a docker image; the pipeline task downloads the latest version as a container runs it:

Build pipeline YAML fragment – perform SAST scan
Software Composition Analysis (SCA)

The inclusion of dependencies such as libraries, plugins, packages etc from external sources is increasingly recognised as a significant source of security issues. For example, if the component is unmaintained it may contain unpatched vulnerabilities. If the component is imported via an uncontrolled, unofficial or insecure route, a modified, malicious version of the component may be unwittingly included. Even if a legitimate route is used, failure to verify the integrity of the component may lead to so-called supply chain attacks if the component supplier has been compromised.

The issue is of sufficient severity that it is referenced by both the OWASP Top 10 and the CWE Top 25 application security weakness lists.

Software Composition Analysis (SCA) is the name given to the process of examining the third party elements which make up a complete software system for the presence of components which are known to contain published vulnerabilities.

The build pipeline is this example performs this operation using a pre-packaged Azure Task, which exposes functionality provided by Snyk Security Scan for Container Images. This is a service provided by Snyk which offers a free tier, permitting up to 100 container tests per month for an unlimited number of developers. The service requires creation of a Snyk account, which the Azure platform connects to using an API token through a service connection which configured within the Azure pipeline project.

When executed, the security scan examines the docker image and its dependencies for known vulnerable components.

Build pipeline YAML fragment – perform SCA scan

Container Registry

Once the security scans have been completed, the application docker container is pushed to the Azure Container Registry:

Build pipeline YAML fragment – push container to registry
Confirming a successful container push via the Azure Container Registry web portal

Release Pipeline

Here I’ve tried out visual generation the YAML file – the end result is the same, but the process is a little more intuitive as compared to building up the YAML file manually, especially when multiple sequential stages with their own tasks are involved.

Building up the release pipeline using the visual interface
Triggering a release

With a real system, it’s likely that there would be multiple development, test, QA, and live deployment environments, probably under the control of a release manager, depending on the size and maturity of the team. As already mentioned, this exercise uses a single live environment, and in an attempt to simulate a typical QA/sign-off scenario, I’ve chosen to trigger creation of release for deployment manually, rather than automating it.

Triggering a release initiates the release pipeline – the stages and tasks within the pipeline are outlined below:

Dynamic Application Security Testing (DAST)

The purpose of Dynamic application security testing is to identify types of vulnerabilities in web applications that can only be found while the application is actually running. It is a type of black box test because it is performed without knowledge of the internal source code or application architecture. DAST scanning can identify issues such as input/output validation weaknesses, configuration errors, susceptibility to injection attacks, etc.

In this example, I’ve chosen to use the Zed Attack Proxy (ZAP) API Scan, an open source utility developed by OWASP. This variant of ZAP is specifically tuned to scan applications which present their services via an API, so security issues such as cross site scripting which are generally more applicable to applications with a web UI are not considered.

The scanner is provided as a docker container, so the pipeline includes tasks which handle the creation and deletion of a dedicated Azure container to house the scanner on the fly. Additional support tasks convert the output from the scanner into a format that the Azure platform can parse for display.

Release pipeline DAST testing tasks – OWASP ZAP API Scanner

Although the visual builder has been used to construct this part of the pipeline, its trivial matter to inspect the YAML script if required, should any fine-tuning or debugging be required:

Release pipeline DAST testing tasks – underlying YAML file
Deployment

The final stage of the pipeline causes the application to be pushed to an Azure container instance, assigned an IP address and executed so that the API is publicly accessible via the internet (Azure’s DNS ensures that a fixed domain name can be used, despite the IP address being allocated dynamically).

Release pipeline YAML fragment – deploy to Azure container instance
Container deployed and running

Happy Path Testing

Unit Test Results

Unit test results are accessible from the web dashboard:

Results from the suite of JUnit test suite executed during the build process

SAST Security Testing Results

The ShiftLeft SAST scan generates a comprehensive set of test results grouped by vulnerability severity, which can be viewed via a dedicated page on the web portal. Vulnerabilities are tagged with a link to the specific line(s) of source code identified as the root cause.

SCA Security Testing Results

Similar to the SAST scan, the Snyk container SCA scan also generates a comprehensive set of test results grouped by vulnerability severity, which can be viewed via a dedicated page on the web portal.

Each report entry describes what the vulnerable component is, what the dependency path is and provides references to published details about the vulnerability itself:

DAST Security Testing Results

The results from the ZAP API DAST scanner are converted into a form that can be displayed in a similar manner to unit tests:

ZAP API Scan DAST Testing results
ZAP API Scan DAST Testing results

The results contain observations which relate to the fact that, in its current state, the target application does actually return an HTML error page with an HTTP 500 response code if it receives a request which is incorrectly formatted, or can’t be mapped to a method. Strictly speaking, an API wouldn’t do this, but rather than fix this behaviour, I’ve left it as-is and just treated the results as a baseline to compare subsequent results against.

Live API Testing

Now that the application is running, live functional testing can begin. Because testing against the API will involve having relatively fine control over the structure of the HTTP requests, I’ve chosen to interact with it using an interception proxy (Portswigger Burp Suite – Community Edition) rather than a web browser or a command line tool such as Curl. Burp Suite provides the ability to capture, modify and send requests with full control over the request type, header values, body content etc.

Live API functional testing – query by ID
Live API functional testing – query by username
Live API functional testing – update by ID PUT request
Live API functional testing – query by ID after update

Introducing an SQL Injection Vulnerability

Let’s now deliberately introduce an SQL injection vulnerability into the application software, to assess the sensitivity and effectiveness of the pipeline SAST and DAST security controls.

I’ve inserted an additional API method findByRole which allows the details for a User to be queried by supplying a role type as search parameter.

Instead of using Spring’s object relational mapping to handle the database query for us, I’ve hand crafted an SQL statement using input parsed from the HTTP request. The raw userRole parameter value is included in the statement without performing any data sanitisation.

Re-Testing

Updated – Live API Testing Results

First we test the new API endpoint to verify that it returns expected results when called under normal circumstances:

Testing the new API method

As a result of the dangerous coding practices used to implement the new API method, the application code is now vulnerable to SQL injection. A suitably crafted input parameter value can be used to “break out” of the string object used internally to hold the SQL statement query clause, causing the supplied value to be interpreted as an SQL command rather than a condition.

For example, the following parameter value – shown in URL encoded and decoded form using Burp Suite’s Decoder view – uses single quote characters to cause the SQL statement construction code to build a query which should cause all entries to be returned from the database, despite the request containing a non-existent userRole search term:

…the vulnerable source code fails to filter out the single quote character and as a result, the DBMS treats the string value after it as a real “OR” condition, ignores the role search term of ‘DOESNOTEXIST’ and returns all entries where ‘1’=’1′, i.e. every entry in the database:

SELECT a.id as id, a.username as username, a.role as role, a.password as password FROM User a WHERE role = 'DOESNOTEXIST' OR '1' = '1' --

If we modifying the request within the interception proxy by inserting the above payload and resending it to the application, we get the following result:

SQL Injection being used to dump all data from the database

Updated – SAST Security Testing Results

The code change responsible for introducing the SQLi vulnerability triggered the build pipeline, and so a re-run of the SAST security testing was performed.

We can see from the new results that the SQL injection vulnerability was detected by ShiftLeft’s SAST-Scan and classified with a HIGH severity rating. The offending lines of code are identified within the report item, together with links to the relevant Java source file, and an explanation of what the vulnerability is and why it is considered to be a weakness.

Updated – DAST Security Testing Results

The SQLI vulnerability I introduced is not particularly subtle – any SAST scanner should easily be able to detect an error of this nature. However, the same type of vulnerability could have been introduced using code with a more complex execution path, which might have made the flaw more difficult to detect using static analysis methods.

To simulate a scenario where the SQLi vulnerability has been missed by the SAST scan for some reason, I triggered a new release causing the DAST security scan to run and examined the results.

As the following illustration shows, the number of vulnerabilities detected by the ZAP API scanner increased from the baseline of 2, to 5:

We can see that the DAST scan has successfully detected the SQL Injection vulnerability through active testing of the API:

… the test report entry provides the details of the HTTP request that was used by ZAP to elicit a response which proved SQL injection was possible, as well as description of the vulnerability and general advice for developers tasked with engineering a fix, and for avoiding introducing this type of vulnerability in future:

Conclusions

Returning to the assessment criteria identified in the objectives at the start of this post, which outlines queries and concerns that might justifiably be expressed by stakeholders when considering a shift to a cloud-native DevSecOps approach, as a means of achieving greater focus on application security, earlier in the development lifecycle:

Bottleneck: Will it slow the development process process down ?
The automated, continuous and parallel nature of the security testing carried out within the pipeline should help to streamline the development process. Feedback on the discovery of security vulnerabilities is provided soon after introduction, rather than days, weeks or even months later perhaps when the product is much closer to release and when the development team has perhaps begun to disperse to other projects. Near-immediate security issue feedback means remediation can potentially take place within a single sprint iteration, rather than later in the project when the additional disruption is much more likely to introduce significant project delays.

Feedback: will it allow developers to find out about security issues soon after they’ve been introduced, when remediation is comparatively easier and cheaper ?

The automated, continuous nature of the security testing carried out within the pipeline whenever code changes are committed to the source repository means that security issues can be flagged up to the originating developer very quickly after its introduction. Although the pipeline constructed for this exercise is relatively simple, the architecture can easily be extended to integrate with messaging and ticket management systems, to ensure that vulnerability alerts are channelled to the correct audience and maintained throughout the remediation lifecycle.

Constructive: will it provide information that can be understood by a development audience so it can be acted upon ?

The information presented in the “Test Results” sections above demonstrate clearly that each of the security testing stages generate vulnerability reports containing entries which are informative and relevant. Crucially, each entry is supported by root cause indicators and actionable remediation advice, which helps developers who might not be familiar with software application vulnerability concepts understand why the vulnerability exists and what needs to be done to address it.

Overhead: how much additional workload will it cause for the development team ?

Arguably, the immediacy of the feedback provided by implementing continuous, automated security testing early in the development process should mean that the workload on developers is reduced in magnitude. Security issues are communicated back the originator much sooner, meaning that developers can tackle remediation when the relevant implementation details are still fresh in their minds, rather than having to recall the details at a later date – a process which inevitably takes longer and is likely to be more error prone.

None of the free/open source variants of security testing tools selected for this exercise proved to be invasive, or required any additional dependencies or modifications to be added to the application source code to support security testing.

Scalable: how easy is to change, adapt, grow / shrink ?

Personally, I found the Azure platform is well documented and highly intuitive to use. The modular nature of the YAML-based pipeline job and task definitions make the business of adding and removing functionality trivial and clear error logs simplify the troubleshooting process during pipeline development. Being script-based means that the infrastructure which defines the pipeline architecture can be snapshotted and tagged in the same way as any other software artefact . For this reason, it’s is easy to envisage how an organisation could adapt pipelines to suit their particular business requirements as needed.

Experimentation: learning leads to innovation: developers are problem solvers – does it give developers the freedom try out different ways to fix security bugs, quickly, in an isolated environment without fear of breaking the release process, learning from mistakes without fear of retribution ?

The highly configurable and scalable nature of the platform allows experimental, ad-hoc pipelines and deployment environments to be stood up and later decommissioned quickly and without affecting the main development stream. This would also be the case with a conventional, on-premise pipeline with e.g. a “Git Flow” style feature/hotfix/release branching strategy, but the point here is the ease with which pre-canned Azure tasks can be included, tested and removed – a feature which eliminates much of the time, effort and risk from “what if?” experimentation.

Costs: What are the initial costs ? What does the ongoing spend profile looks like – is it easy to quantify and predict ?

Although absolute initial and ongoing costs will obviously depend on factors such as the size of the development team, the size and complexity of the software applications, number of active projects, frequency of deployments etc, the automated cost metrics provided as standard by the Azure platform make it very easy to determine what the current accumulated costs are and what the predicted spend profile looks like over time. Breakdowns by resource help to identify exactly which of the utilised services are costing the most, and historical data allows the root cause of spikes in costs to be analysed, helping to identify and tune out inefficiencies in the pipeline.

Azure platform cost management dashboard

Metrics: how easy is it to assess and report on the level of business risk exposure from the results generated from security testing in the pipeline ?

The test results generated by the SAST, SCA and DAST pipeline stages are sufficiently detailed to provide vulnerability data input into threat modelling activities. Assuming the organisation has good understanding of the relative value its information assets, the nature and disposition of the trust boundaries that exist between them and the potential threats they may face, then the overall risk to the organisation’s business risk can be calculated using the preferred risk assessment technique. Continuous security testing means vulnerability data is kept up to date, so stakeholders can be confident that the calculated level of risk is accurate. Risk assessment data can then also be used prioritise vulnerability remediation activity, ensuring that the effort is being expended in the most effective manner.

ROI: How can the organisation’s return on investment be measured using the results generated ?

Probably the most immediate and comprehendible means of assessing the success of adopting a DevSecOps approach is by comparing the results of “before” and “after” penetration tests. There should be a measurable decrease in the number of vulnerabilities discovered late in the development process, with a corresponding increase in security issues being detected earlier in the lifecycle. If the team has an efficient time tracking process in place, it should be relatively straightforward to compute the reduction in the overall cost of remediation over time.