Detecting Attacks against Azure DevOps

By Matthew Lucas on 5 April, 2022

Matthew Lucas

5 April, 2022

This post will cover detection opportunities specific to the attack path discussed in the previous blog.

In this path, a malicious Azure Active Directory application was registered from a low privileged foothold. Through spear-phishing, this application allowed the theft of a developer's PAT token and access to the repositories and pipelines under their control. From there, malicious code was deployed and run in an Azure Pipeline, exfiltrating privileged Service Principal credentials.

Detection within Azure DevOps

First, let us set the stage for our telemetry sources within Azure DevOps. The intended method of logging activities within Azure DevOps is the built-in "Audit Logs". These can be streamed into a Log Analytics Workspace, upon which Azure Sentinel can enable alerting and threat hunting. These can also be directly streamed to a SIEM. Unfortunately, as we will see, the actual telemetry this produces is insufficient for detecting multiple elements of this attack chain. This is likely because the "Audit Log" is intended for monitoring the control plane of Azure DevOps rather than the actions performed on the contents of the repositories and pipelines, leaving many of our malicious actions outside of its traditional remit.

Whilst not intended for logging of malicious actions, Azure DevOps' "Service Hooks" functionality does cover a lot of the actions relevant to this attack chain. Primarily, this is because most of our actions so far have been near-identical to normal developer actions, which falls within the remit of these "Service Hooks". Webhooks allow Azure DevOps to send alerts to various sources such as Slack, Function Apps, or simply a generic HTTP(S) webhook. For demonstration purposes, this final option is the one we will use, sending data again to our Burp Collaborator server. In our example, we have enabled webhooks for the following events for all pipelines and repos in our project:

  • Build completed
  • Code pushed
  • Run stage state changed
  • Run state changed

Finally, as not all of the stages of our attack chain took place solely within Azure DevOps, we will also look at some of the traces left behind in Azure Active Directory.

Phishing a PAT

Our attack path begins outside of Azure DevOps, within the realm of Azure AD. When we register our application and generate the necessary Service Principal secrets, the following activities appear in Azure AD's Audit logs:

  • Add application
  • Add owner to application
  • Add service principal
  • Update service principal
  • Update application - Certificates and secrets management
  • Update application
    • [{"ResourceAppId":"[REDACTED]","RequiredAppPermissions":[{"EntitlementId":"e1fe6dd8-ba31-4d61-89e7-88639da4683d","DirectAccessGrant":false,"ImpersonationAccessGrants":[20]}],"EncodingVersion":1}]

Defenders should consider which users in your organisation are expected to be creating applications, generating secrets and DevOps API permissions for these. A compromised member of the development team may be incomparable to background noise in this regard, but a member of HR creating enterprise applications is relatively unlikely. Additionally, this stage of the attack is only possible if the Azure Active Directory User Setting “Users can register applications” is set to “Yes”. Whilst this is the case by default, setting it to “No” would force an attacker down a different path to gain initial access to an Azure DevOps organisation.

Later on, when we phish our target user, we see the following Activities in the Azure AD Audit Logs:

  • Add app role assignment grant to user
    • Target: App name
  • Consent to Application
    • Target: App name
  • Add delegated permission grant
    • Target: Azure DevOps
    • DelegatedPermissionGrant.Scope: "user_impersonation"

Finally, once the PAT token is created, the targeted user will receive an email to this effect. If the phishing context doesn't suitably explain this, this should alert the user that their DevOps access has been compromised.

Azure DevOps' "Audit Logs" claims to log the creation and changes made to any PAT tokens. In practice, it was not found to generate events for PATs created within the web portal, but did succeed in flagging PATs created through our malicious application, triggering the following activity:

  • ActionId: Token.PatCreateEvent
  • UserAgent: python-requests/2.22.0
  • Scopes: ["app_token"]

The latter two elements should not be assumed to be present in all possible attack scenarios, as it would be relatively trivial for an attacker to alter the "User Agent" of the app and reduce the scope to avoid arousing suspicion.

Targeting the Pipeline

The first half of our attack in this section involves simply pulling, editing, and pushing code into a new branch. "Audit Logs" were found to be completely silent on any of these actions, although this is likely because these are beyond the scope of what it keeps track of. Unfortunately, since "Audit Logs" are the only log source intended for direct streaming to a Log Analytics Workspace or SIEM, this means none of these solutions would receive any information about these actions. The primary useful source of telemetry in fact was found to be the Azure DevOps webhooks. Any push to any repo resulted in an HTTP record sent to our server, as shown below:

POST / HTTP/1.1
X-VSS-ActivityId: [REDACTED]
X-VSS-SubscriptionId: [REDACTED]
User-Agent: VSServices/16.195.32010.8 (w3wp.exe)
Content-Type: application/json; charset=utf-8
Host: [REDACTED].burp.[REDACTED]
Request-Context: appId=cid-v1:[REDACTED]
Request-Id: [REDACTED]
Content-Length: 5600
Expect: 100-continue
Connection: Keep-Alive

{"subscriptionId":"[REDACTED]",
"notificationId":2,
"id":"d645caa2-0cbe-432c-9db7-423fdcb7dd4a",
"eventType":"git.push",
"publisherId":"tfs",
"message":{"text":"Matthew Lucas pushed updates to incomplete-branch-protection:malicious-branch\r\n(https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/#version=GBmalicious-branch)",
"html":"Matthew Lucas pushed updates to <a href=\"https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/\">incomplete-branch-protection</a>:<a href=\"https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/#version=GBmalicious-branch\">malicious-branch</a>",
"markdown":"Matthew Lucas pushed updates to [incomplete-branch-protection](https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/):[malicious-branch](https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/#version=GBmalicious-branch)"},
"detailedMessage":{"text":"Matthew Lucas pushed a commit to incomplete-branch-protection:malicious-branch\r\n - another commit 41ddb1e6 (https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/commit/41ddb1e67899395896ae8ffe2662b6e25a6e6599)"...
"resource":{"commits":[{"commitId":"41ddb1e67899395896ae8ffe2662b6e25a6e6599","author":{"name":"matt","email":"[REDACTED]@outlook.com","date":"2022-01-06T16:44:37Z"},"committer":{"name":"matt","email":"[REDACTED]@outlook.com","date":"2022-01-06T16:44:37Z"},"comment":"another commit","url":"https://dev.azure.com/devopsattacks/_apis/git/repositories/1bb3c427-fee7-45c4-99b7-3e908ccf2958/commits/41ddb1e67899395896ae8ffe2662b6e25a6e6599"}],"refUpdates":[{"name":"refs/heads/malicious-branch","oldObjectId":"44753db83d69dcc4d529a4da2fda274d21420348","newObjectId":"41ddb1e67899395896ae8ffe2662b6e25a6e6599"}]...

Interestingly, this telemetry displayed an additional potential method of exposing an attacker, as it included the email address stored in git. In the case of my test environment, this did not match the target Azure DevOps environment (see 'author' on the final line).

For the second half of our attack, which involved manual running of a target Pipeline, the "Audit Logs" do provide some usable telemetry. In this instance, there was a record made of a pipeline being run:

  • Service Connection "sp-devops-logging-test" of type azurerm executed in project incomplete-branch-protection.

However, a crucial issue is that no element of this log entry includes details of who initiated the pipeline run, nor the branch the pipeline is being run from. In essence, it is assumed that the entity performing this action is the Service Principal - which is not particularly useful for identifying a compromised developer account.

Again, webhooks can fill in these important details as can be seen below, with an initial request when the pipeline begins revealing the branch that was used:

POST / HTTP/1.1
X-VSS-ActivityId: [REDACTED]
X-VSS-SubscriptionId: [REDACTED]
User-Agent: VSServices/16.195.32010.8 (w3wp.exe)
Content-Type: application/json; charset=utf-8
Host: [REDACTED].burp.[REDACTED]
Request-Context: appId=[REDACTED]
Request-Id: [REDACTED]
Content-Length: 2413
Expect: 100-continue
Connection: Keep-Alive

{"subscriptionId":"[REDACTED]",
"notificationId":4,"id":"b2b99787-6658-47f1-95c5-525de3d18c1f",
"eventType":"ms.vss-pipelines.run-state-changed-event",
"publisherId":"pipelines",
"message":{"text":"Run 20220106.7 in progress.","html":"Run <a href=\"https://dev.azure.com/devopsattacks/d6fab4fc-3907-4804-ac4b-85fb2c7f8fd9/_build/results?buildId=7\">20220106.7</a> in progress."...
...{"run":{"_links":{"self":{"href":"https://dev.azure.com/devopsattacks/d6fab4fc-3907-4804-ac4b-85fb2c7f8fd9/_apis/pipelines/1/runs/7"}..."resources":{"repositories":{"self":{"repository":{"id":"1bb3c427-fee7-45c4-99b7-3e908ccf2958","type":"azureReposGit"},"refName":"refs/heads/malicious-branch","version":"56ab81b3b0f5ed1b3140700d1f1dd3d74be000ae"}}},"id":7,"name":"20220106.7"},

And a final webhook request when our build finishes revealing the identity of the Pipeline caller:

POST / HTTP/1.1
X-VSS-ActivityId: [REDACTED]
X-VSS-SubscriptionId: [REDACTED]
User-Agent: VSServices/16.195.32010.8 (w3wp.exe)
Content-Type: application/json; charset=utf-8
Host: [REDACTED].burp.[REDACTED]
Request-Context: [REDACTED]
Request-Id: [REDACTED]
Content-Length: 5173
Expect: 100-continue

...
{"text":"Build 20220106.7 failed","html":"Build <a href=\"https://dev.azure.com/devopsattacks/web/build.aspx?pcguid=[REDACTED]&amp;builduri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f7\">20220106.7</a>
...
,"requestedFor":{"displayName":"Matthew Lucas","id":"[REDACTED]","uniqueName":"matthew.lucas@[REDACTED].onmicrosoft.com"}}]},"resourceVersion":"1.0","resourceContainers":{"collection":{"id":"[REDACTED]","baseUrl":"https://dev.azure.com/devopsattacks/"},"account":{"id":"[REDACTED]","baseUrl":"https://dev.azure.com/devopsattacks/"},"project":{"id":"[REDACTED]","baseUrl":"https://dev.azure.com/devopsattacks/"}},"createdDate":"2022-01-06T16:58:32.533708Z"}

Finally, should an attacker log in with the Service Principal, then there will be a record in the Service Principal's sign-in logs. Quite simply, there should not be a reason for anyone to directly log in as a pipeline's Service Principal so any records of this should be treated as suspicious.

Cleaning up after ourselves

The final step of our attack - an attacker attempting to cover their tracks - can in fact generate more telemetry through webhooks. As the act of deleting a branch is itself a "git push", it is unsurprising that this triggers our web hook:

POST / HTTP/1.1
X-VSS-ActivityId: [REDACTED]
X-VSS-SubscriptionId: [REDACTED]
User-Agent: VSServices/16.195.32010.8 (w3wp.exe)
Content-Type: application/json; charset=utf-8
Host: [REDACTED].burp.[REDACTED]
Request-Context: [REDACTED]
Request-Id: [REDACTED]
Content-Length: 4267
Expect: 100-continue
Connection: Keep-Alive

{"subscriptionId":"[REDACTED]",
"notificationId":5,
"id":"e768f7fb-59c3-4617-b8b9-abe38fdf33b9",
"eventType":"git.push",
"publisherId":"tfs",
"message":{"text":"Matthew Lucas deleted incomplete-branch-protection:malicious-branch\r\n(https://dev.azure.com/devopsattacks/incomplete-branch-protection/_git/incomplete-branch-protection/#version=GBmalicious-branch)",...

Deleted branches can be recovered within Azure DevOps, but only if the exact name is known and has not since been reused for a new branch. By recording branch deletions in this way, it becomes possible to view the previous deleted malicious code. In the Azure DevOps web portal, this can be done by searching for this branch in the relevant repository with the exact name, which brings up the option to restore it.

Finally, we briefly touched on the deletion of pipeline runs as only being possible within the Azure DevOps web portal. It should be noted that it also only applies within the Azure DevOps web portal as the following command shows all runs up until their expiration date, deleted or otherwise:

az pipeline runs list --project=$proj

Unfortunately, this does not let us view the output of the runs on the agent in the way that the web portal does.

Conclusion

Whilst transitions to infrastructure as code have reduced the requirement for individual users to hold dangerous privileges over a cloud environment, these privileges have had to go somewhere. In essence, developers still hold these privileges – but with some extra steps. And depending on the exact implementation of these steps, they can act either as a genuine blocker to an attacker, or just a minor complication and nuisance.

Built-in logging and detection tools in Azure DevOps prove to be a mixed bag when it comes to recording the basic telemetry to even begin to infer an attack has taken place. Through relying on multiple data sources, we can still keep an eye on just what is happening within an organisation's repositories and their pipelines. Converting this into effective detection logic remains non-trivial, as the majority of malicious actions are effectively identical to the typical day-to-day actions taken by developers to simply perform their role. Thus, organisations should put in the effort to establish the much needed context behind what actions are legitimately performed by developers on a day-to-day basis and flag any dangerous outliers that could signify a security incident had occurred or is in progress.