Pull Request bots with Azure DevOps

Adding bespoke functionality to your Azure DevOps CI solution

"Why can't I do [x] with Azure DevOps?"

With the best will in the world, CI/CD systems such as Azure DevOps and github, among others, can't always cater for every feature everybody wants.

If you're a hobbyist or otherwise in complete control of a single project it's an obvious (if not entirely painless) choice to use a different system that offers you the functionality you need.

But what if you're locked in to the system you're on due to a choice by your employer, or some other integration that you're relying on?

That's the problem faced to us recently whereby a not-so-straightforward setup of CI pipelines resulted in a cumbersome experience when queueing Integration Test pipelines against build output artefacts which were generated from a Pull Request in our main product.

For simplicity's sake, this is a rough overview of what we have

(Very) Simplified overview of our CI setup

Basically, our main branch in the main product git repo will not accept new commits other than when it's come in from a Pull Request.

The Pull Request configuration for that branch has various quality gates configured on it, but the main thing of interest here is that we do build that Pull Request. Doing this allows us to test the result of building the proposed change, merged with main at the time of the build, which is the build we would get should we merge the changes. Testing in this way allows us to fail fast, and increase our confidence that we have not left main in a state that would be unreleasable.

The next thing to note is that we have a separate repository for our System Integration tests. There are reasons for this which aren't really relevant for this blog post. The setup we have here is that we can queue the System Integration Test pipeline by telling it the build number of the main product which we want to test, this pipeline will then retrieve the correct build artefacts, deploy it to the test hosts and run all of the integration tests.

So the question was raised, can we use a bot to orchestrate this for us? The immediate answer was "maybe but I don't know how to", because all of our build agents run on private networks and are not accessible via the internet. This meant we couldn't run a jenkins (or equivalent) instance on a server in our farm, and setup an Azure DevOps Service Connection to post events to it.

However, Azure DevOps does now allow you to setup an "Incoming WebHook", which pipelines can consume to use as a trigger.

Setting up the WebHooks

In your Azure DevOps project, click the "⚙️ Settings" link in the lower left portion of the view. From the settings page, first click the "Service connections" option under the heading Pipelines. Then click to Create a new service connection, and choose Incoming WebHook from the list of options. Click next, and then fill in the pane with a chosen WebHook Name. When I do this, I provide the same value for the Service connection name. (TODO: Provide info about the Secret)

Click save.

Still in the Settings section for your project, you now want to set up some Service Hooks to call in to the Web Hook you just created.

Cilck "Servce hooks" under the General heading and then click "Create subscription". Select "Web Hooks" from the list of options and click Next.

For my purposes I created two service subscriptions, one for "Pull request created" and another for "Pull request commented on"

The URL to post to should be in the format

https://dev.azure.com/{organizationName}/_apis/public/distributedtask/webhooks/{WebHookName}?api-version=6.0-preview

So what now?

Now you've done that. Your WebHook will start to receive events. It doesn't matter that you're not consuming the events yet. Once some events have occurred that will have triggered the Service Hooks, you will be able to see the logs of them by clicking the burger button and clicking History

You will then be able to inspect the payload data, and review which items are relevant for what you want to achieve. The yaml pipeline example below shows how to access the value from the WebHook payload.

How do I use it in a pipeline then?

Here is a very simple pipeline which will consume both of the service connections as part of a single pipeline. It determines which tasks to run based on the eventType member of the json payload.

This pipeline will update the original description of the PR (based on the Pull Request created service connection), with some basic text instructing viewers on how to trigger the bot to perform some work.

trigger: none

resources:
  webhooks:
    - webhook: ExampleWebHook
      connection: ExampleWebHook

pool:
  name: NameOfYourAgentPool

# Because the pullRequestId needs to come from different
# json paths when it's a comment or new PR, it's easiest
# to have the tasks which deal with the relevant job type
# retrieve the value and then update the variable, so that
# other jobs which you may want to call for both incoming
# payload types have a uniform way of accessing the value
# For bash, it is passed through capitlized, ie: ${PULLREQUESTID}
variables:
  isNew: $[eq( '${{ parameters.ExampleWebHook.eventType}}', 'git.pullrequest.created')]
  isComment: $[eq( '${{ parameters.ExampleWebHook.eventType}}', 'ms.vss-code.git-pullrequest-comment-event')]
  updatePrDescription: false
  runAutomation: false
  pullRequestId: ""
  updateComment: false

# Unfortunately using the azure-cli command to post updates to the
# description of the PR does not work unless you run the command
# from within a git repo that is relevant for the PR
# I don't understand why - because you can retrieve the info for the
# id fine, you just can't update it 🤷🏻‍♂️
# Perhaps you would work around this restraint by using
# az devops invoke syntax instead and specifying the correct
# values.
steps:
- checkout: self
- checkout: git://WebHookExample/my_app

# Obviously, you need to ensure that your build agents have
# azure-cli installed, and has the azure-devops extension
- script: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject) --use-git-aliases false
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

# values can be be fished out of the posted JSON using template expression
# syntax
# ie, ${{ parameters.ExampleWebHook.resource.comment.content }}
# Trying to get at the whole json object with
#      ${{convertToJson(parameters.ExampleWebHook)}}
# unfortunately gives us output without correct formatting so
# we can't put it through jq
# the result of ${{convertToJson(...)}} appears to give a json
# document where string values are not correctly surrounded by quotes
- task: Bash@3
  displayName: New Pull Request
  condition: and(succeeded(), eq(variables.isNew, true))
  inputs:
    targetType: 'inline'
    script: |
      # Don't do much here, just set the pullRequestId as a variable
      # accessible to other tasks, and turn on the bool which causes
      # the task to run which will update the PR description
      echo 'New PR was created'
      echo '${{ parameters.ExampleWebHook.resource.title }}'
      echo "##vso[task.setvariable variable=updatePrDescription]true"
      echo "##vso[task.setvariable variable=pullRequestId]${{ parameters.ExampleWebHook.resource.pullRequestId }}"
        
- task: Bash@3
  displayName: New Comment Added
  condition: and(succeeded(), eq(variables.isComment, true))
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/ExampleWebHook'
    script: |
      set -e

      readonly comment='${{ parameters.ExampleWebHook.resource.comment.content }}'
      commands=()

      echo "New comment was added: ${comment}"
      if [[ "${comment}" == '!'* ]]; then
        read -r -a commands <<< "${comment}"
      else
        exit 0
      fi

      if [ ${#commands[@]} -eq 0 ]; then
        echo "No commands found, we shouldn't get here" >&2
        exit 1
      fi

      # As well as triggering automation against a PR build, there
      # may also be an automation/system integration test branch which 
      # contains improvements related to the PR
      if [[ ${commands[0]} == '!testbranch' ]]; then
        git checkout "${commands[1]}"
        echo '##vso[task.setvariable variable=updatePrDescription]true'
        echo '##vso[task.setvariable variable=updateComment]true'
        echo '##vso[task.setvariable variable=pullRequestId]${{ parameters.ExampleWebHook.resource.pullRequest.pullRequestId }}'
      fi

      if [[ ${commands[0]} == '!runautomation' ]]; then
        echo "We need to run automation!"
        echo '##vso[task.setvariable variable=runAutomation]true'
        echo '##vso[task.setvariable variable=updateComment]true'
        echo '##vso[task.setvariable variable=pullRequestId]${{ parameters.ExampleWebHook.resource.pullRequest.pullRequestId }}'
      fi
- task: Bash@3
  displayName: Run Automation
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
  condition: and(succeeded(), eq(variables.runAutomation, true))
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/ExampleWebHook'
    script: |
      # This task doesn't actually run anything, for example purposes it just parses the selected options
      # that the user checked/unchecked in the PR Description before commenting !runautomation
      current_description=$(az repos pr show --id ${PULLREQUESTID} | jq -cr '.description')

      on_off() {
        case $1 in
          "[x]")
            echo true
            return 0
        esac
        echo false
        return 1
      }

      # We're not interested in any of the content of the desciption field until
      # we find one of the header lines we appended to the end of the original
      # description when the PR was created
      state="description"

      while IFS= read -r line; do
        # When you untick a markdown option in the description, instead of
        # setting the data as "- [] Option", it puts a space between the square
        # brackets which makes parsing annoying, we we deliberately replace
        # it with the variable expansion when splitting the line in to tokens
        read -r -a tokens <<<  "${line//"[ ]"/[]}"

        if [[ "${line}" != "##"* ]]; then
          case $state in
            "features")
              echo "Feature: ${tokens[2]} is $(on_off "${tokens[1]}")"
              ;;
            "hosts")
              echo "Host: ${tokens[2]} is $(on_off "${tokens[1]}")"
              ;;
          esac
        fi

        case $line in
          "##### Automation branch:"*)
            echo "Running for branch: ${tokens[4]}"
            ;;
          "##### Features")
            state="features"
            ;;
          "##### Hosts")
            state="hosts"
            ;;
        esac
      done <<< "${current_description}"
- task: Bash@3
  displayName: Update Pull Request Comment
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
  condition: and(succeeded(), eq(variables.updateComment, true))
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/ExampleWebHook'
    script: |
      # Annoyingly, when the webhook is told about a comment being
      # posted, we don't get told the thread id, so we have to fish
      # it out for ourselves from the _links section href, it is the
      # last element in the url so we can use basename to strip the
      # prefix off for us
      readonly threadId="$(basename "${{ parameters.ExampleWebHook.resource.comment._links.threads.href }}")"
      readonly repositoryId="${{ parameters.ExampleWebHook.resource.pullRequest.repository.id }}"
      readonly commentId="${{ parameters.ExampleWebHook.resource.comment.id }}"

      postdata=$(mktemp)
      cleanup() {
        rm -f "${postdata}"
      }
      trap cleanup EXIT

      # You will probably want to have something a little more informative than
      # just "Done it" as the comment which is posted
      cat > "${postdata}" <<EOD
      {
        "content": "Done it",
        "parentCommentId": ${commentId},
        "commentType": "system"
      }
      EOD

      az devops invoke --area git --resource pullRequestThreads --http-method POST --api-version 6.0 \
        --route-parameters pullRequestId=${PULLREQUESTID} repositoryId=${repositoryId} threadId=${threadId}/comments \
        --in-file "${postdata}"

      # When we've posted the comment, we can also immediately Resolve the
      # poster's comment, to present "All comments must be resolved" quality
      # gate preventing PR completion
      cat > "${postdata}" <<EOD
      {
        "status": "fixed"
      }
      EOD

      az devops invoke --area git --resource pullRequestThreads --http-method PATCH --api-version 6.0 \
        --route-parameters pullRequestId=${PULLREQUESTID} repositoryId=${repositoryId} threadId=${threadId} \
        --in-file "${postdata}"
- task: Bash@3
  displayName: Update Pull Request Description
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
  condition: and(succeeded(), eq(variables.updatePrDescription, true))
  inputs:
    targetType: 'inline'
    workingDirectory: '$(Build.SourcesDirectory)/ExampleWebHook'
    script: |
      set -e
      readonly automation_hosts=("Example Host 1" "Example Host 2" "Example Host 3")
      readonly current_branch=$(git rev-parse --abbrev-ref HEAD)

      current_description=$(az repos pr show --id ${PULLREQUESTID} | jq -cr '.description')

      new_description_lines=()
      while IFS= read -r line; do
        if [[ "$line" == *"#### Automation Settings"* ]]; then
          # We're going to rebuild the automation settings
          break
        fi
        new_description_lines+=("$line")
      done <<< "${current_description}"

      new_description_lines+=("#### Automation Settings")

      new_description_lines+=("##### Automation branch: ${current_branch}")
      new_description_lines+=("Use command '!testbranch <name>' to switch")

      new_description_lines+=("##### Features")

      # Our repo which is hosting this pipeline is our System Integration tests
      # repo, so we can simply list the files that exist in our Gherkin features
      # folder, and default them all to be set to on.  Users may untick any required
      # before triggering a run with the !runautomation comment
      for feature in features/*.feature; do
        new_description_lines+=("- [x] $feature")
      done

      new_description_lines+=("##### Hosts")

      for host in "${automation_hosts[@]}"; do
        new_description_lines+=("- [x] ${host}")
      done

      # This is really annoying. If you run a command to update a PR from within
      # a git folder, it can only find pull requests for this specific repo, not
      # a PR for a different repo. *shrug*
      pushd "$(Build.SourcesDirectory)/my_app"
      az repos pr update --id ${PULLREQUESTID} --description "${new_description_lines[@]}"

Final steps

When your pipeline such as the above runs, you may need to grant it permission in Azure DevOps to checkout your additional repositories as required.

You will also need to grant the build service account permission to contribute to Pull Requests.

To do this, go to your Project Settings and select the Repositories section. Click on your repo which is getting the Pull Requests, and then click Permissions.

In here, select the "{project name} Build Service" account, and change "Contribute to pull requests" from "Not set" to "Allowed"

I hope you find this useful!