Building Microsoft Dynamics 365 Business Central Apps with Azure DevOps

Last time out we were discussing defining your build pipeline in a YAML file. That post was an intro to what pipelines are and the benefits of defining the tasks that it runs in a YAML file alongside your other source code. Now we’ll turn our attention to some Business Central specific considerations.

Objectives

We’re start by defining the key objectives of the build process:

  1. Download a particular version of the source AL code
  2. Create an appropriate Docker container to publish the app into and run the tests against (see below for more about what “appropriate” means in this context)
  3. Acquire the alc.exe compiler from the container and use it to compile the AL code into two apps (the main app and a dependent app that contains the tests)
  4. Acquire and install any necessary dependencies, install the main app and test app (see here)
  5. Execute the tests and export the results
  6. Upload test results, main app and test app to the build
  7. Remove the Docker container

Environment

Microsoft-hosted or self-hosted?

Microsoft give you a menu of different hosted build agents to execute your pipelines on and 1,800 minutes of build time per month for free. The obvious attraction of this option is not having to build and maintain your own infrastructure to run builds and you just pay for the time you use (assuming you exceed the free limit). The obvious downside is that you can’t prepare that environment as you’d like e.g. Docker images must be downloaded each time as part of the job. I can’t comment too much on this option as it isn’t something we’ve experimented with so far.

We host our own server that runs several build agents. The main driver for the decision at the time was that it allowed us to persist Docker images between builds (NAV images are approx. 15GB, although BC images are smaller) and save a substantial amount of time on each build.

Azure DevOps Build Agents

With smaller Docker images these days it ought to be increasingly feasible to run BC builds in a sensible amount of time.

Installing and Connecting the Build Agent

From the list of build agents (at https://dev.azure.com/<your organisation>/_settings/agentpools) you’ll see the link to Download the agent. Simply download and extract onto your build server. Run config.cmd and follow the instructions to connect the agent to your DevOps organisation.

You’ll need a Personal Access Token to authenticate. See here if you need a refresher on how to create those.

Triggers

The majority of our builds are triggered by some new code being pushed to server branch i.e. continuous integration builds. DevOps handles downloading this version of the source code to the build agent to work on. This is defined by the trigger section of the .azue-pipelines.yml file:

trigger:
  branches:
    include:
      - "*"

We are also starting to schedule more builds. This is useful for building our apps against insider builds of Business Central. Which brings me onto how we define the Docker image that we are going to build against.

Environment.json

Our apps include a json file that defines some parameters that are used by the build.

  • The Docker image to build against
  • The user name and password to create the new container with
  • Translations (country and language) that must be present in the app
  • Details of the Azure DevOps project/repo to acquire dependencies listed in app.json from (as described here)

Which Docker Image?

As a rule I develop and test against sandbox images (mcr.microsoft.com/businesscentral/sandbox). They are the closest thing to testing on SaaS that you can get without actually having a SaaS tenant. We always develop against the worldwide (W1) image and build against all of the localisations that we are planning to support.

The sandbox image has very little data in it, which is great for downloading new versions of the image and creating new containers but does mean that you have to handle more of the data setup in your tests than you would for an on-prem image. Yes, tests should be data-agnostic and run in an empty company but we still need to work around some bugs in standard library functions.

Branch per Docker Image

This approach allows us to have a separate branch for each different Docker image that we want to build our app against. We have country/xyz branches where “xyz” is the Docker tag for the localisation that we need to support i.e. country/es, country/ca, country/nz

At any moment these branches should be a single commit ahead of the feature branch we are working on, the only difference being the Docker image that is used. We can then rebase these branches on top of whichever commit we want to build. When we push those branches to the server continuous integration builds will be kicked off for each country.

PowerShell Tasks

It won’t come as much of a surprise that the majority of tasks performed by the build are PowerShell scripts. You’ve got some different options for defining these scripts:

  1. Define them in .ps1 files alongside your source code
  2. Define them in .ps1 files that are saved on the build server (assuming that you are self-hosting the agent)
  3. Maintain the scripts somewhere else and share them with the build server

We started with #2 and have recently moved onto #3. All our scripts are now bundled into a PowerShell module which is published on the PowerShell Gallery. The module is installed and updated on the build server. Maybe I’ll post some more about our approach to PowerShell development, our build process for it and testing with Pester another time.

We use inline PowerShell tasks to import our module and run a command on the source like this:

steps:
- task: PowerShell@1
  displayName: 'Create packages and execute tests'
  inputs:
    scriptType: inlineScript
    inlineScript: 'Import-Module Tecman.Tfs.Tools;Run-ALBuildProcess ''$(Build.SourcesDirectory)'' ''$(Build.ArtifactsStagingDirectory)'' $(Build.BuildID) $true'

Compiling the App

Acquiring the Compiler

If you’ve read the output from the creation of a new Docker container then you’ve probably noticed that the corresponding version of the Visual Studio Code extension is included with the container. It is hosted at http://<containername&gt;:8080/<name of vsix file>. You can get the precise URL to the file by inspecting the logs with docker logs <container name>.

Use PowerShell’s Download-File function to download the vsix to a local file. The .vsix file, like a .app file, is a archive file containing the source of the extension. You can use Expand-Archive on the file to extract the contents of the .vsix to a local folder and find alc.exe in the extracted files. You’ll need to rename the file to .zip first to convince Expand-Archive that it is a format it can expand.

function Get-CompilerFromContainer
{
    Param(
        [Parameter(Mandatory=$true)]
        [string]$ContainerName
    )

    $VsixPath = Get-VSCodeExtensionFromContainer -ContainerName $ContainerName
    if (!(Test-path "$VsixPath\Extract")){
        Rename-Item $VsixPath "$VsixPath.zip"
        Create-EmptyDirectory "$VsixPath\Extract"
        Expand-Archive -Path "$VsixPath.zip" -DestinationPath "$VsixPath\Extract"
    }
    
    "$VsixPath\Extract\extension\bin\alc.exe"
}

function Get-VSCodeExtensionFromContainer {
    Param(
        [Parameter(Mandatory=$false)]
        [string]$ContainerName = (Get-ContainerFromLaunchJson)
    )

    $Logs = docker logs $ContainerName
    $VsixUrl = $Logs.item($Logs.indexOf('Files:') + 1)
    $VsixName = (Split-Path $VsixUrl -Leaf).TrimEnd('.vsix')
    $VsixPath = Join-Path (Split-Path (Get-TFSConfigPath) -Parent) $VsixName
    $VsixFile = (Join-Path -Path $VsixPath -ChildPath $VsixName) + '.vsix'
    if (!(Test-Path $VsixPath)){
        New-Item -Path $VsixPath -ItemType Directory
        Download-File -sourceUrl $VsixUrl -destinationFile $VsixFile
    }

    $VsixFile
} 

The above includes some code to save the extracted .vsix files into the AppData folder on the build server to save us downloading and extracting a version of the VS Code extensions that we’ve already got. Over time the .vsix file has grown in size and we can save ourselves some time and disk space by reusing the copy that we’ve already extracted.

Compiling

Having got your hands on the right version of alc.exe you can run it with something like the below:

Start-Process -FilePath $CompilerPath -ArgumentList (('/project:"{0}"' -f $SourcePath),('/packagecachepath:"{0}"' -f (Join-Path $SourcePath '.alpackages')),('/assemblyProbingPaths:"{0}"' -f (Join-Path $SourcePath '.netpackages'))) -Wait

Assuming the app builds successfully you’ll see a .app file in the root of the source directory. You can now grab that app file and publish it into the container using the navcontainerhelper module.

Testing and Uploading the Results

Having created a container, got the VS Code extension and published the app (with any dependencies) it’s time to run the tests. I’ve been writing about using navcontainerhelper to execute the tests in the container quite a lot lately so I won’t go into all that again.

Suffice to say that we use navcontainerhelper to execute the tests and export the results to XUnit format. We then use the “Publish test results” task to upload those results to the build on Azure DevOps.

- task: PowerShell@1
  displayName: 'Error on test failure'
  inputs:
    scriptType: inlineScript
    inlineScript: 'Import-Module Tecman.Tfs.Tools;Error-OnTestFailure $(Build.BuildID)'

You might notice Error-OnTestFailure in that inlineScript. The purpose of that is to throw an error if any of the tests fail otherwise the build will be reported as successful, even with failed tests. I suspect setting the AzureDevOps parameter on the Run-TestsInNavContainer function is the better way to do this now though.

Uploading the App(s)

If the tests have run successfully then we can upload the app files to the build artefacts. Simply copy the app files into the artefacts directory – defined by the $(Build.ArtifactsStagingDirectory) variable and run the Publish Build Artifacts task.

- task: PublishBuildArtifacts@1
  displayName: 'Publish App Package'
  inputs:
    ArtifactName: 'App Package' 

Removing the Docker Container

Finally we’re going to remove the Docker container with a inline PowerShell script. Notice the condition property that is attached to this task. In this case we’re just defining that the task should always be run – even if an earlier task has failed. It is possible to get smarter with conditions e.g. only running certain tasks if the build has been triggered in a certain way, or from a particular branch.

- task: PowerShell@1
  displayName: 'Remove Docker build container'
  inputs:
    scriptType: inlineScript
    inlineScript: 'Import-Module Tecman.Tfs.Tools;Remove-ALBuildContainer $(Build.BuildID)'

  condition: always() 

Writing Your Own YAML Pipeline

If you’re reading this post and wondering how on earth you are supposed to know what to type into your blank .azure-pipelines.yml file then remember that the Azure Pipelines extension for VS Code give you intellisense. Just create the file and hit Ctrl+Space to see the what’s what.

Conclusion

This post has been a bit of mixed bag, a rummage through our build pipeline toolkit, but hopefully some of it has been useful. As ever, the best way to learn is to get stuck in and try it out for yourself.

Working with Azure DevOps Pipelines in YAML

Overview

This post is an update to a post I made about YAML pipelines here. We’ll also take the opportunity to discuss why you might want to define a pipeline with YAML.

Wait…What?

What the heck are we talking about? (skip this bit if you do know what we’re talking about) A pipeline defines a series of tasks, running on defined environments that are performed with your code. In Azure DevOps they come in two flavours:

  • Build – for us that means, taking our AL source code, splitting it into two (test app and production app), compiling them, signing them, publishing and installing into a new container and running the tests and saving the .app files as artefacts of the build
  • Release – taking the built software and deploying it into one or more test and/or production environments – we don’t currently use release pipelines

Pipeline as Code, Why?

Defining the steps involved in your pipeline in a YAML file is sometimes called “pipeline as code” because the YAML file is checked-in to your repository alongside your source code.

The benefit is that your pipeline is version controlled. You can view its history, compare versions, blame/annotate etc. You could also have different versions of your pipeline in different branches and include it in a pull request.

The downside is of having yet another markup language to learn. What are you supposed to put in this file anyway?

Defining the Pipeline

Let’s consider two ways of creating and maintaining your pipeline file. I’m sticking to Visual Studio Code and Azure Repos/Pipelines in Azure DevOps as that’s what I’m familiar with. Loads of other options are available, loads of them supported in Azure DevOps.

In Azure DevOps

The features in Azure DevOps and the UI change frequently as they add new stuff. Microsoft announced loads of changes, including a new YAML editing experience (below) and YAML release pipelines, at Build 2019. You can browse through and watch sessions here: https://www.microsoft.com/en-us/build search for DevOps to jump to the sessions related to this post.

I’ve got a Hello World app with the AL code hosted in Azure Repos. Let’s walk through creating the pipeline file in the UI. Select Builds from the Pipelines menu and hit the “New pipeline” button.

Choose where you want this pipeline to fetch the source code from. In my case it’s in an Azure Repos Git repository.

And I’ve only got one in this project, so I’ll select that.

I don’t have an existing pipeline file, so I’ll create a starter pipeline.

And there it is.

Great…but what does all that mean?

Firstly, this is a pretty neat editor. It works a lot like Visual Studio Code. Maybe it even is Visual Studio Code behind the web page, for all I know. You can hover over different parts of the file and get tooltips about what they do. You also get intellisense when you hit Ctrl+Space giving you some info about the valid options for this part of the file.

Briefly, this pipeline will:

  • Trigger a build when changes are pushed to the master branch
  • Run the build on a hosted ubuntu agent (this is the “we love Linux, we love open-source” Microsoft after all)
  • Run a script to echo “Hello, World!”
  • Run another script to echo some more text

Let’s save and run the pipeline. I’ll commit straight to the master branch for now.

I’m bounced over to see the build that has been scheduled and can watch it run. This is the result:

You can click into each of the steps to see the logs for that step.

In Visual Studio Code

Notice that the file created above was automatically named .azure-pipelines.yml. That is the magic name that Azure DevOps will automatically recognise as defining a pipeline. That means if you create a file with that name and push it to Azure Repos it will automatically create a pipeline using that file as the definition for you.

When I flick back to Visual Studio Code I’ve got a commit waiting to be fetched into my local repo which was created when I saved the pipeline file. Now that I’ve got .azure-pipelines.yml locally I can edit it and source control it just like anything else.

To get the same editing experience as you had online you’re going to want to grab the Azure Pipelines extension for Visual Studio Code. That will recognise that the file is a pipeline definition and give you all the intellisense and more-info goodness you had in the browser.

Further Reading

For more information about what you can do with the yml file check out: https://aka.ms/yaml otherwise I’ll follow up with something more Business Central specific in another post.

Chaining Builds in Azure DevOps

We are triggering a lot of builds in Azure DevOps these days. If anyone so much as looks at an AL file we start a new build.

OK, that’s a small exaggeration, but we do use our build pipelines for:

  • Continuous integration i.e. whenever code is pushed up to Azure DevOps we start a build
  • Verifying our apps compile and run against different localisations (more of that another time)
  • Checking that a dependent app hasn’t been broken by some changes (what we’re going to talk about now)
  • Building our app against different upcoming versions of Business Central (this is an idea that we haven’t implemented yet)

Background Reading

If you haven’t got a clue what I’m talking about you might find a little background reading useful. These might get you started:

Overview

We’re considering a similar problem to the one I wrote about in the last post on package management – but from the other end. The question then was, “how do I fetch packages (apps) that my app depends on?” Although not explicitly stated, a benefit of the package management approach is that you’ll find out pretty quickly if there are any breaking changes in the dependency that you must handle in your app.

Obviously, you want to minimise the number of times you make a breaking change in the first place but if you can’t avoid it then change the major version no. and do your best to let any dependent devs know how it will affect them e.g. if you’re going to change how an API works, give people some notice…I’m looking at you Microsoft 😉

But what if we’re developing the dependency and not the dependent app? There will be no trigger to build the dependent app and check that it still works.

Chaining Builds

Azure DevOps allows you to trigger a new build on completion of another build. In our scenario we’ve got two apps that are built from two separate Git repositories in the same Azure DevOps project. One is dependent upon the other.

It doesn’t really matter for the purposes of this post what the apps do or why they are split into two but, for the curious, the dependent app provides a little slice of extra functionality for on-prem customers that cannot be supported for SaaS. Consequently the dependency (which has the core functionality supported both for SaaS and on-prem) is developed far more frequently than the dependent app.

Build Triggers.JPG

We want to check that when we push changes to the dependency that the dependent app still works i.e. it compiles, publishes, installs and the tests still run.

You can add a “Build Completion” trigger to pipeline for the dependent app. This defines that when the dependency app is built (filtered by branch) that a build for the dependency kicks off.

That way if we’ve inadvertently made some breaking change we gives ourselves a chance to catch it before our customers do.

Limitations

Currently the triggering and to-be-triggered build pipelines must be in the same Azure DevOps project – which is a shame. I’d love to be able to trigger builds across different projects in the same organisation. No doubt this would be possible to achieve through the API – maybe I’ll attempt it some day – but I’d rather this was supported in the UI.

An Approach to Package Management in Dynamics 365 Business Central

TL;DL

We use PowerShell to call the Azure DevOps API and retrieve Build Artefacts from the last successful build of the repository/repositories that we’re dependent on.

Background

Over the last few years I’ve moved into a role where I’m managing a development team more than I’m writing code myself. I’ve spent a lot of that time looking at tools and practices in the broader software development community. After all, whether you’re writing C/AL, AL, PowerShell or JavaScript it’s all code and it’s unlikely that we’ll face any challenges that haven’t already been faced in one way or another in a different setting.

In that time we’ve introduced:

Package Management

The next thing to talk about is package management. I’ve written about the benefits of trying to avoid dependencies between your apps before (see here). However, if app A relies on app B and you cannot foresee ever deploying A without B then you have a dependency. There is no point trying to code your way round the problems that avoiding the dependency will create.

Accepting that your app has one or more dependencies – and most of our apps have at least one – opens up a bunch of questions and presents some interesting challenges.

Most obviously you need to know, where can I get the .app files for the apps that I am dependent on? Is it at least the minimum version required by my app? Is this the correct app for the version of the Dynamics NAV / Dynamics 365 Business Central that I am developing against? Are the apps that I depend on themselves dependent on other apps? If so, where do I get those from? Is there another layer of dependencies below that? Is it really turtles all the way down?

These are the sorts of questions that you don’t want to have to worry about when you are setting up an environment to develop in. Docker gives us a slick way to quickly create disposable development and testing environments. We don’t want to burn all the time that Docker saves us searching for, publishing and installing app files before we can start work.

This is what a package manager is for. The developer just needs to declare what their app depends on and leave the package manager to retrieve and install the appropriate packages.

The Goal

Why are we talking about this? What are we trying to achieve?

We want to keep the maintenance of all apps separate. When writing app A I shouldn’t need to know or care about the development of app B beyond my use of its API. I just need to know:

  • The minimum version that includes the functionality that I need – this will go into my app.json file
  • I can acquire that, or a later, version of the app from somewhere as and when I need it

I want to be able to specify my dependencies and with the minimum of fuss download and install those apps into my Docker container.

We’ve got a PowerShell command to do just that.

Get-ALDependencies -Container BCOnPrem -Install

There are a few jigsaw pieces we need to gather before we can start putting it all together.

Locating the Apps

We need somewhere to store the latest version of the apps that we might depend upon. There is usually some central, public repository where the packages are hosted – think of the PowerShell Gallery or Docker Hub for example.

We don’t have an equivalent repository for AL apps. AppSource performs that function for Business Central SaaS but that’s not much use to us while we are developing or if the apps we need aren’t on AppSource. We’re going to need to set something up ourselves.

You could just use a network folder. Or maybe SharePoint. Or some custom web service that you created. Our choice is Azure DevOps build artefacts. For a few reasons:

  • We’ve already got all of our AL code going through build pipelines anyway. The build creates the .app files, digitally signs them and stores them as build artefacts
  • The artefacts are only stored if all the tests ran successfully which ought to give us more confidence relying on them
  • The build automatically increments the app version so it should always be clear which version of the app is later and we shouldn’t get caught in app version purgatory when upgrading an app that we’re dependent on
  • We’re already making use of Azure DevOp’s REST API for loads of other stuff – it was easy to add some commands to retrieve the build artefacts (hence my earlier post on getting started with the API)

Identifying the Repository

There is a challenge here. In the app.json file we identify dependencies by app name, id and publisher. To find a build – and its artefacts – we need to know the project and repository name in Azure DevOps.

Seeing as we can’t add extra details into the app.json file itself we hold these details in a separate json file – environment.json. This file can have an array of dependency objects with a:

  • name – which should match the name of the dependency in the app.json file
  • project – the Azure DevOps project to to find this app in
  • repo – the Git repository in that project to find this app in

Once we know the right repository we can use the Azure DevOps API to find the most recent successful build and download its artefacts.

I’m aware that we could use Azure DevOps to create proper releases, rather than downloading apps that are still in development. We probably should – maybe I’ll come back and update this post some day. For now, we find that using the artefacts from builds is fine for the two main purposes we use them: creating local development environments and creating a Docker container as part of a build. We have a separate, manual process for uploading new released versions to SharePoint for now.

The Code

So much for the theory, let’s look at some code. In brief we:

  1. Read app.json and iterate through the dependencies
  2. For each dependency, find the corresponding entry in the environment.json file and read the project and repo for that dependency
  3. Download the app from the last successful build for that repo
  4. Acquire the app.json of the dependency
  5. Repeat steps 2-5 recursively for each branch of the dependency tree
  6. Optionally publish and install the apps that have been found (starting at the bottom of the tree and working up)

A few notes about the code:

  • It’s not all here – particularly the definition of Invoke-TFSAPI. That is just a wrapper for the Invoke-WebRequest command which adds the authentication headers (as previously described)
  • These functions are split across different files and grouped into a module, I’ve bundled them into a single file here for ease

(The PowerShell is hosted here if you can’t see it embedded below: https://gist.github.com/jimmymcp/37c6f9a9981b6f503a6fecb905b03672)

Getting Started with the Azure DevOps API

Azure DevOps is pretty sweet. Manage your code, backlog, sprints, builds – the whole caboodle. Also, it has a comprehensive REST API so you can access your data and integrate with DevOps from anywhere you like.

Ever since we started with DevOps (VSTS, TFS) we created some PowerShell scripts to integrate with it for Dynamics NAV development. They’ve become an indispensable part of a developer’s day.

(I’m going to refer to “Azure DevOps” as ADO from now on. I know ADO is an already familiar acronym in software development but I don’t want to type “Azure DevOps” every time and just referring to it as “DevOps” makes no sense. I’m not sure “Azure DevOps” makes much sense as a name anyway. Surely DevOps refers to the practice, not the tool you use to achieve it? Anyway…digression over.)

Authentication

The first thing you’re going to need to do is authenticate with your instance of ADO. You can create a Personal Access Token to authenticate your requests.

ADO Profile Menu.JPG

Sign in to your ADO instance, click on your profile (top right) and select Security from the menu.

Click “New Token” to create a new Personal Access Token. Give it a name, I’ll call mine “Azure Barbara” (only marginally sillier than “Azure DevOps”).

There are a bunch of “scopes” (25, at the time of writing) to which you can grant this token access. You can define which scopes an API call authorised with this token should have access to. For the sake of this example, I’ll choose “Full access”. Choose an expiration date for this key and hit Create.

Your token will be created and displayed. You need to copy this token somewhere safe. This is the only opportunity you will have to view the token. If you lose it you’ll need to create a new one.

Calling the API

Now that you’ve got an access token you can go ahead and call the API. The API is well documented here: https://docs.microsoft.com/en-us/rest/api/azure/devops/core/?view=azure-devops-rest-5.0

As a test, I’ll list the team projects in my instance. Open up PowerShell…

function Create-BasicAuthHeader {
  Param(
    [Parameter(Mandatory=$true)]
    [string]$Name,
    [Parameter(Mandatory=$true)]
    [string]$PAT
)

  $Auth = '{0}:{1}' -f $Name, $PAT
  $Auth = [System.Text.Encoding]::UTF8.GetBytes($Auth)
  $Auth = [System.Convert]::ToBase64String($Auth)
  $Header = @{Authorization=("Basic {0}" -f $Auth)} 
  $Header
}

Invoke-WebRequest -Uri 'https://dev.azure.com/<ADO organisation name>/_apis/projects' -Headers (Create-BasicAuthHeader 'Azure Barabara' '<personal access token>') -Method Get

Replace <ADO organisation name> with the name of your organisation in ADO. Also put your token name and value into the script in place of Barbara.

The Create-BasicAuthHeader function creates an authentication header which is passed by Invoke-WebRequest. If all is well you’ll get some JSON back. Something like this. I’ve got one project in my ADO instance called “Hello World”.

{"count":1,"value":[{"id":"<GUID>","name":"Hello World","url":"https://dev.azure.com/<my ADO instance>/_apis/projects/<project GUID>","state":"wellFormed","revision":471004199,"visibility":"private","lastUpdateTime":"2019-02-28T16:21:42.417Z"}]}

Nice. Next time we can set about something that is actually useful. To whet your appetite, these are some of the things that we use it for.

  • Finding the latest successful build for a given project and Git repo and downloading the build artefacts (the .app files that were created)
  • Reading a given file from a given project and Git repo – we use it to find app.json to download dependency apps recursively in the build process (more on that later)
  • Retrieving CAL objects that were modified by a given changeset #
  • Creating work items, iterations and other ADO entities