TL;DL
We use PowerShell to call the Azure DevOps API and retrieve Build Artefacts from the last successful build of the repository/repositories that we’re dependent on.
Background
Over the last few years I’ve moved into a role where I’m managing a development team more than I’m writing code myself. I’ve spent a lot of that time looking at tools and practices in the broader software development community. After all, whether you’re writing C/AL, AL, PowerShell or JavaScript it’s all code and it’s unlikely that we’ll face any challenges that haven’t already been faced in one way or another in a different setting.
In that time we’ve introduced:
- Source control – first TFVC before migrating to Git. I’ve written about our experiences here
- Code review (now, specifically pull requests in Azure DevOps)
- Automated testing
- Some CI/CD principles with all our AL code going through build pipelines defined in yaml now
Package Management
The next thing to talk about is package management. I’ve written about the benefits of trying to avoid dependencies between your apps before (see here). However, if app A relies on app B and you cannot foresee ever deploying A without B then you have a dependency. There is no point trying to code your way round the problems that avoiding the dependency will create.
Accepting that your app has one or more dependencies – and most of our apps have at least one – opens up a bunch of questions and presents some interesting challenges.
Most obviously you need to know, where can I get the .app files for the apps that I am dependent on? Is it at least the minimum version required by my app? Is this the correct app for the version of the Dynamics NAV / Dynamics 365 Business Central that I am developing against? Are the apps that I depend on themselves dependent on other apps? If so, where do I get those from? Is there another layer of dependencies below that? Is it really turtles all the way down?
These are the sorts of questions that you don’t want to have to worry about when you are setting up an environment to develop in. Docker gives us a slick way to quickly create disposable development and testing environments. We don’t want to burn all the time that Docker saves us searching for, publishing and installing app files before we can start work.
This is what a package manager is for. The developer just needs to declare what their app depends on and leave the package manager to retrieve and install the appropriate packages.
The Goal
Why are we talking about this? What are we trying to achieve?
We want to keep the maintenance of all apps separate. When writing app A I shouldn’t need to know or care about the development of app B beyond my use of its API. I just need to know:
- The minimum version that includes the functionality that I need – this will go into my app.json file
- I can acquire that, or a later, version of the app from somewhere as and when I need it
I want to be able to specify my dependencies and with the minimum of fuss download and install those apps into my Docker container.
We’ve got a PowerShell command to do just that.
Get-ALDependencies -Container BCOnPrem -Install
There are a few jigsaw pieces we need to gather before we can start putting it all together.
Locating the Apps
We need somewhere to store the latest version of the apps that we might depend upon. There is usually some central, public repository where the packages are hosted – think of the PowerShell Gallery or Docker Hub for example.
We don’t have an equivalent repository for AL apps. AppSource performs that function for Business Central SaaS but that’s not much use to us while we are developing or if the apps we need aren’t on AppSource. We’re going to need to set something up ourselves.
You could just use a network folder. Or maybe SharePoint. Or some custom web service that you created. Our choice is Azure DevOps build artefacts. For a few reasons:
- We’ve already got all of our AL code going through build pipelines anyway. The build creates the .app files, digitally signs them and stores them as build artefacts
- The artefacts are only stored if all the tests ran successfully which ought to give us more confidence relying on them
- The build automatically increments the app version so it should always be clear which version of the app is later and we shouldn’t get caught in app version purgatory when upgrading an app that we’re dependent on
- We’re already making use of Azure DevOp’s REST API for loads of other stuff – it was easy to add some commands to retrieve the build artefacts (hence my earlier post on getting started with the API)
Identifying the Repository
There is a challenge here. In the app.json file we identify dependencies by app name, id and publisher. To find a build – and its artefacts – we need to know the project and repository name in Azure DevOps.
Seeing as we can’t add extra details into the app.json file itself we hold these details in a separate json file – environment.json. This file can have an array of dependency objects with a:
- name – which should match the name of the dependency in the app.json file
- project – the Azure DevOps project to to find this app in
- repo – the Git repository in that project to find this app in
Once we know the right repository we can use the Azure DevOps API to find the most recent successful build and download its artefacts.
I’m aware that we could use Azure DevOps to create proper releases, rather than downloading apps that are still in development. We probably should – maybe I’ll come back and update this post some day. For now, we find that using the artefacts from builds is fine for the two main purposes we use them: creating local development environments and creating a Docker container as part of a build. We have a separate, manual process for uploading new released versions to SharePoint for now.
The Code
So much for the theory, let’s look at some code. In brief we:
- Read app.json and iterate through the dependencies
- For each dependency, find the corresponding entry in the environment.json file and read the project and repo for that dependency
- Download the app from the last successful build for that repo
- Acquire the app.json of the dependency
- Repeat steps 2-5 recursively for each branch of the dependency tree
- Optionally publish and install the apps that have been found (starting at the bottom of the tree and working up)
A few notes about the code:
- It’s not all here – particularly the definition of Invoke-TFSAPI. That is just a wrapper for the Invoke-WebRequest command which adds the authentication headers (as previously described)
- These functions are split across different files and grouped into a module, I’ve bundled them into a single file here for ease
(The PowerShell is hosted here if you can’t see it embedded below: https://gist.github.com/jimmymcp/37c6f9a9981b6f503a6fecb905b03672)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function Get-ALDependencies { | |
| Param( | |
| [Parameter(Mandatory=$false)] | |
| [string]$SourcePath = (Get-Location), | |
| [Parameter(MAndatory=$false)] | |
| [string]$ContainerName = (Split-Path (Get-Location) -Leaf), | |
| [Parameter(Mandatory=$false)] | |
| [switch]$Install | |
| ) | |
| if (!([IO.Directory]::Exists((Join-Path $SourcePath '.alpackages')))) { | |
| Create-EmptyDirectory (Join-Path $SourcePath '.alpackages') | |
| } | |
| $AppJson = ConvertFrom-Json (Get-Content (Join-Path $SourcePath 'app.json') -Raw) | |
| Get-ALDependenciesFromAppJson -AppJson $AppJson -SourcePath $SourcePath -ContainerName $ContainerName -Install:$Install | |
| } | |
| function Get-ALDependenciesFromAppJson { | |
| Param( | |
| [Parameter(Mandatory=$true)] | |
| $AppJson, | |
| [Parameter(Mandatory=$false)] | |
| [string]$SourcePath = (Get-Location), | |
| [Parameter(Mandatory=$false)] | |
| [string]$RepositoryName, | |
| [Parameter(Mandatory=$false)] | |
| [string]$ContainerName, | |
| [Parameter(Mandatory=$false)] | |
| [switch]$Install | |
| ) | |
| foreach ($Dependency in $AppJson.dependencies) { | |
| $EnvDependency = Get-DependencyFromEnvironment -SourcePath $SourcePath -Name $Dependency.name | |
| $Apps = Get-AppFromLastSuccessfulBuild -ProjectName $EnvDependency.project -RepositoryName $EnvDependency.repo | |
| $DependencyAppJson = Get-AppJsonForProjectAndRepo -ProjectName $EnvDependency.project -RepositoryName $EnvDependency.repo | |
| Get-ALDependenciesFromAppJson -AppJson $DependencyAppJson -SourcePath $SourcePath -RepositoryName $RepositoryName -ContainerName $ContainerName -Install:$Install | |
| foreach ($App in $Apps) { | |
| if (!$App.FullName.Contains('Tests')) { | |
| Copy-Item $App.FullName (Join-Path (Join-Path $SourcePath '.alpackages') $App.Name) | |
| if ($Install.IsPresent) { | |
| try { | |
| Publish-NavContainerApp -containerName $ContainerName -appFile $App.FullName -sync -install | |
| } | |
| catch { | |
| if (!($_.Exception.Message.Contains('already published'))) { | |
| throw $_.Exception.Message | |
| } | |
| } | |
| } | |
| } | |
| } | |
| } | |
| } | |
| function Get-AppJsonForProjectAndRepo { | |
| Param( | |
| [Parameter(Mandatory=$true)] | |
| [string]$ProjectName, | |
| [Parameter(Mandatory=$false)] | |
| [string]$RepositoryName | |
| ) | |
| $VSTSProjectName = (Get-VSTSProjects | where name -like ('*{0}*' -f $ProjectName)).name | |
| $AppContent = Invoke-TFSAPI ('{0}{1}/_apis/git/repositories/{2}/items?path=app.json' -f (Get-TFSCollectionURL), $VSTSProjectName, (Get-RepositoryId -ProjectName $VSTSProjectName -RepositoryName $RepositoryName)) -GetContents | |
| $AppJson = ConvertFrom-Json $AppContent | |
| $AppJson | |
| } | |
| function Get-DependencyFromEnvironment { | |
| Param( | |
| [Parameter(Mandatory=$true)] | |
| [string]$SourcePath, | |
| [Parameter(Mandatory=$true)] | |
| [string]$Name | |
| ) | |
| Get-EnvironmentKeyValue -SourcePath $SourcePath -KeyName 'dependencies' | where name -eq $Name | |
| } | |
| function Get-EnvironmentKeyValue { | |
| Param( | |
| [Parameter(Mandatory=$false)] | |
| [string]$SourcePath = (Get-Location), | |
| [Parameter(Mandatory=$true)] | |
| [string]$KeyName | |
| ) | |
| if (!(Test-Path (Join-Path $SourcePath 'environment.json'))) { | |
| return '' | |
| } | |
| $JsonContent = Get-Content (Join-Path $SourcePath 'environment.json') -Raw | |
| $Json = ConvertFrom-Json $JsonContent | |
| $Json.PSObject.Properties.Item($KeyName).Value | |
| } | |
| function Get-VSTSProjects { | |
| (Invoke-TFSAPI -Url ('{0}_apis/projects?$top=1000' -f (Get-TFSCollectionURL))).value | |
| } | |
| function Get-RepositoryId { | |
| Param( | |
| [Parameter(Mandatory=$true)] | |
| [string]$ProjectName, | |
| [Parameter(Mandatory=$false)] | |
| [string]$RepositoryName | |
| ) | |
| $Repos = Invoke-TFSAPI ('{0}{1}/_apis/git/repositories' -f (Get-TFSCollectionURL), $ProjectName) | |
| if ($RepositoryName -ne '') { | |
| $Id = ($Repos.value | where name -like ('*{0}*' -f $RepositoryName)).id | |
| } | |
| else { | |
| $Id = $Repos.value.item(0).id | |
| } | |
| if ($Id -eq '' -or $Id -eq $null) { | |
| $Id = Get-RepositoryId -ProjectName $ProjectName -RepositoryName '' | |
| } | |
| $Id | |
| } |
James Pearson 