Measuring Code Coverage in Business Central with AL Test Runner

v0.5.0 of AL Test Runner adds some capability to measure code coverage per object and highlight the lines of code that were hit in the previous test run.

This is an example of what we’re aiming for. Running one or more tests, seeing the list of objects and the percentage of their code lines that were hit, opening those objects and highlighting those lines. This should help identify objects and code paths aren’t covered by any tests. I don’t believe code coverage should be a target in itself (maybe more on that in a separate post) but it can be a useful tool to see where you might want to bolster your test suite.

Overview

Code coverage is started and stopped before each run of run or more tests. The Test Runner Service app is called to download the code coverage details and they are saved to a JSON file. This file is read and summarised into a per-object percentage which is output with the test results. Only objects which are defined in the workspace are included – so you won’t include standard objects, but you will see test objects.

The path to each object is included so you can Alt+Click to navigate to it. A new Toggle Code Coverage command (Ctrl+Alt+C) allows you to switch the highlighting for lines which have been hit on and off.

Setup

Install the Test Runner Service app with the command in Visual Studio Code. If it is already installed you will need to uninstall and unpublish the existing version first
In the AL Test Runner config.json file
- Set the path to save the code coverage JSON file to in the codeCoveragePath key. This path is relative to the folder that contains your test code e.g. .//.altestrunner//codecoverage.json to save it within the .altestrunner folder
- Set the testRunnerServiceUrl key to the OData URL of the Test Runner web service e.g. http://test:7048/BC/ODataV4/TestRunner?company=My%20Company&tenant=default for a container named test, default tenant and company called My Company
In Visual Studio Code Settings
- Enable Code Coverage
- [Edit: this is now optional – see 0.5.3 update below] Select the path to the code coverage file relative to your app code i.e. if you have your test extension in a separate top level folder you might set it to ../tests/.altestrunner/codecoverage.json This allows AL Test Runner to find and display the code coverage details from an object in your app code

0.5.1 Update

Use the Exclude Code Coverage Files to define file paths that should be excluded from the code coverage summary. This is a regex pattern which is matched against the file paths. For example, setting this to “Tests” will exclude any files with “Tests” in their path.

Test Folder Name – specify the name of the folder which contains the test app. Previously if you worked in a multi-root workspace and had an editor open at the production app it would attempt to run tests in the production app, create a config file, ask you which company to test in, prompt for credentials…meh. With this setting AL Test Runner will always run tests in the app which is contained in the folder with the name given in this setting.

0.5.3 Update

Some of the early feedback from people who were trying to enable code coverage was that it was a bit of a game. And not in a good, fun way. More like Ludo. You’re trying to line up all your pieces but every time you think you’ve got them where you want them someone else lands on them and messes everything up.

From 0.5.3 it isn’t necessary to set the code coverage path in VS Code’s settings (see setup #3 above). If this path is not set then the extension will attempt to find the codecoverage.json file somewhere in the workspace.

The codeCoveragePath key in the AL Test Runner config file is still required, but has a default value which will be fine in most cases.

Ideas

Ideas and feedback welcome as issues (or better yet, pull requests) in the repo on GitHub. These are some that I might work on.

Maybe a setting to enter a glob pattern to include/exclude certain files and/or paths from the summary
Smoother setup of the different settings that are required – I’ve tried to provide sensible default values but they are a few things to enter correctly to get it working
Currently the code coverage file is overwritten with each test run – it would be more useful if this was cumulative so that running a single test didn’t overwrite all the results from a previous test run. If you want a complete overview you have to run the whole suite – but then maybe that isn’t a bad thing
Perhaps an overall code coverage percentage for the whole project as well as per app (with the above caveat that I’m sceptical about fixed code coverage targets)
A CodeLens at the top of an object with its percentage code coverage that toggles the highlighting on/off when clicked

Debugging Business Central Tests with AL Test Runner

TL;DR

Install the Test Runner Service app (see https://github.com/jimmymcp/test-runner-service; direct download of the app file from here) or use the “Install Test Runner Service” command from VS Code to install into the Docker container specified in the config file
Set the URL to the test runner service in the testRunnerServiceUrl key of the AL Test Runner config file
Define a debug configuration of request type ‘attach’ in launch.json to attach the debugger to the service tier that you want to debug (should be the same service tier as specified by the testRunnerServiceUrl key)

Overview

From v0.4.0 of the AL Test Runner app it is possible to debug Business Central tests without leaving Visual Studio Code. There’s a lot of scope for improvements but if you’re interested in trying it out it’s included in the marketplace version now.

Test Runner Service App

This is a very simple app that exposes a codeunit as a web service to accept a codeunit ID and test name to run. Those values are passed to a test runner codeunit (codeunit isolation) to actually run the tests. This is so that the tests are executed in a session type of WebService which the debugger can attach to (the PowerShell runner creates a session type of ClientService).

The app is in the per tenant object range: 79150-79160 to be precise (a number picked pretty much at random). If that clashes with some other object ranges present in the database you can clone the repo and renumber the codeunits if you want. The source is here: https://github.com/jimmymcp/test-runner-service

You can use the Install Test Runner Service command in VS Code to automatically download the app and install into the container specified in the AL Test Runner config file.

The app is not code signed so you’ll need to use the -SkipVerification switch when you install it.

testRunnerServiceUrl

A new key is required in the AL Test Runner config file. This specifies the OData endpoint of the test runner service that is exposed by the Test Runner Service app. The service will be called from the VS Code terminal – so consider where the terminal is runner and where the service is hosted.

We develop against local Docker containers so the local VS Code instance will be able to access the web service without any trouble. If you develop against a remote Docker host make sure that the OData port is available externally. If you use VS Code remote development remember that the PowerShell session will be running on the VS Code server host.

The url will be in the format:

http[s]://[BC host]:[OData port]/[BC instance]/ODataV4/TestRunner?company=[BC company]

for example against a local Docker container called bc with OData exposed on the default port of 7048 and a company name of CRONUS International Ltd.:

"testRunnerServiceUrl": "http://bc:7048/BC/ODataV4/TestRunner?company=CRONUS%20International%20Ltd."

Debug Configuration

You will need a debug configuration of type attach in the launch.json file. This should attach the debugger to the same service as identified by the testRunnerServiceUrl key. breakOnNext should be set WebServiceClient. Currently UserPassword authentication is the only authentication method supported.

{
    "name": "Attach bc",
    "type": "al",
    "request": "attach",
    "server": "http://bc",
    "serverInstance": "bc",
    "authentication": "UserPassword",
    "breakOnError": true,
    "breakOnRecordWrite": false,
    "enableSqlInformationDebugger": true,
    "enableLongRunningSqlStatements": true,
    "longRunningSqlStatementsThreshold": 500,
    "numberOfSqlStatements": 10,
    "breakOnNext": "WebServiceClient"
}

Debugging

Codelens actions will be added at the top of test codeunits and before each test method. Set a breakpoint in the test method that you want to debug or allow the debugger to break on an error.

Clicking on Debug Test (Ctrl+Alt+D) will attach the first debug configuration specified in launch.json and call the web service to run the test with the Test Runner Service app.

Attaching the debugger and running a test from VS Code

Step in/out/over as usual. When the code execution has finished if an error was encountered the error message and callstack will be displayed in the terminal.

Limitations

There are some limitations to running tests in a web service session. Most importantly TestPage variables are not supported. There may also be some differences in the behaviour of tests in web services and the PowerShell runner.

Testing Against a Remote Docker Host with AL Test Runner

Apologies for another post about AL Test Runner. If you don’t use or care about the extension you can probably stop reading now and come back next time. It isn’t my intention to keep banging on about it – but the latest version (v0.2.1) does plug a significant gap.

Next time I’ll move onto a different subject – some thoughts about how we use Git to manage our code effectively.

Developing Against a Remote Docker Container

While I still prefer developing against a local Docker container I know that many others publish their apps to a container hosted somewhere else. In which case your options for running tests against that container are:

Using the Remote Development capability of VS Code to open a terminal and execute PowerShell on the remote host – discussed here and favoured by Tobias Fenster (although his views on The Beautiful Game may make you suspicious of any of his opinions 😉)
Enabling PS-Remoting and opening a PowerShell session to the host to execute some commands over the network – today’s topic

Again, shout out to Max and colleagues for opening a pull request with their changes to enable this and for testing these latest mods.

Enable PS Remoting

Firstly, you’re going to need to be able to open a PowerShell session to the Docker host with:

New-PSSession <computer name>

I won’t pretend to understand the intricacies of setting this up in different scenarios – you should probably read the blog of someone who knows what they are talking about if you need help with it.

The solution will likely include:

Opening a PowerShell session on the host as administrator and running Enable-PSRemoting
Making sure the firewall is open to the port that you are connecting over
Passing a credential and possibly an authentication type to New-PSSession

To connect to my test server in Azure I run the following:

New-PSSession <server name> -Credential (Get-Credential) -Authentication Basic

AL Test Runner Config

They are several new keys in the AL Test Runner config file to accommodate remote containers. There are also a few new commands to help create the required config.

The Open Config File command will open the config JSON file or create it, if it doesn’t already exist. Set Container Credential and Set VM Credential can be used to set the credentials used to connect to the container and the remote host respectively.

The required config keys are:

dockerHost – the name of the server that is hosting the Docker containers. This name will be used to create the remote PowerShell session. Leaving this blank implies that containers are hosted locally and the extension will work as before
vmUserName / vmSecurePassword – the credentials used to connect to the Docker host
remoteContainerName – the name of the container to run tests against
newPSSessionOptions – switches and parameters that should be added to New-PSSession to open the session to the Docker host (see below)

The extension uses New-PSSession to open the PowerShell session to the Docker host. The ComputerName and Credential parameters will populated from the dockerHost and vmUserName / vmSecurePassword config values respectively.

Any additional parameters that must be specified should be added to the newPSSessionOptions config key. As in my case I run

New-PSSession <server name> -Credential <credential> -Authentication Basic

I need to set newPSSessionOptions in the config file to “-Authentication Basic”. You can use this key for -useSSL, -Port, -SessionOption or whatever else you need to open the session.

With the config complete you should be able to execute tests, output the results and decorate the test codeunits as if you were working locally. Beautiful.

As ever, feedback, suggestions and contributions welcome. Hosted on GitHub.

Remote Development with VS Code and AL Test Runner

The most obvious limitation of the AL Test Runner extension for VS Code has been that you need to run VS Code on the Docker host machine. That’s fine for us because we do all our development on local Docker containers but I’m aware that this isn’t everyone’s preferred process.

Local Repo and VS Code, Remote Docker Host

I guess if you’re not hosting the Docker container locally then you are hosting it on some remote server – maybe on your own hardware or maybe in Azure or another cloud. To get AL Test Runner working in this scenario you’d need the AL Test Runner PowerShell module imported on the host and PS Remoting enabled to execute PowerShell on the host from your local VS Code terminal.

This post isn’t about getting that working. It’s not supported yet – although I do have a pull request to review from a team that are using it like this (thanks Max).

Remote Development with VS Code

An alternative approach is to use remote development with VS Code. The files that you are working on and the Docker host are remote but you are using VS Code locally. Kind of like RemoteDesktop Apps – the benefit of running on a server and using its resources but with the experience of an app that is running locally.

Install Open SSH server on the remote machine, install some VS Code extensions (using an insider build of VS Code – for now) and connect over SSH to the machine. Some magic happens at the other end and a few spells, invocations and minutes later a VS Code Server is installed on the remote machine.

I won’t go into the setup. I mostly followed this excellent blog post from Tobias Fenster and used aka.ms/getbc to create a new VM in Azure to test with.

It allows you to work in a local VS Code window but access the file system of the server and execute commands on it. Install VS Code extensions and PowerShell modules as if you are working locally and they are installed on the remote.

It is smooth. Impressively so. You quickly forget that you aren’t just working with files and extensions on your local machine. This clip shows:

Selecting a folder on the remote server from my recent history
Connecting via SSH
Entering the password for the remote account that I am authenticating as (a local account as my VM in Azure is not joined to a domain)
Running all the tests in the project and working with VS Code as I would do locally

Connecting to remote host via SSH and running some tests

I still prefer actual local development, but I have to admit that this is pretty great.

AL Test Runner

I span up a remote development scenario out of curiosity but I also wanted to test how/if AL Test Runner would work. It works almost seamlessly. Almost. There is just a little stretch of exposing stitching – but it’s easy to work around.

Once you’ve opened a remote development window you’ll need to:

Install the AL extension
Install the AL Test Runner extension
Install the navcontainerhelper PowerShell module (you can use install-module in the integrated terminal)

If you try to run tests you’ll find that it appears to hang indefinitely. Actually it has popped a window to enter the credentials to connect to the server with – but you can’t see it and it won’t continue until you dismiss the window.

If you’re interested in trying it the workaround for now is to manually edit the config.json file in the .altestrunner folder.

When you first install the extension you won’t have a config.json file. Running a test, any test is enough to create it. You’ll also notice that the command appears to hang in the terminal. You can kill that terminal once the file has been created.

Open config.json and enter the userName to authenticate with BC. Next you need to enter the securePassword (this is not your plain text password). You can get the secure password running the following in the terminal:

ConvertTo-SecureString 'your password' -AsPlainText -Force | ConvertFrom-SecureString

Copy the huge string from the output into the securePassword key of the config file. After that you should be good to go.

At some point I’ll also work on the ability to use remote PowerShell to execute tests on a remote Docker host from your local machine. After all, Max has already done most of the work for me 🙂

AL Test Runner for Visual Studio Code

TL;DR

I’ve written an extension for VS Code to help run your AL tests in local Docker containers. Search for “AL Test Runner” in the extension marketplace or click here. Feedback, bugs, feature suggestions all gratefully received on the GitHub repo or james@jpearson.blog

Intro

As soon as Freddy added capability to the navcontainerhelper module to execute automated tests I was excited about the potential for:

Making test execution in our build pipeline simpler and more reliable
Running tests from Visual Studio Code as part while developing

I’ve written about both aspects in the past, but especially #1 recently – about incorporating automated tests into your Azure DevOps pipeline.

This post is about #2 – incorporating running tests as early as possible into your development cycle.

Finding Bugs ASAP

You’ve probably heard the idea – and it’s common sense even if you haven’t – that the cost of finding a bug in your software increases the later in the development/deployment cycle you find it.

If you realise you made a silly mistake in code that you wrote 2 minutes ago – there’s likely no harm done. Realise there is a bug in software that is now live in customers’ databases and the implications could be much greater. Potentially annoyed customers, data that now needs fixing, support cases, having to rush out a hotfix etc.

We’ve all been there. It’s not a nice place to visit. I once deleted all the (hundreds of thousands of) records in the Purch. Rcpt. Line table with a Rec.DELETEALL on a temporary table…turns out it wasn’t temporary…and I was working in the live database.

Writing automated tests can help catch problems before you release them out into the wild. They force you to think about the expected behaviour of the code and then test whether it actually behaves like that. Hopefully if the code that we push to a branch in Azure DevOps has a bug it will cause a test to fail, the artifacts won’t be published, the developer will get an email and the customer won’t be the hapless recipient of our mistake. No harm done.

However, the rising cost of finding a bug over time still applies. Especially if the developer has started working on something else or gone home. Getting back your head back into the code, reproducing and finding the bug and fixing it are harder if you’ve had a break from the code than if you went looking for it straight away.

Running Tests from VS Code

That’s why I’m keen that we run tests from VS Code as we are writing them. Write a test, see it fail, write the code, see the test pass, repeat.

I’ve written about this before. You can use tasks in VS Code to execute the required PowerShell to run the tests. The task gives you access to the current file and line no. so that you can fancy stuff like running only the current test or test codeunit.

AL Test Runner

However, I was keen to improve on this and so have started work on a VS Code extension – AL Test Runner.

The goals are to:

Make it as simple as possible to run the current test, tests in the current codeunit or all tests in the extension with commands and keyboard shortcuts
Cache the test results
Decorate test methods according to the latest test results – pass, fail or untested
Provide extra details e.g. error message and callstack when hovering over the test name
Add a snippet to make it easier to create new tests with placeholders for GIVEN, WHEN and THEN statements

Important: this is for running tests with the navcontainerhelper PowerShell module against a local Docker container. Please make sure that you are using the latest version of navcontainerhelper.

Getting Started

Download the extension from the extension marketplace in VS Code and reload the window.
Open a folder containing an AL project
Open a test codeunit, you should notice that the names of test methods are decorated with an amber background (as there are no results available for those tests)
- The colours for passing, failing and untested tests are configurable if you don’t like them or they don’t fit with your VS Code theme. Alternatively you can turn test decoration off altogether if you don’t like it
Place the cursor in a test method and run the “AL Test Runner: Run Current Test” command (Ctrl+Alt+T)
You should be prompted to select a debug configuration (from launch.json), company name, test suite name and credentials as appropriate (depends if you’re running BC14 or BC15, if you have multiple companies, authentication type etc.)
- I’ve noticed that sometimes the output isn’t displayed in the new terminal when it is first created – I don’t know why. Subsequent commands always seem to show up fine 🤷‍♂️
Use the “ttestprocedure” to create new test methods

.gitignore

If you’re using Git then I’d recommend adding the .altestrunner folder to your .gitignore file:

.altestrunner/

Committing the config file and the test results xml files doesn’t feel like a great idea.