Code Coverage Updates in VS Code with AL Test Runner

Intro

For a while now AL Test Runner has been able to download the code coverage details after running your tests, output a summary of the objects that were hit with some stats and then highlight the lines which were hit in the previous test run or the last time you ran all the tests. More in the docs.

Recently, VS Code has added an API for test extensions to feed data into and some UI to show the coverage. It’s pretty cool.

Test Coverage

The first thing you’ll notice is this “Test Coverage” panel which is displayed after the tests have run. It displays a tree of the ojbects which have been hit by the run and the percentage coverage (in statement coverage terms).

If you click on a file in the tree it will open the file in the editor and you will see lines which were hit highlighted in the gutter.

In fact, these highlights will continue to be shown as you navigate around your source code. I’m leaving the “Code Coverage: Off/Previous/All” item in the status bar as this highlights each whole line and is much easier to see if you want to zoom out and get an impression of coverage of the whole file.

Coverage from Previous Runs

Coverage from previous test runs is stored and can be accessed from the Test Results pane (usually shown at the bottom of the screen). It might be useful to switch between test coverage results for test runs to see how to coverage % has changed over time (with the usual caveat about not using code coverage as a target).

The decorations in the gutter to indicate which lines have been covered in the current file are only shown when the latest test coverage is being displayed. That makes sense because the code coverage detail is all based on line numbers. Once you’ve made some changes to a file those line numbers are obsolete.

Performance Profiling Tests with AL Test Runner

This is a feature that has been in AL Test Runner for a months now but I haven’t got round to blogging about it. I haven’t done much blogging or work on AL Test Runner for a while now. I started a new job at the start of the year and had more important stuff keeping me occupied, but Johannes (https://github.com/jwikman) has prompted me into some action with some contributions recently – thanks very much for that 🥳

Scenario

My original scenario was that we had some poorly performing code. Complex code. Complex code I hadn’t been involved with much before. Complex code that would be easy for me to break functionally, while trying to improve the performance.

My first task was to surround the code with some integration tests. I find this an effective way to learn some existing code. You have to learn how to construct the GIVENs – what is the data structure and what setup is required for each test? It also gives you an easy way to step through the code and see what’s going on when certain processes are run. Crucially it also gave me some confidence that I wasn’t completely screwing up the app that I was working on while I was changing it.

Comparing Performance

OK, so I’ve got some tests to validate the functional behaviour before and after my changes, but what about performance? We have the Performance Toolkit, but I think that is less about the performance of a single process and more about concurrency. The obvious choice is to use the Performance Profiler.

I wanted to leave the existing code paths intact and just use a setup field as a rudimentary feature flag to switch between the old code and the new code. But, I didn’t want to be opening the client to run the performance profiler page or initialize and download snapshot profiles in between each test. I wondered if it was possible to could automate capturing a profile and downloading it to the workspace somehow.

It was 🙂

Setup

There is a new setting “Enable Performance Profiler” which defaults to true. This uses some new functions in the Test Runner Service app. With each test run the performance profile is captured and downloaded to the .altestrunner folder in your test project.

This should all be handled automatically. The Test Runner Service app should be downloaded when required and the serviceUrl in the config file set automatically. Check the docs for the required setup if not though.

Use

There is a new icon in the status bar which will open the performance profile viewer with the latest trace.

With this I could run two tests with the old and the new and compare the results side by side. If you want to do that just take a copy of the first trace file as it will be overwritten by the second test.

The trace, of course, has other benefits too. It is much easier to see the whole callstack and use the links on the right hand side to jump straight into the code. This also gives the potential for other features. I could maybe do something that allows you to choose two tests to run and download separate traces for them all in one action? Or read through the trace file to update the test coverage map? Let me know whether either of those sounds interesting or if you have other ideas or issues.

Pre-Releases & GitHub Actions for Visual Studio Code Extensions

Intro

This post is going to be a bit of a brain dump about developing my VS Code extension, branching strategy for pre-releases and releases and using GitHub actions to stitch it all together.

If you’re only here for the AL / Business Central content then you might want to give this one a miss. Then again, Microsoft are increasingly using GitHub for AL projects themselves (e.g. AL-Go for GitHub) – so it might be worth a look after all.

Objectives

What am I trying to achieve? I want to have a short turn around of:

Have an idea for a new feature
Implement the feature
Test it and make it available for others to test
Release

I use the extension pretty much every day at work so I am my own biggest customer. I want to write some new feature and start working with it in a pre-release myself to find any issues before I release it.

I also want to have a little fun with a side-project – learn a little typescript, practice some CI/CD, GitHub Actions and Application Insights. If anyone else finds the extension useful as well then that’s a bonus.

Overview

This is my workflow. I want to get the feature into the pre-release version of the extension on the marketplace quickly. That way I will get the new pre-release myself from the marketplace and use it in my daily work. I’ll make any fixes or improvements in updates to the pre-release before merging the code to the release version and publishing to the marketplace.

GitHub Actions

The GitHub actions definition is fairly self-explanatory. The yaml is bellow, or here if you prefer. Run whenever some code is pushed. Build, test, package with npm and vsce. Run the PowerShell tests with Pester. Upload the built extension as an artifact. If the pre-release branch is being built then use vsce to publish to the marketplace with the --pre-release switch.

The actions definition in the master branch is similar but publishes to the marketplace without the --pre-release switch.

name: CI

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
  pull_request:
    branches: [ master ]
  workflow_dispatch:

jobs:
  build:
    runs-on: windows-latest

    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: npm install, build and test
        run: |
          npm install
          npm run build
          npm test
      - name: package with vsce
        run: |
          npm install -g vsce
          vsce package
      - name: run pester tests
        shell: pwsh
        run: |
          Set-PSRepository psgallery -InstallationPolicy Trusted
          Install-Module Pester
          Install-Module bccontainerhelper
          gci *ALTestRunner.psm1 -Recurse | % {$_.FullName; Import-Module $_.FullName}
          Invoke-Pester
      - name: Upload a Build Artifact
        uses: actions/upload-artifact@v2.1.4
        with:
          name: AL Test Runner
          path: ./*.vsix

      - name: Publish to marketplace
        if: github.ref == 'refs/heads/pre-release'
        run: |
          vsce publish -p ${{ secrets.VSCE_PAT }} --pre-release

The personal access token for my Visual Studio account (used to publish to the marketplace) is stored in a repository secret.

You can create and update these from the settings for the repository. You can read more about creating the personal access token and the option for publishing extensions to the marketplace here: https://code.visualstudio.com/api/working-with-extensions/publishing-extension

Conclusions

It is rewarding to make some changes to the extension, push them to GitHub and then 10-15 minutes later be able to use them in a new version of the extension which has been automatically published, downloaded and installed. It allows you to publish more frequently and with more confidence.

AL Test Runner Pre-Release Version

TL;DR

There is now a pre-release version of the AL Test Runner extension for Visual Studio Code. It will have the latest (and possibly unstable) features.

Pre-Releases

VS Code recently added support for pre-release versions of extensions. You can install a pre-release by clicking on the “Switch to Pre-Release Version” button from the extension details within VS Code. See https://code.visualstudio.com/updates/v1_63#_pre-release-extensions for more details.

Up ’til now I have typically packaged a new version of the extension and used it myself for a week or two to check that it isn’t horribly broken before I push an update to the marketplace. Having a pre-release version will give me a better way to use the extension myself but also get feedback from anyone who is interested in being a beta tester. GitHub issues are the best place to log requests or bugs.

What’s in the Pre-Release?

There are few things which are currently in the pre-release but not in the release version.

Debug All Tests

Bit niche, but I have actually found it useful on a couple of occasions. There is an icon at the top of the Test Explorer view and a command in the command palette to debug all the tests, so I decided to add support for it in my extension.

A new version of the Test Runner Service app is required to support this. Install with the "Install Test Runner Service" command from inside VS Code or download the latest version from here: https://github.com/jimmymcp/test-runner-service/raw/master/James%20Pearson_Test%20Runner%20Service.app

Publishing Apps using PowerShell

There is a new setting to publish apps to the container using PowerShell (the bccontainerhelper module) rather than the publish command in VS Code.

Why? A couple of reasons.

I can’t know whether the app has compiled and published successfully when using the AL: Publish command. If publishing the app fails then VS Code is left thinking that the tests are running when in reality they never started. You need to manually cancel the test run before you can start another from the Test Explorer. Publishing from PowerShell gives a little more control
I’m toying with the idea of automating test runs in the background while developing, something along the lines that Luc suggested here: https://github.com/jimmymcp/al-test-runner/issues/42. This would require a more reliable to compile and publish the app(s) than just triggering the AL: Publish command and hoping that it worked

testRunnerCodeunitId

There is a new key in the AL Test Runner config.json file to specify the id of the test runner codeunit id to use. It defaults to the codeunit isolation runner but you can override with another if you like.

Various

Various other improvements – updated Pester tests, updated GitHub actions. Take a look on GitHub if you are interested.

Test Explorer in Visual Studio Code

The July 2021 release of Visual Studio Code (1.59) introduced a new testing API and Test Explorer UI. From v0.6.0 this API is used by AL Test Runner.

Improvements

UI

The biggest improvement is the Test Explorer view which shows your test codeunits, their test methods and the status of each.

Hovering over a test gives you three icons to run, debug or open an editor at the test.

You can run and debug all the tests in a given codeunit by hovering over the codeunit name or run and debug all tests at the top.

The filter box allows you to easily find specific tests, which I’ve found useful in projects which several test codeunits and hundreds of tests.

You can also filter to only show failed tests or only test which are present in the codeunit in the current editor. The explorer supports different ways of sorting and displaying the tests.

Icons are added into the gutter alongside test methods in the editor. Left click to run the test or right click to see this context menu with more options.

The old “Run Test” and “Debug Test” codelens actions are also still added above the test definition.

Commands & Shortcuts

A whole set of new commands are introduced with keyboard chords beginning with Ctrl + ; The existing AL Test Runner keyboard shortcuts still work but there are some nice options in the new set – like “Test: Rerun Last Run” to repeat the last run test without having to navigate to it again.

Using the Test Explorer

Using the Test Explorer is pretty self-explanatory if you’ve already been using AL Test Runner. When you open your workspace/folder the tests should be automatically discovered and loaded into the Test Explorer view. On first opening all of the tests will have no status i.e. neither pass or fail – but results from now on will be persisted.

Running one or more tests – regardless of where you run them from (Test Explorer, Command Palette, CodeLens, Keyboard Shortcut) – will start a test run. You’ll see “Running tests…” in the Status Bar.

Once the test(s) have finished running you’ll see the results at the top of the Test Explorer, “x / y tests passed (z %)”, and the status icons by each test will be updated.

If the tests do not actually run e.g. because your container isn’t started then the test run will not finish and “Running tests” will continue to spin at the bottom of the screen. You can stop the run manually from the top of the Test Explorer, fix the problem and go again.