Measuring Code Coverage in Business Central with AL Test Runner

v0.5.0 of AL Test Runner adds some capability to measure code coverage per object and highlight the lines of code that were hit in the previous test run.

This is an example of what we’re aiming for. Running one or more tests, seeing the list of objects and the percentage of their code lines that were hit, opening those objects and highlighting those lines. This should help identify objects and code paths aren’t covered by any tests. I don’t believe code coverage should be a target in itself (maybe more on that in a separate post) but it can be a useful tool to see where you might want to bolster your test suite.

Overview

Code coverage is started and stopped before each run of run or more tests. The Test Runner Service app is called to download the code coverage details and they are saved to a JSON file. This file is read and summarised into a per-object percentage which is output with the test results. Only objects which are defined in the workspace are included – so you won’t include standard objects, but you will see test objects.

The path to each object is included so you can Alt+Click to navigate to it. A new Toggle Code Coverage command (Ctrl+Alt+C) allows you to switch the highlighting for lines which have been hit on and off.

Setup

  1. Install the Test Runner Service app with the command in Visual Studio Code. If it is already installed you will need to uninstall and unpublish the existing version first
  2. In the AL Test Runner config.json file
    • Set the path to save the code coverage JSON file to in the codeCoveragePath key. This path is relative to the folder that contains your test code e.g. .//.altestrunner//codecoverage.json to save it within the .altestrunner folder
    • Set the testRunnerServiceUrl key to the OData URL of the Test Runner web service e.g. http://test:7048/BC/ODataV4/TestRunner?company=My%20Company&tenant=default for a container named test, default tenant and company called My Company
  3. In Visual Studio Code Settings
    • Enable Code Coverage
    • [Edit: this is now optional – see 0.5.3 update below] Select the path to the code coverage file relative to your app code i.e. if you have your test extension in a separate top level folder you might set it to ../tests/.altestrunner/codecoverage.json This allows AL Test Runner to find and display the code coverage details from an object in your app code

0.5.1 Update

Use the Exclude Code Coverage Files to define file paths that should be excluded from the code coverage summary. This is a regex pattern which is matched against the file paths. For example, setting this to “Tests” will exclude any files with “Tests” in their path.

Test Folder Name – specify the name of the folder which contains the test app. Previously if you worked in a multi-root workspace and had an editor open at the production app it would attempt to run tests in the production app, create a config file, ask you which company to test in, prompt for credentials…meh. With this setting AL Test Runner will always run tests in the app which is contained in the folder with the name given in this setting.

0.5.3 Update

Some of the early feedback from people who were trying to enable code coverage was that it was a bit of a game. And not in a good, fun way. More like Ludo. You’re trying to line up all your pieces but every time you think you’ve got them where you want them someone else lands on them and messes everything up.

From 0.5.3 it isn’t necessary to set the code coverage path in VS Code’s settings (see setup #3 above). If this path is not set then the extension will attempt to find the codecoverage.json file somewhere in the workspace.

The codeCoveragePath key in the AL Test Runner config file is still required, but has a default value which will be fine in most cases.

Ideas

Ideas and feedback welcome as issues (or better yet, pull requests) in the repo on GitHub. These are some that I might work on.

  • Maybe a setting to enter a glob pattern to include/exclude certain files and/or paths from the summary
  • Smoother setup of the different settings that are required – I’ve tried to provide sensible default values but they are a few things to enter correctly to get it working
  • Currently the code coverage file is overwritten with each test run – it would be more useful if this was cumulative so that running a single test didn’t overwrite all the results from a previous test run. If you want a complete overview you have to run the whole suite – but then maybe that isn’t a bad thing
  • Perhaps an overall code coverage percentage for the whole project as well as per app (with the above caveat that I’m sceptical about fixed code coverage targets)
  • A CodeLens at the top of an object with its percentage code coverage that toggles the highlighting on/off when clicked

11 thoughts on “Measuring Code Coverage in Business Central with AL Test Runner

  1. I am getting an error when running the tests. I am trying to get the code coverage part working but run into this error:
    “Downloading code coverage to .//.altestrunner//codecoverage.json
    Cannot bind argument to parameter ‘InputObject’ because it is null.”
    I cannot see the codecoverage json file anywhere.

    Like

    1. Hi,

      Have you installed the Test Runner Service app into the container and set the testRunnerServiceUrl in the config file? If you are running Docker locally (on the same machine as VS Code) then you should be able to install the app with the command in VS Code. Otherwise download it from here: https://github.com/jimmymcp/test-runner-service/raw/master/James%20Pearson_Test%20Runner%20Service.app

      Related info here: https://jpearson.blog/2020/07/30/debugging-business-central-tests-with-al-test-runner/

      Liked by 1 person

      1. Hi James.
        Thanks for the reply.

        I have installed the Test Runner Service and I have verified that by trying to install it again. It tells me that it cannot install the service because it is already installed for the tenant default. I am running Docker locally.

        I get the same error when trying to debug a test. I followed your instructions in the blog post ‘Debugging Business Central Tests with AL Test Runner’ and I get the same error ‘Cannot bind argument to parameter ‘InputObject’ because it is null.’.

        I noticed that in the ‘Web Service’ tab in BC the OData V4 URL field is filled with this url: ‘https://docs.microsoft.com/en-gb/dynamics365/business-central/dev-itpro/developer/devenv-creating-and-interacting-with-odatav4-unbound-action’. I believe that might be causing the issue since the app will not find the service. Could that be the case? That would explain that both the code coverage and the debugging of the tests gets the same error.

        Regards
        Simmi

        Like

      2. This is the value:
        “testRunnerServiceUrl”: “https://tollakerfi17:7048/BC/ODataV4/TestRunner?Company=island&tenant=default”
        I copied the Cronus company to another company because the Icelandic character in it was causing me problems. I named the new company ‘island’. After I did that the error I had with the Icelandic character disappeared.

        BTW: In your ‘debugging’ blog post you mention that the url should contain ‘..TestRunner_RunTest..’ and not just ‘..TestRunner..’. Unrelated, but do I need to change that when switching from being able to debug and wanting code coverage?

        Like

      3. Great, glad that it’s working. No, _RunTest is not required any more. The same URL can be used for debugging and code coverage.

        Like

      4. But, it isn’t working 😀
        I am still getting the ‘Cannot bind argument to parameter ‘InputObject’ because it is null.’ error and I have no idea why.

        Like

      5. Ah my bad 🙄

        So, the test runner service app has been installed? Can you see it in the list of web services in BC? And the web service is available at the URL you posted? I noticed that you’ve put https in the URL – is that right? Are the web services being served over https or http?

        Like

  2. Hi James

    I am trying to set up the code coverage but I am getting this error:

    “Resource not found for the segment ‘TestRunner_GetCodeCoverage’. CorrelationId: 6290b49f-b616-42bc-851a-a96118d1dff5.”

    Do you have any idea what could be causing this?

    Best Regards

    Like

Leave a Reply to Gummi Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s