Measuring Code Coverage in Business Central with AL Test Runner

development 3 Minutes

v0.5.0 of AL Test Runner adds some capability to measure code coverage per object and highlight the lines of code that were hit in the previous test run.

This is an example of what we’re aiming for. Running one or more tests, seeing the list of objects and the percentage of their code lines that were hit, opening those objects and highlighting those lines. This should help identify objects and code paths aren’t covered by any tests. I don’t believe code coverage should be a target in itself (maybe more on that in a separate post) but it can be a useful tool to see where you might want to bolster your test suite.

  • Example use of code coverage

Overview

Code coverage is started and stopped before each run of run or more tests. The Test Runner Service app is called to download the code coverage details and they are saved to a JSON file. This file is read and summarised into a per-object percentage which is output with the test results. Only objects which are defined in the workspace are included – so you won’t include standard objects, but you will see test objects.

The path to each object is included so you can Alt+Click to navigate to it. A new Toggle Code Coverage command (Ctrl+Alt+C) allows you to switch the highlighting for lines which have been hit on and off.

Setup

  1. Install the Test Runner Service app with the command in Visual Studio Code. If it is already installed you will need to uninstall and unpublish the existing version first
  2. In the AL Test Runner config.json file
    • Set the path to save the code coverage JSON file to in the codeCoveragePath key. This path is relative to the folder that contains your test code e.g. .//.altestrunner//codecoverage.json to save it within the .altestrunner folder
    • Set the testRunnerServiceUrl key to the OData URL of the Test Runner web service e.g. http://test:7048/BC/ODataV4/TestRunner?company=My%20Company&tenant=default for a container named test, default tenant and company called My Company
  3. In Visual Studio Code Settings
    • Enable Code Coverage
    • [Edit: this is now optional – see 0.5.3 update below] Select the path to the code coverage file relative to your app code i.e. if you have your test extension in a separate top level folder you might set it to ../tests/.altestrunner/codecoverage.json This allows AL Test Runner to find and display the code coverage details from an object in your app code

0.5.1 Update

Use the Exclude Code Coverage Files to define file paths that should be excluded from the code coverage summary. This is a regex pattern which is matched against the file paths. For example, setting this to “Tests” will exclude any files with “Tests” in their path.

Test Folder Name – specify the name of the folder which contains the test app. Previously if you worked in a multi-root workspace and had an editor open at the production app it would attempt to run tests in the production app, create a config file, ask you which company to test in, prompt for credentials…meh. With this setting AL Test Runner will always run tests in the app which is contained in the folder with the name given in this setting.

0.5.3 Update

Some of the early feedback from people who were trying to enable code coverage was that it was a bit of a game. And not in a good, fun way. More like Ludo. You’re trying to line up all your pieces but every time you think you’ve got them where you want them someone else lands on them and messes everything up.

From 0.5.3 it isn’t necessary to set the code coverage path in VS Code’s settings (see setup #3 above). If this path is not set then the extension will attempt to find the codecoverage.json file somewhere in the workspace.

The codeCoveragePath key in the AL Test Runner config file is still required, but has a default value which will be fine in most cases.

Ideas

Ideas and feedback welcome as issues (or better yet, pull requests) in the repo on GitHub. These are some that I might work on.

  • Maybe a setting to enter a glob pattern to include/exclude certain files and/or paths from the summary
  • Smoother setup of the different settings that are required – I’ve tried to provide sensible default values but they are a few things to enter correctly to get it working
  • Currently the code coverage file is overwritten with each test run – it would be more useful if this was cumulative so that running a single test didn’t overwrite all the results from a previous test run. If you want a complete overview you have to run the whole suite – but then maybe that isn’t a bad thing
  • Perhaps an overall code coverage percentage for the whole project as well as per app (with the above caveat that I’m sceptical about fixed code coverage targets)
  • A CodeLens at the top of an object with its percentage code coverage that toggles the highlighting on/off when clicked

Published by james

Published

20 thoughts on “Measuring Code Coverage in Business Central with AL Test Runner”

  1. Pingback: Measuring Code Coverage in Business Central with AL Test Runner - 365 Community

  3. I am getting an error when running the tests. I am trying to get the code coverage part working but run into this error:
    “Downloading code coverage to .//.altestrunner//codecoverage.json
    Cannot bind argument to parameter ‘InputObject’ because it is null.”
    I cannot see the codecoverage json file anywhere.

    Like

    Reply

    1. Hi,

      Have you installed the Test Runner Service app into the container and set the testRunnerServiceUrl in the config file? If you are running Docker locally (on the same machine as VS Code) then you should be able to install the app with the command in VS Code. Otherwise download it from here: https://github.com/jimmymcp/test-runner-service/raw/master/James%20Pearson_Test%20Runner%20Service.app

      Related info here: https://jpearson.blog/2020/07/30/debugging-business-central-tests-with-al-test-runner/

      Liked by 1 person

      Reply

      1. Hi James.
        Thanks for the reply.

        I have installed the Test Runner Service and I have verified that by trying to install it again. It tells me that it cannot install the service because it is already installed for the tenant default. I am running Docker locally.

        I get the same error when trying to debug a test. I followed your instructions in the blog post ‘Debugging Business Central Tests with AL Test Runner’ and I get the same error ‘Cannot bind argument to parameter ‘InputObject’ because it is null.’.

        I noticed that in the ‘Web Service’ tab in BC the OData V4 URL field is filled with this url: ‘https://docs.microsoft.com/en-gb/dynamics365/business-central/dev-itpro/developer/devenv-creating-and-interacting-with-odatav4-unbound-action’. I believe that might be causing the issue since the app will not find the service. Could that be the case? That would explain that both the code coverage and the debugging of the tests gets the same error.

        Regards
        Simmi

        Like

      3. This is the value:
        “testRunnerServiceUrl”: “https://tollakerfi17:7048/BC/ODataV4/TestRunner?Company=island&tenant=default”
        I copied the Cronus company to another company because the Icelandic character in it was causing me problems. I named the new company ‘island’. After I did that the error I had with the Icelandic character disappeared.

        BTW: In your ‘debugging’ blog post you mention that the url should contain ‘..TestRunner_RunTest..’ and not just ‘..TestRunner..’. Unrelated, but do I need to change that when switching from being able to debug and wanting code coverage?

        Like

      4. Great, glad that it’s working. No, _RunTest is not required any more. The same URL can be used for debugging and code coverage.

        Like

      5. But, it isn’t working 😀
        I am still getting the ‘Cannot bind argument to parameter ‘InputObject’ because it is null.’ error and I have no idea why.

        Like

      6. Ah my bad 🙄

        So, the test runner service app has been installed? Can you see it in the list of web services in BC? And the web service is available at the URL you posted? I noticed that you’ve put https in the URL – is that right? Are the web services being served over https or http?

        Like

  4. Hi James

    I am trying to set up the code coverage but I am getting this error:

    “Resource not found for the segment ‘TestRunner_GetCodeCoverage’. CorrelationId: 6290b49f-b616-42bc-851a-a96118d1dff5.”

    Do you have any idea what could be causing this?

    Best Regards

    Like

    Reply

  5. James, it looks like the standaard CC functionality is broken in BC 18x. We use CC in our build pipeline when invoking units tests through PS. Can you confirm this is still working in your extension?

    Like

    Reply

  6. Is there an ability in any way to generate code coverage report when pipelines are run?

    We have Automated AL tests run and report file, and we are looking to expand that to also have code coverage report.

    Thanks

    Like

    Reply

  7. It is generating the codecoverage.json correctly and without any error but in the output it always displays:

    Code Coverage 0% (0/0)

    Like

    Reply

    1. Hi,

      Please check the .altestrunner/config.json file. What value do you have for codeCoveragePath? Also, if you have a multiroot workspace with separate production and test apps do you have the name of the folder containing the test code in the Test Folder Name setting in VS Code settings?

      Like

      Reply

  8. Hello James, I am trying to run AL Automated testing and get code coverage. But i don’t get it. I am getting error. Please can you give me a demo project with Testing so i can run and check it. Thank you

    Like

    Reply

Leave a comment