Measuring Code Coverage in Business Central with AL Test Runner

v0.5.0 of AL Test Runner adds some capability to measure code coverage per object and highlight the lines of code that were hit in the previous test run.

This is an example of what we’re aiming for. Running one or more tests, seeing the list of objects and the percentage of their code lines that were hit, opening those objects and highlighting those lines. This should help identify objects and code paths aren’t covered by any tests. I don’t believe code coverage should be a target in itself (maybe more on that in a separate post) but it can be a useful tool to see where you might want to bolster your test suite.

Overview

Code coverage is started and stopped before each run of run or more tests. The Test Runner Service app is called to download the code coverage details and they are saved to a JSON file. This file is read and summarised into a per-object percentage which is output with the test results. Only objects which are defined in the workspace are included – so you won’t include standard objects, but you will see test objects.

The path to each object is included so you can Alt+Click to navigate to it. A new Toggle Code Coverage command (Ctrl+Alt+C) allows you to switch the highlighting for lines which have been hit on and off.

Setup

  1. Install the Test Runner Service app with the command in Visual Studio Code. If it is already installed you will need to uninstall and unpublish the existing version first
  2. In the AL Test Runner config.json file
    • Set the path to save the code coverage JSON file to in the codeCoveragePath key. This path is relative to the folder that contains your test code e.g. .//.altestrunner//codecoverage.json to save it within the .altestrunner folder
    • Set the testRunnerServiceUrl key to the OData URL of the Test Runner web service e.g. http://test:7048/BC/ODataV4/TestRunner?company=My%20Company&tenant=default for a container named test, default tenant and company called My Company
  3. In Visual Studio Code Settings
    • Enable Code Coverage
    • Select the path to the code coverage file relative to your app code i.e. if you have your test extension in a separate top level folder you might set it to ../tests/.altestrunner/codecoverage.json This allows AL Test Runner to find and display the code coverage details from an object in your app code

0.5.1 Update

Use the Exclude Code Coverage Files to define file paths that should be excluded from the code coverage summary. This is a regex pattern which is matched against the file paths. For example, setting this to “Tests” will exclude any files with “Tests” in their path.

Test Folder Name – specify the name of the folder which contains the test app. Previously if you worked in a multi-root workspace and had an editor open at the production app it would attempt to run tests in the production app, create a config file, ask you which company to test in, prompt for credentials…meh. With this setting AL Test Runner will always run tests in the app which is contained in the folder with the name given in this setting.

Ideas

Ideas and feedback welcome as issues (or better yet, pull requests) in the repo on GitHub. These are some that I might work on.

  • Maybe a setting to enter a glob pattern to include/exclude certain files and/or paths from the summary
  • Smoother setup of the different settings that are required – I’ve tried to provide sensible default values but they are a few things to enter correctly to get it working
  • Currently the code coverage file is overwritten with each test run – it would be more useful if this was cumulative so that running a single test didn’t overwrite all the results from a previous test run. If you want a complete overview you have to run the whole suite – but then maybe that isn’t a bad thing
  • Perhaps an overall code coverage percentage for the whole project as well as per app (with the above caveat that I’m sceptical about fixed code coverage targets)
  • A CodeLens at the top of an object with its percentage code coverage that toggles the highlighting on/off when clicked

Part 2a: (Slightly) More Elegant Error Handling in Business Central

One of the underrated advantages to doing a little blogging is that you can write about a subject you know a little about and have people who actually know what they are talking about reply to tell you a better way to do it.

There’s probably a minimum threshold of credibility on the subject you need in order for people to post serious replies. If I posted some nonsense about my keyhole surgery technique I doubt I’d get helpful corrections from the Royal College of Surgeons. It would also represent something of a departure from my usual DevOps, Git, testing and BC development posts.

Anyway, I had some useful comments on my previous post about error handling – thanks.

Use the Error Message Table

Henrik Helgesen pointed out you can skip all the codeunits, activation, context, finishing… and just use the Error Message table directly. It has some a bunch of LogXYZ methods for recording errors or messages.

It has a method to determine if there are error messages to show and a method to show them. Nice and simple and, for the scenario that I outlined, probably more appropriate.

LogTestField()

In the last post I complained about the lack of TestField functionality in the Error Message Mgt. codeunit. Kilian replied to say that the method exists in BC17. I doubt that has anything to do with my post – but I’m happy to take some credit if required. It has the signature that you’d expect.

procedure LogTestField(SourceVariant: Variant; SourceFieldNo: Integer) IsLogged: Boolean

That makes the error handling code far less verbose and, crucially, we don’t have to provide a label or any translations for the error message – that’s handled by the framework.

Consistent Behaviour With Base App

LogTestField in the base app

How useful is it for partners to invest in this sort of error handling if the base app doesn’t use it? A consistent user experience might still be better than improving our error handling but departing from standard paradigms in the process. Fair point.

Although actually, it looks like more of this is coming to the base app. This is a snip of the results when searching for “LogTestField” in the base app in BC 17.1.17104.0-W1.

49 results in 2 files. OK, so probably still a long way to go to make this the default user experience but its a start. I’m hopeful that this is an area that Microsoft will pay some attention to over the next few versions, improve the framework and make it easier for us to follow their lead.

Part 2: (Slightly) More Elegant Error Handling in Business Central

Part 2 of the series that I said that I was going to write has been a long time coming. If you don’t know what I’m talking about you might want to read the first post in the series here. Unfortunately it is possible that this series reflects the functionality that it is describing: full of early promise, but on closer inspection a little convoluted and disappointing. I’ll leave you to be the judge of that.

Unfortunately, I’ve just found the framework a little annoying to work with. Maybe I’m missing the correct way to use it but it seems like there are too many objects involved without providing the functions that I was expecting. Then again, if I am missing the best way to use it then that illustrates my point – it’s just not very friendly to work with. I’ll try to make some constructive suggestions as we go.

Scenario

A quick reminder of what we’re trying to achieve here. I’ve got a journal page to record all the video calls for work and family that I’m having.* Before the journal is posted the lines are checked for lots of potential errors. Rather than presenting the user with one error at a time we are trying to batch them all together and present them in a list to be resolved all at once. I’m using the error message handling framework to do it.

*to be clear, I’m not using it. I’m sad…but not that sad**

Overview

Some basic principles to bear in mind when dealing with the Error Message Mgt. codeunit

  1. You need to trap all the errors
    • The framework provides a way of collecting messages and displaying them to the user in a list page. It doesn’t fundamentally change how error handling in BC works. If you encounter an un-trapped error the code execution will stop, transaction be rolled back etc.
    • Obviously that includes TestField() and FieldError() calls, not just Error()
  2. The Error Message framework must be activated before calling the code that you want to trap errors for
  3. Call PushContext to set the current context for which you are handling errors
  4. Call Finish to indicate that the previous context is complete
  5. You need to determine whether there are any errors to display and then, if so, display them

Example

This is where the posting of my journal batch begins. We need to activate the error handling framework and if an error is trapped in the posting codeunit then show the errors that have been collected.

There is a ShowErrors method in the Error Message Management codeunit, but its only for on-prem. Don’t ask, I don’t know. You need to use if Codeunit.Run (or a TryFunction I suppose – although don’t) to determine whether to there are any errors to show. There is a HasErrors method in the Error Message Handler codeunit but that’s also only for on-prem. Still don’t ask.

procedure Post()
var
    VideoCallBatchPost: Codeunit "Video Call Post Batch";
    ErrorMessageMgt: Codeunit "Error Message Management";
    ErrorMessageHandler: Codeunit "Error Message Handler";
begin
    ErrorMessageMgt.Activate(ErrorMessageHandler);
    if not VideoCallBatchPost.Run(Rec) then
        ErrorMessageHandler.ShowErrors();
end;

It would have been nice if there was a way to do with without declaring an extra two codeunits – but I don’t think there is.

Onto the next level on the callstack. Call PushContext with a record that gives the context within which the errors are being collected. Run the code that we want to collect errors from and then Finish. If any errors have been encountered the Finish method will throw an error with a blank error message to ensure that the transaction is rolled back to the last commit.

If Finish is called when GuiAllowed is false then SendTraceTag is called to “send a trace tag to the telemetry service”. Interesting.

local procedure PostBatch(VideoCallBatch: Record "Video Call Batch")
var
    VideoJnlLine: Record "Video Journal Line";
    ErrorMessageMgt: Codeunit "Error Message Management";
    ErrorContextElement: Codeunit "Error Context Element";
begin
    ErrorMessageMgt.PushContext(ErrorContextElement, VideoCallBatch, 0, '');
    VideoJnlLine.SetRange("Batch Name", VideoCallBatch.Name);
    VideoJnlLine.FindSet();
    repeat
       VideoJnlLine.Post();
    until VideoJnlLine.Next() = 0;

    ErrorMessageMgt.Finish(VideoCallBatch);
end;

Now into the journal line posting and all the checks that are performed on each line. I won’t copy out the entire function – it’d be a bit tedious and you can check the source code afterwards if you’re interested.

local procedure Check(var VideoJnlLine: Record "Video Journal Line")
var
    GLSetup: Record "General Ledger Setup";
    ErrorMessageMgt: Codeunit "Error Message Management";
begin
    if VideoJnlLine."No. of Participants" = 0 then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("No. of Participants"), StrSubstNo('%1 must not be 0', VideoJnlLine.FieldCaption("No. of Participants")), VideoJnlLine, VideoJnlLine.FieldNo("No. of Participants"), '');

    if VideoJnlLine."Duration (mins)" = 0 then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("Duration (mins)"), StrSubstNo('%1 must not be 0', VideoJnlLine.FieldCaption("Duration (mins)")), VideoJnlLine, VideoJnlLine.FieldNo("Duration (mins)"), '');

    if VideoJnlLine."Posting Date" = 0D then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("Posting Date"), StrSubstNo('%1 must be set', VideoJnlLine.FieldCaption("Posting Date")), VideoJnlLine, VideoJnlLine.FieldNo("Posting Date"), '');

This is where it really starts to get a bit messy. The TestFields are gone, replaced with calls to LogErrorMessage. LogError and LogSimpleErrorMessage are alternatives with slightly different signatures. Pass in the field no, error message, record and “help article code” related to the error and they will be collected by the framework.

If any errors have been logged then the Finish function (see above) will throw an (untrapped) error and prevent the journal from actually being posted.

Conclusion

I really tried to enjoy working with this. I’d like to have better error handling in our apps – but I don’t think we’re going to get round to introducing this sort of thing any time soon. The parameters on this method are too clunky.

  • It requires a “context” field no. and a “source” field no. – I’m still not clear what the difference is
  • I have to provide the error message text. That’s a problem. With TestField I can leave the system to generate the correct error text, in whatever language the client is set to. This way I have to create a label (I didn’t in my example because I’m lazy) and then translate it into different languages
  • I don’t know what I’m supposed to provide for “help article code”
  • I was hoping for an ErrorMessageMgt.TestField method. Couldn’t I just pass in my record and the field no. that I’m testing? I want to leave the framework to determine if an error needs to be logged and, if so, the correct text

You can view the changes that I’ve made since the first blog post here: https://github.com/jimmymcp/error-message-mgt/commit/6d768c5552c0fad433e75dea15a7d1d064cb040c

I’d love someone to tell me that I’ve missed how easy this framework is to work with and they’ve had a great time with it. It looked like it was going to be great but left me a bit flat. Like a roast potato that you’ve saved for your last mouthful at Sunday lunchtime only to discover it’s actually a parsnip.

Debugging Business Central Tests with AL Test Runner

TL;DR

  1. Install the Test Runner Service app (see https://github.com/jimmymcp/test-runner-service; direct download of the app file from here) or use the “Install Test Runner Service” command from VS Code to install into the Docker container specified in the config file
  2. Set the URL to the test runner service in the testRunnerServiceUrl key of the AL Test Runner config file
  3. Define a debug configuration of request type ‘attach’ in launch.json to attach the debugger to the service tier that you want to debug (should be the same service tier as specified by the testRunnerServiceUrl key)

Overview

From v0.4.0 of the AL Test Runner app it is possible to debug Business Central tests without leaving Visual Studio Code. There’s a lot of scope for improvements but if you’re interested in trying it out it’s included in the marketplace version now.

Test Runner Service App

This is a very simple app that exposes a codeunit as a web service to accept a codeunit ID and test name to run. Those values are passed to a test runner codeunit (codeunit isolation) to actually run the tests. This is so that the tests are executed in a session type of WebService which the debugger can attach to (the PowerShell runner creates a session type of ClientService).

The app is in the per tenant object range: 79150-79160 to be precise (a number picked pretty much at random). If that clashes with some other object ranges present in the database you can clone the repo and renumber the codeunits if you want. The source is here: https://github.com/jimmymcp/test-runner-service

You can use the Install Test Runner Service command in VS Code to automatically download the app and install into the container specified in the AL Test Runner config file.

The app is not code signed so you’ll need to use the -SkipVerification switch when you install it.

testRunnerServiceUrl

A new key is required in the AL Test Runner config file. This specifies the OData endpoint of the test runner service that is exposed by the Test Runner Service app. The service will be called from the VS Code terminal – so consider where the terminal is runner and where the service is hosted.

We develop against local Docker containers so the local VS Code instance will be able to access the web service without any trouble. If you develop against a remote Docker host make sure that the OData port is available externally. If you use VS Code remote development remember that the PowerShell session will be running on the VS Code server host.

The url will be in the format:

http[s]://[BC host]:[OData port]/[BC instance]/ODataV4/TestRunner_RunTest?company=[BC company]

for example against a local Docker container called bc with OData exposed on the default port of 7048 and a company name of CRONUS International Ltd.:

"testRunnerServiceUrl": "http://bc:7048/BC/ODataV4/TestRunner_RunTest?company=CRONUS%20International%20Ltd."

Debug Configuration

You will need a debug configuration of type attach in the launch.json file. This should attach the debugger to the same service as identified by the testRunnerServiceUrl key. breakOnNext should be set WebServiceClient. Currently UserPassword authentication is the only authentication method supported.

{
    "name": "Attach bc",
    "type": "al",
    "request": "attach",
    "server": "http://bc",
    "serverInstance": "bc",
    "authentication": "UserPassword",
    "breakOnError": true,
    "breakOnRecordWrite": false,
    "enableSqlInformationDebugger": true,
    "enableLongRunningSqlStatements": true,
    "longRunningSqlStatementsThreshold": 500,
    "numberOfSqlStatements": 10,
    "breakOnNext": "WebServiceClient"
}

Debugging

Codelens actions will be added at the top of test codeunits and before each test method. Set a breakpoint in the test method that you want to debug or allow the debugger to break on an error.

Clicking on Debug Test (Ctrl+Alt+D) will attach the first debug configuration specified in launch.json and call the web service to run the test with the Test Runner Service app.

Attaching the debugger and running a test from VS Code

Step in/out/over as usual. When the code execution has finished if an error was encountered the error message and callstack will be displayed in the terminal.

Limitations

There are some limitations to running tests in a web service session. Most importantly TestPage variables are not supported. There may also be some differences in the behaviour of tests in web services and the PowerShell runner.

Tip: Evaluating DateTime with Type Helper

Dates. What a nightmare. Day/Month/Year? Month/Day/Year? 24 hour time? 12 hour time? It’s almost enough to make you sympathetic to the idea of decimal time…almost.

Type Helper codeunit to the rescue. It has a method to allow you to evaluate the text of a date, time or datetime into the corresponding type according to a format that you specify.

AVariant := DateResult;
FormatString := 'ddMMyy';
if TypeHelper.Evaluate(AVariant, DateText, FormatString, '') then
  DateResult := AVariant

The first parameter is of type Variant. The actual data type that the variant contains determines whether the method will attempt to evaluate to a date, time or datetime. Unfortunately because that parameter is passed by reference (var) you have to declare a variant variable and then assign its value to another variable afterwards – but apart from that its pretty self explanatory.

See https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings for info about the formats you can use. Don’t do what I did and miss the distinction between lowercase ‘m’ (minute) and uppercase ‘M’ (month) *facepalm*