Test Explorer in Visual Studio Code

The July 2021 release of Visual Studio Code (1.59) introduced a new testing API and Test Explorer UI. From v0.6.0 this API is used by AL Test Runner.

Test Explorer Demo

Improvements

UI

The biggest improvement is the Test Explorer view which shows your test codeunits, their test methods and the status of each.

Hovering over a test gives you three icons to run, debug or open an editor at the test.

You can run and debug all the tests in a given codeunit by hovering over the codeunit name or run and debug all tests at the top.

The filter box allows you to easily find specific tests, which I’ve found useful in projects which several test codeunits and hundreds of tests.

You can also filter to only show failed tests or only test which are present in the codeunit in the current editor. The explorer supports different ways of sorting and displaying the tests.

Icons are added into the gutter alongside test methods in the editor. Left click to run the test or right click to see this context menu with more options.

The old “Run Test” and “Debug Test” codelens actions are also still added above the test definition.

Commands & Shortcuts

A whole set of new commands are introduced with keyboard chords beginning with Ctrl + ; The existing AL Test Runner keyboard shortcuts still work but there are some nice options in the new set – like “Test: Rerun Last Run” to repeat the last run test without having to navigate to it again.

Using the Test Explorer

Using the Test Explorer is pretty self-explanatory if you’ve already been using AL Test Runner. When you open your workspace/folder the tests should be automatically discovered and loaded into the Test Explorer view. On first opening all of the tests will have no status i.e. neither pass or fail – but results from now on will be persisted.

Running one or more tests – regardless of where you run them from (Test Explorer, Command Palette, CodeLens, Keyboard Shortcut) – will start a test run. You’ll see “Running tests…” in the Status Bar.

Once the test(s) have finished running you’ll see the results at the top of the Test Explorer, “x / y tests passed (z %)”, and the status icons by each test will be updated.

If the tests do not actually run e.g. because your container isn’t started then the test run will not finish and “Running tests” will continue to spin at the bottom of the screen. You can stop the run manually from the top of the Test Explorer, fix the problem and go again.

Using Code Coverage in Business Central Development

Intro

Sample code coverage summary

In the latest version of AL Test Runner I’ve added an overall percentage code coverage and totals for number of lines hit and number of lines. I’ve hesitated whether to add this in previous versions. Let me explain why.

Measuring Code Coverage

First, what these stats actually are. From right to left:

Code Coverage 79% (501/636)
  1. The total number of code lines in objects which were hit by the tests
  2. The total number of lines hit by the tests
  3. The percentage of the code lines hit in objects which were hit at least once

Notice that the stats only include objects which were hit by your tests. You might have a codeunit with thousands of lines of code, but if it isn’t hit at all by your tests it won’t count in the figures. That’s just how the code coverage stats come back from Business Central. Take a look at the file that is downloaded from the test runner if you’re interested (by default it’s saved as codecoverage.json in the .altestrunner folder).

It is important to bear this is mind when you are looking at the headline code coverage figure. If you have hundreds of objects and your tests only hit the code in one of them, but all of the code in that object – the code coverage % will be a misleading 100%. (If you don’t like that you’ll have to take it up with Microsoft, not me).

What Code Coverage Isn’t Good For

OK, but assuming that my tests hit at least some of the code in the most important objects then the overall percentage should be more or less accurate right? In which case we should be able to get an idea of how good the tests are for this project? No.

Code Coverage ≠ Test Quality

The fact that one or more tests hits a block of code does not tell you anything about how good those tests are. The tests could be completely meaningless and the code coverage % alone would not tell you. For example;

procedure CalcStandardDeviation(Values: List of [Decimal]): Decimal
var
    Value, Sum, Mean, SumOfVariance : Decimal;
begin
    foreach Value in Values do
        Sum += Value;
    Mean := Sum / Values.Count();
    foreach Value in Values do
        SumOfVariance += Power((Value - Mean), 2);
    exit(SumOfVariance / Values.Count());
end;

[Test]
procedure TestCalcStandardDeviation()
var
    Values: List of [Decimal];
begin
    Values.Add(1);
    Values.Add(3);
    Values.Add(8);
    Values.Add(12);

    CalcStandardDeviation(Values);
end;

Code coverage? 100% ✅

Does the code work? No ❌ The calculation of the standard deviation is wrong. It is a pointless test, it executes the code but doesn’t verify the result and so doesn’t identify the problem. (In case you’re wondering the result should be the square root of SumOfVariance).

Setting a Target for Code Coverage

What target should we set for code coverage in our projects? Don’t.

Why not? There are a couple of good reasons.

  1. There is likely to be some code in your project that you don’t want to test
  2. You might inadvertently encourage some undesired behaviour from your developers

Why Wouldn’t You Test Some of Your Code?

Personally, I try to avoid testing any code on pages. Tests which rely on test page objects take significantly longer to run, they can’t be debugged with AL Test Runner and I try to minimise the code that I write in pages anyway. Usually I don’t test any of:

  • Code in action triggers
  • Lookup, Drilldown, AssistEdit or page field validation triggers
  • OnOpen, OnClose, OnAfterGetRecord
  • …you get the idea, any of the code on a page

You might also choose not to test code that calls a 3rd party service. You don’t want your tests to become dependent on some other service being available, it is likely to slow the test down and you might end up paying for consumption of the service.

I would test the code that handles the response from the 3rd party but not the code that actually calls it e.g. not the code that sends the HTTP request or writes to a file.

Triggers in Install or Upgrade codeunits will not be tested. You can test the code that is called from those triggers, but not the triggers themselves.

Developing to a Target

When a measure becomes a target, it ceases to be a good measure.

Marilyn Strathern

If we already know that we have some code that we will not write tests for then it doesn’t make a lot of sense to set a hard target of 100%. But, what other number can you pick? Imagine two apps:

  1. An app that is purely responsible for handling communication with some Azure Functions. Perhaps the majority of the code in that app is working with HTTP clients, headers and responses. It might not be practical to achieve code coverage of more than 50%
  2. An app that implements a new sales price mechanism. It is pure AL code and the code is almost entirely in codeunits. It might be perfectly reasonable to expect code coverage of 95%

It doesn’t make sense to have a headline target for the developers to work to on both projects. Let’s say we’ve agreed as a team that we must have code coverage of at least 75%. We might incentivise developers on the first project to write some nonsense tests just to artificially boost the code coverage.

Meanwhile on the second project some developers might feel safe skipping writing tests for some important new code because the code coverage is already at 80%.

Neither of these scenarios is great, but, in fairness, the developers are doing what we’ve asked them to.

What is Code Coverage Good For?

So what is code coverage good for? It helps to identify objects that have a lot of lines which aren’t hit by tests. That’s why the output is split by object and includes the path to the source file. You can jump to the source file with Alt+Click.

Highlight the lines which were hit by the previous test run with the Toggle Code Coverage command. That way you can make an informed opinion about whether you ought to write some more tests for this part of the code or whether it is fine as it is.

50% code coverage might be fine when 1 out of 2 lines has been hit. It might not be fine when 360 out of 720 lines have been hit – but that’s for you to decide.

Further Reading

https://martinfowler.com/bliki/TestCoverage.html

Get Errors from a Docker Container Event Log

“You cannot sign in due to a technical issue. Contact your system administrator.”

Business Central

Terrific. This is in a local Docker container, so I am the system administrator. Give me a second while I contact myself…

…nope, myself didn’t know what the problem was either.

It could be that the license has expired, maybe there is something wrong with the tenant, the service tier hasn’t been able to start, who knows? You should probably start by looking for errors in the event log of the container.

Maybe I’m missing a trick and there is an easier way to do this(?) but I look through the event log with PowerShell. You can run this command inside the container:

Get-EventLog Application -EntryType Error

That will return all the errors that have been logged in the Application log. Two problems though:

  1. The list might be massive
  2. You can’t see the full text of the messages

You can add the Newest parameter to specify the number of most recent messages that you want to return. Then you probably want to write the full text of the message so that you can actually read it.

Get-EventLog Application -EntryType Error -Newest 1 | % {$_.Message}

Cool – although you still need to open a PowerShell prompt inside the container to run those commands. It would be nice if we didn’t need to do that. We can use docker exec to run a command against a local container from the outside.

docker exec [container name] powershell 'Get-EventLog Application -EntryType Error -Newest 1 | % {$_.Message}'

Now we’re getting somewhere. But of course, you don’t want to be typing all that each time. I’ve declared a PowerShell function in my profile file (run code $profile in a PowerShell prompt to open the profile file in VS Code).

function Get-ContainerErrors {
  param(
    [Parameter(Mandatory = $true)]
    [string]$ContainerName,
    [Parameter(Mandatory = $false)]
    [int]$Newest = 1
  )
  docker exec $ContainerName powershell ("Get-EventLog Application -EntryType Error -Newest $Newest" + ' | % {$_.Message;''**********''}')
}

Declaring it in the profile file means that the function will always be available in a PowerShell prompt. The container name parameter must be supplied and optionally I can ask for more than just the latest one error. The string of asterisks is just to indicate where one log message ends and another begins.

Part 2a: (Slightly) More Elegant Error Handling in Business Central

One of the underrated advantages to doing a little blogging is that you can write about a subject you know a little about and have people who actually know what they are talking about reply to tell you a better way to do it.

There’s probably a minimum threshold of credibility on the subject you need in order for people to post serious replies. If I posted some nonsense about my keyhole surgery technique I doubt I’d get helpful corrections from the Royal College of Surgeons. It would also represent something of a departure from my usual DevOps, Git, testing and BC development posts.

Anyway, I had some useful comments on my previous post about error handling – thanks.

Use the Error Message Table

Henrik Helgesen pointed out you can skip all the codeunits, activation, context, finishing… and just use the Error Message table directly. It has some a bunch of LogXYZ methods for recording errors or messages.

It has a method to determine if there are error messages to show and a method to show them. Nice and simple and, for the scenario that I outlined, probably more appropriate.

LogTestField()

In the last post I complained about the lack of TestField functionality in the Error Message Mgt. codeunit. Kilian replied to say that the method exists in BC17. I doubt that has anything to do with my post – but I’m happy to take some credit if required. It has the signature that you’d expect.

procedure LogTestField(SourceVariant: Variant; SourceFieldNo: Integer) IsLogged: Boolean

That makes the error handling code far less verbose and, crucially, we don’t have to provide a label or any translations for the error message – that’s handled by the framework.

Consistent Behaviour With Base App

LogTestField in the base app

How useful is it for partners to invest in this sort of error handling if the base app doesn’t use it? A consistent user experience might still be better than improving our error handling but departing from standard paradigms in the process. Fair point.

Although actually, it looks like more of this is coming to the base app. This is a snip of the results when searching for “LogTestField” in the base app in BC 17.1.17104.0-W1.

49 results in 2 files. OK, so probably still a long way to go to make this the default user experience but its a start. I’m hopeful that this is an area that Microsoft will pay some attention to over the next few versions, improve the framework and make it easier for us to follow their lead.

Part 2: (Slightly) More Elegant Error Handling in Business Central

Part 2 of the series that I said that I was going to write has been a long time coming. If you don’t know what I’m talking about you might want to read the first post in the series here. Unfortunately it is possible that this series reflects the functionality that it is describing: full of early promise, but on closer inspection a little convoluted and disappointing. I’ll leave you to be the judge of that.

Unfortunately, I’ve just found the framework a little annoying to work with. Maybe I’m missing the correct way to use it but it seems like there are too many objects involved without providing the functions that I was expecting. Then again, if I am missing the best way to use it then that illustrates my point – it’s just not very friendly to work with. I’ll try to make some constructive suggestions as we go.

Scenario

A quick reminder of what we’re trying to achieve here. I’ve got a journal page to record all the video calls for work and family that I’m having.* Before the journal is posted the lines are checked for lots of potential errors. Rather than presenting the user with one error at a time we are trying to batch them all together and present them in a list to be resolved all at once. I’m using the error message handling framework to do it.

*to be clear, I’m not using it. I’m sad…but not that sad**

Overview

Some basic principles to bear in mind when dealing with the Error Message Mgt. codeunit

  1. You need to trap all the errors
    • The framework provides a way of collecting messages and displaying them to the user in a list page. It doesn’t fundamentally change how error handling in BC works. If you encounter an un-trapped error the code execution will stop, transaction be rolled back etc.
    • Obviously that includes TestField() and FieldError() calls, not just Error()
  2. The Error Message framework must be activated before calling the code that you want to trap errors for
  3. Call PushContext to set the current context for which you are handling errors
  4. Call Finish to indicate that the previous context is complete
  5. You need to determine whether there are any errors to display and then, if so, display them

Example

This is where the posting of my journal batch begins. We need to activate the error handling framework and if an error is trapped in the posting codeunit then show the errors that have been collected.

There is a ShowErrors method in the Error Message Management codeunit, but its only for on-prem. Don’t ask, I don’t know. You need to use if Codeunit.Run (or a TryFunction I suppose – although don’t) to determine whether to there are any errors to show. There is a HasErrors method in the Error Message Handler codeunit but that’s also only for on-prem. Still don’t ask.

procedure Post()
var
    VideoCallBatchPost: Codeunit "Video Call Post Batch";
    ErrorMessageMgt: Codeunit "Error Message Management";
    ErrorMessageHandler: Codeunit "Error Message Handler";
begin
    ErrorMessageMgt.Activate(ErrorMessageHandler);
    if not VideoCallBatchPost.Run(Rec) then
        ErrorMessageHandler.ShowErrors();
end;

It would have been nice if there was a way to do with without declaring an extra two codeunits – but I don’t think there is.

Onto the next level on the callstack. Call PushContext with a record that gives the context within which the errors are being collected. Run the code that we want to collect errors from and then Finish. If any errors have been encountered the Finish method will throw an error with a blank error message to ensure that the transaction is rolled back to the last commit.

If Finish is called when GuiAllowed is false then SendTraceTag is called to “send a trace tag to the telemetry service”. Interesting.

local procedure PostBatch(VideoCallBatch: Record "Video Call Batch")
var
    VideoJnlLine: Record "Video Journal Line";
    ErrorMessageMgt: Codeunit "Error Message Management";
    ErrorContextElement: Codeunit "Error Context Element";
begin
    ErrorMessageMgt.PushContext(ErrorContextElement, VideoCallBatch, 0, '');
    VideoJnlLine.SetRange("Batch Name", VideoCallBatch.Name);
    VideoJnlLine.FindSet();
    repeat
       VideoJnlLine.Post();
    until VideoJnlLine.Next() = 0;

    ErrorMessageMgt.Finish(VideoCallBatch);
end;

Now into the journal line posting and all the checks that are performed on each line. I won’t copy out the entire function – it’d be a bit tedious and you can check the source code afterwards if you’re interested.

local procedure Check(var VideoJnlLine: Record "Video Journal Line")
var
    GLSetup: Record "General Ledger Setup";
    ErrorMessageMgt: Codeunit "Error Message Management";
begin
    if VideoJnlLine."No. of Participants" = 0 then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("No. of Participants"), StrSubstNo('%1 must not be 0', VideoJnlLine.FieldCaption("No. of Participants")), VideoJnlLine, VideoJnlLine.FieldNo("No. of Participants"), '');

    if VideoJnlLine."Duration (mins)" = 0 then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("Duration (mins)"), StrSubstNo('%1 must not be 0', VideoJnlLine.FieldCaption("Duration (mins)")), VideoJnlLine, VideoJnlLine.FieldNo("Duration (mins)"), '');

    if VideoJnlLine."Posting Date" = 0D then
        ErrorMessageMgt.LogErrorMessage(VideoJnlLine.FieldNo("Posting Date"), StrSubstNo('%1 must be set', VideoJnlLine.FieldCaption("Posting Date")), VideoJnlLine, VideoJnlLine.FieldNo("Posting Date"), '');

This is where it really starts to get a bit messy. The TestFields are gone, replaced with calls to LogErrorMessage. LogError and LogSimpleErrorMessage are alternatives with slightly different signatures. Pass in the field no, error message, record and “help article code” related to the error and they will be collected by the framework.

If any errors have been logged then the Finish function (see above) will throw an (untrapped) error and prevent the journal from actually being posted.

Conclusion

I really tried to enjoy working with this. I’d like to have better error handling in our apps – but I don’t think we’re going to get round to introducing this sort of thing any time soon. The parameters on this method are too clunky.

  • It requires a “context” field no. and a “source” field no. – I’m still not clear what the difference is
  • I have to provide the error message text. That’s a problem. With TestField I can leave the system to generate the correct error text, in whatever language the client is set to. This way I have to create a label (I didn’t in my example because I’m lazy) and then translate it into different languages
  • I don’t know what I’m supposed to provide for “help article code”
  • I was hoping for an ErrorMessageMgt.TestField method. Couldn’t I just pass in my record and the field no. that I’m testing? I want to leave the framework to determine if an error needs to be logged and, if so, the correct text

You can view the changes that I’ve made since the first blog post here: https://github.com/jimmymcp/error-message-mgt/commit/6d768c5552c0fad433e75dea15a7d1d064cb040c

I’d love someone to tell me that I’ve missed how easy this framework is to work with and they’ve had a great time with it. It looked like it was going to be great but left me a bit flat. Like a roast potato that you’ve saved for your last mouthful at Sunday lunchtime only to discover it’s actually a parsnip.