Using Code Coverage in Business Central Development

Intro

Sample code coverage summary

In the latest version of AL Test Runner I’ve added an overall percentage code coverage and totals for number of lines hit and number of lines. I’ve hesitated whether to add this in previous versions. Let me explain why.

Measuring Code Coverage

First, what these stats actually are. From right to left:

Code Coverage 79% (501/636)
  1. The total number of code lines in objects which were hit by the tests
  2. The total number of lines hit by the tests
  3. The percentage of the code lines hit in objects which were hit at least once

Notice that the stats only include objects which were hit by your tests. You might have a codeunit with thousands of lines of code, but if it isn’t hit at all by your tests it won’t count in the figures. That’s just how the code coverage stats come back from Business Central. Take a look at the file that is downloaded from the test runner if you’re interested (by default it’s saved as codecoverage.json in the .altestrunner folder).

It is important to bear this is mind when you are looking at the headline code coverage figure. If you have hundreds of objects and your tests only hit the code in one of them, but all of the code in that object – the code coverage % will be a misleading 100%. (If you don’t like that you’ll have to take it up with Microsoft, not me).

What Code Coverage Isn’t Good For

OK, but assuming that my tests hit at least some of the code in the most important objects then the overall percentage should be more or less accurate right? In which case we should be able to get an idea of how good the tests are for this project? No.

Code Coverage ≠ Test Quality

The fact that one or more tests hits a block of code does not tell you anything about how good those tests are. The tests could be completely meaningless and the code coverage % alone would not tell you. For example;

procedure CalcStandardDeviation(Values: List of [Decimal]): Decimal
var
    Value, Sum, Mean, SumOfVariance : Decimal;
begin
    foreach Value in Values do
        Sum += Value;
    Mean := Sum / Values.Count();
    foreach Value in Values do
        SumOfVariance += Power((Value - Mean), 2);
    exit(SumOfVariance / Values.Count());
end;

[Test]
procedure TestCalcStandardDeviation()
var
    Values: List of [Decimal];
begin
    Values.Add(1);
    Values.Add(3);
    Values.Add(8);
    Values.Add(12);

    CalcStandardDeviation(Values);
end;

Code coverage? 100% ✅

Does the code work? No ❌ The calculation of the standard deviation is wrong. It is a pointless test, it executes the code but doesn’t verify the result and so doesn’t identify the problem. (In case you’re wondering the result should be the square root of SumOfVariance).

Setting a Target for Code Coverage

What target should we set for code coverage in our projects? Don’t.

Why not? There are a couple of good reasons.

  1. There is likely to be some code in your project that you don’t want to test
  2. You might inadvertently encourage some undesired behaviour from your developers

Why Wouldn’t You Test Some of Your Code?

Personally, I try to avoid testing any code on pages. Tests which rely on test page objects take significantly longer to run, they can’t be debugged with AL Test Runner and I try to minimise the code that I write in pages anyway. Usually I don’t test any of:

  • Code in action triggers
  • Lookup, Drilldown, AssistEdit or page field validation triggers
  • OnOpen, OnClose, OnAfterGetRecord
  • …you get the idea, any of the code on a page

You might also choose not to test code that calls a 3rd party service. You don’t want your tests to become dependent on some other service being available, it is likely to slow the test down and you might end up paying for consumption of the service.

I would test the code that handles the response from the 3rd party but not the code that actually calls it e.g. not the code that sends the HTTP request or writes to a file.

Triggers in Install or Upgrade codeunits will not be tested. You can test the code that is called from those triggers, but not the triggers themselves.

Developing to a Target

When a measure becomes a target, it ceases to be a good measure.

Marilyn Strathern

If we already know that we have some code that we will not write tests for then it doesn’t make a lot of sense to set a hard target of 100%. But, what other number can you pick? Imagine two apps:

  1. An app that is purely responsible for handling communication with some Azure Functions. Perhaps the majority of the code in that app is working with HTTP clients, headers and responses. It might not be practical to achieve code coverage of more than 50%
  2. An app that implements a new sales price mechanism. It is pure AL code and the code is almost entirely in codeunits. It might be perfectly reasonable to expect code coverage of 95%

It doesn’t make sense to have a headline target for the developers to work to on both projects. Let’s say we’ve agreed as a team that we must have code coverage of at least 75%. We might incentivise developers on the first project to write some nonsense tests just to artificially boost the code coverage.

Meanwhile on the second project some developers might feel safe skipping writing tests for some important new code because the code coverage is already at 80%.

Neither of these scenarios is great, but, in fairness, the developers are doing what we’ve asked them to.

What is Code Coverage Good For?

So what is code coverage good for? It helps to identify objects that have a lot of lines which aren’t hit by tests. That’s why the output is split by object and includes the path to the source file. You can jump to the source file with Alt+Click.

Highlight the lines which were hit by the previous test run with the Toggle Code Coverage command. That way you can make an informed opinion about whether you ought to write some more tests for this part of the code or whether it is fine as it is.

50% code coverage might be fine when 1 out of 2 lines has been hit. It might not be fine when 360 out of 720 lines have been hit – but that’s for you to decide.

Further Reading

https://martinfowler.com/bliki/TestCoverage.html

Get Errors from a Docker Container Event Log

“You cannot sign in due to a technical issue. Contact your system administrator.”

Business Central

Terrific. This is in a local Docker container, so I am the system administrator. Give me a second while I contact myself…

…nope, myself didn’t know what the problem was either.

It could be that the license has expired, maybe there is something wrong with the tenant, the service tier hasn’t been able to start, who knows? You should probably start by looking for errors in the event log of the container.

Maybe I’m missing a trick and there is an easier way to do this(?) but I look through the event log with PowerShell. You can run this command inside the container:

Get-EventLog Application -EntryType Error

That will return all the errors that have been logged in the Application log. Two problems though:

  1. The list might be massive
  2. You can’t see the full text of the messages

You can add the Newest parameter to specify the number of most recent messages that you want to return. Then you probably want to write the full text of the message so that you can actually read it.

Get-EventLog Application -EntryType Error -Newest 1 | % {$_.Message}

Cool – although you still need to open a PowerShell prompt inside the container to run those commands. It would be nice if we didn’t need to do that. We can use docker exec to run a command against a local container from the outside.

docker exec [container name] powershell 'Get-EventLog Application -EntryType Error -Newest 1 | % {$_.Message}'

Now we’re getting somewhere. But of course, you don’t want to be typing all that each time. I’ve declared a PowerShell function in my profile file (run code $profile in a PowerShell prompt to open the profile file in VS Code).

function Get-ContainerErrors {
  param(
    [Parameter(Mandatory = $true)]
    [string]$ContainerName,
    [Parameter(Mandatory = $false)]
    [int]$Newest = 1
  )
  docker exec $ContainerName powershell ("Get-EventLog Application -EntryType Error -Newest $Newest" + ' | % {$_.Message;''**********''}')
}

Declaring it in the profile file means that the function will always be available in a PowerShell prompt. The container name parameter must be supplied and optionally I can ask for more than just the latest one error. The string of asterisks is just to indicate where one log message ends and another begins.

Show Table Data from AL Test Runner

Why?

If you are writing Business Central automated tests then you are probably creating some data as part of those tests. Sometimes it can be useful to see the records that have been created, especially if you are trying to create the correct scenario for the GIVEN part of your test and are experimenting with the library codeunits in that part of the system.

Of course you can debug your tests and most of the time being able to see the variable values in the debugger is all you need. Other times it would be better to be able to see the whole table e.g. “why were there no Reservation Entries in the filter?” Opening the table, applying the filters and taking a look yourself can be faster than trawling through the debugger and tweaking the test code. See here for a little background.

What?

While you are debugging your test you can open a new web client session and navigate directly to the table by adding &table=[table id] to the end of the URL. You don’t need anything just to do that – just a browser. v0.5.4 of AL Test Runner adds a new command to launch the browser to the correct table id from the code. Just right click on the variable name and choose Show Table Data.

Table Locks

Depending on what data you are reading and when you are reading it you might find that the table is locked and the web client will not open it. There are a few things you can do in that case:

  • Try stepping through the code until the lock has been released on the table that you are interested in
  • Add a Commit after the data has been inserted (don’t worry – even code that is explicitly committed during a test is rolled back at the end)
  • Use a SQL query to read uncommitted data

Update Test Runner Service

You need v0.1.2 of the Test Runner Service app (because it adds a new method to return the table id for a given table name). You can install it using the Install Test Runner Service command in VS Code. If you’ve already got an older version installed you’ll need to start the app data upgrade yourself to complete the installation. You can get the app from here: https://github.com/jimmymcp/test-runner-service/raw/master/James%20Pearson_Test%20Runner%20Service.app

Video

If you prefer your demos prefaced with questionable accents I’ve got you covered here…

Increase ODataServicesOperationTimeout for Longer Debugging

TL;DR

Invoke-ScriptInBCContainer [container name] {Set-NavServerConfiguration bc -KeyName ODataServicesOperationTimeout -KeyValue 00:20:00 -ApplyTo All}

Timeout Error

If you use AL Test Runner to debug your tests then you are using the OData services to run the test in the background. OData calls have a timeout that is determined by the ODataServicesOperationTimeout key in the service tier configuration. The timeout is set to 8 minutes by default.

This means that a debug session will be closed after 8 minutes and if you haven’t finished you will receive an error like this:

The operation has been canceled because it took longer to generate rows than the specified threshold (00:08:00). Refine your filters to include less data and try again.

Debugging something for more than 8 minutes isn’t a happy place to be – but it happens. You step into a posting routine, the guts of some complex calculations or you aren’t really sure where to start and have to step through loads of code to try to narrow down the field of investigation. That was me this morning.

You can increase the timeout, for longer, blissfully uninterrupted debugging sessions. Yay. You can use the above command (on the Docker host) to set a new value for the timeout in hh:mm:ss. I don’t know if there is a maximum limit on the timeout but if you need more than 20 minutes, as per the example, then you have my sympathy!

Upgrade Your StrMenus to Confirmation Pages

Exhibit A: StrMenu when posting a sales document

Prompting the User

Sometimes we want to ask users to make a decision before executing some business logic. Often we just want a yes/no answer and can use a simple Confirm.

But what if you can’t phrase the question to give a yes/no answer? Or if there are more than two options for the user to select between?

Traditionally we’ve used the built-in StrMenu command for that. Think of the menu that you’re given when posting a sales, purchase or warehouse document.

StrMenu is nice and easy to use. Just provide a comma-separated string of the options that you want the user to select between optionally with a default and some instructions and get an integer value back. 0 indicates that the user clicked the Cancel button, otherwise the return value corresponds to the options in the string.

Selection := StrMenu(ShipInvoiceQst, DefaultOption);
Ship := Selection in [1, 3];
Invoice := Selection in [2, 3];
if Selection = 0 then
    exit(false);

That’s cool, it works nicely. Let’s not over-complicate things unnecessarily, but you might want to try out an alternative in some situations.

Shortcomings

What’s not to love? Give the user some options, get their selection back. Well…

UI

You’ve got no control over the prompt that the user is given. That might be a good thing if you don’t care what the prompt looks like, or might be a problem if you’ve got specific requirements.

Extensibility

It’s difficult to allow someone else to extend. You could have an event before and after the StrMenu I suppose – allow someone to change the string and read the return value. If you’ve got more than one subscriber this is going to get messy pretty quickly though.

Maintenance

Comma-separated values – you should put the values in a label so that it can be translated, but what if you want to use those values more than once? Maintain them and translate them in more than one place? A set of possible values is a first class citizen in AL now that we’ve got enums.

Confirmation Pages

An alternative to StrMenu is to use a confirmation page to display the values defined by an enum. Have that enum implement an interface and you’ve got a better recipe for maintaining your code and allowing others to extend it.

Example

Imagine that you are creating something to allow users to interact with phone numbers. We’ll have some assist-edit or drilldown code on a phone number field and allow the user to take some action.

Initially we decide that the user will be able to either:

  • Create a new contact and add the phone no. to it
  • Add the phone no. to an existing contact

Phone No. Action Enum

First, create an enum that defines the available options.

enum 50100 "Phone No. Action"
{
    Extensible = true;

    value(0; "Create a new contact")
    {
        Caption = 'Create a new contact';
    }
    value(1; "Add to existing contact")
    {
        Caption = 'Add to existing contact';
    }
}

I’ve left Extensible = true because I want to allow other devs to add new phone no. capabilities later.

ConfirmationDialog Page

Now a page to present the options in the enum to the user. Because it’s a page we’ve more control over what is displayed and how it works. Add instructional text, add the phone no. for information etc. The PageType is ConfirmationDialog. When you add an enum control to the page it will be rendered as radio buttons rather than the usual dropdown list.

page 50100 "Select Phone No. Action"
{
    PageType = ConfirmationDialog;
    ApplicationArea = All;
    UsageCategory = Administration;
    Caption = 'Select Phone No. Action';

    layout
    {
        area(Content)
        {
            group(General)
            {
                ShowCaption = false;
                InstructionalText = 'What would like to do?';
                field(PhoneNoCtrl; PhoneNo)
                {
                    ApplicationArea = All;
                    Editable = false;
                    Caption = 'Phone No.';
                }
                field(PhoneNoAction; PhoneNoAction)
                {
                    ApplicationArea = All;
                    ShowCaption = false;
                }
            }
        }
    }

    var
        PhoneNoAction: Enum "Phone No. Action";
        PhoneNo: Text;

    internal procedure Intialize(PhoneNo2: Text)
    begin
        PhoneNo := PhoneNo2;
    end;

    internal procedure GetResult(): Enum "Phone No. Action";
    begin
        exit(PhoneNoAction);
    end;
}
Sample ConfirmationDialog page

Which gives a page that looks like this. OK, looking good. I’ve also created a codeunit which initialises and calls the page and then retrieves the result.

Having fetched the selected enum value we can call code to create a new contact, have the user select an existing contact or whatever we want.

That’s great, but it could still be better. If another dev adds an enum value it will be displayed on the page correctly but how will their code get called? In a case statement? Should they subscribe to an event? No.

Phone No. Action Interface

We should define an interface that we expect all Phone No. Actions to implement. For now we probably only need one method. I’ll just call it Handle. You could imagine this being more complex in a real example depending on the type of number (Mobile, Landline), country code or some other factors.

interface "Phone No. Action"
{
    procedure Handle(PhoneNo: Text);
}

Now we make the Phone No. Action enum implement that interface. I’ve created a new codeunit for each action. Those codeunits will also implement the interface and have the guts of the business logic for each action.

enum 50100 "Phone No. Action" implements "Phone No. Action"
{
    Extensible = true;

    value(0; "Create a new contact")
    {
        Caption = 'Create a new contact';
        Implementation = "Phone No. Action" = "Create New Contact";
    }
    value(1; "Add to existing contact")
    {
        Caption = 'Add to existing contact';
        Implementation = "Phone No. Action" = "Add to Exist Contact";
    }
}

Calling the Confirmation Page and Interface

Finally, this is some sample code that calls the confirmation page to have the user make a selection and then handle their response. Assign the selected action to the interface variable and call its Handle method.

procedure Select(PhoneNo: Text)
var
    SelectAction: Page "Select Phone No. Action";
    PhoneNoAction: Interface "Phone No. Action";
begin
    SelectAction.Intialize(PhoneNo);
    if SelectAction.RunModal() = Action::OK then begin
        PhoneNoAction := SelectAction.GetResult();
        PhoneNoAction.Handle(PhoneNo);
    end;
end;

It should be pretty easy to extend this in the future. Let’s say that you want to add an option to call the number or send a message. Just

  1. Extend the enum with the new action
  2. Create a new codeunit which implements the interface and makes the phone call, sends the message or does whatever you need
  3. Link your new enum value with the new codeunit
  4. (there is no step 4, just three easy steps)

Conclusion

Obviously, this is a lot more work than just using StrMenu, especially if some of these concepts are new to you. I’m not going to tell you that you ought to do this or that it is necessarily always worth the extra effort – but it’s good to have the option.

Maybe you’d start with a StrMenu and later refactor it into a confirmation dialog when you know that you’ve got a case for it. Go wild.