Scheduling Azure DevOps Pipelines with YAML

I had the pleasure of presenting some thoughts about developing apps for SaaS with James Crowter to the Dutch Dynamics Community yesterday. We were sharing some of our experiences of the maintenance challenge that comes with having published apps on AppSource.

How can you continuously test your apps against past, current and upcoming versions of Business Central? Perhaps two ways:

  1. Slowly drive yourself to despair with the monotony of creating different versions of Business Central environments and testing manually
  2. Automate as much of the tedious infrastructure and repetitive testing work as possible so you can concentrate on some fun stuff instead

We have two main reasons to trigger the execution of the pipeline for a given branch of an app in Azure DevOps:

  1. We have changed some code
  2. Microsoft have changed some code that we depend on

If we have changed some of our own code we should run it through the pipeline to ensure that it passes our checks, the automated tests run and that the resulting .app files are versioned and signed correctly. It is easy to overlook some of these tasks and/or inadvertently break some existing functionality when making our changes. The pipeline is there to have our back.

At the same time, Microsoft are making changes to the base and system applications that we rely on. Even if we don’t have any planned changes for our apps we may need to make some code changes to accommodate what Microsoft have done to the ground underneath our feet.

With a bit of luck we’ll see this sort of thing:

warning AL0432: Method 'FilterReservFor' is marked for removal. Reason: Replaced by ProdOrderLine.SetReservationFilters(FilterReservEntry)

warning AL0432: Method 'CreateReservEntryFor' is marked for removal. Reason: Replaced by CreateReservEntryFor(ForType, ForSubtype, ForID, ForBatchName, ForProdOrderLine, ForRefNo, ForQtyPerUOM, Quantity, QuantityBase, ForReservEntry)

We’re using a method that Microsoft are making obsolete and will be removed at some point in the future. No need to panic, but be aware that you should switch to the new method. Very civilised. Thanks.

With less luck we’ll find that Microsoft have introduced a change that breaks our app in some way – with a compilation error or unintended behaviour. Either way, it’s something that we want to know about.

Scheduling pipelines can help with that.

Typically we:

  • Develop against a W1 version of the latest sandbox image, run pipelines against our latest commits against mcr.microsoft.com/businesscentral/sandbox with a continuous integration trigger
  • Migrate changes backwards to BC14 and BC13 compatible versions of our apps, run pipelines against appropriate Docker images for those versions
  • Have separate branches which we rebase onto the latest commit to run pipelines against bcinsider.azurecr.io/bcsandbox and bcinsider.azurecr.io/bcsandbox-master with a schedule

The continuous integration trigger is straightforward enough. At the top of our .azure-pipelines.yml we have:

trigger:
  - '*'

The schedule is defined in a separate section of the yml file, like this:

schedules:
  - cron: 0 3 * * Sun
    displayName: Schedule insider builds
    branches:
      include: ['build/insider', 'build/insider-master']
    always: true

Those branches are the ones that are set to build against the insider Docker images. I hadn’t come across cron before, but it’s pretty simple. The schedule is defined as:

  • Minute
  • Hour
  • Day of month
  • Month
  • Day of week

Our schedule comes out as 03:00 every Sunday. Asterisks stand for any value. https://crontab.guru/ is useful for getting your head around the format.

The branches key defines which branches are included in the schedule and the always indicates that we always want to run the pipeline, even if there haven’t been any code changes since it was last run.

AL Test Runner 0.1.15

This is a brief update about the AL Test Runner extension that I’ve been working on for VS Code. I’ve had some great feedback and suggestions via GitHub, a pull request correcting me being a muppet and 300-odd installs so far. Thanks to everyone who has got involved to improve it.

This version includes a new setting “Publish Before Test” which allows you to invoke either the “Publish without debugging” or “Rapid Application Publish without debugging” commands in the AL Language extension.

If you’re aiming for a quick write test->run test->write code->run test cycle then you might like to try setting this option to “Rapid application publish”. Create a test->hit Ctrl+Alt+T (rapid publish and run the current test)->see the test fail->write the code->run the test…

Rapid application publishing before running tests

Obviously, the above is a silly example. Don’t make your tests pass by simply commenting out any code that causes it to fail 😉 You’ll still need to do a full publish from time to time as not all changes are pushed in a rapid publish.

Wish List

The main improvement to the extension would come in the form of support for remote development. There are, I think, two key scenarios:

  • Visual Studio Code is running locally but Docker host is not i.e. apps are being published to some central server
    • The ALTestRunner module would need to be installed on the Docker host (maybe served from the PowerShell Gallery?) and the PowerShell commands invoked on the server (through SSH? through a PowerShell session?)
  • Remote Visual Studio Code development – where both VS Code and the Docker host are running remotely somewhere
    • I don’t know enough about this way of working yet (we’ve only ever developed on Docker containers running on our own laptops) but with AL Test Runner installed on the remote end maybe this sort of works already?

In both cases I think the results file would need to passed back somehow to the developer’s machine in order to decorate the local copy of the AL file that they are working on. If you know more about these scenarios than I do or have other suggestions I’d be glad to hear from you.

I’m planning a few other bits and pieces like adding the path to file/line of a failing test in the Output window so that you can click and jump straight to it.

And with that I’m going to sign off for this year. Merry Christmas and see you in 2020!

AL Test Runner for Visual Studio Code

TL;DR

I’ve written an extension for VS Code to help run your AL tests in local Docker containers. Search for “AL Test Runner” in the extension marketplace or click here. Feedback, bugs, feature suggestions all gratefully received on the GitHub repo or james@jpearson.blog

Intro

As soon as Freddy added capability to the navcontainerhelper module to execute automated tests I was excited about the potential for:

  1. Making test execution in our build pipeline simpler and more reliable
  2. Running tests from Visual Studio Code as part while developing

I’ve written about both aspects in the past, but especially #1 recently – about incorporating automated tests into your Azure DevOps pipeline.

This post is about #2 – incorporating running tests as early as possible into your development cycle.

Finding Bugs ASAP

You’ve probably heard the idea – and it’s common sense even if you haven’t – that the cost of finding a bug in your software increases the later in the development/deployment cycle you find it.

If you realise you made a silly mistake in code that you wrote 2 minutes ago – there’s likely no harm done. Realise there is a bug in software that is now live in customers’ databases and the implications could be much greater. Potentially annoyed customers, data that now needs fixing, support cases, having to rush out a hotfix etc.

We’ve all been there. It’s not a nice place to visit. I once deleted all the (hundreds of thousands of) records in the Purch. Rcpt. Line table with a Rec.DELETEALL on a temporary table…turns out it wasn’t temporary…and I was working in the live database.

Writing automated tests can help catch problems before you release them out into the wild. They force you to think about the expected behaviour of the code and then test whether it actually behaves like that. Hopefully if the code that we push to a branch in Azure DevOps has a bug it will cause a test to fail, the artifacts won’t be published, the developer will get an email and the customer won’t be the hapless recipient of our mistake. No harm done.

However, the rising cost of finding a bug over time still applies. Especially if the developer has started working on something else or gone home. Getting back your head back into the code, reproducing and finding the bug and fixing it are harder if you’ve had a break from the code than if you went looking for it straight away.

Running Tests from VS Code

That’s why I’m keen that we run tests from VS Code as we are writing them. Write a test, see it fail, write the code, see the test pass, repeat.

I’ve written about this before. You can use tasks in VS Code to execute the required PowerShell to run the tests. The task gives you access to the current file and line no. so that you can fancy stuff like running only the current test or test codeunit.

AL Test Runner

However, I was keen to improve on this and so have started work on a VS Code extension – AL Test Runner.

Running the current test with AL Test Runner and navcontainerhelper

The goals are to:

  • Make it as simple as possible to run the current test, tests in the current codeunit or all tests in the extension with commands and keyboard shortcuts
  • Cache the test results
  • Decorate test methods according to the latest test results – pass, fail or untested
  • Provide extra details e.g. error message and callstack when hovering over the test name
  • Add a snippet to make it easier to create new tests with placeholders for GIVEN, WHEN and THEN statements

Important: this is for running tests with the navcontainerhelper PowerShell module against a local Docker container. Please make sure that you are using the latest version of navcontainerhelper.

Getting Started

  • Download the extension from the extension marketplace in VS Code and reload the window.
  • Open a folder containing an AL project
  • Open a test codeunit, you should notice that the names of test methods are decorated with an amber background (as there are no results available for those tests)
    • The colours for passing, failing and untested tests are configurable if you don’t like them or they don’t fit with your VS Code theme. Alternatively you can turn test decoration off altogether if you don’t like it
  • Place the cursor in a test method and run the “AL Test Runner: Run Current Test” command (Ctrl+Alt+T)
  • You should be prompted to select a debug configuration (from launch.json), company name, test suite name and credentials as appropriate (depends if you’re running BC14 or BC15, if you have multiple companies, authentication type etc.)
    • I’ve noticed that sometimes the output isn’t displayed in the new terminal when it is first created – I don’t know why. Subsequent commands always seem to show up fine 🤷‍♂️
  • Use the “ttestprocedure” to create new test methods

.gitignore

If you’re using Git then I’d recommend adding the .altestrunner folder to your .gitignore file:

.altestrunner/

Committing the config file and the test results xml files doesn’t feel like a great idea.

You Can Ditch Our Build Helper for Dynamics 365 Business Central

I’m a bit of a minimalist when it comes to tooling, so I’m always happy to ditch a tool because its functionality can be provided by something else I’m already using.

In a previous post I described how we use our Build Helper AL app to prep a test suite with the test codeunits and methods that you want to run. Either as part of a CI/CD pipeline or to run from VS Code.

Freddy K has updated the navcontainerhelper PowerShell module and improved the testing capabilities – see this post for full details.

The new extensionId parameter for the Run-TestsInBCContainer function removes the need to prepare the test suite before running the tests. Happily, that means we can dispense with downloading, publishing, installing, synchronising and calling the Build Helper app.

The next version of our own PowerShell module will read the app id from app.json and use the extensionId parameter to run the tests. Shout out to Freddy for making it easier than ever to run the tests from the shell 👍

Stop Writing Automated Tests and Get On With Some Real Code

To be fair, these weren’t the exact words that were used, but a view was expressed from the keynote stage at Directions last week along these lines. Frustration that developers now have to concern themselves with infrastructure, like Docker, and writing automated tests rather than “real” code.

I couldn’t resist a short post in response to this view.

If It Doesn’t Add Value, Stop Doing It!

First, no one is forcing you to write automated tests – apart from Microsoft, who want them with your AppSource app submission. Even then, I haven’t heard of Microsoft rejecting an app because it wasn’t accompanied by enough automated tests.

I’m an advocate of developers taking responsibility for their own practices. Don’t follow a best practice simply because someone else tells you it’s a best practice. You know your scenario, your team, your code and your customers better than anyone else. You are best placed to judge whether implementing a new practice is worth the cost of getting started with it.

AppSource aside, if you are complaining about the amount of time you have to spend on writing tests then you have no one to blame but yourself. Or maybe your boss. If you don’t see the value in writing automated tests then you probably should stop wasting your time writing them!

Automated Tests vs “Real” Code

Part of the frustration with tests seemed to be that they aren’t even “real” code. If by “real” code we are referring to the code that we deliver and sell to customers then no, tests aren’t real code.

But what are we trying to achieve? Surely working, maintainable code that adds value for our customers.

We might invest in lots of things in pursuit of that goal. Time spent manually testing, sufficient hardware to develop and test the code on, an internet connection to communicate with each other and the customer, office space to work in, training courses and materials, coffee. We’re not selling these things to the customer either but no one would question that they are necessary to achieve the goal of delivering working software. Especially the coffee.

Whether or not automated tests are “real” code is the wrong question. The important judgement is whether the time spent on writing them makes a big enough contribution to the quality of the product that you eventually ship.

I won’t make the case for automated testing here. That’s for a different post. Or a different book. Suffice to say, I do think it is worth the investment.

But We’ve Got a Backlog of Code Not Covered By Tests

One problem you might have is that you’ve got a backlog of legacy code that isn’t covered by any automated tests. Trying to write tests to cover it all will take ages. This frustration also seemed to be expressed by the speaker at Directions. It even got a round of applause from some of the Directions audience.

My response would be the same – you are best placed to make a pragmatic judgement. Of course it would be nice to have 100% code coverage of your tens of thousands of lines of legacy code – but if you’ll have to stop developing new features for six months to achieve it, is it worth it? Probably not.

Automated tests should give you confidence that the code works as expected. If you are already confident that your existing code works then there might be limited value in writing a suite of tests to prove it.

Try starting with tests to cover new features that you develop or bug fixes. With these cases you’ve got some code that you aren’t confident works as expected – or that you know doesn’t. Take the opportunity to document and prove the expected behaviour with some tests. Over time you’ll build a valuable suite of tests that you can run to demonstrate that each new release of your product works and that bugs haven’t been reintroduced.

With some practice you’ll find that you can use the library codeunits to create scenarios with little test code e.g. you can create a customer, item and sales order, post it and get the posted sales invoice in 2 lines of code.

Interested? More here