Testing¶
Charms should have tests to verify that they are functioning correctly. This document describes some of the various types of testing you may want to consider – their meaning, recommended coverage, and recommended tooling in the context of a charm.
Unit testing¶
A unit test is a test that targets an individual unit of code (function, method, class, etc.) independently. In the context of a charm, it refers to testing charm code against mock Juju APIs and mocked-out workloads as a way to validate isolated behaviour without external interactions.
Unit tests are intended to be isolating and fast to complete. These are the tests you would run every time before committing code changes.
Coverage. Unit testing a charm should cover:
how relation data is modified as a result of an event
what pebble services are running as a result of an event
which configuration files are written and their contents, as a result of an event
Tools. Unit testing a charm can be done using:
state transition testing, using the
ops
unit testing framework
Examples.
Interface testing¶
In the context of a charm, interface tests help validate charm library behavior without individual charm code against mock Juju APIs.
See more: Interface tests
Integration testing¶
An integration test is a test that targets multiple software components in interaction. In the context of a charm, it checks that the charm operates as expected when Juju-deployed by a user in a test model in a real controller.
Integration tests should be focused on a single charm. Sometimes an integration test requires multiple charms to be deployed for adequate testing, but ideally integration tests should not become end-to-end tests.
Integration tests typically take significantly longer to run than unit tests.
Coverage.
Charm actions
Charm integrations
Charm configurations
That the workload is up and running, and responsive
Upgrade sequence
Regression test: upgrade stable/candidate/beta/edge from charmhub with the locally-built charm.
Caution
When writing an integration test, it is not sufficient to simply check that Juju reports that running the action was successful; rather, additional checks need to be executed to ensure that whatever the action was intended to achieve worked.
Tools.
pytest-operator and/or
zaza
pytest-operator
¶
pytest-operator
is a Python library that provides Juju plugins for the generic Python library pytest
to facilitate the integration testing of charms.
See more:
pytest-operator
It builds a fixture called ops_test
that helps you interact with Juju through constructs that wrap around python-libjuju
.
See more:
It also provides convenient markers and command line parameters (e.g., the @pytest.mark.skip_if_deployed
marker in combination with the --no-deploy
configuration helps you skip, e.g., a deployment test in the case where you already have a deployment).
Examples.