Code Coverage: Ensuring Quality in Development Testing
Application development requires continuous and comprehensive testing. When we neglect testing functionalities and scenarios, we open the door to the emergence of bugs. Through Continuous Integration (CI), we implement test automation that runs as part of the pipeline.
Beyond passing automated tests, we must ensure they are comprehensive enough to cover all possibilities.
To achieve this assurance, we employ Code Coverage.
What is Code Coverage?
Code Coverage is a metric we can analyze after conducting automated tests. This metric indicates what part of the total code has been tested by the tests. In other words, after running automated tests, we obtain metrics showing which part of the code has truly been executed by them.
There are different types of Code Coverage, depending on what we analyze. To better understand, let’s use a source code example.
/* coffee.js */
export function calcCoffeeIngredient(coffeeName, cup = 1) {
let espresso, water;
if (coffeeName === 'espresso') {
espresso = 30 * cup;
return { espresso };
}
if (coffeeName === 'americano') {
espresso = 30 * cup; water = 70 * cup;
return { espresso, water };
}
return {};
}
export function isValidCoffee(name) {
return ['espresso', 'americano', 'mocha'].includes(name);
}In this code, we define a couple of simple functions—one that returns ingredients based on the type of coffee and another that indicates which types of coffee we accept as valid.
We conduct the following automated tests on this code:
/* coffee.test.js */
import { describe, expect, assert, it } from 'vitest';
import { calcCoffeeIngredient } from '../src/coffee-incomplete';
describe('Coffee', () => {
it('should have espresso', () => {
const result = calcCoffeeIngredient('espresso', 2);
expect(result).to.deep.equal({ espresso: 60 });
});
it('should have nothing', () => {
const result = calcCoffeeIngredient('unknown');
expect(result).to.deep.equal({});
});
});In the test, we request ingredients for 2 espressos (it should give us 60 in total, as each one requires 30). Then, we request ingredients for an ‘unknown’ coffee, which should give us an empty result.
This test is testing the above code, but what Code Coverage is it achieving?
Line Coverage
To begin, we have the Line Coverage metric. This metric simply calculates the total executable lines of the original code and the total lines we have actually executed with the test. In the original code, the executable lines are highlighted:
/* coffee.js */
export function calcCoffeeIngredient(coffeeName, cup = 1) {
let espresso, water;
#14C8BE
}
if (coffeeName === 'americano') {
espresso = 30 * cup; water = 70 * cup;
return { espresso, water };
}
return {};
}
export function isValidCoffee(name) {
return ['espresso', 'americano', 'mocha'].includes(name);
}When conducting this test, in the first part, we check if the name is ‘espresso,’ enter the first condition, and finish the test. In the second part, we check if the name is ‘espresso’ (it’s not), and then check if it’s ‘americano’ (it’s not), returning an empty set.
Therefore, in the test, we are not accessing the code of the second condition (which we enter if it’s ‘americano’). We also don’t test if the coffee type name is valid, as checked in another function. In total, we have gone through 5 out of the 8 executable lines, giving us a Line Coverage of 62.5%.
Line Coverage does not consider lines that are for definition and not executed. Line Coverage provides a good initial approximation that the code is running, but it’s not sufficient. As we can see, there is an entire function that has not been tested. With Line Coverage alone, we might think we tested it, but only at 62.5%. For a more comprehensive view, we need to use other types of Coverage, such as understanding how many functions have been tested.
Statement Coverage
This type of Coverage indicates the executed statements within the code. At first, it may seem the same as Line Coverage, and often it is, but it’s possible for a line to have more than one statement. For example, in our code, this line:
espresso = 30 * cup; water = 70 * cup;
Here, in a single line, we have two statements. This leaves the total statements at 9, and our test only executes 5, resulting in a Coverage of 55.55%.
Function Coverage
Function Coverage indicates the total declared functions in the code that are actually called by the tests.
In our case, we have two:
export function calcCoffeeIngredient(coffeeName, cup = 1)
export function isValidCoffee(name)Given these two functions, the test calls only one of them. The Function Coverage is 50%.
If the test called both functions, we would have 100% Function Coverage. However, internally, entire pieces of code may not be queried, as we use conditionals and loops.
Branch Coverage
Branch Coverage checks, within the code, how many sections separated by conditions or loops have actually been tested. In the above code, we have the following branches:
export function calcCoffeeIngredient(coffeeName, cup = 1) {
// ...
if (coffeeName === 'espresso') {
// ...
return { espresso };
}
if (coffeeName === 'americano') {
// ...
return { espresso, water };
}
return {};
}Given these branches, we have the following options:
- Call the function with only the coffee name.
- Call the function with the coffee name and the number of cups.
- The coffee is ‘espresso.’
- The coffee is ‘americano.’
- The coffee is of another type.
The test we conducted goes through branches 1, 2, 3, and 5, leaving branch 4 untested. Therefore, the Branch Coverage is 80%.
There are other types of Coverage, for example, some that also try to evaluate the complexity of code and weigh the % of executed complexity, giving more weight to more complex code. But these 4 we’ve mentioned are basic and common in many Code Coverage tools.
Depending on the programming language and environment, different tools must be used to extract metrics and obtain values. There is no one-size-fits-all solution.
Code Coverage and Continuous Integration
As we have seen, relying on only one type of Coverage would provide a partial idea. If we only count lines or statements, it may be that tests are not accessing a vital piece of code or function with fewer lines, giving the appearance that we have tested a lot of code but missing basic functionalities. Looking only at branches, we may miss important internal functions, and focusing only on functions, we may miss important internal branches.
During CI, we need to use tools that allow us to calculate the Code Coverage we are achieving in an application and provide several Coverage metrics. Given these results, we need to add conditions to the pipeline that halt its execution without a minimum Coverage in the obtained metrics.
When enough tests are included to increase Coverage, and the pipeline does not fail, we can proceed with normal execution and installation in different environments. This way, we ensure that the code has been properly tested and has a certain quality automatically.
Although we would all like an ideal world where there is 100% Coverage in all types, we must adapt conditions to reality. In general, reaching 80% Coverage in new developments is already considered a very good metric. Additionally, there may be cases where tests are being conducted for a finished project (for example, in cases of legacy applications), and we may have to start from lower minimum metrics, such as 30%, and gradually increase as these tests are developed.
In general, conditions must be adapted to the reality and situation of each case.
Conclusions
Code Coverage is a powerful metric for ensuring the quality of projects and automating testing. It allows us to stop untested code before it reaches productive environments and forces us to think about a testing program where we truly check everything the application can do, without leaving out sections or functionalities.
With Code Coverage, we can improve the quality of deliveries and tests, ensuring that new functionalities and developments are not made without exhaustive automated testing that provides security and confidence.
If you have questions, need advice, or want to discuss how we can collaborate, feel free to contact us.