Higher Strength Combinatorial Interaction Testing
This page is supplementary to the paper entitled "Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing" ([PDF] [BIB]), which is currently to appear in the proceedings of the ESEC/FSE 2013.
Abstract
Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints and should, ideally, report a test priority order that maximises earliest fault detection. We show that we can achieve the highest strength CIT in 5.65 minutes on average. This was previously thought to be too computationally expensive to be feasible. Furthermore, we show that higher strength suites find more faults, while prioritisations using lower strengths are no worse at achieving early fault revelation.
Experimental Setup
Subject Programs
We use five Unix utilities from the Software Infrastructure Repository (SIR), maintained by University of Nebraska Lincoln: flex, grep, gzip, sed, and make.
Test Suites
The test suites we use are subsets of those provided by SIR. It can be downloaded from here. For each of the subjects we provide:
-
An annotated TSL file (from SIR) :
_v0_#_small.tsl
Description: The v0_# matches the associated TSL name from SIR for that subject. The leftmost column in this file is a mapping to the covering array values given as sequential numbers starting at 0. For instance, the first parameter if binary will be 0,1 and the next parameter's values will start at 2. The single and error parameters (and any other parameters that we did not include in our model) are not numbered and not used in our experiments. -
An input model file for CASA:
_v0_small.model
Description: The first row is the strength (t) and can be changed to 3,4 etc. to generate higher strength arrays. The second row is the number of factors (TSL parameters in the model). The last row lists the number of values (TSL choices) for each of the parameters. This model matches the associated TSL file. -
The constraint file in CASA input format:
_v0_small.constraints
Description: The first row of this file is the number of constraints. For each of the constraints there are two rows. The first row gives the arity of the constraint (i.e. how many values will be included in this constraint). The next row is list of the constraints between values. The - signs indicate excludes. For instance, a row with a 2, followed by a row that has -10 -12 encodes the binary constraint that values 10 and 12 cannot appear together in a test. Please see the CASA tool for full documentation.
Combinatorial Interaction Testing Tool
We use CASA (Covering Arrays by Simulated Annealing) tool to generate combinatorial interaction test suites from the TSL files.
Full Results
This section contains the full APCC data for all experiments and plots for APCC and fault detection.
Subject | Generation Strength | Prioritsation Strengths | Subject | Generation Strength | Prioritsation Strengths |
flex | 2 | 2, 3, 4, 5 | gzip | 2 | 2, 3, 4, 5 |
flex | 3 | 2, 3, 4, 5 | gzip | 3 | 2, 3, 4, 5 |
flex | 4 | 2, 3, 4, 5 | gzip | 4 | 2, 3, 4, 5 |
flex | 5 | 2, 3, 4, 5 | gzip | 5 | 2, 3, 4, 5 |
grep | 2 | 2, 3, 4, 5 | sed | 2 | 2, 3, 4, 5 |
grep | 3 | 2, 3, 4, 5 | sed | 3 | 2, 3, 4, 5 |
grep | 4 | 2, 3, 4, 5 | sed | 4 | 2, 3, 4, 5 |
grep | 5 | 2, 3, 4, 5 | sed | 5 | 2, 3, 4, 5 |
flex
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
4 | 93.023 | 82.851 | 71.707 | 60.886 | 51.033 |
5 | 97.674 | 89.340 | 78.722 | 67.791 | 57.518 |
6 | 100.000 | 93.163 | 83.403 | 72.852 | 62.623 |
7 | 100.000 | 94.206 | 85.516 | 75.762 | 66.013 |
8 | 100.000 | 95.017 | 87.113 | 77.908 | 68.466 |
9 | 100.000 | 95.829 | 88.849 | 80.407 | 71.487 |
10 | 100.000 | 96.408 | 90.086 | 82.192 | 73.659 |
11 | 100.000 | 96.987 | 91.240 | 83.759 | 75.468 |
12 | 100.000 | 97.451 | 92.394 | 85.598 | 77.854 |
13 | 100.000 | 97.798 | 93.160 | 86.739 | 79.287 |
14 | 100.000 | 98.030 | 93.760 | 87.730 | 80.618 |
15 | 100.000 | 98.262 | 94.268 | 88.491 | 81.588 |
16 | 100.000 | 98.494 | 94.831 | 89.396 | 82.795 |
17 | 100.000 | 98.725 | 95.458 | 90.470 | 84.280 |
18 | 100.000 | 98.957 | 95.994 | 91.305 | 85.375 |
19 | 100.000 | 99.189 | 96.594 | 92.307 | 86.742 |
20 | 100.000 | 99.305 | 96.843 | 92.677 | 87.212 |
21 | 100.000 | 99.421 | 97.175 | 93.258 | 88.023 |
22 | 100.000 | 99.537 | 97.434 | 93.656 | 88.545 |
23 | 100.000 | 99.652 | 97.711 | 94.097 | 89.130 |
24 | 100.000 | 99.768 | 98.006 | 94.592 | 89.818 |
25 | 100.000 | 99.884 | 98.274 | 95.012 | 90.372 |
26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |


flex
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 93.023 | 82.271 | 70.747 | 59.812 | 50.024 |
5 | 97.674 | 89.108 | 78.418 | 67.553 | 57.408 |
6 | 100.000 | 93.048 | 83.310 | 72.881 | 62.799 |
7 | 100.000 | 93.975 | 85.240 | 75.594 | 66.014 |
8 | 100.000 | 94.670 | 86.753 | 77.807 | 68.727 |
9 | 100.000 | 95.481 | 88.433 | 80.170 | 71.538 |
10 | 100.000 | 96.176 | 89.901 | 82.268 | 74.067 |
11 | 100.000 | 96.871 | 91.341 | 84.300 | 76.498 |
12 | 100.000 | 97.219 | 92.218 | 85.721 | 78.377 |
13 | 100.000 | 97.567 | 93.021 | 86.959 | 79.971 |
14 | 100.000 | 97.914 | 93.806 | 88.142 | 81.462 |
15 | 100.000 | 98.262 | 94.581 | 89.308 | 82.934 |
16 | 100.000 | 98.610 | 95.320 | 90.379 | 84.256 |
17 | 100.000 | 98.841 | 95.855 | 91.219 | 85.365 |
18 | 100.000 | 99.073 | 96.381 | 92.021 | 86.389 |
19 | 100.000 | 99.189 | 96.705 | 92.589 | 87.195 |
20 | 100.000 | 99.305 | 97.009 | 93.109 | 87.925 |
21 | 100.000 | 99.421 | 97.268 | 93.507 | 88.448 |
22 | 100.000 | 99.537 | 97.526 | 93.899 | 88.950 |
23 | 100.000 | 99.652 | 97.775 | 94.271 | 89.426 |
24 | 100.000 | 99.768 | 98.025 | 94.640 | 89.896 |
25 | 100.000 | 99.884 | 98.274 | 95.010 | 90.366 |
26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |


flex
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 93.023 | 81.808 | 69.934 | 58.848 | 49.062 |
5 | 97.674 | 88.528 | 77.375 | 66.284 | 56.108 |
6 | 97.674 | 90.151 | 80.504 | 70.398 | 60.699 |
7 | 97.674 | 91.078 | 82.424 | 73.093 | 63.894 |
8 | 97.674 | 91.889 | 84.187 | 75.652 | 66.997 |
9 | 97.674 | 92.352 | 85.267 | 77.317 | 69.125 |
10 | 100.000 | 96.060 | 89.670 | 81.958 | 73.716 |
11 | 100.000 | 96.408 | 90.538 | 83.356 | 75.562 |
12 | 100.000 | 97.103 | 91.978 | 85.384 | 77.980 |
13 | 100.000 | 97.451 | 92.791 | 86.642 | 79.600 |
14 | 100.000 | 97.798 | 93.557 | 87.793 | 81.063 |
15 | 100.000 | 98.146 | 94.341 | 88.982 | 82.571 |
16 | 100.000 | 98.494 | 95.126 | 90.168 | 84.073 |
17 | 100.000 | 98.725 | 95.671 | 91.019 | 85.181 |
18 | 100.000 | 98.957 | 96.178 | 91.780 | 86.151 |
19 | 100.000 | 99.189 | 96.705 | 92.582 | 87.175 |
20 | 100.000 | 99.305 | 97.000 | 93.078 | 87.863 |
21 | 100.000 | 99.421 | 97.268 | 93.498 | 88.418 |
22 | 100.000 | 99.537 | 97.526 | 93.896 | 88.940 |
23 | 100.000 | 99.652 | 97.775 | 94.271 | 89.426 |
24 | 100.000 | 99.768 | 98.025 | 94.642 | 89.902 |
25 | 100.000 | 99.884 | 98.274 | 95.012 | 90.372 |
26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |


flex
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 90.698 | 79.954 | 68.993 | 58.624 | 49.270 |
5 | 93.023 | 84.125 | 74.328 | 64.519 | 55.269 |
6 | 95.349 | 87.717 | 78.685 | 69.298 | 60.196 |
7 | 97.674 | 91.194 | 82.766 | 73.665 | 64.635 |
8 | 97.674 | 92.236 | 84.870 | 76.556 | 68.001 |
9 | 97.674 | 92.816 | 86.135 | 78.421 | 70.313 |
10 | 97.674 | 93.395 | 87.409 | 80.302 | 72.642 |
11 | 100.000 | 96.640 | 90.972 | 83.914 | 76.172 |
12 | 100.000 | 97.335 | 92.394 | 85.900 | 78.531 |
13 | 100.000 | 97.683 | 93.178 | 87.092 | 80.054 |
14 | 100.000 | 98.030 | 93.963 | 88.282 | 81.565 |
15 | 100.000 | 98.262 | 94.508 | 89.137 | 82.686 |
16 | 100.000 | 98.378 | 94.858 | 89.773 | 83.600 |
17 | 100.000 | 98.610 | 95.403 | 90.628 | 84.721 |
18 | 100.000 | 98.725 | 95.708 | 91.150 | 85.456 |
19 | 100.000 | 98.957 | 96.197 | 91.860 | 86.337 |
20 | 100.000 | 99.189 | 96.741 | 92.705 | 87.423 |
21 | 100.000 | 99.421 | 97.268 | 93.507 | 88.448 |
22 | 100.000 | 99.537 | 97.517 | 93.879 | 88.924 |
23 | 100.000 | 99.652 | 97.766 | 94.251 | 89.400 |
24 | 100.000 | 99.768 | 98.015 | 94.621 | 89.870 |
25 | 100.000 | 99.884 | 98.265 | 94.990 | 90.340 |
26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |


flex
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 86.047 | 69.757 | 55.183 | 43.241 | 33.810 |
3 | 90.698 | 77.520 | 64.405 | 52.735 | 42.856 |
4 | 95.349 | 84.820 | 72.851 | 61.352 | 51.071 |
5 | 100.000 | 91.309 | 79.886 | 68.304 | 57.631 |
6 | 100.000 | 93.048 | 83.283 | 72.814 | 62.698 |
7 | 100.000 | 94.322 | 85.821 | 76.250 | 66.635 |
8 | 100.000 | 95.249 | 87.723 | 78.901 | 69.762 |
9 | 100.000 | 96.060 | 89.357 | 81.147 | 72.387 |
10 | 100.000 | 96.756 | 90.843 | 83.296 | 75.004 |
11 | 100.000 | 97.103 | 91.609 | 84.434 | 76.427 |
12 | 100.000 | 97.451 | 92.458 | 85.785 | 78.194 |
13 | 100.000 | 97.798 | 93.252 | 86.997 | 79.738 |
14 | 100.000 | 98.146 | 93.981 | 88.041 | 81.010 |
15 | 100.000 | 98.378 | 94.609 | 89.104 | 82.459 |
16 | 100.000 | 98.610 | 95.255 | 90.212 | 83.980 |
17 | 100.000 | 98.841 | 95.818 | 91.119 | 85.194 |
18 | 100.000 | 98.957 | 96.114 | 91.621 | 85.902 |
19 | 100.000 | 99.073 | 96.372 | 92.019 | 86.424 |
20 | 100.000 | 99.189 | 96.649 | 92.459 | 87.009 |
21 | 100.000 | 99.305 | 96.917 | 92.877 | 87.558 |
22 | 100.000 | 99.421 | 97.203 | 93.347 | 88.198 |
23 | 100.000 | 99.537 | 97.471 | 93.772 | 88.770 |
24 | 100.000 | 99.652 | 97.738 | 94.191 | 89.325 |
25 | 100.000 | 99.768 | 98.052 | 94.746 | 90.125 |
26 | 100.000 | 99.884 | 98.301 | 95.117 | 90.601 |
27 | 100.000 | 100.000 | 98.569 | 95.537 | 91.156 |
28 | 100.000 | 100.000 | 98.643 | 95.745 | 91.529 |
29 | 100.000 | 100.000 | 98.726 | 95.972 | 91.924 |
30 | 100.000 | 100.000 | 98.735 | 96.003 | 91.985 |
31 | 100.000 | 100.000 | 98.800 | 96.180 | 92.294 |
32 | 100.000 | 100.000 | 98.837 | 96.282 | 92.474 |
33 | 100.000 | 100.000 | 98.911 | 96.494 | 92.860 |
34 | 100.000 | 100.000 | 98.994 | 96.726 | 93.273 |
35 | 100.000 | 100.000 | 99.086 | 96.980 | 93.715 |
36 | 100.000 | 100.000 | 99.169 | 97.223 | 94.164 |
37 | 100.000 | 100.000 | 99.178 | 97.250 | 94.215 |
38 | 100.000 | 100.000 | 99.188 | 97.277 | 94.266 |
39 | 100.000 | 100.000 | 99.197 | 97.306 | 94.320 |
40 | 100.000 | 100.000 | 99.289 | 97.560 | 94.767 |
41 | 100.000 | 100.000 | 99.400 | 97.875 | 95.334 |
42 | 100.000 | 100.000 | 99.409 | 97.905 | 95.395 |
43 | 100.000 | 100.000 | 99.492 | 98.136 | 95.799 |
44 | 100.000 | 100.000 | 99.502 | 98.164 | 95.854 |
45 | 100.000 | 100.000 | 99.511 | 98.191 | 95.905 |
46 | 100.000 | 100.000 | 99.585 | 98.403 | 96.292 |
47 | 100.000 | 100.000 | 99.631 | 98.538 | 96.538 |
48 | 100.000 | 100.000 | 99.640 | 98.565 | 96.589 |
49 | 100.000 | 100.000 | 99.732 | 98.817 | 97.027 |
50 | 100.000 | 100.000 | 99.788 | 98.979 | 97.326 |
51 | 100.000 | 100.000 | 99.852 | 99.159 | 97.647 |
52 | 100.000 | 100.000 | 99.862 | 99.187 | 97.698 |
53 | 100.000 | 100.000 | 99.898 | 99.289 | 97.877 |
54 | 100.000 | 100.000 | 99.954 | 99.447 | 98.166 |
55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |


flex
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
4 | 95.349 | 84.936 | 73.064 | 61.614 | 51.341 |
5 | 97.674 | 88.992 | 78.187 | 67.247 | 57.070 |
6 | 100.000 | 92.816 | 82.895 | 72.377 | 62.284 |
7 | 100.000 | 94.090 | 85.452 | 75.862 | 66.303 |
8 | 100.000 | 95.017 | 87.390 | 78.600 | 69.560 |
9 | 100.000 | 95.829 | 89.098 | 81.031 | 72.476 |
10 | 100.000 | 96.524 | 90.566 | 83.128 | 75.005 |
11 | 100.000 | 97.219 | 91.987 | 85.116 | 77.367 |
12 | 100.000 | 97.451 | 92.624 | 86.204 | 78.862 |
13 | 100.000 | 97.683 | 93.261 | 87.292 | 80.357 |
14 | 100.000 | 98.030 | 94.046 | 88.476 | 81.849 |
15 | 100.000 | 98.378 | 94.821 | 89.644 | 83.331 |
16 | 100.000 | 98.494 | 95.172 | 90.290 | 84.276 |
17 | 100.000 | 98.610 | 95.523 | 90.935 | 85.217 |
18 | 100.000 | 98.841 | 96.040 | 91.718 | 86.220 |
19 | 100.000 | 98.957 | 96.372 | 92.312 | 87.076 |
20 | 100.000 | 98.957 | 96.511 | 92.687 | 87.721 |
21 | 100.000 | 99.073 | 96.825 | 93.226 | 88.469 |
22 | 100.000 | 99.073 | 96.945 | 93.559 | 89.056 |
23 | 100.000 | 99.189 | 97.231 | 94.045 | 89.748 |
24 | 100.000 | 99.189 | 97.332 | 94.324 | 90.237 |
25 | 100.000 | 99.189 | 97.425 | 94.577 | 90.677 |
26 | 100.000 | 99.189 | 97.517 | 94.836 | 91.139 |
27 | 100.000 | 99.305 | 97.775 | 95.236 | 91.668 |
28 | 100.000 | 99.305 | 97.849 | 95.451 | 92.061 |
29 | 100.000 | 99.305 | 97.923 | 95.668 | 92.464 |
30 | 100.000 | 99.421 | 98.181 | 96.066 | 92.987 |
31 | 100.000 | 99.537 | 98.431 | 96.443 | 93.480 |
32 | 100.000 | 99.652 | 98.680 | 96.813 | 93.950 |
33 | 100.000 | 99.768 | 98.929 | 97.181 | 94.413 |
34 | 100.000 | 99.884 | 99.178 | 97.549 | 94.879 |
35 | 100.000 | 100.000 | 99.428 | 97.915 | 95.335 |
36 | 100.000 | 100.000 | 99.483 | 98.071 | 95.617 |
37 | 100.000 | 100.000 | 99.538 | 98.228 | 95.902 |
38 | 100.000 | 100.000 | 99.594 | 98.385 | 96.183 |
39 | 100.000 | 100.000 | 99.649 | 98.548 | 96.485 |
40 | 100.000 | 100.000 | 99.695 | 98.682 | 96.735 |
41 | 100.000 | 100.000 | 99.742 | 98.824 | 97.005 |
42 | 100.000 | 100.000 | 99.788 | 98.957 | 97.250 |
43 | 100.000 | 100.000 | 99.834 | 99.093 | 97.500 |
44 | 100.000 | 100.000 | 99.871 | 99.199 | 97.694 |
45 | 100.000 | 100.000 | 99.908 | 99.303 | 97.881 |
46 | 100.000 | 100.000 | 99.917 | 99.330 | 97.932 |
47 | 100.000 | 100.000 | 99.926 | 99.357 | 97.982 |
48 | 100.000 | 100.000 | 99.935 | 99.385 | 98.037 |
49 | 100.000 | 100.000 | 99.945 | 99.415 | 98.095 |
50 | 100.000 | 100.000 | 99.954 | 99.442 | 98.146 |
51 | 100.000 | 100.000 | 99.963 | 99.470 | 98.200 |
52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.305 |
54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |


flex
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 88.372 | 74.739 | 61.885 | 50.692 | 41.300 |
4 | 93.023 | 82.039 | 70.331 | 59.310 | 49.515 |
5 | 97.674 | 88.644 | 77.605 | 66.592 | 56.452 |
6 | 97.674 | 90.267 | 80.809 | 70.883 | 61.312 |
7 | 100.000 | 93.975 | 85.276 | 75.688 | 66.165 |
8 | 100.000 | 94.670 | 86.781 | 77.877 | 68.838 |
9 | 100.000 | 95.249 | 88.064 | 79.782 | 71.205 |
10 | 100.000 | 95.944 | 89.523 | 81.856 | 73.695 |
11 | 100.000 | 96.524 | 90.741 | 83.602 | 75.818 |
12 | 100.000 | 97.219 | 92.191 | 85.649 | 78.258 |
13 | 100.000 | 97.451 | 92.800 | 86.673 | 79.658 |
14 | 100.000 | 97.567 | 93.151 | 87.325 | 80.621 |
15 | 100.000 | 98.030 | 94.138 | 88.755 | 82.373 |
16 | 100.000 | 98.262 | 94.720 | 89.705 | 83.649 |
17 | 100.000 | 98.262 | 94.849 | 90.065 | 84.285 |
18 | 100.000 | 98.378 | 95.200 | 90.703 | 85.206 |
19 | 100.000 | 98.610 | 95.744 | 91.558 | 86.327 |
20 | 100.000 | 98.725 | 96.077 | 92.152 | 87.179 |
21 | 100.000 | 98.841 | 96.381 | 92.677 | 87.923 |
22 | 100.000 | 98.957 | 96.695 | 93.223 | 88.698 |
23 | 100.000 | 99.189 | 97.194 | 93.962 | 89.635 |
24 | 100.000 | 99.189 | 97.286 | 94.222 | 90.098 |
25 | 100.000 | 99.189 | 97.388 | 94.502 | 90.589 |
26 | 100.000 | 99.305 | 97.692 | 95.016 | 91.299 |
27 | 100.000 | 99.305 | 97.785 | 95.269 | 91.739 |
28 | 100.000 | 99.305 | 97.840 | 95.435 | 92.048 |
29 | 100.000 | 99.305 | 97.905 | 95.621 | 92.389 |
30 | 100.000 | 99.305 | 97.960 | 95.787 | 92.698 |
31 | 100.000 | 99.305 | 98.015 | 95.952 | 93.004 |
32 | 100.000 | 99.421 | 98.274 | 96.355 | 93.543 |
33 | 100.000 | 99.421 | 98.329 | 96.518 | 93.848 |
34 | 100.000 | 99.421 | 98.385 | 96.678 | 94.137 |
35 | 100.000 | 99.421 | 98.440 | 96.836 | 94.426 |
36 | 100.000 | 99.421 | 98.495 | 96.991 | 94.705 |
37 | 100.000 | 99.537 | 98.745 | 97.365 | 95.188 |
38 | 100.000 | 99.537 | 98.791 | 97.497 | 95.430 |
39 | 100.000 | 99.537 | 98.837 | 97.628 | 95.669 |
40 | 100.000 | 99.652 | 99.086 | 97.996 | 96.132 |
41 | 100.000 | 99.768 | 99.335 | 98.364 | 96.595 |
42 | 100.000 | 99.884 | 99.585 | 98.731 | 97.058 |
43 | 100.000 | 100.000 | 99.834 | 99.099 | 97.521 |
44 | 100.000 | 100.000 | 99.871 | 99.203 | 97.708 |
45 | 100.000 | 100.000 | 99.908 | 99.305 | 97.888 |
46 | 100.000 | 100.000 | 99.917 | 99.333 | 97.943 |
47 | 100.000 | 100.000 | 99.926 | 99.362 | 97.997 |
48 | 100.000 | 100.000 | 99.935 | 99.390 | 98.052 |
49 | 100.000 | 100.000 | 99.945 | 99.417 | 98.102 |
50 | 100.000 | 100.000 | 99.954 | 99.444 | 98.153 |
51 | 100.000 | 100.000 | 99.963 | 99.471 | 98.204 |
52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.306 |
54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |


flex
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 79.070 | 62.109 | 48.777 | 38.407 | 30.360 |
3 | 86.047 | 71.958 | 59.356 | 48.629 | 39.714 |
4 | 90.698 | 79.374 | 68.014 | 57.508 | 48.199 |
5 | 95.349 | 86.327 | 75.898 | 65.511 | 55.852 |
6 | 95.349 | 87.370 | 78.095 | 68.622 | 59.549 |
7 | 95.349 | 88.181 | 79.830 | 71.124 | 62.579 |
8 | 95.349 | 88.760 | 81.169 | 73.159 | 65.140 |
9 | 100.000 | 95.133 | 87.898 | 79.628 | 71.093 |
10 | 100.000 | 95.829 | 89.338 | 81.663 | 73.534 |
11 | 100.000 | 96.408 | 90.557 | 83.410 | 75.656 |
12 | 100.000 | 96.987 | 91.775 | 85.156 | 77.779 |
13 | 100.000 | 97.451 | 92.781 | 86.637 | 79.619 |
14 | 100.000 | 97.683 | 93.363 | 87.595 | 80.921 |
15 | 100.000 | 97.798 | 93.714 | 88.240 | 81.862 |
16 | 100.000 | 98.030 | 94.277 | 89.147 | 83.075 |
17 | 100.000 | 98.030 | 94.397 | 89.481 | 83.662 |
18 | 100.000 | 98.146 | 94.738 | 90.099 | 84.558 |
19 | 100.000 | 98.262 | 95.071 | 90.692 | 85.407 |
20 | 100.000 | 98.262 | 95.191 | 91.023 | 85.987 |
21 | 100.000 | 98.494 | 95.754 | 91.921 | 87.171 |
22 | 100.000 | 98.494 | 95.864 | 92.227 | 87.708 |
23 | 100.000 | 98.494 | 95.975 | 92.533 | 88.246 |
24 | 100.000 | 98.725 | 96.474 | 93.276 | 89.193 |
25 | 100.000 | 98.957 | 96.972 | 94.014 | 90.127 |
26 | 100.000 | 98.957 | 97.028 | 94.178 | 90.432 |
27 | 100.000 | 98.957 | 97.083 | 94.339 | 90.730 |
28 | 100.000 | 99.073 | 97.360 | 94.789 | 91.346 |
29 | 100.000 | 99.073 | 97.415 | 94.947 | 91.631 |
30 | 100.000 | 99.073 | 97.480 | 95.127 | 91.951 |
31 | 100.000 | 99.073 | 97.591 | 95.422 | 92.454 |
32 | 100.000 | 99.073 | 97.637 | 95.556 | 92.699 |
33 | 100.000 | 99.189 | 97.905 | 95.982 | 93.275 |
34 | 100.000 | 99.189 | 97.960 | 96.144 | 93.574 |
35 | 100.000 | 99.305 | 98.228 | 96.567 | 94.140 |
36 | 100.000 | 99.305 | 98.283 | 96.726 | 94.430 |
37 | 100.000 | 99.421 | 98.532 | 97.100 | 94.913 |
38 | 100.000 | 99.537 | 98.791 | 97.494 | 95.423 |
39 | 100.000 | 99.537 | 98.837 | 97.628 | 95.669 |
40 | 100.000 | 99.652 | 99.086 | 97.998 | 96.139 |
41 | 100.000 | 99.768 | 99.335 | 98.366 | 96.603 |
42 | 100.000 | 99.884 | 99.585 | 98.734 | 97.065 |
43 | 100.000 | 99.884 | 99.622 | 98.838 | 97.253 |
44 | 100.000 | 100.000 | 99.871 | 99.204 | 97.712 |
45 | 100.000 | 100.000 | 99.908 | 99.306 | 97.891 |
46 | 100.000 | 100.000 | 99.917 | 99.334 | 97.946 |
47 | 100.000 | 100.000 | 99.926 | 99.363 | 98.001 |
48 | 100.000 | 100.000 | 99.935 | 99.390 | 98.052 |
49 | 100.000 | 100.000 | 99.945 | 99.417 | 98.103 |
50 | 100.000 | 100.000 | 99.954 | 99.444 | 98.154 |
51 | 100.000 | 100.000 | 99.963 | 99.471 | 98.204 |
52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.306 |
54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |


flex
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 86.047 | 69.757 | 55.183 | 43.241 | 33.810 |
3 | 90.698 | 77.520 | 64.405 | 52.735 | 42.856 |
4 | 95.349 | 84.820 | 72.833 | 61.307 | 51.003 |
5 | 100.000 | 91.425 | 80.070 | 68.500 | 57.802 |
6 | 100.000 | 93.048 | 83.264 | 72.769 | 62.629 |
7 | 100.000 | 94.206 | 85.544 | 75.832 | 66.127 |
8 | 100.000 | 95.133 | 87.437 | 78.456 | 69.205 |
9 | 100.000 | 95.944 | 89.117 | 80.815 | 72.003 |
10 | 100.000 | 96.756 | 90.778 | 83.131 | 74.739 |
11 | 100.000 | 97.219 | 91.849 | 84.772 | 76.828 |
12 | 100.000 | 97.567 | 92.615 | 85.911 | 78.255 |
13 | 100.000 | 97.914 | 93.400 | 87.101 | 79.766 |
14 | 100.000 | 98.146 | 93.972 | 88.031 | 81.015 |
15 | 100.000 | 98.378 | 94.517 | 88.875 | 82.102 |
16 | 100.000 | 98.610 | 95.108 | 89.841 | 83.390 |
17 | 100.000 | 98.841 | 95.671 | 90.744 | 84.591 |
18 | 100.000 | 98.957 | 96.031 | 91.392 | 85.504 |
19 | 100.000 | 99.073 | 96.354 | 91.962 | 86.316 |
20 | 100.000 | 99.189 | 96.695 | 92.574 | 87.191 |
21 | 100.000 | 99.305 | 96.954 | 92.968 | 87.700 |
22 | 100.000 | 99.421 | 97.314 | 93.636 | 88.681 |
23 | 100.000 | 99.537 | 97.655 | 94.240 | 89.528 |
24 | 100.000 | 99.652 | 97.905 | 94.610 | 89.998 |
25 | 100.000 | 99.768 | 98.154 | 94.982 | 90.474 |
26 | 100.000 | 99.884 | 98.403 | 95.360 | 90.971 |
27 | 100.000 | 100.000 | 98.661 | 95.756 | 91.486 |
28 | 100.000 | 100.000 | 98.698 | 95.858 | 91.666 |
29 | 100.000 | 100.000 | 98.782 | 96.089 | 92.074 |
30 | 100.000 | 100.000 | 98.883 | 96.367 | 92.558 |
31 | 100.000 | 100.000 | 98.975 | 96.635 | 93.047 |
32 | 100.000 | 100.000 | 99.040 | 96.816 | 93.370 |
33 | 100.000 | 100.000 | 99.105 | 97.007 | 93.724 |
34 | 100.000 | 100.000 | 99.151 | 97.147 | 93.992 |
35 | 100.000 | 100.000 | 99.169 | 97.211 | 94.126 |
36 | 100.000 | 100.000 | 99.188 | 97.276 | 94.267 |
37 | 100.000 | 100.000 | 99.206 | 97.337 | 94.389 |
38 | 100.000 | 100.000 | 99.225 | 97.398 | 94.515 |
39 | 100.000 | 100.000 | 99.271 | 97.533 | 94.762 |
40 | 100.000 | 100.000 | 99.308 | 97.646 | 94.980 |
41 | 100.000 | 100.000 | 99.326 | 97.707 | 95.102 |
42 | 100.000 | 100.000 | 99.335 | 97.735 | 95.157 |
43 | 100.000 | 100.000 | 99.409 | 97.949 | 95.550 |
44 | 100.000 | 100.000 | 99.446 | 98.053 | 95.737 |
45 | 100.000 | 100.000 | 99.465 | 98.112 | 95.853 |
46 | 100.000 | 100.000 | 99.483 | 98.170 | 95.969 |
47 | 100.000 | 100.000 | 99.502 | 98.230 | 96.092 |
48 | 100.000 | 100.000 | 99.511 | 98.263 | 96.161 |
49 | 100.000 | 100.000 | 99.520 | 98.291 | 96.216 |
50 | 100.000 | 100.000 | 99.548 | 98.376 | 96.381 |
51 | 100.000 | 100.000 | 99.594 | 98.519 | 96.656 |
52 | 100.000 | 100.000 | 99.612 | 98.575 | 96.764 |
53 | 100.000 | 100.000 | 99.668 | 98.734 | 97.051 |
54 | 100.000 | 100.000 | 99.677 | 98.767 | 97.124 |
55 | 100.000 | 100.000 | 99.686 | 98.799 | 97.194 |
56 | 100.000 | 100.000 | 99.686 | 98.810 | 97.231 |
57 | 100.000 | 100.000 | 99.695 | 98.841 | 97.298 |
58 | 100.000 | 100.000 | 99.723 | 98.927 | 97.466 |
59 | 100.000 | 100.000 | 99.742 | 98.983 | 97.575 |
60 | 100.000 | 100.000 | 99.760 | 99.047 | 97.708 |
61 | 100.000 | 100.000 | 99.778 | 99.107 | 97.832 |
62 | 100.000 | 100.000 | 99.788 | 99.139 | 97.897 |
63 | 100.000 | 100.000 | 99.797 | 99.170 | 97.963 |
64 | 100.000 | 100.000 | 99.806 | 99.204 | 98.041 |
65 | 100.000 | 100.000 | 99.825 | 99.262 | 98.153 |
66 | 100.000 | 100.000 | 99.834 | 99.299 | 98.240 |
67 | 100.000 | 100.000 | 99.843 | 99.332 | 98.312 |
68 | 100.000 | 100.000 | 99.852 | 99.359 | 98.364 |
69 | 100.000 | 100.000 | 99.852 | 99.366 | 98.386 |
70 | 100.000 | 100.000 | 99.852 | 99.372 | 98.410 |
71 | 100.000 | 100.000 | 99.871 | 99.427 | 98.515 |
72 | 100.000 | 100.000 | 99.880 | 99.457 | 98.577 |
73 | 100.000 | 100.000 | 99.880 | 99.473 | 98.632 |
74 | 100.000 | 100.000 | 99.889 | 99.501 | 98.689 |
75 | 100.000 | 100.000 | 99.908 | 99.554 | 98.786 |
76 | 100.000 | 100.000 | 99.917 | 99.583 | 98.845 |
77 | 100.000 | 100.000 | 99.917 | 99.588 | 98.860 |
78 | 100.000 | 100.000 | 99.917 | 99.599 | 98.901 |
79 | 100.000 | 100.000 | 99.926 | 99.633 | 98.978 |
80 | 100.000 | 100.000 | 99.935 | 99.660 | 99.029 |
81 | 100.000 | 100.000 | 99.945 | 99.689 | 99.083 |
82 | 100.000 | 100.000 | 99.945 | 99.700 | 99.125 |
83 | 100.000 | 100.000 | 99.945 | 99.704 | 99.140 |
84 | 100.000 | 100.000 | 99.945 | 99.710 | 99.163 |
85 | 100.000 | 100.000 | 99.945 | 99.718 | 99.189 |
86 | 100.000 | 100.000 | 99.954 | 99.745 | 99.240 |
87 | 100.000 | 100.000 | 99.954 | 99.748 | 99.253 |
88 | 100.000 | 100.000 | 99.954 | 99.754 | 99.275 |
89 | 100.000 | 100.000 | 99.954 | 99.760 | 99.298 |
90 | 100.000 | 100.000 | 99.963 | 99.790 | 99.357 |
91 | 100.000 | 100.000 | 99.963 | 99.796 | 99.379 |
92 | 100.000 | 100.000 | 99.963 | 99.804 | 99.409 |
93 | 100.000 | 100.000 | 99.963 | 99.810 | 99.432 |
94 | 100.000 | 100.000 | 99.963 | 99.816 | 99.451 |
95 | 100.000 | 100.000 | 99.963 | 99.821 | 99.470 |
96 | 100.000 | 100.000 | 99.972 | 99.848 | 99.521 |
97 | 100.000 | 100.000 | 99.982 | 99.875 | 99.572 |
98 | 100.000 | 100.000 | 99.982 | 99.883 | 99.603 |
99 | 100.000 | 100.000 | 99.982 | 99.890 | 99.626 |
100 | 100.000 | 100.000 | 99.982 | 99.895 | 99.645 |
101 | 100.000 | 100.000 | 99.982 | 99.900 | 99.664 |
102 | 100.000 | 100.000 | 99.991 | 99.930 | 99.726 |
103 | 100.000 | 100.000 | 99.991 | 99.934 | 99.742 |
104 | 100.000 | 100.000 | 99.991 | 99.944 | 99.775 |
105 | 100.000 | 100.000 | 99.991 | 99.949 | 99.794 |
106 | 100.000 | 100.000 | 100.000 | 99.976 | 99.844 |
107 | 100.000 | 100.000 | 100.000 | 99.979 | 99.856 |
108 | 100.000 | 100.000 | 100.000 | 99.985 | 99.879 |
109 | 100.000 | 100.000 | 100.000 | 99.993 | 99.906 |
110 | 100.000 | 100.000 | 100.000 | 99.997 | 99.922 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |


flex
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
5 | 97.674 | 89.224 | 78.593 | 67.727 | 57.546 |
6 | 97.674 | 90.498 | 81.215 | 71.361 | 61.782 |
7 | 100.000 | 94.206 | 85.646 | 76.075 | 66.494 |
8 | 100.000 | 95.249 | 87.760 | 78.985 | 69.883 |
9 | 100.000 | 95.944 | 89.255 | 81.158 | 72.540 |
10 | 100.000 | 96.756 | 90.907 | 83.454 | 75.250 |
11 | 100.000 | 97.219 | 91.969 | 85.072 | 77.303 |
12 | 100.000 | 97.567 | 92.809 | 86.401 | 79.038 |
13 | 100.000 | 97.798 | 93.427 | 87.444 | 80.464 |
14 | 100.000 | 98.146 | 94.194 | 88.587 | 81.900 |
15 | 100.000 | 98.262 | 94.581 | 89.324 | 82.989 |
16 | 100.000 | 98.494 | 95.135 | 90.196 | 84.126 |
17 | 100.000 | 98.610 | 95.486 | 90.829 | 85.030 |
18 | 100.000 | 98.841 | 96.012 | 91.632 | 86.059 |
19 | 100.000 | 98.957 | 96.354 | 92.246 | 86.937 |
20 | 100.000 | 99.073 | 96.695 | 92.851 | 87.788 |
21 | 100.000 | 99.073 | 96.815 | 93.184 | 88.375 |
22 | 100.000 | 99.073 | 96.926 | 93.495 | 88.926 |
23 | 100.000 | 99.189 | 97.212 | 93.963 | 89.563 |
24 | 100.000 | 99.305 | 97.489 | 94.413 | 90.179 |
25 | 100.000 | 99.421 | 97.766 | 94.868 | 90.812 |
26 | 100.000 | 99.421 | 97.849 | 95.114 | 91.268 |
27 | 100.000 | 99.421 | 97.932 | 95.351 | 91.693 |
28 | 100.000 | 99.537 | 98.191 | 95.748 | 92.216 |
29 | 100.000 | 99.652 | 98.449 | 96.147 | 92.742 |
30 | 100.000 | 99.652 | 98.514 | 96.330 | 93.069 |
31 | 100.000 | 99.768 | 98.763 | 96.699 | 93.539 |
32 | 100.000 | 99.884 | 99.012 | 97.082 | 94.049 |
33 | 100.000 | 100.000 | 99.262 | 97.448 | 94.508 |
34 | 100.000 | 100.000 | 99.326 | 97.635 | 94.849 |
35 | 100.000 | 100.000 | 99.372 | 97.773 | 95.109 |
36 | 100.000 | 100.000 | 99.409 | 97.880 | 95.308 |
37 | 100.000 | 100.000 | 99.446 | 97.986 | 95.500 |
38 | 100.000 | 100.000 | 99.483 | 98.103 | 95.733 |
39 | 100.000 | 100.000 | 99.520 | 98.211 | 95.931 |
40 | 100.000 | 100.000 | 99.548 | 98.304 | 96.123 |
41 | 100.000 | 100.000 | 99.575 | 98.385 | 96.270 |
42 | 100.000 | 100.000 | 99.603 | 98.472 | 96.442 |
43 | 100.000 | 100.000 | 99.631 | 98.551 | 96.586 |
44 | 100.000 | 100.000 | 99.658 | 98.631 | 96.734 |
45 | 100.000 | 100.000 | 99.686 | 98.720 | 96.911 |
46 | 100.000 | 100.000 | 99.705 | 98.781 | 97.037 |
47 | 100.000 | 100.000 | 99.723 | 98.846 | 97.172 |
48 | 100.000 | 100.000 | 99.742 | 98.907 | 97.299 |
49 | 100.000 | 100.000 | 99.760 | 98.970 | 97.428 |
50 | 100.000 | 100.000 | 99.778 | 99.027 | 97.540 |
51 | 100.000 | 100.000 | 99.797 | 99.087 | 97.660 |
52 | 100.000 | 100.000 | 99.815 | 99.147 | 97.783 |
53 | 100.000 | 100.000 | 99.825 | 99.180 | 97.856 |
54 | 100.000 | 100.000 | 99.834 | 99.218 | 97.944 |
55 | 100.000 | 100.000 | 99.843 | 99.247 | 98.002 |
56 | 100.000 | 100.000 | 99.852 | 99.278 | 98.068 |
57 | 100.000 | 100.000 | 99.862 | 99.314 | 98.148 |
58 | 100.000 | 100.000 | 99.871 | 99.343 | 98.206 |
59 | 100.000 | 100.000 | 99.880 | 99.370 | 98.257 |
60 | 100.000 | 100.000 | 99.889 | 99.403 | 98.330 |
61 | 100.000 | 100.000 | 99.898 | 99.442 | 98.420 |
62 | 100.000 | 100.000 | 99.908 | 99.469 | 98.471 |
63 | 100.000 | 100.000 | 99.917 | 99.501 | 98.540 |
64 | 100.000 | 100.000 | 99.926 | 99.528 | 98.591 |
65 | 100.000 | 100.000 | 99.935 | 99.555 | 98.642 |
66 | 100.000 | 100.000 | 99.945 | 99.583 | 98.697 |
67 | 100.000 | 100.000 | 99.954 | 99.619 | 98.776 |
68 | 100.000 | 100.000 | 99.963 | 99.653 | 98.853 |
69 | 100.000 | 100.000 | 99.972 | 99.681 | 98.908 |
70 | 100.000 | 100.000 | 99.982 | 99.708 | 98.958 |
71 | 100.000 | 100.000 | 99.991 | 99.735 | 99.009 |
72 | 100.000 | 100.000 | 100.000 | 99.763 | 99.060 |
73 | 100.000 | 100.000 | 100.000 | 99.771 | 99.090 |
74 | 100.000 | 100.000 | 100.000 | 99.780 | 99.124 |
75 | 100.000 | 100.000 | 100.000 | 99.791 | 99.161 |
76 | 100.000 | 100.000 | 100.000 | 99.797 | 99.184 |
77 | 100.000 | 100.000 | 100.000 | 99.804 | 99.211 |
78 | 100.000 | 100.000 | 100.000 | 99.811 | 99.237 |
79 | 100.000 | 100.000 | 100.000 | 99.818 | 99.260 |
80 | 100.000 | 100.000 | 100.000 | 99.824 | 99.284 |
81 | 100.000 | 100.000 | 100.000 | 99.836 | 99.328 |
82 | 100.000 | 100.000 | 100.000 | 99.844 | 99.355 |
83 | 100.000 | 100.000 | 100.000 | 99.850 | 99.378 |
84 | 100.000 | 100.000 | 100.000 | 99.860 | 99.415 |
85 | 100.000 | 100.000 | 100.000 | 99.869 | 99.444 |
86 | 100.000 | 100.000 | 100.000 | 99.874 | 99.465 |
87 | 100.000 | 100.000 | 100.000 | 99.881 | 99.491 |
88 | 100.000 | 100.000 | 100.000 | 99.885 | 99.507 |
89 | 100.000 | 100.000 | 100.000 | 99.896 | 99.546 |
90 | 100.000 | 100.000 | 100.000 | 99.900 | 99.561 |
91 | 100.000 | 100.000 | 100.000 | 99.903 | 99.574 |
92 | 100.000 | 100.000 | 100.000 | 99.906 | 99.586 |
93 | 100.000 | 100.000 | 100.000 | 99.913 | 99.609 |
94 | 100.000 | 100.000 | 100.000 | 99.917 | 99.625 |
95 | 100.000 | 100.000 | 100.000 | 99.920 | 99.637 |
96 | 100.000 | 100.000 | 100.000 | 99.927 | 99.664 |
97 | 100.000 | 100.000 | 100.000 | 99.932 | 99.683 |
98 | 100.000 | 100.000 | 100.000 | 99.938 | 99.703 |
99 | 100.000 | 100.000 | 100.000 | 99.943 | 99.723 |
100 | 100.000 | 100.000 | 100.000 | 99.950 | 99.749 |
101 | 100.000 | 100.000 | 100.000 | 99.956 | 99.772 |
102 | 100.000 | 100.000 | 100.000 | 99.960 | 99.788 |
103 | 100.000 | 100.000 | 100.000 | 99.966 | 99.807 |
104 | 100.000 | 100.000 | 100.000 | 99.969 | 99.818 |
105 | 100.000 | 100.000 | 100.000 | 99.976 | 99.845 |
106 | 100.000 | 100.000 | 100.000 | 99.982 | 99.869 |
107 | 100.000 | 100.000 | 100.000 | 99.986 | 99.884 |
108 | 100.000 | 100.000 | 100.000 | 99.988 | 99.889 |
109 | 100.000 | 100.000 | 100.000 | 99.994 | 99.912 |
110 | 100.000 | 100.000 | 100.000 | 99.997 | 99.923 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |


flex
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
5 | 97.674 | 89.108 | 78.399 | 67.511 | 57.346 |
6 | 97.674 | 90.498 | 81.169 | 71.255 | 61.628 |
7 | 97.674 | 91.425 | 83.135 | 74.060 | 64.987 |
8 | 100.000 | 95.133 | 87.584 | 78.814 | 69.754 |
9 | 100.000 | 95.713 | 88.877 | 80.744 | 72.165 |
10 | 100.000 | 96.524 | 90.538 | 83.065 | 74.914 |
11 | 100.000 | 96.987 | 91.600 | 84.684 | 76.973 |
12 | 100.000 | 97.219 | 92.227 | 85.751 | 78.435 |
13 | 100.000 | 97.683 | 93.261 | 87.292 | 80.357 |
14 | 100.000 | 98.030 | 94.046 | 88.486 | 81.883 |
15 | 100.000 | 98.378 | 94.831 | 89.674 | 83.391 |
16 | 100.000 | 98.378 | 94.978 | 90.079 | 84.096 |
17 | 100.000 | 98.378 | 95.108 | 90.439 | 84.728 |
18 | 100.000 | 98.494 | 95.468 | 91.099 | 85.683 |
19 | 100.000 | 98.610 | 95.800 | 91.690 | 86.525 |
20 | 100.000 | 98.725 | 96.114 | 92.236 | 87.302 |
21 | 100.000 | 98.725 | 96.215 | 92.520 | 87.804 |
22 | 100.000 | 98.957 | 96.723 | 93.281 | 88.774 |
23 | 100.000 | 99.073 | 97.000 | 93.736 | 89.407 |
24 | 100.000 | 99.073 | 97.083 | 93.972 | 89.829 |
25 | 100.000 | 99.189 | 97.360 | 94.424 | 90.452 |
26 | 100.000 | 99.305 | 97.628 | 94.853 | 91.038 |
27 | 100.000 | 99.421 | 97.895 | 95.280 | 91.617 |
28 | 100.000 | 99.421 | 97.941 | 95.422 | 91.891 |
29 | 100.000 | 99.537 | 98.218 | 95.866 | 92.486 |
30 | 100.000 | 99.537 | 98.265 | 96.008 | 92.759 |
31 | 100.000 | 99.537 | 98.311 | 96.147 | 93.024 |
32 | 100.000 | 99.537 | 98.348 | 96.264 | 93.251 |
33 | 100.000 | 99.537 | 98.403 | 96.422 | 93.539 |
34 | 100.000 | 99.537 | 98.440 | 96.537 | 93.761 |
35 | 100.000 | 99.537 | 98.477 | 96.649 | 93.975 |
36 | 100.000 | 99.652 | 98.726 | 97.021 | 94.452 |
37 | 100.000 | 99.652 | 98.763 | 97.133 | 94.663 |
38 | 100.000 | 99.652 | 98.809 | 97.264 | 94.900 |
39 | 100.000 | 99.768 | 99.058 | 97.634 | 95.371 |
40 | 100.000 | 99.768 | 99.095 | 97.742 | 95.572 |
41 | 100.000 | 99.884 | 99.345 | 98.110 | 96.036 |
42 | 100.000 | 100.000 | 99.594 | 98.476 | 96.496 |
43 | 100.000 | 100.000 | 99.612 | 98.541 | 96.633 |
44 | 100.000 | 100.000 | 99.631 | 98.604 | 96.765 |
45 | 100.000 | 100.000 | 99.658 | 98.689 | 96.926 |
46 | 100.000 | 100.000 | 99.686 | 98.771 | 97.081 |
47 | 100.000 | 100.000 | 99.695 | 98.812 | 97.178 |
48 | 100.000 | 100.000 | 99.714 | 98.872 | 97.301 |
49 | 100.000 | 100.000 | 99.723 | 98.912 | 97.395 |
50 | 100.000 | 100.000 | 99.751 | 98.993 | 97.547 |
51 | 100.000 | 100.000 | 99.760 | 99.030 | 97.634 |
52 | 100.000 | 100.000 | 99.778 | 99.089 | 97.749 |
53 | 100.000 | 100.000 | 99.797 | 99.147 | 97.866 |
54 | 100.000 | 100.000 | 99.815 | 99.205 | 97.983 |
55 | 100.000 | 100.000 | 99.825 | 99.242 | 98.066 |
56 | 100.000 | 100.000 | 99.843 | 99.298 | 98.175 |
57 | 100.000 | 100.000 | 99.852 | 99.333 | 98.255 |
58 | 100.000 | 100.000 | 99.871 | 99.388 | 98.356 |
59 | 100.000 | 100.000 | 99.871 | 99.399 | 98.397 |
60 | 100.000 | 100.000 | 99.880 | 99.431 | 98.467 |
61 | 100.000 | 100.000 | 99.880 | 99.443 | 98.509 |
62 | 100.000 | 100.000 | 99.889 | 99.474 | 98.575 |
63 | 100.000 | 100.000 | 99.898 | 99.505 | 98.642 |
64 | 100.000 | 100.000 | 99.898 | 99.516 | 98.681 |
65 | 100.000 | 100.000 | 99.908 | 99.546 | 98.744 |
66 | 100.000 | 100.000 | 99.908 | 99.555 | 98.778 |
67 | 100.000 | 100.000 | 99.908 | 99.564 | 98.808 |
68 | 100.000 | 100.000 | 99.908 | 99.572 | 98.839 |
69 | 100.000 | 100.000 | 99.908 | 99.580 | 98.870 |
70 | 100.000 | 100.000 | 99.908 | 99.589 | 98.900 |
71 | 100.000 | 100.000 | 99.908 | 99.596 | 98.927 |
72 | 100.000 | 100.000 | 99.917 | 99.624 | 98.982 |
73 | 100.000 | 100.000 | 99.917 | 99.631 | 99.008 |
74 | 100.000 | 100.000 | 99.926 | 99.659 | 99.062 |
75 | 100.000 | 100.000 | 99.935 | 99.687 | 99.113 |
76 | 100.000 | 100.000 | 99.935 | 99.693 | 99.137 |
77 | 100.000 | 100.000 | 99.945 | 99.720 | 99.188 |
78 | 100.000 | 100.000 | 99.954 | 99.747 | 99.239 |
79 | 100.000 | 100.000 | 99.954 | 99.753 | 99.262 |
80 | 100.000 | 100.000 | 99.954 | 99.759 | 99.285 |
81 | 100.000 | 100.000 | 99.954 | 99.766 | 99.308 |
82 | 100.000 | 100.000 | 99.963 | 99.793 | 99.358 |
83 | 100.000 | 100.000 | 99.972 | 99.820 | 99.409 |
84 | 100.000 | 100.000 | 99.982 | 99.847 | 99.460 |
85 | 100.000 | 100.000 | 99.991 | 99.874 | 99.511 |
86 | 100.000 | 100.000 | 100.000 | 99.901 | 99.561 |
87 | 100.000 | 100.000 | 100.000 | 99.906 | 99.581 |
88 | 100.000 | 100.000 | 100.000 | 99.911 | 99.600 |
89 | 100.000 | 100.000 | 100.000 | 99.917 | 99.620 |
90 | 100.000 | 100.000 | 100.000 | 99.922 | 99.639 |
91 | 100.000 | 100.000 | 100.000 | 99.927 | 99.659 |
92 | 100.000 | 100.000 | 100.000 | 99.932 | 99.679 |
93 | 100.000 | 100.000 | 100.000 | 99.938 | 99.699 |
94 | 100.000 | 100.000 | 100.000 | 99.942 | 99.714 |
95 | 100.000 | 100.000 | 100.000 | 99.946 | 99.730 |
96 | 100.000 | 100.000 | 100.000 | 99.950 | 99.747 |
97 | 100.000 | 100.000 | 100.000 | 99.954 | 99.762 |
98 | 100.000 | 100.000 | 100.000 | 99.958 | 99.777 |
99 | 100.000 | 100.000 | 100.000 | 99.963 | 99.793 |
100 | 100.000 | 100.000 | 100.000 | 99.967 | 99.808 |
101 | 100.000 | 100.000 | 100.000 | 99.971 | 99.825 |
102 | 100.000 | 100.000 | 100.000 | 99.974 | 99.836 |
103 | 100.000 | 100.000 | 100.000 | 99.977 | 99.848 |
104 | 100.000 | 100.000 | 100.000 | 99.980 | 99.860 |
105 | 100.000 | 100.000 | 100.000 | 99.983 | 99.872 |
106 | 100.000 | 100.000 | 100.000 | 99.986 | 99.884 |
107 | 100.000 | 100.000 | 100.000 | 99.990 | 99.895 |
108 | 100.000 | 100.000 | 100.000 | 99.993 | 99.907 |
109 | 100.000 | 100.000 | 100.000 | 99.996 | 99.918 |
110 | 100.000 | 100.000 | 100.000 | 99.999 | 99.930 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |


flex
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 79.070 | 62.109 | 48.777 | 38.407 | 30.360 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 90.698 | 79.374 | 68.014 | 57.508 | 48.199 |
5 | 93.023 | 83.314 | 72.925 | 62.880 | 53.659 |
6 | 95.349 | 87.486 | 78.252 | 68.754 | 59.628 |
7 | 97.674 | 91.194 | 82.756 | 73.646 | 64.612 |
8 | 97.674 | 91.889 | 84.252 | 75.820 | 67.271 |
9 | 100.000 | 95.597 | 88.655 | 80.467 | 71.882 |
10 | 100.000 | 96.524 | 90.529 | 83.049 | 74.900 |
11 | 100.000 | 96.987 | 91.554 | 84.578 | 76.819 |
12 | 100.000 | 97.219 | 92.209 | 85.710 | 78.380 |
13 | 100.000 | 97.683 | 93.215 | 87.193 | 80.227 |
14 | 100.000 | 98.030 | 94.018 | 88.434 | 81.828 |
15 | 100.000 | 98.262 | 94.581 | 89.341 | 83.042 |
16 | 100.000 | 98.262 | 94.729 | 89.742 | 83.734 |
17 | 100.000 | 98.262 | 94.877 | 90.140 | 84.416 |
18 | 100.000 | 98.262 | 94.969 | 90.404 | 84.895 |
19 | 100.000 | 98.262 | 95.052 | 90.645 | 85.335 |
20 | 100.000 | 98.262 | 95.154 | 90.926 | 85.830 |
21 | 100.000 | 98.494 | 95.717 | 91.823 | 87.010 |
22 | 100.000 | 98.725 | 96.252 | 92.653 | 88.089 |
23 | 100.000 | 98.841 | 96.520 | 93.084 | 88.683 |
24 | 100.000 | 98.957 | 96.815 | 93.584 | 89.384 |
25 | 100.000 | 99.073 | 97.083 | 94.015 | 89.978 |
26 | 100.000 | 99.073 | 97.175 | 94.272 | 90.429 |
27 | 100.000 | 99.073 | 97.231 | 94.436 | 90.737 |
28 | 100.000 | 99.073 | 97.295 | 94.622 | 91.074 |
29 | 100.000 | 99.073 | 97.341 | 94.763 | 91.346 |
30 | 100.000 | 99.189 | 97.600 | 95.163 | 91.877 |
31 | 100.000 | 99.189 | 97.665 | 95.350 | 92.216 |
32 | 100.000 | 99.305 | 97.923 | 95.750 | 92.747 |
33 | 100.000 | 99.305 | 97.969 | 95.889 | 93.011 |
34 | 100.000 | 99.305 | 98.006 | 96.003 | 93.230 |
35 | 100.000 | 99.305 | 98.043 | 96.112 | 93.436 |
36 | 100.000 | 99.305 | 98.080 | 96.222 | 93.645 |
37 | 100.000 | 99.305 | 98.089 | 96.265 | 93.749 |
38 | 100.000 | 99.305 | 98.117 | 96.354 | 93.925 |
39 | 100.000 | 99.421 | 98.366 | 96.726 | 94.405 |
40 | 100.000 | 99.537 | 98.615 | 97.098 | 94.883 |
41 | 100.000 | 99.652 | 98.865 | 97.469 | 95.357 |
42 | 100.000 | 99.652 | 98.883 | 97.533 | 95.490 |
43 | 100.000 | 99.652 | 98.902 | 97.596 | 95.622 |
44 | 100.000 | 99.652 | 98.911 | 97.636 | 95.716 |
45 | 100.000 | 99.652 | 98.920 | 97.677 | 95.815 |
46 | 100.000 | 99.652 | 98.948 | 97.760 | 95.971 |
47 | 100.000 | 99.652 | 98.966 | 97.820 | 96.093 |
48 | 100.000 | 99.652 | 98.994 | 97.900 | 96.242 |
49 | 100.000 | 99.652 | 99.012 | 97.958 | 96.355 |
50 | 100.000 | 99.652 | 99.012 | 97.968 | 96.393 |
51 | 100.000 | 99.652 | 99.012 | 97.976 | 96.425 |
52 | 100.000 | 99.652 | 99.022 | 98.011 | 96.501 |
53 | 100.000 | 99.652 | 99.040 | 98.067 | 96.611 |
54 | 100.000 | 99.768 | 99.289 | 98.434 | 97.070 |
55 | 100.000 | 99.884 | 99.538 | 98.799 | 97.527 |
56 | 100.000 | 99.884 | 99.548 | 98.836 | 97.609 |
57 | 100.000 | 99.884 | 99.557 | 98.871 | 97.689 |
58 | 100.000 | 100.000 | 99.806 | 99.237 | 98.145 |
59 | 100.000 | 100.000 | 99.825 | 99.292 | 98.251 |
60 | 100.000 | 100.000 | 99.834 | 99.324 | 98.320 |
61 | 100.000 | 100.000 | 99.843 | 99.355 | 98.387 |
62 | 100.000 | 100.000 | 99.852 | 99.385 | 98.450 |
63 | 100.000 | 100.000 | 99.862 | 99.416 | 98.513 |
64 | 100.000 | 100.000 | 99.862 | 99.425 | 98.547 |
65 | 100.000 | 100.000 | 99.862 | 99.432 | 98.574 |
66 | 100.000 | 100.000 | 99.862 | 99.442 | 98.607 |
67 | 100.000 | 100.000 | 99.862 | 99.448 | 98.631 |
68 | 100.000 | 100.000 | 99.880 | 99.502 | 98.733 |
69 | 100.000 | 100.000 | 99.889 | 99.534 | 98.802 |
70 | 100.000 | 100.000 | 99.889 | 99.542 | 98.829 |
71 | 100.000 | 100.000 | 99.889 | 99.549 | 98.855 |
72 | 100.000 | 100.000 | 99.889 | 99.554 | 98.875 |
73 | 100.000 | 100.000 | 99.889 | 99.560 | 98.898 |
74 | 100.000 | 100.000 | 99.889 | 99.565 | 98.915 |
75 | 100.000 | 100.000 | 99.889 | 99.571 | 98.939 |
76 | 100.000 | 100.000 | 99.908 | 99.625 | 99.040 |
77 | 100.000 | 100.000 | 99.908 | 99.631 | 99.063 |
78 | 100.000 | 100.000 | 99.908 | 99.639 | 99.089 |
79 | 100.000 | 100.000 | 99.908 | 99.644 | 99.108 |
80 | 100.000 | 100.000 | 99.908 | 99.649 | 99.128 |
81 | 100.000 | 100.000 | 99.908 | 99.654 | 99.147 |
82 | 100.000 | 100.000 | 99.908 | 99.660 | 99.170 |
83 | 100.000 | 100.000 | 99.908 | 99.666 | 99.189 |
84 | 100.000 | 100.000 | 99.908 | 99.671 | 99.209 |
85 | 100.000 | 100.000 | 99.908 | 99.675 | 99.225 |
86 | 100.000 | 100.000 | 99.908 | 99.679 | 99.240 |
87 | 100.000 | 100.000 | 99.908 | 99.683 | 99.256 |
88 | 100.000 | 100.000 | 99.908 | 99.688 | 99.272 |
89 | 100.000 | 100.000 | 99.908 | 99.692 | 99.287 |
90 | 100.000 | 100.000 | 99.908 | 99.696 | 99.302 |
91 | 100.000 | 100.000 | 99.908 | 99.700 | 99.318 |
92 | 100.000 | 100.000 | 99.908 | 99.703 | 99.330 |
93 | 100.000 | 100.000 | 99.908 | 99.706 | 99.342 |
94 | 100.000 | 100.000 | 99.917 | 99.733 | 99.393 |
95 | 100.000 | 100.000 | 99.917 | 99.736 | 99.404 |
96 | 100.000 | 100.000 | 99.926 | 99.764 | 99.455 |
97 | 100.000 | 100.000 | 99.935 | 99.791 | 99.506 |
98 | 100.000 | 100.000 | 99.945 | 99.818 | 99.557 |
99 | 100.000 | 100.000 | 99.954 | 99.845 | 99.608 |
100 | 100.000 | 100.000 | 99.954 | 99.848 | 99.619 |
101 | 100.000 | 100.000 | 99.963 | 99.875 | 99.670 |
102 | 100.000 | 100.000 | 99.972 | 99.902 | 99.721 |
103 | 100.000 | 100.000 | 99.972 | 99.905 | 99.732 |
104 | 100.000 | 100.000 | 99.972 | 99.908 | 99.744 |
105 | 100.000 | 100.000 | 99.982 | 99.935 | 99.795 |
106 | 100.000 | 100.000 | 99.991 | 99.963 | 99.845 |
107 | 100.000 | 100.000 | 100.000 | 99.990 | 99.896 |
108 | 100.000 | 100.000 | 100.000 | 99.993 | 99.907 |
109 | 100.000 | 100.000 | 100.000 | 99.996 | 99.919 |
110 | 100.000 | 100.000 | 100.000 | 99.999 | 99.930 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |


flex
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
4 | 90.698 | 80.185 | 69.371 | 59.041 | 49.654 |
5 | 95.349 | 86.790 | 76.645 | 66.321 | 56.585 |
6 | 100.000 | 93.163 | 83.486 | 73.051 | 62.925 |
7 | 100.000 | 94.322 | 85.876 | 76.378 | 66.822 |
8 | 100.000 | 95.365 | 87.935 | 79.158 | 70.018 |
9 | 100.000 | 96.176 | 89.597 | 81.480 | 72.770 |
10 | 100.000 | 96.871 | 90.954 | 83.288 | 74.819 |
11 | 100.000 | 97.335 | 91.997 | 84.858 | 76.797 |
12 | 100.000 | 97.683 | 92.754 | 85.966 | 78.159 |
13 | 100.000 | 97.914 | 93.298 | 86.817 | 79.267 |
14 | 100.000 | 98.146 | 93.908 | 87.839 | 80.660 |
15 | 100.000 | 98.378 | 94.498 | 88.802 | 81.938 |
16 | 100.000 | 98.610 | 95.080 | 89.746 | 83.196 |
17 | 100.000 | 98.841 | 95.661 | 90.688 | 84.445 |
18 | 100.000 | 98.957 | 95.920 | 91.085 | 84.964 |
19 | 100.000 | 99.073 | 96.243 | 91.648 | 85.757 |
20 | 100.000 | 99.189 | 96.612 | 92.332 | 86.750 |
21 | 100.000 | 99.305 | 96.889 | 92.779 | 87.355 |
22 | 100.000 | 99.421 | 97.166 | 93.223 | 87.950 |
23 | 100.000 | 99.537 | 97.415 | 93.588 | 88.405 |
24 | 100.000 | 99.652 | 97.665 | 93.963 | 88.892 |
25 | 100.000 | 99.768 | 97.960 | 94.456 | 89.569 |
26 | 100.000 | 99.884 | 98.218 | 94.854 | 90.091 |
27 | 100.000 | 100.000 | 98.468 | 95.225 | 90.564 |
28 | 100.000 | 100.000 | 98.495 | 95.305 | 90.711 |
29 | 100.000 | 100.000 | 98.615 | 95.634 | 91.284 |
30 | 100.000 | 100.000 | 98.745 | 95.990 | 91.906 |
31 | 100.000 | 100.000 | 98.818 | 96.202 | 92.289 |
32 | 100.000 | 100.000 | 98.883 | 96.382 | 92.608 |
33 | 100.000 | 100.000 | 98.929 | 96.520 | 92.869 |
34 | 100.000 | 100.000 | 98.975 | 96.655 | 93.116 |
35 | 100.000 | 100.000 | 99.012 | 96.766 | 93.327 |
36 | 100.000 | 100.000 | 99.058 | 96.899 | 93.571 |
37 | 100.000 | 100.000 | 99.105 | 97.033 | 93.815 |
38 | 100.000 | 100.000 | 99.142 | 97.147 | 94.037 |
39 | 100.000 | 100.000 | 99.151 | 97.180 | 94.108 |
40 | 100.000 | 100.000 | 99.188 | 97.286 | 94.302 |
41 | 100.000 | 100.000 | 99.262 | 97.501 | 94.699 |
42 | 100.000 | 100.000 | 99.280 | 97.569 | 94.845 |
43 | 100.000 | 100.000 | 99.298 | 97.633 | 94.978 |
44 | 100.000 | 100.000 | 99.326 | 97.718 | 95.144 |
45 | 100.000 | 100.000 | 99.326 | 97.733 | 95.195 |
46 | 100.000 | 100.000 | 99.354 | 97.816 | 95.355 |
47 | 100.000 | 100.000 | 99.382 | 97.898 | 95.511 |
48 | 100.000 | 100.000 | 99.428 | 98.026 | 95.737 |
49 | 100.000 | 100.000 | 99.465 | 98.137 | 95.946 |
50 | 100.000 | 100.000 | 99.483 | 98.194 | 96.058 |
51 | 100.000 | 100.000 | 99.511 | 98.276 | 96.213 |
52 | 100.000 | 100.000 | 99.511 | 98.285 | 96.243 |
53 | 100.000 | 100.000 | 99.520 | 98.325 | 96.340 |
54 | 100.000 | 100.000 | 99.529 | 98.354 | 96.399 |
55 | 100.000 | 100.000 | 99.548 | 98.409 | 96.500 |
56 | 100.000 | 100.000 | 99.594 | 98.543 | 96.747 |
57 | 100.000 | 100.000 | 99.594 | 98.548 | 96.768 |
58 | 100.000 | 100.000 | 99.603 | 98.577 | 96.827 |
59 | 100.000 | 100.000 | 99.631 | 98.664 | 96.996 |
60 | 100.000 | 100.000 | 99.658 | 98.748 | 97.158 |
61 | 100.000 | 100.000 | 99.686 | 98.828 | 97.305 |
62 | 100.000 | 100.000 | 99.695 | 98.864 | 97.386 |
63 | 100.000 | 100.000 | 99.695 | 98.868 | 97.404 |
64 | 100.000 | 100.000 | 99.705 | 98.898 | 97.467 |
65 | 100.000 | 100.000 | 99.714 | 98.926 | 97.521 |
66 | 100.000 | 100.000 | 99.714 | 98.929 | 97.533 |
67 | 100.000 | 100.000 | 99.732 | 98.990 | 97.656 |
68 | 100.000 | 100.000 | 99.751 | 99.053 | 97.789 |
69 | 100.000 | 100.000 | 99.760 | 99.086 | 97.858 |
70 | 100.000 | 100.000 | 99.760 | 99.089 | 97.869 |
71 | 100.000 | 100.000 | 99.760 | 99.098 | 97.903 |
72 | 100.000 | 100.000 | 99.778 | 99.153 | 98.008 |
73 | 100.000 | 100.000 | 99.788 | 99.180 | 98.059 |
74 | 100.000 | 100.000 | 99.797 | 99.213 | 98.129 |
75 | 100.000 | 100.000 | 99.806 | 99.242 | 98.188 |
76 | 100.000 | 100.000 | 99.825 | 99.295 | 98.286 |
77 | 100.000 | 100.000 | 99.834 | 99.322 | 98.336 |
78 | 100.000 | 100.000 | 99.843 | 99.350 | 98.391 |
79 | 100.000 | 100.000 | 99.843 | 99.354 | 98.407 |
80 | 100.000 | 100.000 | 99.852 | 99.381 | 98.457 |
81 | 100.000 | 100.000 | 99.862 | 99.410 | 98.516 |
82 | 100.000 | 100.000 | 99.862 | 99.413 | 98.524 |
83 | 100.000 | 100.000 | 99.871 | 99.448 | 98.604 |
84 | 100.000 | 100.000 | 99.871 | 99.454 | 98.627 |
85 | 100.000 | 100.000 | 99.880 | 99.481 | 98.677 |
86 | 100.000 | 100.000 | 99.880 | 99.483 | 98.686 |
87 | 100.000 | 100.000 | 99.880 | 99.488 | 98.702 |
88 | 100.000 | 100.000 | 99.880 | 99.488 | 98.702 |
89 | 100.000 | 100.000 | 99.889 | 99.518 | 98.764 |
90 | 100.000 | 100.000 | 99.908 | 99.579 | 98.891 |
91 | 100.000 | 100.000 | 99.908 | 99.583 | 98.907 |
92 | 100.000 | 100.000 | 99.908 | 99.587 | 98.918 |
93 | 100.000 | 100.000 | 99.908 | 99.591 | 98.934 |
94 | 100.000 | 100.000 | 99.917 | 99.622 | 99.000 |
95 | 100.000 | 100.000 | 99.935 | 99.677 | 99.106 |
96 | 100.000 | 100.000 | 99.935 | 99.677 | 99.106 |
97 | 100.000 | 100.000 | 99.935 | 99.688 | 99.143 |
98 | 100.000 | 100.000 | 99.935 | 99.691 | 99.155 |
99 | 100.000 | 100.000 | 99.945 | 99.718 | 99.206 |
100 | 100.000 | 100.000 | 99.945 | 99.721 | 99.218 |
101 | 100.000 | 100.000 | 99.945 | 99.724 | 99.229 |
102 | 100.000 | 100.000 | 99.945 | 99.724 | 99.230 |
103 | 100.000 | 100.000 | 99.945 | 99.729 | 99.249 |
104 | 100.000 | 100.000 | 99.945 | 99.733 | 99.265 |
105 | 100.000 | 100.000 | 99.954 | 99.761 | 99.321 |
106 | 100.000 | 100.000 | 99.954 | 99.766 | 99.336 |
107 | 100.000 | 100.000 | 99.954 | 99.768 | 99.344 |
108 | 100.000 | 100.000 | 99.954 | 99.769 | 99.348 |
109 | 100.000 | 100.000 | 99.954 | 99.773 | 99.364 |
110 | 100.000 | 100.000 | 99.963 | 99.801 | 99.420 |
111 | 100.000 | 100.000 | 99.963 | 99.805 | 99.435 |
112 | 100.000 | 100.000 | 99.963 | 99.806 | 99.440 |
113 | 100.000 | 100.000 | 99.963 | 99.809 | 99.452 |
114 | 100.000 | 100.000 | 99.963 | 99.813 | 99.464 |
115 | 100.000 | 100.000 | 99.963 | 99.815 | 99.473 |
116 | 100.000 | 100.000 | 99.963 | 99.819 | 99.488 |
117 | 100.000 | 100.000 | 99.963 | 99.821 | 99.496 |
118 | 100.000 | 100.000 | 99.963 | 99.822 | 99.500 |
119 | 100.000 | 100.000 | 99.963 | 99.826 | 99.516 |
120 | 100.000 | 100.000 | 99.963 | 99.828 | 99.525 |
121 | 100.000 | 100.000 | 99.963 | 99.829 | 99.530 |
122 | 100.000 | 100.000 | 99.963 | 99.833 | 99.545 |
123 | 100.000 | 100.000 | 99.963 | 99.834 | 99.550 |
124 | 100.000 | 100.000 | 99.972 | 99.861 | 99.601 |
125 | 100.000 | 100.000 | 99.972 | 99.861 | 99.603 |
126 | 100.000 | 100.000 | 99.972 | 99.861 | 99.603 |
127 | 100.000 | 100.000 | 99.972 | 99.863 | 99.608 |
128 | 100.000 | 100.000 | 99.982 | 99.891 | 99.662 |
129 | 100.000 | 100.000 | 99.982 | 99.894 | 99.675 |
130 | 100.000 | 100.000 | 99.982 | 99.897 | 99.687 |
131 | 100.000 | 100.000 | 99.982 | 99.899 | 99.695 |
132 | 100.000 | 100.000 | 99.982 | 99.899 | 99.696 |
133 | 100.000 | 100.000 | 99.982 | 99.903 | 99.712 |
134 | 100.000 | 100.000 | 99.982 | 99.905 | 99.720 |
135 | 100.000 | 100.000 | 99.982 | 99.907 | 99.728 |
136 | 100.000 | 100.000 | 99.982 | 99.908 | 99.733 |
137 | 100.000 | 100.000 | 99.982 | 99.911 | 99.745 |
138 | 100.000 | 100.000 | 99.982 | 99.914 | 99.753 |
139 | 100.000 | 100.000 | 99.982 | 99.916 | 99.761 |
140 | 100.000 | 100.000 | 99.982 | 99.919 | 99.772 |
141 | 100.000 | 100.000 | 99.982 | 99.919 | 99.773 |
142 | 100.000 | 100.000 | 99.982 | 99.920 | 99.777 |
143 | 100.000 | 100.000 | 99.982 | 99.922 | 99.785 |
144 | 100.000 | 100.000 | 99.982 | 99.924 | 99.794 |
145 | 100.000 | 100.000 | 99.982 | 99.924 | 99.796 |
146 | 100.000 | 100.000 | 99.982 | 99.926 | 99.804 |
147 | 100.000 | 100.000 | 99.991 | 99.953 | 99.854 |
148 | 100.000 | 100.000 | 99.991 | 99.953 | 99.855 |
149 | 100.000 | 100.000 | 99.991 | 99.955 | 99.863 |
150 | 100.000 | 100.000 | 99.991 | 99.956 | 99.867 |
151 | 100.000 | 100.000 | 99.991 | 99.956 | 99.867 |
152 | 100.000 | 100.000 | 99.991 | 99.958 | 99.875 |
153 | 100.000 | 100.000 | 99.991 | 99.959 | 99.880 |
154 | 100.000 | 100.000 | 99.991 | 99.959 | 99.881 |
155 | 100.000 | 100.000 | 99.991 | 99.961 | 99.889 |
156 | 100.000 | 100.000 | 99.991 | 99.964 | 99.896 |
157 | 100.000 | 100.000 | 99.991 | 99.964 | 99.898 |
158 | 100.000 | 100.000 | 99.991 | 99.965 | 99.902 |
159 | 100.000 | 100.000 | 99.991 | 99.965 | 99.903 |
160 | 100.000 | 100.000 | 99.991 | 99.965 | 99.904 |
161 | 100.000 | 100.000 | 99.991 | 99.965 | 99.905 |
162 | 100.000 | 100.000 | 99.991 | 99.967 | 99.913 |
163 | 100.000 | 100.000 | 99.991 | 99.968 | 99.918 |
164 | 100.000 | 100.000 | 99.991 | 99.969 | 99.922 |
165 | 100.000 | 100.000 | 99.991 | 99.969 | 99.923 |
166 | 100.000 | 100.000 | 99.991 | 99.969 | 99.924 |
167 | 100.000 | 100.000 | 100.000 | 99.996 | 99.975 |
168 | 100.000 | 100.000 | 100.000 | 99.997 | 99.979 |
169 | 100.000 | 100.000 | 100.000 | 99.998 | 99.984 |
170 | 100.000 | 100.000 | 100.000 | 99.998 | 99.984 |
171 | 100.000 | 100.000 | 100.000 | 99.998 | 99.985 |
172 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
173 | 100.000 | 100.000 | 100.000 | 99.998 | 99.987 |
174 | 100.000 | 100.000 | 100.000 | 99.998 | 99.988 |
175 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
176 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
177 | 100.000 | 100.000 | 100.000 | 99.999 | 99.994 |
178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


flex
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
4 | 93.023 | 82.619 | 71.291 | 60.381 | 50.518 |
5 | 95.349 | 86.559 | 76.230 | 65.817 | 56.069 |
6 | 97.674 | 90.382 | 80.947 | 70.967 | 61.313 |
7 | 97.674 | 91.425 | 83.153 | 74.099 | 65.039 |
8 | 100.000 | 95.249 | 87.778 | 79.027 | 69.942 |
9 | 100.000 | 96.176 | 89.661 | 81.632 | 72.997 |
10 | 100.000 | 96.871 | 91.138 | 83.755 | 75.571 |
11 | 100.000 | 97.219 | 92.015 | 85.177 | 77.453 |
12 | 100.000 | 97.567 | 92.837 | 86.465 | 79.132 |
13 | 100.000 | 97.914 | 93.658 | 87.745 | 80.785 |
14 | 100.000 | 98.146 | 94.258 | 88.733 | 82.106 |
15 | 100.000 | 98.378 | 94.849 | 89.712 | 83.437 |
16 | 100.000 | 98.378 | 95.034 | 90.202 | 84.266 |
17 | 100.000 | 98.610 | 95.588 | 91.081 | 85.426 |
18 | 100.000 | 98.725 | 95.938 | 91.726 | 86.367 |
19 | 100.000 | 98.841 | 96.280 | 92.330 | 87.214 |
20 | 100.000 | 98.957 | 96.612 | 92.923 | 88.063 |
21 | 100.000 | 99.189 | 97.111 | 93.656 | 88.981 |
22 | 100.000 | 99.305 | 97.406 | 94.158 | 89.689 |
23 | 100.000 | 99.305 | 97.498 | 94.418 | 90.154 |
24 | 100.000 | 99.305 | 97.581 | 94.650 | 90.562 |
25 | 100.000 | 99.421 | 97.840 | 95.051 | 91.095 |
26 | 100.000 | 99.537 | 98.098 | 95.446 | 91.611 |
27 | 100.000 | 99.537 | 98.172 | 95.659 | 91.996 |
28 | 100.000 | 99.652 | 98.421 | 96.026 | 92.456 |
29 | 100.000 | 99.768 | 98.671 | 96.391 | 92.911 |
30 | 100.000 | 99.884 | 98.920 | 96.768 | 93.404 |
31 | 100.000 | 99.884 | 98.985 | 96.955 | 93.743 |
32 | 100.000 | 100.000 | 99.234 | 97.322 | 94.206 |
33 | 100.000 | 100.000 | 99.289 | 97.480 | 94.488 |
34 | 100.000 | 100.000 | 99.335 | 97.616 | 94.742 |
35 | 100.000 | 100.000 | 99.382 | 97.758 | 95.013 |
36 | 100.000 | 100.000 | 99.428 | 97.900 | 95.288 |
37 | 100.000 | 100.000 | 99.465 | 98.020 | 95.527 |
38 | 100.000 | 100.000 | 99.502 | 98.136 | 95.753 |
39 | 100.000 | 100.000 | 99.538 | 98.240 | 95.940 |
40 | 100.000 | 100.000 | 99.575 | 98.355 | 96.167 |
41 | 100.000 | 100.000 | 99.612 | 98.460 | 96.354 |
42 | 100.000 | 100.000 | 99.649 | 98.570 | 96.563 |
43 | 100.000 | 100.000 | 99.686 | 98.684 | 96.782 |
44 | 100.000 | 100.000 | 99.723 | 98.791 | 96.980 |
45 | 100.000 | 100.000 | 99.751 | 98.878 | 97.151 |
46 | 100.000 | 100.000 | 99.778 | 98.961 | 97.307 |
47 | 100.000 | 100.000 | 99.806 | 99.051 | 97.489 |
48 | 100.000 | 100.000 | 99.834 | 99.139 | 97.663 |
49 | 100.000 | 100.000 | 99.852 | 99.196 | 97.775 |
50 | 100.000 | 100.000 | 99.871 | 99.254 | 97.892 |
51 | 100.000 | 100.000 | 99.880 | 99.281 | 97.943 |
52 | 100.000 | 100.000 | 99.889 | 99.316 | 98.019 |
53 | 100.000 | 100.000 | 99.898 | 99.351 | 98.100 |
54 | 100.000 | 100.000 | 99.908 | 99.382 | 98.167 |
55 | 100.000 | 100.000 | 99.917 | 99.409 | 98.218 |
56 | 100.000 | 100.000 | 99.926 | 99.438 | 98.273 |
57 | 100.000 | 100.000 | 99.935 | 99.475 | 98.360 |
58 | 100.000 | 100.000 | 99.945 | 99.505 | 98.421 |
59 | 100.000 | 100.000 | 99.954 | 99.532 | 98.473 |
60 | 100.000 | 100.000 | 99.963 | 99.560 | 98.527 |
61 | 100.000 | 100.000 | 99.972 | 99.588 | 98.578 |
62 | 100.000 | 100.000 | 99.982 | 99.616 | 98.632 |
63 | 100.000 | 100.000 | 99.991 | 99.643 | 98.683 |
64 | 100.000 | 100.000 | 100.000 | 99.671 | 98.737 |
65 | 100.000 | 100.000 | 100.000 | 99.675 | 98.752 |
66 | 100.000 | 100.000 | 100.000 | 99.685 | 98.789 |
67 | 100.000 | 100.000 | 100.000 | 99.694 | 98.819 |
68 | 100.000 | 100.000 | 100.000 | 99.699 | 98.839 |
69 | 100.000 | 100.000 | 100.000 | 99.708 | 98.873 |
70 | 100.000 | 100.000 | 100.000 | 99.717 | 98.904 |
71 | 100.000 | 100.000 | 100.000 | 99.726 | 98.938 |
72 | 100.000 | 100.000 | 100.000 | 99.736 | 98.975 |
73 | 100.000 | 100.000 | 100.000 | 99.743 | 98.997 |
74 | 100.000 | 100.000 | 100.000 | 99.748 | 99.017 |
75 | 100.000 | 100.000 | 100.000 | 99.753 | 99.036 |
76 | 100.000 | 100.000 | 100.000 | 99.759 | 99.059 |
77 | 100.000 | 100.000 | 100.000 | 99.761 | 99.067 |
78 | 100.000 | 100.000 | 100.000 | 99.767 | 99.087 |
79 | 100.000 | 100.000 | 100.000 | 99.772 | 99.106 |
80 | 100.000 | 100.000 | 100.000 | 99.779 | 99.132 |
81 | 100.000 | 100.000 | 100.000 | 99.788 | 99.162 |
82 | 100.000 | 100.000 | 100.000 | 99.793 | 99.182 |
83 | 100.000 | 100.000 | 100.000 | 99.799 | 99.205 |
84 | 100.000 | 100.000 | 100.000 | 99.803 | 99.222 |
85 | 100.000 | 100.000 | 100.000 | 99.806 | 99.233 |
86 | 100.000 | 100.000 | 100.000 | 99.816 | 99.267 |
87 | 100.000 | 100.000 | 100.000 | 99.823 | 99.293 |
88 | 100.000 | 100.000 | 100.000 | 99.828 | 99.313 |
89 | 100.000 | 100.000 | 100.000 | 99.832 | 99.329 |
90 | 100.000 | 100.000 | 100.000 | 99.838 | 99.348 |
91 | 100.000 | 100.000 | 100.000 | 99.841 | 99.360 |
92 | 100.000 | 100.000 | 100.000 | 99.846 | 99.380 |
93 | 100.000 | 100.000 | 100.000 | 99.851 | 99.399 |
94 | 100.000 | 100.000 | 100.000 | 99.855 | 99.415 |
95 | 100.000 | 100.000 | 100.000 | 99.856 | 99.419 |
96 | 100.000 | 100.000 | 100.000 | 99.860 | 99.435 |
97 | 100.000 | 100.000 | 100.000 | 99.865 | 99.450 |
98 | 100.000 | 100.000 | 100.000 | 99.869 | 99.466 |
99 | 100.000 | 100.000 | 100.000 | 99.871 | 99.474 |
100 | 100.000 | 100.000 | 100.000 | 99.877 | 99.497 |
101 | 100.000 | 100.000 | 100.000 | 99.880 | 99.508 |
102 | 100.000 | 100.000 | 100.000 | 99.885 | 99.528 |
103 | 100.000 | 100.000 | 100.000 | 99.890 | 99.544 |
104 | 100.000 | 100.000 | 100.000 | 99.894 | 99.559 |
105 | 100.000 | 100.000 | 100.000 | 99.897 | 99.571 |
106 | 100.000 | 100.000 | 100.000 | 99.900 | 99.583 |
107 | 100.000 | 100.000 | 100.000 | 99.901 | 99.587 |
108 | 100.000 | 100.000 | 100.000 | 99.902 | 99.591 |
109 | 100.000 | 100.000 | 100.000 | 99.905 | 99.604 |
110 | 100.000 | 100.000 | 100.000 | 99.908 | 99.616 |
111 | 100.000 | 100.000 | 100.000 | 99.913 | 99.632 |
112 | 100.000 | 100.000 | 100.000 | 99.915 | 99.641 |
113 | 100.000 | 100.000 | 100.000 | 99.916 | 99.646 |
114 | 100.000 | 100.000 | 100.000 | 99.916 | 99.648 |
115 | 100.000 | 100.000 | 100.000 | 99.919 | 99.660 |
116 | 100.000 | 100.000 | 100.000 | 99.922 | 99.671 |
117 | 100.000 | 100.000 | 100.000 | 99.924 | 99.679 |
118 | 100.000 | 100.000 | 100.000 | 99.927 | 99.691 |
119 | 100.000 | 100.000 | 100.000 | 99.928 | 99.696 |
120 | 100.000 | 100.000 | 100.000 | 99.934 | 99.719 |
121 | 100.000 | 100.000 | 100.000 | 99.938 | 99.731 |
122 | 100.000 | 100.000 | 100.000 | 99.940 | 99.739 |
123 | 100.000 | 100.000 | 100.000 | 99.941 | 99.744 |
124 | 100.000 | 100.000 | 100.000 | 99.944 | 99.756 |
125 | 100.000 | 100.000 | 100.000 | 99.944 | 99.757 |
126 | 100.000 | 100.000 | 100.000 | 99.946 | 99.765 |
127 | 100.000 | 100.000 | 100.000 | 99.947 | 99.769 |
128 | 100.000 | 100.000 | 100.000 | 99.947 | 99.770 |
129 | 100.000 | 100.000 | 100.000 | 99.948 | 99.775 |
130 | 100.000 | 100.000 | 100.000 | 99.948 | 99.775 |
131 | 100.000 | 100.000 | 100.000 | 99.950 | 99.783 |
132 | 100.000 | 100.000 | 100.000 | 99.951 | 99.788 |
133 | 100.000 | 100.000 | 100.000 | 99.955 | 99.803 |
134 | 100.000 | 100.000 | 100.000 | 99.959 | 99.819 |
135 | 100.000 | 100.000 | 100.000 | 99.960 | 99.824 |
136 | 100.000 | 100.000 | 100.000 | 99.963 | 99.832 |
137 | 100.000 | 100.000 | 100.000 | 99.963 | 99.833 |
138 | 100.000 | 100.000 | 100.000 | 99.964 | 99.837 |
139 | 100.000 | 100.000 | 100.000 | 99.966 | 99.846 |
140 | 100.000 | 100.000 | 100.000 | 99.967 | 99.851 |
141 | 100.000 | 100.000 | 100.000 | 99.969 | 99.859 |
142 | 100.000 | 100.000 | 100.000 | 99.970 | 99.864 |
143 | 100.000 | 100.000 | 100.000 | 99.970 | 99.864 |
144 | 100.000 | 100.000 | 100.000 | 99.971 | 99.869 |
145 | 100.000 | 100.000 | 100.000 | 99.973 | 99.877 |
146 | 100.000 | 100.000 | 100.000 | 99.974 | 99.881 |
147 | 100.000 | 100.000 | 100.000 | 99.977 | 99.893 |
148 | 100.000 | 100.000 | 100.000 | 99.980 | 99.904 |
149 | 100.000 | 100.000 | 100.000 | 99.982 | 99.912 |
150 | 100.000 | 100.000 | 100.000 | 99.982 | 99.913 |
151 | 100.000 | 100.000 | 100.000 | 99.983 | 99.918 |
152 | 100.000 | 100.000 | 100.000 | 99.985 | 99.926 |
153 | 100.000 | 100.000 | 100.000 | 99.988 | 99.934 |
154 | 100.000 | 100.000 | 100.000 | 99.991 | 99.945 |
155 | 100.000 | 100.000 | 100.000 | 99.991 | 99.946 |
156 | 100.000 | 100.000 | 100.000 | 99.992 | 99.952 |
157 | 100.000 | 100.000 | 100.000 | 99.994 | 99.959 |
158 | 100.000 | 100.000 | 100.000 | 99.996 | 99.967 |
159 | 100.000 | 100.000 | 100.000 | 99.997 | 99.972 |
160 | 100.000 | 100.000 | 100.000 | 99.997 | 99.972 |
161 | 100.000 | 100.000 | 100.000 | 99.997 | 99.973 |
162 | 100.000 | 100.000 | 100.000 | 99.997 | 99.974 |
163 | 100.000 | 100.000 | 100.000 | 99.997 | 99.975 |
164 | 100.000 | 100.000 | 100.000 | 99.997 | 99.977 |
165 | 100.000 | 100.000 | 100.000 | 99.997 | 99.978 |
166 | 100.000 | 100.000 | 100.000 | 99.997 | 99.979 |
167 | 100.000 | 100.000 | 100.000 | 99.997 | 99.980 |
168 | 100.000 | 100.000 | 100.000 | 99.998 | 99.985 |
169 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
170 | 100.000 | 100.000 | 100.000 | 99.999 | 99.990 |
171 | 100.000 | 100.000 | 100.000 | 99.999 | 99.991 |
172 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
173 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
174 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
175 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
176 | 100.000 | 100.000 | 100.000 | 99.999 | 99.994 |
177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


flex
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 90.698 | 79.722 | 68.596 | 58.165 | 48.823 |
5 | 95.349 | 86.327 | 75.870 | 65.445 | 55.754 |
6 | 97.674 | 90.267 | 80.809 | 70.881 | 61.305 |
7 | 97.674 | 90.846 | 82.110 | 72.833 | 63.749 |
8 | 100.000 | 94.322 | 86.135 | 77.062 | 67.971 |
9 | 100.000 | 95.017 | 87.630 | 79.227 | 70.602 |
10 | 100.000 | 96.060 | 89.689 | 82.005 | 73.792 |
11 | 100.000 | 96.640 | 90.907 | 83.753 | 75.921 |
12 | 100.000 | 97.103 | 91.951 | 85.325 | 77.902 |
13 | 100.000 | 97.567 | 92.966 | 86.827 | 79.771 |
14 | 100.000 | 97.914 | 93.788 | 88.109 | 81.431 |
15 | 100.000 | 98.030 | 94.175 | 88.844 | 82.513 |
16 | 100.000 | 98.146 | 94.517 | 89.467 | 83.422 |
17 | 100.000 | 98.262 | 94.849 | 90.067 | 84.294 |
18 | 100.000 | 98.494 | 95.384 | 90.898 | 85.376 |
19 | 100.000 | 98.725 | 95.920 | 91.727 | 86.451 |
20 | 100.000 | 98.725 | 96.012 | 91.993 | 86.934 |
21 | 100.000 | 98.957 | 96.529 | 92.778 | 87.944 |
22 | 100.000 | 99.073 | 96.852 | 93.343 | 88.740 |
23 | 100.000 | 99.189 | 97.157 | 93.864 | 89.474 |
24 | 100.000 | 99.189 | 97.240 | 94.099 | 89.892 |
25 | 100.000 | 99.305 | 97.526 | 94.572 | 90.544 |
26 | 100.000 | 99.305 | 97.609 | 94.805 | 90.960 |
27 | 100.000 | 99.305 | 97.692 | 95.036 | 91.368 |
28 | 100.000 | 99.305 | 97.757 | 95.226 | 91.718 |
29 | 100.000 | 99.305 | 97.821 | 95.413 | 92.062 |
30 | 100.000 | 99.305 | 97.886 | 95.599 | 92.399 |
31 | 100.000 | 99.305 | 97.932 | 95.737 | 92.661 |
32 | 100.000 | 99.305 | 97.988 | 95.894 | 92.945 |
33 | 100.000 | 99.305 | 98.034 | 96.031 | 93.199 |
34 | 100.000 | 99.421 | 98.292 | 96.427 | 93.715 |
35 | 100.000 | 99.421 | 98.320 | 96.519 | 93.904 |
36 | 100.000 | 99.421 | 98.348 | 96.610 | 94.087 |
37 | 100.000 | 99.537 | 98.597 | 96.982 | 94.564 |
38 | 100.000 | 99.537 | 98.615 | 97.050 | 94.713 |
39 | 100.000 | 99.537 | 98.652 | 97.161 | 94.923 |
40 | 100.000 | 99.652 | 98.902 | 97.531 | 95.394 |
41 | 100.000 | 99.652 | 98.938 | 97.639 | 95.596 |
42 | 100.000 | 99.768 | 99.188 | 98.008 | 96.062 |
43 | 100.000 | 99.884 | 99.437 | 98.376 | 96.529 |
44 | 100.000 | 99.884 | 99.474 | 98.482 | 96.724 |
45 | 100.000 | 99.884 | 99.502 | 98.568 | 96.891 |
46 | 100.000 | 100.000 | 99.751 | 98.933 | 97.347 |
47 | 100.000 | 100.000 | 99.769 | 98.997 | 97.480 |
48 | 100.000 | 100.000 | 99.806 | 99.101 | 97.667 |
49 | 100.000 | 100.000 | 99.825 | 99.163 | 97.794 |
50 | 100.000 | 100.000 | 99.843 | 99.224 | 97.920 |
51 | 100.000 | 100.000 | 99.852 | 99.262 | 98.007 |
52 | 100.000 | 100.000 | 99.871 | 99.319 | 98.120 |
53 | 100.000 | 100.000 | 99.880 | 99.354 | 98.200 |
54 | 100.000 | 100.000 | 99.898 | 99.408 | 98.302 |
55 | 100.000 | 100.000 | 99.898 | 99.421 | 98.348 |
56 | 100.000 | 100.000 | 99.898 | 99.432 | 98.388 |
57 | 100.000 | 100.000 | 99.908 | 99.465 | 98.457 |
58 | 100.000 | 100.000 | 99.908 | 99.476 | 98.497 |
59 | 100.000 | 100.000 | 99.908 | 99.487 | 98.533 |
60 | 100.000 | 100.000 | 99.908 | 99.496 | 98.567 |
61 | 100.000 | 100.000 | 99.908 | 99.504 | 98.597 |
62 | 100.000 | 100.000 | 99.908 | 99.513 | 98.627 |
63 | 100.000 | 100.000 | 99.908 | 99.520 | 98.654 |
64 | 100.000 | 100.000 | 99.908 | 99.527 | 98.680 |
65 | 100.000 | 100.000 | 99.908 | 99.534 | 98.707 |
66 | 100.000 | 100.000 | 99.908 | 99.541 | 98.730 |
67 | 100.000 | 100.000 | 99.917 | 99.568 | 98.780 |
68 | 100.000 | 100.000 | 99.917 | 99.574 | 98.803 |
69 | 100.000 | 100.000 | 99.926 | 99.601 | 98.854 |
70 | 100.000 | 100.000 | 99.926 | 99.607 | 98.877 |
71 | 100.000 | 100.000 | 99.935 | 99.634 | 98.928 |
72 | 100.000 | 100.000 | 99.945 | 99.662 | 98.979 |
73 | 100.000 | 100.000 | 99.945 | 99.668 | 99.002 |
74 | 100.000 | 100.000 | 99.954 | 99.695 | 99.053 |
75 | 100.000 | 100.000 | 99.954 | 99.701 | 99.076 |
76 | 100.000 | 100.000 | 99.963 | 99.728 | 99.127 |
77 | 100.000 | 100.000 | 99.972 | 99.755 | 99.177 |
78 | 100.000 | 100.000 | 99.982 | 99.782 | 99.229 |
79 | 100.000 | 100.000 | 99.991 | 99.809 | 99.280 |
80 | 100.000 | 100.000 | 99.991 | 99.816 | 99.303 |
81 | 100.000 | 100.000 | 100.000 | 99.843 | 99.354 |
82 | 100.000 | 100.000 | 100.000 | 99.848 | 99.374 |
83 | 100.000 | 100.000 | 100.000 | 99.853 | 99.393 |
84 | 100.000 | 100.000 | 100.000 | 99.858 | 99.413 |
85 | 100.000 | 100.000 | 100.000 | 99.864 | 99.432 |
86 | 100.000 | 100.000 | 100.000 | 99.869 | 99.451 |
87 | 100.000 | 100.000 | 100.000 | 99.873 | 99.467 |
88 | 100.000 | 100.000 | 100.000 | 99.877 | 99.482 |
89 | 100.000 | 100.000 | 100.000 | 99.881 | 99.498 |
90 | 100.000 | 100.000 | 100.000 | 99.885 | 99.514 |
91 | 100.000 | 100.000 | 100.000 | 99.890 | 99.530 |
92 | 100.000 | 100.000 | 100.000 | 99.894 | 99.547 |
93 | 100.000 | 100.000 | 100.000 | 99.898 | 99.562 |
94 | 100.000 | 100.000 | 100.000 | 99.902 | 99.578 |
95 | 100.000 | 100.000 | 100.000 | 99.906 | 99.593 |
96 | 100.000 | 100.000 | 100.000 | 99.910 | 99.609 |
97 | 100.000 | 100.000 | 100.000 | 99.915 | 99.624 |
98 | 100.000 | 100.000 | 100.000 | 99.918 | 99.636 |
99 | 100.000 | 100.000 | 100.000 | 99.921 | 99.648 |
100 | 100.000 | 100.000 | 100.000 | 99.924 | 99.660 |
101 | 100.000 | 100.000 | 100.000 | 99.927 | 99.671 |
102 | 100.000 | 100.000 | 100.000 | 99.930 | 99.684 |
103 | 100.000 | 100.000 | 100.000 | 99.933 | 99.696 |
104 | 100.000 | 100.000 | 100.000 | 99.936 | 99.708 |
105 | 100.000 | 100.000 | 100.000 | 99.940 | 99.721 |
106 | 100.000 | 100.000 | 100.000 | 99.943 | 99.733 |
107 | 100.000 | 100.000 | 100.000 | 99.945 | 99.742 |
108 | 100.000 | 100.000 | 100.000 | 99.947 | 99.750 |
109 | 100.000 | 100.000 | 100.000 | 99.949 | 99.758 |
110 | 100.000 | 100.000 | 100.000 | 99.951 | 99.766 |
111 | 100.000 | 100.000 | 100.000 | 99.953 | 99.774 |
112 | 100.000 | 100.000 | 100.000 | 99.955 | 99.782 |
113 | 100.000 | 100.000 | 100.000 | 99.957 | 99.790 |
114 | 100.000 | 100.000 | 100.000 | 99.959 | 99.799 |
115 | 100.000 | 100.000 | 100.000 | 99.961 | 99.807 |
116 | 100.000 | 100.000 | 100.000 | 99.964 | 99.815 |
117 | 100.000 | 100.000 | 100.000 | 99.966 | 99.823 |
118 | 100.000 | 100.000 | 100.000 | 99.968 | 99.831 |
119 | 100.000 | 100.000 | 100.000 | 99.970 | 99.840 |
120 | 100.000 | 100.000 | 100.000 | 99.972 | 99.848 |
121 | 100.000 | 100.000 | 100.000 | 99.974 | 99.857 |
122 | 100.000 | 100.000 | 100.000 | 99.976 | 99.865 |
123 | 100.000 | 100.000 | 100.000 | 99.978 | 99.873 |
124 | 100.000 | 100.000 | 100.000 | 99.980 | 99.881 |
125 | 100.000 | 100.000 | 100.000 | 99.982 | 99.889 |
126 | 100.000 | 100.000 | 100.000 | 99.984 | 99.897 |
127 | 100.000 | 100.000 | 100.000 | 99.986 | 99.905 |
128 | 100.000 | 100.000 | 100.000 | 99.989 | 99.913 |
129 | 100.000 | 100.000 | 100.000 | 99.991 | 99.921 |
130 | 100.000 | 100.000 | 100.000 | 99.993 | 99.929 |
131 | 100.000 | 100.000 | 100.000 | 99.994 | 99.934 |
132 | 100.000 | 100.000 | 100.000 | 99.995 | 99.939 |
133 | 100.000 | 100.000 | 100.000 | 99.996 | 99.944 |
134 | 100.000 | 100.000 | 100.000 | 99.997 | 99.949 |
135 | 100.000 | 100.000 | 100.000 | 99.998 | 99.954 |
136 | 100.000 | 100.000 | 100.000 | 99.999 | 99.959 |
137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.966 |
141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
144 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
145 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
146 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
147 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
148 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
149 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
150 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
151 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
152 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
153 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
154 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
155 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
156 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
157 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
158 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
159 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
160 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
161 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
162 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
163 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
164 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
165 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
166 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
167 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
168 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
169 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
170 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
171 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
172 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
173 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
174 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
175 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
176 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


flex
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
5 | 95.349 | 86.327 | 75.852 | 65.403 | 55.691 |
6 | 95.349 | 87.601 | 78.445 | 68.973 | 59.835 |
7 | 95.349 | 88.528 | 80.384 | 71.714 | 63.102 |
8 | 95.349 | 89.108 | 81.723 | 73.754 | 65.676 |
9 | 97.674 | 92.932 | 86.357 | 78.710 | 70.634 |
10 | 97.674 | 93.163 | 87.012 | 79.849 | 72.214 |
11 | 97.674 | 93.859 | 88.461 | 81.905 | 74.684 |
12 | 97.674 | 94.670 | 90.067 | 84.089 | 77.221 |
13 | 100.000 | 97.914 | 93.640 | 87.720 | 80.778 |
14 | 100.000 | 98.030 | 94.018 | 88.436 | 81.834 |
15 | 100.000 | 98.262 | 94.628 | 89.451 | 83.208 |
16 | 100.000 | 98.378 | 95.006 | 90.161 | 84.244 |
17 | 100.000 | 98.494 | 95.357 | 90.801 | 85.172 |
18 | 100.000 | 98.494 | 95.486 | 91.157 | 85.795 |
19 | 100.000 | 98.494 | 95.597 | 91.468 | 86.346 |
20 | 100.000 | 98.610 | 95.901 | 91.991 | 87.084 |
21 | 100.000 | 98.610 | 95.994 | 92.247 | 87.536 |
22 | 100.000 | 98.841 | 96.492 | 92.989 | 88.483 |
23 | 100.000 | 98.841 | 96.594 | 93.272 | 88.980 |
24 | 100.000 | 99.073 | 97.101 | 94.033 | 89.950 |
25 | 100.000 | 99.073 | 97.185 | 94.264 | 90.360 |
26 | 100.000 | 99.073 | 97.240 | 94.434 | 90.684 |
27 | 100.000 | 99.073 | 97.286 | 94.576 | 90.956 |
28 | 100.000 | 99.073 | 97.323 | 94.691 | 91.183 |
29 | 100.000 | 99.073 | 97.406 | 94.925 | 91.597 |
30 | 100.000 | 99.073 | 97.434 | 95.017 | 91.787 |
31 | 100.000 | 99.073 | 97.461 | 95.109 | 91.974 |
32 | 100.000 | 99.189 | 97.729 | 95.533 | 92.543 |
33 | 100.000 | 99.189 | 97.775 | 95.672 | 92.807 |
34 | 100.000 | 99.189 | 97.794 | 95.736 | 92.942 |
35 | 100.000 | 99.305 | 98.043 | 96.111 | 93.430 |
36 | 100.000 | 99.305 | 98.052 | 96.151 | 93.525 |
37 | 100.000 | 99.305 | 98.098 | 96.284 | 93.769 |
38 | 100.000 | 99.305 | 98.145 | 96.416 | 94.010 |
39 | 100.000 | 99.305 | 98.154 | 96.454 | 94.098 |
40 | 100.000 | 99.305 | 98.181 | 96.540 | 94.267 |
41 | 100.000 | 99.305 | 98.218 | 96.646 | 94.462 |
42 | 100.000 | 99.421 | 98.477 | 97.038 | 94.965 |
43 | 100.000 | 99.421 | 98.514 | 97.144 | 95.161 |
44 | 100.000 | 99.421 | 98.532 | 97.205 | 95.284 |
45 | 100.000 | 99.421 | 98.551 | 97.264 | 95.403 |
46 | 100.000 | 99.421 | 98.560 | 97.299 | 95.484 |
47 | 100.000 | 99.537 | 98.809 | 97.666 | 95.944 |
48 | 100.000 | 99.537 | 98.818 | 97.703 | 96.031 |
49 | 100.000 | 99.652 | 99.068 | 98.071 | 96.494 |
50 | 100.000 | 99.652 | 99.077 | 98.105 | 96.571 |
51 | 100.000 | 99.652 | 99.086 | 98.141 | 96.650 |
52 | 100.000 | 99.768 | 99.335 | 98.507 | 97.110 |
53 | 100.000 | 99.884 | 99.585 | 98.874 | 97.569 |
54 | 100.000 | 99.884 | 99.603 | 98.931 | 97.681 |
55 | 100.000 | 99.884 | 99.612 | 98.963 | 97.748 |
56 | 100.000 | 99.884 | 99.612 | 98.972 | 97.783 |
57 | 100.000 | 99.884 | 99.622 | 99.006 | 97.858 |
58 | 100.000 | 100.000 | 99.871 | 99.372 | 98.314 |
59 | 100.000 | 100.000 | 99.889 | 99.429 | 98.426 |
60 | 100.000 | 100.000 | 99.889 | 99.439 | 98.460 |
61 | 100.000 | 100.000 | 99.889 | 99.450 | 98.500 |
62 | 100.000 | 100.000 | 99.889 | 99.457 | 98.527 |
63 | 100.000 | 100.000 | 99.889 | 99.466 | 98.557 |
64 | 100.000 | 100.000 | 99.889 | 99.473 | 98.584 |
65 | 100.000 | 100.000 | 99.889 | 99.480 | 98.611 |
66 | 100.000 | 100.000 | 99.908 | 99.533 | 98.709 |
67 | 100.000 | 100.000 | 99.908 | 99.542 | 98.739 |
68 | 100.000 | 100.000 | 99.908 | 99.548 | 98.762 |
69 | 100.000 | 100.000 | 99.908 | 99.556 | 98.792 |
70 | 100.000 | 100.000 | 99.908 | 99.563 | 98.815 |
71 | 100.000 | 100.000 | 99.908 | 99.569 | 98.838 |
72 | 100.000 | 100.000 | 99.908 | 99.573 | 98.855 |
73 | 100.000 | 100.000 | 99.908 | 99.578 | 98.875 |
74 | 100.000 | 100.000 | 99.908 | 99.584 | 98.897 |
75 | 100.000 | 100.000 | 99.908 | 99.590 | 98.917 |
76 | 100.000 | 100.000 | 99.908 | 99.595 | 98.936 |
77 | 100.000 | 100.000 | 99.908 | 99.599 | 98.953 |
78 | 100.000 | 100.000 | 99.908 | 99.603 | 98.969 |
79 | 100.000 | 100.000 | 99.908 | 99.608 | 98.988 |
80 | 100.000 | 100.000 | 99.908 | 99.614 | 99.008 |
81 | 100.000 | 100.000 | 99.908 | 99.617 | 99.021 |
82 | 100.000 | 100.000 | 99.908 | 99.621 | 99.037 |
83 | 100.000 | 100.000 | 99.908 | 99.625 | 99.052 |
84 | 100.000 | 100.000 | 99.908 | 99.629 | 99.068 |
85 | 100.000 | 100.000 | 99.908 | 99.633 | 99.084 |
86 | 100.000 | 100.000 | 99.908 | 99.637 | 99.097 |
87 | 100.000 | 100.000 | 99.908 | 99.640 | 99.109 |
88 | 100.000 | 100.000 | 99.908 | 99.644 | 99.124 |
89 | 100.000 | 100.000 | 99.908 | 99.647 | 99.137 |
90 | 100.000 | 100.000 | 99.908 | 99.651 | 99.152 |
91 | 100.000 | 100.000 | 99.908 | 99.653 | 99.161 |
92 | 100.000 | 100.000 | 99.917 | 99.680 | 99.212 |
93 | 100.000 | 100.000 | 99.917 | 99.683 | 99.224 |
94 | 100.000 | 100.000 | 99.917 | 99.687 | 99.236 |
95 | 100.000 | 100.000 | 99.917 | 99.689 | 99.245 |
96 | 100.000 | 100.000 | 99.917 | 99.691 | 99.254 |
97 | 100.000 | 100.000 | 99.917 | 99.695 | 99.269 |
98 | 100.000 | 100.000 | 99.917 | 99.697 | 99.278 |
99 | 100.000 | 100.000 | 99.917 | 99.700 | 99.290 |
100 | 100.000 | 100.000 | 99.926 | 99.727 | 99.341 |
101 | 100.000 | 100.000 | 99.926 | 99.729 | 99.349 |
102 | 100.000 | 100.000 | 99.935 | 99.756 | 99.400 |
103 | 100.000 | 100.000 | 99.945 | 99.783 | 99.450 |
104 | 100.000 | 100.000 | 99.945 | 99.785 | 99.459 |
105 | 100.000 | 100.000 | 99.954 | 99.813 | 99.510 |
106 | 100.000 | 100.000 | 99.963 | 99.840 | 99.560 |
107 | 100.000 | 100.000 | 99.972 | 99.867 | 99.611 |
108 | 100.000 | 100.000 | 99.982 | 99.894 | 99.661 |
109 | 100.000 | 100.000 | 99.991 | 99.921 | 99.712 |
110 | 100.000 | 100.000 | 100.000 | 99.948 | 99.763 |
111 | 100.000 | 100.000 | 100.000 | 99.950 | 99.771 |
112 | 100.000 | 100.000 | 100.000 | 99.952 | 99.779 |
113 | 100.000 | 100.000 | 100.000 | 99.954 | 99.788 |
114 | 100.000 | 100.000 | 100.000 | 99.956 | 99.796 |
115 | 100.000 | 100.000 | 100.000 | 99.958 | 99.804 |
116 | 100.000 | 100.000 | 100.000 | 99.959 | 99.809 |
117 | 100.000 | 100.000 | 100.000 | 99.963 | 99.821 |
118 | 100.000 | 100.000 | 100.000 | 99.965 | 99.829 |
119 | 100.000 | 100.000 | 100.000 | 99.967 | 99.837 |
120 | 100.000 | 100.000 | 100.000 | 99.968 | 99.842 |
121 | 100.000 | 100.000 | 100.000 | 99.970 | 99.850 |
122 | 100.000 | 100.000 | 100.000 | 99.972 | 99.858 |
123 | 100.000 | 100.000 | 100.000 | 99.974 | 99.866 |
124 | 100.000 | 100.000 | 100.000 | 99.976 | 99.874 |
125 | 100.000 | 100.000 | 100.000 | 99.978 | 99.882 |
126 | 100.000 | 100.000 | 100.000 | 99.980 | 99.890 |
127 | 100.000 | 100.000 | 100.000 | 99.982 | 99.898 |
128 | 100.000 | 100.000 | 100.000 | 99.984 | 99.906 |
129 | 100.000 | 100.000 | 100.000 | 99.986 | 99.914 |
130 | 100.000 | 100.000 | 100.000 | 99.989 | 99.922 |
131 | 100.000 | 100.000 | 100.000 | 99.990 | 99.926 |
132 | 100.000 | 100.000 | 100.000 | 99.992 | 99.934 |
133 | 100.000 | 100.000 | 100.000 | 99.994 | 99.942 |
134 | 100.000 | 100.000 | 100.000 | 99.996 | 99.950 |
135 | 100.000 | 100.000 | 100.000 | 99.997 | 99.954 |
136 | 100.000 | 100.000 | 100.000 | 99.999 | 99.962 |
137 | 100.000 | 100.000 | 100.000 | 99.999 | 99.963 |
138 | 100.000 | 100.000 | 100.000 | 99.999 | 99.965 |
139 | 100.000 | 100.000 | 100.000 | 99.999 | 99.966 |
140 | 100.000 | 100.000 | 100.000 | 99.999 | 99.967 |
141 | 100.000 | 100.000 | 100.000 | 99.999 | 99.968 |
142 | 100.000 | 100.000 | 100.000 | 99.999 | 99.969 |
143 | 100.000 | 100.000 | 100.000 | 99.999 | 99.970 |
144 | 100.000 | 100.000 | 100.000 | 99.999 | 99.972 |
145 | 100.000 | 100.000 | 100.000 | 99.999 | 99.973 |
146 | 100.000 | 100.000 | 100.000 | 99.999 | 99.974 |
147 | 100.000 | 100.000 | 100.000 | 99.999 | 99.974 |
148 | 100.000 | 100.000 | 100.000 | 99.999 | 99.975 |
149 | 100.000 | 100.000 | 100.000 | 99.999 | 99.976 |
150 | 100.000 | 100.000 | 100.000 | 99.999 | 99.977 |
151 | 100.000 | 100.000 | 100.000 | 99.999 | 99.978 |
152 | 100.000 | 100.000 | 100.000 | 99.999 | 99.979 |
153 | 100.000 | 100.000 | 100.000 | 99.999 | 99.980 |
154 | 100.000 | 100.000 | 100.000 | 99.999 | 99.981 |
155 | 100.000 | 100.000 | 100.000 | 99.999 | 99.982 |
156 | 100.000 | 100.000 | 100.000 | 99.999 | 99.983 |
157 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
158 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
159 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
160 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
161 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
162 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
163 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
164 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
165 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
166 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
167 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
168 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
169 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
170 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
171 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
172 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
173 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
174 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
175 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
176 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
178 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
179 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


grep
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 64.286 | 34.687 | 19.287 | 11.372 | 7.125 |
4 | 71.429 | 41.794 | 24.402 | 14.787 | 9.398 |
5 | 80.952 | 48.900 | 29.141 | 17.945 | 11.545 |
6 | 85.714 | 53.807 | 32.895 | 20.632 | 13.457 |
7 | 90.476 | 58.545 | 36.696 | 23.463 | 15.504 |
8 | 95.238 | 63.283 | 40.497 | 26.293 | 17.551 |
9 | 97.619 | 67.174 | 44.064 | 29.071 | 19.589 |
10 | 100.000 | 70.558 | 47.208 | 31.602 | 21.506 |
11 | 100.000 | 72.927 | 50.023 | 34.046 | 23.407 |
12 | 100.000 | 75.296 | 52.816 | 36.481 | 25.314 |
13 | 100.000 | 77.327 | 55.256 | 38.663 | 27.073 |
14 | 100.000 | 79.188 | 57.602 | 40.802 | 28.803 |
15 | 100.000 | 80.880 | 59.784 | 42.840 | 30.490 |
16 | 100.000 | 82.572 | 61.872 | 44.759 | 32.080 |
17 | 100.000 | 83.926 | 63.749 | 46.603 | 33.661 |
18 | 100.000 | 85.110 | 65.415 | 48.287 | 35.148 |
19 | 100.000 | 86.125 | 66.753 | 49.601 | 36.311 |
20 | 100.000 | 87.140 | 68.207 | 51.102 | 37.667 |
21 | 100.000 | 87.986 | 69.521 | 52.517 | 38.978 |
22 | 100.000 | 88.832 | 70.695 | 53.718 | 40.076 |
23 | 100.000 | 89.679 | 72.032 | 55.171 | 41.426 |
24 | 100.000 | 90.525 | 73.393 | 56.662 | 42.811 |
25 | 100.000 | 91.371 | 74.801 | 58.227 | 44.272 |
26 | 100.000 | 92.047 | 75.692 | 59.107 | 45.058 |
27 | 100.000 | 92.724 | 76.771 | 60.308 | 46.210 |
28 | 100.000 | 93.401 | 78.038 | 61.782 | 47.617 |
29 | 100.000 | 94.078 | 79.024 | 62.833 | 48.612 |
30 | 100.000 | 94.755 | 80.009 | 63.884 | 49.607 |
31 | 100.000 | 95.262 | 80.784 | 64.753 | 50.471 |
32 | 100.000 | 95.770 | 81.628 | 65.723 | 51.429 |
33 | 100.000 | 96.277 | 82.543 | 66.822 | 52.536 |
34 | 100.000 | 96.785 | 83.388 | 67.803 | 53.514 |
35 | 100.000 | 97.293 | 84.233 | 68.785 | 54.496 |
36 | 100.000 | 97.800 | 85.031 | 69.680 | 55.375 |
37 | 100.000 | 98.308 | 85.734 | 70.409 | 56.057 |
38 | 100.000 | 98.646 | 86.415 | 71.272 | 56.963 |
39 | 100.000 | 98.985 | 87.025 | 72.034 | 57.773 |
40 | 100.000 | 99.323 | 87.541 | 72.613 | 58.352 |
41 | 100.000 | 99.662 | 88.104 | 73.294 | 59.073 |
42 | 100.000 | 99.831 | 88.433 | 73.717 | 59.535 |
43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |


grep
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
3 | 64.286 | 34.518 | 19.193 | 11.340 | 7.118 |
4 | 71.429 | 41.455 | 24.214 | 14.723 | 9.383 |
5 | 76.190 | 46.870 | 28.578 | 17.881 | 11.587 |
6 | 83.333 | 52.961 | 33.060 | 21.039 | 13.782 |
7 | 85.714 | 57.191 | 36.931 | 24.009 | 15.922 |
8 | 88.095 | 61.421 | 40.779 | 26.958 | 18.050 |
9 | 90.476 | 65.313 | 44.392 | 29.795 | 20.134 |
10 | 92.857 | 68.697 | 47.489 | 32.277 | 22.022 |
11 | 95.238 | 71.912 | 50.493 | 34.738 | 23.920 |
12 | 97.619 | 74.958 | 53.332 | 37.086 | 25.752 |
13 | 97.619 | 76.819 | 55.655 | 39.215 | 27.492 |
14 | 97.619 | 78.511 | 57.837 | 41.263 | 29.195 |
15 | 97.619 | 80.034 | 59.808 | 43.129 | 30.767 |
16 | 97.619 | 81.049 | 61.427 | 44.850 | 32.313 |
17 | 97.619 | 82.064 | 62.928 | 46.432 | 33.757 |
18 | 97.619 | 83.249 | 64.524 | 48.008 | 35.142 |
19 | 97.619 | 84.264 | 66.025 | 49.579 | 36.565 |
20 | 97.619 | 85.279 | 67.480 | 51.091 | 37.940 |
21 | 97.619 | 86.294 | 68.888 | 52.523 | 39.233 |
22 | 97.619 | 87.310 | 70.296 | 53.954 | 40.526 |
23 | 97.619 | 88.325 | 71.680 | 55.338 | 41.755 |
24 | 97.619 | 88.832 | 72.712 | 56.603 | 43.013 |
25 | 97.619 | 89.509 | 73.792 | 57.815 | 44.185 |
26 | 97.619 | 90.186 | 74.871 | 59.016 | 45.337 |
27 | 100.000 | 92.217 | 76.420 | 60.195 | 46.253 |
28 | 100.000 | 92.893 | 77.452 | 61.326 | 47.338 |
29 | 100.000 | 93.570 | 78.461 | 62.410 | 48.361 |
30 | 100.000 | 94.247 | 79.446 | 63.460 | 49.356 |
31 | 100.000 | 94.924 | 80.432 | 64.511 | 50.350 |
32 | 100.000 | 95.431 | 81.300 | 65.530 | 51.364 |
33 | 100.000 | 95.770 | 82.074 | 66.527 | 52.390 |
34 | 100.000 | 96.108 | 82.802 | 67.460 | 53.363 |
35 | 100.000 | 96.785 | 83.693 | 68.339 | 54.149 |
36 | 100.000 | 97.293 | 84.491 | 69.251 | 55.064 |
37 | 100.000 | 97.800 | 85.265 | 70.120 | 55.928 |
38 | 100.000 | 98.308 | 86.016 | 70.945 | 56.739 |
39 | 100.000 | 98.646 | 86.649 | 71.733 | 57.566 |
40 | 100.000 | 99.154 | 87.353 | 72.463 | 58.248 |
41 | 100.000 | 99.492 | 87.916 | 73.138 | 58.957 |
42 | 100.000 | 99.831 | 88.433 | 73.717 | 59.535 |
43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |


grep
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
3 | 61.905 | 33.503 | 18.841 | 11.233 | 7.090 |
4 | 71.429 | 41.455 | 24.261 | 14.771 | 9.413 |
5 | 80.952 | 49.239 | 29.517 | 18.208 | 11.687 |
6 | 85.714 | 54.653 | 33.881 | 21.366 | 13.891 |
7 | 88.095 | 59.391 | 38.104 | 24.503 | 16.094 |
8 | 90.476 | 63.621 | 41.952 | 27.452 | 18.222 |
9 | 90.476 | 65.990 | 44.721 | 29.864 | 20.118 |
10 | 92.857 | 68.866 | 47.583 | 32.309 | 22.030 |
11 | 95.238 | 71.912 | 50.446 | 34.684 | 23.881 |
12 | 100.000 | 76.142 | 53.778 | 37.221 | 25.778 |
13 | 100.000 | 77.834 | 55.913 | 39.210 | 27.435 |
14 | 100.000 | 79.357 | 57.907 | 41.113 | 29.044 |
15 | 100.000 | 80.372 | 59.573 | 42.866 | 30.592 |
16 | 100.000 | 81.387 | 61.098 | 44.480 | 32.062 |
17 | 100.000 | 82.741 | 62.905 | 46.239 | 33.578 |
18 | 100.000 | 83.926 | 64.547 | 47.896 | 35.046 |
19 | 100.000 | 84.941 | 66.002 | 49.408 | 36.422 |
20 | 100.000 | 85.787 | 67.316 | 50.839 | 37.761 |
21 | 100.000 | 86.802 | 68.771 | 52.340 | 39.116 |
22 | 100.000 | 87.986 | 70.366 | 53.917 | 40.502 |
23 | 100.000 | 88.832 | 71.633 | 55.289 | 41.794 |
24 | 100.000 | 89.848 | 73.041 | 56.715 | 43.079 |
25 | 100.000 | 90.863 | 74.449 | 58.142 | 44.365 |
26 | 100.000 | 91.709 | 75.669 | 59.423 | 45.554 |
27 | 100.000 | 92.724 | 77.030 | 60.774 | 46.756 |
28 | 100.000 | 93.401 | 78.109 | 61.975 | 47.908 |
29 | 100.000 | 94.078 | 79.141 | 63.107 | 48.993 |
30 | 100.000 | 94.416 | 79.939 | 64.131 | 50.038 |
31 | 100.000 | 95.093 | 80.924 | 65.181 | 51.033 |
32 | 100.000 | 95.601 | 81.746 | 66.120 | 51.963 |
33 | 100.000 | 95.939 | 82.426 | 66.983 | 52.869 |
34 | 100.000 | 96.447 | 83.247 | 67.921 | 53.799 |
35 | 100.000 | 96.954 | 84.045 | 68.822 | 54.691 |
36 | 100.000 | 97.462 | 84.819 | 69.691 | 55.554 |
37 | 100.000 | 98.139 | 85.711 | 70.570 | 56.340 |
38 | 100.000 | 98.477 | 86.274 | 71.251 | 57.061 |
39 | 100.000 | 98.816 | 86.861 | 71.948 | 57.773 |
40 | 100.000 | 99.323 | 87.565 | 72.677 | 58.456 |
41 | 100.000 | 99.662 | 88.081 | 73.256 | 59.034 |
42 | 100.000 | 99.831 | 88.409 | 73.680 | 59.497 |
43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |


grep
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 59.524 | 32.318 | 18.348 | 11.045 | 7.025 |
4 | 66.667 | 39.594 | 23.627 | 14.562 | 9.346 |
5 | 73.810 | 46.701 | 28.789 | 18.026 | 11.648 |
6 | 78.571 | 51.777 | 32.895 | 21.050 | 13.797 |
7 | 80.952 | 54.992 | 36.039 | 23.666 | 15.800 |
8 | 83.333 | 58.376 | 39.207 | 26.235 | 17.752 |
9 | 88.095 | 62.944 | 42.891 | 29.012 | 19.781 |
10 | 88.095 | 65.144 | 45.519 | 31.333 | 21.623 |
11 | 88.095 | 67.343 | 48.170 | 33.676 | 23.476 |
12 | 90.476 | 70.051 | 50.634 | 35.757 | 25.159 |
13 | 92.857 | 73.096 | 53.426 | 38.046 | 26.945 |
14 | 92.857 | 74.450 | 55.256 | 39.842 | 28.500 |
15 | 97.619 | 78.680 | 58.517 | 42.282 | 30.312 |
16 | 97.619 | 80.034 | 60.347 | 44.078 | 31.864 |
17 | 100.000 | 82.910 | 62.905 | 46.164 | 33.510 |
18 | 100.000 | 84.095 | 64.547 | 47.810 | 34.961 |
19 | 100.000 | 84.941 | 65.931 | 49.322 | 36.357 |
20 | 100.000 | 86.125 | 67.597 | 50.995 | 37.828 |
21 | 100.000 | 86.971 | 68.935 | 52.448 | 39.177 |
22 | 100.000 | 87.817 | 70.202 | 53.809 | 40.453 |
23 | 100.000 | 88.832 | 71.656 | 55.311 | 41.809 |
24 | 100.000 | 90.017 | 73.229 | 56.855 | 43.166 |
25 | 100.000 | 90.863 | 74.519 | 58.238 | 44.450 |
26 | 100.000 | 91.878 | 75.880 | 59.611 | 45.696 |
27 | 100.000 | 92.555 | 76.983 | 60.844 | 46.876 |
28 | 100.000 | 93.401 | 78.179 | 62.088 | 48.026 |
29 | 100.000 | 94.078 | 79.165 | 63.155 | 49.056 |
30 | 100.000 | 94.416 | 79.962 | 64.179 | 50.101 |
31 | 100.000 | 94.755 | 80.666 | 65.069 | 51.022 |
32 | 100.000 | 95.262 | 81.488 | 66.007 | 51.952 |
33 | 100.000 | 95.770 | 82.309 | 66.945 | 52.882 |
34 | 100.000 | 96.447 | 83.271 | 67.959 | 53.838 |
35 | 100.000 | 97.124 | 84.233 | 68.972 | 54.794 |
36 | 100.000 | 97.631 | 85.007 | 69.841 | 55.658 |
37 | 100.000 | 97.970 | 85.570 | 70.522 | 56.379 |
38 | 100.000 | 98.308 | 86.157 | 71.219 | 57.091 |
39 | 100.000 | 98.816 | 86.861 | 71.948 | 57.773 |
40 | 100.000 | 99.323 | 87.565 | 72.677 | 58.456 |
41 | 100.000 | 99.662 | 88.081 | 73.256 | 59.034 |
42 | 100.000 | 99.831 | 88.409 | 73.680 | 59.497 |
43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |


grep
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 66.667 | 35.533 | 19.521 | 11.426 | 7.134 |
4 | 76.190 | 43.486 | 24.941 | 14.964 | 9.457 |
5 | 85.714 | 51.100 | 30.103 | 18.369 | 11.724 |
6 | 90.476 | 57.191 | 34.937 | 21.747 | 14.010 |
7 | 92.857 | 61.929 | 39.160 | 24.883 | 16.212 |
8 | 95.238 | 66.328 | 43.149 | 27.913 | 18.377 |
9 | 97.619 | 70.051 | 46.574 | 30.615 | 20.386 |
10 | 100.000 | 73.435 | 49.789 | 33.242 | 22.385 |
11 | 100.000 | 75.296 | 52.112 | 35.376 | 24.134 |
12 | 100.000 | 77.157 | 54.575 | 37.671 | 25.996 |
13 | 100.000 | 79.019 | 56.875 | 39.778 | 27.727 |
14 | 100.000 | 80.541 | 59.033 | 41.880 | 29.488 |
15 | 100.000 | 81.895 | 61.004 | 43.842 | 31.165 |
16 | 100.000 | 83.249 | 62.764 | 45.536 | 32.626 |
17 | 100.000 | 84.433 | 64.406 | 47.182 | 34.077 |
18 | 100.000 | 85.448 | 65.814 | 48.619 | 35.379 |
19 | 100.000 | 86.464 | 67.222 | 50.051 | 36.672 |
20 | 100.000 | 87.479 | 68.583 | 51.402 | 37.874 |
21 | 100.000 | 88.494 | 70.014 | 52.866 | 39.194 |
22 | 100.000 | 89.340 | 71.258 | 54.179 | 40.411 |
23 | 100.000 | 90.186 | 72.642 | 55.713 | 41.844 |
24 | 100.000 | 91.032 | 74.050 | 57.257 | 43.268 |
25 | 100.000 | 91.709 | 75.364 | 58.774 | 44.692 |
26 | 100.000 | 92.386 | 76.396 | 59.895 | 45.753 |
27 | 100.000 | 93.063 | 77.288 | 60.774 | 46.539 |
28 | 100.000 | 93.570 | 78.179 | 61.847 | 47.628 |
29 | 100.000 | 94.078 | 78.954 | 62.715 | 48.492 |
30 | 100.000 | 94.585 | 79.798 | 63.707 | 49.494 |
31 | 100.000 | 95.093 | 80.526 | 64.474 | 50.215 |
32 | 100.000 | 95.601 | 81.253 | 65.240 | 50.937 |
33 | 100.000 | 96.108 | 82.074 | 66.179 | 51.867 |
34 | 100.000 | 96.447 | 82.708 | 66.967 | 52.693 |
35 | 100.000 | 96.785 | 83.388 | 67.825 | 53.587 |
36 | 100.000 | 97.124 | 83.881 | 68.361 | 54.114 |
37 | 100.000 | 97.462 | 84.420 | 69.004 | 54.796 |
38 | 100.000 | 97.800 | 85.031 | 69.771 | 55.615 |
39 | 100.000 | 97.970 | 85.617 | 70.629 | 56.555 |
40 | 100.000 | 98.139 | 86.133 | 71.374 | 57.383 |
41 | 100.000 | 98.308 | 86.509 | 71.894 | 57.977 |
42 | 100.000 | 98.477 | 86.908 | 72.452 | 58.609 |
43 | 100.000 | 98.646 | 87.353 | 73.101 | 59.352 |
44 | 100.000 | 98.816 | 87.682 | 73.540 | 59.848 |
45 | 100.000 | 98.985 | 88.034 | 74.007 | 60.362 |
46 | 100.000 | 99.154 | 88.409 | 74.532 | 60.967 |
47 | 100.000 | 99.323 | 88.761 | 75.031 | 61.553 |
48 | 100.000 | 99.492 | 89.254 | 75.739 | 62.343 |
49 | 100.000 | 99.662 | 89.653 | 76.296 | 62.975 |
50 | 100.000 | 99.831 | 89.911 | 76.591 | 63.280 |
51 | 100.000 | 100.000 | 90.216 | 76.977 | 63.704 |
52 | 100.000 | 100.000 | 90.357 | 77.261 | 64.090 |
53 | 100.000 | 100.000 | 90.802 | 78.055 | 65.031 |
54 | 100.000 | 100.000 | 90.990 | 78.441 | 65.553 |
55 | 100.000 | 100.000 | 91.061 | 78.586 | 65.754 |
56 | 100.000 | 100.000 | 91.107 | 78.687 | 65.904 |
57 | 100.000 | 100.000 | 91.248 | 78.961 | 66.263 |
58 | 100.000 | 100.000 | 91.436 | 79.336 | 66.764 |
59 | 100.000 | 100.000 | 91.741 | 79.888 | 67.437 |
60 | 100.000 | 100.000 | 91.929 | 80.264 | 67.938 |
61 | 100.000 | 100.000 | 92.046 | 80.505 | 68.269 |
62 | 100.000 | 100.000 | 92.281 | 80.961 | 68.861 |
63 | 100.000 | 100.000 | 92.374 | 81.165 | 69.153 |
64 | 100.000 | 100.000 | 92.562 | 81.545 | 69.667 |
65 | 100.000 | 100.000 | 92.679 | 81.781 | 69.987 |
66 | 100.000 | 100.000 | 92.726 | 81.883 | 70.137 |
67 | 100.000 | 100.000 | 92.844 | 82.130 | 70.485 |
68 | 100.000 | 100.000 | 92.867 | 82.189 | 70.583 |
69 | 100.000 | 100.000 | 92.914 | 82.290 | 70.732 |
70 | 100.000 | 100.000 | 92.985 | 82.441 | 70.949 |
71 | 100.000 | 100.000 | 93.102 | 82.682 | 71.284 |
72 | 100.000 | 100.000 | 93.172 | 82.843 | 71.524 |
73 | 100.000 | 100.000 | 93.290 | 83.084 | 71.855 |
74 | 100.000 | 100.000 | 93.407 | 83.320 | 72.175 |
75 | 100.000 | 100.000 | 93.454 | 83.427 | 72.339 |
76 | 100.000 | 100.000 | 93.595 | 83.722 | 72.750 |
77 | 100.000 | 100.000 | 93.688 | 83.915 | 73.018 |
78 | 100.000 | 100.000 | 93.759 | 84.065 | 73.234 |
79 | 100.000 | 100.000 | 93.782 | 84.119 | 73.318 |
80 | 100.000 | 100.000 | 94.017 | 84.521 | 73.789 |
81 | 100.000 | 100.000 | 94.111 | 84.714 | 74.057 |
82 | 100.000 | 100.000 | 94.392 | 85.245 | 74.727 |
83 | 100.000 | 100.000 | 94.674 | 85.781 | 75.398 |
84 | 100.000 | 100.000 | 94.721 | 85.904 | 75.601 |
85 | 100.000 | 100.000 | 94.791 | 86.076 | 75.873 |
86 | 100.000 | 100.000 | 94.979 | 86.462 | 76.395 |
87 | 100.000 | 100.000 | 95.190 | 86.880 | 76.940 |
88 | 100.000 | 100.000 | 95.237 | 87.014 | 77.171 |
89 | 100.000 | 100.000 | 95.331 | 87.229 | 77.491 |
90 | 100.000 | 100.000 | 95.448 | 87.464 | 77.813 |
91 | 100.000 | 100.000 | 95.565 | 87.711 | 78.161 |
92 | 100.000 | 100.000 | 95.636 | 87.856 | 78.362 |
93 | 100.000 | 100.000 | 95.753 | 88.113 | 78.734 |
94 | 100.000 | 100.000 | 95.847 | 88.306 | 79.002 |
95 | 100.000 | 100.000 | 95.870 | 88.381 | 79.141 |
96 | 100.000 | 100.000 | 95.964 | 88.590 | 79.448 |
97 | 100.000 | 100.000 | 95.988 | 88.671 | 79.599 |
98 | 100.000 | 100.000 | 96.058 | 88.832 | 79.844 |
99 | 100.000 | 100.000 | 96.293 | 89.277 | 80.411 |
100 | 100.000 | 100.000 | 96.387 | 89.491 | 80.724 |
101 | 100.000 | 100.000 | 96.457 | 89.647 | 80.953 |
102 | 100.000 | 100.000 | 96.621 | 89.990 | 81.423 |
103 | 100.000 | 100.000 | 96.809 | 90.370 | 81.937 |
104 | 100.000 | 100.000 | 96.832 | 90.424 | 82.020 |
105 | 100.000 | 100.000 | 96.973 | 90.698 | 82.379 |
106 | 100.000 | 100.000 | 97.255 | 91.212 | 83.013 |
107 | 100.000 | 100.000 | 97.349 | 91.405 | 83.281 |
108 | 100.000 | 100.000 | 97.396 | 91.507 | 83.431 |
109 | 100.000 | 100.000 | 97.489 | 91.722 | 83.751 |
110 | 100.000 | 100.000 | 97.630 | 92.022 | 84.174 |
111 | 100.000 | 100.000 | 97.724 | 92.220 | 84.455 |
112 | 100.000 | 100.000 | 97.748 | 92.285 | 84.566 |
113 | 100.000 | 100.000 | 97.794 | 92.386 | 84.716 |
114 | 100.000 | 100.000 | 97.888 | 92.590 | 85.012 |
115 | 100.000 | 100.000 | 97.912 | 92.649 | 85.112 |
116 | 100.000 | 100.000 | 97.959 | 92.762 | 85.289 |
117 | 100.000 | 100.000 | 98.053 | 92.955 | 85.558 |
118 | 100.000 | 100.000 | 98.076 | 93.025 | 85.682 |
119 | 100.000 | 100.000 | 98.170 | 93.228 | 85.977 |
120 | 100.000 | 100.000 | 98.264 | 93.421 | 86.246 |
121 | 100.000 | 100.000 | 98.358 | 93.625 | 86.534 |
122 | 100.000 | 100.000 | 98.381 | 93.738 | 86.761 |
123 | 100.000 | 100.000 | 98.405 | 93.797 | 86.859 |
124 | 100.000 | 100.000 | 98.451 | 93.898 | 87.009 |
125 | 100.000 | 100.000 | 98.522 | 94.054 | 87.239 |
126 | 100.000 | 100.000 | 98.710 | 94.429 | 87.740 |
127 | 100.000 | 100.000 | 98.756 | 94.531 | 87.889 |
128 | 100.000 | 100.000 | 98.850 | 94.724 | 88.158 |
129 | 100.000 | 100.000 | 98.921 | 94.874 | 88.374 |
130 | 100.000 | 100.000 | 98.968 | 94.976 | 88.524 |
131 | 100.000 | 100.000 | 99.085 | 95.233 | 88.895 |
132 | 100.000 | 100.000 | 99.108 | 95.303 | 89.019 |
133 | 100.000 | 100.000 | 99.179 | 95.459 | 89.249 |
134 | 100.000 | 100.000 | 99.202 | 95.512 | 89.332 |
135 | 100.000 | 100.000 | 99.249 | 95.646 | 89.563 |
136 | 100.000 | 100.000 | 99.273 | 95.711 | 89.674 |
137 | 100.000 | 100.000 | 99.320 | 95.823 | 89.852 |
138 | 100.000 | 100.000 | 99.413 | 96.016 | 90.120 |
139 | 100.000 | 100.000 | 99.460 | 96.129 | 90.297 |
140 | 100.000 | 100.000 | 99.484 | 96.193 | 90.410 |
141 | 100.000 | 100.000 | 99.578 | 96.397 | 90.706 |
142 | 100.000 | 100.000 | 99.648 | 96.552 | 90.935 |
143 | 100.000 | 100.000 | 99.695 | 96.708 | 91.213 |
144 | 100.000 | 100.000 | 99.742 | 96.810 | 91.362 |
145 | 100.000 | 100.000 | 99.812 | 96.981 | 91.632 |
146 | 100.000 | 100.000 | 99.883 | 97.132 | 91.849 |
147 | 100.000 | 100.000 | 99.906 | 97.201 | 91.974 |
148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |


grep
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
4 | 73.810 | 42.640 | 24.707 | 14.911 | 9.448 |
5 | 80.952 | 49.915 | 29.986 | 18.428 | 11.768 |
6 | 83.333 | 54.146 | 33.904 | 21.447 | 13.938 |
7 | 85.714 | 57.868 | 37.424 | 24.251 | 16.014 |
8 | 92.857 | 63.621 | 41.600 | 27.216 | 18.108 |
9 | 95.238 | 67.343 | 45.002 | 29.891 | 20.097 |
10 | 95.238 | 69.543 | 47.701 | 32.293 | 21.996 |
11 | 100.000 | 73.942 | 51.103 | 34.813 | 23.846 |
12 | 100.000 | 76.142 | 53.754 | 37.156 | 25.699 |
13 | 100.000 | 78.173 | 56.288 | 39.456 | 27.550 |
14 | 100.000 | 80.203 | 58.799 | 41.735 | 29.390 |
15 | 100.000 | 81.557 | 60.770 | 43.697 | 31.067 |
16 | 100.000 | 82.910 | 62.694 | 45.611 | 32.714 |
17 | 100.000 | 84.095 | 64.313 | 47.236 | 34.155 |
18 | 100.000 | 85.448 | 66.002 | 48.812 | 35.486 |
19 | 100.000 | 86.464 | 67.480 | 50.362 | 36.899 |
20 | 100.000 | 87.479 | 68.935 | 51.863 | 38.255 |
21 | 100.000 | 88.325 | 70.272 | 53.316 | 39.605 |
22 | 100.000 | 89.171 | 71.563 | 54.699 | 40.888 |
23 | 100.000 | 90.017 | 72.853 | 56.104 | 42.208 |
24 | 100.000 | 90.863 | 74.097 | 57.439 | 43.462 |
25 | 100.000 | 91.709 | 75.340 | 58.758 | 44.688 |
26 | 100.000 | 92.386 | 76.466 | 60.008 | 45.862 |
27 | 100.000 | 93.063 | 77.546 | 61.209 | 47.014 |
28 | 100.000 | 93.739 | 78.531 | 62.286 | 48.065 |
29 | 100.000 | 94.247 | 79.423 | 63.326 | 49.089 |
30 | 100.000 | 94.247 | 80.033 | 64.350 | 50.234 |
31 | 100.000 | 94.755 | 80.924 | 65.412 | 51.303 |
32 | 100.000 | 95.262 | 81.793 | 66.420 | 52.299 |
33 | 100.000 | 95.431 | 82.450 | 67.374 | 53.331 |
34 | 100.000 | 95.939 | 83.294 | 68.361 | 54.324 |
35 | 100.000 | 96.447 | 84.092 | 69.283 | 55.260 |
36 | 100.000 | 96.954 | 84.866 | 70.146 | 56.113 |
37 | 100.000 | 97.293 | 85.547 | 71.004 | 57.006 |
38 | 100.000 | 97.462 | 86.110 | 71.830 | 57.921 |
39 | 100.000 | 97.800 | 86.720 | 72.581 | 58.709 |
40 | 100.000 | 97.800 | 87.142 | 73.315 | 59.567 |
41 | 100.000 | 98.139 | 87.752 | 74.066 | 60.355 |
42 | 100.000 | 98.139 | 88.104 | 74.704 | 61.115 |
43 | 100.000 | 98.139 | 88.456 | 75.331 | 61.867 |
44 | 100.000 | 98.308 | 88.878 | 75.932 | 62.550 |
45 | 100.000 | 98.646 | 89.371 | 76.500 | 63.153 |
46 | 100.000 | 98.816 | 89.770 | 77.063 | 63.796 |
47 | 100.000 | 98.985 | 90.145 | 77.583 | 64.390 |
48 | 100.000 | 99.154 | 90.521 | 78.103 | 64.983 |
49 | 100.000 | 99.323 | 90.896 | 78.623 | 65.577 |
50 | 100.000 | 99.323 | 91.154 | 79.116 | 66.207 |
51 | 100.000 | 99.492 | 91.506 | 79.615 | 66.794 |
52 | 100.000 | 99.492 | 91.741 | 80.065 | 67.372 |
53 | 100.000 | 99.492 | 91.976 | 80.500 | 67.916 |
54 | 100.000 | 99.662 | 92.304 | 80.950 | 68.439 |
55 | 100.000 | 99.831 | 92.633 | 81.374 | 68.902 |
56 | 100.000 | 99.831 | 92.867 | 81.835 | 69.505 |
57 | 100.000 | 99.831 | 93.078 | 82.253 | 70.050 |
58 | 100.000 | 99.831 | 93.266 | 82.628 | 70.551 |
59 | 100.000 | 99.831 | 93.454 | 82.998 | 71.041 |
60 | 100.000 | 99.831 | 93.641 | 83.395 | 71.591 |
61 | 100.000 | 99.831 | 93.829 | 83.770 | 72.092 |
62 | 100.000 | 99.831 | 94.017 | 84.151 | 72.606 |
63 | 100.000 | 100.000 | 94.275 | 84.446 | 72.911 |
64 | 100.000 | 100.000 | 94.439 | 84.784 | 73.373 |
65 | 100.000 | 100.000 | 94.603 | 85.100 | 73.784 |
66 | 100.000 | 100.000 | 94.744 | 85.384 | 74.170 |
67 | 100.000 | 100.000 | 94.885 | 85.700 | 74.625 |
68 | 100.000 | 100.000 | 95.026 | 85.974 | 74.984 |
69 | 100.000 | 100.000 | 95.167 | 86.253 | 75.356 |
70 | 100.000 | 100.000 | 95.307 | 86.553 | 75.777 |
71 | 100.000 | 100.000 | 95.448 | 86.837 | 76.156 |
72 | 100.000 | 100.000 | 95.589 | 87.121 | 76.535 |
73 | 100.000 | 100.000 | 95.730 | 87.400 | 76.907 |
74 | 100.000 | 100.000 | 95.870 | 87.695 | 77.318 |
75 | 100.000 | 100.000 | 96.011 | 88.001 | 77.752 |
76 | 100.000 | 100.000 | 96.129 | 88.253 | 78.111 |
77 | 100.000 | 100.000 | 96.246 | 88.483 | 78.418 |
78 | 100.000 | 100.000 | 96.363 | 88.719 | 78.738 |
79 | 100.000 | 100.000 | 96.481 | 88.976 | 79.111 |
80 | 100.000 | 100.000 | 96.574 | 89.180 | 79.405 |
81 | 100.000 | 100.000 | 96.668 | 89.373 | 79.673 |
82 | 100.000 | 100.000 | 96.762 | 89.641 | 80.110 |
83 | 100.000 | 100.000 | 96.856 | 89.872 | 80.467 |
84 | 100.000 | 100.000 | 96.950 | 90.070 | 80.750 |
85 | 100.000 | 100.000 | 97.044 | 90.269 | 81.031 |
86 | 100.000 | 100.000 | 97.137 | 90.499 | 81.388 |
87 | 100.000 | 100.000 | 97.231 | 90.714 | 81.708 |
88 | 100.000 | 100.000 | 97.302 | 90.864 | 81.924 |
89 | 100.000 | 100.000 | 97.372 | 91.009 | 82.125 |
90 | 100.000 | 100.000 | 97.443 | 91.175 | 82.379 |
91 | 100.000 | 100.000 | 97.513 | 91.357 | 82.671 |
92 | 100.000 | 100.000 | 97.583 | 91.534 | 82.952 |
93 | 100.000 | 100.000 | 97.654 | 91.700 | 83.205 |
94 | 100.000 | 100.000 | 97.724 | 91.856 | 83.435 |
95 | 100.000 | 100.000 | 97.794 | 92.017 | 83.679 |
96 | 100.000 | 100.000 | 97.865 | 92.188 | 83.949 |
97 | 100.000 | 100.000 | 97.935 | 92.360 | 84.219 |
98 | 100.000 | 100.000 | 98.006 | 92.510 | 84.435 |
99 | 100.000 | 100.000 | 98.076 | 92.660 | 84.652 |
100 | 100.000 | 100.000 | 98.146 | 92.815 | 84.881 |
101 | 100.000 | 100.000 | 98.217 | 92.966 | 85.097 |
102 | 100.000 | 100.000 | 98.287 | 93.148 | 85.391 |
103 | 100.000 | 100.000 | 98.358 | 93.346 | 85.722 |
104 | 100.000 | 100.000 | 98.428 | 93.502 | 85.952 |
105 | 100.000 | 100.000 | 98.475 | 93.604 | 86.101 |
106 | 100.000 | 100.000 | 98.522 | 93.722 | 86.294 |
107 | 100.000 | 100.000 | 98.569 | 93.829 | 86.458 |
108 | 100.000 | 100.000 | 98.616 | 93.931 | 86.608 |
109 | 100.000 | 100.000 | 98.663 | 94.049 | 86.798 |
110 | 100.000 | 100.000 | 98.710 | 94.156 | 86.963 |
111 | 100.000 | 100.000 | 98.756 | 94.268 | 87.141 |
112 | 100.000 | 100.000 | 98.803 | 94.381 | 87.322 |
113 | 100.000 | 100.000 | 98.850 | 94.504 | 87.527 |
114 | 100.000 | 100.000 | 98.897 | 94.606 | 87.677 |
115 | 100.000 | 100.000 | 98.944 | 94.708 | 87.827 |
116 | 100.000 | 100.000 | 98.991 | 94.810 | 87.976 |
117 | 100.000 | 100.000 | 99.038 | 94.923 | 88.154 |
118 | 100.000 | 100.000 | 99.085 | 95.030 | 88.317 |
119 | 100.000 | 100.000 | 99.132 | 95.137 | 88.481 |
120 | 100.000 | 100.000 | 99.179 | 95.239 | 88.631 |
121 | 100.000 | 100.000 | 99.226 | 95.394 | 88.908 |
122 | 100.000 | 100.000 | 99.273 | 95.518 | 89.110 |
123 | 100.000 | 100.000 | 99.320 | 95.636 | 89.302 |
124 | 100.000 | 100.000 | 99.366 | 95.737 | 89.452 |
125 | 100.000 | 100.000 | 99.413 | 95.861 | 89.654 |
126 | 100.000 | 100.000 | 99.460 | 95.968 | 89.818 |
127 | 100.000 | 100.000 | 99.507 | 96.086 | 90.007 |
128 | 100.000 | 100.000 | 99.531 | 96.156 | 90.131 |
129 | 100.000 | 100.000 | 99.554 | 96.225 | 90.256 |
130 | 100.000 | 100.000 | 99.578 | 96.279 | 90.340 |
131 | 100.000 | 100.000 | 99.601 | 96.343 | 90.454 |
132 | 100.000 | 100.000 | 99.625 | 96.429 | 90.621 |
133 | 100.000 | 100.000 | 99.648 | 96.488 | 90.719 |
134 | 100.000 | 100.000 | 99.672 | 96.542 | 90.802 |
135 | 100.000 | 100.000 | 99.695 | 96.601 | 90.900 |
136 | 100.000 | 100.000 | 99.718 | 96.660 | 90.998 |
137 | 100.000 | 100.000 | 99.742 | 96.719 | 91.096 |
138 | 100.000 | 100.000 | 99.765 | 96.772 | 91.179 |
139 | 100.000 | 100.000 | 99.789 | 96.837 | 91.290 |
140 | 100.000 | 100.000 | 99.812 | 96.890 | 91.373 |
141 | 100.000 | 100.000 | 99.836 | 96.955 | 91.486 |
142 | 100.000 | 100.000 | 99.859 | 97.014 | 91.586 |
143 | 100.000 | 100.000 | 99.883 | 97.110 | 91.778 |
144 | 100.000 | 100.000 | 99.906 | 97.164 | 91.862 |
145 | 100.000 | 100.000 | 99.930 | 97.223 | 91.960 |
146 | 100.000 | 100.000 | 99.953 | 97.282 | 92.058 |
147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |


grep
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
3 | 57.143 | 31.641 | 18.254 | 11.072 | 7.053 |
4 | 66.667 | 39.425 | 23.557 | 14.557 | 9.357 |
5 | 76.190 | 47.208 | 28.860 | 18.042 | 11.661 |
6 | 80.952 | 52.623 | 33.224 | 21.200 | 13.865 |
7 | 85.714 | 57.530 | 37.166 | 24.122 | 15.966 |
8 | 88.095 | 61.083 | 40.544 | 26.851 | 18.013 |
9 | 90.476 | 64.129 | 43.548 | 29.382 | 19.970 |
10 | 90.476 | 66.328 | 46.293 | 31.832 | 21.898 |
11 | 95.238 | 71.066 | 50.047 | 34.615 | 23.916 |
12 | 97.619 | 74.281 | 53.050 | 37.076 | 25.813 |
13 | 97.619 | 76.142 | 55.397 | 39.237 | 27.581 |
14 | 97.619 | 77.834 | 57.626 | 41.344 | 29.330 |
15 | 100.000 | 80.711 | 60.300 | 43.590 | 31.115 |
16 | 100.000 | 82.403 | 62.412 | 45.542 | 32.733 |
17 | 100.000 | 83.249 | 63.820 | 47.113 | 34.201 |
18 | 100.000 | 84.772 | 65.767 | 48.952 | 35.755 |
19 | 100.000 | 85.956 | 67.433 | 50.630 | 37.234 |
20 | 100.000 | 87.310 | 69.193 | 52.319 | 38.684 |
21 | 100.000 | 88.494 | 70.765 | 53.868 | 40.050 |
22 | 100.000 | 89.171 | 71.938 | 55.209 | 41.336 |
23 | 100.000 | 90.017 | 73.205 | 56.565 | 42.600 |
24 | 100.000 | 90.863 | 74.449 | 57.884 | 43.828 |
25 | 100.000 | 91.540 | 75.575 | 59.134 | 45.002 |
26 | 100.000 | 92.047 | 76.513 | 60.265 | 46.138 |
27 | 100.000 | 92.893 | 77.710 | 61.498 | 47.264 |
28 | 100.000 | 93.232 | 78.461 | 62.474 | 48.287 |
29 | 100.000 | 93.570 | 79.235 | 63.471 | 49.309 |
30 | 100.000 | 94.078 | 80.103 | 64.490 | 50.323 |
31 | 100.000 | 94.416 | 80.807 | 65.412 | 51.310 |
32 | 100.000 | 94.755 | 81.511 | 66.334 | 52.283 |
33 | 100.000 | 95.093 | 82.168 | 67.192 | 53.213 |
34 | 100.000 | 95.262 | 82.708 | 67.986 | 54.102 |
35 | 100.000 | 95.431 | 83.200 | 68.709 | 54.927 |
36 | 100.000 | 95.939 | 83.951 | 69.551 | 55.772 |
37 | 100.000 | 96.447 | 84.702 | 70.388 | 56.608 |
38 | 100.000 | 96.447 | 85.031 | 70.994 | 57.353 |
39 | 100.000 | 96.616 | 85.500 | 71.664 | 58.104 |
40 | 100.000 | 96.785 | 85.969 | 72.334 | 58.853 |
41 | 100.000 | 97.124 | 86.579 | 73.079 | 59.628 |
42 | 100.000 | 97.293 | 87.025 | 73.728 | 60.371 |
43 | 100.000 | 97.462 | 87.471 | 74.366 | 61.094 |
44 | 100.000 | 97.800 | 88.034 | 75.058 | 61.840 |
45 | 100.000 | 97.800 | 88.315 | 75.594 | 62.511 |
46 | 100.000 | 97.970 | 88.738 | 76.200 | 63.206 |
47 | 100.000 | 98.139 | 89.137 | 76.779 | 63.879 |
48 | 100.000 | 98.646 | 89.864 | 77.545 | 64.600 |
49 | 100.000 | 98.816 | 90.263 | 78.114 | 65.257 |
50 | 100.000 | 98.985 | 90.662 | 78.671 | 65.889 |
51 | 100.000 | 99.154 | 91.061 | 79.229 | 66.522 |
52 | 100.000 | 99.154 | 91.295 | 79.685 | 67.114 |
53 | 100.000 | 99.154 | 91.530 | 80.140 | 67.705 |
54 | 100.000 | 99.154 | 91.741 | 80.559 | 68.251 |
55 | 100.000 | 99.154 | 91.952 | 80.972 | 68.791 |
56 | 100.000 | 99.154 | 92.140 | 81.363 | 69.327 |
57 | 100.000 | 99.154 | 92.281 | 81.701 | 69.830 |
58 | 100.000 | 99.154 | 92.468 | 82.076 | 70.331 |
59 | 100.000 | 99.154 | 92.656 | 82.451 | 70.832 |
60 | 100.000 | 99.154 | 92.820 | 82.800 | 71.319 |
61 | 100.000 | 99.154 | 93.031 | 83.191 | 71.807 |
62 | 100.000 | 99.323 | 93.360 | 83.631 | 72.302 |
63 | 100.000 | 99.492 | 93.688 | 84.071 | 72.805 |
64 | 100.000 | 99.492 | 93.853 | 84.414 | 73.279 |
65 | 100.000 | 99.492 | 94.017 | 84.751 | 73.741 |
66 | 100.000 | 99.492 | 94.181 | 85.089 | 74.203 |
67 | 100.000 | 99.662 | 94.486 | 85.497 | 74.671 |
68 | 100.000 | 99.662 | 94.627 | 85.808 | 75.119 |
69 | 100.000 | 99.662 | 94.791 | 86.135 | 75.557 |
70 | 100.000 | 99.662 | 94.908 | 86.408 | 75.964 |
71 | 100.000 | 99.662 | 95.073 | 86.725 | 76.374 |
72 | 100.000 | 99.662 | 95.214 | 87.019 | 76.785 |
73 | 100.000 | 99.662 | 95.354 | 87.314 | 77.197 |
74 | 100.000 | 99.662 | 95.495 | 87.604 | 77.595 |
75 | 100.000 | 99.831 | 95.800 | 87.990 | 78.018 |
76 | 100.000 | 100.000 | 96.105 | 88.376 | 78.442 |
77 | 100.000 | 100.000 | 96.246 | 88.660 | 78.828 |
78 | 100.000 | 100.000 | 96.340 | 88.896 | 79.193 |
79 | 100.000 | 100.000 | 96.457 | 89.148 | 79.553 |
80 | 100.000 | 100.000 | 96.551 | 89.368 | 79.888 |
81 | 100.000 | 100.000 | 96.668 | 89.609 | 80.221 |
82 | 100.000 | 100.000 | 96.786 | 89.850 | 80.556 |
83 | 100.000 | 100.000 | 96.856 | 90.038 | 80.861 |
84 | 100.000 | 100.000 | 96.950 | 90.247 | 81.169 |
85 | 100.000 | 100.000 | 97.044 | 90.451 | 81.458 |
86 | 100.000 | 100.000 | 97.137 | 90.655 | 81.750 |
87 | 100.000 | 100.000 | 97.231 | 90.858 | 82.048 |
88 | 100.000 | 100.000 | 97.325 | 91.057 | 82.329 |
89 | 100.000 | 100.000 | 97.419 | 91.250 | 82.597 |
90 | 100.000 | 100.000 | 97.489 | 91.421 | 82.865 |
91 | 100.000 | 100.000 | 97.560 | 91.593 | 83.133 |
92 | 100.000 | 100.000 | 97.630 | 91.759 | 83.390 |
93 | 100.000 | 100.000 | 97.701 | 91.925 | 83.647 |
94 | 100.000 | 100.000 | 97.771 | 92.086 | 83.890 |
95 | 100.000 | 100.000 | 97.841 | 92.247 | 84.130 |
96 | 100.000 | 100.000 | 97.912 | 92.403 | 84.359 |
97 | 100.000 | 100.000 | 97.982 | 92.558 | 84.592 |
98 | 100.000 | 100.000 | 98.053 | 92.714 | 84.822 |
99 | 100.000 | 100.000 | 98.099 | 92.842 | 85.040 |
100 | 100.000 | 100.000 | 98.170 | 92.992 | 85.258 |
101 | 100.000 | 100.000 | 98.240 | 93.142 | 85.474 |
102 | 100.000 | 100.000 | 98.311 | 93.293 | 85.691 |
103 | 100.000 | 100.000 | 98.381 | 93.443 | 85.907 |
104 | 100.000 | 100.000 | 98.451 | 93.587 | 86.109 |
105 | 100.000 | 100.000 | 98.498 | 93.711 | 86.310 |
106 | 100.000 | 100.000 | 98.545 | 93.834 | 86.516 |
107 | 100.000 | 100.000 | 98.592 | 93.957 | 86.717 |
108 | 100.000 | 100.000 | 98.639 | 94.075 | 86.906 |
109 | 100.000 | 100.000 | 98.663 | 94.172 | 87.098 |
110 | 100.000 | 100.000 | 98.710 | 94.284 | 87.276 |
111 | 100.000 | 100.000 | 98.756 | 94.397 | 87.455 |
112 | 100.000 | 100.000 | 98.803 | 94.510 | 87.632 |
113 | 100.000 | 100.000 | 98.850 | 94.622 | 87.810 |
114 | 100.000 | 100.000 | 98.897 | 94.735 | 87.989 |
115 | 100.000 | 100.000 | 98.944 | 94.847 | 88.167 |
116 | 100.000 | 100.000 | 98.991 | 94.960 | 88.344 |
117 | 100.000 | 100.000 | 99.015 | 95.046 | 88.513 |
118 | 100.000 | 100.000 | 99.061 | 95.153 | 88.679 |
119 | 100.000 | 100.000 | 99.108 | 95.260 | 88.844 |
120 | 100.000 | 100.000 | 99.155 | 95.362 | 88.993 |
121 | 100.000 | 100.000 | 99.202 | 95.464 | 89.143 |
122 | 100.000 | 100.000 | 99.249 | 95.566 | 89.293 |
123 | 100.000 | 100.000 | 99.296 | 95.668 | 89.443 |
124 | 100.000 | 100.000 | 99.343 | 95.770 | 89.593 |
125 | 100.000 | 100.000 | 99.390 | 95.872 | 89.742 |
126 | 100.000 | 100.000 | 99.437 | 95.973 | 89.892 |
127 | 100.000 | 100.000 | 99.484 | 96.075 | 90.042 |
128 | 100.000 | 100.000 | 99.531 | 96.177 | 90.194 |
129 | 100.000 | 100.000 | 99.554 | 96.252 | 90.332 |
130 | 100.000 | 100.000 | 99.578 | 96.327 | 90.471 |
131 | 100.000 | 100.000 | 99.601 | 96.397 | 90.597 |
132 | 100.000 | 100.000 | 99.625 | 96.467 | 90.722 |
133 | 100.000 | 100.000 | 99.648 | 96.536 | 90.846 |
134 | 100.000 | 100.000 | 99.672 | 96.601 | 90.957 |
135 | 100.000 | 100.000 | 99.695 | 96.665 | 91.068 |
136 | 100.000 | 100.000 | 99.718 | 96.729 | 91.179 |
137 | 100.000 | 100.000 | 99.742 | 96.794 | 91.292 |
138 | 100.000 | 100.000 | 99.765 | 96.858 | 91.405 |
139 | 100.000 | 100.000 | 99.789 | 96.922 | 91.516 |
140 | 100.000 | 100.000 | 99.812 | 96.981 | 91.614 |
141 | 100.000 | 100.000 | 99.836 | 97.035 | 91.697 |
142 | 100.000 | 100.000 | 99.859 | 97.089 | 91.780 |
143 | 100.000 | 100.000 | 99.883 | 97.142 | 91.863 |
144 | 100.000 | 100.000 | 99.906 | 97.196 | 91.947 |
145 | 100.000 | 100.000 | 99.930 | 97.249 | 92.030 |
146 | 100.000 | 100.000 | 99.953 | 97.303 | 92.113 |
147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |


grep
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 42.857 | 21.658 | 12.107 | 7.292 | 4.656 |
3 | 52.381 | 29.272 | 17.269 | 10.696 | 6.923 |
4 | 59.524 | 36.041 | 22.220 | 14.074 | 9.200 |
5 | 71.429 | 44.501 | 27.616 | 17.533 | 11.476 |
6 | 78.571 | 51.100 | 32.426 | 20.830 | 13.716 |
7 | 80.952 | 56.176 | 36.861 | 24.052 | 15.944 |
8 | 85.714 | 61.421 | 41.107 | 27.157 | 18.130 |
9 | 90.476 | 66.159 | 44.932 | 30.025 | 20.212 |
10 | 95.238 | 70.728 | 48.545 | 32.727 | 22.192 |
11 | 95.238 | 73.096 | 51.337 | 35.162 | 24.099 |
12 | 97.619 | 76.311 | 54.411 | 37.698 | 26.044 |
13 | 100.000 | 79.188 | 57.109 | 39.966 | 27.840 |
14 | 100.000 | 80.372 | 58.893 | 41.810 | 29.467 |
15 | 100.000 | 81.726 | 60.863 | 43.772 | 31.144 |
16 | 100.000 | 83.080 | 62.740 | 45.633 | 32.755 |
17 | 100.000 | 84.095 | 64.336 | 47.327 | 34.283 |
18 | 100.000 | 85.618 | 66.283 | 49.166 | 35.836 |
19 | 100.000 | 86.633 | 67.785 | 50.732 | 37.251 |
20 | 100.000 | 87.648 | 69.310 | 52.324 | 38.684 |
21 | 100.000 | 88.325 | 70.507 | 53.697 | 39.997 |
22 | 100.000 | 89.509 | 72.055 | 55.214 | 41.336 |
23 | 100.000 | 90.186 | 73.158 | 56.453 | 42.526 |
24 | 100.000 | 90.694 | 74.097 | 57.579 | 43.653 |
25 | 100.000 | 91.371 | 75.153 | 58.748 | 44.777 |
26 | 100.000 | 92.047 | 76.208 | 59.916 | 45.901 |
27 | 100.000 | 92.555 | 77.100 | 60.978 | 46.970 |
28 | 100.000 | 92.724 | 77.687 | 61.852 | 47.946 |
29 | 100.000 | 93.232 | 78.578 | 62.903 | 48.995 |
30 | 100.000 | 93.739 | 79.446 | 63.922 | 50.016 |
31 | 100.000 | 94.078 | 80.150 | 64.828 | 50.972 |
32 | 100.000 | 94.585 | 80.948 | 65.750 | 51.907 |
33 | 100.000 | 94.755 | 81.534 | 66.608 | 52.847 |
34 | 100.000 | 94.755 | 81.863 | 67.230 | 53.627 |
35 | 100.000 | 95.431 | 82.872 | 68.313 | 54.650 |
36 | 100.000 | 95.601 | 83.365 | 69.037 | 55.471 |
37 | 100.000 | 95.939 | 84.045 | 69.894 | 56.364 |
38 | 100.000 | 95.939 | 84.374 | 70.500 | 57.109 |
39 | 100.000 | 96.277 | 85.007 | 71.294 | 57.947 |
40 | 100.000 | 96.785 | 85.805 | 72.195 | 58.838 |
41 | 100.000 | 97.124 | 86.415 | 72.951 | 59.637 |
42 | 100.000 | 97.631 | 87.213 | 73.851 | 60.525 |
43 | 100.000 | 97.800 | 87.635 | 74.468 | 61.240 |
44 | 100.000 | 97.970 | 88.081 | 75.111 | 61.975 |
45 | 100.000 | 97.970 | 88.339 | 75.610 | 62.616 |
46 | 100.000 | 97.970 | 88.620 | 76.135 | 63.275 |
47 | 100.000 | 97.970 | 88.902 | 76.671 | 63.946 |
48 | 100.000 | 98.139 | 89.301 | 77.245 | 64.613 |
49 | 100.000 | 98.308 | 89.700 | 77.814 | 65.270 |
50 | 100.000 | 98.477 | 90.122 | 78.409 | 65.941 |
51 | 100.000 | 98.646 | 90.521 | 78.972 | 66.585 |
52 | 100.000 | 98.646 | 90.756 | 79.422 | 67.163 |
53 | 100.000 | 98.646 | 90.967 | 79.840 | 67.716 |
54 | 100.000 | 98.816 | 91.366 | 80.398 | 68.349 |
55 | 100.000 | 98.816 | 91.577 | 80.811 | 68.891 |
56 | 100.000 | 98.816 | 91.788 | 81.224 | 69.432 |
57 | 100.000 | 98.816 | 91.976 | 81.610 | 69.954 |
58 | 100.000 | 98.816 | 92.163 | 81.985 | 70.455 |
59 | 100.000 | 98.816 | 92.328 | 82.328 | 70.928 |
60 | 100.000 | 98.985 | 92.679 | 82.811 | 71.483 |
61 | 100.000 | 98.985 | 92.867 | 83.181 | 71.973 |
62 | 100.000 | 98.985 | 93.055 | 83.545 | 72.454 |
63 | 100.000 | 99.154 | 93.360 | 83.958 | 72.942 |
64 | 100.000 | 99.154 | 93.501 | 84.258 | 73.366 |
65 | 100.000 | 99.323 | 93.829 | 84.698 | 73.869 |
66 | 100.000 | 99.323 | 93.993 | 85.025 | 74.307 |
67 | 100.000 | 99.492 | 94.298 | 85.432 | 74.782 |
68 | 100.000 | 99.492 | 94.439 | 85.727 | 75.193 |
69 | 100.000 | 99.492 | 94.580 | 86.022 | 75.603 |
70 | 100.000 | 99.492 | 94.721 | 86.317 | 76.014 |
71 | 100.000 | 99.492 | 94.862 | 86.607 | 76.413 |
72 | 100.000 | 99.662 | 95.190 | 87.046 | 76.909 |
73 | 100.000 | 99.662 | 95.331 | 87.330 | 77.297 |
74 | 100.000 | 99.662 | 95.495 | 87.652 | 77.719 |
75 | 100.000 | 99.662 | 95.612 | 87.904 | 78.079 |
76 | 100.000 | 99.662 | 95.730 | 88.151 | 78.427 |
77 | 100.000 | 99.662 | 95.824 | 88.365 | 78.747 |
78 | 100.000 | 99.662 | 95.941 | 88.612 | 79.093 |
79 | 100.000 | 99.662 | 96.035 | 88.826 | 79.413 |
80 | 100.000 | 99.662 | 96.176 | 89.111 | 79.792 |
81 | 100.000 | 99.662 | 96.269 | 89.320 | 80.102 |
82 | 100.000 | 99.662 | 96.387 | 89.561 | 80.437 |
83 | 100.000 | 99.662 | 96.481 | 89.770 | 80.746 |
84 | 100.000 | 99.662 | 96.574 | 89.979 | 81.053 |
85 | 100.000 | 99.662 | 96.668 | 90.183 | 81.349 |
86 | 100.000 | 99.662 | 96.762 | 90.387 | 81.645 |
87 | 100.000 | 99.662 | 96.856 | 90.596 | 81.948 |
88 | 100.000 | 99.662 | 96.950 | 90.799 | 82.244 |
89 | 100.000 | 99.662 | 97.044 | 91.003 | 82.540 |
90 | 100.000 | 99.662 | 97.091 | 91.143 | 82.780 |
91 | 100.000 | 99.831 | 97.372 | 91.486 | 83.152 |
92 | 100.000 | 99.831 | 97.466 | 91.689 | 83.440 |
93 | 100.000 | 99.831 | 97.489 | 91.786 | 83.634 |
94 | 100.000 | 99.831 | 97.536 | 91.915 | 83.853 |
95 | 100.000 | 99.831 | 97.560 | 92.011 | 84.045 |
96 | 100.000 | 99.831 | 97.630 | 92.172 | 84.289 |
97 | 100.000 | 99.831 | 97.724 | 92.365 | 84.557 |
98 | 100.000 | 99.831 | 97.794 | 92.521 | 84.790 |
99 | 100.000 | 99.831 | 97.841 | 92.644 | 84.995 |
100 | 100.000 | 99.831 | 97.912 | 92.799 | 85.225 |
101 | 100.000 | 99.831 | 97.982 | 92.955 | 85.454 |
102 | 100.000 | 99.831 | 98.053 | 93.110 | 85.683 |
103 | 100.000 | 99.831 | 98.099 | 93.234 | 85.885 |
104 | 100.000 | 99.831 | 98.146 | 93.357 | 86.087 |
105 | 100.000 | 99.831 | 98.193 | 93.475 | 86.279 |
106 | 100.000 | 99.831 | 98.217 | 93.561 | 86.445 |
107 | 100.000 | 99.831 | 98.287 | 93.711 | 86.662 |
108 | 100.000 | 99.831 | 98.358 | 93.861 | 86.878 |
109 | 100.000 | 99.831 | 98.428 | 94.011 | 87.094 |
110 | 100.000 | 100.000 | 98.686 | 94.306 | 87.399 |
111 | 100.000 | 100.000 | 98.756 | 94.456 | 87.616 |
112 | 100.000 | 100.000 | 98.803 | 94.574 | 87.804 |
113 | 100.000 | 100.000 | 98.827 | 94.654 | 87.958 |
114 | 100.000 | 100.000 | 98.874 | 94.767 | 88.135 |
115 | 100.000 | 100.000 | 98.921 | 94.880 | 88.313 |
116 | 100.000 | 100.000 | 98.968 | 94.992 | 88.490 |
117 | 100.000 | 100.000 | 99.038 | 95.137 | 88.692 |
118 | 100.000 | 100.000 | 99.085 | 95.244 | 88.858 |
119 | 100.000 | 100.000 | 99.132 | 95.351 | 89.023 |
120 | 100.000 | 100.000 | 99.179 | 95.453 | 89.175 |
121 | 100.000 | 100.000 | 99.202 | 95.523 | 89.300 |
122 | 100.000 | 100.000 | 99.249 | 95.625 | 89.450 |
123 | 100.000 | 100.000 | 99.273 | 95.695 | 89.574 |
124 | 100.000 | 100.000 | 99.320 | 95.796 | 89.724 |
125 | 100.000 | 100.000 | 99.366 | 95.898 | 89.874 |
126 | 100.000 | 100.000 | 99.413 | 96.000 | 90.023 |
127 | 100.000 | 100.000 | 99.460 | 96.102 | 90.173 |
128 | 100.000 | 100.000 | 99.507 | 96.204 | 90.323 |
129 | 100.000 | 100.000 | 99.531 | 96.274 | 90.447 |
130 | 100.000 | 100.000 | 99.578 | 96.376 | 90.597 |
131 | 100.000 | 100.000 | 99.601 | 96.440 | 90.710 |
132 | 100.000 | 100.000 | 99.625 | 96.504 | 90.820 |
133 | 100.000 | 100.000 | 99.648 | 96.569 | 90.931 |
134 | 100.000 | 100.000 | 99.672 | 96.633 | 91.042 |
135 | 100.000 | 100.000 | 99.695 | 96.697 | 91.153 |
136 | 100.000 | 100.000 | 99.718 | 96.756 | 91.251 |
137 | 100.000 | 100.000 | 99.742 | 96.815 | 91.349 |
138 | 100.000 | 100.000 | 99.765 | 96.874 | 91.447 |
139 | 100.000 | 100.000 | 99.789 | 96.928 | 91.531 |
140 | 100.000 | 100.000 | 99.812 | 96.981 | 91.614 |
141 | 100.000 | 100.000 | 99.836 | 97.035 | 91.697 |
142 | 100.000 | 100.000 | 99.859 | 97.089 | 91.780 |
143 | 100.000 | 100.000 | 99.883 | 97.142 | 91.863 |
144 | 100.000 | 100.000 | 99.906 | 97.196 | 91.947 |
145 | 100.000 | 100.000 | 99.930 | 97.249 | 92.030 |
146 | 100.000 | 100.000 | 99.953 | 97.303 | 92.113 |
147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |


grep
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 66.667 | 35.533 | 19.521 | 11.426 | 7.134 |
4 | 76.190 | 43.486 | 24.941 | 14.964 | 9.457 |
5 | 80.952 | 49.577 | 29.775 | 18.342 | 11.743 |
6 | 85.714 | 55.668 | 34.608 | 21.720 | 14.028 |
7 | 92.857 | 62.098 | 39.301 | 24.964 | 16.249 |
8 | 95.238 | 66.836 | 43.524 | 28.100 | 18.451 |
9 | 97.619 | 70.389 | 46.715 | 30.594 | 20.310 |
10 | 100.000 | 73.435 | 49.648 | 33.049 | 22.218 |
11 | 100.000 | 75.635 | 52.229 | 35.301 | 23.994 |
12 | 100.000 | 77.496 | 54.364 | 37.151 | 25.464 |
13 | 100.000 | 79.357 | 56.617 | 39.183 | 27.119 |
14 | 100.000 | 81.049 | 58.611 | 40.952 | 28.552 |
15 | 100.000 | 82.572 | 60.676 | 42.947 | 30.237 |
16 | 100.000 | 83.926 | 62.694 | 44.957 | 31.943 |
17 | 100.000 | 85.279 | 64.735 | 46.989 | 33.661 |
18 | 100.000 | 86.464 | 66.448 | 48.737 | 35.207 |
19 | 100.000 | 87.648 | 68.067 | 50.319 | 36.566 |
20 | 100.000 | 88.663 | 69.451 | 51.702 | 37.796 |
21 | 100.000 | 89.509 | 70.695 | 53.016 | 39.013 |
22 | 100.000 | 90.355 | 72.102 | 54.560 | 40.437 |
23 | 100.000 | 91.201 | 73.322 | 55.836 | 41.615 |
24 | 100.000 | 92.047 | 74.542 | 57.107 | 42.780 |
25 | 100.000 | 92.893 | 75.809 | 58.426 | 43.974 |
26 | 100.000 | 93.570 | 76.795 | 59.466 | 44.945 |
27 | 100.000 | 94.247 | 77.945 | 60.742 | 46.138 |
28 | 100.000 | 94.755 | 78.672 | 61.509 | 46.859 |
29 | 100.000 | 95.262 | 79.587 | 62.619 | 47.983 |
30 | 100.000 | 95.770 | 80.432 | 63.616 | 48.986 |
31 | 100.000 | 96.108 | 80.995 | 64.286 | 49.683 |
32 | 100.000 | 96.447 | 81.699 | 65.181 | 50.613 |
33 | 100.000 | 96.785 | 82.426 | 66.131 | 51.617 |
34 | 100.000 | 97.124 | 82.942 | 66.715 | 52.207 |
35 | 100.000 | 97.462 | 83.458 | 67.294 | 52.786 |
36 | 100.000 | 97.800 | 84.186 | 68.222 | 53.746 |
37 | 100.000 | 98.139 | 85.007 | 69.267 | 54.798 |
38 | 100.000 | 98.477 | 85.687 | 70.141 | 55.726 |
39 | 100.000 | 98.816 | 86.204 | 70.720 | 56.305 |
40 | 100.000 | 99.154 | 86.861 | 71.546 | 57.163 |
41 | 100.000 | 99.323 | 87.166 | 71.932 | 57.586 |
42 | 100.000 | 99.492 | 87.494 | 72.361 | 58.062 |
43 | 100.000 | 99.662 | 88.175 | 73.379 | 59.179 |
44 | 100.000 | 99.831 | 88.503 | 73.803 | 59.641 |
45 | 100.000 | 100.000 | 88.832 | 74.243 | 60.136 |
46 | 100.000 | 100.000 | 89.043 | 74.650 | 60.665 |
47 | 100.000 | 100.000 | 89.230 | 75.015 | 61.142 |
48 | 100.000 | 100.000 | 89.559 | 75.631 | 61.904 |
49 | 100.000 | 100.000 | 89.747 | 76.007 | 62.405 |
50 | 100.000 | 100.000 | 89.840 | 76.205 | 62.687 |
51 | 100.000 | 100.000 | 89.958 | 76.436 | 62.994 |
52 | 100.000 | 100.000 | 90.075 | 76.671 | 63.313 |
53 | 100.000 | 100.000 | 90.521 | 77.449 | 64.223 |
54 | 100.000 | 100.000 | 90.756 | 77.948 | 64.900 |
55 | 100.000 | 100.000 | 90.943 | 78.312 | 65.377 |
56 | 100.000 | 100.000 | 91.389 | 79.090 | 66.287 |
57 | 100.000 | 100.000 | 91.436 | 79.197 | 66.452 |
58 | 100.000 | 100.000 | 91.600 | 79.524 | 66.890 |
59 | 100.000 | 100.000 | 91.835 | 79.969 | 67.458 |
60 | 100.000 | 100.000 | 92.046 | 80.355 | 67.935 |
61 | 100.000 | 100.000 | 92.140 | 80.569 | 68.251 |
62 | 100.000 | 100.000 | 92.210 | 80.714 | 68.452 |
63 | 100.000 | 100.000 | 92.351 | 81.020 | 68.887 |
64 | 100.000 | 100.000 | 92.398 | 81.127 | 69.052 |
65 | 100.000 | 100.000 | 92.445 | 81.250 | 69.253 |
66 | 100.000 | 100.000 | 92.445 | 81.283 | 69.336 |
67 | 100.000 | 100.000 | 92.586 | 81.577 | 69.747 |
68 | 100.000 | 100.000 | 92.633 | 81.690 | 69.924 |
69 | 100.000 | 100.000 | 92.679 | 81.803 | 70.102 |
70 | 100.000 | 100.000 | 92.726 | 81.910 | 70.265 |
71 | 100.000 | 100.000 | 92.773 | 82.012 | 70.414 |
72 | 100.000 | 100.000 | 92.820 | 82.114 | 70.564 |
73 | 100.000 | 100.000 | 92.938 | 82.371 | 70.934 |
74 | 100.000 | 100.000 | 92.985 | 82.489 | 71.125 |
75 | 100.000 | 100.000 | 93.055 | 82.660 | 71.393 |
76 | 100.000 | 100.000 | 93.196 | 82.955 | 71.788 |
77 | 100.000 | 100.000 | 93.407 | 83.374 | 72.334 |
78 | 100.000 | 100.000 | 93.430 | 83.438 | 72.445 |
79 | 100.000 | 100.000 | 93.618 | 83.813 | 72.942 |
80 | 100.000 | 100.000 | 93.712 | 84.006 | 73.210 |
81 | 100.000 | 100.000 | 93.993 | 84.553 | 73.917 |
82 | 100.000 | 100.000 | 94.040 | 84.687 | 74.142 |
83 | 100.000 | 100.000 | 94.064 | 84.757 | 74.268 |
84 | 100.000 | 100.000 | 94.087 | 84.821 | 74.379 |
85 | 100.000 | 100.000 | 94.158 | 84.966 | 74.581 |
86 | 100.000 | 100.000 | 94.181 | 85.030 | 74.692 |
87 | 100.000 | 100.000 | 94.369 | 85.373 | 75.117 |
88 | 100.000 | 100.000 | 94.369 | 85.406 | 75.200 |
89 | 100.000 | 100.000 | 94.510 | 85.690 | 75.587 |
90 | 100.000 | 100.000 | 94.627 | 85.920 | 75.894 |
91 | 100.000 | 100.000 | 94.674 | 86.070 | 76.160 |
92 | 100.000 | 100.000 | 94.721 | 86.172 | 76.310 |
93 | 100.000 | 100.000 | 94.862 | 86.456 | 76.696 |
94 | 100.000 | 100.000 | 94.932 | 86.633 | 76.970 |
95 | 100.000 | 100.000 | 95.026 | 86.826 | 77.238 |
96 | 100.000 | 100.000 | 95.096 | 86.977 | 77.454 |
97 | 100.000 | 100.000 | 95.214 | 87.245 | 77.850 |
98 | 100.000 | 100.000 | 95.354 | 87.550 | 78.285 |
99 | 100.000 | 100.000 | 95.354 | 87.582 | 78.372 |
100 | 100.000 | 100.000 | 95.378 | 87.641 | 78.470 |
101 | 100.000 | 100.000 | 95.495 | 87.883 | 78.801 |
102 | 100.000 | 100.000 | 95.589 | 88.086 | 79.096 |
103 | 100.000 | 100.000 | 95.659 | 88.231 | 79.298 |
104 | 100.000 | 100.000 | 95.706 | 88.333 | 79.448 |
105 | 100.000 | 100.000 | 95.753 | 88.446 | 79.625 |
106 | 100.000 | 100.000 | 95.894 | 88.719 | 79.984 |
107 | 100.000 | 100.000 | 96.176 | 89.255 | 80.655 |
108 | 100.000 | 100.000 | 96.269 | 89.448 | 80.927 |
109 | 100.000 | 100.000 | 96.340 | 89.604 | 81.156 |
110 | 100.000 | 100.000 | 96.340 | 89.636 | 81.247 |
111 | 100.000 | 100.000 | 96.340 | 89.652 | 81.293 |
112 | 100.000 | 100.000 | 96.387 | 89.754 | 81.443 |
113 | 100.000 | 100.000 | 96.481 | 89.958 | 81.735 |
114 | 100.000 | 100.000 | 96.481 | 89.968 | 81.769 |
115 | 100.000 | 100.000 | 96.527 | 90.076 | 81.933 |
116 | 100.000 | 100.000 | 96.692 | 90.397 | 82.355 |
117 | 100.000 | 100.000 | 96.762 | 90.569 | 82.621 |
118 | 100.000 | 100.000 | 96.786 | 90.622 | 82.704 |
119 | 100.000 | 100.000 | 96.809 | 90.676 | 82.788 |
120 | 100.000 | 100.000 | 96.809 | 90.698 | 82.847 |
121 | 100.000 | 100.000 | 96.856 | 90.799 | 82.996 |
122 | 100.000 | 100.000 | 96.903 | 90.901 | 83.146 |
123 | 100.000 | 100.000 | 96.950 | 91.014 | 83.324 |
124 | 100.000 | 100.000 | 96.997 | 91.121 | 83.488 |
125 | 100.000 | 100.000 | 97.067 | 91.293 | 83.756 |
126 | 100.000 | 100.000 | 97.114 | 91.400 | 83.921 |
127 | 100.000 | 100.000 | 97.114 | 91.421 | 83.984 |
128 | 100.000 | 100.000 | 97.137 | 91.507 | 84.145 |
129 | 100.000 | 100.000 | 97.161 | 91.598 | 84.319 |
130 | 100.000 | 100.000 | 97.161 | 91.609 | 84.350 |
131 | 100.000 | 100.000 | 97.161 | 91.641 | 84.433 |
132 | 100.000 | 100.000 | 97.255 | 91.850 | 84.731 |
133 | 100.000 | 100.000 | 97.255 | 91.893 | 84.838 |
134 | 100.000 | 100.000 | 97.302 | 91.995 | 84.988 |
135 | 100.000 | 100.000 | 97.325 | 92.054 | 85.086 |
136 | 100.000 | 100.000 | 97.396 | 92.204 | 85.302 |
137 | 100.000 | 100.000 | 97.396 | 92.210 | 85.319 |
138 | 100.000 | 100.000 | 97.396 | 92.285 | 85.502 |
139 | 100.000 | 100.000 | 97.396 | 92.295 | 85.534 |
140 | 100.000 | 100.000 | 97.419 | 92.376 | 85.687 |
141 | 100.000 | 100.000 | 97.419 | 92.392 | 85.731 |
142 | 100.000 | 100.000 | 97.513 | 92.596 | 86.024 |
143 | 100.000 | 100.000 | 97.536 | 92.660 | 86.136 |
144 | 100.000 | 100.000 | 97.583 | 92.762 | 86.286 |
145 | 100.000 | 100.000 | 97.607 | 92.815 | 86.369 |
146 | 100.000 | 100.000 | 97.630 | 92.869 | 86.453 |
147 | 100.000 | 100.000 | 97.654 | 92.923 | 86.536 |
148 | 100.000 | 100.000 | 97.654 | 92.923 | 86.540 |
149 | 100.000 | 100.000 | 97.654 | 92.928 | 86.556 |
150 | 100.000 | 100.000 | 97.701 | 93.030 | 86.706 |
151 | 100.000 | 100.000 | 97.748 | 93.142 | 86.884 |
152 | 100.000 | 100.000 | 97.771 | 93.207 | 86.998 |
153 | 100.000 | 100.000 | 97.818 | 93.330 | 87.200 |
154 | 100.000 | 100.000 | 97.818 | 93.346 | 87.246 |
155 | 100.000 | 100.000 | 97.888 | 93.496 | 87.462 |
156 | 100.000 | 100.000 | 97.888 | 93.496 | 87.466 |
157 | 100.000 | 100.000 | 97.912 | 93.577 | 87.616 |
158 | 100.000 | 100.000 | 97.912 | 93.582 | 87.632 |
159 | 100.000 | 100.000 | 97.912 | 93.614 | 87.721 |
160 | 100.000 | 100.000 | 97.912 | 93.641 | 87.797 |
161 | 100.000 | 100.000 | 97.912 | 93.646 | 87.814 |
162 | 100.000 | 100.000 | 97.959 | 93.748 | 87.963 |
163 | 100.000 | 100.000 | 97.959 | 93.759 | 87.995 |
164 | 100.000 | 100.000 | 98.006 | 93.861 | 88.145 |
165 | 100.000 | 100.000 | 98.029 | 93.920 | 88.243 |
166 | 100.000 | 100.000 | 98.053 | 93.995 | 88.383 |
167 | 100.000 | 100.000 | 98.099 | 94.102 | 88.552 |
168 | 100.000 | 100.000 | 98.123 | 94.172 | 88.681 |
169 | 100.000 | 100.000 | 98.123 | 94.193 | 88.740 |
170 | 100.000 | 100.000 | 98.170 | 94.295 | 88.890 |
171 | 100.000 | 100.000 | 98.170 | 94.376 | 89.090 |
172 | 100.000 | 100.000 | 98.193 | 94.429 | 89.173 |
173 | 100.000 | 100.000 | 98.287 | 94.633 | 89.465 |
174 | 100.000 | 100.000 | 98.358 | 94.778 | 89.667 |
175 | 100.000 | 100.000 | 98.498 | 95.073 | 90.077 |
176 | 100.000 | 100.000 | 98.592 | 95.260 | 90.332 |
177 | 100.000 | 100.000 | 98.592 | 95.266 | 90.349 |
178 | 100.000 | 100.000 | 98.639 | 95.368 | 90.499 |
179 | 100.000 | 100.000 | 98.639 | 95.373 | 90.515 |
180 | 100.000 | 100.000 | 98.686 | 95.528 | 90.789 |
181 | 100.000 | 100.000 | 98.733 | 95.630 | 90.939 |
182 | 100.000 | 100.000 | 98.756 | 95.684 | 91.022 |
183 | 100.000 | 100.000 | 98.780 | 95.737 | 91.105 |
184 | 100.000 | 100.000 | 98.780 | 95.754 | 91.155 |
185 | 100.000 | 100.000 | 98.803 | 95.813 | 91.253 |
186 | 100.000 | 100.000 | 98.827 | 95.866 | 91.336 |
187 | 100.000 | 100.000 | 98.850 | 95.952 | 91.499 |
188 | 100.000 | 100.000 | 98.850 | 95.963 | 91.531 |
189 | 100.000 | 100.000 | 98.874 | 96.032 | 91.658 |
190 | 100.000 | 100.000 | 98.874 | 96.048 | 91.704 |
191 | 100.000 | 100.000 | 98.921 | 96.150 | 91.856 |
192 | 100.000 | 100.000 | 98.921 | 96.161 | 91.887 |
193 | 100.000 | 100.000 | 98.921 | 96.172 | 91.919 |
194 | 100.000 | 100.000 | 98.921 | 96.193 | 91.980 |
195 | 100.000 | 100.000 | 98.921 | 96.220 | 92.056 |
196 | 100.000 | 100.000 | 98.944 | 96.274 | 92.139 |
197 | 100.000 | 100.000 | 98.944 | 96.290 | 92.185 |
198 | 100.000 | 100.000 | 98.968 | 96.349 | 92.283 |
199 | 100.000 | 100.000 | 99.015 | 96.451 | 92.433 |
200 | 100.000 | 100.000 | 99.015 | 96.472 | 92.492 |
201 | 100.000 | 100.000 | 99.061 | 96.574 | 92.642 |
202 | 100.000 | 100.000 | 99.085 | 96.628 | 92.725 |
203 | 100.000 | 100.000 | 99.108 | 96.681 | 92.808 |
204 | 100.000 | 100.000 | 99.108 | 96.729 | 92.932 |
205 | 100.000 | 100.000 | 99.108 | 96.740 | 92.964 |
206 | 100.000 | 100.000 | 99.108 | 96.756 | 93.008 |
207 | 100.000 | 100.000 | 99.108 | 96.767 | 93.040 |
208 | 100.000 | 100.000 | 99.108 | 96.772 | 93.056 |
209 | 100.000 | 100.000 | 99.108 | 96.778 | 93.073 |
210 | 100.000 | 100.000 | 99.108 | 96.794 | 93.119 |
211 | 100.000 | 100.000 | 99.132 | 96.847 | 93.204 |
212 | 100.000 | 100.000 | 99.155 | 96.901 | 93.287 |
213 | 100.000 | 100.000 | 99.155 | 96.906 | 93.306 |
214 | 100.000 | 100.000 | 99.155 | 96.928 | 93.365 |
215 | 100.000 | 100.000 | 99.155 | 96.933 | 93.384 |
216 | 100.000 | 100.000 | 99.179 | 96.992 | 93.482 |
217 | 100.000 | 100.000 | 99.179 | 96.997 | 93.498 |
218 | 100.000 | 100.000 | 99.226 | 97.121 | 93.700 |
219 | 100.000 | 100.000 | 99.226 | 97.121 | 93.707 |
220 | 100.000 | 100.000 | 99.226 | 97.126 | 93.724 |
221 | 100.000 | 100.000 | 99.320 | 97.341 | 94.044 |
222 | 100.000 | 100.000 | 99.320 | 97.357 | 94.088 |
223 | 100.000 | 100.000 | 99.343 | 97.410 | 94.171 |
224 | 100.000 | 100.000 | 99.343 | 97.416 | 94.188 |
225 | 100.000 | 100.000 | 99.343 | 97.432 | 94.238 |
226 | 100.000 | 100.000 | 99.343 | 97.432 | 94.242 |
227 | 100.000 | 100.000 | 99.390 | 97.560 | 94.460 |
228 | 100.000 | 100.000 | 99.413 | 97.614 | 94.543 |
229 | 100.000 | 100.000 | 99.460 | 97.721 | 94.706 |
230 | 100.000 | 100.000 | 99.460 | 97.732 | 94.737 |
231 | 100.000 | 100.000 | 99.460 | 97.737 | 94.756 |
232 | 100.000 | 100.000 | 99.460 | 97.743 | 94.772 |
233 | 100.000 | 100.000 | 99.554 | 97.946 | 95.068 |
234 | 100.000 | 100.000 | 99.554 | 97.963 | 95.114 |
235 | 100.000 | 100.000 | 99.578 | 98.016 | 95.198 |
236 | 100.000 | 100.000 | 99.578 | 98.038 | 95.257 |
237 | 100.000 | 100.000 | 99.578 | 98.043 | 95.273 |
238 | 100.000 | 100.000 | 99.578 | 98.043 | 95.275 |
239 | 100.000 | 100.000 | 99.578 | 98.064 | 95.334 |
240 | 100.000 | 100.000 | 99.578 | 98.086 | 95.394 |
241 | 100.000 | 100.000 | 99.578 | 98.091 | 95.410 |
242 | 100.000 | 100.000 | 99.578 | 98.097 | 95.427 |
243 | 100.000 | 100.000 | 99.578 | 98.107 | 95.458 |
244 | 100.000 | 100.000 | 99.601 | 98.161 | 95.542 |
245 | 100.000 | 100.000 | 99.625 | 98.215 | 95.625 |
246 | 100.000 | 100.000 | 99.625 | 98.215 | 95.627 |
247 | 100.000 | 100.000 | 99.672 | 98.327 | 95.804 |
248 | 100.000 | 100.000 | 99.695 | 98.397 | 95.926 |
249 | 100.000 | 100.000 | 99.695 | 98.408 | 95.961 |
250 | 100.000 | 100.000 | 99.695 | 98.418 | 95.993 |
251 | 100.000 | 100.000 | 99.695 | 98.429 | 96.024 |
252 | 100.000 | 100.000 | 99.695 | 98.450 | 96.087 |
253 | 100.000 | 100.000 | 99.742 | 98.552 | 96.237 |
254 | 100.000 | 100.000 | 99.742 | 98.558 | 96.253 |
255 | 100.000 | 100.000 | 99.742 | 98.568 | 96.289 |
256 | 100.000 | 100.000 | 99.742 | 98.601 | 96.376 |
257 | 100.000 | 100.000 | 99.742 | 98.606 | 96.392 |
258 | 100.000 | 100.000 | 99.765 | 98.660 | 96.475 |
259 | 100.000 | 100.000 | 99.765 | 98.665 | 96.492 |
260 | 100.000 | 100.000 | 99.812 | 98.778 | 96.670 |
261 | 100.000 | 100.000 | 99.812 | 98.788 | 96.701 |
262 | 100.000 | 100.000 | 99.836 | 98.853 | 96.812 |
263 | 100.000 | 100.000 | 99.836 | 98.858 | 96.829 |
264 | 100.000 | 100.000 | 99.836 | 98.858 | 96.836 |
265 | 100.000 | 100.000 | 99.859 | 98.912 | 96.919 |
266 | 100.000 | 100.000 | 99.859 | 98.917 | 96.936 |
267 | 100.000 | 100.000 | 99.859 | 98.944 | 97.010 |
268 | 100.000 | 100.000 | 99.859 | 98.944 | 97.012 |
269 | 100.000 | 100.000 | 99.859 | 98.954 | 97.043 |
270 | 100.000 | 100.000 | 99.859 | 98.965 | 97.075 |
271 | 100.000 | 100.000 | 99.906 | 99.067 | 97.224 |
272 | 100.000 | 100.000 | 99.930 | 99.121 | 97.308 |
273 | 100.000 | 100.000 | 99.930 | 99.185 | 97.470 |
274 | 100.000 | 100.000 | 99.930 | 99.185 | 97.474 |
275 | 100.000 | 100.000 | 99.930 | 99.190 | 97.491 |
276 | 100.000 | 100.000 | 99.930 | 99.196 | 97.507 |
277 | 100.000 | 100.000 | 99.930 | 99.217 | 97.568 |
278 | 100.000 | 100.000 | 99.930 | 99.223 | 97.585 |
279 | 100.000 | 100.000 | 99.930 | 99.233 | 97.616 |
280 | 100.000 | 100.000 | 99.930 | 99.239 | 97.633 |
281 | 100.000 | 100.000 | 99.930 | 99.239 | 97.635 |
282 | 100.000 | 100.000 | 99.930 | 99.244 | 97.653 |
283 | 100.000 | 100.000 | 99.930 | 99.260 | 97.698 |
284 | 100.000 | 100.000 | 99.930 | 99.287 | 97.770 |
285 | 100.000 | 100.000 | 99.930 | 99.308 | 97.829 |
286 | 100.000 | 100.000 | 99.930 | 99.319 | 97.860 |
287 | 100.000 | 100.000 | 99.930 | 99.324 | 97.877 |
288 | 100.000 | 100.000 | 99.930 | 99.335 | 97.909 |
289 | 100.000 | 100.000 | 99.930 | 99.335 | 97.910 |
290 | 100.000 | 100.000 | 99.930 | 99.341 | 97.927 |
291 | 100.000 | 100.000 | 99.930 | 99.351 | 97.962 |
292 | 100.000 | 100.000 | 99.930 | 99.351 | 97.964 |
293 | 100.000 | 100.000 | 99.930 | 99.362 | 97.995 |
294 | 100.000 | 100.000 | 99.930 | 99.367 | 98.012 |
295 | 100.000 | 100.000 | 99.930 | 99.367 | 98.014 |
296 | 100.000 | 100.000 | 99.930 | 99.373 | 98.031 |
297 | 100.000 | 100.000 | 99.930 | 99.378 | 98.047 |
298 | 100.000 | 100.000 | 99.930 | 99.383 | 98.064 |
299 | 100.000 | 100.000 | 99.930 | 99.389 | 98.081 |
300 | 100.000 | 100.000 | 99.930 | 99.394 | 98.097 |
301 | 100.000 | 100.000 | 99.930 | 99.399 | 98.119 |
302 | 100.000 | 100.000 | 99.930 | 99.410 | 98.154 |
303 | 100.000 | 100.000 | 99.930 | 99.421 | 98.186 |
304 | 100.000 | 100.000 | 99.930 | 99.442 | 98.245 |
305 | 100.000 | 100.000 | 99.930 | 99.464 | 98.308 |
306 | 100.000 | 100.000 | 99.930 | 99.469 | 98.325 |
307 | 100.000 | 100.000 | 99.930 | 99.485 | 98.371 |
308 | 100.000 | 100.000 | 99.930 | 99.496 | 98.402 |
309 | 100.000 | 100.000 | 99.953 | 99.550 | 98.485 |
310 | 100.000 | 100.000 | 99.953 | 99.555 | 98.502 |
311 | 100.000 | 100.000 | 99.953 | 99.560 | 98.519 |
312 | 100.000 | 100.000 | 99.953 | 99.566 | 98.535 |
313 | 100.000 | 100.000 | 99.953 | 99.566 | 98.537 |
314 | 100.000 | 100.000 | 99.953 | 99.571 | 98.554 |
315 | 100.000 | 100.000 | 99.953 | 99.582 | 98.589 |
316 | 100.000 | 100.000 | 100.000 | 99.694 | 98.767 |
317 | 100.000 | 100.000 | 100.000 | 99.716 | 98.826 |
318 | 100.000 | 100.000 | 100.000 | 99.727 | 98.857 |
319 | 100.000 | 100.000 | 100.000 | 99.732 | 98.874 |
320 | 100.000 | 100.000 | 100.000 | 99.743 | 98.905 |
321 | 100.000 | 100.000 | 100.000 | 99.748 | 98.922 |
322 | 100.000 | 100.000 | 100.000 | 99.748 | 98.931 |
323 | 100.000 | 100.000 | 100.000 | 99.748 | 98.935 |
324 | 100.000 | 100.000 | 100.000 | 99.753 | 98.951 |
325 | 100.000 | 100.000 | 100.000 | 99.759 | 98.968 |
326 | 100.000 | 100.000 | 100.000 | 99.759 | 98.970 |
327 | 100.000 | 100.000 | 100.000 | 99.759 | 98.977 |
328 | 100.000 | 100.000 | 100.000 | 99.759 | 98.981 |
329 | 100.000 | 100.000 | 100.000 | 99.769 | 99.016 |
330 | 100.000 | 100.000 | 100.000 | 99.780 | 99.048 |
331 | 100.000 | 100.000 | 100.000 | 99.786 | 99.064 |
332 | 100.000 | 100.000 | 100.000 | 99.791 | 99.081 |
333 | 100.000 | 100.000 | 100.000 | 99.812 | 99.140 |
334 | 100.000 | 100.000 | 100.000 | 99.823 | 99.172 |
335 | 100.000 | 100.000 | 100.000 | 99.828 | 99.188 |
336 | 100.000 | 100.000 | 100.000 | 99.834 | 99.205 |
337 | 100.000 | 100.000 | 100.000 | 99.845 | 99.236 |
338 | 100.000 | 100.000 | 100.000 | 99.866 | 99.295 |
339 | 100.000 | 100.000 | 100.000 | 99.866 | 99.299 |
340 | 100.000 | 100.000 | 100.000 | 99.866 | 99.301 |
341 | 100.000 | 100.000 | 100.000 | 99.877 | 99.336 |
342 | 100.000 | 100.000 | 100.000 | 99.877 | 99.338 |
343 | 100.000 | 100.000 | 100.000 | 99.887 | 99.369 |
344 | 100.000 | 100.000 | 100.000 | 99.893 | 99.386 |
345 | 100.000 | 100.000 | 100.000 | 99.903 | 99.417 |
346 | 100.000 | 100.000 | 100.000 | 99.914 | 99.449 |
347 | 100.000 | 100.000 | 100.000 | 99.920 | 99.466 |
348 | 100.000 | 100.000 | 100.000 | 99.920 | 99.469 |
349 | 100.000 | 100.000 | 100.000 | 99.930 | 99.504 |
350 | 100.000 | 100.000 | 100.000 | 99.936 | 99.521 |
351 | 100.000 | 100.000 | 100.000 | 99.941 | 99.538 |
352 | 100.000 | 100.000 | 100.000 | 99.946 | 99.554 |
353 | 100.000 | 100.000 | 100.000 | 99.957 | 99.586 |
354 | 100.000 | 100.000 | 100.000 | 99.973 | 99.630 |
355 | 100.000 | 100.000 | 100.000 | 99.979 | 99.647 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |


grep
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
4 | 71.429 | 41.794 | 24.472 | 14.857 | 9.438 |
5 | 78.571 | 48.900 | 29.634 | 18.321 | 11.741 |
6 | 85.714 | 55.330 | 34.374 | 21.613 | 13.991 |
7 | 88.095 | 59.391 | 38.128 | 24.530 | 16.112 |
8 | 92.857 | 63.959 | 41.835 | 27.334 | 18.159 |
9 | 97.619 | 68.866 | 45.730 | 30.197 | 20.214 |
10 | 100.000 | 72.420 | 49.061 | 32.867 | 22.215 |
11 | 100.000 | 74.958 | 51.971 | 35.355 | 24.140 |
12 | 100.000 | 77.157 | 54.622 | 37.708 | 26.009 |
13 | 100.000 | 79.188 | 57.156 | 39.998 | 27.844 |
14 | 100.000 | 81.049 | 59.456 | 42.110 | 29.582 |
15 | 100.000 | 82.572 | 61.520 | 44.116 | 31.283 |
16 | 100.000 | 84.095 | 63.491 | 45.987 | 32.864 |
17 | 100.000 | 85.279 | 65.275 | 47.820 | 34.475 |
18 | 100.000 | 86.464 | 66.917 | 49.467 | 35.927 |
19 | 100.000 | 87.479 | 68.466 | 51.102 | 37.408 |
20 | 100.000 | 88.663 | 70.108 | 52.748 | 38.859 |
21 | 100.000 | 89.679 | 71.563 | 54.254 | 40.228 |
22 | 100.000 | 90.694 | 72.994 | 55.718 | 41.548 |
23 | 100.000 | 91.371 | 74.214 | 57.128 | 42.896 |
24 | 100.000 | 92.047 | 75.340 | 58.394 | 44.104 |
25 | 100.000 | 92.724 | 76.420 | 59.605 | 45.276 |
26 | 100.000 | 93.401 | 77.475 | 60.790 | 46.430 |
27 | 100.000 | 93.739 | 78.297 | 61.879 | 47.567 |
28 | 100.000 | 94.416 | 79.282 | 62.930 | 48.562 |
29 | 100.000 | 94.585 | 79.962 | 63.932 | 49.653 |
30 | 100.000 | 95.093 | 80.831 | 64.951 | 50.667 |
31 | 100.000 | 95.601 | 81.699 | 65.964 | 51.674 |
32 | 100.000 | 96.108 | 82.543 | 66.962 | 52.688 |
33 | 100.000 | 96.616 | 83.365 | 67.894 | 53.607 |
34 | 100.000 | 96.954 | 84.069 | 68.801 | 54.563 |
35 | 100.000 | 97.124 | 84.679 | 69.696 | 55.537 |
36 | 100.000 | 97.631 | 85.453 | 70.565 | 56.401 |
37 | 100.000 | 97.970 | 86.133 | 71.428 | 57.309 |
38 | 100.000 | 98.308 | 86.767 | 72.205 | 58.112 |
39 | 100.000 | 98.646 | 87.400 | 72.988 | 58.927 |
40 | 100.000 | 98.816 | 87.870 | 73.701 | 59.767 |
41 | 100.000 | 98.985 | 88.339 | 74.377 | 60.529 |
42 | 100.000 | 99.323 | 88.855 | 74.956 | 61.107 |
43 | 100.000 | 99.323 | 89.183 | 75.551 | 61.832 |
44 | 100.000 | 99.492 | 89.606 | 76.157 | 62.528 |
45 | 100.000 | 99.492 | 89.934 | 76.752 | 63.252 |
46 | 100.000 | 99.492 | 90.239 | 77.320 | 63.959 |
47 | 100.000 | 99.492 | 90.544 | 77.883 | 64.656 |
48 | 100.000 | 99.492 | 90.826 | 78.414 | 65.325 |
49 | 100.000 | 99.492 | 91.084 | 78.907 | 65.956 |
50 | 100.000 | 99.492 | 91.342 | 79.406 | 66.599 |
51 | 100.000 | 99.492 | 91.577 | 79.862 | 67.191 |
52 | 100.000 | 99.492 | 91.811 | 80.307 | 67.759 |
53 | 100.000 | 99.662 | 92.140 | 80.746 | 68.255 |
54 | 100.000 | 99.831 | 92.468 | 81.207 | 68.802 |
55 | 100.000 | 99.831 | 92.679 | 81.626 | 69.347 |
56 | 100.000 | 99.831 | 92.891 | 82.028 | 69.863 |
57 | 100.000 | 99.831 | 93.102 | 82.441 | 70.403 |
58 | 100.000 | 99.831 | 93.313 | 82.859 | 70.956 |
59 | 100.000 | 99.831 | 93.501 | 83.213 | 71.406 |
60 | 100.000 | 99.831 | 93.688 | 83.583 | 71.894 |
61 | 100.000 | 100.000 | 93.970 | 83.926 | 72.265 |
62 | 100.000 | 100.000 | 94.158 | 84.296 | 72.756 |
63 | 100.000 | 100.000 | 94.345 | 84.671 | 73.257 |
64 | 100.000 | 100.000 | 94.533 | 85.025 | 73.706 |
65 | 100.000 | 100.000 | 94.697 | 85.352 | 74.144 |
66 | 100.000 | 100.000 | 94.862 | 85.700 | 74.631 |
67 | 100.000 | 100.000 | 95.002 | 85.985 | 75.010 |
68 | 100.000 | 100.000 | 95.143 | 86.290 | 75.444 |
69 | 100.000 | 100.000 | 95.284 | 86.585 | 75.857 |
70 | 100.000 | 100.000 | 95.425 | 86.875 | 76.258 |
71 | 100.000 | 100.000 | 95.565 | 87.148 | 76.617 |
72 | 100.000 | 100.000 | 95.683 | 87.389 | 76.953 |
73 | 100.000 | 100.000 | 95.800 | 87.636 | 77.301 |
74 | 100.000 | 100.000 | 95.917 | 87.883 | 77.649 |
75 | 100.000 | 100.000 | 96.035 | 88.129 | 77.996 |
76 | 100.000 | 100.000 | 96.152 | 88.387 | 78.368 |
77 | 100.000 | 100.000 | 96.269 | 88.623 | 78.688 |
78 | 100.000 | 100.000 | 96.387 | 88.864 | 79.023 |
79 | 100.000 | 100.000 | 96.481 | 89.057 | 79.291 |
80 | 100.000 | 100.000 | 96.574 | 89.271 | 79.614 |
81 | 100.000 | 100.000 | 96.668 | 89.480 | 79.919 |
82 | 100.000 | 100.000 | 96.762 | 89.673 | 80.188 |
83 | 100.000 | 100.000 | 96.856 | 89.888 | 80.507 |
84 | 100.000 | 100.000 | 96.950 | 90.092 | 80.796 |
85 | 100.000 | 100.000 | 97.044 | 90.285 | 81.064 |
86 | 100.000 | 100.000 | 97.137 | 90.478 | 81.332 |
87 | 100.000 | 100.000 | 97.231 | 90.703 | 81.678 |
88 | 100.000 | 100.000 | 97.325 | 90.917 | 81.998 |
89 | 100.000 | 100.000 | 97.419 | 91.110 | 82.266 |
90 | 100.000 | 100.000 | 97.489 | 91.261 | 82.482 |
91 | 100.000 | 100.000 | 97.560 | 91.432 | 82.754 |
92 | 100.000 | 100.000 | 97.630 | 91.588 | 82.984 |
93 | 100.000 | 100.000 | 97.701 | 91.732 | 83.185 |
94 | 100.000 | 100.000 | 97.771 | 91.893 | 83.429 |
95 | 100.000 | 100.000 | 97.841 | 92.054 | 83.671 |
96 | 100.000 | 100.000 | 97.912 | 92.204 | 83.888 |
97 | 100.000 | 100.000 | 97.982 | 92.392 | 84.193 |
98 | 100.000 | 100.000 | 98.053 | 92.547 | 84.422 |
99 | 100.000 | 100.000 | 98.123 | 92.697 | 84.639 |
100 | 100.000 | 100.000 | 98.193 | 92.848 | 84.855 |
101 | 100.000 | 100.000 | 98.264 | 93.030 | 85.147 |
102 | 100.000 | 100.000 | 98.334 | 93.207 | 85.428 |
103 | 100.000 | 100.000 | 98.381 | 93.319 | 85.606 |
104 | 100.000 | 100.000 | 98.428 | 93.432 | 85.783 |
105 | 100.000 | 100.000 | 98.475 | 93.561 | 85.998 |
106 | 100.000 | 100.000 | 98.522 | 93.684 | 86.201 |
107 | 100.000 | 100.000 | 98.569 | 93.813 | 86.419 |
108 | 100.000 | 100.000 | 98.616 | 93.915 | 86.569 |
109 | 100.000 | 100.000 | 98.663 | 94.038 | 86.771 |
110 | 100.000 | 100.000 | 98.710 | 94.167 | 86.989 |
111 | 100.000 | 100.000 | 98.756 | 94.279 | 87.168 |
112 | 100.000 | 100.000 | 98.803 | 94.402 | 87.370 |
113 | 100.000 | 100.000 | 98.850 | 94.504 | 87.520 |
114 | 100.000 | 100.000 | 98.897 | 94.606 | 87.669 |
115 | 100.000 | 100.000 | 98.944 | 94.708 | 87.819 |
116 | 100.000 | 100.000 | 98.991 | 94.831 | 88.021 |
117 | 100.000 | 100.000 | 99.038 | 94.933 | 88.171 |
118 | 100.000 | 100.000 | 99.085 | 95.046 | 88.350 |
119 | 100.000 | 100.000 | 99.132 | 95.158 | 88.527 |
120 | 100.000 | 100.000 | 99.179 | 95.271 | 88.705 |
121 | 100.000 | 100.000 | 99.226 | 95.373 | 88.855 |
122 | 100.000 | 100.000 | 99.249 | 95.427 | 88.938 |
123 | 100.000 | 100.000 | 99.273 | 95.485 | 89.036 |
124 | 100.000 | 100.000 | 99.296 | 95.550 | 89.149 |
125 | 100.000 | 100.000 | 99.320 | 95.609 | 89.247 |
126 | 100.000 | 100.000 | 99.343 | 95.679 | 89.373 |
127 | 100.000 | 100.000 | 99.366 | 95.743 | 89.485 |
128 | 100.000 | 100.000 | 99.390 | 95.813 | 89.611 |
129 | 100.000 | 100.000 | 99.413 | 95.866 | 89.694 |
130 | 100.000 | 100.000 | 99.437 | 95.941 | 89.833 |
131 | 100.000 | 100.000 | 99.460 | 96.038 | 90.025 |
132 | 100.000 | 100.000 | 99.484 | 96.113 | 90.164 |
133 | 100.000 | 100.000 | 99.507 | 96.172 | 90.262 |
134 | 100.000 | 100.000 | 99.531 | 96.225 | 90.345 |
135 | 100.000 | 100.000 | 99.554 | 96.290 | 90.456 |
136 | 100.000 | 100.000 | 99.578 | 96.365 | 90.595 |
137 | 100.000 | 100.000 | 99.601 | 96.445 | 90.747 |
138 | 100.000 | 100.000 | 99.625 | 96.510 | 90.857 |
139 | 100.000 | 100.000 | 99.648 | 96.585 | 90.996 |
140 | 100.000 | 100.000 | 99.672 | 96.638 | 91.079 |
141 | 100.000 | 100.000 | 99.695 | 96.708 | 91.205 |
142 | 100.000 | 100.000 | 99.718 | 96.767 | 91.303 |
143 | 100.000 | 100.000 | 99.742 | 96.821 | 91.386 |
144 | 100.000 | 100.000 | 99.765 | 96.906 | 91.549 |
145 | 100.000 | 100.000 | 99.789 | 96.976 | 91.673 |
146 | 100.000 | 100.000 | 99.812 | 97.030 | 91.756 |
147 | 100.000 | 100.000 | 99.836 | 97.089 | 91.858 |
148 | 100.000 | 100.000 | 99.859 | 97.142 | 91.941 |
149 | 100.000 | 100.000 | 99.883 | 97.201 | 92.039 |
150 | 100.000 | 100.000 | 99.906 | 97.266 | 92.150 |
151 | 100.000 | 100.000 | 99.930 | 97.325 | 92.248 |
152 | 100.000 | 100.000 | 99.953 | 97.378 | 92.331 |
153 | 100.000 | 100.000 | 99.977 | 97.432 | 92.415 |
154 | 100.000 | 100.000 | 100.000 | 97.501 | 92.540 |
155 | 100.000 | 100.000 | 100.000 | 97.512 | 92.572 |
156 | 100.000 | 100.000 | 100.000 | 97.518 | 92.588 |
157 | 100.000 | 100.000 | 100.000 | 97.534 | 92.633 |
158 | 100.000 | 100.000 | 100.000 | 97.544 | 92.666 |
159 | 100.000 | 100.000 | 100.000 | 97.560 | 92.712 |
160 | 100.000 | 100.000 | 100.000 | 97.566 | 92.731 |
161 | 100.000 | 100.000 | 100.000 | 97.593 | 92.805 |
162 | 100.000 | 100.000 | 100.000 | 97.614 | 92.864 |
163 | 100.000 | 100.000 | 100.000 | 97.630 | 92.910 |
164 | 100.000 | 100.000 | 100.000 | 97.668 | 93.006 |
165 | 100.000 | 100.000 | 100.000 | 97.700 | 93.093 |
166 | 100.000 | 100.000 | 100.000 | 97.716 | 93.138 |
167 | 100.000 | 100.000 | 100.000 | 97.727 | 93.169 |
168 | 100.000 | 100.000 | 100.000 | 97.748 | 93.228 |
169 | 100.000 | 100.000 | 100.000 | 97.759 | 93.260 |
170 | 100.000 | 100.000 | 100.000 | 97.775 | 93.306 |
171 | 100.000 | 100.000 | 100.000 | 97.780 | 93.326 |
172 | 100.000 | 100.000 | 100.000 | 97.786 | 93.345 |
173 | 100.000 | 100.000 | 100.000 | 97.807 | 93.404 |
174 | 100.000 | 100.000 | 100.000 | 97.812 | 93.420 |
175 | 100.000 | 100.000 | 100.000 | 97.823 | 93.452 |
176 | 100.000 | 100.000 | 100.000 | 97.845 | 93.511 |
177 | 100.000 | 100.000 | 100.000 | 97.855 | 93.543 |
178 | 100.000 | 100.000 | 100.000 | 97.877 | 93.605 |
179 | 100.000 | 100.000 | 100.000 | 97.946 | 93.777 |
180 | 100.000 | 100.000 | 100.000 | 97.963 | 93.822 |
181 | 100.000 | 100.000 | 100.000 | 97.979 | 93.866 |
182 | 100.000 | 100.000 | 100.000 | 97.984 | 93.885 |
183 | 100.000 | 100.000 | 100.000 | 98.005 | 93.944 |
184 | 100.000 | 100.000 | 100.000 | 98.016 | 93.975 |
185 | 100.000 | 100.000 | 100.000 | 98.027 | 94.010 |
186 | 100.000 | 100.000 | 100.000 | 98.059 | 94.094 |
187 | 100.000 | 100.000 | 100.000 | 98.075 | 94.140 |
188 | 100.000 | 100.000 | 100.000 | 98.102 | 94.212 |
189 | 100.000 | 100.000 | 100.000 | 98.113 | 94.243 |
190 | 100.000 | 100.000 | 100.000 | 98.123 | 94.279 |
191 | 100.000 | 100.000 | 100.000 | 98.198 | 94.458 |
192 | 100.000 | 100.000 | 100.000 | 98.204 | 94.475 |
193 | 100.000 | 100.000 | 100.000 | 98.247 | 94.585 |
194 | 100.000 | 100.000 | 100.000 | 98.257 | 94.621 |
195 | 100.000 | 100.000 | 100.000 | 98.322 | 94.780 |
196 | 100.000 | 100.000 | 100.000 | 98.333 | 94.811 |
197 | 100.000 | 100.000 | 100.000 | 98.365 | 94.898 |
198 | 100.000 | 100.000 | 100.000 | 98.370 | 94.915 |
199 | 100.000 | 100.000 | 100.000 | 98.386 | 94.961 |
200 | 100.000 | 100.000 | 100.000 | 98.402 | 95.007 |
201 | 100.000 | 100.000 | 100.000 | 98.429 | 95.079 |
202 | 100.000 | 100.000 | 100.000 | 98.445 | 95.125 |
203 | 100.000 | 100.000 | 100.000 | 98.467 | 95.185 |
204 | 100.000 | 100.000 | 100.000 | 98.483 | 95.233 |
205 | 100.000 | 100.000 | 100.000 | 98.499 | 95.277 |
206 | 100.000 | 100.000 | 100.000 | 98.515 | 95.325 |
207 | 100.000 | 100.000 | 100.000 | 98.531 | 95.370 |
208 | 100.000 | 100.000 | 100.000 | 98.542 | 95.401 |
209 | 100.000 | 100.000 | 100.000 | 98.552 | 95.447 |
210 | 100.000 | 100.000 | 100.000 | 98.568 | 95.493 |
211 | 100.000 | 100.000 | 100.000 | 98.601 | 95.577 |
212 | 100.000 | 100.000 | 100.000 | 98.606 | 95.593 |
213 | 100.000 | 100.000 | 100.000 | 98.649 | 95.704 |
214 | 100.000 | 100.000 | 100.000 | 98.660 | 95.736 |
215 | 100.000 | 100.000 | 100.000 | 98.665 | 95.754 |
216 | 100.000 | 100.000 | 100.000 | 98.670 | 95.771 |
217 | 100.000 | 100.000 | 100.000 | 98.676 | 95.787 |
218 | 100.000 | 100.000 | 100.000 | 98.697 | 95.847 |
219 | 100.000 | 100.000 | 100.000 | 98.713 | 95.891 |
220 | 100.000 | 100.000 | 100.000 | 98.719 | 95.908 |
221 | 100.000 | 100.000 | 100.000 | 98.724 | 95.924 |
222 | 100.000 | 100.000 | 100.000 | 98.745 | 95.984 |
223 | 100.000 | 100.000 | 100.000 | 98.751 | 96.000 |
224 | 100.000 | 100.000 | 100.000 | 98.756 | 96.017 |
225 | 100.000 | 100.000 | 100.000 | 98.761 | 96.033 |
226 | 100.000 | 100.000 | 100.000 | 98.778 | 96.080 |
227 | 100.000 | 100.000 | 100.000 | 98.778 | 96.089 |
228 | 100.000 | 100.000 | 100.000 | 98.788 | 96.120 |
229 | 100.000 | 100.000 | 100.000 | 98.831 | 96.231 |
230 | 100.000 | 100.000 | 100.000 | 98.837 | 96.248 |
231 | 100.000 | 100.000 | 100.000 | 98.842 | 96.266 |
232 | 100.000 | 100.000 | 100.000 | 98.853 | 96.298 |
233 | 100.000 | 100.000 | 100.000 | 98.879 | 96.370 |
234 | 100.000 | 100.000 | 100.000 | 98.912 | 96.457 |
235 | 100.000 | 100.000 | 100.000 | 98.917 | 96.474 |
236 | 100.000 | 100.000 | 100.000 | 98.922 | 96.490 |
237 | 100.000 | 100.000 | 100.000 | 98.944 | 96.549 |
238 | 100.000 | 100.000 | 100.000 | 98.949 | 96.568 |
239 | 100.000 | 100.000 | 100.000 | 98.971 | 96.627 |
240 | 100.000 | 100.000 | 100.000 | 99.003 | 96.714 |
241 | 100.000 | 100.000 | 100.000 | 99.013 | 96.745 |
242 | 100.000 | 100.000 | 100.000 | 99.019 | 96.762 |
243 | 100.000 | 100.000 | 100.000 | 99.035 | 96.808 |
244 | 100.000 | 100.000 | 100.000 | 99.040 | 96.825 |
245 | 100.000 | 100.000 | 100.000 | 99.056 | 96.869 |
246 | 100.000 | 100.000 | 100.000 | 99.056 | 96.873 |
247 | 100.000 | 100.000 | 100.000 | 99.078 | 96.932 |
248 | 100.000 | 100.000 | 100.000 | 99.089 | 96.967 |
249 | 100.000 | 100.000 | 100.000 | 99.094 | 96.986 |
250 | 100.000 | 100.000 | 100.000 | 99.105 | 97.017 |
251 | 100.000 | 100.000 | 100.000 | 99.121 | 97.065 |
252 | 100.000 | 100.000 | 100.000 | 99.131 | 97.097 |
253 | 100.000 | 100.000 | 100.000 | 99.137 | 97.115 |
254 | 100.000 | 100.000 | 100.000 | 99.147 | 97.150 |
255 | 100.000 | 100.000 | 100.000 | 99.164 | 97.197 |
256 | 100.000 | 100.000 | 100.000 | 99.169 | 97.213 |
257 | 100.000 | 100.000 | 100.000 | 99.180 | 97.245 |
258 | 100.000 | 100.000 | 100.000 | 99.190 | 97.276 |
259 | 100.000 | 100.000 | 100.000 | 99.212 | 97.339 |
260 | 100.000 | 100.000 | 100.000 | 99.217 | 97.356 |
261 | 100.000 | 100.000 | 100.000 | 99.282 | 97.518 |
262 | 100.000 | 100.000 | 100.000 | 99.292 | 97.550 |
263 | 100.000 | 100.000 | 100.000 | 99.303 | 97.581 |
264 | 100.000 | 100.000 | 100.000 | 99.303 | 97.585 |
265 | 100.000 | 100.000 | 100.000 | 99.330 | 97.661 |
266 | 100.000 | 100.000 | 100.000 | 99.330 | 97.664 |
267 | 100.000 | 100.000 | 100.000 | 99.335 | 97.681 |
268 | 100.000 | 100.000 | 100.000 | 99.341 | 97.701 |
269 | 100.000 | 100.000 | 100.000 | 99.351 | 97.733 |
270 | 100.000 | 100.000 | 100.000 | 99.357 | 97.750 |
271 | 100.000 | 100.000 | 100.000 | 99.378 | 97.809 |
272 | 100.000 | 100.000 | 100.000 | 99.389 | 97.840 |
273 | 100.000 | 100.000 | 100.000 | 99.394 | 97.857 |
274 | 100.000 | 100.000 | 100.000 | 99.405 | 97.888 |
275 | 100.000 | 100.000 | 100.000 | 99.410 | 97.905 |
276 | 100.000 | 100.000 | 100.000 | 99.416 | 97.923 |
277 | 100.000 | 100.000 | 100.000 | 99.421 | 97.942 |
278 | 100.000 | 100.000 | 100.000 | 99.426 | 97.958 |
279 | 100.000 | 100.000 | 100.000 | 99.432 | 97.975 |
280 | 100.000 | 100.000 | 100.000 | 99.442 | 98.007 |
281 | 100.000 | 100.000 | 100.000 | 99.448 | 98.023 |
282 | 100.000 | 100.000 | 100.000 | 99.469 | 98.082 |
283 | 100.000 | 100.000 | 100.000 | 99.501 | 98.169 |
284 | 100.000 | 100.000 | 100.000 | 99.501 | 98.173 |
285 | 100.000 | 100.000 | 100.000 | 99.501 | 98.175 |
286 | 100.000 | 100.000 | 100.000 | 99.512 | 98.206 |
287 | 100.000 | 100.000 | 100.000 | 99.534 | 98.265 |
288 | 100.000 | 100.000 | 100.000 | 99.544 | 98.297 |
289 | 100.000 | 100.000 | 100.000 | 99.555 | 98.328 |
290 | 100.000 | 100.000 | 100.000 | 99.555 | 98.330 |
291 | 100.000 | 100.000 | 100.000 | 99.566 | 98.362 |
292 | 100.000 | 100.000 | 100.000 | 99.571 | 98.378 |
293 | 100.000 | 100.000 | 100.000 | 99.576 | 98.395 |
294 | 100.000 | 100.000 | 100.000 | 99.587 | 98.426 |
295 | 100.000 | 100.000 | 100.000 | 99.593 | 98.443 |
296 | 100.000 | 100.000 | 100.000 | 99.598 | 98.460 |
297 | 100.000 | 100.000 | 100.000 | 99.598 | 98.463 |
298 | 100.000 | 100.000 | 100.000 | 99.609 | 98.495 |
299 | 100.000 | 100.000 | 100.000 | 99.609 | 98.498 |
300 | 100.000 | 100.000 | 100.000 | 99.619 | 98.530 |
301 | 100.000 | 100.000 | 100.000 | 99.625 | 98.547 |
302 | 100.000 | 100.000 | 100.000 | 99.635 | 98.582 |
303 | 100.000 | 100.000 | 100.000 | 99.646 | 98.613 |
304 | 100.000 | 100.000 | 100.000 | 99.651 | 98.630 |
305 | 100.000 | 100.000 | 100.000 | 99.662 | 98.661 |
306 | 100.000 | 100.000 | 100.000 | 99.668 | 98.678 |
307 | 100.000 | 100.000 | 100.000 | 99.668 | 98.685 |
308 | 100.000 | 100.000 | 100.000 | 99.700 | 98.772 |
309 | 100.000 | 100.000 | 100.000 | 99.721 | 98.831 |
310 | 100.000 | 100.000 | 100.000 | 99.727 | 98.848 |
311 | 100.000 | 100.000 | 100.000 | 99.732 | 98.866 |
312 | 100.000 | 100.000 | 100.000 | 99.753 | 98.926 |
313 | 100.000 | 100.000 | 100.000 | 99.764 | 98.957 |
314 | 100.000 | 100.000 | 100.000 | 99.769 | 98.974 |
315 | 100.000 | 100.000 | 100.000 | 99.780 | 99.005 |
316 | 100.000 | 100.000 | 100.000 | 99.786 | 99.022 |
317 | 100.000 | 100.000 | 100.000 | 99.818 | 99.109 |
318 | 100.000 | 100.000 | 100.000 | 99.823 | 99.125 |
319 | 100.000 | 100.000 | 100.000 | 99.823 | 99.129 |
320 | 100.000 | 100.000 | 100.000 | 99.823 | 99.133 |
321 | 100.000 | 100.000 | 100.000 | 99.823 | 99.135 |
322 | 100.000 | 100.000 | 100.000 | 99.828 | 99.151 |
323 | 100.000 | 100.000 | 100.000 | 99.828 | 99.153 |
324 | 100.000 | 100.000 | 100.000 | 99.828 | 99.160 |
325 | 100.000 | 100.000 | 100.000 | 99.828 | 99.162 |
326 | 100.000 | 100.000 | 100.000 | 99.828 | 99.164 |
327 | 100.000 | 100.000 | 100.000 | 99.839 | 99.196 |
328 | 100.000 | 100.000 | 100.000 | 99.861 | 99.255 |
329 | 100.000 | 100.000 | 100.000 | 99.861 | 99.257 |
330 | 100.000 | 100.000 | 100.000 | 99.866 | 99.273 |
331 | 100.000 | 100.000 | 100.000 | 99.871 | 99.295 |
332 | 100.000 | 100.000 | 100.000 | 99.877 | 99.312 |
333 | 100.000 | 100.000 | 100.000 | 99.887 | 99.344 |
334 | 100.000 | 100.000 | 100.000 | 99.887 | 99.345 |
335 | 100.000 | 100.000 | 100.000 | 99.898 | 99.377 |
336 | 100.000 | 100.000 | 100.000 | 99.909 | 99.408 |
337 | 100.000 | 100.000 | 100.000 | 99.920 | 99.440 |
338 | 100.000 | 100.000 | 100.000 | 99.925 | 99.456 |
339 | 100.000 | 100.000 | 100.000 | 99.925 | 99.458 |
340 | 100.000 | 100.000 | 100.000 | 99.930 | 99.475 |
341 | 100.000 | 100.000 | 100.000 | 99.941 | 99.510 |
342 | 100.000 | 100.000 | 100.000 | 99.946 | 99.527 |
343 | 100.000 | 100.000 | 100.000 | 99.946 | 99.530 |
344 | 100.000 | 100.000 | 100.000 | 99.946 | 99.532 |
345 | 100.000 | 100.000 | 100.000 | 99.952 | 99.549 |
346 | 100.000 | 100.000 | 100.000 | 99.962 | 99.580 |
347 | 100.000 | 100.000 | 100.000 | 99.968 | 99.597 |
348 | 100.000 | 100.000 | 100.000 | 99.973 | 99.614 |
349 | 100.000 | 100.000 | 100.000 | 99.979 | 99.630 |
350 | 100.000 | 100.000 | 100.000 | 99.984 | 99.647 |
351 | 100.000 | 100.000 | 100.000 | 99.984 | 99.650 |
352 | 100.000 | 100.000 | 100.000 | 99.984 | 99.654 |
353 | 100.000 | 100.000 | 100.000 | 99.989 | 99.671 |
354 | 100.000 | 100.000 | 100.000 | 99.989 | 99.673 |
355 | 100.000 | 100.000 | 100.000 | 99.995 | 99.689 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |


grep
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
3 | 59.524 | 32.826 | 18.700 | 11.211 | 7.088 |
4 | 69.048 | 40.778 | 24.097 | 14.728 | 9.400 |
5 | 76.190 | 47.547 | 29.071 | 18.128 | 11.687 |
6 | 80.952 | 53.299 | 33.693 | 21.420 | 13.947 |
7 | 83.333 | 57.868 | 37.799 | 24.503 | 16.131 |
8 | 88.095 | 62.944 | 41.952 | 27.575 | 18.309 |
9 | 90.476 | 66.328 | 45.096 | 30.122 | 20.251 |
10 | 95.238 | 70.897 | 48.733 | 32.851 | 22.250 |
11 | 97.619 | 74.112 | 51.807 | 35.387 | 24.195 |
12 | 97.619 | 75.973 | 54.223 | 37.633 | 26.028 |
13 | 100.000 | 79.019 | 57.039 | 39.955 | 27.842 |
14 | 100.000 | 80.541 | 59.057 | 41.896 | 29.489 |
15 | 100.000 | 81.387 | 60.535 | 43.553 | 31.022 |
16 | 100.000 | 82.741 | 62.459 | 45.456 | 32.653 |
17 | 100.000 | 84.095 | 64.313 | 47.284 | 34.236 |
18 | 100.000 | 84.772 | 65.580 | 48.753 | 35.634 |
19 | 100.000 | 85.787 | 67.081 | 50.319 | 37.049 |
20 | 100.000 | 86.633 | 68.419 | 51.772 | 38.399 |
21 | 100.000 | 87.310 | 69.568 | 53.107 | 39.703 |
22 | 100.000 | 88.494 | 71.164 | 54.683 | 41.088 |
23 | 100.000 | 89.340 | 72.478 | 56.104 | 42.410 |
24 | 100.000 | 90.017 | 73.627 | 57.407 | 43.656 |
25 | 100.000 | 90.186 | 74.402 | 58.522 | 44.829 |
26 | 100.000 | 90.863 | 75.504 | 59.756 | 46.008 |
27 | 100.000 | 91.709 | 76.725 | 61.037 | 47.198 |
28 | 100.000 | 92.386 | 77.757 | 62.168 | 48.287 |
29 | 100.000 | 92.555 | 78.390 | 63.096 | 49.287 |
30 | 100.000 | 93.232 | 79.399 | 64.200 | 50.356 |
31 | 100.000 | 93.570 | 80.103 | 65.101 | 51.297 |
32 | 100.000 | 93.909 | 80.784 | 65.975 | 52.226 |
33 | 100.000 | 94.585 | 81.746 | 66.994 | 53.193 |
34 | 100.000 | 95.093 | 82.543 | 67.916 | 54.128 |
35 | 100.000 | 95.431 | 83.200 | 68.752 | 55.020 |
36 | 100.000 | 95.601 | 83.717 | 69.508 | 55.872 |
37 | 100.000 | 95.939 | 84.374 | 70.334 | 56.738 |
38 | 100.000 | 96.616 | 85.312 | 71.304 | 57.642 |
39 | 100.000 | 96.954 | 85.946 | 72.103 | 58.489 |
40 | 100.000 | 97.293 | 86.579 | 72.902 | 59.339 |
41 | 100.000 | 97.293 | 86.884 | 73.497 | 60.101 |
42 | 100.000 | 97.293 | 87.189 | 74.066 | 60.811 |
43 | 100.000 | 97.800 | 87.893 | 74.838 | 61.590 |
44 | 100.000 | 97.800 | 88.198 | 75.401 | 62.287 |
45 | 100.000 | 97.800 | 88.480 | 75.937 | 62.958 |
46 | 100.000 | 97.970 | 88.878 | 76.521 | 63.646 |
47 | 100.000 | 97.970 | 89.160 | 77.047 | 64.305 |
48 | 100.000 | 98.477 | 89.887 | 77.814 | 65.026 |
49 | 100.000 | 98.816 | 90.427 | 78.446 | 65.684 |
50 | 100.000 | 98.816 | 90.662 | 78.902 | 66.276 |
51 | 100.000 | 98.985 | 91.037 | 79.433 | 66.894 |
52 | 100.000 | 98.985 | 91.272 | 79.888 | 67.485 |
53 | 100.000 | 99.154 | 91.647 | 80.409 | 68.079 |
54 | 100.000 | 99.154 | 91.811 | 80.784 | 68.621 |
55 | 100.000 | 99.154 | 91.999 | 81.181 | 69.166 |
56 | 100.000 | 99.154 | 92.210 | 81.599 | 69.712 |
57 | 100.000 | 99.323 | 92.539 | 82.065 | 70.272 |
58 | 100.000 | 99.492 | 92.891 | 82.553 | 70.838 |
59 | 100.000 | 99.492 | 93.078 | 82.939 | 71.363 |
60 | 100.000 | 99.492 | 93.290 | 83.341 | 71.879 |
61 | 100.000 | 99.492 | 93.477 | 83.717 | 72.380 |
62 | 100.000 | 99.492 | 93.665 | 84.092 | 72.881 |
63 | 100.000 | 99.492 | 93.853 | 84.462 | 73.371 |
64 | 100.000 | 99.492 | 94.017 | 84.805 | 73.847 |
65 | 100.000 | 99.662 | 94.345 | 85.245 | 74.342 |
66 | 100.000 | 99.831 | 94.650 | 85.647 | 74.804 |
67 | 100.000 | 99.831 | 94.791 | 85.947 | 75.228 |
68 | 100.000 | 99.831 | 94.932 | 86.247 | 75.651 |
69 | 100.000 | 99.831 | 95.073 | 86.548 | 76.075 |
70 | 100.000 | 99.831 | 95.214 | 86.843 | 76.485 |
71 | 100.000 | 99.831 | 95.307 | 87.089 | 76.874 |
72 | 100.000 | 99.831 | 95.425 | 87.352 | 77.260 |
73 | 100.000 | 99.831 | 95.542 | 87.609 | 77.632 |
74 | 100.000 | 99.831 | 95.636 | 87.845 | 77.996 |
75 | 100.000 | 99.831 | 95.753 | 88.103 | 78.368 |
76 | 100.000 | 99.831 | 95.894 | 88.381 | 78.741 |
77 | 100.000 | 99.831 | 96.011 | 88.633 | 79.102 |
78 | 100.000 | 99.831 | 96.105 | 88.859 | 79.446 |
79 | 100.000 | 99.831 | 96.222 | 89.105 | 79.794 |
80 | 100.000 | 99.831 | 96.340 | 89.352 | 80.141 |
81 | 100.000 | 99.831 | 96.457 | 89.598 | 80.489 |
82 | 100.000 | 99.831 | 96.527 | 89.797 | 80.818 |
83 | 100.000 | 99.831 | 96.621 | 90.017 | 81.151 |
84 | 100.000 | 99.831 | 96.715 | 90.226 | 81.460 |
85 | 100.000 | 99.831 | 96.809 | 90.429 | 81.748 |
86 | 100.000 | 99.831 | 96.879 | 90.612 | 82.044 |
87 | 100.000 | 99.831 | 96.973 | 90.810 | 82.329 |
88 | 100.000 | 99.831 | 97.067 | 91.009 | 82.608 |
89 | 100.000 | 99.831 | 97.114 | 91.164 | 82.886 |
90 | 100.000 | 100.000 | 97.372 | 91.459 | 83.191 |
91 | 100.000 | 100.000 | 97.466 | 91.657 | 83.477 |
92 | 100.000 | 100.000 | 97.560 | 91.850 | 83.745 |
93 | 100.000 | 100.000 | 97.607 | 92.000 | 84.010 |
94 | 100.000 | 100.000 | 97.677 | 92.172 | 84.280 |
95 | 100.000 | 100.000 | 97.748 | 92.338 | 84.533 |
96 | 100.000 | 100.000 | 97.818 | 92.504 | 84.790 |
97 | 100.000 | 100.000 | 97.888 | 92.671 | 85.047 |
98 | 100.000 | 100.000 | 97.959 | 92.831 | 85.288 |
99 | 100.000 | 100.000 | 98.029 | 92.992 | 85.534 |
100 | 100.000 | 100.000 | 98.099 | 93.148 | 85.763 |
101 | 100.000 | 100.000 | 98.170 | 93.298 | 85.979 |
102 | 100.000 | 100.000 | 98.217 | 93.427 | 86.197 |
103 | 100.000 | 100.000 | 98.287 | 93.577 | 86.414 |
104 | 100.000 | 100.000 | 98.358 | 93.727 | 86.630 |
105 | 100.000 | 100.000 | 98.428 | 93.877 | 86.847 |
106 | 100.000 | 100.000 | 98.498 | 94.027 | 87.063 |
107 | 100.000 | 100.000 | 98.569 | 94.177 | 87.279 |
108 | 100.000 | 100.000 | 98.639 | 94.327 | 87.496 |
109 | 100.000 | 100.000 | 98.686 | 94.451 | 87.701 |
110 | 100.000 | 100.000 | 98.733 | 94.574 | 87.906 |
111 | 100.000 | 100.000 | 98.780 | 94.697 | 88.111 |
112 | 100.000 | 100.000 | 98.827 | 94.821 | 88.319 |
113 | 100.000 | 100.000 | 98.874 | 94.944 | 88.520 |
114 | 100.000 | 100.000 | 98.921 | 95.057 | 88.705 |
115 | 100.000 | 100.000 | 98.968 | 95.169 | 88.883 |
116 | 100.000 | 100.000 | 99.015 | 95.282 | 89.060 |
117 | 100.000 | 100.000 | 99.061 | 95.389 | 89.225 |
118 | 100.000 | 100.000 | 99.061 | 95.453 | 89.387 |
119 | 100.000 | 100.000 | 99.108 | 95.561 | 89.552 |
120 | 100.000 | 100.000 | 99.155 | 95.668 | 89.717 |
121 | 100.000 | 100.000 | 99.179 | 95.754 | 89.887 |
122 | 100.000 | 100.000 | 99.226 | 95.861 | 90.055 |
123 | 100.000 | 100.000 | 99.249 | 95.947 | 90.221 |
124 | 100.000 | 100.000 | 99.249 | 96.011 | 90.377 |
125 | 100.000 | 100.000 | 99.273 | 96.091 | 90.532 |
126 | 100.000 | 100.000 | 99.320 | 96.193 | 90.682 |
127 | 100.000 | 100.000 | 99.343 | 96.274 | 90.837 |
128 | 100.000 | 100.000 | 99.366 | 96.354 | 90.992 |
129 | 100.000 | 100.000 | 99.413 | 96.456 | 91.142 |
130 | 100.000 | 100.000 | 99.437 | 96.536 | 91.296 |
131 | 100.000 | 100.000 | 99.484 | 96.638 | 91.446 |
132 | 100.000 | 100.000 | 99.484 | 96.692 | 91.581 |
133 | 100.000 | 100.000 | 99.507 | 96.767 | 91.719 |
134 | 100.000 | 100.000 | 99.531 | 96.842 | 91.860 |
135 | 100.000 | 100.000 | 99.554 | 96.917 | 92.000 |
136 | 100.000 | 100.000 | 99.578 | 96.987 | 92.126 |
137 | 100.000 | 100.000 | 99.601 | 97.056 | 92.252 |
138 | 100.000 | 100.000 | 99.625 | 97.126 | 92.378 |
139 | 100.000 | 100.000 | 99.648 | 97.190 | 92.488 |
140 | 100.000 | 100.000 | 99.672 | 97.255 | 92.599 |
141 | 100.000 | 100.000 | 99.695 | 97.319 | 92.712 |
142 | 100.000 | 100.000 | 99.718 | 97.384 | 92.825 |
143 | 100.000 | 100.000 | 99.742 | 97.448 | 92.938 |
144 | 100.000 | 100.000 | 99.765 | 97.512 | 93.052 |
145 | 100.000 | 100.000 | 99.789 | 97.577 | 93.163 |
146 | 100.000 | 100.000 | 99.789 | 97.614 | 93.263 |
147 | 100.000 | 100.000 | 99.812 | 97.673 | 93.361 |
148 | 100.000 | 100.000 | 99.836 | 97.732 | 93.459 |
149 | 100.000 | 100.000 | 99.859 | 97.791 | 93.557 |
150 | 100.000 | 100.000 | 99.883 | 97.850 | 93.655 |
151 | 100.000 | 100.000 | 99.883 | 97.888 | 93.752 |
152 | 100.000 | 100.000 | 99.883 | 97.925 | 93.851 |
153 | 100.000 | 100.000 | 99.883 | 97.963 | 93.955 |
154 | 100.000 | 100.000 | 99.906 | 98.022 | 94.057 |
155 | 100.000 | 100.000 | 99.906 | 98.054 | 94.144 |
156 | 100.000 | 100.000 | 99.906 | 98.086 | 94.230 |
157 | 100.000 | 100.000 | 99.930 | 98.140 | 94.314 |
158 | 100.000 | 100.000 | 99.953 | 98.193 | 94.397 |
159 | 100.000 | 100.000 | 99.953 | 98.225 | 94.484 |
160 | 100.000 | 100.000 | 99.977 | 98.279 | 94.567 |
161 | 100.000 | 100.000 | 99.977 | 98.311 | 94.654 |
162 | 100.000 | 100.000 | 99.977 | 98.343 | 94.745 |
163 | 100.000 | 100.000 | 99.977 | 98.375 | 94.835 |
164 | 100.000 | 100.000 | 100.000 | 98.429 | 94.918 |
165 | 100.000 | 100.000 | 100.000 | 98.456 | 94.990 |
166 | 100.000 | 100.000 | 100.000 | 98.483 | 95.063 |
167 | 100.000 | 100.000 | 100.000 | 98.509 | 95.135 |
168 | 100.000 | 100.000 | 100.000 | 98.536 | 95.211 |
169 | 100.000 | 100.000 | 100.000 | 98.563 | 95.283 |
170 | 100.000 | 100.000 | 100.000 | 98.590 | 95.355 |
171 | 100.000 | 100.000 | 100.000 | 98.617 | 95.427 |
172 | 100.000 | 100.000 | 100.000 | 98.638 | 95.490 |
173 | 100.000 | 100.000 | 100.000 | 98.660 | 95.549 |
174 | 100.000 | 100.000 | 100.000 | 98.681 | 95.608 |
175 | 100.000 | 100.000 | 100.000 | 98.702 | 95.667 |
176 | 100.000 | 100.000 | 100.000 | 98.724 | 95.728 |
177 | 100.000 | 100.000 | 100.000 | 98.745 | 95.791 |
178 | 100.000 | 100.000 | 100.000 | 98.767 | 95.850 |
179 | 100.000 | 100.000 | 100.000 | 98.788 | 95.911 |
180 | 100.000 | 100.000 | 100.000 | 98.810 | 95.971 |
181 | 100.000 | 100.000 | 100.000 | 98.831 | 96.033 |
182 | 100.000 | 100.000 | 100.000 | 98.853 | 96.093 |
183 | 100.000 | 100.000 | 100.000 | 98.874 | 96.152 |
184 | 100.000 | 100.000 | 100.000 | 98.896 | 96.211 |
185 | 100.000 | 100.000 | 100.000 | 98.917 | 96.274 |
186 | 100.000 | 100.000 | 100.000 | 98.938 | 96.335 |
187 | 100.000 | 100.000 | 100.000 | 98.960 | 96.394 |
188 | 100.000 | 100.000 | 100.000 | 98.981 | 96.453 |
189 | 100.000 | 100.000 | 100.000 | 99.003 | 96.512 |
190 | 100.000 | 100.000 | 100.000 | 99.024 | 96.572 |
191 | 100.000 | 100.000 | 100.000 | 99.046 | 96.631 |
192 | 100.000 | 100.000 | 100.000 | 99.067 | 96.690 |
193 | 100.000 | 100.000 | 100.000 | 99.089 | 96.751 |
194 | 100.000 | 100.000 | 100.000 | 99.105 | 96.797 |
195 | 100.000 | 100.000 | 100.000 | 99.121 | 96.843 |
196 | 100.000 | 100.000 | 100.000 | 99.137 | 96.893 |
197 | 100.000 | 100.000 | 100.000 | 99.153 | 96.945 |
198 | 100.000 | 100.000 | 100.000 | 99.169 | 96.991 |
199 | 100.000 | 100.000 | 100.000 | 99.185 | 97.038 |
200 | 100.000 | 100.000 | 100.000 | 99.201 | 97.086 |
201 | 100.000 | 100.000 | 100.000 | 99.217 | 97.132 |
202 | 100.000 | 100.000 | 100.000 | 99.233 | 97.178 |
203 | 100.000 | 100.000 | 100.000 | 99.249 | 97.224 |
204 | 100.000 | 100.000 | 100.000 | 99.265 | 97.271 |
205 | 100.000 | 100.000 | 100.000 | 99.282 | 97.317 |
206 | 100.000 | 100.000 | 100.000 | 99.298 | 97.365 |
207 | 100.000 | 100.000 | 100.000 | 99.314 | 97.409 |
208 | 100.000 | 100.000 | 100.000 | 99.324 | 97.441 |
209 | 100.000 | 100.000 | 100.000 | 99.335 | 97.472 |
210 | 100.000 | 100.000 | 100.000 | 99.346 | 97.504 |
211 | 100.000 | 100.000 | 100.000 | 99.357 | 97.535 |
212 | 100.000 | 100.000 | 100.000 | 99.367 | 97.566 |
213 | 100.000 | 100.000 | 100.000 | 99.378 | 97.598 |
214 | 100.000 | 100.000 | 100.000 | 99.389 | 97.629 |
215 | 100.000 | 100.000 | 100.000 | 99.399 | 97.661 |
216 | 100.000 | 100.000 | 100.000 | 99.410 | 97.694 |
217 | 100.000 | 100.000 | 100.000 | 99.421 | 97.725 |
218 | 100.000 | 100.000 | 100.000 | 99.432 | 97.761 |
219 | 100.000 | 100.000 | 100.000 | 99.442 | 97.792 |
220 | 100.000 | 100.000 | 100.000 | 99.453 | 97.823 |
221 | 100.000 | 100.000 | 100.000 | 99.464 | 97.855 |
222 | 100.000 | 100.000 | 100.000 | 99.475 | 97.886 |
223 | 100.000 | 100.000 | 100.000 | 99.485 | 97.918 |
224 | 100.000 | 100.000 | 100.000 | 99.496 | 97.949 |
225 | 100.000 | 100.000 | 100.000 | 99.507 | 97.981 |
226 | 100.000 | 100.000 | 100.000 | 99.517 | 98.012 |
227 | 100.000 | 100.000 | 100.000 | 99.528 | 98.044 |
228 | 100.000 | 100.000 | 100.000 | 99.539 | 98.075 |
229 | 100.000 | 100.000 | 100.000 | 99.550 | 98.106 |
230 | 100.000 | 100.000 | 100.000 | 99.560 | 98.138 |
231 | 100.000 | 100.000 | 100.000 | 99.571 | 98.171 |
232 | 100.000 | 100.000 | 100.000 | 99.582 | 98.204 |
233 | 100.000 | 100.000 | 100.000 | 99.593 | 98.236 |
234 | 100.000 | 100.000 | 100.000 | 99.603 | 98.267 |
235 | 100.000 | 100.000 | 100.000 | 99.614 | 98.299 |
236 | 100.000 | 100.000 | 100.000 | 99.625 | 98.334 |
237 | 100.000 | 100.000 | 100.000 | 99.635 | 98.365 |
238 | 100.000 | 100.000 | 100.000 | 99.646 | 98.397 |
239 | 100.000 | 100.000 | 100.000 | 99.657 | 98.428 |
240 | 100.000 | 100.000 | 100.000 | 99.668 | 98.463 |
241 | 100.000 | 100.000 | 100.000 | 99.678 | 98.495 |
242 | 100.000 | 100.000 | 100.000 | 99.684 | 98.513 |
243 | 100.000 | 100.000 | 100.000 | 99.689 | 98.530 |
244 | 100.000 | 100.000 | 100.000 | 99.694 | 98.558 |
245 | 100.000 | 100.000 | 100.000 | 99.700 | 98.574 |
246 | 100.000 | 100.000 | 100.000 | 99.705 | 98.591 |
247 | 100.000 | 100.000 | 100.000 | 99.710 | 98.608 |
248 | 100.000 | 100.000 | 100.000 | 99.716 | 98.624 |
249 | 100.000 | 100.000 | 100.000 | 99.721 | 98.641 |
250 | 100.000 | 100.000 | 100.000 | 99.727 | 98.657 |
251 | 100.000 | 100.000 | 100.000 | 99.732 | 98.676 |
252 | 100.000 | 100.000 | 100.000 | 99.737 | 98.694 |
253 | 100.000 | 100.000 | 100.000 | 99.743 | 98.711 |
254 | 100.000 | 100.000 | 100.000 | 99.748 | 98.730 |
255 | 100.000 | 100.000 | 100.000 | 99.753 | 98.746 |
256 | 100.000 | 100.000 | 100.000 | 99.759 | 98.763 |
257 | 100.000 | 100.000 | 100.000 | 99.764 | 98.780 |
258 | 100.000 | 100.000 | 100.000 | 99.769 | 98.796 |
259 | 100.000 | 100.000 | 100.000 | 99.775 | 98.813 |
260 | 100.000 | 100.000 | 100.000 | 99.780 | 98.831 |
261 | 100.000 | 100.000 | 100.000 | 99.786 | 98.848 |
262 | 100.000 | 100.000 | 100.000 | 99.791 | 98.865 |
263 | 100.000 | 100.000 | 100.000 | 99.796 | 98.881 |
264 | 100.000 | 100.000 | 100.000 | 99.802 | 98.898 |
265 | 100.000 | 100.000 | 100.000 | 99.807 | 98.915 |
266 | 100.000 | 100.000 | 100.000 | 99.812 | 98.931 |
267 | 100.000 | 100.000 | 100.000 | 99.818 | 98.948 |
268 | 100.000 | 100.000 | 100.000 | 99.823 | 98.964 |
269 | 100.000 | 100.000 | 100.000 | 99.828 | 98.981 |
270 | 100.000 | 100.000 | 100.000 | 99.834 | 98.998 |
271 | 100.000 | 100.000 | 100.000 | 99.839 | 99.014 |
272 | 100.000 | 100.000 | 100.000 | 99.845 | 99.031 |
273 | 100.000 | 100.000 | 100.000 | 99.850 | 99.050 |
274 | 100.000 | 100.000 | 100.000 | 99.855 | 99.066 |
275 | 100.000 | 100.000 | 100.000 | 99.861 | 99.083 |
276 | 100.000 | 100.000 | 100.000 | 99.866 | 99.099 |
277 | 100.000 | 100.000 | 100.000 | 99.871 | 99.116 |
278 | 100.000 | 100.000 | 100.000 | 99.877 | 99.133 |
279 | 100.000 | 100.000 | 100.000 | 99.882 | 99.151 |
280 | 100.000 | 100.000 | 100.000 | 99.887 | 99.168 |
281 | 100.000 | 100.000 | 100.000 | 99.893 | 99.184 |
282 | 100.000 | 100.000 | 100.000 | 99.898 | 99.201 |
283 | 100.000 | 100.000 | 100.000 | 99.903 | 99.221 |
284 | 100.000 | 100.000 | 100.000 | 99.909 | 99.240 |
285 | 100.000 | 100.000 | 100.000 | 99.914 | 99.257 |
286 | 100.000 | 100.000 | 100.000 | 99.920 | 99.275 |
287 | 100.000 | 100.000 | 100.000 | 99.925 | 99.292 |
288 | 100.000 | 100.000 | 100.000 | 99.930 | 99.308 |
289 | 100.000 | 100.000 | 100.000 | 99.936 | 99.325 |
290 | 100.000 | 100.000 | 100.000 | 99.941 | 99.342 |
291 | 100.000 | 100.000 | 100.000 | 99.946 | 99.358 |
292 | 100.000 | 100.000 | 100.000 | 99.952 | 99.377 |
293 | 100.000 | 100.000 | 100.000 | 99.957 | 99.393 |
294 | 100.000 | 100.000 | 100.000 | 99.962 | 99.410 |
295 | 100.000 | 100.000 | 100.000 | 99.968 | 99.429 |
296 | 100.000 | 100.000 | 100.000 | 99.973 | 99.447 |
297 | 100.000 | 100.000 | 100.000 | 99.979 | 99.464 |
298 | 100.000 | 100.000 | 100.000 | 99.984 | 99.480 |
299 | 100.000 | 100.000 | 100.000 | 99.989 | 99.497 |
300 | 100.000 | 100.000 | 100.000 | 99.995 | 99.514 |
301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.530 |
302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.532 |
303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.540 |
304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.543 |
305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.545 |
306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.552 |
308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.554 |
309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.556 |
310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.560 |
311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.565 |
313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.569 |
314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.573 |
315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.577 |
316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.580 |
317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.584 |
318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.588 |
319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.591 |
320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.599 |
321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.604 |
323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.606 |
324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.610 |
325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.614 |
326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.617 |
328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.619 |
329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.621 |
330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.625 |
332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.636 |
334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.639 |
336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.645 |
338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.650 |
340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.675 |
345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.689 |
351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.697 |
354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |


grep
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 59.524 | 32.487 | 18.442 | 11.077 | 7.033 |
4 | 64.286 | 37.563 | 22.548 | 14.101 | 9.181 |
5 | 71.429 | 43.993 | 27.241 | 17.345 | 11.402 |
6 | 76.190 | 49.577 | 31.699 | 20.535 | 13.614 |
7 | 85.714 | 56.853 | 36.603 | 23.806 | 15.826 |
8 | 88.095 | 60.406 | 39.934 | 26.487 | 17.843 |
9 | 90.476 | 64.129 | 43.383 | 29.221 | 19.879 |
10 | 92.857 | 66.836 | 46.011 | 31.494 | 21.702 |
11 | 92.857 | 69.205 | 48.780 | 33.902 | 23.590 |
12 | 92.857 | 71.743 | 51.783 | 36.486 | 25.575 |
13 | 92.857 | 74.112 | 54.528 | 38.867 | 27.444 |
14 | 95.238 | 76.819 | 57.062 | 41.033 | 29.192 |
15 | 95.238 | 78.680 | 59.362 | 43.134 | 30.913 |
16 | 95.238 | 79.526 | 60.770 | 44.689 | 32.358 |
17 | 95.238 | 80.372 | 62.224 | 46.292 | 33.828 |
18 | 97.619 | 83.418 | 64.969 | 48.512 | 35.547 |
19 | 97.619 | 84.772 | 66.729 | 50.212 | 37.018 |
20 | 100.000 | 86.971 | 68.724 | 51.965 | 38.497 |
21 | 100.000 | 87.986 | 70.202 | 53.498 | 39.884 |
22 | 100.000 | 89.002 | 71.633 | 54.962 | 41.204 |
23 | 100.000 | 89.679 | 72.736 | 56.206 | 42.404 |
24 | 100.000 | 90.525 | 74.003 | 57.557 | 43.660 |
25 | 100.000 | 91.032 | 74.988 | 58.748 | 44.842 |
26 | 100.000 | 91.878 | 76.255 | 60.099 | 46.097 |
27 | 100.000 | 92.217 | 77.030 | 61.117 | 47.168 |
28 | 100.000 | 92.724 | 77.921 | 62.174 | 48.228 |
29 | 100.000 | 93.232 | 78.789 | 63.198 | 49.259 |
30 | 100.000 | 93.570 | 79.493 | 64.115 | 50.236 |
31 | 100.000 | 94.078 | 80.338 | 65.112 | 51.247 |
32 | 100.000 | 94.247 | 80.878 | 65.911 | 52.148 |
33 | 100.000 | 94.247 | 81.300 | 66.656 | 53.030 |
34 | 100.000 | 94.247 | 81.628 | 67.289 | 53.831 |
35 | 100.000 | 94.755 | 82.567 | 68.393 | 54.909 |
36 | 100.000 | 94.924 | 83.153 | 69.251 | 55.848 |
37 | 100.000 | 95.431 | 83.951 | 70.168 | 56.775 |
38 | 100.000 | 95.601 | 84.444 | 70.897 | 57.605 |
39 | 100.000 | 95.939 | 85.054 | 71.664 | 58.428 |
40 | 100.000 | 96.277 | 85.687 | 72.457 | 59.265 |
41 | 100.000 | 96.447 | 86.086 | 73.052 | 59.977 |
42 | 100.000 | 96.616 | 86.532 | 73.701 | 60.725 |
43 | 100.000 | 96.616 | 86.861 | 74.296 | 61.449 |
44 | 100.000 | 97.293 | 87.823 | 75.304 | 62.393 |
45 | 100.000 | 97.293 | 88.104 | 75.830 | 63.051 |
46 | 100.000 | 97.293 | 88.339 | 76.296 | 63.663 |
47 | 100.000 | 97.462 | 88.738 | 76.870 | 64.330 |
48 | 100.000 | 97.800 | 89.301 | 77.551 | 65.052 |
49 | 100.000 | 97.970 | 89.700 | 78.119 | 65.708 |
50 | 100.000 | 98.646 | 90.591 | 79.020 | 66.546 |
51 | 100.000 | 98.816 | 90.967 | 79.556 | 67.175 |
52 | 100.000 | 98.816 | 91.178 | 79.980 | 67.739 |
53 | 100.000 | 98.816 | 91.389 | 80.403 | 68.303 |
54 | 100.000 | 98.816 | 91.553 | 80.757 | 68.800 |
55 | 100.000 | 98.816 | 91.741 | 81.138 | 69.314 |
56 | 100.000 | 98.816 | 91.882 | 81.454 | 69.775 |
57 | 100.000 | 98.816 | 92.093 | 81.872 | 70.320 |
58 | 100.000 | 98.985 | 92.398 | 82.301 | 70.840 |
59 | 100.000 | 99.154 | 92.726 | 82.757 | 71.378 |
60 | 100.000 | 99.154 | 92.891 | 83.100 | 71.851 |
61 | 100.000 | 99.154 | 93.055 | 83.443 | 72.325 |
62 | 100.000 | 99.323 | 93.407 | 83.926 | 72.879 |
63 | 100.000 | 99.323 | 93.595 | 84.296 | 73.369 |
64 | 100.000 | 99.323 | 93.759 | 84.634 | 73.832 |
65 | 100.000 | 99.492 | 94.111 | 85.111 | 74.374 |
66 | 100.000 | 99.492 | 94.181 | 85.320 | 74.725 |
67 | 100.000 | 99.492 | 94.275 | 85.556 | 75.093 |
68 | 100.000 | 99.492 | 94.416 | 85.851 | 75.503 |
69 | 100.000 | 99.492 | 94.533 | 86.113 | 75.888 |
70 | 100.000 | 99.492 | 94.627 | 86.349 | 76.252 |
71 | 100.000 | 99.492 | 94.744 | 86.612 | 76.635 |
72 | 100.000 | 99.662 | 95.073 | 87.052 | 77.131 |
73 | 100.000 | 99.662 | 95.167 | 87.277 | 77.477 |
74 | 100.000 | 99.662 | 95.260 | 87.502 | 77.821 |
75 | 100.000 | 99.662 | 95.401 | 87.786 | 78.207 |
76 | 100.000 | 99.662 | 95.542 | 88.065 | 78.582 |
77 | 100.000 | 99.662 | 95.683 | 88.349 | 78.961 |
78 | 100.000 | 99.662 | 95.777 | 88.564 | 79.281 |
79 | 100.000 | 99.662 | 95.870 | 88.773 | 79.592 |
80 | 100.000 | 99.662 | 95.988 | 89.014 | 79.929 |
81 | 100.000 | 99.662 | 96.082 | 89.223 | 80.237 |
82 | 100.000 | 99.662 | 96.199 | 89.464 | 80.572 |
83 | 100.000 | 99.831 | 96.504 | 89.850 | 80.996 |
84 | 100.000 | 99.831 | 96.598 | 90.060 | 81.304 |
85 | 100.000 | 99.831 | 96.668 | 90.236 | 81.586 |
86 | 100.000 | 99.831 | 96.739 | 90.419 | 81.874 |
87 | 100.000 | 99.831 | 96.832 | 90.622 | 82.172 |
88 | 100.000 | 99.831 | 96.926 | 90.826 | 82.468 |
89 | 100.000 | 99.831 | 96.997 | 90.992 | 82.726 |
90 | 100.000 | 99.831 | 97.067 | 91.159 | 82.984 |
91 | 100.000 | 99.831 | 97.161 | 91.357 | 83.266 |
92 | 100.000 | 99.831 | 97.231 | 91.529 | 83.529 |
93 | 100.000 | 99.831 | 97.278 | 91.657 | 83.749 |
94 | 100.000 | 99.831 | 97.349 | 91.823 | 84.002 |
95 | 100.000 | 99.831 | 97.396 | 91.952 | 84.221 |
96 | 100.000 | 99.831 | 97.443 | 92.081 | 84.439 |
97 | 100.000 | 99.831 | 97.536 | 92.274 | 84.707 |
98 | 100.000 | 99.831 | 97.630 | 92.467 | 84.975 |
99 | 100.000 | 99.831 | 97.677 | 92.596 | 85.190 |
100 | 100.000 | 99.831 | 97.677 | 92.660 | 85.352 |
101 | 100.000 | 99.831 | 97.724 | 92.783 | 85.558 |
102 | 100.000 | 99.831 | 97.771 | 92.907 | 85.763 |
103 | 100.000 | 99.831 | 97.841 | 93.062 | 85.994 |
104 | 100.000 | 99.831 | 97.888 | 93.185 | 86.199 |
105 | 100.000 | 99.831 | 97.959 | 93.341 | 86.429 |
106 | 100.000 | 99.831 | 98.029 | 93.496 | 86.658 |
107 | 100.000 | 99.831 | 98.099 | 93.652 | 86.887 |
108 | 100.000 | 99.831 | 98.146 | 93.775 | 87.089 |
109 | 100.000 | 99.831 | 98.146 | 93.839 | 87.244 |
110 | 100.000 | 99.831 | 98.170 | 93.925 | 87.411 |
111 | 100.000 | 100.000 | 98.428 | 94.220 | 87.716 |
112 | 100.000 | 100.000 | 98.475 | 94.338 | 87.906 |
113 | 100.000 | 100.000 | 98.522 | 94.456 | 88.097 |
114 | 100.000 | 100.000 | 98.592 | 94.606 | 88.313 |
115 | 100.000 | 100.000 | 98.663 | 94.756 | 88.529 |
116 | 100.000 | 100.000 | 98.710 | 94.874 | 88.720 |
117 | 100.000 | 100.000 | 98.780 | 95.024 | 88.936 |
118 | 100.000 | 100.000 | 98.780 | 95.078 | 89.075 |
119 | 100.000 | 100.000 | 98.850 | 95.228 | 89.291 |
120 | 100.000 | 100.000 | 98.850 | 95.282 | 89.430 |
121 | 100.000 | 100.000 | 98.874 | 95.362 | 89.583 |
122 | 100.000 | 100.000 | 98.921 | 95.475 | 89.763 |
123 | 100.000 | 100.000 | 98.968 | 95.587 | 89.940 |
124 | 100.000 | 100.000 | 99.015 | 95.700 | 90.118 |
125 | 100.000 | 100.000 | 99.061 | 95.813 | 90.295 |
126 | 100.000 | 100.000 | 99.085 | 95.888 | 90.438 |
127 | 100.000 | 100.000 | 99.085 | 95.931 | 90.552 |
128 | 100.000 | 100.000 | 99.132 | 96.038 | 90.719 |
129 | 100.000 | 100.000 | 99.179 | 96.145 | 90.883 |
130 | 100.000 | 100.000 | 99.226 | 96.252 | 91.048 |
131 | 100.000 | 100.000 | 99.273 | 96.359 | 91.213 |
132 | 100.000 | 100.000 | 99.273 | 96.402 | 91.323 |
133 | 100.000 | 100.000 | 99.296 | 96.472 | 91.453 |
134 | 100.000 | 100.000 | 99.320 | 96.542 | 91.581 |
135 | 100.000 | 100.000 | 99.343 | 96.611 | 91.708 |
136 | 100.000 | 100.000 | 99.366 | 96.681 | 91.834 |
137 | 100.000 | 100.000 | 99.413 | 96.783 | 91.986 |
138 | 100.000 | 100.000 | 99.437 | 96.853 | 92.111 |
139 | 100.000 | 100.000 | 99.460 | 96.922 | 92.237 |
140 | 100.000 | 100.000 | 99.484 | 96.992 | 92.363 |
141 | 100.000 | 100.000 | 99.531 | 97.094 | 92.513 |
142 | 100.000 | 100.000 | 99.554 | 97.158 | 92.629 |
143 | 100.000 | 100.000 | 99.554 | 97.190 | 92.720 |
144 | 100.000 | 100.000 | 99.601 | 97.292 | 92.869 |
145 | 100.000 | 100.000 | 99.648 | 97.394 | 93.019 |
146 | 100.000 | 100.000 | 99.648 | 97.421 | 93.102 |
147 | 100.000 | 100.000 | 99.648 | 97.453 | 93.189 |
148 | 100.000 | 100.000 | 99.672 | 97.518 | 93.302 |
149 | 100.000 | 100.000 | 99.672 | 97.550 | 93.389 |
150 | 100.000 | 100.000 | 99.672 | 97.577 | 93.465 |
151 | 100.000 | 100.000 | 99.672 | 97.603 | 93.541 |
152 | 100.000 | 100.000 | 99.672 | 97.630 | 93.615 |
153 | 100.000 | 100.000 | 99.695 | 97.689 | 93.715 |
154 | 100.000 | 100.000 | 99.718 | 97.748 | 93.814 |
155 | 100.000 | 100.000 | 99.718 | 97.775 | 93.888 |
156 | 100.000 | 100.000 | 99.742 | 97.834 | 93.986 |
157 | 100.000 | 100.000 | 99.742 | 97.861 | 94.058 |
158 | 100.000 | 100.000 | 99.765 | 97.920 | 94.156 |
159 | 100.000 | 100.000 | 99.789 | 97.979 | 94.254 |
160 | 100.000 | 100.000 | 99.812 | 98.038 | 94.352 |
161 | 100.000 | 100.000 | 99.836 | 98.097 | 94.451 |
162 | 100.000 | 100.000 | 99.859 | 98.156 | 94.549 |
163 | 100.000 | 100.000 | 99.859 | 98.182 | 94.621 |
164 | 100.000 | 100.000 | 99.859 | 98.209 | 94.693 |
165 | 100.000 | 100.000 | 99.883 | 98.268 | 94.791 |
166 | 100.000 | 100.000 | 99.883 | 98.295 | 94.863 |
167 | 100.000 | 100.000 | 99.883 | 98.322 | 94.935 |
168 | 100.000 | 100.000 | 99.883 | 98.343 | 94.998 |
169 | 100.000 | 100.000 | 99.883 | 98.365 | 95.061 |
170 | 100.000 | 100.000 | 99.883 | 98.386 | 95.124 |
171 | 100.000 | 100.000 | 99.883 | 98.408 | 95.186 |
172 | 100.000 | 100.000 | 99.883 | 98.429 | 95.249 |
173 | 100.000 | 100.000 | 99.906 | 98.483 | 95.336 |
174 | 100.000 | 100.000 | 99.906 | 98.504 | 95.397 |
175 | 100.000 | 100.000 | 99.906 | 98.526 | 95.458 |
176 | 100.000 | 100.000 | 99.906 | 98.536 | 95.505 |
177 | 100.000 | 100.000 | 99.906 | 98.558 | 95.564 |
178 | 100.000 | 100.000 | 99.906 | 98.579 | 95.623 |
179 | 100.000 | 100.000 | 99.906 | 98.601 | 95.682 |
180 | 100.000 | 100.000 | 99.906 | 98.622 | 95.741 |
181 | 100.000 | 100.000 | 99.906 | 98.644 | 95.800 |
182 | 100.000 | 100.000 | 99.906 | 98.665 | 95.860 |
183 | 100.000 | 100.000 | 99.930 | 98.719 | 95.945 |
184 | 100.000 | 100.000 | 99.930 | 98.740 | 96.004 |
185 | 100.000 | 100.000 | 99.953 | 98.794 | 96.089 |
186 | 100.000 | 100.000 | 99.953 | 98.810 | 96.141 |
187 | 100.000 | 100.000 | 99.953 | 98.831 | 96.200 |
188 | 100.000 | 100.000 | 99.953 | 98.853 | 96.259 |
189 | 100.000 | 100.000 | 99.953 | 98.874 | 96.318 |
190 | 100.000 | 100.000 | 99.953 | 98.896 | 96.377 |
191 | 100.000 | 100.000 | 99.953 | 98.917 | 96.437 |
192 | 100.000 | 100.000 | 99.977 | 98.971 | 96.520 |
193 | 100.000 | 100.000 | 100.000 | 99.024 | 96.603 |
194 | 100.000 | 100.000 | 100.000 | 99.040 | 96.653 |
195 | 100.000 | 100.000 | 100.000 | 99.056 | 96.701 |
196 | 100.000 | 100.000 | 100.000 | 99.072 | 96.749 |
197 | 100.000 | 100.000 | 100.000 | 99.089 | 96.797 |
198 | 100.000 | 100.000 | 100.000 | 99.105 | 96.845 |
199 | 100.000 | 100.000 | 100.000 | 99.121 | 96.893 |
200 | 100.000 | 100.000 | 100.000 | 99.137 | 96.940 |
201 | 100.000 | 100.000 | 100.000 | 99.153 | 96.986 |
202 | 100.000 | 100.000 | 100.000 | 99.169 | 97.032 |
203 | 100.000 | 100.000 | 100.000 | 99.185 | 97.078 |
204 | 100.000 | 100.000 | 100.000 | 99.201 | 97.124 |
205 | 100.000 | 100.000 | 100.000 | 99.217 | 97.171 |
206 | 100.000 | 100.000 | 100.000 | 99.233 | 97.217 |
207 | 100.000 | 100.000 | 100.000 | 99.249 | 97.263 |
208 | 100.000 | 100.000 | 100.000 | 99.265 | 97.309 |
209 | 100.000 | 100.000 | 100.000 | 99.282 | 97.354 |
210 | 100.000 | 100.000 | 100.000 | 99.292 | 97.389 |
211 | 100.000 | 100.000 | 100.000 | 99.303 | 97.424 |
212 | 100.000 | 100.000 | 100.000 | 99.314 | 97.459 |
213 | 100.000 | 100.000 | 100.000 | 99.324 | 97.492 |
214 | 100.000 | 100.000 | 100.000 | 99.335 | 97.526 |
215 | 100.000 | 100.000 | 100.000 | 99.346 | 97.559 |
216 | 100.000 | 100.000 | 100.000 | 99.357 | 97.592 |
217 | 100.000 | 100.000 | 100.000 | 99.367 | 97.626 |
218 | 100.000 | 100.000 | 100.000 | 99.378 | 97.659 |
219 | 100.000 | 100.000 | 100.000 | 99.389 | 97.690 |
220 | 100.000 | 100.000 | 100.000 | 99.399 | 97.722 |
221 | 100.000 | 100.000 | 100.000 | 99.410 | 97.753 |
222 | 100.000 | 100.000 | 100.000 | 99.421 | 97.785 |
223 | 100.000 | 100.000 | 100.000 | 99.432 | 97.816 |
224 | 100.000 | 100.000 | 100.000 | 99.442 | 97.848 |
225 | 100.000 | 100.000 | 100.000 | 99.453 | 97.879 |
226 | 100.000 | 100.000 | 100.000 | 99.464 | 97.910 |
227 | 100.000 | 100.000 | 100.000 | 99.475 | 97.942 |
228 | 100.000 | 100.000 | 100.000 | 99.485 | 97.973 |
229 | 100.000 | 100.000 | 100.000 | 99.496 | 98.005 |
230 | 100.000 | 100.000 | 100.000 | 99.507 | 98.036 |
231 | 100.000 | 100.000 | 100.000 | 99.517 | 98.068 |
232 | 100.000 | 100.000 | 100.000 | 99.528 | 98.099 |
233 | 100.000 | 100.000 | 100.000 | 99.539 | 98.130 |
234 | 100.000 | 100.000 | 100.000 | 99.550 | 98.162 |
235 | 100.000 | 100.000 | 100.000 | 99.560 | 98.193 |
236 | 100.000 | 100.000 | 100.000 | 99.571 | 98.225 |
237 | 100.000 | 100.000 | 100.000 | 99.582 | 98.256 |
238 | 100.000 | 100.000 | 100.000 | 99.593 | 98.288 |
239 | 100.000 | 100.000 | 100.000 | 99.603 | 98.319 |
240 | 100.000 | 100.000 | 100.000 | 99.614 | 98.351 |
241 | 100.000 | 100.000 | 100.000 | 99.625 | 98.382 |
242 | 100.000 | 100.000 | 100.000 | 99.635 | 98.413 |
243 | 100.000 | 100.000 | 100.000 | 99.646 | 98.445 |
244 | 100.000 | 100.000 | 100.000 | 99.657 | 98.476 |
245 | 100.000 | 100.000 | 100.000 | 99.662 | 98.497 |
246 | 100.000 | 100.000 | 100.000 | 99.668 | 98.517 |
247 | 100.000 | 100.000 | 100.000 | 99.673 | 98.535 |
248 | 100.000 | 100.000 | 100.000 | 99.678 | 98.554 |
249 | 100.000 | 100.000 | 100.000 | 99.684 | 98.572 |
250 | 100.000 | 100.000 | 100.000 | 99.689 | 98.591 |
251 | 100.000 | 100.000 | 100.000 | 99.694 | 98.609 |
252 | 100.000 | 100.000 | 100.000 | 99.700 | 98.628 |
253 | 100.000 | 100.000 | 100.000 | 99.705 | 98.645 |
254 | 100.000 | 100.000 | 100.000 | 99.710 | 98.661 |
255 | 100.000 | 100.000 | 100.000 | 99.710 | 98.670 |
256 | 100.000 | 100.000 | 100.000 | 99.716 | 98.687 |
257 | 100.000 | 100.000 | 100.000 | 99.721 | 98.704 |
258 | 100.000 | 100.000 | 100.000 | 99.727 | 98.720 |
259 | 100.000 | 100.000 | 100.000 | 99.732 | 98.737 |
260 | 100.000 | 100.000 | 100.000 | 99.737 | 98.754 |
261 | 100.000 | 100.000 | 100.000 | 99.743 | 98.770 |
262 | 100.000 | 100.000 | 100.000 | 99.748 | 98.787 |
263 | 100.000 | 100.000 | 100.000 | 99.753 | 98.804 |
264 | 100.000 | 100.000 | 100.000 | 99.759 | 98.820 |
265 | 100.000 | 100.000 | 100.000 | 99.764 | 98.837 |
266 | 100.000 | 100.000 | 100.000 | 99.769 | 98.853 |
267 | 100.000 | 100.000 | 100.000 | 99.775 | 98.870 |
268 | 100.000 | 100.000 | 100.000 | 99.780 | 98.887 |
269 | 100.000 | 100.000 | 100.000 | 99.786 | 98.903 |
270 | 100.000 | 100.000 | 100.000 | 99.791 | 98.920 |
271 | 100.000 | 100.000 | 100.000 | 99.796 | 98.937 |
272 | 100.000 | 100.000 | 100.000 | 99.802 | 98.953 |
273 | 100.000 | 100.000 | 100.000 | 99.807 | 98.970 |
274 | 100.000 | 100.000 | 100.000 | 99.812 | 98.987 |
275 | 100.000 | 100.000 | 100.000 | 99.818 | 99.003 |
276 | 100.000 | 100.000 | 100.000 | 99.823 | 99.020 |
277 | 100.000 | 100.000 | 100.000 | 99.828 | 99.037 |
278 | 100.000 | 100.000 | 100.000 | 99.834 | 99.053 |
279 | 100.000 | 100.000 | 100.000 | 99.839 | 99.070 |
280 | 100.000 | 100.000 | 100.000 | 99.845 | 99.086 |
281 | 100.000 | 100.000 | 100.000 | 99.850 | 99.103 |
282 | 100.000 | 100.000 | 100.000 | 99.855 | 99.120 |
283 | 100.000 | 100.000 | 100.000 | 99.861 | 99.136 |
284 | 100.000 | 100.000 | 100.000 | 99.866 | 99.153 |
285 | 100.000 | 100.000 | 100.000 | 99.871 | 99.170 |
286 | 100.000 | 100.000 | 100.000 | 99.877 | 99.186 |
287 | 100.000 | 100.000 | 100.000 | 99.882 | 99.203 |
288 | 100.000 | 100.000 | 100.000 | 99.887 | 99.220 |
289 | 100.000 | 100.000 | 100.000 | 99.893 | 99.236 |
290 | 100.000 | 100.000 | 100.000 | 99.898 | 99.253 |
291 | 100.000 | 100.000 | 100.000 | 99.903 | 99.270 |
292 | 100.000 | 100.000 | 100.000 | 99.909 | 99.286 |
293 | 100.000 | 100.000 | 100.000 | 99.914 | 99.303 |
294 | 100.000 | 100.000 | 100.000 | 99.920 | 99.319 |
295 | 100.000 | 100.000 | 100.000 | 99.925 | 99.336 |
296 | 100.000 | 100.000 | 100.000 | 99.930 | 99.353 |
297 | 100.000 | 100.000 | 100.000 | 99.936 | 99.369 |
298 | 100.000 | 100.000 | 100.000 | 99.941 | 99.386 |
299 | 100.000 | 100.000 | 100.000 | 99.946 | 99.403 |
300 | 100.000 | 100.000 | 100.000 | 99.952 | 99.419 |
301 | 100.000 | 100.000 | 100.000 | 99.957 | 99.436 |
302 | 100.000 | 100.000 | 100.000 | 99.962 | 99.453 |
303 | 100.000 | 100.000 | 100.000 | 99.968 | 99.469 |
304 | 100.000 | 100.000 | 100.000 | 99.973 | 99.486 |
305 | 100.000 | 100.000 | 100.000 | 99.979 | 99.503 |
306 | 100.000 | 100.000 | 100.000 | 99.984 | 99.519 |
307 | 100.000 | 100.000 | 100.000 | 99.989 | 99.536 |
308 | 100.000 | 100.000 | 100.000 | 99.995 | 99.552 |
309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.569 |
310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.577 |
311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.584 |
312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.591 |
313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.597 |
314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.604 |
316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.608 |
317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.612 |
318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.619 |
320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.634 |
324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.645 |
327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.649 |
328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.652 |
329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.656 |
330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.660 |
332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.663 |
334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.667 |
336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.669 |
337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.671 |
338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.675 |
340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.676 |
341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.682 |
344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.689 |
348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.693 |
350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.697 |
352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.702 |
355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |


grep
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
4 | 73.810 | 42.809 | 24.801 | 14.943 | 9.455 |
5 | 83.333 | 50.761 | 30.221 | 18.482 | 11.778 |
6 | 85.714 | 55.499 | 34.444 | 21.618 | 13.980 |
7 | 92.857 | 61.421 | 38.691 | 24.588 | 16.062 |
8 | 95.238 | 65.990 | 42.820 | 27.693 | 18.257 |
9 | 97.619 | 70.051 | 46.574 | 30.610 | 20.378 |
10 | 100.000 | 73.773 | 49.883 | 33.156 | 22.255 |
11 | 100.000 | 75.804 | 52.346 | 35.365 | 24.032 |
12 | 100.000 | 77.834 | 54.974 | 37.773 | 25.959 |
13 | 100.000 | 79.865 | 57.461 | 40.025 | 27.781 |
14 | 100.000 | 81.557 | 59.596 | 42.008 | 29.427 |
15 | 100.000 | 82.910 | 61.427 | 43.805 | 30.982 |
16 | 100.000 | 84.264 | 63.257 | 45.590 | 32.517 |
17 | 100.000 | 85.618 | 65.275 | 47.601 | 34.223 |
18 | 100.000 | 86.802 | 66.964 | 49.327 | 35.758 |
19 | 100.000 | 87.817 | 68.419 | 50.818 | 37.097 |
20 | 100.000 | 88.832 | 69.826 | 52.228 | 38.345 |
21 | 100.000 | 89.679 | 71.187 | 53.708 | 39.710 |
22 | 100.000 | 90.525 | 72.478 | 55.096 | 41.004 |
23 | 100.000 | 91.371 | 73.698 | 56.367 | 42.169 |
24 | 100.000 | 92.217 | 74.918 | 57.648 | 43.359 |
25 | 100.000 | 92.893 | 75.903 | 58.699 | 44.353 |
26 | 100.000 | 93.570 | 76.983 | 59.906 | 45.515 |
27 | 100.000 | 94.078 | 77.992 | 61.123 | 46.715 |
28 | 100.000 | 94.585 | 78.930 | 62.243 | 47.830 |
29 | 100.000 | 95.093 | 79.775 | 63.214 | 48.788 |
30 | 100.000 | 95.601 | 80.690 | 64.308 | 49.888 |
31 | 100.000 | 96.108 | 81.417 | 65.074 | 50.609 |
32 | 100.000 | 96.447 | 82.309 | 66.243 | 51.800 |
33 | 100.000 | 96.785 | 82.802 | 66.779 | 52.327 |
34 | 100.000 | 97.124 | 83.623 | 67.836 | 53.400 |
35 | 100.000 | 97.462 | 84.280 | 68.656 | 54.254 |
36 | 100.000 | 97.800 | 84.984 | 69.557 | 55.199 |
37 | 100.000 | 98.139 | 85.476 | 70.103 | 55.750 |
38 | 100.000 | 98.477 | 85.969 | 70.640 | 56.277 |
39 | 100.000 | 98.646 | 86.368 | 71.192 | 56.897 |
40 | 100.000 | 98.816 | 86.649 | 71.540 | 57.281 |
41 | 100.000 | 98.985 | 87.236 | 72.398 | 58.221 |
42 | 100.000 | 99.154 | 87.565 | 72.838 | 58.716 |
43 | 100.000 | 99.323 | 88.104 | 73.626 | 59.593 |
44 | 100.000 | 99.492 | 88.456 | 74.114 | 60.159 |
45 | 100.000 | 99.662 | 88.761 | 74.500 | 60.582 |
46 | 100.000 | 99.831 | 89.019 | 74.795 | 60.887 |
47 | 100.000 | 100.000 | 89.277 | 75.090 | 61.192 |
48 | 100.000 | 100.000 | 89.465 | 75.433 | 61.618 |
49 | 100.000 | 100.000 | 89.676 | 75.830 | 62.123 |
50 | 100.000 | 100.000 | 89.887 | 76.248 | 62.672 |
51 | 100.000 | 100.000 | 90.099 | 76.661 | 63.212 |
52 | 100.000 | 100.000 | 90.216 | 76.897 | 63.532 |
53 | 100.000 | 100.000 | 90.333 | 77.133 | 63.852 |
54 | 100.000 | 100.000 | 90.544 | 77.556 | 64.416 |
55 | 100.000 | 100.000 | 90.591 | 77.679 | 64.617 |
56 | 100.000 | 100.000 | 90.779 | 78.044 | 65.094 |
57 | 100.000 | 100.000 | 90.967 | 78.409 | 65.571 |
58 | 100.000 | 100.000 | 91.154 | 78.763 | 66.021 |
59 | 100.000 | 100.000 | 91.248 | 78.966 | 66.313 |
60 | 100.000 | 100.000 | 91.412 | 79.342 | 66.851 |
61 | 100.000 | 100.000 | 91.483 | 79.502 | 67.091 |
62 | 100.000 | 100.000 | 91.835 | 80.157 | 67.899 |
63 | 100.000 | 100.000 | 92.163 | 80.752 | 68.624 |
64 | 100.000 | 100.000 | 92.210 | 80.870 | 68.815 |
65 | 100.000 | 100.000 | 92.374 | 81.207 | 69.277 |
66 | 100.000 | 100.000 | 92.445 | 81.363 | 69.506 |
67 | 100.000 | 100.000 | 92.656 | 81.781 | 70.058 |
68 | 100.000 | 100.000 | 92.750 | 81.974 | 70.327 |
69 | 100.000 | 100.000 | 92.961 | 82.382 | 70.856 |
70 | 100.000 | 100.000 | 93.008 | 82.484 | 71.006 |
71 | 100.000 | 100.000 | 93.219 | 82.929 | 71.611 |
72 | 100.000 | 100.000 | 93.360 | 83.202 | 71.970 |
73 | 100.000 | 100.000 | 93.430 | 83.368 | 72.227 |
74 | 100.000 | 100.000 | 93.430 | 83.433 | 72.382 |
75 | 100.000 | 100.000 | 93.454 | 83.497 | 72.493 |
76 | 100.000 | 100.000 | 93.524 | 83.642 | 72.694 |
77 | 100.000 | 100.000 | 93.618 | 83.845 | 72.990 |
78 | 100.000 | 100.000 | 93.665 | 83.979 | 73.216 |
79 | 100.000 | 100.000 | 93.853 | 84.333 | 73.665 |
80 | 100.000 | 100.000 | 94.134 | 84.848 | 74.300 |
81 | 100.000 | 100.000 | 94.228 | 85.062 | 74.620 |
82 | 100.000 | 100.000 | 94.298 | 85.234 | 74.886 |
83 | 100.000 | 100.000 | 94.345 | 85.347 | 75.063 |
84 | 100.000 | 100.000 | 94.369 | 85.406 | 75.161 |
85 | 100.000 | 100.000 | 94.416 | 85.518 | 75.339 |
86 | 100.000 | 100.000 | 94.510 | 85.717 | 75.618 |
87 | 100.000 | 100.000 | 94.603 | 85.899 | 75.858 |
88 | 100.000 | 100.000 | 94.603 | 85.942 | 75.969 |
89 | 100.000 | 100.000 | 94.603 | 85.958 | 76.016 |
90 | 100.000 | 100.000 | 94.627 | 86.033 | 76.154 |
91 | 100.000 | 100.000 | 94.791 | 86.371 | 76.617 |
92 | 100.000 | 100.000 | 94.791 | 86.398 | 76.689 |
93 | 100.000 | 100.000 | 94.838 | 86.521 | 76.894 |
94 | 100.000 | 100.000 | 94.908 | 86.676 | 77.123 |
95 | 100.000 | 100.000 | 94.955 | 86.789 | 77.301 |
96 | 100.000 | 100.000 | 95.049 | 86.982 | 77.569 |
97 | 100.000 | 100.000 | 95.096 | 87.116 | 77.795 |
98 | 100.000 | 100.000 | 95.143 | 87.223 | 77.957 |
99 | 100.000 | 100.000 | 95.167 | 87.282 | 78.055 |
100 | 100.000 | 100.000 | 95.260 | 87.475 | 78.324 |
101 | 100.000 | 100.000 | 95.448 | 87.845 | 78.814 |
102 | 100.000 | 100.000 | 95.495 | 87.974 | 79.026 |
103 | 100.000 | 100.000 | 95.612 | 88.226 | 79.385 |
104 | 100.000 | 100.000 | 95.659 | 88.349 | 79.587 |
105 | 100.000 | 100.000 | 95.753 | 88.564 | 79.899 |
106 | 100.000 | 100.000 | 95.753 | 88.596 | 79.990 |
107 | 100.000 | 100.000 | 95.777 | 88.665 | 80.115 |
108 | 100.000 | 100.000 | 95.824 | 88.778 | 80.293 |
109 | 100.000 | 100.000 | 95.824 | 88.853 | 80.472 |
110 | 100.000 | 100.000 | 95.847 | 88.917 | 80.583 |
111 | 100.000 | 100.000 | 95.894 | 89.030 | 80.761 |
112 | 100.000 | 100.000 | 95.964 | 89.202 | 81.025 |
113 | 100.000 | 100.000 | 96.082 | 89.443 | 81.360 |
114 | 100.000 | 100.000 | 96.082 | 89.486 | 81.471 |
115 | 100.000 | 100.000 | 96.129 | 89.598 | 81.648 |
116 | 100.000 | 100.000 | 96.129 | 89.614 | 81.693 |
117 | 100.000 | 100.000 | 96.152 | 89.673 | 81.791 |
118 | 100.000 | 100.000 | 96.152 | 89.684 | 81.822 |
119 | 100.000 | 100.000 | 96.222 | 89.829 | 82.024 |
120 | 100.000 | 100.000 | 96.246 | 89.888 | 82.125 |
121 | 100.000 | 100.000 | 96.340 | 90.092 | 82.418 |
122 | 100.000 | 100.000 | 96.410 | 90.242 | 82.634 |
123 | 100.000 | 100.000 | 96.410 | 90.269 | 82.708 |
124 | 100.000 | 100.000 | 96.457 | 90.370 | 82.858 |
125 | 100.000 | 100.000 | 96.527 | 90.521 | 83.076 |
126 | 100.000 | 100.000 | 96.574 | 90.622 | 83.229 |
127 | 100.000 | 100.000 | 96.598 | 90.676 | 83.313 |
128 | 100.000 | 100.000 | 96.598 | 90.730 | 83.448 |
129 | 100.000 | 100.000 | 96.668 | 90.891 | 83.692 |
130 | 100.000 | 100.000 | 96.715 | 90.998 | 83.855 |
131 | 100.000 | 100.000 | 96.786 | 91.143 | 84.058 |
132 | 100.000 | 100.000 | 96.809 | 91.202 | 84.158 |
133 | 100.000 | 100.000 | 96.832 | 91.261 | 84.256 |
134 | 100.000 | 100.000 | 96.832 | 91.271 | 84.287 |
135 | 100.000 | 100.000 | 96.856 | 91.325 | 84.370 |
136 | 100.000 | 100.000 | 96.856 | 91.346 | 84.430 |
137 | 100.000 | 100.000 | 96.856 | 91.357 | 84.461 |
138 | 100.000 | 100.000 | 96.879 | 91.421 | 84.572 |
139 | 100.000 | 100.000 | 96.903 | 91.502 | 84.722 |
140 | 100.000 | 100.000 | 96.950 | 91.614 | 84.905 |
141 | 100.000 | 100.000 | 96.950 | 91.625 | 84.936 |
142 | 100.000 | 100.000 | 96.997 | 91.727 | 85.086 |
143 | 100.000 | 100.000 | 97.020 | 91.791 | 85.197 |
144 | 100.000 | 100.000 | 97.044 | 91.845 | 85.280 |
145 | 100.000 | 100.000 | 97.067 | 91.899 | 85.363 |
146 | 100.000 | 100.000 | 97.114 | 92.011 | 85.541 |
147 | 100.000 | 100.000 | 97.231 | 92.242 | 85.848 |
148 | 100.000 | 100.000 | 97.278 | 92.365 | 86.050 |
149 | 100.000 | 100.000 | 97.278 | 92.381 | 86.094 |
150 | 100.000 | 100.000 | 97.325 | 92.499 | 86.286 |
151 | 100.000 | 100.000 | 97.372 | 92.606 | 86.449 |
152 | 100.000 | 100.000 | 97.443 | 92.767 | 86.689 |
153 | 100.000 | 100.000 | 97.443 | 92.815 | 86.813 |
154 | 100.000 | 100.000 | 97.466 | 92.869 | 86.898 |
155 | 100.000 | 100.000 | 97.466 | 92.885 | 86.945 |
156 | 100.000 | 100.000 | 97.466 | 92.896 | 86.976 |
157 | 100.000 | 100.000 | 97.489 | 92.949 | 87.059 |
158 | 100.000 | 100.000 | 97.489 | 92.960 | 87.091 |
159 | 100.000 | 100.000 | 97.513 | 93.041 | 87.242 |
160 | 100.000 | 100.000 | 97.536 | 93.100 | 87.340 |
161 | 100.000 | 100.000 | 97.536 | 93.121 | 87.399 |
162 | 100.000 | 100.000 | 97.536 | 93.142 | 87.459 |
163 | 100.000 | 100.000 | 97.607 | 93.293 | 87.675 |
164 | 100.000 | 100.000 | 97.607 | 93.298 | 87.692 |
165 | 100.000 | 100.000 | 97.630 | 93.352 | 87.775 |
166 | 100.000 | 100.000 | 97.724 | 93.550 | 88.056 |
167 | 100.000 | 100.000 | 97.724 | 93.604 | 88.191 |
168 | 100.000 | 100.000 | 97.724 | 93.614 | 88.222 |
169 | 100.000 | 100.000 | 97.748 | 93.689 | 88.361 |
170 | 100.000 | 100.000 | 97.771 | 93.743 | 88.444 |
171 | 100.000 | 100.000 | 97.818 | 93.845 | 88.594 |
172 | 100.000 | 100.000 | 97.818 | 93.877 | 88.681 |
173 | 100.000 | 100.000 | 97.818 | 93.898 | 88.740 |
174 | 100.000 | 100.000 | 97.841 | 93.952 | 88.823 |
175 | 100.000 | 100.000 | 97.888 | 94.059 | 88.988 |
176 | 100.000 | 100.000 | 97.888 | 94.070 | 89.019 |
177 | 100.000 | 100.000 | 97.888 | 94.070 | 89.023 |
178 | 100.000 | 100.000 | 97.888 | 94.081 | 89.054 |
179 | 100.000 | 100.000 | 97.888 | 94.102 | 89.117 |
180 | 100.000 | 100.000 | 97.888 | 94.129 | 89.189 |
181 | 100.000 | 100.000 | 97.912 | 94.188 | 89.289 |
182 | 100.000 | 100.000 | 97.912 | 94.188 | 89.291 |
183 | 100.000 | 100.000 | 97.912 | 94.204 | 89.339 |
184 | 100.000 | 100.000 | 97.912 | 94.226 | 89.400 |
185 | 100.000 | 100.000 | 97.912 | 94.236 | 89.432 |
186 | 100.000 | 100.000 | 97.912 | 94.236 | 89.434 |
187 | 100.000 | 100.000 | 97.912 | 94.242 | 89.450 |
188 | 100.000 | 100.000 | 97.912 | 94.258 | 89.496 |
189 | 100.000 | 100.000 | 97.912 | 94.258 | 89.504 |
190 | 100.000 | 100.000 | 97.935 | 94.311 | 89.591 |
191 | 100.000 | 100.000 | 97.935 | 94.322 | 89.622 |
192 | 100.000 | 100.000 | 97.935 | 94.333 | 89.654 |
193 | 100.000 | 100.000 | 98.006 | 94.483 | 89.870 |
194 | 100.000 | 100.000 | 98.099 | 94.676 | 90.138 |
195 | 100.000 | 100.000 | 98.099 | 94.687 | 90.170 |
196 | 100.000 | 100.000 | 98.099 | 94.719 | 90.260 |
197 | 100.000 | 100.000 | 98.123 | 94.778 | 90.358 |
198 | 100.000 | 100.000 | 98.170 | 94.880 | 90.508 |
199 | 100.000 | 100.000 | 98.193 | 94.933 | 90.591 |
200 | 100.000 | 100.000 | 98.264 | 95.078 | 90.793 |
201 | 100.000 | 100.000 | 98.358 | 95.271 | 91.061 |
202 | 100.000 | 100.000 | 98.358 | 95.287 | 91.109 |
203 | 100.000 | 100.000 | 98.381 | 95.351 | 91.220 |
204 | 100.000 | 100.000 | 98.428 | 95.480 | 91.434 |
205 | 100.000 | 100.000 | 98.451 | 95.534 | 91.518 |
206 | 100.000 | 100.000 | 98.451 | 95.555 | 91.577 |
207 | 100.000 | 100.000 | 98.451 | 95.566 | 91.612 |
208 | 100.000 | 100.000 | 98.451 | 95.587 | 91.671 |
209 | 100.000 | 100.000 | 98.451 | 95.587 | 91.679 |
210 | 100.000 | 100.000 | 98.451 | 95.587 | 91.680 |
211 | 100.000 | 100.000 | 98.451 | 95.603 | 91.727 |
212 | 100.000 | 100.000 | 98.522 | 95.754 | 91.943 |
213 | 100.000 | 100.000 | 98.522 | 95.759 | 91.961 |
214 | 100.000 | 100.000 | 98.592 | 95.904 | 92.163 |
215 | 100.000 | 100.000 | 98.592 | 95.925 | 92.222 |
216 | 100.000 | 100.000 | 98.592 | 95.936 | 92.254 |
217 | 100.000 | 100.000 | 98.592 | 95.941 | 92.272 |
218 | 100.000 | 100.000 | 98.592 | 95.947 | 92.289 |
219 | 100.000 | 100.000 | 98.592 | 95.952 | 92.307 |
220 | 100.000 | 100.000 | 98.592 | 95.952 | 92.315 |
221 | 100.000 | 100.000 | 98.592 | 95.963 | 92.350 |
222 | 100.000 | 100.000 | 98.616 | 96.016 | 92.433 |
223 | 100.000 | 100.000 | 98.616 | 96.022 | 92.450 |
224 | 100.000 | 100.000 | 98.663 | 96.124 | 92.599 |
225 | 100.000 | 100.000 | 98.663 | 96.140 | 92.646 |
226 | 100.000 | 100.000 | 98.663 | 96.156 | 92.692 |
227 | 100.000 | 100.000 | 98.710 | 96.258 | 92.842 |
228 | 100.000 | 100.000 | 98.710 | 96.258 | 92.849 |
229 | 100.000 | 100.000 | 98.710 | 96.263 | 92.866 |
230 | 100.000 | 100.000 | 98.710 | 96.274 | 92.897 |
231 | 100.000 | 100.000 | 98.710 | 96.295 | 92.956 |
232 | 100.000 | 100.000 | 98.733 | 96.359 | 93.069 |
233 | 100.000 | 100.000 | 98.733 | 96.365 | 93.086 |
234 | 100.000 | 100.000 | 98.733 | 96.376 | 93.117 |
235 | 100.000 | 100.000 | 98.733 | 96.381 | 93.134 |
236 | 100.000 | 100.000 | 98.756 | 96.435 | 93.219 |
237 | 100.000 | 100.000 | 98.756 | 96.445 | 93.250 |
238 | 100.000 | 100.000 | 98.756 | 96.461 | 93.295 |
239 | 100.000 | 100.000 | 98.780 | 96.520 | 93.393 |
240 | 100.000 | 100.000 | 98.780 | 96.526 | 93.411 |
241 | 100.000 | 100.000 | 98.803 | 96.585 | 93.509 |
242 | 100.000 | 100.000 | 98.803 | 96.611 | 93.581 |
243 | 100.000 | 100.000 | 98.827 | 96.670 | 93.679 |
244 | 100.000 | 100.000 | 98.827 | 96.670 | 93.683 |
245 | 100.000 | 100.000 | 98.827 | 96.692 | 93.742 |
246 | 100.000 | 100.000 | 98.827 | 96.703 | 93.774 |
247 | 100.000 | 100.000 | 98.827 | 96.713 | 93.805 |
248 | 100.000 | 100.000 | 98.827 | 96.724 | 93.837 |
249 | 100.000 | 100.000 | 98.827 | 96.729 | 93.853 |
250 | 100.000 | 100.000 | 98.827 | 96.745 | 93.898 |
251 | 100.000 | 100.000 | 98.850 | 96.799 | 93.981 |
252 | 100.000 | 100.000 | 98.850 | 96.831 | 94.071 |
253 | 100.000 | 100.000 | 98.850 | 96.837 | 94.092 |
254 | 100.000 | 100.000 | 98.874 | 96.890 | 94.175 |
255 | 100.000 | 100.000 | 98.874 | 96.912 | 94.236 |
256 | 100.000 | 100.000 | 98.874 | 96.922 | 94.267 |
257 | 100.000 | 100.000 | 98.874 | 96.928 | 94.284 |
258 | 100.000 | 100.000 | 98.874 | 96.939 | 94.327 |
259 | 100.000 | 100.000 | 98.874 | 96.949 | 94.358 |
260 | 100.000 | 100.000 | 98.874 | 96.960 | 94.389 |
261 | 100.000 | 100.000 | 98.874 | 96.965 | 94.408 |
262 | 100.000 | 100.000 | 98.874 | 96.971 | 94.425 |
263 | 100.000 | 100.000 | 98.874 | 96.976 | 94.441 |
264 | 100.000 | 100.000 | 98.874 | 96.987 | 94.473 |
265 | 100.000 | 100.000 | 98.874 | 96.997 | 94.504 |
266 | 100.000 | 100.000 | 98.897 | 97.062 | 94.615 |
267 | 100.000 | 100.000 | 98.897 | 97.083 | 94.674 |
268 | 100.000 | 100.000 | 98.897 | 97.083 | 94.678 |
269 | 100.000 | 100.000 | 98.897 | 97.094 | 94.709 |
270 | 100.000 | 100.000 | 98.897 | 97.094 | 94.711 |
271 | 100.000 | 100.000 | 98.897 | 97.099 | 94.728 |
272 | 100.000 | 100.000 | 98.897 | 97.105 | 94.745 |
273 | 100.000 | 100.000 | 98.944 | 97.207 | 94.894 |
274 | 100.000 | 100.000 | 98.944 | 97.212 | 94.911 |
275 | 100.000 | 100.000 | 98.944 | 97.217 | 94.929 |
276 | 100.000 | 100.000 | 98.944 | 97.217 | 94.933 |
277 | 100.000 | 100.000 | 98.968 | 97.271 | 95.016 |
278 | 100.000 | 100.000 | 98.968 | 97.271 | 95.018 |
279 | 100.000 | 100.000 | 98.968 | 97.271 | 95.026 |
280 | 100.000 | 100.000 | 98.991 | 97.325 | 95.109 |
281 | 100.000 | 100.000 | 98.991 | 97.335 | 95.140 |
282 | 100.000 | 100.000 | 98.991 | 97.351 | 95.185 |
283 | 100.000 | 100.000 | 98.991 | 97.351 | 95.192 |
284 | 100.000 | 100.000 | 98.991 | 97.351 | 95.198 |
285 | 100.000 | 100.000 | 98.991 | 97.357 | 95.214 |
286 | 100.000 | 100.000 | 98.991 | 97.357 | 95.222 |
287 | 100.000 | 100.000 | 98.991 | 97.362 | 95.238 |
288 | 100.000 | 100.000 | 98.991 | 97.362 | 95.240 |
289 | 100.000 | 100.000 | 99.015 | 97.416 | 95.323 |
290 | 100.000 | 100.000 | 99.015 | 97.426 | 95.355 |
291 | 100.000 | 100.000 | 99.015 | 97.432 | 95.371 |
292 | 100.000 | 100.000 | 99.015 | 97.442 | 95.405 |
293 | 100.000 | 100.000 | 99.015 | 97.448 | 95.421 |
294 | 100.000 | 100.000 | 99.038 | 97.507 | 95.519 |
295 | 100.000 | 100.000 | 99.273 | 97.957 | 96.091 |
296 | 100.000 | 100.000 | 99.273 | 97.968 | 96.122 |
297 | 100.000 | 100.000 | 99.296 | 98.022 | 96.205 |
298 | 100.000 | 100.000 | 99.296 | 98.032 | 96.239 |
299 | 100.000 | 100.000 | 99.296 | 98.038 | 96.259 |
300 | 100.000 | 100.000 | 99.296 | 98.048 | 96.290 |
301 | 100.000 | 100.000 | 99.296 | 98.054 | 96.307 |
302 | 100.000 | 100.000 | 99.296 | 98.054 | 96.311 |
303 | 100.000 | 100.000 | 99.296 | 98.054 | 96.313 |
304 | 100.000 | 100.000 | 99.296 | 98.064 | 96.344 |
305 | 100.000 | 100.000 | 99.296 | 98.081 | 96.388 |
306 | 100.000 | 100.000 | 99.296 | 98.091 | 96.420 |
307 | 100.000 | 100.000 | 99.296 | 98.091 | 96.422 |
308 | 100.000 | 100.000 | 99.296 | 98.097 | 96.438 |
309 | 100.000 | 100.000 | 99.320 | 98.150 | 96.522 |
310 | 100.000 | 100.000 | 99.320 | 98.156 | 96.538 |
311 | 100.000 | 100.000 | 99.320 | 98.161 | 96.555 |
312 | 100.000 | 100.000 | 99.320 | 98.161 | 96.557 |
313 | 100.000 | 100.000 | 99.320 | 98.166 | 96.573 |
314 | 100.000 | 100.000 | 99.320 | 98.166 | 96.577 |
315 | 100.000 | 100.000 | 99.320 | 98.177 | 96.609 |
316 | 100.000 | 100.000 | 99.320 | 98.188 | 96.644 |
317 | 100.000 | 100.000 | 99.320 | 98.188 | 96.646 |
318 | 100.000 | 100.000 | 99.320 | 98.188 | 96.653 |
319 | 100.000 | 100.000 | 99.320 | 98.193 | 96.671 |
320 | 100.000 | 100.000 | 99.320 | 98.193 | 96.673 |
321 | 100.000 | 100.000 | 99.320 | 98.209 | 96.718 |
322 | 100.000 | 100.000 | 99.531 | 98.627 | 97.263 |
323 | 100.000 | 100.000 | 99.531 | 98.627 | 97.271 |
324 | 100.000 | 100.000 | 99.554 | 98.681 | 97.354 |
325 | 100.000 | 100.000 | 99.554 | 98.692 | 97.385 |
326 | 100.000 | 100.000 | 99.578 | 98.745 | 97.470 |
327 | 100.000 | 100.000 | 99.578 | 98.745 | 97.476 |
328 | 100.000 | 100.000 | 99.578 | 98.756 | 97.507 |
329 | 100.000 | 100.000 | 99.578 | 98.756 | 97.511 |
330 | 100.000 | 100.000 | 99.578 | 98.767 | 97.542 |
331 | 100.000 | 100.000 | 99.601 | 98.820 | 97.626 |
332 | 100.000 | 100.000 | 99.601 | 98.820 | 97.627 |
333 | 100.000 | 100.000 | 99.601 | 98.826 | 97.644 |
334 | 100.000 | 100.000 | 99.601 | 98.826 | 97.646 |
335 | 100.000 | 100.000 | 99.601 | 98.842 | 97.692 |
336 | 100.000 | 100.000 | 99.601 | 98.847 | 97.709 |
337 | 100.000 | 100.000 | 99.601 | 98.847 | 97.714 |
338 | 100.000 | 100.000 | 99.601 | 98.858 | 97.750 |
339 | 100.000 | 100.000 | 99.601 | 98.863 | 97.766 |
340 | 100.000 | 100.000 | 99.601 | 98.863 | 97.774 |
341 | 100.000 | 100.000 | 99.601 | 98.869 | 97.790 |
342 | 100.000 | 100.000 | 99.601 | 98.869 | 97.794 |
343 | 100.000 | 100.000 | 99.601 | 98.874 | 97.811 |
344 | 100.000 | 100.000 | 99.601 | 98.879 | 97.827 |
345 | 100.000 | 100.000 | 99.601 | 98.879 | 97.831 |
346 | 100.000 | 100.000 | 99.601 | 98.890 | 97.862 |
347 | 100.000 | 100.000 | 99.601 | 98.901 | 97.894 |
348 | 100.000 | 100.000 | 99.601 | 98.906 | 97.910 |
349 | 100.000 | 100.000 | 99.601 | 98.912 | 97.927 |
350 | 100.000 | 100.000 | 99.601 | 98.917 | 97.944 |
351 | 100.000 | 100.000 | 99.601 | 98.917 | 97.947 |
352 | 100.000 | 100.000 | 99.601 | 98.928 | 97.979 |
353 | 100.000 | 100.000 | 99.601 | 98.933 | 97.995 |
354 | 100.000 | 100.000 | 99.601 | 98.944 | 98.027 |
355 | 100.000 | 100.000 | 99.601 | 98.949 | 98.044 |
356 | 100.000 | 100.000 | 99.601 | 98.954 | 98.060 |
357 | 100.000 | 100.000 | 99.648 | 99.056 | 98.210 |
358 | 100.000 | 100.000 | 99.789 | 99.341 | 98.589 |
359 | 100.000 | 100.000 | 99.789 | 99.346 | 98.606 |
360 | 100.000 | 100.000 | 99.789 | 99.346 | 98.609 |
361 | 100.000 | 100.000 | 99.789 | 99.351 | 98.626 |
362 | 100.000 | 100.000 | 99.789 | 99.351 | 98.630 |
363 | 100.000 | 100.000 | 99.789 | 99.351 | 98.633 |
364 | 100.000 | 100.000 | 99.789 | 99.357 | 98.654 |
365 | 100.000 | 100.000 | 99.789 | 99.362 | 98.672 |
366 | 100.000 | 100.000 | 99.789 | 99.362 | 98.674 |
367 | 100.000 | 100.000 | 99.789 | 99.362 | 98.676 |
368 | 100.000 | 100.000 | 99.789 | 99.362 | 98.680 |
369 | 100.000 | 100.000 | 99.789 | 99.373 | 98.715 |
370 | 100.000 | 100.000 | 99.789 | 99.378 | 98.731 |
371 | 100.000 | 100.000 | 99.789 | 99.383 | 98.748 |
372 | 100.000 | 100.000 | 99.789 | 99.389 | 98.765 |
373 | 100.000 | 100.000 | 99.789 | 99.389 | 98.768 |
374 | 100.000 | 100.000 | 99.789 | 99.394 | 98.785 |
375 | 100.000 | 100.000 | 99.812 | 99.448 | 98.868 |
376 | 100.000 | 100.000 | 99.812 | 99.448 | 98.872 |
377 | 100.000 | 100.000 | 99.812 | 99.448 | 98.874 |
378 | 100.000 | 100.000 | 99.812 | 99.485 | 98.970 |
379 | 100.000 | 100.000 | 99.812 | 99.485 | 98.974 |
380 | 100.000 | 100.000 | 99.812 | 99.485 | 98.976 |
381 | 100.000 | 100.000 | 99.812 | 99.496 | 99.007 |
382 | 100.000 | 100.000 | 99.812 | 99.496 | 99.011 |
383 | 100.000 | 100.000 | 99.812 | 99.496 | 99.013 |
384 | 100.000 | 100.000 | 99.812 | 99.501 | 99.029 |
385 | 100.000 | 100.000 | 99.812 | 99.501 | 99.033 |
386 | 100.000 | 100.000 | 99.812 | 99.501 | 99.037 |
387 | 100.000 | 100.000 | 99.812 | 99.501 | 99.040 |
388 | 100.000 | 100.000 | 99.812 | 99.501 | 99.044 |
389 | 100.000 | 100.000 | 99.812 | 99.507 | 99.061 |
390 | 100.000 | 100.000 | 99.812 | 99.512 | 99.077 |
391 | 100.000 | 100.000 | 99.883 | 99.662 | 99.294 |
392 | 100.000 | 100.000 | 99.883 | 99.668 | 99.310 |
393 | 100.000 | 100.000 | 99.883 | 99.668 | 99.314 |
394 | 100.000 | 100.000 | 99.930 | 99.769 | 99.464 |
395 | 100.000 | 100.000 | 99.930 | 99.769 | 99.466 |
396 | 100.000 | 100.000 | 99.930 | 99.769 | 99.467 |
397 | 100.000 | 100.000 | 99.930 | 99.780 | 99.499 |
398 | 100.000 | 100.000 | 99.930 | 99.780 | 99.501 |
399 | 100.000 | 100.000 | 99.930 | 99.780 | 99.504 |
400 | 100.000 | 100.000 | 99.930 | 99.780 | 99.508 |
401 | 100.000 | 100.000 | 99.930 | 99.780 | 99.510 |
402 | 100.000 | 100.000 | 99.930 | 99.786 | 99.527 |
403 | 100.000 | 100.000 | 99.930 | 99.791 | 99.543 |
404 | 100.000 | 100.000 | 99.930 | 99.791 | 99.545 |
405 | 100.000 | 100.000 | 99.930 | 99.796 | 99.562 |
406 | 100.000 | 100.000 | 99.930 | 99.796 | 99.565 |
407 | 100.000 | 100.000 | 99.930 | 99.796 | 99.567 |
408 | 100.000 | 100.000 | 99.930 | 99.796 | 99.571 |
409 | 100.000 | 100.000 | 99.930 | 99.802 | 99.588 |
410 | 100.000 | 100.000 | 99.930 | 99.802 | 99.589 |
411 | 100.000 | 100.000 | 99.930 | 99.802 | 99.593 |
412 | 100.000 | 100.000 | 99.930 | 99.802 | 99.597 |
413 | 100.000 | 100.000 | 99.930 | 99.802 | 99.601 |
414 | 100.000 | 100.000 | 99.930 | 99.812 | 99.632 |
415 | 100.000 | 100.000 | 99.930 | 99.818 | 99.649 |
416 | 100.000 | 100.000 | 99.930 | 99.818 | 99.650 |
417 | 100.000 | 100.000 | 99.930 | 99.818 | 99.652 |
418 | 100.000 | 100.000 | 99.930 | 99.823 | 99.669 |
419 | 100.000 | 100.000 | 99.930 | 99.834 | 99.700 |
420 | 100.000 | 100.000 | 99.930 | 99.839 | 99.717 |
421 | 100.000 | 100.000 | 99.930 | 99.845 | 99.734 |
422 | 100.000 | 100.000 | 99.930 | 99.845 | 99.736 |
423 | 100.000 | 100.000 | 99.930 | 99.845 | 99.739 |
424 | 100.000 | 100.000 | 99.930 | 99.845 | 99.741 |
425 | 100.000 | 100.000 | 99.930 | 99.845 | 99.745 |
426 | 100.000 | 100.000 | 99.930 | 99.845 | 99.747 |
427 | 100.000 | 100.000 | 99.930 | 99.845 | 99.750 |
428 | 100.000 | 100.000 | 99.930 | 99.845 | 99.752 |
429 | 100.000 | 100.000 | 99.930 | 99.845 | 99.756 |
430 | 100.000 | 100.000 | 99.930 | 99.845 | 99.758 |
431 | 100.000 | 100.000 | 99.930 | 99.845 | 99.760 |
432 | 100.000 | 100.000 | 99.930 | 99.845 | 99.761 |
433 | 100.000 | 100.000 | 99.930 | 99.845 | 99.765 |
434 | 100.000 | 100.000 | 99.930 | 99.845 | 99.767 |
435 | 100.000 | 100.000 | 100.000 | 99.995 | 99.983 |
436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


grep
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
4 | 71.429 | 41.794 | 24.472 | 14.857 | 9.438 |
5 | 78.571 | 48.393 | 29.329 | 18.203 | 11.707 |
6 | 80.952 | 52.623 | 33.247 | 21.221 | 13.877 |
7 | 85.714 | 58.037 | 37.588 | 24.358 | 16.070 |
8 | 90.476 | 63.283 | 41.741 | 27.355 | 18.180 |
9 | 95.238 | 68.020 | 45.519 | 30.165 | 20.216 |
10 | 95.238 | 70.728 | 48.592 | 32.760 | 22.196 |
11 | 97.619 | 74.112 | 51.760 | 35.328 | 24.149 |
12 | 100.000 | 76.481 | 54.223 | 37.564 | 25.976 |
13 | 100.000 | 78.511 | 56.710 | 39.805 | 27.781 |
14 | 100.000 | 79.865 | 58.775 | 41.874 | 29.534 |
15 | 100.000 | 81.557 | 60.934 | 43.906 | 31.235 |
16 | 100.000 | 83.249 | 63.069 | 45.906 | 32.909 |
17 | 100.000 | 84.772 | 65.063 | 47.820 | 34.538 |
18 | 100.000 | 86.125 | 66.823 | 49.509 | 35.988 |
19 | 100.000 | 87.140 | 68.325 | 51.096 | 37.439 |
20 | 100.000 | 88.156 | 69.803 | 52.608 | 38.785 |
21 | 100.000 | 88.832 | 71.023 | 54.013 | 40.126 |
22 | 100.000 | 89.679 | 72.313 | 55.418 | 41.450 |
23 | 100.000 | 90.694 | 73.698 | 56.833 | 42.741 |
24 | 100.000 | 91.540 | 74.965 | 58.185 | 43.997 |
25 | 100.000 | 92.047 | 75.997 | 59.455 | 45.265 |
26 | 100.000 | 92.724 | 77.053 | 60.629 | 46.399 |
27 | 100.000 | 93.570 | 78.203 | 61.793 | 47.458 |
28 | 100.000 | 94.078 | 79.141 | 62.924 | 48.594 |
29 | 100.000 | 94.585 | 79.986 | 63.895 | 49.552 |
30 | 100.000 | 94.755 | 80.643 | 64.854 | 50.593 |
31 | 100.000 | 94.924 | 81.276 | 65.777 | 51.586 |
32 | 100.000 | 95.093 | 81.910 | 66.726 | 52.636 |
33 | 100.000 | 95.601 | 82.708 | 67.626 | 53.527 |
34 | 100.000 | 96.108 | 83.505 | 68.533 | 54.430 |
35 | 100.000 | 96.616 | 84.280 | 69.406 | 55.303 |
36 | 100.000 | 96.954 | 84.960 | 70.270 | 56.207 |
37 | 100.000 | 97.462 | 85.687 | 71.085 | 57.026 |
38 | 100.000 | 97.970 | 86.415 | 71.851 | 57.747 |
39 | 100.000 | 98.308 | 87.048 | 72.640 | 58.574 |
40 | 100.000 | 98.308 | 87.471 | 73.374 | 59.432 |
41 | 100.000 | 98.646 | 88.081 | 74.157 | 60.286 |
42 | 100.000 | 98.816 | 88.550 | 74.865 | 61.113 |
43 | 100.000 | 98.985 | 89.019 | 75.540 | 61.875 |
44 | 100.000 | 99.154 | 89.489 | 76.221 | 62.648 |
45 | 100.000 | 99.154 | 89.840 | 76.848 | 63.400 |
46 | 100.000 | 99.154 | 90.169 | 77.444 | 64.133 |
47 | 100.000 | 99.323 | 90.568 | 78.001 | 64.765 |
48 | 100.000 | 99.323 | 90.873 | 78.564 | 65.462 |
49 | 100.000 | 99.323 | 91.178 | 79.127 | 66.159 |
50 | 100.000 | 99.662 | 91.671 | 79.663 | 66.686 |
51 | 100.000 | 99.662 | 91.952 | 80.199 | 67.358 |
52 | 100.000 | 99.662 | 92.187 | 80.666 | 67.972 |
53 | 100.000 | 99.831 | 92.515 | 81.106 | 68.467 |
54 | 100.000 | 99.831 | 92.726 | 81.524 | 69.020 |
55 | 100.000 | 99.831 | 92.938 | 81.947 | 69.584 |
56 | 100.000 | 99.831 | 93.149 | 82.366 | 70.130 |
57 | 100.000 | 100.000 | 93.454 | 82.752 | 70.553 |
58 | 100.000 | 100.000 | 93.641 | 83.138 | 71.078 |
59 | 100.000 | 100.000 | 93.829 | 83.513 | 71.579 |
60 | 100.000 | 100.000 | 93.993 | 83.840 | 72.018 |
61 | 100.000 | 100.000 | 94.158 | 84.178 | 72.480 |
62 | 100.000 | 100.000 | 94.322 | 84.521 | 72.955 |
63 | 100.000 | 100.000 | 94.486 | 84.843 | 73.381 |
64 | 100.000 | 100.000 | 94.650 | 85.170 | 73.819 |
65 | 100.000 | 100.000 | 94.815 | 85.502 | 74.268 |
66 | 100.000 | 100.000 | 94.955 | 85.786 | 74.647 |
67 | 100.000 | 100.000 | 95.096 | 86.092 | 75.082 |
68 | 100.000 | 100.000 | 95.237 | 86.381 | 75.481 |
69 | 100.000 | 100.000 | 95.378 | 86.666 | 75.864 |
70 | 100.000 | 100.000 | 95.519 | 86.971 | 76.299 |
71 | 100.000 | 100.000 | 95.636 | 87.218 | 76.643 |
72 | 100.000 | 100.000 | 95.753 | 87.481 | 77.027 |
73 | 100.000 | 100.000 | 95.870 | 87.738 | 77.399 |
74 | 100.000 | 100.000 | 95.988 | 87.990 | 77.759 |
75 | 100.000 | 100.000 | 96.105 | 88.247 | 78.131 |
76 | 100.000 | 100.000 | 96.222 | 88.489 | 78.462 |
77 | 100.000 | 100.000 | 96.340 | 88.730 | 78.793 |
78 | 100.000 | 100.000 | 96.457 | 88.982 | 79.154 |
79 | 100.000 | 100.000 | 96.551 | 89.191 | 79.459 |
80 | 100.000 | 100.000 | 96.645 | 89.373 | 79.699 |
81 | 100.000 | 100.000 | 96.739 | 89.620 | 80.091 |
82 | 100.000 | 100.000 | 96.832 | 89.824 | 80.389 |
83 | 100.000 | 100.000 | 96.926 | 90.017 | 80.657 |
84 | 100.000 | 100.000 | 97.020 | 90.215 | 80.936 |
85 | 100.000 | 100.000 | 97.114 | 90.440 | 81.280 |
86 | 100.000 | 100.000 | 97.208 | 90.655 | 81.598 |
87 | 100.000 | 100.000 | 97.302 | 90.864 | 81.905 |
88 | 100.000 | 100.000 | 97.396 | 91.062 | 82.187 |
89 | 100.000 | 100.000 | 97.466 | 91.228 | 82.440 |
90 | 100.000 | 100.000 | 97.536 | 91.384 | 82.669 |
91 | 100.000 | 100.000 | 97.607 | 91.539 | 82.900 |
92 | 100.000 | 100.000 | 97.677 | 91.716 | 83.181 |
93 | 100.000 | 100.000 | 97.748 | 91.893 | 83.464 |
94 | 100.000 | 100.000 | 97.818 | 92.059 | 83.721 |
95 | 100.000 | 100.000 | 97.888 | 92.226 | 83.975 |
96 | 100.000 | 100.000 | 97.959 | 92.376 | 84.191 |
97 | 100.000 | 100.000 | 98.029 | 92.531 | 84.420 |
98 | 100.000 | 100.000 | 98.099 | 92.681 | 84.637 |
99 | 100.000 | 100.000 | 98.146 | 92.794 | 84.814 |
100 | 100.000 | 100.000 | 98.193 | 92.912 | 85.005 |
101 | 100.000 | 100.000 | 98.240 | 93.067 | 85.284 |
102 | 100.000 | 100.000 | 98.287 | 93.169 | 85.434 |
103 | 100.000 | 100.000 | 98.334 | 93.293 | 85.635 |
104 | 100.000 | 100.000 | 98.381 | 93.394 | 85.787 |
105 | 100.000 | 100.000 | 98.428 | 93.523 | 86.001 |
106 | 100.000 | 100.000 | 98.475 | 93.636 | 86.179 |
107 | 100.000 | 100.000 | 98.522 | 93.738 | 86.329 |
108 | 100.000 | 100.000 | 98.569 | 93.866 | 86.547 |
109 | 100.000 | 100.000 | 98.616 | 93.995 | 86.761 |
110 | 100.000 | 100.000 | 98.663 | 94.108 | 86.939 |
111 | 100.000 | 100.000 | 98.710 | 94.209 | 87.089 |
112 | 100.000 | 100.000 | 98.756 | 94.333 | 87.292 |
113 | 100.000 | 100.000 | 98.803 | 94.435 | 87.442 |
114 | 100.000 | 100.000 | 98.850 | 94.536 | 87.592 |
115 | 100.000 | 100.000 | 98.897 | 94.649 | 87.769 |
116 | 100.000 | 100.000 | 98.944 | 94.778 | 87.987 |
117 | 100.000 | 100.000 | 98.991 | 94.896 | 88.178 |
118 | 100.000 | 100.000 | 99.038 | 94.998 | 88.328 |
119 | 100.000 | 100.000 | 99.085 | 95.105 | 88.492 |
120 | 100.000 | 100.000 | 99.132 | 95.228 | 88.694 |
121 | 100.000 | 100.000 | 99.179 | 95.330 | 88.844 |
122 | 100.000 | 100.000 | 99.226 | 95.448 | 89.034 |
123 | 100.000 | 100.000 | 99.273 | 95.630 | 89.369 |
124 | 100.000 | 100.000 | 99.320 | 95.732 | 89.519 |
125 | 100.000 | 100.000 | 99.343 | 95.786 | 89.602 |
126 | 100.000 | 100.000 | 99.366 | 95.839 | 89.685 |
127 | 100.000 | 100.000 | 99.390 | 95.914 | 89.824 |
128 | 100.000 | 100.000 | 99.413 | 96.006 | 90.001 |
129 | 100.000 | 100.000 | 99.437 | 96.081 | 90.140 |
130 | 100.000 | 100.000 | 99.460 | 96.145 | 90.253 |
131 | 100.000 | 100.000 | 99.484 | 96.204 | 90.351 |
132 | 100.000 | 100.000 | 99.507 | 96.268 | 90.462 |
133 | 100.000 | 100.000 | 99.531 | 96.354 | 90.624 |
134 | 100.000 | 100.000 | 99.554 | 96.413 | 90.722 |
135 | 100.000 | 100.000 | 99.578 | 96.483 | 90.848 |
136 | 100.000 | 100.000 | 99.601 | 96.552 | 90.974 |
137 | 100.000 | 100.000 | 99.625 | 96.606 | 91.057 |
138 | 100.000 | 100.000 | 99.648 | 96.665 | 91.155 |
139 | 100.000 | 100.000 | 99.672 | 96.735 | 91.290 |
140 | 100.000 | 100.000 | 99.695 | 96.794 | 91.388 |
141 | 100.000 | 100.000 | 99.718 | 96.869 | 91.531 |
142 | 100.000 | 100.000 | 99.742 | 96.922 | 91.614 |
143 | 100.000 | 100.000 | 99.765 | 96.981 | 91.712 |
144 | 100.000 | 100.000 | 99.789 | 97.046 | 91.823 |
145 | 100.000 | 100.000 | 99.812 | 97.121 | 91.965 |
146 | 100.000 | 100.000 | 99.836 | 97.180 | 92.063 |
147 | 100.000 | 100.000 | 99.859 | 97.239 | 92.161 |
148 | 100.000 | 100.000 | 99.883 | 97.298 | 92.259 |
149 | 100.000 | 100.000 | 99.906 | 97.351 | 92.342 |
150 | 100.000 | 100.000 | 99.930 | 97.405 | 92.426 |
151 | 100.000 | 100.000 | 99.953 | 97.459 | 92.509 |
152 | 100.000 | 100.000 | 99.977 | 97.523 | 92.622 |
153 | 100.000 | 100.000 | 100.000 | 97.577 | 92.705 |
154 | 100.000 | 100.000 | 100.000 | 97.582 | 92.723 |
155 | 100.000 | 100.000 | 100.000 | 97.587 | 92.744 |
156 | 100.000 | 100.000 | 100.000 | 97.614 | 92.818 |
157 | 100.000 | 100.000 | 100.000 | 97.641 | 92.892 |
158 | 100.000 | 100.000 | 100.000 | 97.684 | 93.003 |
159 | 100.000 | 100.000 | 100.000 | 97.705 | 93.062 |
160 | 100.000 | 100.000 | 100.000 | 97.716 | 93.093 |
161 | 100.000 | 100.000 | 100.000 | 97.721 | 93.112 |
162 | 100.000 | 100.000 | 100.000 | 97.743 | 93.173 |
163 | 100.000 | 100.000 | 100.000 | 97.764 | 93.232 |
164 | 100.000 | 100.000 | 100.000 | 97.807 | 93.345 |
165 | 100.000 | 100.000 | 100.000 | 97.818 | 93.376 |
166 | 100.000 | 100.000 | 100.000 | 97.829 | 93.408 |
167 | 100.000 | 100.000 | 100.000 | 97.850 | 93.467 |
168 | 100.000 | 100.000 | 100.000 | 97.866 | 93.511 |
169 | 100.000 | 100.000 | 100.000 | 97.877 | 93.543 |
170 | 100.000 | 100.000 | 100.000 | 97.888 | 93.574 |
171 | 100.000 | 100.000 | 100.000 | 97.914 | 93.648 |
172 | 100.000 | 100.000 | 100.000 | 97.920 | 93.666 |
173 | 100.000 | 100.000 | 100.000 | 97.925 | 93.683 |
174 | 100.000 | 100.000 | 100.000 | 97.957 | 93.770 |
175 | 100.000 | 100.000 | 100.000 | 97.968 | 93.801 |
176 | 100.000 | 100.000 | 100.000 | 97.989 | 93.861 |
177 | 100.000 | 100.000 | 100.000 | 98.022 | 93.948 |
178 | 100.000 | 100.000 | 100.000 | 98.064 | 94.058 |
179 | 100.000 | 100.000 | 100.000 | 98.113 | 94.186 |
180 | 100.000 | 100.000 | 100.000 | 98.118 | 94.205 |
181 | 100.000 | 100.000 | 100.000 | 98.150 | 94.291 |
182 | 100.000 | 100.000 | 100.000 | 98.161 | 94.323 |
183 | 100.000 | 100.000 | 100.000 | 98.193 | 94.406 |
184 | 100.000 | 100.000 | 100.000 | 98.209 | 94.452 |
185 | 100.000 | 100.000 | 100.000 | 98.231 | 94.515 |
186 | 100.000 | 100.000 | 100.000 | 98.241 | 94.550 |
187 | 100.000 | 100.000 | 100.000 | 98.247 | 94.567 |
188 | 100.000 | 100.000 | 100.000 | 98.257 | 94.598 |
189 | 100.000 | 100.000 | 100.000 | 98.279 | 94.658 |
190 | 100.000 | 100.000 | 100.000 | 98.295 | 94.704 |
191 | 100.000 | 100.000 | 100.000 | 98.300 | 94.722 |
192 | 100.000 | 100.000 | 100.000 | 98.311 | 94.754 |
193 | 100.000 | 100.000 | 100.000 | 98.327 | 94.800 |
194 | 100.000 | 100.000 | 100.000 | 98.343 | 94.848 |
195 | 100.000 | 100.000 | 100.000 | 98.343 | 94.855 |
196 | 100.000 | 100.000 | 100.000 | 98.365 | 94.915 |
197 | 100.000 | 100.000 | 100.000 | 98.413 | 95.042 |
198 | 100.000 | 100.000 | 100.000 | 98.434 | 95.101 |
199 | 100.000 | 100.000 | 100.000 | 98.440 | 95.118 |
200 | 100.000 | 100.000 | 100.000 | 98.445 | 95.137 |
201 | 100.000 | 100.000 | 100.000 | 98.450 | 95.153 |
202 | 100.000 | 100.000 | 100.000 | 98.456 | 95.172 |
203 | 100.000 | 100.000 | 100.000 | 98.467 | 95.207 |
204 | 100.000 | 100.000 | 100.000 | 98.488 | 95.266 |
205 | 100.000 | 100.000 | 100.000 | 98.520 | 95.353 |
206 | 100.000 | 100.000 | 100.000 | 98.526 | 95.371 |
207 | 100.000 | 100.000 | 100.000 | 98.531 | 95.388 |
208 | 100.000 | 100.000 | 100.000 | 98.536 | 95.407 |
209 | 100.000 | 100.000 | 100.000 | 98.552 | 95.453 |
210 | 100.000 | 100.000 | 100.000 | 98.563 | 95.488 |
211 | 100.000 | 100.000 | 100.000 | 98.574 | 95.519 |
212 | 100.000 | 100.000 | 100.000 | 98.585 | 95.554 |
213 | 100.000 | 100.000 | 100.000 | 98.590 | 95.571 |
214 | 100.000 | 100.000 | 100.000 | 98.606 | 95.616 |
215 | 100.000 | 100.000 | 100.000 | 98.611 | 95.632 |
216 | 100.000 | 100.000 | 100.000 | 98.638 | 95.704 |
217 | 100.000 | 100.000 | 100.000 | 98.660 | 95.767 |
218 | 100.000 | 100.000 | 100.000 | 98.670 | 95.799 |
219 | 100.000 | 100.000 | 100.000 | 98.676 | 95.815 |
220 | 100.000 | 100.000 | 100.000 | 98.676 | 95.823 |
221 | 100.000 | 100.000 | 100.000 | 98.697 | 95.885 |
222 | 100.000 | 100.000 | 100.000 | 98.708 | 95.919 |
223 | 100.000 | 100.000 | 100.000 | 98.724 | 95.963 |
224 | 100.000 | 100.000 | 100.000 | 98.729 | 95.980 |
225 | 100.000 | 100.000 | 100.000 | 98.740 | 96.011 |
226 | 100.000 | 100.000 | 100.000 | 98.740 | 96.013 |
227 | 100.000 | 100.000 | 100.000 | 98.745 | 96.032 |
228 | 100.000 | 100.000 | 100.000 | 98.745 | 96.033 |
229 | 100.000 | 100.000 | 100.000 | 98.756 | 96.065 |
230 | 100.000 | 100.000 | 100.000 | 98.761 | 96.082 |
231 | 100.000 | 100.000 | 100.000 | 98.767 | 96.102 |
232 | 100.000 | 100.000 | 100.000 | 98.778 | 96.137 |
233 | 100.000 | 100.000 | 100.000 | 98.788 | 96.172 |
234 | 100.000 | 100.000 | 100.000 | 98.788 | 96.176 |
235 | 100.000 | 100.000 | 100.000 | 98.799 | 96.207 |
236 | 100.000 | 100.000 | 100.000 | 98.810 | 96.239 |
237 | 100.000 | 100.000 | 100.000 | 98.815 | 96.255 |
238 | 100.000 | 100.000 | 100.000 | 98.820 | 96.272 |
239 | 100.000 | 100.000 | 100.000 | 98.826 | 96.289 |
240 | 100.000 | 100.000 | 100.000 | 98.831 | 96.305 |
241 | 100.000 | 100.000 | 100.000 | 98.869 | 96.401 |
242 | 100.000 | 100.000 | 100.000 | 98.879 | 96.433 |
243 | 100.000 | 100.000 | 100.000 | 98.944 | 96.596 |
244 | 100.000 | 100.000 | 100.000 | 98.954 | 96.629 |
245 | 100.000 | 100.000 | 100.000 | 98.965 | 96.660 |
246 | 100.000 | 100.000 | 100.000 | 98.981 | 96.707 |
247 | 100.000 | 100.000 | 100.000 | 98.987 | 96.723 |
248 | 100.000 | 100.000 | 100.000 | 98.997 | 96.758 |
249 | 100.000 | 100.000 | 100.000 | 99.003 | 96.775 |
250 | 100.000 | 100.000 | 100.000 | 99.013 | 96.808 |
251 | 100.000 | 100.000 | 100.000 | 99.024 | 96.843 |
252 | 100.000 | 100.000 | 100.000 | 99.024 | 96.845 |
253 | 100.000 | 100.000 | 100.000 | 99.035 | 96.877 |
254 | 100.000 | 100.000 | 100.000 | 99.040 | 96.893 |
255 | 100.000 | 100.000 | 100.000 | 99.046 | 96.910 |
256 | 100.000 | 100.000 | 100.000 | 99.051 | 96.927 |
257 | 100.000 | 100.000 | 100.000 | 99.072 | 96.986 |
258 | 100.000 | 100.000 | 100.000 | 99.083 | 97.017 |
259 | 100.000 | 100.000 | 100.000 | 99.083 | 97.019 |
260 | 100.000 | 100.000 | 100.000 | 99.099 | 97.069 |
261 | 100.000 | 100.000 | 100.000 | 99.105 | 97.086 |
262 | 100.000 | 100.000 | 100.000 | 99.126 | 97.149 |
263 | 100.000 | 100.000 | 100.000 | 99.137 | 97.180 |
264 | 100.000 | 100.000 | 100.000 | 99.147 | 97.211 |
265 | 100.000 | 100.000 | 100.000 | 99.153 | 97.228 |
266 | 100.000 | 100.000 | 100.000 | 99.153 | 97.232 |
267 | 100.000 | 100.000 | 100.000 | 99.158 | 97.248 |
268 | 100.000 | 100.000 | 100.000 | 99.164 | 97.265 |
269 | 100.000 | 100.000 | 100.000 | 99.174 | 97.296 |
270 | 100.000 | 100.000 | 100.000 | 99.180 | 97.313 |
271 | 100.000 | 100.000 | 100.000 | 99.180 | 97.315 |
272 | 100.000 | 100.000 | 100.000 | 99.190 | 97.346 |
273 | 100.000 | 100.000 | 100.000 | 99.196 | 97.365 |
274 | 100.000 | 100.000 | 100.000 | 99.196 | 97.367 |
275 | 100.000 | 100.000 | 100.000 | 99.217 | 97.426 |
276 | 100.000 | 100.000 | 100.000 | 99.228 | 97.457 |
277 | 100.000 | 100.000 | 100.000 | 99.233 | 97.476 |
278 | 100.000 | 100.000 | 100.000 | 99.255 | 97.535 |
279 | 100.000 | 100.000 | 100.000 | 99.260 | 97.552 |
280 | 100.000 | 100.000 | 100.000 | 99.271 | 97.583 |
281 | 100.000 | 100.000 | 100.000 | 99.276 | 97.600 |
282 | 100.000 | 100.000 | 100.000 | 99.287 | 97.631 |
283 | 100.000 | 100.000 | 100.000 | 99.292 | 97.653 |
284 | 100.000 | 100.000 | 100.000 | 99.357 | 97.816 |
285 | 100.000 | 100.000 | 100.000 | 99.378 | 97.875 |
286 | 100.000 | 100.000 | 100.000 | 99.378 | 97.879 |
287 | 100.000 | 100.000 | 100.000 | 99.389 | 97.910 |
288 | 100.000 | 100.000 | 100.000 | 99.399 | 97.942 |
289 | 100.000 | 100.000 | 100.000 | 99.416 | 97.986 |
290 | 100.000 | 100.000 | 100.000 | 99.421 | 98.003 |
291 | 100.000 | 100.000 | 100.000 | 99.421 | 98.005 |
292 | 100.000 | 100.000 | 100.000 | 99.421 | 98.007 |
293 | 100.000 | 100.000 | 100.000 | 99.426 | 98.023 |
294 | 100.000 | 100.000 | 100.000 | 99.437 | 98.062 |
295 | 100.000 | 100.000 | 100.000 | 99.448 | 98.093 |
296 | 100.000 | 100.000 | 100.000 | 99.448 | 98.101 |
297 | 100.000 | 100.000 | 100.000 | 99.453 | 98.117 |
298 | 100.000 | 100.000 | 100.000 | 99.453 | 98.119 |
299 | 100.000 | 100.000 | 100.000 | 99.464 | 98.151 |
300 | 100.000 | 100.000 | 100.000 | 99.464 | 98.154 |
301 | 100.000 | 100.000 | 100.000 | 99.469 | 98.171 |
302 | 100.000 | 100.000 | 100.000 | 99.480 | 98.203 |
303 | 100.000 | 100.000 | 100.000 | 99.501 | 98.265 |
304 | 100.000 | 100.000 | 100.000 | 99.507 | 98.282 |
305 | 100.000 | 100.000 | 100.000 | 99.507 | 98.286 |
306 | 100.000 | 100.000 | 100.000 | 99.512 | 98.302 |
307 | 100.000 | 100.000 | 100.000 | 99.517 | 98.319 |
308 | 100.000 | 100.000 | 100.000 | 99.517 | 98.321 |
309 | 100.000 | 100.000 | 100.000 | 99.528 | 98.352 |
310 | 100.000 | 100.000 | 100.000 | 99.539 | 98.384 |
311 | 100.000 | 100.000 | 100.000 | 99.539 | 98.387 |
312 | 100.000 | 100.000 | 100.000 | 99.544 | 98.408 |
313 | 100.000 | 100.000 | 100.000 | 99.566 | 98.467 |
314 | 100.000 | 100.000 | 100.000 | 99.576 | 98.498 |
315 | 100.000 | 100.000 | 100.000 | 99.587 | 98.530 |
316 | 100.000 | 100.000 | 100.000 | 99.593 | 98.547 |
317 | 100.000 | 100.000 | 100.000 | 99.603 | 98.578 |
318 | 100.000 | 100.000 | 100.000 | 99.603 | 98.582 |
319 | 100.000 | 100.000 | 100.000 | 99.609 | 98.598 |
320 | 100.000 | 100.000 | 100.000 | 99.614 | 98.615 |
321 | 100.000 | 100.000 | 100.000 | 99.614 | 98.619 |
322 | 100.000 | 100.000 | 100.000 | 99.619 | 98.635 |
323 | 100.000 | 100.000 | 100.000 | 99.625 | 98.652 |
324 | 100.000 | 100.000 | 100.000 | 99.625 | 98.656 |
325 | 100.000 | 100.000 | 100.000 | 99.635 | 98.687 |
326 | 100.000 | 100.000 | 100.000 | 99.641 | 98.704 |
327 | 100.000 | 100.000 | 100.000 | 99.646 | 98.720 |
328 | 100.000 | 100.000 | 100.000 | 99.646 | 98.724 |
329 | 100.000 | 100.000 | 100.000 | 99.646 | 98.726 |
330 | 100.000 | 100.000 | 100.000 | 99.657 | 98.757 |
331 | 100.000 | 100.000 | 100.000 | 99.662 | 98.778 |
332 | 100.000 | 100.000 | 100.000 | 99.668 | 98.794 |
333 | 100.000 | 100.000 | 100.000 | 99.673 | 98.811 |
334 | 100.000 | 100.000 | 100.000 | 99.673 | 98.813 |
335 | 100.000 | 100.000 | 100.000 | 99.678 | 98.829 |
336 | 100.000 | 100.000 | 100.000 | 99.689 | 98.861 |
337 | 100.000 | 100.000 | 100.000 | 99.694 | 98.878 |
338 | 100.000 | 100.000 | 100.000 | 99.700 | 98.894 |
339 | 100.000 | 100.000 | 100.000 | 99.705 | 98.911 |
340 | 100.000 | 100.000 | 100.000 | 99.710 | 98.927 |
341 | 100.000 | 100.000 | 100.000 | 99.710 | 98.929 |
342 | 100.000 | 100.000 | 100.000 | 99.710 | 98.933 |
343 | 100.000 | 100.000 | 100.000 | 99.721 | 98.968 |
344 | 100.000 | 100.000 | 100.000 | 99.732 | 99.000 |
345 | 100.000 | 100.000 | 100.000 | 99.732 | 99.003 |
346 | 100.000 | 100.000 | 100.000 | 99.732 | 99.005 |
347 | 100.000 | 100.000 | 100.000 | 99.748 | 99.051 |
348 | 100.000 | 100.000 | 100.000 | 99.759 | 99.083 |
349 | 100.000 | 100.000 | 100.000 | 99.764 | 99.099 |
350 | 100.000 | 100.000 | 100.000 | 99.764 | 99.101 |
351 | 100.000 | 100.000 | 100.000 | 99.775 | 99.133 |
352 | 100.000 | 100.000 | 100.000 | 99.775 | 99.140 |
353 | 100.000 | 100.000 | 100.000 | 99.775 | 99.144 |
354 | 100.000 | 100.000 | 100.000 | 99.775 | 99.148 |
355 | 100.000 | 100.000 | 100.000 | 99.780 | 99.164 |
356 | 100.000 | 100.000 | 100.000 | 99.786 | 99.181 |
357 | 100.000 | 100.000 | 100.000 | 99.786 | 99.184 |
358 | 100.000 | 100.000 | 100.000 | 99.786 | 99.188 |
359 | 100.000 | 100.000 | 100.000 | 99.786 | 99.190 |
360 | 100.000 | 100.000 | 100.000 | 99.786 | 99.194 |
361 | 100.000 | 100.000 | 100.000 | 99.791 | 99.210 |
362 | 100.000 | 100.000 | 100.000 | 99.791 | 99.212 |
363 | 100.000 | 100.000 | 100.000 | 99.802 | 99.244 |
364 | 100.000 | 100.000 | 100.000 | 99.807 | 99.260 |
365 | 100.000 | 100.000 | 100.000 | 99.812 | 99.277 |
366 | 100.000 | 100.000 | 100.000 | 99.818 | 99.295 |
367 | 100.000 | 100.000 | 100.000 | 99.828 | 99.327 |
368 | 100.000 | 100.000 | 100.000 | 99.828 | 99.329 |
369 | 100.000 | 100.000 | 100.000 | 99.834 | 99.345 |
370 | 100.000 | 100.000 | 100.000 | 99.834 | 99.347 |
371 | 100.000 | 100.000 | 100.000 | 99.834 | 99.351 |
372 | 100.000 | 100.000 | 100.000 | 99.834 | 99.353 |
373 | 100.000 | 100.000 | 100.000 | 99.850 | 99.399 |
374 | 100.000 | 100.000 | 100.000 | 99.850 | 99.403 |
375 | 100.000 | 100.000 | 100.000 | 99.850 | 99.406 |
376 | 100.000 | 100.000 | 100.000 | 99.850 | 99.412 |
377 | 100.000 | 100.000 | 100.000 | 99.855 | 99.429 |
378 | 100.000 | 100.000 | 100.000 | 99.861 | 99.445 |
379 | 100.000 | 100.000 | 100.000 | 99.866 | 99.462 |
380 | 100.000 | 100.000 | 100.000 | 99.871 | 99.479 |
381 | 100.000 | 100.000 | 100.000 | 99.871 | 99.482 |
382 | 100.000 | 100.000 | 100.000 | 99.882 | 99.514 |
383 | 100.000 | 100.000 | 100.000 | 99.882 | 99.517 |
384 | 100.000 | 100.000 | 100.000 | 99.882 | 99.525 |
385 | 100.000 | 100.000 | 100.000 | 99.893 | 99.556 |
386 | 100.000 | 100.000 | 100.000 | 99.893 | 99.558 |
387 | 100.000 | 100.000 | 100.000 | 99.903 | 99.593 |
388 | 100.000 | 100.000 | 100.000 | 99.903 | 99.597 |
389 | 100.000 | 100.000 | 100.000 | 99.903 | 99.601 |
390 | 100.000 | 100.000 | 100.000 | 99.903 | 99.604 |
391 | 100.000 | 100.000 | 100.000 | 99.909 | 99.621 |
392 | 100.000 | 100.000 | 100.000 | 99.909 | 99.623 |
393 | 100.000 | 100.000 | 100.000 | 99.909 | 99.625 |
394 | 100.000 | 100.000 | 100.000 | 99.914 | 99.641 |
395 | 100.000 | 100.000 | 100.000 | 99.925 | 99.673 |
396 | 100.000 | 100.000 | 100.000 | 99.925 | 99.676 |
397 | 100.000 | 100.000 | 100.000 | 99.930 | 99.693 |
398 | 100.000 | 100.000 | 100.000 | 99.941 | 99.724 |
399 | 100.000 | 100.000 | 100.000 | 99.946 | 99.741 |
400 | 100.000 | 100.000 | 100.000 | 99.946 | 99.745 |
401 | 100.000 | 100.000 | 100.000 | 99.946 | 99.749 |
402 | 100.000 | 100.000 | 100.000 | 99.952 | 99.765 |
403 | 100.000 | 100.000 | 100.000 | 99.952 | 99.769 |
404 | 100.000 | 100.000 | 100.000 | 99.952 | 99.771 |
405 | 100.000 | 100.000 | 100.000 | 99.952 | 99.782 |
406 | 100.000 | 100.000 | 100.000 | 99.952 | 99.784 |
407 | 100.000 | 100.000 | 100.000 | 99.952 | 99.787 |
408 | 100.000 | 100.000 | 100.000 | 99.952 | 99.791 |
409 | 100.000 | 100.000 | 100.000 | 99.952 | 99.795 |
410 | 100.000 | 100.000 | 100.000 | 99.952 | 99.798 |
411 | 100.000 | 100.000 | 100.000 | 99.957 | 99.815 |
412 | 100.000 | 100.000 | 100.000 | 99.962 | 99.832 |
413 | 100.000 | 100.000 | 100.000 | 99.962 | 99.839 |
414 | 100.000 | 100.000 | 100.000 | 99.962 | 99.843 |
415 | 100.000 | 100.000 | 100.000 | 99.968 | 99.859 |
416 | 100.000 | 100.000 | 100.000 | 99.973 | 99.876 |
417 | 100.000 | 100.000 | 100.000 | 99.973 | 99.880 |
418 | 100.000 | 100.000 | 100.000 | 99.979 | 99.896 |
419 | 100.000 | 100.000 | 100.000 | 99.979 | 99.900 |
420 | 100.000 | 100.000 | 100.000 | 99.979 | 99.904 |
421 | 100.000 | 100.000 | 100.000 | 99.979 | 99.906 |
422 | 100.000 | 100.000 | 100.000 | 99.979 | 99.908 |
423 | 100.000 | 100.000 | 100.000 | 99.979 | 99.909 |
424 | 100.000 | 100.000 | 100.000 | 99.979 | 99.911 |
425 | 100.000 | 100.000 | 100.000 | 99.984 | 99.928 |
426 | 100.000 | 100.000 | 100.000 | 99.984 | 99.932 |
427 | 100.000 | 100.000 | 100.000 | 99.995 | 99.963 |
428 | 100.000 | 100.000 | 100.000 | 99.995 | 99.967 |
429 | 100.000 | 100.000 | 100.000 | 99.995 | 99.970 |
430 | 100.000 | 100.000 | 100.000 | 99.995 | 99.974 |
431 | 100.000 | 100.000 | 100.000 | 99.995 | 99.976 |
432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


grep
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
3 | 57.143 | 31.641 | 18.254 | 11.072 | 7.053 |
4 | 64.286 | 38.579 | 23.322 | 14.503 | 9.348 |
5 | 73.810 | 46.024 | 28.414 | 17.903 | 11.626 |
6 | 83.333 | 53.807 | 33.740 | 21.409 | 13.941 |
7 | 88.095 | 59.222 | 38.081 | 24.546 | 16.134 |
8 | 90.476 | 63.113 | 41.764 | 27.457 | 18.267 |
9 | 95.238 | 68.697 | 46.222 | 30.647 | 20.478 |
10 | 100.000 | 73.435 | 50.000 | 33.457 | 22.514 |
11 | 100.000 | 75.465 | 52.557 | 35.784 | 24.384 |
12 | 100.000 | 76.988 | 54.716 | 37.896 | 26.161 |
13 | 100.000 | 79.019 | 57.227 | 40.164 | 27.984 |
14 | 100.000 | 80.372 | 59.338 | 42.282 | 29.767 |
15 | 100.000 | 82.234 | 61.614 | 44.368 | 31.487 |
16 | 100.000 | 83.249 | 63.210 | 46.078 | 33.038 |
17 | 100.000 | 84.772 | 65.204 | 47.992 | 34.667 |
18 | 100.000 | 86.125 | 67.058 | 49.810 | 36.230 |
19 | 100.000 | 87.310 | 68.700 | 51.466 | 37.702 |
20 | 100.000 | 88.325 | 70.155 | 52.994 | 39.107 |
21 | 100.000 | 89.171 | 71.422 | 54.356 | 40.383 |
22 | 100.000 | 89.679 | 72.407 | 55.568 | 41.602 |
23 | 100.000 | 90.355 | 73.534 | 56.849 | 42.841 |
24 | 100.000 | 91.201 | 74.847 | 58.238 | 44.098 |
25 | 100.000 | 91.878 | 75.950 | 59.477 | 45.289 |
26 | 100.000 | 92.555 | 77.053 | 60.715 | 46.480 |
27 | 100.000 | 92.893 | 77.851 | 61.771 | 47.590 |
28 | 100.000 | 93.401 | 78.719 | 62.822 | 48.664 |
29 | 100.000 | 93.909 | 79.634 | 63.911 | 49.751 |
30 | 100.000 | 94.416 | 80.526 | 64.967 | 50.811 |
31 | 100.000 | 94.755 | 81.276 | 65.937 | 51.822 |
32 | 100.000 | 95.262 | 82.145 | 66.956 | 52.836 |
33 | 100.000 | 95.431 | 82.708 | 67.798 | 53.784 |
34 | 100.000 | 95.601 | 83.294 | 68.656 | 54.724 |
35 | 100.000 | 95.939 | 83.975 | 69.535 | 55.665 |
36 | 100.000 | 96.108 | 84.514 | 70.334 | 56.566 |
37 | 100.000 | 96.616 | 85.312 | 71.256 | 57.501 |
38 | 100.000 | 96.954 | 85.946 | 72.050 | 58.339 |
39 | 100.000 | 97.293 | 86.579 | 72.838 | 59.166 |
40 | 100.000 | 97.462 | 87.001 | 73.471 | 59.911 |
41 | 100.000 | 97.631 | 87.447 | 74.125 | 60.669 |
42 | 100.000 | 97.800 | 87.893 | 74.768 | 61.403 |
43 | 100.000 | 97.970 | 88.339 | 75.412 | 62.137 |
44 | 100.000 | 97.970 | 88.620 | 75.958 | 62.840 |
45 | 100.000 | 98.139 | 89.066 | 76.602 | 63.574 |
46 | 100.000 | 98.308 | 89.465 | 77.192 | 64.277 |
47 | 100.000 | 98.308 | 89.747 | 77.722 | 64.946 |
48 | 100.000 | 98.477 | 90.145 | 78.291 | 65.599 |
49 | 100.000 | 98.646 | 90.544 | 78.854 | 66.244 |
50 | 100.000 | 98.816 | 90.896 | 79.358 | 66.845 |
51 | 100.000 | 98.816 | 91.107 | 79.781 | 67.409 |
52 | 100.000 | 99.154 | 91.600 | 80.350 | 68.012 |
53 | 100.000 | 99.323 | 91.976 | 80.864 | 68.593 |
54 | 100.000 | 99.323 | 92.187 | 81.283 | 69.138 |
55 | 100.000 | 99.323 | 92.374 | 81.669 | 69.664 |
56 | 100.000 | 99.323 | 92.562 | 82.055 | 70.189 |
57 | 100.000 | 99.492 | 92.914 | 82.532 | 70.731 |
58 | 100.000 | 99.492 | 93.102 | 82.907 | 71.232 |
59 | 100.000 | 99.492 | 93.290 | 83.272 | 71.709 |
60 | 100.000 | 99.492 | 93.454 | 83.615 | 72.182 |
61 | 100.000 | 99.662 | 93.782 | 84.054 | 72.678 |
62 | 100.000 | 99.662 | 93.947 | 84.392 | 73.140 |
63 | 100.000 | 99.831 | 94.252 | 84.800 | 73.615 |
64 | 100.000 | 99.831 | 94.416 | 85.132 | 74.067 |
65 | 100.000 | 99.831 | 94.580 | 85.465 | 74.518 |
66 | 100.000 | 99.831 | 94.721 | 85.770 | 74.954 |
67 | 100.000 | 99.831 | 94.838 | 86.054 | 75.385 |
68 | 100.000 | 99.831 | 94.979 | 86.355 | 75.809 |
69 | 100.000 | 99.831 | 95.120 | 86.650 | 76.221 |
70 | 100.000 | 99.831 | 95.260 | 86.944 | 76.631 |
71 | 100.000 | 99.831 | 95.401 | 87.239 | 77.042 |
72 | 100.000 | 99.831 | 95.542 | 87.534 | 77.456 |
73 | 100.000 | 99.831 | 95.683 | 87.824 | 77.856 |
74 | 100.000 | 99.831 | 95.800 | 88.092 | 78.251 |
75 | 100.000 | 99.831 | 95.941 | 88.376 | 78.638 |
76 | 100.000 | 99.831 | 96.082 | 88.660 | 79.017 |
77 | 100.000 | 99.831 | 96.222 | 88.939 | 79.389 |
78 | 100.000 | 99.831 | 96.363 | 89.212 | 79.747 |
79 | 100.000 | 99.831 | 96.481 | 89.459 | 80.095 |
80 | 100.000 | 99.831 | 96.574 | 89.684 | 80.439 |
81 | 100.000 | 99.831 | 96.692 | 89.931 | 80.787 |
82 | 100.000 | 99.831 | 96.786 | 90.151 | 81.118 |
83 | 100.000 | 99.831 | 96.879 | 90.360 | 81.426 |
84 | 100.000 | 99.831 | 96.973 | 90.569 | 81.735 |
85 | 100.000 | 99.831 | 97.020 | 90.735 | 82.037 |
86 | 100.000 | 99.831 | 97.067 | 90.901 | 82.334 |
87 | 100.000 | 99.831 | 97.137 | 91.084 | 82.630 |
88 | 100.000 | 99.831 | 97.184 | 91.244 | 82.921 |
89 | 100.000 | 100.000 | 97.443 | 91.539 | 83.226 |
90 | 100.000 | 100.000 | 97.489 | 91.695 | 83.503 |
91 | 100.000 | 100.000 | 97.536 | 91.850 | 83.781 |
92 | 100.000 | 100.000 | 97.630 | 92.049 | 84.060 |
93 | 100.000 | 100.000 | 97.724 | 92.242 | 84.328 |
94 | 100.000 | 100.000 | 97.818 | 92.435 | 84.596 |
95 | 100.000 | 100.000 | 97.888 | 92.606 | 84.862 |
96 | 100.000 | 100.000 | 97.959 | 92.773 | 85.119 |
97 | 100.000 | 100.000 | 98.029 | 92.939 | 85.376 |
98 | 100.000 | 100.000 | 98.099 | 93.100 | 85.621 |
99 | 100.000 | 100.000 | 98.170 | 93.260 | 85.866 |
100 | 100.000 | 100.000 | 98.217 | 93.400 | 86.109 |
101 | 100.000 | 100.000 | 98.287 | 93.561 | 86.355 |
102 | 100.000 | 100.000 | 98.334 | 93.695 | 86.584 |
103 | 100.000 | 100.000 | 98.405 | 93.845 | 86.800 |
104 | 100.000 | 100.000 | 98.475 | 93.995 | 87.017 |
105 | 100.000 | 100.000 | 98.522 | 94.124 | 87.231 |
106 | 100.000 | 100.000 | 98.569 | 94.247 | 87.436 |
107 | 100.000 | 100.000 | 98.616 | 94.370 | 87.642 |
108 | 100.000 | 100.000 | 98.663 | 94.494 | 87.849 |
109 | 100.000 | 100.000 | 98.710 | 94.612 | 88.039 |
110 | 100.000 | 100.000 | 98.756 | 94.730 | 88.230 |
111 | 100.000 | 100.000 | 98.803 | 94.842 | 88.411 |
112 | 100.000 | 100.000 | 98.850 | 94.955 | 88.588 |
113 | 100.000 | 100.000 | 98.897 | 95.067 | 88.770 |
114 | 100.000 | 100.000 | 98.944 | 95.180 | 88.949 |
115 | 100.000 | 100.000 | 98.944 | 95.244 | 89.104 |
116 | 100.000 | 100.000 | 98.991 | 95.351 | 89.269 |
117 | 100.000 | 100.000 | 99.038 | 95.459 | 89.434 |
118 | 100.000 | 100.000 | 99.061 | 95.544 | 89.600 |
119 | 100.000 | 100.000 | 99.061 | 95.609 | 89.763 |
120 | 100.000 | 100.000 | 99.108 | 95.716 | 89.931 |
121 | 100.000 | 100.000 | 99.155 | 95.818 | 90.081 |
122 | 100.000 | 100.000 | 99.179 | 95.898 | 90.234 |
123 | 100.000 | 100.000 | 99.226 | 96.000 | 90.384 |
124 | 100.000 | 100.000 | 99.249 | 96.081 | 90.538 |
125 | 100.000 | 100.000 | 99.296 | 96.183 | 90.687 |
126 | 100.000 | 100.000 | 99.343 | 96.284 | 90.837 |
127 | 100.000 | 100.000 | 99.390 | 96.386 | 90.987 |
128 | 100.000 | 100.000 | 99.413 | 96.461 | 91.126 |
129 | 100.000 | 100.000 | 99.437 | 96.536 | 91.266 |
130 | 100.000 | 100.000 | 99.437 | 96.590 | 91.405 |
131 | 100.000 | 100.000 | 99.460 | 96.665 | 91.545 |
132 | 100.000 | 100.000 | 99.484 | 96.735 | 91.671 |
133 | 100.000 | 100.000 | 99.507 | 96.804 | 91.797 |
134 | 100.000 | 100.000 | 99.531 | 96.869 | 91.908 |
135 | 100.000 | 100.000 | 99.531 | 96.912 | 92.022 |
136 | 100.000 | 100.000 | 99.554 | 96.976 | 92.135 |
137 | 100.000 | 100.000 | 99.554 | 97.019 | 92.246 |
138 | 100.000 | 100.000 | 99.578 | 97.083 | 92.357 |
139 | 100.000 | 100.000 | 99.601 | 97.148 | 92.468 |
140 | 100.000 | 100.000 | 99.625 | 97.212 | 92.581 |
141 | 100.000 | 100.000 | 99.648 | 97.276 | 92.696 |
142 | 100.000 | 100.000 | 99.672 | 97.341 | 92.807 |
143 | 100.000 | 100.000 | 99.695 | 97.405 | 92.918 |
144 | 100.000 | 100.000 | 99.718 | 97.469 | 93.028 |
145 | 100.000 | 100.000 | 99.742 | 97.528 | 93.126 |
146 | 100.000 | 100.000 | 99.765 | 97.587 | 93.224 |
147 | 100.000 | 100.000 | 99.789 | 97.646 | 93.322 |
148 | 100.000 | 100.000 | 99.812 | 97.705 | 93.422 |
149 | 100.000 | 100.000 | 99.812 | 97.743 | 93.522 |
150 | 100.000 | 100.000 | 99.836 | 97.802 | 93.620 |
151 | 100.000 | 100.000 | 99.836 | 97.839 | 93.720 |
152 | 100.000 | 100.000 | 99.859 | 97.898 | 93.818 |
153 | 100.000 | 100.000 | 99.883 | 97.957 | 93.918 |
154 | 100.000 | 100.000 | 99.883 | 97.995 | 94.014 |
155 | 100.000 | 100.000 | 99.906 | 98.054 | 94.112 |
156 | 100.000 | 100.000 | 99.906 | 98.086 | 94.199 |
157 | 100.000 | 100.000 | 99.906 | 98.118 | 94.286 |
158 | 100.000 | 100.000 | 99.930 | 98.172 | 94.369 |
159 | 100.000 | 100.000 | 99.953 | 98.225 | 94.452 |
160 | 100.000 | 100.000 | 99.953 | 98.257 | 94.539 |
161 | 100.000 | 100.000 | 99.953 | 98.290 | 94.626 |
162 | 100.000 | 100.000 | 99.977 | 98.343 | 94.709 |
163 | 100.000 | 100.000 | 99.977 | 98.375 | 94.796 |
164 | 100.000 | 100.000 | 100.000 | 98.429 | 94.880 |
165 | 100.000 | 100.000 | 100.000 | 98.461 | 94.966 |
166 | 100.000 | 100.000 | 100.000 | 98.493 | 95.053 |
167 | 100.000 | 100.000 | 100.000 | 98.526 | 95.140 |
168 | 100.000 | 100.000 | 100.000 | 98.552 | 95.216 |
169 | 100.000 | 100.000 | 100.000 | 98.579 | 95.288 |
170 | 100.000 | 100.000 | 100.000 | 98.606 | 95.368 |
171 | 100.000 | 100.000 | 100.000 | 98.633 | 95.440 |
172 | 100.000 | 100.000 | 100.000 | 98.660 | 95.512 |
173 | 100.000 | 100.000 | 100.000 | 98.686 | 95.588 |
174 | 100.000 | 100.000 | 100.000 | 98.713 | 95.664 |
175 | 100.000 | 100.000 | 100.000 | 98.735 | 95.723 |
176 | 100.000 | 100.000 | 100.000 | 98.756 | 95.793 |
177 | 100.000 | 100.000 | 100.000 | 98.778 | 95.860 |
178 | 100.000 | 100.000 | 100.000 | 98.799 | 95.922 |
179 | 100.000 | 100.000 | 100.000 | 98.820 | 95.984 |
180 | 100.000 | 100.000 | 100.000 | 98.842 | 96.045 |
181 | 100.000 | 100.000 | 100.000 | 98.863 | 96.104 |
182 | 100.000 | 100.000 | 100.000 | 98.885 | 96.167 |
183 | 100.000 | 100.000 | 100.000 | 98.906 | 96.226 |
184 | 100.000 | 100.000 | 100.000 | 98.928 | 96.289 |
185 | 100.000 | 100.000 | 100.000 | 98.949 | 96.348 |
186 | 100.000 | 100.000 | 100.000 | 98.971 | 96.411 |
187 | 100.000 | 100.000 | 100.000 | 98.992 | 96.474 |
188 | 100.000 | 100.000 | 100.000 | 99.013 | 96.533 |
189 | 100.000 | 100.000 | 100.000 | 99.035 | 96.592 |
190 | 100.000 | 100.000 | 100.000 | 99.056 | 96.655 |
191 | 100.000 | 100.000 | 100.000 | 99.078 | 96.714 |
192 | 100.000 | 100.000 | 100.000 | 99.099 | 96.777 |
193 | 100.000 | 100.000 | 100.000 | 99.121 | 96.836 |
194 | 100.000 | 100.000 | 100.000 | 99.142 | 96.895 |
195 | 100.000 | 100.000 | 100.000 | 99.164 | 96.958 |
196 | 100.000 | 100.000 | 100.000 | 99.185 | 97.021 |
197 | 100.000 | 100.000 | 100.000 | 99.201 | 97.067 |
198 | 100.000 | 100.000 | 100.000 | 99.217 | 97.113 |
199 | 100.000 | 100.000 | 100.000 | 99.233 | 97.163 |
200 | 100.000 | 100.000 | 100.000 | 99.249 | 97.213 |
201 | 100.000 | 100.000 | 100.000 | 99.265 | 97.259 |
202 | 100.000 | 100.000 | 100.000 | 99.282 | 97.306 |
203 | 100.000 | 100.000 | 100.000 | 99.298 | 97.354 |
204 | 100.000 | 100.000 | 100.000 | 99.314 | 97.400 |
205 | 100.000 | 100.000 | 100.000 | 99.330 | 97.444 |
206 | 100.000 | 100.000 | 100.000 | 99.346 | 97.491 |
207 | 100.000 | 100.000 | 100.000 | 99.362 | 97.537 |
208 | 100.000 | 100.000 | 100.000 | 99.373 | 97.570 |
209 | 100.000 | 100.000 | 100.000 | 99.383 | 97.603 |
210 | 100.000 | 100.000 | 100.000 | 99.394 | 97.635 |
211 | 100.000 | 100.000 | 100.000 | 99.405 | 97.666 |
212 | 100.000 | 100.000 | 100.000 | 99.416 | 97.698 |
213 | 100.000 | 100.000 | 100.000 | 99.426 | 97.729 |
214 | 100.000 | 100.000 | 100.000 | 99.437 | 97.762 |
215 | 100.000 | 100.000 | 100.000 | 99.448 | 97.794 |
216 | 100.000 | 100.000 | 100.000 | 99.458 | 97.825 |
217 | 100.000 | 100.000 | 100.000 | 99.469 | 97.859 |
218 | 100.000 | 100.000 | 100.000 | 99.480 | 97.890 |
219 | 100.000 | 100.000 | 100.000 | 99.491 | 97.925 |
220 | 100.000 | 100.000 | 100.000 | 99.501 | 97.958 |
221 | 100.000 | 100.000 | 100.000 | 99.512 | 97.994 |
222 | 100.000 | 100.000 | 100.000 | 99.523 | 98.025 |
223 | 100.000 | 100.000 | 100.000 | 99.534 | 98.056 |
224 | 100.000 | 100.000 | 100.000 | 99.544 | 98.090 |
225 | 100.000 | 100.000 | 100.000 | 99.555 | 98.125 |
226 | 100.000 | 100.000 | 100.000 | 99.566 | 98.156 |
227 | 100.000 | 100.000 | 100.000 | 99.576 | 98.188 |
228 | 100.000 | 100.000 | 100.000 | 99.587 | 98.223 |
229 | 100.000 | 100.000 | 100.000 | 99.598 | 98.254 |
230 | 100.000 | 100.000 | 100.000 | 99.609 | 98.288 |
231 | 100.000 | 100.000 | 100.000 | 99.619 | 98.325 |
232 | 100.000 | 100.000 | 100.000 | 99.630 | 98.362 |
233 | 100.000 | 100.000 | 100.000 | 99.641 | 98.393 |
234 | 100.000 | 100.000 | 100.000 | 99.651 | 98.424 |
235 | 100.000 | 100.000 | 100.000 | 99.662 | 98.458 |
236 | 100.000 | 100.000 | 100.000 | 99.673 | 98.489 |
237 | 100.000 | 100.000 | 100.000 | 99.684 | 98.521 |
238 | 100.000 | 100.000 | 100.000 | 99.694 | 98.552 |
239 | 100.000 | 100.000 | 100.000 | 99.705 | 98.584 |
240 | 100.000 | 100.000 | 100.000 | 99.716 | 98.615 |
241 | 100.000 | 100.000 | 100.000 | 99.721 | 98.632 |
242 | 100.000 | 100.000 | 100.000 | 99.727 | 98.648 |
243 | 100.000 | 100.000 | 100.000 | 99.732 | 98.665 |
244 | 100.000 | 100.000 | 100.000 | 99.737 | 98.683 |
245 | 100.000 | 100.000 | 100.000 | 99.743 | 98.700 |
246 | 100.000 | 100.000 | 100.000 | 99.748 | 98.717 |
247 | 100.000 | 100.000 | 100.000 | 99.753 | 98.733 |
248 | 100.000 | 100.000 | 100.000 | 99.759 | 98.750 |
249 | 100.000 | 100.000 | 100.000 | 99.764 | 98.767 |
250 | 100.000 | 100.000 | 100.000 | 99.769 | 98.783 |
251 | 100.000 | 100.000 | 100.000 | 99.775 | 98.802 |
252 | 100.000 | 100.000 | 100.000 | 99.780 | 98.818 |
253 | 100.000 | 100.000 | 100.000 | 99.786 | 98.835 |
254 | 100.000 | 100.000 | 100.000 | 99.791 | 98.852 |
255 | 100.000 | 100.000 | 100.000 | 99.796 | 98.868 |
256 | 100.000 | 100.000 | 100.000 | 99.802 | 98.885 |
257 | 100.000 | 100.000 | 100.000 | 99.807 | 98.902 |
258 | 100.000 | 100.000 | 100.000 | 99.812 | 98.918 |
259 | 100.000 | 100.000 | 100.000 | 99.818 | 98.937 |
260 | 100.000 | 100.000 | 100.000 | 99.823 | 98.953 |
261 | 100.000 | 100.000 | 100.000 | 99.828 | 98.970 |
262 | 100.000 | 100.000 | 100.000 | 99.834 | 98.987 |
263 | 100.000 | 100.000 | 100.000 | 99.839 | 99.003 |
264 | 100.000 | 100.000 | 100.000 | 99.845 | 99.020 |
265 | 100.000 | 100.000 | 100.000 | 99.850 | 99.037 |
266 | 100.000 | 100.000 | 100.000 | 99.855 | 99.053 |
267 | 100.000 | 100.000 | 100.000 | 99.861 | 99.070 |
268 | 100.000 | 100.000 | 100.000 | 99.866 | 99.088 |
269 | 100.000 | 100.000 | 100.000 | 99.871 | 99.107 |
270 | 100.000 | 100.000 | 100.000 | 99.877 | 99.123 |
271 | 100.000 | 100.000 | 100.000 | 99.882 | 99.144 |
272 | 100.000 | 100.000 | 100.000 | 99.887 | 99.160 |
273 | 100.000 | 100.000 | 100.000 | 99.893 | 99.177 |
274 | 100.000 | 100.000 | 100.000 | 99.898 | 99.196 |
275 | 100.000 | 100.000 | 100.000 | 99.903 | 99.216 |
276 | 100.000 | 100.000 | 100.000 | 99.909 | 99.233 |
277 | 100.000 | 100.000 | 100.000 | 99.914 | 99.249 |
278 | 100.000 | 100.000 | 100.000 | 99.920 | 99.266 |
279 | 100.000 | 100.000 | 100.000 | 99.925 | 99.283 |
280 | 100.000 | 100.000 | 100.000 | 99.930 | 99.299 |
281 | 100.000 | 100.000 | 100.000 | 99.936 | 99.316 |
282 | 100.000 | 100.000 | 100.000 | 99.941 | 99.332 |
283 | 100.000 | 100.000 | 100.000 | 99.946 | 99.351 |
284 | 100.000 | 100.000 | 100.000 | 99.952 | 99.368 |
285 | 100.000 | 100.000 | 100.000 | 99.957 | 99.384 |
286 | 100.000 | 100.000 | 100.000 | 99.962 | 99.405 |
287 | 100.000 | 100.000 | 100.000 | 99.968 | 99.421 |
288 | 100.000 | 100.000 | 100.000 | 99.973 | 99.438 |
289 | 100.000 | 100.000 | 100.000 | 99.979 | 99.454 |
290 | 100.000 | 100.000 | 100.000 | 99.984 | 99.471 |
291 | 100.000 | 100.000 | 100.000 | 99.989 | 99.488 |
292 | 100.000 | 100.000 | 100.000 | 99.995 | 99.504 |
293 | 100.000 | 100.000 | 100.000 | 100.000 | 99.521 |
294 | 100.000 | 100.000 | 100.000 | 100.000 | 99.525 |
295 | 100.000 | 100.000 | 100.000 | 100.000 | 99.532 |
296 | 100.000 | 100.000 | 100.000 | 100.000 | 99.536 |
297 | 100.000 | 100.000 | 100.000 | 100.000 | 99.538 |
298 | 100.000 | 100.000 | 100.000 | 100.000 | 99.540 |
299 | 100.000 | 100.000 | 100.000 | 100.000 | 99.541 |
300 | 100.000 | 100.000 | 100.000 | 100.000 | 99.543 |
301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.545 |
302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.547 |
303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.551 |
305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.554 |
306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.562 |
307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.567 |
309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.571 |
310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.575 |
311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.578 |
312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.582 |
313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.586 |
314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.589 |
315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.593 |
316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.597 |
317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.602 |
319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.606 |
320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.610 |
321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.612 |
322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.625 |
325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.632 |
328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.634 |
329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.643 |
331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.649 |
333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.652 |
334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.660 |
337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.671 |
340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.682 |
343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.708 |
351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.712 |
352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.713 |
353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.715 |
354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.723 |
355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.724 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.726 |
357 | 100.000 | 100.000 | 100.000 | 100.000 | 99.728 |
358 | 100.000 | 100.000 | 100.000 | 100.000 | 99.730 |
359 | 100.000 | 100.000 | 100.000 | 100.000 | 99.734 |
360 | 100.000 | 100.000 | 100.000 | 100.000 | 99.737 |
361 | 100.000 | 100.000 | 100.000 | 100.000 | 99.745 |
362 | 100.000 | 100.000 | 100.000 | 100.000 | 99.749 |
363 | 100.000 | 100.000 | 100.000 | 100.000 | 99.752 |
364 | 100.000 | 100.000 | 100.000 | 100.000 | 99.754 |
365 | 100.000 | 100.000 | 100.000 | 100.000 | 99.756 |
366 | 100.000 | 100.000 | 100.000 | 100.000 | 99.760 |
367 | 100.000 | 100.000 | 100.000 | 100.000 | 99.761 |
368 | 100.000 | 100.000 | 100.000 | 100.000 | 99.776 |
369 | 100.000 | 100.000 | 100.000 | 100.000 | 99.778 |
370 | 100.000 | 100.000 | 100.000 | 100.000 | 99.782 |
371 | 100.000 | 100.000 | 100.000 | 100.000 | 99.789 |
372 | 100.000 | 100.000 | 100.000 | 100.000 | 99.793 |
373 | 100.000 | 100.000 | 100.000 | 100.000 | 99.797 |
374 | 100.000 | 100.000 | 100.000 | 100.000 | 99.800 |
375 | 100.000 | 100.000 | 100.000 | 100.000 | 99.802 |
376 | 100.000 | 100.000 | 100.000 | 100.000 | 99.806 |
377 | 100.000 | 100.000 | 100.000 | 100.000 | 99.810 |
378 | 100.000 | 100.000 | 100.000 | 100.000 | 99.813 |
379 | 100.000 | 100.000 | 100.000 | 100.000 | 99.817 |
380 | 100.000 | 100.000 | 100.000 | 100.000 | 99.821 |
381 | 100.000 | 100.000 | 100.000 | 100.000 | 99.822 |
382 | 100.000 | 100.000 | 100.000 | 100.000 | 99.826 |
383 | 100.000 | 100.000 | 100.000 | 100.000 | 99.830 |
384 | 100.000 | 100.000 | 100.000 | 100.000 | 99.835 |
385 | 100.000 | 100.000 | 100.000 | 100.000 | 99.837 |
386 | 100.000 | 100.000 | 100.000 | 100.000 | 99.841 |
387 | 100.000 | 100.000 | 100.000 | 100.000 | 99.843 |
388 | 100.000 | 100.000 | 100.000 | 100.000 | 99.847 |
389 | 100.000 | 100.000 | 100.000 | 100.000 | 99.850 |
390 | 100.000 | 100.000 | 100.000 | 100.000 | 99.852 |
391 | 100.000 | 100.000 | 100.000 | 100.000 | 99.854 |
392 | 100.000 | 100.000 | 100.000 | 100.000 | 99.858 |
393 | 100.000 | 100.000 | 100.000 | 100.000 | 99.861 |
394 | 100.000 | 100.000 | 100.000 | 100.000 | 99.863 |
395 | 100.000 | 100.000 | 100.000 | 100.000 | 99.865 |
396 | 100.000 | 100.000 | 100.000 | 100.000 | 99.869 |
397 | 100.000 | 100.000 | 100.000 | 100.000 | 99.872 |
398 | 100.000 | 100.000 | 100.000 | 100.000 | 99.874 |
399 | 100.000 | 100.000 | 100.000 | 100.000 | 99.876 |
400 | 100.000 | 100.000 | 100.000 | 100.000 | 99.880 |
401 | 100.000 | 100.000 | 100.000 | 100.000 | 99.883 |
402 | 100.000 | 100.000 | 100.000 | 100.000 | 99.887 |
403 | 100.000 | 100.000 | 100.000 | 100.000 | 99.889 |
404 | 100.000 | 100.000 | 100.000 | 100.000 | 99.893 |
405 | 100.000 | 100.000 | 100.000 | 100.000 | 99.895 |
406 | 100.000 | 100.000 | 100.000 | 100.000 | 99.896 |
407 | 100.000 | 100.000 | 100.000 | 100.000 | 99.900 |
408 | 100.000 | 100.000 | 100.000 | 100.000 | 99.908 |
409 | 100.000 | 100.000 | 100.000 | 100.000 | 99.911 |
410 | 100.000 | 100.000 | 100.000 | 100.000 | 99.913 |
411 | 100.000 | 100.000 | 100.000 | 100.000 | 99.917 |
412 | 100.000 | 100.000 | 100.000 | 100.000 | 99.920 |
413 | 100.000 | 100.000 | 100.000 | 100.000 | 99.922 |
414 | 100.000 | 100.000 | 100.000 | 100.000 | 99.924 |
415 | 100.000 | 100.000 | 100.000 | 100.000 | 99.928 |
416 | 100.000 | 100.000 | 100.000 | 100.000 | 99.930 |
417 | 100.000 | 100.000 | 100.000 | 100.000 | 99.932 |
418 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
419 | 100.000 | 100.000 | 100.000 | 100.000 | 99.937 |
420 | 100.000 | 100.000 | 100.000 | 100.000 | 99.941 |
421 | 100.000 | 100.000 | 100.000 | 100.000 | 99.945 |
422 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
423 | 100.000 | 100.000 | 100.000 | 100.000 | 99.948 |
424 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
425 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
426 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
427 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
428 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
429 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
430 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
431 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


grep
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
2 | 42.857 | 21.658 | 12.107 | 7.292 | 4.656 |
3 | 54.762 | 30.288 | 17.644 | 10.831 | 6.970 |
4 | 61.905 | 36.717 | 22.337 | 14.074 | 9.191 |
5 | 69.048 | 42.978 | 26.889 | 17.238 | 11.375 |
6 | 73.810 | 48.562 | 31.347 | 20.428 | 13.586 |
7 | 80.952 | 54.992 | 36.039 | 23.672 | 15.807 |
8 | 80.952 | 57.360 | 38.878 | 26.165 | 17.760 |
9 | 80.952 | 59.729 | 41.671 | 28.610 | 19.683 |
10 | 83.333 | 62.944 | 44.721 | 31.124 | 21.617 |
11 | 85.714 | 66.497 | 48.076 | 33.816 | 23.629 |
12 | 90.476 | 71.066 | 51.689 | 36.518 | 25.610 |
13 | 92.857 | 73.773 | 54.247 | 38.711 | 27.376 |
14 | 92.857 | 75.635 | 56.546 | 40.813 | 29.097 |
15 | 95.238 | 78.173 | 58.916 | 42.866 | 30.780 |
16 | 95.238 | 79.357 | 60.605 | 44.587 | 32.308 |
17 | 95.238 | 80.880 | 62.529 | 46.405 | 33.854 |
18 | 100.000 | 85.110 | 65.791 | 48.845 | 35.666 |
19 | 100.000 | 85.787 | 67.034 | 50.276 | 37.025 |
20 | 100.000 | 86.802 | 68.536 | 51.842 | 38.440 |
21 | 100.000 | 87.648 | 69.873 | 53.295 | 39.790 |
22 | 100.000 | 88.663 | 71.305 | 54.775 | 41.139 |
23 | 100.000 | 89.679 | 72.877 | 56.421 | 42.606 |
24 | 100.000 | 90.355 | 73.956 | 57.648 | 43.808 |
25 | 100.000 | 91.371 | 75.411 | 59.144 | 45.156 |
26 | 100.000 | 91.878 | 76.396 | 60.335 | 46.338 |
27 | 100.000 | 92.386 | 77.311 | 61.434 | 47.449 |
28 | 100.000 | 93.232 | 78.555 | 62.753 | 48.677 |
29 | 100.000 | 93.739 | 79.423 | 63.787 | 49.729 |
30 | 100.000 | 93.909 | 80.033 | 64.683 | 50.715 |
31 | 100.000 | 94.247 | 80.760 | 65.621 | 51.699 |
32 | 100.000 | 94.247 | 81.206 | 66.399 | 52.605 |
33 | 100.000 | 94.416 | 81.793 | 67.251 | 53.538 |
34 | 100.000 | 94.585 | 82.309 | 68.012 | 54.400 |
35 | 100.000 | 95.431 | 83.482 | 69.213 | 55.499 |
36 | 100.000 | 95.939 | 84.280 | 70.136 | 56.434 |
37 | 100.000 | 96.277 | 84.913 | 70.945 | 57.305 |
38 | 100.000 | 96.785 | 85.758 | 71.921 | 58.274 |
39 | 100.000 | 97.124 | 86.368 | 72.693 | 59.108 |
40 | 100.000 | 97.462 | 86.978 | 73.455 | 59.920 |
41 | 100.000 | 97.462 | 87.236 | 73.964 | 60.586 |
42 | 100.000 | 97.631 | 87.705 | 74.639 | 61.348 |
43 | 100.000 | 97.631 | 87.940 | 75.106 | 61.967 |
44 | 100.000 | 97.800 | 88.386 | 75.749 | 62.701 |
45 | 100.000 | 97.800 | 88.644 | 76.248 | 63.345 |
46 | 100.000 | 98.308 | 89.395 | 77.079 | 64.170 |
47 | 100.000 | 98.308 | 89.747 | 77.712 | 64.922 |
48 | 100.000 | 98.477 | 90.169 | 78.312 | 65.606 |
49 | 100.000 | 98.477 | 90.286 | 78.623 | 66.091 |
50 | 100.000 | 98.646 | 90.685 | 79.186 | 66.738 |
51 | 100.000 | 98.646 | 90.873 | 79.583 | 67.284 |
52 | 100.000 | 98.646 | 91.084 | 80.006 | 67.848 |
53 | 100.000 | 98.646 | 91.295 | 80.430 | 68.412 |
54 | 100.000 | 98.646 | 91.506 | 80.848 | 68.965 |
55 | 100.000 | 98.985 | 92.046 | 81.486 | 69.634 |
56 | 100.000 | 98.985 | 92.234 | 81.862 | 70.139 |
57 | 100.000 | 98.985 | 92.445 | 82.280 | 70.684 |
58 | 100.000 | 98.985 | 92.586 | 82.596 | 71.143 |
59 | 100.000 | 99.154 | 92.938 | 83.084 | 71.709 |
60 | 100.000 | 99.323 | 93.243 | 83.513 | 72.228 |
61 | 100.000 | 99.323 | 93.430 | 83.888 | 72.730 |
62 | 100.000 | 99.323 | 93.571 | 84.199 | 73.175 |
63 | 100.000 | 99.323 | 93.759 | 84.569 | 73.665 |
64 | 100.000 | 99.323 | 93.876 | 84.848 | 74.085 |
65 | 100.000 | 99.323 | 94.040 | 85.186 | 74.547 |
66 | 100.000 | 99.323 | 94.134 | 85.432 | 74.939 |
67 | 100.000 | 99.323 | 94.252 | 85.700 | 75.335 |
68 | 100.000 | 99.323 | 94.345 | 85.931 | 75.694 |
69 | 100.000 | 99.323 | 94.416 | 86.129 | 76.023 |
70 | 100.000 | 99.323 | 94.557 | 86.419 | 76.423 |
71 | 100.000 | 99.323 | 94.697 | 86.708 | 76.822 |
72 | 100.000 | 99.323 | 94.838 | 86.998 | 77.221 |
73 | 100.000 | 99.323 | 94.955 | 87.255 | 77.593 |
74 | 100.000 | 99.323 | 95.026 | 87.448 | 77.909 |
75 | 100.000 | 99.323 | 95.073 | 87.609 | 78.200 |
76 | 100.000 | 99.323 | 95.143 | 87.802 | 78.516 |
77 | 100.000 | 99.492 | 95.448 | 88.194 | 78.958 |
78 | 100.000 | 99.662 | 95.753 | 88.585 | 79.398 |
79 | 100.000 | 99.662 | 95.847 | 88.805 | 79.731 |
80 | 100.000 | 99.831 | 96.176 | 89.228 | 80.193 |
81 | 100.000 | 99.831 | 96.269 | 89.443 | 80.515 |
82 | 100.000 | 99.831 | 96.340 | 89.625 | 80.809 |
83 | 100.000 | 99.831 | 96.481 | 89.909 | 81.188 |
84 | 100.000 | 99.831 | 96.574 | 90.124 | 81.508 |
85 | 100.000 | 99.831 | 96.715 | 90.408 | 81.887 |
86 | 100.000 | 99.831 | 96.809 | 90.612 | 82.187 |
87 | 100.000 | 99.831 | 96.809 | 90.698 | 82.394 |
88 | 100.000 | 99.831 | 96.903 | 90.901 | 82.689 |
89 | 100.000 | 99.831 | 96.973 | 91.073 | 82.958 |
90 | 100.000 | 99.831 | 97.067 | 91.271 | 83.242 |
91 | 100.000 | 99.831 | 97.137 | 91.437 | 83.499 |
92 | 100.000 | 99.831 | 97.208 | 91.604 | 83.756 |
93 | 100.000 | 99.831 | 97.255 | 91.738 | 83.986 |
94 | 100.000 | 99.831 | 97.302 | 91.872 | 84.215 |
95 | 100.000 | 99.831 | 97.396 | 92.070 | 84.496 |
96 | 100.000 | 99.831 | 97.466 | 92.231 | 84.742 |
97 | 100.000 | 99.831 | 97.536 | 92.392 | 84.986 |
98 | 100.000 | 99.831 | 97.583 | 92.521 | 85.204 |
99 | 100.000 | 99.831 | 97.677 | 92.714 | 85.473 |
100 | 100.000 | 99.831 | 97.771 | 92.907 | 85.741 |
101 | 100.000 | 99.831 | 97.818 | 93.030 | 85.948 |
102 | 100.000 | 99.831 | 97.888 | 93.185 | 86.181 |
103 | 100.000 | 99.831 | 97.935 | 93.309 | 86.386 |
104 | 100.000 | 99.831 | 98.006 | 93.464 | 86.617 |
105 | 100.000 | 99.831 | 98.076 | 93.620 | 86.848 |
106 | 100.000 | 99.831 | 98.146 | 93.775 | 87.078 |
107 | 100.000 | 99.831 | 98.217 | 93.931 | 87.307 |
108 | 100.000 | 99.831 | 98.264 | 94.049 | 87.501 |
109 | 100.000 | 99.831 | 98.264 | 94.113 | 87.656 |
110 | 100.000 | 99.831 | 98.334 | 94.263 | 87.873 |
111 | 100.000 | 100.000 | 98.592 | 94.558 | 88.178 |
112 | 100.000 | 100.000 | 98.639 | 94.676 | 88.368 |
113 | 100.000 | 100.000 | 98.710 | 94.826 | 88.585 |
114 | 100.000 | 100.000 | 98.756 | 94.939 | 88.768 |
115 | 100.000 | 100.000 | 98.780 | 95.019 | 88.923 |
116 | 100.000 | 100.000 | 98.827 | 95.132 | 89.103 |
117 | 100.000 | 100.000 | 98.897 | 95.276 | 89.308 |
118 | 100.000 | 100.000 | 98.921 | 95.357 | 89.461 |
119 | 100.000 | 100.000 | 98.944 | 95.437 | 89.615 |
120 | 100.000 | 100.000 | 99.015 | 95.582 | 89.820 |
121 | 100.000 | 100.000 | 99.061 | 95.695 | 89.998 |
122 | 100.000 | 100.000 | 99.108 | 95.807 | 90.175 |
123 | 100.000 | 100.000 | 99.155 | 95.920 | 90.353 |
124 | 100.000 | 100.000 | 99.202 | 96.032 | 90.530 |
125 | 100.000 | 100.000 | 99.226 | 96.107 | 90.673 |
126 | 100.000 | 100.000 | 99.273 | 96.215 | 90.841 |
127 | 100.000 | 100.000 | 99.273 | 96.258 | 90.955 |
128 | 100.000 | 100.000 | 99.296 | 96.333 | 91.094 |
129 | 100.000 | 100.000 | 99.320 | 96.408 | 91.233 |
130 | 100.000 | 100.000 | 99.366 | 96.515 | 91.397 |
131 | 100.000 | 100.000 | 99.413 | 96.622 | 91.562 |
132 | 100.000 | 100.000 | 99.460 | 96.729 | 91.727 |
133 | 100.000 | 100.000 | 99.460 | 96.762 | 91.821 |
134 | 100.000 | 100.000 | 99.484 | 96.831 | 91.947 |
135 | 100.000 | 100.000 | 99.484 | 96.869 | 92.047 |
136 | 100.000 | 100.000 | 99.507 | 96.939 | 92.172 |
137 | 100.000 | 100.000 | 99.507 | 96.976 | 92.272 |
138 | 100.000 | 100.000 | 99.531 | 97.046 | 92.398 |
139 | 100.000 | 100.000 | 99.554 | 97.115 | 92.524 |
140 | 100.000 | 100.000 | 99.601 | 97.217 | 92.673 |
141 | 100.000 | 100.000 | 99.648 | 97.319 | 92.823 |
142 | 100.000 | 100.000 | 99.672 | 97.384 | 92.938 |
143 | 100.000 | 100.000 | 99.672 | 97.421 | 93.034 |
144 | 100.000 | 100.000 | 99.672 | 97.453 | 93.121 |
145 | 100.000 | 100.000 | 99.672 | 97.485 | 93.208 |
146 | 100.000 | 100.000 | 99.672 | 97.512 | 93.287 |
147 | 100.000 | 100.000 | 99.672 | 97.544 | 93.374 |
148 | 100.000 | 100.000 | 99.672 | 97.577 | 93.461 |
149 | 100.000 | 100.000 | 99.672 | 97.609 | 93.548 |
150 | 100.000 | 100.000 | 99.672 | 97.641 | 93.635 |
151 | 100.000 | 100.000 | 99.695 | 97.705 | 93.746 |
152 | 100.000 | 100.000 | 99.695 | 97.732 | 93.822 |
153 | 100.000 | 100.000 | 99.695 | 97.759 | 93.898 |
154 | 100.000 | 100.000 | 99.695 | 97.786 | 93.973 |
155 | 100.000 | 100.000 | 99.695 | 97.812 | 94.049 |
156 | 100.000 | 100.000 | 99.718 | 97.871 | 94.149 |
157 | 100.000 | 100.000 | 99.718 | 97.898 | 94.223 |
158 | 100.000 | 100.000 | 99.742 | 97.957 | 94.323 |
159 | 100.000 | 100.000 | 99.765 | 98.016 | 94.421 |
160 | 100.000 | 100.000 | 99.765 | 98.043 | 94.493 |
161 | 100.000 | 100.000 | 99.789 | 98.102 | 94.591 |
162 | 100.000 | 100.000 | 99.812 | 98.161 | 94.689 |
163 | 100.000 | 100.000 | 99.836 | 98.220 | 94.787 |
164 | 100.000 | 100.000 | 99.836 | 98.247 | 94.859 |
165 | 100.000 | 100.000 | 99.836 | 98.274 | 94.931 |
166 | 100.000 | 100.000 | 99.836 | 98.300 | 95.003 |
167 | 100.000 | 100.000 | 99.859 | 98.359 | 95.101 |
168 | 100.000 | 100.000 | 99.883 | 98.418 | 95.199 |
169 | 100.000 | 100.000 | 99.906 | 98.477 | 95.297 |
170 | 100.000 | 100.000 | 99.906 | 98.504 | 95.370 |
171 | 100.000 | 100.000 | 99.906 | 98.526 | 95.432 |
172 | 100.000 | 100.000 | 99.906 | 98.547 | 95.495 |
173 | 100.000 | 100.000 | 99.906 | 98.568 | 95.558 |
174 | 100.000 | 100.000 | 99.906 | 98.590 | 95.621 |
175 | 100.000 | 100.000 | 99.906 | 98.611 | 95.684 |
176 | 100.000 | 100.000 | 99.906 | 98.633 | 95.745 |
177 | 100.000 | 100.000 | 99.906 | 98.654 | 95.806 |
178 | 100.000 | 100.000 | 99.906 | 98.676 | 95.865 |
179 | 100.000 | 100.000 | 99.906 | 98.697 | 95.924 |
180 | 100.000 | 100.000 | 99.906 | 98.719 | 95.984 |
181 | 100.000 | 100.000 | 99.906 | 98.740 | 96.043 |
182 | 100.000 | 100.000 | 99.906 | 98.761 | 96.102 |
183 | 100.000 | 100.000 | 99.906 | 98.783 | 96.161 |
184 | 100.000 | 100.000 | 99.906 | 98.804 | 96.220 |
185 | 100.000 | 100.000 | 99.906 | 98.826 | 96.279 |
186 | 100.000 | 100.000 | 99.930 | 98.879 | 96.364 |
187 | 100.000 | 100.000 | 99.930 | 98.901 | 96.424 |
188 | 100.000 | 100.000 | 99.930 | 98.922 | 96.483 |
189 | 100.000 | 100.000 | 99.930 | 98.944 | 96.542 |
190 | 100.000 | 100.000 | 99.930 | 98.965 | 96.601 |
191 | 100.000 | 100.000 | 99.930 | 98.987 | 96.660 |
192 | 100.000 | 100.000 | 99.930 | 99.008 | 96.719 |
193 | 100.000 | 100.000 | 99.953 | 99.062 | 96.805 |
194 | 100.000 | 100.000 | 99.977 | 99.115 | 96.888 |
195 | 100.000 | 100.000 | 100.000 | 99.169 | 96.971 |
196 | 100.000 | 100.000 | 100.000 | 99.185 | 97.019 |
197 | 100.000 | 100.000 | 100.000 | 99.201 | 97.067 |
198 | 100.000 | 100.000 | 100.000 | 99.217 | 97.115 |
199 | 100.000 | 100.000 | 100.000 | 99.233 | 97.163 |
200 | 100.000 | 100.000 | 100.000 | 99.249 | 97.211 |
201 | 100.000 | 100.000 | 100.000 | 99.265 | 97.259 |
202 | 100.000 | 100.000 | 100.000 | 99.282 | 97.306 |
203 | 100.000 | 100.000 | 100.000 | 99.298 | 97.352 |
204 | 100.000 | 100.000 | 100.000 | 99.314 | 97.398 |
205 | 100.000 | 100.000 | 100.000 | 99.330 | 97.444 |
206 | 100.000 | 100.000 | 100.000 | 99.346 | 97.491 |
207 | 100.000 | 100.000 | 100.000 | 99.362 | 97.537 |
208 | 100.000 | 100.000 | 100.000 | 99.373 | 97.572 |
209 | 100.000 | 100.000 | 100.000 | 99.383 | 97.607 |
210 | 100.000 | 100.000 | 100.000 | 99.394 | 97.642 |
211 | 100.000 | 100.000 | 100.000 | 99.405 | 97.676 |
212 | 100.000 | 100.000 | 100.000 | 99.416 | 97.709 |
213 | 100.000 | 100.000 | 100.000 | 99.426 | 97.742 |
214 | 100.000 | 100.000 | 100.000 | 99.437 | 97.775 |
215 | 100.000 | 100.000 | 100.000 | 99.448 | 97.809 |
216 | 100.000 | 100.000 | 100.000 | 99.458 | 97.842 |
217 | 100.000 | 100.000 | 100.000 | 99.469 | 97.875 |
218 | 100.000 | 100.000 | 100.000 | 99.480 | 97.909 |
219 | 100.000 | 100.000 | 100.000 | 99.491 | 97.942 |
220 | 100.000 | 100.000 | 100.000 | 99.501 | 97.975 |
221 | 100.000 | 100.000 | 100.000 | 99.512 | 98.007 |
222 | 100.000 | 100.000 | 100.000 | 99.523 | 98.038 |
223 | 100.000 | 100.000 | 100.000 | 99.534 | 98.069 |
224 | 100.000 | 100.000 | 100.000 | 99.544 | 98.101 |
225 | 100.000 | 100.000 | 100.000 | 99.555 | 98.132 |
226 | 100.000 | 100.000 | 100.000 | 99.566 | 98.164 |
227 | 100.000 | 100.000 | 100.000 | 99.576 | 98.195 |
228 | 100.000 | 100.000 | 100.000 | 99.587 | 98.227 |
229 | 100.000 | 100.000 | 100.000 | 99.598 | 98.258 |
230 | 100.000 | 100.000 | 100.000 | 99.609 | 98.289 |
231 | 100.000 | 100.000 | 100.000 | 99.619 | 98.321 |
232 | 100.000 | 100.000 | 100.000 | 99.630 | 98.352 |
233 | 100.000 | 100.000 | 100.000 | 99.641 | 98.384 |
234 | 100.000 | 100.000 | 100.000 | 99.651 | 98.415 |
235 | 100.000 | 100.000 | 100.000 | 99.662 | 98.447 |
236 | 100.000 | 100.000 | 100.000 | 99.673 | 98.478 |
237 | 100.000 | 100.000 | 100.000 | 99.684 | 98.510 |
238 | 100.000 | 100.000 | 100.000 | 99.684 | 98.524 |
239 | 100.000 | 100.000 | 100.000 | 99.689 | 98.547 |
240 | 100.000 | 100.000 | 100.000 | 99.694 | 98.567 |
241 | 100.000 | 100.000 | 100.000 | 99.700 | 98.587 |
242 | 100.000 | 100.000 | 100.000 | 99.705 | 98.608 |
243 | 100.000 | 100.000 | 100.000 | 99.710 | 98.628 |
244 | 100.000 | 100.000 | 100.000 | 99.716 | 98.648 |
245 | 100.000 | 100.000 | 100.000 | 99.716 | 98.659 |
246 | 100.000 | 100.000 | 100.000 | 99.721 | 98.678 |
247 | 100.000 | 100.000 | 100.000 | 99.727 | 98.696 |
248 | 100.000 | 100.000 | 100.000 | 99.732 | 98.715 |
249 | 100.000 | 100.000 | 100.000 | 99.737 | 98.733 |
250 | 100.000 | 100.000 | 100.000 | 99.743 | 98.750 |
251 | 100.000 | 100.000 | 100.000 | 99.748 | 98.767 |
252 | 100.000 | 100.000 | 100.000 | 99.753 | 98.783 |
253 | 100.000 | 100.000 | 100.000 | 99.759 | 98.800 |
254 | 100.000 | 100.000 | 100.000 | 99.764 | 98.817 |
255 | 100.000 | 100.000 | 100.000 | 99.769 | 98.833 |
256 | 100.000 | 100.000 | 100.000 | 99.775 | 98.850 |
257 | 100.000 | 100.000 | 100.000 | 99.780 | 98.866 |
258 | 100.000 | 100.000 | 100.000 | 99.786 | 98.883 |
259 | 100.000 | 100.000 | 100.000 | 99.791 | 98.900 |
260 | 100.000 | 100.000 | 100.000 | 99.796 | 98.916 |
261 | 100.000 | 100.000 | 100.000 | 99.802 | 98.933 |
262 | 100.000 | 100.000 | 100.000 | 99.807 | 98.950 |
263 | 100.000 | 100.000 | 100.000 | 99.812 | 98.966 |
264 | 100.000 | 100.000 | 100.000 | 99.818 | 98.983 |
265 | 100.000 | 100.000 | 100.000 | 99.823 | 99.000 |
266 | 100.000 | 100.000 | 100.000 | 99.828 | 99.016 |
267 | 100.000 | 100.000 | 100.000 | 99.834 | 99.033 |
268 | 100.000 | 100.000 | 100.000 | 99.839 | 99.050 |
269 | 100.000 | 100.000 | 100.000 | 99.845 | 99.066 |
270 | 100.000 | 100.000 | 100.000 | 99.850 | 99.083 |
271 | 100.000 | 100.000 | 100.000 | 99.855 | 99.099 |
272 | 100.000 | 100.000 | 100.000 | 99.861 | 99.116 |
273 | 100.000 | 100.000 | 100.000 | 99.866 | 99.133 |
274 | 100.000 | 100.000 | 100.000 | 99.871 | 99.149 |
275 | 100.000 | 100.000 | 100.000 | 99.877 | 99.166 |
276 | 100.000 | 100.000 | 100.000 | 99.882 | 99.183 |
277 | 100.000 | 100.000 | 100.000 | 99.887 | 99.199 |
278 | 100.000 | 100.000 | 100.000 | 99.893 | 99.216 |
279 | 100.000 | 100.000 | 100.000 | 99.898 | 99.233 |
280 | 100.000 | 100.000 | 100.000 | 99.903 | 99.249 |
281 | 100.000 | 100.000 | 100.000 | 99.909 | 99.266 |
282 | 100.000 | 100.000 | 100.000 | 99.914 | 99.283 |
283 | 100.000 | 100.000 | 100.000 | 99.920 | 99.299 |
284 | 100.000 | 100.000 | 100.000 | 99.925 | 99.316 |
285 | 100.000 | 100.000 | 100.000 | 99.930 | 99.332 |
286 | 100.000 | 100.000 | 100.000 | 99.936 | 99.349 |
287 | 100.000 | 100.000 | 100.000 | 99.941 | 99.366 |
288 | 100.000 | 100.000 | 100.000 | 99.946 | 99.382 |
289 | 100.000 | 100.000 | 100.000 | 99.952 | 99.399 |
290 | 100.000 | 100.000 | 100.000 | 99.957 | 99.416 |
291 | 100.000 | 100.000 | 100.000 | 99.962 | 99.432 |
292 | 100.000 | 100.000 | 100.000 | 99.968 | 99.449 |
293 | 100.000 | 100.000 | 100.000 | 99.973 | 99.466 |
294 | 100.000 | 100.000 | 100.000 | 99.979 | 99.482 |
295 | 100.000 | 100.000 | 100.000 | 99.984 | 99.499 |
296 | 100.000 | 100.000 | 100.000 | 99.989 | 99.516 |
297 | 100.000 | 100.000 | 100.000 | 99.995 | 99.532 |
298 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
299 | 100.000 | 100.000 | 100.000 | 100.000 | 99.556 |
300 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.571 |
302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.578 |
303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.586 |
304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.593 |
305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.608 |
307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.643 |
312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.650 |
314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.669 |
319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.676 |
321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.702 |
328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.710 |
330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.713 |
331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.717 |
332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.721 |
333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.724 |
334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.728 |
335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.732 |
336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.736 |
337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.739 |
338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.743 |
339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.747 |
340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.750 |
341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.754 |
342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.758 |
343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.761 |
344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.765 |
345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.769 |
346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.773 |
347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.776 |
348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.780 |
349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.784 |
350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.787 |
351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.791 |
352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.795 |
353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.798 |
354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.802 |
355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.806 |
356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.810 |
357 | 100.000 | 100.000 | 100.000 | 100.000 | 99.813 |
358 | 100.000 | 100.000 | 100.000 | 100.000 | 99.817 |
359 | 100.000 | 100.000 | 100.000 | 100.000 | 99.821 |
360 | 100.000 | 100.000 | 100.000 | 100.000 | 99.824 |
361 | 100.000 | 100.000 | 100.000 | 100.000 | 99.828 |
362 | 100.000 | 100.000 | 100.000 | 100.000 | 99.832 |
363 | 100.000 | 100.000 | 100.000 | 100.000 | 99.835 |
364 | 100.000 | 100.000 | 100.000 | 100.000 | 99.839 |
365 | 100.000 | 100.000 | 100.000 | 100.000 | 99.843 |
366 | 100.000 | 100.000 | 100.000 | 100.000 | 99.847 |
367 | 100.000 | 100.000 | 100.000 | 100.000 | 99.850 |
368 | 100.000 | 100.000 | 100.000 | 100.000 | 99.854 |
369 | 100.000 | 100.000 | 100.000 | 100.000 | 99.858 |
370 | 100.000 | 100.000 | 100.000 | 100.000 | 99.861 |
371 | 100.000 | 100.000 | 100.000 | 100.000 | 99.865 |
372 | 100.000 | 100.000 | 100.000 | 100.000 | 99.869 |
373 | 100.000 | 100.000 | 100.000 | 100.000 | 99.872 |
374 | 100.000 | 100.000 | 100.000 | 100.000 | 99.876 |
375 | 100.000 | 100.000 | 100.000 | 100.000 | 99.880 |
376 | 100.000 | 100.000 | 100.000 | 100.000 | 99.883 |
377 | 100.000 | 100.000 | 100.000 | 100.000 | 99.887 |
378 | 100.000 | 100.000 | 100.000 | 100.000 | 99.891 |
379 | 100.000 | 100.000 | 100.000 | 100.000 | 99.895 |
380 | 100.000 | 100.000 | 100.000 | 100.000 | 99.896 |
381 | 100.000 | 100.000 | 100.000 | 100.000 | 99.898 |
382 | 100.000 | 100.000 | 100.000 | 100.000 | 99.900 |
383 | 100.000 | 100.000 | 100.000 | 100.000 | 99.902 |
384 | 100.000 | 100.000 | 100.000 | 100.000 | 99.904 |
385 | 100.000 | 100.000 | 100.000 | 100.000 | 99.906 |
386 | 100.000 | 100.000 | 100.000 | 100.000 | 99.908 |
387 | 100.000 | 100.000 | 100.000 | 100.000 | 99.909 |
388 | 100.000 | 100.000 | 100.000 | 100.000 | 99.911 |
389 | 100.000 | 100.000 | 100.000 | 100.000 | 99.913 |
390 | 100.000 | 100.000 | 100.000 | 100.000 | 99.915 |
391 | 100.000 | 100.000 | 100.000 | 100.000 | 99.917 |
392 | 100.000 | 100.000 | 100.000 | 100.000 | 99.919 |
393 | 100.000 | 100.000 | 100.000 | 100.000 | 99.920 |
394 | 100.000 | 100.000 | 100.000 | 100.000 | 99.922 |
395 | 100.000 | 100.000 | 100.000 | 100.000 | 99.924 |
396 | 100.000 | 100.000 | 100.000 | 100.000 | 99.926 |
397 | 100.000 | 100.000 | 100.000 | 100.000 | 99.928 |
398 | 100.000 | 100.000 | 100.000 | 100.000 | 99.930 |
399 | 100.000 | 100.000 | 100.000 | 100.000 | 99.932 |
400 | 100.000 | 100.000 | 100.000 | 100.000 | 99.933 |
401 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
402 | 100.000 | 100.000 | 100.000 | 100.000 | 99.937 |
403 | 100.000 | 100.000 | 100.000 | 100.000 | 99.939 |
404 | 100.000 | 100.000 | 100.000 | 100.000 | 99.941 |
405 | 100.000 | 100.000 | 100.000 | 100.000 | 99.943 |
406 | 100.000 | 100.000 | 100.000 | 100.000 | 99.945 |
407 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
408 | 100.000 | 100.000 | 100.000 | 100.000 | 99.948 |
409 | 100.000 | 100.000 | 100.000 | 100.000 | 99.950 |
410 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
411 | 100.000 | 100.000 | 100.000 | 100.000 | 99.954 |
412 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
413 | 100.000 | 100.000 | 100.000 | 100.000 | 99.957 |
414 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
415 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
416 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
417 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
418 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
419 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
420 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
421 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
422 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
423 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
424 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
425 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
426 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
427 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
428 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
429 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
430 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
431 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


gzip
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
7 | 94.737 | 86.022 | 76.557 | 67.361 | 58.835 |
8 | 97.368 | 89.555 | 80.347 | 71.159 | 62.542 |
9 | 100.000 | 93.088 | 84.092 | 74.853 | 66.091 |
10 | 100.000 | 93.856 | 85.664 | 77.056 | 68.721 |
11 | 100.000 | 94.624 | 87.147 | 79.050 | 71.033 |
12 | 100.000 | 95.392 | 88.616 | 81.004 | 73.276 |
13 | 100.000 | 96.160 | 90.100 | 82.990 | 75.565 |
14 | 100.000 | 96.928 | 91.569 | 84.927 | 77.757 |
15 | 100.000 | 97.696 | 93.038 | 86.869 | 79.963 |
16 | 100.000 | 98.464 | 94.506 | 88.806 | 82.154 |
17 | 100.000 | 99.232 | 96.034 | 90.906 | 84.627 |
18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |


gzip
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.347 | 62.970 | 52.942 | 44.301 |
5 | 89.474 | 78.648 | 68.243 | 58.664 | 50.095 |
6 | 92.105 | 82.488 | 72.738 | 63.484 | 54.998 |
7 | 94.737 | 86.022 | 76.601 | 67.465 | 58.989 |
8 | 97.368 | 89.555 | 80.376 | 71.230 | 62.647 |
9 | 100.000 | 93.088 | 84.122 | 74.918 | 66.180 |
10 | 100.000 | 93.856 | 85.679 | 77.095 | 68.783 |
11 | 100.000 | 94.624 | 87.206 | 79.194 | 71.258 |
12 | 100.000 | 95.392 | 88.704 | 81.219 | 73.611 |
13 | 100.000 | 96.160 | 90.203 | 83.249 | 75.983 |
14 | 100.000 | 96.928 | 91.686 | 85.236 | 78.272 |
15 | 100.000 | 97.696 | 93.155 | 87.177 | 80.477 |
16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |


gzip
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 78.947 | 61.290 | 47.474 | 36.905 | 28.873 |
3 | 84.211 | 69.585 | 56.962 | 46.428 | 37.793 |
4 | 86.842 | 75.115 | 64.233 | 54.358 | 45.643 |
5 | 89.474 | 78.802 | 68.478 | 58.904 | 50.294 |
6 | 92.105 | 82.488 | 72.723 | 63.448 | 54.940 |
7 | 94.737 | 86.022 | 76.572 | 67.396 | 58.886 |
8 | 97.368 | 89.555 | 80.405 | 71.303 | 62.763 |
9 | 100.000 | 93.088 | 84.151 | 74.989 | 66.290 |
10 | 100.000 | 93.856 | 85.679 | 77.093 | 68.778 |
11 | 100.000 | 94.624 | 87.206 | 79.194 | 71.260 |
12 | 100.000 | 95.392 | 88.719 | 81.261 | 73.688 |
13 | 100.000 | 96.160 | 90.217 | 83.294 | 76.066 |
14 | 100.000 | 96.928 | 91.686 | 85.240 | 78.285 |
15 | 100.000 | 97.696 | 93.155 | 87.181 | 80.490 |
16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |


gzip
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 89.474 | 79.724 | 70.535 | 61.935 | 54.002 |
7 | 92.105 | 83.257 | 74.354 | 65.811 | 57.839 |
8 | 94.737 | 86.790 | 78.173 | 69.682 | 61.660 |
9 | 97.368 | 90.323 | 81.933 | 73.411 | 65.262 |
10 | 100.000 | 93.856 | 85.693 | 77.135 | 68.854 |
11 | 100.000 | 94.624 | 87.221 | 79.237 | 71.335 |
12 | 100.000 | 95.392 | 88.719 | 81.272 | 73.719 |
13 | 100.000 | 96.160 | 90.217 | 83.294 | 76.066 |
14 | 100.000 | 96.928 | 91.686 | 85.240 | 78.285 |
15 | 100.000 | 97.696 | 93.155 | 87.181 | 80.490 |
16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |


gzip
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.347 | 62.970 | 52.942 | 44.301 |
5 | 89.474 | 78.648 | 68.199 | 58.561 | 49.940 |
6 | 92.105 | 82.335 | 72.429 | 63.075 | 54.547 |
7 | 94.737 | 86.022 | 76.601 | 67.463 | 58.983 |
8 | 97.368 | 89.555 | 80.479 | 71.477 | 63.025 |
9 | 100.000 | 93.088 | 84.269 | 75.277 | 66.739 |
10 | 100.000 | 93.702 | 85.517 | 77.042 | 68.885 |
11 | 100.000 | 94.316 | 86.722 | 78.683 | 70.814 |
12 | 100.000 | 94.931 | 87.985 | 80.480 | 73.005 |
13 | 100.000 | 95.545 | 89.189 | 82.120 | 74.929 |
14 | 100.000 | 96.006 | 90.115 | 83.420 | 76.510 |
15 | 100.000 | 96.467 | 91.040 | 84.700 | 78.027 |
16 | 100.000 | 96.928 | 91.965 | 85.980 | 79.541 |
17 | 100.000 | 97.389 | 92.891 | 87.260 | 81.056 |
18 | 100.000 | 97.849 | 93.816 | 88.544 | 82.587 |
19 | 100.000 | 98.157 | 94.477 | 89.530 | 83.849 |
20 | 100.000 | 98.464 | 95.138 | 90.500 | 85.057 |
21 | 100.000 | 98.771 | 95.799 | 91.470 | 86.268 |
22 | 100.000 | 99.078 | 96.460 | 92.443 | 87.489 |
23 | 100.000 | 99.386 | 97.121 | 93.419 | 88.717 |
24 | 100.000 | 99.539 | 97.488 | 94.005 | 89.501 |
25 | 100.000 | 99.693 | 97.870 | 94.628 | 90.342 |
26 | 100.000 | 99.846 | 98.237 | 95.212 | 91.119 |
27 | 100.000 | 100.000 | 98.605 | 95.797 | 91.896 |
28 | 100.000 | 100.000 | 98.707 | 96.073 | 92.373 |
29 | 100.000 | 100.000 | 98.796 | 96.310 | 92.782 |
30 | 100.000 | 100.000 | 98.884 | 96.555 | 93.220 |
31 | 100.000 | 100.000 | 98.972 | 96.793 | 93.630 |
32 | 100.000 | 100.000 | 99.060 | 97.034 | 94.054 |
33 | 100.000 | 100.000 | 99.163 | 97.318 | 94.556 |
34 | 100.000 | 100.000 | 99.222 | 97.489 | 94.871 |
35 | 100.000 | 100.000 | 99.324 | 97.764 | 95.346 |
36 | 100.000 | 100.000 | 99.412 | 98.002 | 95.757 |
37 | 100.000 | 100.000 | 99.471 | 98.164 | 96.046 |
38 | 100.000 | 100.000 | 99.559 | 98.401 | 96.456 |
39 | 100.000 | 100.000 | 99.618 | 98.564 | 96.743 |
40 | 100.000 | 100.000 | 99.706 | 98.801 | 97.153 |
41 | 100.000 | 100.000 | 99.765 | 98.963 | 97.441 |
42 | 100.000 | 100.000 | 99.824 | 99.126 | 97.728 |
43 | 100.000 | 100.000 | 99.882 | 99.288 | 98.015 |
44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |


gzip
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
8 | 97.368 | 89.555 | 80.552 | 71.650 | 63.284 |
9 | 100.000 | 93.088 | 84.342 | 75.451 | 67.002 |
10 | 100.000 | 93.702 | 85.590 | 77.218 | 69.154 |
11 | 100.000 | 94.316 | 86.810 | 78.916 | 71.203 |
12 | 100.000 | 94.931 | 88.014 | 80.568 | 73.164 |
13 | 100.000 | 95.545 | 89.219 | 82.207 | 75.088 |
14 | 100.000 | 96.160 | 90.423 | 83.852 | 77.030 |
15 | 100.000 | 96.774 | 91.627 | 85.483 | 78.929 |
16 | 100.000 | 97.389 | 92.832 | 87.122 | 80.852 |
17 | 100.000 | 98.003 | 94.036 | 88.749 | 82.737 |
18 | 100.000 | 98.464 | 94.962 | 90.037 | 84.281 |
19 | 100.000 | 98.771 | 95.623 | 91.007 | 85.489 |
20 | 100.000 | 98.925 | 96.005 | 91.630 | 86.331 |
21 | 100.000 | 99.078 | 96.387 | 92.267 | 87.219 |
22 | 100.000 | 99.232 | 96.769 | 92.900 | 88.095 |
23 | 100.000 | 99.386 | 97.150 | 93.523 | 88.938 |
24 | 100.000 | 99.539 | 97.532 | 94.147 | 89.786 |
25 | 100.000 | 99.693 | 97.914 | 94.770 | 90.627 |
26 | 100.000 | 99.846 | 98.296 | 95.393 | 91.468 |
27 | 100.000 | 100.000 | 98.678 | 96.016 | 92.310 |
28 | 100.000 | 100.000 | 98.766 | 96.257 | 92.736 |
29 | 100.000 | 100.000 | 98.854 | 96.495 | 93.146 |
30 | 100.000 | 100.000 | 98.942 | 96.732 | 93.556 |
31 | 100.000 | 100.000 | 99.031 | 96.969 | 93.965 |
32 | 100.000 | 100.000 | 99.119 | 97.207 | 94.377 |
33 | 100.000 | 100.000 | 99.207 | 97.444 | 94.787 |
34 | 100.000 | 100.000 | 99.295 | 97.681 | 95.197 |
35 | 100.000 | 100.000 | 99.383 | 97.919 | 95.607 |
36 | 100.000 | 100.000 | 99.471 | 98.156 | 96.018 |
37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.305 |
38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.593 |
39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.880 |
40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.168 |
41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.455 |
42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |


gzip
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
7 | 94.737 | 86.022 | 76.675 | 67.637 | 59.246 |
8 | 97.368 | 89.555 | 80.567 | 71.684 | 63.334 |
9 | 100.000 | 93.088 | 84.327 | 75.417 | 66.951 |
10 | 100.000 | 93.702 | 85.576 | 77.180 | 69.091 |
11 | 100.000 | 94.316 | 86.795 | 78.878 | 71.139 |
12 | 100.000 | 94.931 | 87.999 | 80.531 | 73.107 |
13 | 100.000 | 95.545 | 89.204 | 82.172 | 75.036 |
14 | 100.000 | 96.160 | 90.408 | 83.813 | 76.966 |
15 | 100.000 | 96.774 | 91.613 | 85.451 | 78.884 |
16 | 100.000 | 97.389 | 92.817 | 87.082 | 80.782 |
17 | 100.000 | 98.003 | 94.022 | 88.709 | 82.665 |
18 | 100.000 | 98.618 | 95.226 | 90.336 | 84.548 |
19 | 100.000 | 98.771 | 95.623 | 91.007 | 85.488 |
20 | 100.000 | 98.925 | 96.005 | 91.634 | 86.344 |
21 | 100.000 | 99.078 | 96.387 | 92.259 | 87.192 |
22 | 100.000 | 99.232 | 96.769 | 92.884 | 88.041 |
23 | 100.000 | 99.386 | 97.150 | 93.506 | 88.883 |
24 | 100.000 | 99.539 | 97.532 | 94.129 | 89.724 |
25 | 100.000 | 99.693 | 97.914 | 94.752 | 90.565 |
26 | 100.000 | 99.846 | 98.296 | 95.375 | 91.408 |
27 | 100.000 | 100.000 | 98.678 | 95.997 | 92.248 |
28 | 100.000 | 100.000 | 98.766 | 96.245 | 92.692 |
29 | 100.000 | 100.000 | 98.854 | 96.486 | 93.116 |
30 | 100.000 | 100.000 | 98.942 | 96.728 | 93.541 |
31 | 100.000 | 100.000 | 99.031 | 96.965 | 93.951 |
32 | 100.000 | 100.000 | 99.119 | 97.202 | 94.361 |
33 | 100.000 | 100.000 | 99.207 | 97.440 | 94.770 |
34 | 100.000 | 100.000 | 99.295 | 97.677 | 95.180 |
35 | 100.000 | 100.000 | 99.383 | 97.915 | 95.590 |
36 | 100.000 | 100.000 | 99.471 | 98.152 | 96.002 |
37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.304 |
38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.592 |
39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.879 |
40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.167 |
41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.454 |
42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |


gzip
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
7 | 94.737 | 85.868 | 76.293 | 67.053 | 58.532 |
8 | 97.368 | 89.401 | 80.185 | 71.096 | 62.610 |
9 | 100.000 | 93.088 | 84.224 | 75.173 | 66.585 |
10 | 100.000 | 93.702 | 85.532 | 77.076 | 68.937 |
11 | 100.000 | 94.316 | 86.780 | 78.837 | 71.071 |
12 | 100.000 | 94.931 | 87.985 | 80.493 | 73.042 |
13 | 100.000 | 95.545 | 89.189 | 82.136 | 74.978 |
14 | 100.000 | 96.160 | 90.394 | 83.777 | 76.909 |
15 | 100.000 | 96.774 | 91.598 | 85.410 | 78.812 |
16 | 100.000 | 97.389 | 92.803 | 87.037 | 80.696 |
17 | 100.000 | 98.003 | 94.007 | 88.664 | 82.580 |
18 | 100.000 | 98.464 | 94.932 | 89.950 | 84.117 |
19 | 100.000 | 98.771 | 95.593 | 90.920 | 85.327 |
20 | 100.000 | 98.925 | 95.990 | 91.591 | 86.267 |
21 | 100.000 | 99.078 | 96.387 | 92.263 | 87.206 |
22 | 100.000 | 99.232 | 96.769 | 92.890 | 88.062 |
23 | 100.000 | 99.386 | 97.150 | 93.514 | 88.911 |
24 | 100.000 | 99.539 | 97.532 | 94.137 | 89.753 |
25 | 100.000 | 99.693 | 97.914 | 94.760 | 90.594 |
26 | 100.000 | 99.846 | 98.296 | 95.383 | 91.434 |
27 | 100.000 | 100.000 | 98.678 | 96.006 | 92.274 |
28 | 100.000 | 100.000 | 98.766 | 96.253 | 92.719 |
29 | 100.000 | 100.000 | 98.854 | 96.495 | 93.144 |
30 | 100.000 | 100.000 | 98.942 | 96.732 | 93.556 |
31 | 100.000 | 100.000 | 99.031 | 96.969 | 93.968 |
32 | 100.000 | 100.000 | 99.119 | 97.207 | 94.378 |
33 | 100.000 | 100.000 | 99.207 | 97.444 | 94.787 |
34 | 100.000 | 100.000 | 99.295 | 97.681 | 95.197 |
35 | 100.000 | 100.000 | 99.383 | 97.919 | 95.607 |
36 | 100.000 | 100.000 | 99.471 | 98.156 | 96.017 |
37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.305 |
38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.593 |
39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.880 |
40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.168 |
41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.455 |
42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |


gzip
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
6 | 92.105 | 82.181 | 72.150 | 62.732 | 54.193 |
7 | 94.737 | 86.022 | 76.542 | 67.329 | 58.789 |
8 | 97.368 | 89.555 | 80.288 | 71.013 | 62.310 |
9 | 100.000 | 93.088 | 84.122 | 74.908 | 66.153 |
10 | 100.000 | 93.856 | 85.693 | 77.119 | 68.805 |
11 | 100.000 | 94.624 | 87.192 | 79.152 | 71.182 |
12 | 100.000 | 95.392 | 88.690 | 81.182 | 73.554 |
13 | 100.000 | 96.160 | 90.159 | 83.120 | 75.744 |
14 | 100.000 | 96.928 | 91.657 | 85.142 | 78.092 |
15 | 100.000 | 97.696 | 93.126 | 87.088 | 80.310 |
16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.693 |
17 | 100.000 | 99.232 | 96.093 | 91.068 | 84.910 |
18 | 100.000 | 100.000 | 97.562 | 93.005 | 87.101 |
19 | 100.000 | 100.000 | 97.679 | 93.330 | 87.673 |
20 | 100.000 | 100.000 | 97.767 | 93.581 | 88.132 |
21 | 100.000 | 100.000 | 97.855 | 93.819 | 88.545 |
22 | 100.000 | 100.000 | 97.944 | 94.060 | 88.969 |
23 | 100.000 | 100.000 | 98.061 | 94.383 | 89.536 |
24 | 100.000 | 100.000 | 98.135 | 94.588 | 89.903 |
25 | 100.000 | 100.000 | 98.223 | 94.825 | 90.316 |
26 | 100.000 | 100.000 | 98.281 | 94.987 | 90.606 |
27 | 100.000 | 100.000 | 98.370 | 95.225 | 91.015 |
28 | 100.000 | 100.000 | 98.443 | 95.434 | 91.397 |
29 | 100.000 | 100.000 | 98.502 | 95.596 | 91.684 |
30 | 100.000 | 100.000 | 98.575 | 95.797 | 92.038 |
31 | 100.000 | 100.000 | 98.663 | 96.034 | 92.449 |
32 | 100.000 | 100.000 | 98.751 | 96.271 | 92.858 |
33 | 100.000 | 100.000 | 98.840 | 96.509 | 93.269 |
34 | 100.000 | 100.000 | 98.913 | 96.710 | 93.621 |
35 | 100.000 | 100.000 | 98.957 | 96.835 | 93.851 |
36 | 100.000 | 100.000 | 99.031 | 97.036 | 94.204 |
37 | 100.000 | 100.000 | 99.119 | 97.274 | 94.616 |
38 | 100.000 | 100.000 | 99.177 | 97.436 | 94.904 |
39 | 100.000 | 100.000 | 99.236 | 97.598 | 95.191 |
40 | 100.000 | 100.000 | 99.266 | 97.687 | 95.363 |
41 | 100.000 | 100.000 | 99.295 | 97.777 | 95.535 |
42 | 100.000 | 100.000 | 99.310 | 97.827 | 95.642 |
43 | 100.000 | 100.000 | 99.324 | 97.878 | 95.750 |
44 | 100.000 | 100.000 | 99.368 | 98.004 | 95.980 |
45 | 100.000 | 100.000 | 99.383 | 98.055 | 96.087 |
46 | 100.000 | 100.000 | 99.383 | 98.069 | 96.137 |
47 | 100.000 | 100.000 | 99.398 | 98.119 | 96.244 |
48 | 100.000 | 100.000 | 99.486 | 98.357 | 96.654 |
49 | 100.000 | 100.000 | 99.486 | 98.371 | 96.703 |
50 | 100.000 | 100.000 | 99.545 | 98.533 | 96.991 |
51 | 100.000 | 100.000 | 99.574 | 98.623 | 97.163 |
52 | 100.000 | 100.000 | 99.633 | 98.785 | 97.450 |
53 | 100.000 | 100.000 | 99.662 | 98.874 | 97.622 |
54 | 100.000 | 100.000 | 99.706 | 99.000 | 97.852 |
55 | 100.000 | 100.000 | 99.780 | 99.201 | 98.204 |
56 | 100.000 | 100.000 | 99.794 | 99.251 | 98.311 |
57 | 100.000 | 100.000 | 99.838 | 99.377 | 98.540 |
58 | 100.000 | 100.000 | 99.853 | 99.428 | 98.648 |
59 | 100.000 | 100.000 | 99.853 | 99.442 | 98.697 |
60 | 100.000 | 100.000 | 99.882 | 99.531 | 98.869 |
61 | 100.000 | 100.000 | 99.882 | 99.546 | 98.919 |
62 | 100.000 | 100.000 | 99.927 | 99.671 | 99.148 |
63 | 100.000 | 100.000 | 99.956 | 99.761 | 99.320 |
64 | 100.000 | 100.000 | 99.956 | 99.775 | 99.370 |
65 | 100.000 | 100.000 | 99.971 | 99.826 | 99.477 |
66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |


gzip
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
7 | 94.737 | 86.022 | 76.572 | 67.396 | 58.886 |
8 | 97.368 | 89.555 | 80.449 | 71.408 | 62.923 |
9 | 100.000 | 93.088 | 84.254 | 75.242 | 66.687 |
10 | 100.000 | 93.856 | 85.781 | 77.352 | 69.194 |
11 | 100.000 | 94.624 | 87.294 | 79.423 | 71.636 |
12 | 100.000 | 95.392 | 88.793 | 81.464 | 74.038 |
13 | 100.000 | 96.160 | 90.276 | 83.460 | 76.359 |
14 | 100.000 | 96.928 | 91.745 | 85.422 | 78.628 |
15 | 100.000 | 97.696 | 93.214 | 87.360 | 80.822 |
16 | 100.000 | 98.464 | 94.683 | 89.297 | 83.015 |
17 | 100.000 | 99.232 | 96.152 | 91.234 | 85.209 |
18 | 100.000 | 100.000 | 97.620 | 93.172 | 87.402 |
19 | 100.000 | 100.000 | 97.709 | 93.409 | 87.817 |
20 | 100.000 | 100.000 | 97.797 | 93.665 | 88.286 |
21 | 100.000 | 100.000 | 97.885 | 93.902 | 88.699 |
22 | 100.000 | 100.000 | 97.973 | 94.139 | 89.112 |
23 | 100.000 | 100.000 | 98.061 | 94.381 | 89.536 |
24 | 100.000 | 100.000 | 98.149 | 94.618 | 89.946 |
25 | 100.000 | 100.000 | 98.237 | 94.855 | 90.357 |
26 | 100.000 | 100.000 | 98.325 | 95.097 | 90.781 |
27 | 100.000 | 100.000 | 98.414 | 95.334 | 91.191 |
28 | 100.000 | 100.000 | 98.502 | 95.571 | 91.601 |
29 | 100.000 | 100.000 | 98.590 | 95.809 | 92.012 |
30 | 100.000 | 100.000 | 98.678 | 96.046 | 92.422 |
31 | 100.000 | 100.000 | 98.766 | 96.288 | 92.846 |
32 | 100.000 | 100.000 | 98.854 | 96.525 | 93.255 |
33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.688 |
34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.098 |
35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.508 |
36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.917 |
37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.147 |
38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.377 |
39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.606 |
40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.836 |
41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.066 |
42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.295 |
43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.525 |
44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.756 |
45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.985 |
46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.216 |
47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.676 |
49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.906 |
50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.365 |
52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.595 |
53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |


gzip
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
7 | 94.737 | 85.714 | 76.058 | 66.807 | 58.315 |
8 | 97.368 | 89.247 | 79.950 | 70.852 | 62.398 |
9 | 100.000 | 92.934 | 83.975 | 74.896 | 66.323 |
10 | 100.000 | 93.856 | 85.752 | 77.283 | 69.092 |
11 | 100.000 | 94.624 | 87.221 | 79.239 | 71.341 |
12 | 100.000 | 95.392 | 88.690 | 81.194 | 73.590 |
13 | 100.000 | 96.160 | 90.159 | 83.146 | 75.828 |
14 | 100.000 | 96.928 | 91.627 | 85.098 | 78.066 |
15 | 100.000 | 97.696 | 93.096 | 87.039 | 80.271 |
16 | 100.000 | 98.464 | 94.565 | 88.980 | 82.477 |
17 | 100.000 | 99.232 | 96.034 | 90.922 | 84.681 |
18 | 100.000 | 100.000 | 97.503 | 92.863 | 86.886 |
19 | 100.000 | 100.000 | 97.650 | 93.251 | 87.543 |
20 | 100.000 | 100.000 | 97.797 | 93.638 | 88.200 |
21 | 100.000 | 100.000 | 97.885 | 93.888 | 88.651 |
22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.102 |
23 | 100.000 | 100.000 | 98.061 | 94.385 | 89.547 |
24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.993 |
25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.404 |
26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.228 |
28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.638 |
29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.049 |
30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.460 |
31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.870 |
32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.281 |
33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.692 |
34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.151 |
38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.380 |
39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.610 |
40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.840 |
41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.069 |
42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.299 |
43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.529 |
44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.758 |
45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.988 |
46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.218 |
47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.677 |
49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.907 |
50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.366 |
52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.596 |
53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |


gzip
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
7 | 94.737 | 85.714 | 76.058 | 66.807 | 58.315 |
8 | 97.368 | 89.247 | 79.950 | 70.852 | 62.398 |
9 | 100.000 | 92.934 | 83.975 | 74.896 | 66.323 |
10 | 100.000 | 93.856 | 85.752 | 77.283 | 69.092 |
11 | 100.000 | 94.624 | 87.221 | 79.239 | 71.341 |
12 | 100.000 | 95.392 | 88.690 | 81.194 | 73.590 |
13 | 100.000 | 96.160 | 90.159 | 83.146 | 75.828 |
14 | 100.000 | 96.928 | 91.627 | 85.098 | 78.066 |
15 | 100.000 | 97.696 | 93.096 | 87.039 | 80.271 |
16 | 100.000 | 98.464 | 94.565 | 88.980 | 82.477 |
17 | 100.000 | 99.232 | 96.034 | 90.922 | 84.681 |
18 | 100.000 | 100.000 | 97.503 | 92.863 | 86.886 |
19 | 100.000 | 100.000 | 97.650 | 93.251 | 87.543 |
20 | 100.000 | 100.000 | 97.797 | 93.638 | 88.200 |
21 | 100.000 | 100.000 | 97.885 | 93.888 | 88.651 |
22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.102 |
23 | 100.000 | 100.000 | 98.061 | 94.385 | 89.547 |
24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.993 |
25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.404 |
26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.228 |
28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.639 |
29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.050 |
30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.461 |
31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.871 |
32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.282 |
33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.692 |
34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.151 |
38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.380 |
39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.610 |
40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.840 |
41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.069 |
42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.299 |
43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.529 |
44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.758 |
45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.988 |
46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.218 |
47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.677 |
49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.907 |
50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.366 |
52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.596 |
53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |


gzip
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
8 | 97.368 | 89.555 | 80.449 | 71.408 | 62.925 |
9 | 100.000 | 93.088 | 84.195 | 75.088 | 66.434 |
10 | 100.000 | 93.856 | 85.664 | 77.030 | 68.638 |
11 | 100.000 | 94.624 | 87.147 | 79.036 | 70.987 |
12 | 100.000 | 95.392 | 88.646 | 81.057 | 73.327 |
13 | 100.000 | 96.160 | 90.129 | 83.051 | 75.641 |
14 | 100.000 | 96.928 | 91.627 | 85.071 | 77.980 |
15 | 100.000 | 97.696 | 93.111 | 87.061 | 80.282 |
16 | 100.000 | 98.464 | 94.683 | 89.279 | 82.953 |
17 | 100.000 | 99.232 | 96.152 | 91.222 | 85.166 |
18 | 100.000 | 100.000 | 97.620 | 93.159 | 87.358 |
19 | 100.000 | 100.000 | 97.679 | 93.318 | 87.632 |
20 | 100.000 | 100.000 | 97.767 | 93.555 | 88.045 |
21 | 100.000 | 100.000 | 97.855 | 93.792 | 88.455 |
22 | 100.000 | 100.000 | 97.914 | 93.963 | 88.771 |
23 | 100.000 | 100.000 | 97.973 | 94.121 | 89.044 |
24 | 100.000 | 100.000 | 98.002 | 94.212 | 89.223 |
25 | 100.000 | 100.000 | 98.090 | 94.458 | 89.662 |
26 | 100.000 | 100.000 | 98.179 | 94.695 | 90.072 |
27 | 100.000 | 100.000 | 98.193 | 94.738 | 90.152 |
28 | 100.000 | 100.000 | 98.281 | 94.975 | 90.565 |
29 | 100.000 | 100.000 | 98.340 | 95.133 | 90.838 |
30 | 100.000 | 100.000 | 98.428 | 95.375 | 91.264 |
31 | 100.000 | 100.000 | 98.472 | 95.505 | 91.507 |
32 | 100.000 | 100.000 | 98.531 | 95.663 | 91.781 |
33 | 100.000 | 100.000 | 98.561 | 95.746 | 91.933 |
34 | 100.000 | 100.000 | 98.649 | 95.983 | 92.344 |
35 | 100.000 | 100.000 | 98.678 | 96.066 | 92.496 |
36 | 100.000 | 100.000 | 98.766 | 96.304 | 92.906 |
37 | 100.000 | 100.000 | 98.781 | 96.355 | 93.012 |
38 | 100.000 | 100.000 | 98.810 | 96.438 | 93.163 |
39 | 100.000 | 100.000 | 98.854 | 96.559 | 93.381 |
40 | 100.000 | 100.000 | 98.913 | 96.722 | 93.669 |
41 | 100.000 | 100.000 | 99.001 | 96.959 | 94.079 |
42 | 100.000 | 100.000 | 99.031 | 97.042 | 94.230 |
43 | 100.000 | 100.000 | 99.045 | 97.085 | 94.309 |
44 | 100.000 | 100.000 | 99.104 | 97.247 | 94.597 |
45 | 100.000 | 100.000 | 99.148 | 97.369 | 94.812 |
46 | 100.000 | 100.000 | 99.163 | 97.411 | 94.892 |
47 | 100.000 | 100.000 | 99.207 | 97.533 | 95.108 |
48 | 100.000 | 100.000 | 99.236 | 97.616 | 95.261 |
49 | 100.000 | 100.000 | 99.266 | 97.700 | 95.412 |
50 | 100.000 | 100.000 | 99.354 | 97.937 | 95.822 |
51 | 100.000 | 100.000 | 99.354 | 97.945 | 95.850 |
52 | 100.000 | 100.000 | 99.354 | 97.949 | 95.865 |
53 | 100.000 | 100.000 | 99.354 | 97.957 | 95.894 |
54 | 100.000 | 100.000 | 99.354 | 97.959 | 95.901 |
55 | 100.000 | 100.000 | 99.412 | 98.117 | 96.176 |
56 | 100.000 | 100.000 | 99.412 | 98.124 | 96.197 |
57 | 100.000 | 100.000 | 99.471 | 98.286 | 96.485 |
58 | 100.000 | 100.000 | 99.530 | 98.448 | 96.772 |
59 | 100.000 | 100.000 | 99.589 | 98.610 | 97.059 |
60 | 100.000 | 100.000 | 99.618 | 98.698 | 97.224 |
61 | 100.000 | 100.000 | 99.633 | 98.740 | 97.304 |
62 | 100.000 | 100.000 | 99.633 | 98.746 | 97.325 |
63 | 100.000 | 100.000 | 99.692 | 98.911 | 97.620 |
64 | 100.000 | 100.000 | 99.692 | 98.915 | 97.634 |
65 | 100.000 | 100.000 | 99.692 | 98.919 | 97.649 |
66 | 100.000 | 100.000 | 99.721 | 99.002 | 97.800 |
67 | 100.000 | 100.000 | 99.750 | 99.085 | 97.951 |
68 | 100.000 | 100.000 | 99.780 | 99.168 | 98.102 |
69 | 100.000 | 100.000 | 99.794 | 99.211 | 98.182 |
70 | 100.000 | 100.000 | 99.794 | 99.215 | 98.196 |
71 | 100.000 | 100.000 | 99.794 | 99.221 | 98.218 |
72 | 100.000 | 100.000 | 99.838 | 99.343 | 98.434 |
73 | 100.000 | 100.000 | 99.838 | 99.347 | 98.448 |
74 | 100.000 | 100.000 | 99.868 | 99.430 | 98.599 |
75 | 100.000 | 100.000 | 99.868 | 99.434 | 98.614 |
76 | 100.000 | 100.000 | 99.868 | 99.434 | 98.615 |
77 | 100.000 | 100.000 | 99.882 | 99.477 | 98.694 |
78 | 100.000 | 100.000 | 99.882 | 99.487 | 98.730 |
79 | 100.000 | 100.000 | 99.882 | 99.493 | 98.751 |
80 | 100.000 | 100.000 | 99.882 | 99.499 | 98.773 |
81 | 100.000 | 100.000 | 99.882 | 99.507 | 98.801 |
82 | 100.000 | 100.000 | 99.912 | 99.590 | 98.953 |
83 | 100.000 | 100.000 | 99.912 | 99.594 | 98.967 |
84 | 100.000 | 100.000 | 99.912 | 99.594 | 98.968 |
85 | 100.000 | 100.000 | 99.912 | 99.598 | 98.983 |
86 | 100.000 | 100.000 | 99.912 | 99.606 | 99.011 |
87 | 100.000 | 100.000 | 99.912 | 99.613 | 99.033 |
88 | 100.000 | 100.000 | 99.912 | 99.613 | 99.034 |
89 | 100.000 | 100.000 | 99.912 | 99.623 | 99.069 |
90 | 100.000 | 100.000 | 99.941 | 99.706 | 99.220 |
91 | 100.000 | 100.000 | 99.956 | 99.753 | 99.314 |
92 | 100.000 | 100.000 | 99.956 | 99.753 | 99.314 |
93 | 100.000 | 100.000 | 99.956 | 99.757 | 99.329 |
94 | 100.000 | 100.000 | 99.956 | 99.761 | 99.344 |
95 | 100.000 | 100.000 | 99.956 | 99.769 | 99.373 |
96 | 100.000 | 100.000 | 99.956 | 99.775 | 99.394 |
97 | 100.000 | 100.000 | 99.985 | 99.858 | 99.545 |
98 | 100.000 | 100.000 | 99.985 | 99.858 | 99.546 |
99 | 100.000 | 100.000 | 99.985 | 99.858 | 99.547 |
100 | 100.000 | 100.000 | 99.985 | 99.864 | 99.568 |
101 | 100.000 | 100.000 | 99.985 | 99.870 | 99.590 |
102 | 100.000 | 100.000 | 100.000 | 99.913 | 99.669 |
103 | 100.000 | 100.000 | 100.000 | 99.919 | 99.691 |
104 | 100.000 | 100.000 | 100.000 | 99.919 | 99.691 |
105 | 100.000 | 100.000 | 100.000 | 99.919 | 99.692 |
106 | 100.000 | 100.000 | 100.000 | 99.921 | 99.700 |
107 | 100.000 | 100.000 | 100.000 | 99.925 | 99.714 |
108 | 100.000 | 100.000 | 100.000 | 99.927 | 99.722 |
109 | 100.000 | 100.000 | 100.000 | 99.931 | 99.737 |
110 | 100.000 | 100.000 | 100.000 | 99.935 | 99.751 |
111 | 100.000 | 100.000 | 100.000 | 99.943 | 99.780 |
112 | 100.000 | 100.000 | 100.000 | 99.945 | 99.788 |
113 | 100.000 | 100.000 | 100.000 | 99.949 | 99.802 |
114 | 100.000 | 100.000 | 100.000 | 99.951 | 99.810 |
115 | 100.000 | 100.000 | 100.000 | 99.955 | 99.825 |
116 | 100.000 | 100.000 | 100.000 | 99.955 | 99.825 |
117 | 100.000 | 100.000 | 100.000 | 99.959 | 99.840 |
118 | 100.000 | 100.000 | 100.000 | 99.959 | 99.841 |
119 | 100.000 | 100.000 | 100.000 | 99.963 | 99.856 |
120 | 100.000 | 100.000 | 100.000 | 99.968 | 99.870 |
121 | 100.000 | 100.000 | 100.000 | 99.976 | 99.899 |
122 | 100.000 | 100.000 | 100.000 | 99.980 | 99.914 |
123 | 100.000 | 100.000 | 100.000 | 99.984 | 99.929 |
124 | 100.000 | 100.000 | 100.000 | 99.984 | 99.929 |
125 | 100.000 | 100.000 | 100.000 | 99.988 | 99.944 |
126 | 100.000 | 100.000 | 100.000 | 99.988 | 99.945 |
127 | 100.000 | 100.000 | 100.000 | 99.988 | 99.946 |
128 | 100.000 | 100.000 | 100.000 | 99.992 | 99.960 |
129 | 100.000 | 100.000 | 100.000 | 99.992 | 99.961 |
130 | 100.000 | 100.000 | 100.000 | 99.992 | 99.962 |
131 | 100.000 | 100.000 | 100.000 | 99.992 | 99.963 |
132 | 100.000 | 100.000 | 100.000 | 99.994 | 99.970 |
133 | 100.000 | 100.000 | 100.000 | 99.994 | 99.971 |
134 | 100.000 | 100.000 | 100.000 | 99.994 | 99.972 |
135 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
136 | 100.000 | 100.000 | 100.000 | 99.998 | 99.987 |
137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


gzip
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
10 | 100.000 | 93.856 | 85.870 | 77.559 | 69.501 |
11 | 100.000 | 94.624 | 87.338 | 79.511 | 71.739 |
12 | 100.000 | 95.392 | 88.807 | 81.478 | 74.025 |
13 | 100.000 | 96.160 | 90.276 | 83.428 | 76.257 |
14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.525 |
15 | 100.000 | 97.696 | 93.214 | 87.335 | 80.745 |
16 | 100.000 | 98.464 | 94.683 | 89.283 | 82.970 |
17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
19 | 100.000 | 100.000 | 97.709 | 93.421 | 87.851 |
20 | 100.000 | 100.000 | 97.797 | 93.663 | 88.276 |
21 | 100.000 | 100.000 | 97.885 | 93.900 | 88.688 |
22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.098 |
23 | 100.000 | 100.000 | 98.061 | 94.379 | 89.523 |
24 | 100.000 | 100.000 | 98.149 | 94.616 | 89.934 |
25 | 100.000 | 100.000 | 98.237 | 94.853 | 90.346 |
26 | 100.000 | 100.000 | 98.325 | 95.091 | 90.757 |
27 | 100.000 | 100.000 | 98.414 | 95.328 | 91.167 |
28 | 100.000 | 100.000 | 98.502 | 95.569 | 91.592 |
29 | 100.000 | 100.000 | 98.590 | 95.807 | 92.002 |
30 | 100.000 | 100.000 | 98.678 | 96.044 | 92.413 |
31 | 100.000 | 100.000 | 98.766 | 96.286 | 92.838 |
32 | 100.000 | 100.000 | 98.854 | 96.523 | 93.248 |
33 | 100.000 | 100.000 | 98.942 | 96.764 | 93.672 |
34 | 100.000 | 100.000 | 99.031 | 97.002 | 94.082 |
35 | 100.000 | 100.000 | 99.119 | 97.239 | 94.492 |
36 | 100.000 | 100.000 | 99.207 | 97.476 | 94.902 |
37 | 100.000 | 100.000 | 99.266 | 97.639 | 95.190 |
38 | 100.000 | 100.000 | 99.324 | 97.797 | 95.464 |
39 | 100.000 | 100.000 | 99.383 | 97.959 | 95.752 |
40 | 100.000 | 100.000 | 99.442 | 98.121 | 96.039 |
41 | 100.000 | 100.000 | 99.486 | 98.247 | 96.269 |
42 | 100.000 | 100.000 | 99.530 | 98.373 | 96.499 |
43 | 100.000 | 100.000 | 99.574 | 98.499 | 96.728 |
44 | 100.000 | 100.000 | 99.618 | 98.625 | 96.958 |
45 | 100.000 | 100.000 | 99.662 | 98.750 | 97.187 |
46 | 100.000 | 100.000 | 99.706 | 98.876 | 97.417 |
47 | 100.000 | 100.000 | 99.750 | 99.002 | 97.647 |
48 | 100.000 | 100.000 | 99.794 | 99.128 | 97.876 |
49 | 100.000 | 100.000 | 99.824 | 99.211 | 98.028 |
50 | 100.000 | 100.000 | 99.853 | 99.298 | 98.193 |
51 | 100.000 | 100.000 | 99.882 | 99.385 | 98.359 |
52 | 100.000 | 100.000 | 99.912 | 99.468 | 98.510 |
53 | 100.000 | 100.000 | 99.941 | 99.556 | 98.675 |
54 | 100.000 | 100.000 | 99.971 | 99.639 | 98.826 |
55 | 100.000 | 100.000 | 99.985 | 99.686 | 98.920 |
56 | 100.000 | 100.000 | 100.000 | 99.732 | 99.013 |
57 | 100.000 | 100.000 | 100.000 | 99.736 | 99.028 |
58 | 100.000 | 100.000 | 100.000 | 99.738 | 99.036 |
59 | 100.000 | 100.000 | 100.000 | 99.740 | 99.044 |
60 | 100.000 | 100.000 | 100.000 | 99.742 | 99.052 |
61 | 100.000 | 100.000 | 100.000 | 99.757 | 99.101 |
62 | 100.000 | 100.000 | 100.000 | 99.767 | 99.137 |
63 | 100.000 | 100.000 | 100.000 | 99.767 | 99.138 |
64 | 100.000 | 100.000 | 100.000 | 99.771 | 99.152 |
65 | 100.000 | 100.000 | 100.000 | 99.773 | 99.160 |
66 | 100.000 | 100.000 | 100.000 | 99.777 | 99.175 |
67 | 100.000 | 100.000 | 100.000 | 99.783 | 99.197 |
68 | 100.000 | 100.000 | 100.000 | 99.797 | 99.246 |
69 | 100.000 | 100.000 | 100.000 | 99.801 | 99.261 |
70 | 100.000 | 100.000 | 100.000 | 99.805 | 99.276 |
71 | 100.000 | 100.000 | 100.000 | 99.805 | 99.277 |
72 | 100.000 | 100.000 | 100.000 | 99.813 | 99.305 |
73 | 100.000 | 100.000 | 100.000 | 99.817 | 99.320 |
74 | 100.000 | 100.000 | 100.000 | 99.821 | 99.335 |
75 | 100.000 | 100.000 | 100.000 | 99.821 | 99.336 |
76 | 100.000 | 100.000 | 100.000 | 99.826 | 99.350 |
77 | 100.000 | 100.000 | 100.000 | 99.832 | 99.372 |
78 | 100.000 | 100.000 | 100.000 | 99.834 | 99.379 |
79 | 100.000 | 100.000 | 100.000 | 99.838 | 99.394 |
80 | 100.000 | 100.000 | 100.000 | 99.840 | 99.402 |
81 | 100.000 | 100.000 | 100.000 | 99.840 | 99.402 |
82 | 100.000 | 100.000 | 100.000 | 99.844 | 99.417 |
83 | 100.000 | 100.000 | 100.000 | 99.844 | 99.418 |
84 | 100.000 | 100.000 | 100.000 | 99.848 | 99.433 |
85 | 100.000 | 100.000 | 100.000 | 99.856 | 99.461 |
86 | 100.000 | 100.000 | 100.000 | 99.860 | 99.476 |
87 | 100.000 | 100.000 | 100.000 | 99.862 | 99.484 |
88 | 100.000 | 100.000 | 100.000 | 99.870 | 99.512 |
89 | 100.000 | 100.000 | 100.000 | 99.878 | 99.541 |
90 | 100.000 | 100.000 | 100.000 | 99.878 | 99.542 |
91 | 100.000 | 100.000 | 100.000 | 99.886 | 99.571 |
92 | 100.000 | 100.000 | 100.000 | 99.890 | 99.585 |
93 | 100.000 | 100.000 | 100.000 | 99.899 | 99.614 |
94 | 100.000 | 100.000 | 100.000 | 99.903 | 99.629 |
95 | 100.000 | 100.000 | 100.000 | 99.907 | 99.643 |
96 | 100.000 | 100.000 | 100.000 | 99.907 | 99.644 |
97 | 100.000 | 100.000 | 100.000 | 99.907 | 99.645 |
98 | 100.000 | 100.000 | 100.000 | 99.913 | 99.666 |
99 | 100.000 | 100.000 | 100.000 | 99.913 | 99.667 |
100 | 100.000 | 100.000 | 100.000 | 99.923 | 99.703 |
101 | 100.000 | 100.000 | 100.000 | 99.929 | 99.724 |
102 | 100.000 | 100.000 | 100.000 | 99.935 | 99.746 |
103 | 100.000 | 100.000 | 100.000 | 99.941 | 99.767 |
104 | 100.000 | 100.000 | 100.000 | 99.947 | 99.789 |
105 | 100.000 | 100.000 | 100.000 | 99.955 | 99.818 |
106 | 100.000 | 100.000 | 100.000 | 99.955 | 99.818 |
107 | 100.000 | 100.000 | 100.000 | 99.963 | 99.847 |
108 | 100.000 | 100.000 | 100.000 | 99.963 | 99.848 |
109 | 100.000 | 100.000 | 100.000 | 99.963 | 99.849 |
110 | 100.000 | 100.000 | 100.000 | 99.966 | 99.856 |
111 | 100.000 | 100.000 | 100.000 | 99.968 | 99.864 |
112 | 100.000 | 100.000 | 100.000 | 99.970 | 99.871 |
113 | 100.000 | 100.000 | 100.000 | 99.970 | 99.872 |
114 | 100.000 | 100.000 | 100.000 | 99.972 | 99.880 |
115 | 100.000 | 100.000 | 100.000 | 99.972 | 99.880 |
116 | 100.000 | 100.000 | 100.000 | 99.976 | 99.895 |
117 | 100.000 | 100.000 | 100.000 | 99.980 | 99.910 |
118 | 100.000 | 100.000 | 100.000 | 99.980 | 99.910 |
119 | 100.000 | 100.000 | 100.000 | 99.980 | 99.911 |
120 | 100.000 | 100.000 | 100.000 | 99.980 | 99.912 |
121 | 100.000 | 100.000 | 100.000 | 99.980 | 99.913 |
122 | 100.000 | 100.000 | 100.000 | 99.988 | 99.941 |
123 | 100.000 | 100.000 | 100.000 | 99.988 | 99.942 |
124 | 100.000 | 100.000 | 100.000 | 99.988 | 99.943 |
125 | 100.000 | 100.000 | 100.000 | 99.992 | 99.958 |
126 | 100.000 | 100.000 | 100.000 | 99.992 | 99.958 |
127 | 100.000 | 100.000 | 100.000 | 99.992 | 99.959 |
128 | 100.000 | 100.000 | 100.000 | 99.992 | 99.960 |
129 | 100.000 | 100.000 | 100.000 | 99.992 | 99.961 |
130 | 100.000 | 100.000 | 100.000 | 99.996 | 99.975 |
131 | 100.000 | 100.000 | 100.000 | 99.996 | 99.976 |
132 | 100.000 | 100.000 | 100.000 | 99.996 | 99.977 |
133 | 100.000 | 100.000 | 100.000 | 99.996 | 99.978 |
134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


gzip
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
10 | 100.000 | 93.856 | 85.870 | 77.563 | 69.515 |
11 | 100.000 | 94.624 | 87.338 | 79.525 | 71.783 |
12 | 100.000 | 95.392 | 88.807 | 81.487 | 74.051 |
13 | 100.000 | 96.160 | 90.276 | 83.438 | 76.289 |
14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.526 |
15 | 100.000 | 97.696 | 93.214 | 87.341 | 80.762 |
16 | 100.000 | 98.464 | 94.683 | 89.289 | 82.988 |
17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
19 | 100.000 | 100.000 | 97.709 | 93.425 | 87.865 |
20 | 100.000 | 100.000 | 97.797 | 93.667 | 88.291 |
21 | 100.000 | 100.000 | 97.885 | 93.908 | 88.717 |
22 | 100.000 | 100.000 | 97.973 | 94.149 | 89.143 |
23 | 100.000 | 100.000 | 98.061 | 94.391 | 89.569 |
24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.995 |
25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.406 |
26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.226 |
28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.636 |
29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.046 |
30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.456 |
31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.866 |
32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.277 |
33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.687 |
34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.097 |
35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.507 |
36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.917 |
37 | 100.000 | 100.000 | 99.266 | 97.643 | 95.205 |
38 | 100.000 | 100.000 | 99.324 | 97.805 | 95.492 |
39 | 100.000 | 100.000 | 99.383 | 97.967 | 95.780 |
40 | 100.000 | 100.000 | 99.442 | 98.130 | 96.067 |
41 | 100.000 | 100.000 | 99.501 | 98.292 | 96.355 |
42 | 100.000 | 100.000 | 99.545 | 98.418 | 96.584 |
43 | 100.000 | 100.000 | 99.589 | 98.543 | 96.814 |
44 | 100.000 | 100.000 | 99.633 | 98.669 | 97.044 |
45 | 100.000 | 100.000 | 99.677 | 98.795 | 97.273 |
46 | 100.000 | 100.000 | 99.706 | 98.882 | 97.438 |
47 | 100.000 | 100.000 | 99.736 | 98.969 | 97.603 |
48 | 100.000 | 100.000 | 99.765 | 99.057 | 97.768 |
49 | 100.000 | 100.000 | 99.794 | 99.144 | 97.934 |
50 | 100.000 | 100.000 | 99.824 | 99.231 | 98.099 |
51 | 100.000 | 100.000 | 99.853 | 99.314 | 98.250 |
52 | 100.000 | 100.000 | 99.882 | 99.397 | 98.401 |
53 | 100.000 | 100.000 | 99.912 | 99.481 | 98.553 |
54 | 100.000 | 100.000 | 99.941 | 99.564 | 98.704 |
55 | 100.000 | 100.000 | 99.971 | 99.647 | 98.855 |
56 | 100.000 | 100.000 | 99.985 | 99.694 | 98.948 |
57 | 100.000 | 100.000 | 100.000 | 99.740 | 99.041 |
58 | 100.000 | 100.000 | 100.000 | 99.755 | 99.091 |
59 | 100.000 | 100.000 | 100.000 | 99.769 | 99.140 |
60 | 100.000 | 100.000 | 100.000 | 99.783 | 99.190 |
61 | 100.000 | 100.000 | 100.000 | 99.797 | 99.239 |
62 | 100.000 | 100.000 | 100.000 | 99.807 | 99.275 |
63 | 100.000 | 100.000 | 100.000 | 99.817 | 99.310 |
64 | 100.000 | 100.000 | 100.000 | 99.828 | 99.346 |
65 | 100.000 | 100.000 | 100.000 | 99.838 | 99.381 |
66 | 100.000 | 100.000 | 100.000 | 99.848 | 99.417 |
67 | 100.000 | 100.000 | 100.000 | 99.858 | 99.452 |
68 | 100.000 | 100.000 | 100.000 | 99.868 | 99.488 |
69 | 100.000 | 100.000 | 100.000 | 99.878 | 99.523 |
70 | 100.000 | 100.000 | 100.000 | 99.888 | 99.559 |
71 | 100.000 | 100.000 | 100.000 | 99.899 | 99.594 |
72 | 100.000 | 100.000 | 100.000 | 99.909 | 99.630 |
73 | 100.000 | 100.000 | 100.000 | 99.919 | 99.665 |
74 | 100.000 | 100.000 | 100.000 | 99.929 | 99.701 |
75 | 100.000 | 100.000 | 100.000 | 99.939 | 99.736 |
76 | 100.000 | 100.000 | 100.000 | 99.949 | 99.773 |
77 | 100.000 | 100.000 | 100.000 | 99.959 | 99.808 |
78 | 100.000 | 100.000 | 100.000 | 99.970 | 99.844 |
79 | 100.000 | 100.000 | 100.000 | 99.980 | 99.879 |
80 | 100.000 | 100.000 | 100.000 | 99.988 | 99.908 |
81 | 100.000 | 100.000 | 100.000 | 99.994 | 99.929 |
82 | 100.000 | 100.000 | 100.000 | 100.000 | 99.951 |
83 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
84 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
85 | 100.000 | 100.000 | 100.000 | 100.000 | 99.954 |
86 | 100.000 | 100.000 | 100.000 | 100.000 | 99.955 |
87 | 100.000 | 100.000 | 100.000 | 100.000 | 99.955 |
88 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
89 | 100.000 | 100.000 | 100.000 | 100.000 | 99.957 |
90 | 100.000 | 100.000 | 100.000 | 100.000 | 99.958 |
91 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
92 | 100.000 | 100.000 | 100.000 | 100.000 | 99.960 |
93 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
94 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
95 | 100.000 | 100.000 | 100.000 | 100.000 | 99.962 |
96 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
97 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
98 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
99 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
100 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
101 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
102 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
103 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
104 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
105 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
106 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
107 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
108 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
109 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
110 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
112 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
113 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
114 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
115 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
116 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
117 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
118 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
119 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
120 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
121 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
122 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
123 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
124 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
125 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
126 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
127 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
128 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
129 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
130 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
131 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
132 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
133 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


gzip
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
10 | 100.000 | 93.856 | 85.870 | 77.563 | 69.515 |
11 | 100.000 | 94.624 | 87.338 | 79.525 | 71.783 |
12 | 100.000 | 95.392 | 88.807 | 81.487 | 74.051 |
13 | 100.000 | 96.160 | 90.276 | 83.438 | 76.289 |
14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.526 |
15 | 100.000 | 97.696 | 93.214 | 87.339 | 80.758 |
16 | 100.000 | 98.464 | 94.683 | 89.289 | 82.989 |
17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
19 | 100.000 | 100.000 | 97.709 | 93.425 | 87.865 |
20 | 100.000 | 100.000 | 97.797 | 93.667 | 88.291 |
21 | 100.000 | 100.000 | 97.885 | 93.908 | 88.717 |
22 | 100.000 | 100.000 | 97.973 | 94.149 | 89.143 |
23 | 100.000 | 100.000 | 98.061 | 94.391 | 89.570 |
24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.996 |
25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.406 |
26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.817 |
27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.227 |
28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.638 |
29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.049 |
30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.459 |
31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.870 |
32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.280 |
33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.691 |
34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
37 | 100.000 | 100.000 | 99.266 | 97.643 | 95.209 |
38 | 100.000 | 100.000 | 99.324 | 97.805 | 95.496 |
39 | 100.000 | 100.000 | 99.383 | 97.967 | 95.783 |
40 | 100.000 | 100.000 | 99.442 | 98.130 | 96.071 |
41 | 100.000 | 100.000 | 99.486 | 98.255 | 96.301 |
42 | 100.000 | 100.000 | 99.530 | 98.381 | 96.530 |
43 | 100.000 | 100.000 | 99.559 | 98.468 | 96.695 |
44 | 100.000 | 100.000 | 99.589 | 98.556 | 96.860 |
45 | 100.000 | 100.000 | 99.618 | 98.643 | 97.025 |
46 | 100.000 | 100.000 | 99.647 | 98.730 | 97.190 |
47 | 100.000 | 100.000 | 99.677 | 98.813 | 97.342 |
48 | 100.000 | 100.000 | 99.706 | 98.896 | 97.493 |
49 | 100.000 | 100.000 | 99.736 | 98.980 | 97.644 |
50 | 100.000 | 100.000 | 99.765 | 99.063 | 97.795 |
51 | 100.000 | 100.000 | 99.794 | 99.146 | 97.946 |
52 | 100.000 | 100.000 | 99.824 | 99.229 | 98.097 |
53 | 100.000 | 100.000 | 99.853 | 99.312 | 98.248 |
54 | 100.000 | 100.000 | 99.882 | 99.395 | 98.399 |
55 | 100.000 | 100.000 | 99.912 | 99.479 | 98.550 |
56 | 100.000 | 100.000 | 99.927 | 99.525 | 98.644 |
57 | 100.000 | 100.000 | 99.941 | 99.572 | 98.737 |
58 | 100.000 | 100.000 | 99.956 | 99.619 | 98.830 |
59 | 100.000 | 100.000 | 99.971 | 99.665 | 98.923 |
60 | 100.000 | 100.000 | 99.985 | 99.712 | 99.017 |
61 | 100.000 | 100.000 | 100.000 | 99.759 | 99.110 |
62 | 100.000 | 100.000 | 100.000 | 99.773 | 99.160 |
63 | 100.000 | 100.000 | 100.000 | 99.787 | 99.209 |
64 | 100.000 | 100.000 | 100.000 | 99.797 | 99.245 |
65 | 100.000 | 100.000 | 100.000 | 99.807 | 99.280 |
66 | 100.000 | 100.000 | 100.000 | 99.817 | 99.316 |
67 | 100.000 | 100.000 | 100.000 | 99.828 | 99.351 |
68 | 100.000 | 100.000 | 100.000 | 99.838 | 99.387 |
69 | 100.000 | 100.000 | 100.000 | 99.848 | 99.422 |
70 | 100.000 | 100.000 | 100.000 | 99.858 | 99.458 |
71 | 100.000 | 100.000 | 100.000 | 99.868 | 99.493 |
72 | 100.000 | 100.000 | 100.000 | 99.878 | 99.529 |
73 | 100.000 | 100.000 | 100.000 | 99.888 | 99.564 |
74 | 100.000 | 100.000 | 100.000 | 99.899 | 99.600 |
75 | 100.000 | 100.000 | 100.000 | 99.909 | 99.635 |
76 | 100.000 | 100.000 | 100.000 | 99.919 | 99.671 |
77 | 100.000 | 100.000 | 100.000 | 99.929 | 99.706 |
78 | 100.000 | 100.000 | 100.000 | 99.939 | 99.742 |
79 | 100.000 | 100.000 | 100.000 | 99.947 | 99.770 |
80 | 100.000 | 100.000 | 100.000 | 99.953 | 99.792 |
81 | 100.000 | 100.000 | 100.000 | 99.959 | 99.813 |
82 | 100.000 | 100.000 | 100.000 | 99.966 | 99.835 |
83 | 100.000 | 100.000 | 100.000 | 99.972 | 99.856 |
84 | 100.000 | 100.000 | 100.000 | 99.978 | 99.878 |
85 | 100.000 | 100.000 | 100.000 | 99.984 | 99.900 |
86 | 100.000 | 100.000 | 100.000 | 99.988 | 99.914 |
87 | 100.000 | 100.000 | 100.000 | 99.992 | 99.929 |
88 | 100.000 | 100.000 | 100.000 | 99.996 | 99.944 |
89 | 100.000 | 100.000 | 100.000 | 100.000 | 99.958 |
90 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
91 | 100.000 | 100.000 | 100.000 | 100.000 | 99.960 |
92 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
93 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
94 | 100.000 | 100.000 | 100.000 | 100.000 | 99.962 |
95 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
96 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
97 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
98 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
99 | 100.000 | 100.000 | 100.000 | 100.000 | 99.966 |
100 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
101 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
102 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
103 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
104 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
105 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
106 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
107 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
108 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
109 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
110 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
112 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
113 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
114 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
115 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
116 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
117 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
118 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
119 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
120 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
121 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
122 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
123 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
124 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
125 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
126 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
127 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
128 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
129 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
130 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
131 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
132 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
133 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |


sed
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 57.500 | 31.395 | 18.105 | 11.144 | 7.277 |
3 | 70.000 | 42.359 | 25.843 | 16.375 | 10.836 |
4 | 75.000 | 49.668 | 32.050 | 20.986 | 14.150 |
5 | 82.500 | 56.478 | 37.430 | 25.080 | 17.220 |
6 | 87.500 | 61.960 | 41.982 | 28.594 | 19.891 |
7 | 90.000 | 66.279 | 46.182 | 32.097 | 22.655 |
8 | 92.500 | 69.601 | 49.348 | 34.818 | 24.901 |
9 | 95.000 | 72.591 | 52.142 | 37.237 | 26.933 |
10 | 97.500 | 75.415 | 54.852 | 39.656 | 29.000 |
11 | 100.000 | 78.239 | 57.438 | 41.889 | 30.886 |
12 | 100.000 | 79.734 | 59.756 | 44.345 | 33.123 |
13 | 100.000 | 81.063 | 61.577 | 46.189 | 34.801 |
14 | 100.000 | 82.392 | 63.439 | 48.112 | 36.577 |
15 | 100.000 | 83.555 | 65.053 | 49.770 | 38.109 |
16 | 100.000 | 84.551 | 66.377 | 51.090 | 39.308 |
17 | 100.000 | 85.548 | 67.763 | 52.520 | 40.640 |
18 | 100.000 | 86.545 | 69.150 | 53.951 | 41.971 |
19 | 100.000 | 87.375 | 70.536 | 55.531 | 43.514 |
20 | 100.000 | 88.206 | 71.674 | 56.689 | 44.578 |
21 | 100.000 | 89.037 | 72.812 | 57.846 | 45.642 |
22 | 100.000 | 89.701 | 73.846 | 59.013 | 46.803 |
23 | 100.000 | 90.365 | 74.798 | 60.022 | 47.765 |
24 | 100.000 | 91.030 | 75.874 | 61.266 | 49.024 |
25 | 100.000 | 91.528 | 76.619 | 62.089 | 49.840 |
26 | 100.000 | 92.027 | 77.488 | 63.152 | 50.964 |
27 | 100.000 | 92.525 | 78.233 | 63.975 | 51.779 |
28 | 100.000 | 93.023 | 79.019 | 64.868 | 52.673 |
29 | 100.000 | 93.522 | 79.806 | 65.761 | 53.567 |
30 | 100.000 | 94.020 | 80.633 | 66.741 | 54.582 |
31 | 100.000 | 94.352 | 81.233 | 67.498 | 55.408 |
32 | 100.000 | 94.684 | 81.833 | 68.242 | 56.199 |
33 | 100.000 | 95.017 | 82.351 | 68.838 | 56.814 |
34 | 100.000 | 95.349 | 82.909 | 69.512 | 57.528 |
35 | 100.000 | 95.681 | 83.468 | 70.194 | 58.265 |
36 | 100.000 | 96.013 | 84.026 | 70.868 | 58.979 |
37 | 100.000 | 96.346 | 84.606 | 71.592 | 59.770 |
38 | 100.000 | 96.678 | 85.164 | 72.266 | 60.484 |
39 | 100.000 | 96.844 | 85.599 | 72.894 | 61.207 |
40 | 100.000 | 97.010 | 85.992 | 73.452 | 61.853 |
41 | 100.000 | 97.176 | 86.261 | 73.792 | 62.245 |
42 | 100.000 | 97.342 | 86.530 | 74.114 | 62.592 |
43 | 100.000 | 97.508 | 86.944 | 74.722 | 63.316 |
44 | 100.000 | 97.674 | 87.378 | 75.350 | 64.039 |
45 | 100.000 | 97.841 | 87.772 | 75.921 | 64.718 |
46 | 100.000 | 98.007 | 88.041 | 76.244 | 65.065 |
47 | 100.000 | 98.173 | 88.434 | 76.802 | 65.711 |
48 | 100.000 | 98.339 | 88.744 | 77.224 | 66.213 |
49 | 100.000 | 98.505 | 89.075 | 77.670 | 66.726 |
50 | 100.000 | 98.671 | 89.427 | 78.158 | 67.294 |
51 | 100.000 | 98.837 | 89.758 | 78.605 | 67.807 |
52 | 100.000 | 99.003 | 90.068 | 79.014 | 68.276 |
53 | 100.000 | 99.169 | 90.379 | 79.424 | 68.746 |
54 | 100.000 | 99.336 | 90.648 | 79.763 | 69.137 |
55 | 100.000 | 99.502 | 91.082 | 80.408 | 69.905 |
56 | 100.000 | 99.668 | 91.393 | 80.817 | 70.374 |
57 | 100.000 | 99.834 | 91.724 | 81.264 | 70.887 |
58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |


sed
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
5 | 80.000 | 55.316 | 37.016 | 24.964 | 17.203 |
6 | 85.000 | 60.465 | 41.548 | 28.677 | 20.122 |
7 | 87.500 | 64.452 | 45.396 | 31.915 | 22.716 |
8 | 92.500 | 69.269 | 49.328 | 35.016 | 25.149 |
9 | 95.000 | 72.924 | 52.866 | 38.035 | 27.599 |
10 | 97.500 | 75.914 | 55.659 | 40.454 | 29.631 |
11 | 100.000 | 78.738 | 58.225 | 42.646 | 31.462 |
12 | 100.000 | 80.066 | 60.087 | 44.568 | 33.238 |
13 | 100.000 | 81.395 | 61.949 | 46.491 | 35.015 |
14 | 100.000 | 82.392 | 63.542 | 48.274 | 36.743 |
15 | 100.000 | 83.389 | 65.032 | 49.899 | 38.315 |
16 | 100.000 | 84.385 | 66.418 | 51.329 | 39.646 |
17 | 100.000 | 85.382 | 67.805 | 52.760 | 40.978 |
18 | 100.000 | 86.213 | 69.088 | 54.199 | 42.404 |
19 | 100.000 | 87.209 | 70.432 | 55.560 | 43.657 |
20 | 100.000 | 88.040 | 71.612 | 56.804 | 44.843 |
21 | 100.000 | 88.870 | 72.750 | 57.962 | 45.907 |
22 | 100.000 | 89.701 | 73.888 | 59.120 | 46.971 |
23 | 100.000 | 90.365 | 74.922 | 60.286 | 48.132 |
24 | 100.000 | 91.030 | 75.936 | 61.423 | 49.270 |
25 | 100.000 | 91.694 | 76.929 | 62.519 | 50.353 |
26 | 100.000 | 92.359 | 77.881 | 63.528 | 51.315 |
27 | 100.000 | 92.857 | 78.709 | 64.508 | 52.330 |
28 | 100.000 | 93.355 | 79.495 | 65.401 | 53.224 |
29 | 100.000 | 93.854 | 80.240 | 66.224 | 54.040 |
30 | 100.000 | 94.020 | 80.799 | 67.080 | 55.043 |
31 | 100.000 | 94.352 | 81.419 | 67.874 | 55.912 |
32 | 100.000 | 94.684 | 82.019 | 68.618 | 56.704 |
33 | 100.000 | 95.017 | 82.599 | 69.338 | 57.484 |
34 | 100.000 | 95.349 | 83.178 | 70.045 | 58.232 |
35 | 100.000 | 95.681 | 83.737 | 70.719 | 58.946 |
36 | 100.000 | 95.847 | 84.171 | 71.348 | 59.669 |
37 | 100.000 | 96.013 | 84.606 | 71.976 | 60.393 |
38 | 100.000 | 96.346 | 85.123 | 72.572 | 61.008 |
39 | 100.000 | 96.678 | 85.640 | 73.167 | 61.623 |
40 | 100.000 | 97.010 | 86.158 | 73.763 | 62.238 |
41 | 100.000 | 97.176 | 86.571 | 74.366 | 62.951 |
42 | 100.000 | 97.342 | 86.965 | 74.925 | 63.597 |
43 | 100.000 | 97.508 | 87.337 | 75.462 | 64.242 |
44 | 100.000 | 97.674 | 87.709 | 76.004 | 64.898 |
45 | 100.000 | 97.841 | 88.061 | 76.492 | 65.465 |
46 | 100.000 | 98.007 | 88.413 | 76.971 | 66.013 |
47 | 100.000 | 98.173 | 88.765 | 77.459 | 66.580 |
48 | 100.000 | 98.339 | 89.096 | 77.906 | 67.093 |
49 | 100.000 | 98.505 | 89.427 | 78.353 | 67.607 |
50 | 100.000 | 98.671 | 89.737 | 78.762 | 68.076 |
51 | 100.000 | 98.837 | 90.048 | 79.171 | 68.545 |
52 | 100.000 | 99.003 | 90.358 | 79.585 | 69.025 |
53 | 100.000 | 99.169 | 90.668 | 79.994 | 69.495 |
54 | 100.000 | 99.336 | 90.958 | 80.371 | 69.933 |
55 | 100.000 | 99.502 | 91.227 | 80.693 | 70.280 |
56 | 100.000 | 99.668 | 91.496 | 81.016 | 70.628 |
57 | 100.000 | 99.834 | 91.765 | 81.338 | 70.976 |
58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |


sed
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
5 | 77.500 | 54.319 | 36.706 | 24.881 | 17.185 |
6 | 82.500 | 59.468 | 41.237 | 28.594 | 20.103 |
7 | 85.000 | 63.289 | 44.920 | 31.700 | 22.596 |
8 | 90.000 | 67.774 | 48.624 | 34.723 | 25.040 |
9 | 95.000 | 72.591 | 52.555 | 37.824 | 27.473 |
10 | 97.500 | 75.415 | 55.204 | 40.165 | 29.480 |
11 | 100.000 | 78.073 | 57.645 | 42.319 | 31.342 |
12 | 100.000 | 79.070 | 59.197 | 44.039 | 33.007 |
13 | 100.000 | 80.233 | 60.852 | 45.776 | 34.638 |
14 | 100.000 | 81.561 | 62.632 | 47.550 | 36.238 |
15 | 100.000 | 82.724 | 64.246 | 49.208 | 37.770 |
16 | 100.000 | 83.721 | 65.673 | 50.717 | 39.200 |
17 | 100.000 | 84.718 | 67.080 | 52.190 | 40.586 |
18 | 100.000 | 85.714 | 68.487 | 53.662 | 41.972 |
19 | 100.000 | 86.545 | 69.708 | 54.985 | 43.256 |
20 | 100.000 | 87.209 | 70.764 | 56.192 | 44.472 |
21 | 100.000 | 88.206 | 72.088 | 57.511 | 45.671 |
22 | 100.000 | 89.037 | 73.246 | 58.715 | 46.802 |
23 | 100.000 | 89.867 | 74.384 | 59.873 | 47.866 |
24 | 100.000 | 90.698 | 75.522 | 61.030 | 48.929 |
25 | 100.000 | 91.196 | 76.350 | 62.019 | 49.965 |
26 | 100.000 | 91.528 | 77.033 | 62.924 | 50.967 |
27 | 100.000 | 92.027 | 77.840 | 63.871 | 51.948 |
28 | 100.000 | 92.691 | 78.792 | 64.880 | 52.910 |
29 | 100.000 | 93.189 | 79.578 | 65.790 | 53.848 |
30 | 100.000 | 93.522 | 80.219 | 66.621 | 54.761 |
31 | 100.000 | 94.020 | 81.006 | 67.514 | 55.654 |
32 | 100.000 | 94.352 | 81.626 | 68.304 | 56.513 |
33 | 100.000 | 94.850 | 82.371 | 69.127 | 57.329 |
34 | 100.000 | 95.183 | 82.930 | 69.818 | 58.086 |
35 | 100.000 | 95.349 | 83.364 | 70.446 | 58.810 |
36 | 100.000 | 95.515 | 83.799 | 71.075 | 59.533 |
37 | 100.000 | 95.681 | 84.192 | 71.645 | 60.212 |
38 | 100.000 | 95.847 | 84.585 | 72.204 | 60.857 |
39 | 100.000 | 96.013 | 84.978 | 72.762 | 61.503 |
40 | 100.000 | 96.179 | 85.371 | 73.320 | 62.148 |
41 | 100.000 | 96.512 | 85.889 | 73.916 | 62.764 |
42 | 100.000 | 96.844 | 86.406 | 74.511 | 63.379 |
43 | 100.000 | 97.176 | 86.923 | 75.106 | 63.994 |
44 | 100.000 | 97.508 | 87.441 | 75.702 | 64.609 |
45 | 100.000 | 97.841 | 87.958 | 76.297 | 65.225 |
46 | 100.000 | 98.007 | 88.310 | 76.789 | 65.805 |
47 | 100.000 | 98.173 | 88.661 | 77.277 | 66.372 |
48 | 100.000 | 98.339 | 89.013 | 77.765 | 66.940 |
49 | 100.000 | 98.505 | 89.365 | 78.245 | 67.487 |
50 | 100.000 | 98.671 | 89.696 | 78.692 | 68.000 |
51 | 100.000 | 98.837 | 90.027 | 79.138 | 68.514 |
52 | 100.000 | 99.003 | 90.337 | 79.552 | 68.994 |
53 | 100.000 | 99.169 | 90.648 | 79.961 | 69.463 |
54 | 100.000 | 99.336 | 90.958 | 80.371 | 69.933 |
55 | 100.000 | 99.502 | 91.227 | 80.710 | 70.324 |
56 | 100.000 | 99.668 | 91.496 | 81.032 | 70.672 |
57 | 100.000 | 99.834 | 91.765 | 81.355 | 71.020 |
58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |


sed
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 70.000 | 41.694 | 25.347 | 16.123 | 10.733 |
4 | 75.000 | 48.505 | 31.181 | 20.543 | 13.968 |
5 | 75.000 | 52.658 | 35.837 | 24.472 | 17.008 |
6 | 77.500 | 56.312 | 39.623 | 27.838 | 19.773 |
7 | 85.000 | 62.791 | 44.527 | 31.506 | 22.539 |
8 | 90.000 | 67.608 | 48.500 | 34.677 | 25.050 |
9 | 92.500 | 70.764 | 51.500 | 37.282 | 27.227 |
10 | 95.000 | 73.588 | 54.149 | 39.623 | 29.235 |
11 | 95.000 | 75.083 | 56.218 | 41.732 | 31.157 |
12 | 97.500 | 77.741 | 58.680 | 43.923 | 33.062 |
13 | 97.500 | 79.070 | 60.501 | 45.776 | 34.761 |
14 | 100.000 | 81.728 | 62.880 | 47.823 | 36.501 |
15 | 100.000 | 82.724 | 64.349 | 49.411 | 38.029 |
16 | 100.000 | 83.721 | 65.798 | 50.961 | 39.514 |
17 | 100.000 | 84.551 | 67.101 | 52.417 | 40.935 |
18 | 100.000 | 85.548 | 68.529 | 53.926 | 42.365 |
19 | 100.000 | 86.545 | 69.915 | 55.357 | 43.696 |
20 | 100.000 | 87.209 | 70.929 | 56.494 | 44.834 |
21 | 100.000 | 88.040 | 72.108 | 57.739 | 46.020 |
22 | 100.000 | 88.704 | 73.102 | 58.835 | 47.103 |
23 | 100.000 | 89.037 | 73.743 | 59.666 | 48.016 |
24 | 100.000 | 89.535 | 74.529 | 60.576 | 48.954 |
25 | 100.000 | 90.365 | 75.667 | 61.733 | 50.018 |
26 | 100.000 | 91.196 | 76.805 | 62.891 | 51.082 |
27 | 100.000 | 91.694 | 77.571 | 63.760 | 51.964 |
28 | 100.000 | 92.193 | 78.336 | 64.628 | 52.847 |
29 | 100.000 | 92.857 | 79.288 | 65.637 | 53.809 |
30 | 100.000 | 93.355 | 80.074 | 66.530 | 54.702 |
31 | 100.000 | 93.854 | 80.861 | 67.423 | 55.596 |
32 | 100.000 | 94.186 | 81.440 | 68.143 | 56.377 |
33 | 100.000 | 94.518 | 82.019 | 68.862 | 57.157 |
34 | 100.000 | 94.850 | 82.619 | 69.607 | 57.949 |
35 | 100.000 | 95.017 | 83.054 | 70.235 | 58.672 |
36 | 100.000 | 95.183 | 83.426 | 70.773 | 59.318 |
37 | 100.000 | 95.515 | 84.006 | 71.480 | 60.066 |
38 | 100.000 | 95.847 | 84.564 | 72.154 | 60.779 |
39 | 100.000 | 96.013 | 84.958 | 72.712 | 61.425 |
40 | 100.000 | 96.179 | 85.351 | 73.270 | 62.071 |
41 | 100.000 | 96.346 | 85.744 | 73.829 | 62.716 |
42 | 100.000 | 96.512 | 86.096 | 74.317 | 63.284 |
43 | 100.000 | 96.844 | 86.613 | 74.912 | 63.899 |
44 | 100.000 | 97.176 | 87.130 | 75.508 | 64.515 |
45 | 100.000 | 97.508 | 87.647 | 76.103 | 65.130 |
46 | 100.000 | 97.841 | 88.165 | 76.699 | 65.745 |
47 | 100.000 | 98.173 | 88.682 | 77.294 | 66.360 |
48 | 100.000 | 98.339 | 89.013 | 77.749 | 66.894 |
49 | 100.000 | 98.505 | 89.365 | 78.229 | 67.441 |
50 | 100.000 | 98.671 | 89.696 | 78.675 | 67.954 |
51 | 100.000 | 98.837 | 90.006 | 79.089 | 68.435 |
52 | 100.000 | 99.003 | 90.317 | 79.498 | 68.904 |
53 | 100.000 | 99.169 | 90.627 | 79.907 | 69.373 |
54 | 100.000 | 99.336 | 90.937 | 80.317 | 69.843 |
55 | 100.000 | 99.502 | 91.227 | 80.693 | 70.280 |
56 | 100.000 | 99.668 | 91.496 | 81.032 | 70.672 |
57 | 100.000 | 99.834 | 91.765 | 81.355 | 71.020 |
58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |


sed
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
5 | 85.000 | 56.146 | 36.478 | 24.286 | 16.699 |
6 | 85.000 | 60.299 | 41.134 | 28.214 | 19.739 |
7 | 90.000 | 65.449 | 45.665 | 31.927 | 22.657 |
8 | 92.500 | 69.435 | 49.514 | 35.165 | 25.251 |
9 | 95.000 | 72.757 | 52.700 | 37.919 | 27.531 |
10 | 97.500 | 75.748 | 55.493 | 40.338 | 29.563 |
11 | 100.000 | 78.405 | 57.914 | 42.460 | 31.390 |
12 | 100.000 | 79.734 | 59.735 | 44.304 | 33.068 |
13 | 100.000 | 81.063 | 61.556 | 46.148 | 34.746 |
14 | 100.000 | 82.392 | 63.418 | 48.071 | 36.523 |
15 | 100.000 | 83.555 | 65.156 | 49.915 | 38.236 |
16 | 100.000 | 84.718 | 66.770 | 51.573 | 39.768 |
17 | 100.000 | 85.714 | 68.156 | 53.004 | 41.099 |
18 | 100.000 | 86.711 | 69.584 | 54.514 | 42.529 |
19 | 100.000 | 87.542 | 70.722 | 55.671 | 43.593 |
20 | 100.000 | 88.372 | 72.026 | 57.143 | 45.053 |
21 | 100.000 | 89.203 | 73.164 | 58.301 | 46.117 |
22 | 100.000 | 90.033 | 74.302 | 59.459 | 47.181 |
23 | 100.000 | 90.698 | 75.336 | 60.634 | 48.363 |
24 | 100.000 | 91.362 | 76.371 | 61.800 | 49.524 |
25 | 100.000 | 92.027 | 77.364 | 62.895 | 50.607 |
26 | 100.000 | 92.525 | 78.274 | 64.024 | 51.799 |
27 | 100.000 | 93.023 | 79.040 | 64.893 | 52.682 |
28 | 100.000 | 93.522 | 79.826 | 65.802 | 53.619 |
29 | 100.000 | 94.020 | 80.612 | 66.696 | 54.513 |
30 | 100.000 | 94.352 | 81.212 | 67.448 | 55.327 |
31 | 100.000 | 94.684 | 81.730 | 68.044 | 55.943 |
32 | 100.000 | 95.017 | 82.288 | 68.718 | 56.656 |
33 | 100.000 | 95.349 | 82.888 | 69.462 | 57.448 |
34 | 100.000 | 95.681 | 83.489 | 70.215 | 58.263 |
35 | 100.000 | 96.013 | 84.047 | 70.897 | 58.999 |
36 | 100.000 | 96.346 | 84.564 | 71.492 | 59.615 |
37 | 100.000 | 96.512 | 84.833 | 71.823 | 59.985 |
38 | 100.000 | 96.678 | 85.206 | 72.357 | 60.620 |
39 | 100.000 | 96.844 | 85.496 | 72.725 | 61.035 |
40 | 100.000 | 97.010 | 85.930 | 73.370 | 61.802 |
41 | 100.000 | 97.176 | 86.323 | 73.945 | 62.491 |
42 | 100.000 | 97.342 | 86.716 | 74.503 | 63.137 |
43 | 100.000 | 97.508 | 87.109 | 75.061 | 63.783 |
44 | 100.000 | 97.674 | 87.399 | 75.429 | 64.197 |
45 | 100.000 | 97.841 | 87.813 | 76.041 | 64.930 |
46 | 100.000 | 98.007 | 88.185 | 76.566 | 65.542 |
47 | 100.000 | 98.173 | 88.496 | 76.984 | 66.032 |
48 | 100.000 | 98.339 | 88.827 | 77.430 | 66.545 |
49 | 100.000 | 98.505 | 89.179 | 77.918 | 67.113 |
50 | 100.000 | 98.671 | 89.448 | 78.257 | 67.504 |
51 | 100.000 | 98.837 | 89.758 | 78.667 | 67.974 |
52 | 100.000 | 99.003 | 90.068 | 79.076 | 68.443 |
53 | 100.000 | 99.169 | 90.420 | 79.564 | 69.011 |
54 | 100.000 | 99.336 | 90.772 | 80.052 | 69.579 |
55 | 100.000 | 99.502 | 91.165 | 80.623 | 70.257 |
56 | 100.000 | 99.668 | 91.517 | 81.111 | 70.825 |
57 | 100.000 | 99.834 | 91.786 | 81.433 | 71.173 |
58 | 100.000 | 100.000 | 92.055 | 81.764 | 71.544 |
59 | 100.000 | 100.000 | 92.158 | 81.987 | 71.867 |
60 | 100.000 | 100.000 | 92.220 | 82.124 | 72.069 |
61 | 100.000 | 100.000 | 92.262 | 82.219 | 72.216 |
62 | 100.000 | 100.000 | 92.882 | 83.273 | 73.432 |
63 | 100.000 | 100.000 | 93.503 | 84.328 | 74.648 |
64 | 100.000 | 100.000 | 93.710 | 84.745 | 75.214 |
65 | 100.000 | 100.000 | 93.834 | 85.006 | 75.581 |
66 | 100.000 | 100.000 | 93.896 | 85.142 | 75.783 |
67 | 100.000 | 100.000 | 94.103 | 85.535 | 76.288 |
68 | 100.000 | 100.000 | 94.269 | 85.883 | 76.777 |
69 | 100.000 | 100.000 | 94.413 | 86.185 | 77.199 |
70 | 100.000 | 100.000 | 94.455 | 86.280 | 77.346 |
71 | 100.000 | 100.000 | 94.682 | 86.722 | 77.923 |
72 | 100.000 | 100.000 | 94.765 | 86.904 | 78.192 |
73 | 100.000 | 100.000 | 94.807 | 86.999 | 78.339 |
74 | 100.000 | 100.000 | 94.972 | 87.346 | 78.828 |
75 | 100.000 | 100.000 | 95.034 | 87.483 | 79.030 |
76 | 100.000 | 100.000 | 95.076 | 87.578 | 79.177 |
77 | 100.000 | 100.000 | 95.241 | 87.909 | 79.622 |
78 | 100.000 | 100.000 | 95.303 | 88.045 | 79.824 |
79 | 100.000 | 100.000 | 95.345 | 88.140 | 79.971 |
80 | 100.000 | 100.000 | 95.448 | 88.368 | 80.304 |
81 | 100.000 | 100.000 | 95.593 | 88.670 | 80.726 |
82 | 100.000 | 100.000 | 95.634 | 88.773 | 80.896 |
83 | 100.000 | 100.000 | 95.717 | 88.947 | 81.142 |
84 | 100.000 | 100.000 | 95.820 | 89.187 | 81.509 |
85 | 100.000 | 100.000 | 95.903 | 89.369 | 81.778 |
86 | 100.000 | 100.000 | 95.924 | 89.422 | 81.870 |
87 | 100.000 | 100.000 | 95.986 | 89.567 | 82.095 |
88 | 100.000 | 100.000 | 96.027 | 89.662 | 82.242 |
89 | 100.000 | 100.000 | 96.069 | 89.757 | 82.389 |
90 | 100.000 | 100.000 | 96.131 | 89.894 | 82.591 |
91 | 100.000 | 100.000 | 96.172 | 89.993 | 82.750 |
92 | 100.000 | 100.000 | 96.234 | 90.134 | 82.964 |
93 | 100.000 | 100.000 | 96.338 | 90.353 | 83.277 |
94 | 100.000 | 100.000 | 96.379 | 90.448 | 83.424 |
95 | 100.000 | 100.000 | 96.420 | 90.543 | 83.571 |
96 | 100.000 | 100.000 | 96.462 | 90.638 | 83.718 |
97 | 100.000 | 100.000 | 96.524 | 90.787 | 83.955 |
98 | 100.000 | 100.000 | 96.586 | 90.923 | 84.157 |
99 | 100.000 | 100.000 | 96.648 | 91.060 | 84.359 |
100 | 100.000 | 100.000 | 96.689 | 91.155 | 84.506 |
101 | 100.000 | 100.000 | 96.710 | 91.205 | 84.586 |
102 | 100.000 | 100.000 | 96.814 | 91.428 | 84.910 |
103 | 100.000 | 100.000 | 96.855 | 91.523 | 85.057 |
104 | 100.000 | 100.000 | 96.917 | 91.668 | 85.282 |
105 | 100.000 | 100.000 | 96.938 | 91.721 | 85.374 |
106 | 100.000 | 100.000 | 96.979 | 91.817 | 85.521 |
107 | 100.000 | 100.000 | 97.103 | 92.077 | 85.889 |
108 | 100.000 | 100.000 | 97.145 | 92.172 | 86.036 |
109 | 100.000 | 100.000 | 97.186 | 92.267 | 86.183 |
110 | 100.000 | 100.000 | 97.289 | 92.491 | 86.506 |
111 | 100.000 | 100.000 | 97.352 | 92.631 | 86.720 |
112 | 100.000 | 100.000 | 97.414 | 92.784 | 86.966 |
113 | 100.000 | 100.000 | 97.434 | 92.838 | 87.058 |
114 | 100.000 | 100.000 | 97.496 | 92.999 | 87.327 |
115 | 100.000 | 100.000 | 97.538 | 93.103 | 87.497 |
116 | 100.000 | 100.000 | 97.579 | 93.198 | 87.644 |
117 | 100.000 | 100.000 | 97.600 | 93.251 | 87.737 |
118 | 100.000 | 100.000 | 97.662 | 93.388 | 87.938 |
119 | 100.000 | 100.000 | 97.724 | 93.524 | 88.140 |
120 | 100.000 | 100.000 | 97.765 | 93.636 | 88.331 |
121 | 100.000 | 100.000 | 97.827 | 93.772 | 88.533 |
122 | 100.000 | 100.000 | 97.848 | 93.822 | 88.613 |
123 | 100.000 | 100.000 | 97.910 | 93.959 | 88.815 |
124 | 100.000 | 100.000 | 97.931 | 94.021 | 88.931 |
125 | 100.000 | 100.000 | 98.034 | 94.240 | 89.243 |
126 | 100.000 | 100.000 | 98.076 | 94.335 | 89.390 |
127 | 100.000 | 100.000 | 98.283 | 94.728 | 89.895 |
128 | 100.000 | 100.000 | 98.324 | 94.823 | 90.042 |
129 | 100.000 | 100.000 | 98.345 | 94.881 | 90.145 |
130 | 100.000 | 100.000 | 98.365 | 94.951 | 90.281 |
131 | 100.000 | 100.000 | 98.407 | 95.046 | 90.429 |
132 | 100.000 | 100.000 | 98.448 | 95.141 | 90.576 |
133 | 100.000 | 100.000 | 98.510 | 95.294 | 90.821 |
134 | 100.000 | 100.000 | 98.572 | 95.431 | 91.023 |
135 | 100.000 | 100.000 | 98.614 | 95.526 | 91.170 |
136 | 100.000 | 100.000 | 98.655 | 95.621 | 91.317 |
137 | 100.000 | 100.000 | 98.696 | 95.716 | 91.465 |
138 | 100.000 | 100.000 | 98.738 | 95.828 | 91.655 |
139 | 100.000 | 100.000 | 98.800 | 95.964 | 91.857 |
140 | 100.000 | 100.000 | 98.821 | 96.018 | 91.950 |
141 | 100.000 | 100.000 | 98.862 | 96.130 | 92.141 |
142 | 100.000 | 100.000 | 98.883 | 96.183 | 92.233 |
143 | 100.000 | 100.000 | 98.903 | 96.237 | 92.325 |
144 | 100.000 | 100.000 | 98.965 | 96.373 | 92.527 |
145 | 100.000 | 100.000 | 99.007 | 96.477 | 92.697 |
146 | 100.000 | 100.000 | 99.028 | 96.526 | 92.778 |
147 | 100.000 | 100.000 | 99.069 | 96.622 | 92.925 |
148 | 100.000 | 100.000 | 99.110 | 96.717 | 93.072 |
149 | 100.000 | 100.000 | 99.152 | 96.828 | 93.263 |
150 | 100.000 | 100.000 | 99.234 | 97.027 | 93.575 |
151 | 100.000 | 100.000 | 99.255 | 97.081 | 93.668 |
152 | 100.000 | 100.000 | 99.297 | 97.176 | 93.815 |
153 | 100.000 | 100.000 | 99.317 | 97.238 | 93.930 |
154 | 100.000 | 100.000 | 99.359 | 97.333 | 94.077 |
155 | 100.000 | 100.000 | 99.379 | 97.387 | 94.170 |
156 | 100.000 | 100.000 | 99.421 | 97.490 | 94.340 |
157 | 100.000 | 100.000 | 99.462 | 97.585 | 94.487 |
158 | 100.000 | 100.000 | 99.545 | 97.767 | 94.756 |
159 | 100.000 | 100.000 | 99.565 | 97.821 | 94.848 |
160 | 100.000 | 100.000 | 99.586 | 97.875 | 94.941 |
161 | 100.000 | 100.000 | 99.607 | 97.924 | 95.021 |
162 | 100.000 | 100.000 | 99.628 | 97.978 | 95.113 |
163 | 100.000 | 100.000 | 99.669 | 98.073 | 95.261 |
164 | 100.000 | 100.000 | 99.690 | 98.135 | 95.376 |
165 | 100.000 | 100.000 | 99.710 | 98.193 | 95.479 |
166 | 100.000 | 100.000 | 99.752 | 98.288 | 95.626 |
167 | 100.000 | 100.000 | 99.855 | 98.511 | 95.950 |
168 | 100.000 | 100.000 | 99.897 | 98.615 | 96.120 |
169 | 100.000 | 100.000 | 99.959 | 98.755 | 96.334 |
170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |


sed
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
4 | 75.000 | 48.837 | 31.450 | 20.692 | 14.035 |
5 | 80.000 | 55.648 | 37.285 | 25.113 | 17.270 |
6 | 87.500 | 62.458 | 42.541 | 29.045 | 20.207 |
7 | 90.000 | 65.781 | 45.976 | 32.147 | 22.801 |
8 | 92.500 | 69.435 | 49.555 | 35.235 | 25.329 |
9 | 95.000 | 72.425 | 52.473 | 37.841 | 27.542 |
10 | 97.500 | 75.249 | 55.121 | 40.181 | 29.550 |
11 | 97.500 | 76.910 | 57.397 | 42.472 | 31.608 |
12 | 100.000 | 79.568 | 59.797 | 44.556 | 33.392 |
13 | 100.000 | 81.063 | 61.825 | 46.595 | 35.236 |
14 | 100.000 | 82.558 | 63.811 | 48.555 | 36.982 |
15 | 100.000 | 83.555 | 65.529 | 50.490 | 38.825 |
16 | 100.000 | 84.884 | 67.308 | 52.264 | 40.426 |
17 | 100.000 | 86.047 | 68.881 | 53.852 | 41.880 |
18 | 100.000 | 87.043 | 70.308 | 55.361 | 43.310 |
19 | 100.000 | 88.040 | 71.695 | 56.800 | 44.662 |
20 | 100.000 | 89.037 | 73.081 | 58.231 | 45.993 |
21 | 100.000 | 89.867 | 74.281 | 59.513 | 47.222 |
22 | 100.000 | 90.698 | 75.419 | 60.671 | 48.286 |
23 | 100.000 | 91.196 | 76.329 | 61.808 | 49.498 |
24 | 100.000 | 91.860 | 77.323 | 62.904 | 50.582 |
25 | 100.000 | 91.860 | 77.943 | 63.958 | 51.798 |
26 | 100.000 | 92.359 | 78.771 | 64.938 | 52.813 |
27 | 100.000 | 92.857 | 79.599 | 65.918 | 53.828 |
28 | 100.000 | 93.355 | 80.426 | 66.898 | 54.843 |
29 | 100.000 | 93.854 | 81.233 | 67.853 | 55.845 |
30 | 100.000 | 94.186 | 81.937 | 68.796 | 56.891 |
31 | 100.000 | 94.518 | 82.599 | 69.661 | 57.838 |
32 | 100.000 | 94.850 | 83.220 | 70.454 | 58.708 |
33 | 100.000 | 95.183 | 83.840 | 71.253 | 59.587 |
34 | 100.000 | 95.515 | 84.440 | 72.009 | 60.412 |
35 | 100.000 | 95.847 | 84.999 | 72.683 | 61.126 |
36 | 100.000 | 96.013 | 85.433 | 73.328 | 61.893 |
37 | 100.000 | 96.179 | 85.868 | 73.957 | 62.617 |
38 | 100.000 | 96.346 | 86.303 | 74.585 | 63.340 |
39 | 100.000 | 96.512 | 86.737 | 75.231 | 64.107 |
40 | 100.000 | 96.678 | 87.172 | 75.871 | 64.864 |
41 | 100.000 | 96.844 | 87.565 | 76.430 | 65.509 |
42 | 100.000 | 97.010 | 87.958 | 76.988 | 66.155 |
43 | 100.000 | 97.176 | 88.351 | 77.546 | 66.800 |
44 | 100.000 | 97.342 | 88.703 | 78.034 | 67.368 |
45 | 100.000 | 97.508 | 89.054 | 78.522 | 67.936 |
46 | 100.000 | 97.674 | 89.406 | 79.010 | 68.504 |
47 | 100.000 | 97.841 | 89.758 | 79.502 | 69.084 |
48 | 100.000 | 98.007 | 90.110 | 79.990 | 69.652 |
49 | 100.000 | 98.173 | 90.461 | 80.478 | 70.219 |
50 | 100.000 | 98.339 | 90.813 | 80.966 | 70.787 |
51 | 100.000 | 98.505 | 91.165 | 81.454 | 71.355 |
52 | 100.000 | 98.671 | 91.496 | 81.901 | 71.868 |
53 | 100.000 | 98.837 | 91.827 | 82.347 | 72.381 |
54 | 100.000 | 99.003 | 92.158 | 82.810 | 72.938 |
55 | 100.000 | 99.169 | 92.489 | 83.265 | 73.474 |
56 | 100.000 | 99.336 | 92.820 | 83.716 | 73.998 |
57 | 100.000 | 99.502 | 93.131 | 84.125 | 74.468 |
58 | 100.000 | 99.668 | 93.441 | 84.543 | 74.958 |
59 | 100.000 | 99.834 | 93.751 | 84.952 | 75.427 |
60 | 100.000 | 100.000 | 94.062 | 85.370 | 75.917 |
61 | 100.000 | 100.000 | 94.269 | 85.763 | 76.422 |
62 | 100.000 | 100.000 | 94.475 | 86.156 | 76.926 |
63 | 100.000 | 100.000 | 94.682 | 86.548 | 77.431 |
64 | 100.000 | 100.000 | 94.827 | 86.859 | 77.876 |
65 | 100.000 | 100.000 | 94.972 | 87.169 | 78.321 |
66 | 100.000 | 100.000 | 95.096 | 87.429 | 78.688 |
67 | 100.000 | 100.000 | 95.220 | 87.690 | 79.055 |
68 | 100.000 | 100.000 | 95.345 | 87.950 | 79.422 |
69 | 100.000 | 100.000 | 95.448 | 88.178 | 79.758 |
70 | 100.000 | 100.000 | 95.551 | 88.401 | 80.081 |
71 | 100.000 | 100.000 | 95.655 | 88.624 | 80.405 |
72 | 100.000 | 100.000 | 95.758 | 88.848 | 80.728 |
73 | 100.000 | 100.000 | 95.841 | 89.042 | 81.030 |
74 | 100.000 | 100.000 | 95.924 | 89.224 | 81.299 |
75 | 100.000 | 100.000 | 96.007 | 89.406 | 81.567 |
76 | 100.000 | 100.000 | 96.089 | 89.604 | 81.880 |
77 | 100.000 | 100.000 | 96.172 | 89.786 | 82.148 |
78 | 100.000 | 100.000 | 96.255 | 89.968 | 82.417 |
79 | 100.000 | 100.000 | 96.317 | 90.105 | 82.619 |
80 | 100.000 | 100.000 | 96.379 | 90.245 | 82.833 |
81 | 100.000 | 100.000 | 96.441 | 90.382 | 83.035 |
82 | 100.000 | 100.000 | 96.503 | 90.522 | 83.249 |
83 | 100.000 | 100.000 | 96.565 | 90.667 | 83.474 |
84 | 100.000 | 100.000 | 96.627 | 90.820 | 83.719 |
85 | 100.000 | 100.000 | 96.689 | 90.965 | 83.944 |
86 | 100.000 | 100.000 | 96.752 | 91.114 | 84.179 |
87 | 100.000 | 100.000 | 96.814 | 91.267 | 84.425 |
88 | 100.000 | 100.000 | 96.876 | 91.403 | 84.626 |
89 | 100.000 | 100.000 | 96.938 | 91.540 | 84.828 |
90 | 100.000 | 100.000 | 97.000 | 91.676 | 85.030 |
91 | 100.000 | 100.000 | 97.062 | 91.812 | 85.232 |
92 | 100.000 | 100.000 | 97.124 | 91.949 | 85.434 |
93 | 100.000 | 100.000 | 97.186 | 92.085 | 85.636 |
94 | 100.000 | 100.000 | 97.248 | 92.222 | 85.837 |
95 | 100.000 | 100.000 | 97.310 | 92.358 | 86.039 |
96 | 100.000 | 100.000 | 97.372 | 92.495 | 86.241 |
97 | 100.000 | 100.000 | 97.434 | 92.631 | 86.443 |
98 | 100.000 | 100.000 | 97.496 | 92.768 | 86.645 |
99 | 100.000 | 100.000 | 97.558 | 92.904 | 86.847 |
100 | 100.000 | 100.000 | 97.621 | 93.041 | 87.048 |
101 | 100.000 | 100.000 | 97.683 | 93.177 | 87.250 |
102 | 100.000 | 100.000 | 97.745 | 93.313 | 87.452 |
103 | 100.000 | 100.000 | 97.807 | 93.466 | 87.698 |
104 | 100.000 | 100.000 | 97.869 | 93.603 | 87.900 |
105 | 100.000 | 100.000 | 97.931 | 93.739 | 88.101 |
106 | 100.000 | 100.000 | 97.993 | 93.876 | 88.303 |
107 | 100.000 | 100.000 | 98.055 | 94.012 | 88.505 |
108 | 100.000 | 100.000 | 98.117 | 94.149 | 88.707 |
109 | 100.000 | 100.000 | 98.158 | 94.244 | 88.854 |
110 | 100.000 | 100.000 | 98.200 | 94.347 | 89.024 |
111 | 100.000 | 100.000 | 98.241 | 94.447 | 89.184 |
112 | 100.000 | 100.000 | 98.283 | 94.542 | 89.331 |
113 | 100.000 | 100.000 | 98.324 | 94.653 | 89.522 |
114 | 100.000 | 100.000 | 98.365 | 94.765 | 89.712 |
115 | 100.000 | 100.000 | 98.407 | 94.868 | 89.883 |
116 | 100.000 | 100.000 | 98.448 | 94.963 | 90.030 |
117 | 100.000 | 100.000 | 98.490 | 95.075 | 90.221 |
118 | 100.000 | 100.000 | 98.531 | 95.170 | 90.368 |
119 | 100.000 | 100.000 | 98.572 | 95.265 | 90.515 |
120 | 100.000 | 100.000 | 98.614 | 95.360 | 90.662 |
121 | 100.000 | 100.000 | 98.655 | 95.455 | 90.809 |
122 | 100.000 | 100.000 | 98.696 | 95.559 | 90.979 |
123 | 100.000 | 100.000 | 98.738 | 95.654 | 91.127 |
124 | 100.000 | 100.000 | 98.779 | 95.749 | 91.274 |
125 | 100.000 | 100.000 | 98.821 | 95.844 | 91.421 |
126 | 100.000 | 100.000 | 98.862 | 95.943 | 91.580 |
127 | 100.000 | 100.000 | 98.903 | 96.047 | 91.750 |
128 | 100.000 | 100.000 | 98.945 | 96.142 | 91.897 |
129 | 100.000 | 100.000 | 98.986 | 96.241 | 92.057 |
130 | 100.000 | 100.000 | 99.028 | 96.336 | 92.204 |
131 | 100.000 | 100.000 | 99.069 | 96.431 | 92.351 |
132 | 100.000 | 100.000 | 99.110 | 96.526 | 92.498 |
133 | 100.000 | 100.000 | 99.152 | 96.622 | 92.645 |
134 | 100.000 | 100.000 | 99.193 | 96.717 | 92.792 |
135 | 100.000 | 100.000 | 99.234 | 96.812 | 92.939 |
136 | 100.000 | 100.000 | 99.276 | 96.907 | 93.087 |
137 | 100.000 | 100.000 | 99.317 | 97.019 | 93.277 |
138 | 100.000 | 100.000 | 99.338 | 97.089 | 93.414 |
139 | 100.000 | 100.000 | 99.359 | 97.151 | 93.529 |
140 | 100.000 | 100.000 | 99.379 | 97.213 | 93.645 |
141 | 100.000 | 100.000 | 99.400 | 97.275 | 93.760 |
142 | 100.000 | 100.000 | 99.421 | 97.337 | 93.876 |
143 | 100.000 | 100.000 | 99.441 | 97.391 | 93.968 |
144 | 100.000 | 100.000 | 99.462 | 97.440 | 94.048 |
145 | 100.000 | 100.000 | 99.483 | 97.498 | 94.153 |
146 | 100.000 | 100.000 | 99.503 | 97.556 | 94.256 |
147 | 100.000 | 100.000 | 99.524 | 97.614 | 94.360 |
148 | 100.000 | 100.000 | 99.545 | 97.664 | 94.440 |
149 | 100.000 | 100.000 | 99.565 | 97.726 | 94.555 |
150 | 100.000 | 100.000 | 99.586 | 97.775 | 94.636 |
151 | 100.000 | 100.000 | 99.607 | 97.829 | 94.728 |
152 | 100.000 | 100.000 | 99.628 | 97.879 | 94.808 |
153 | 100.000 | 100.000 | 99.648 | 97.932 | 94.901 |
154 | 100.000 | 100.000 | 99.669 | 97.982 | 94.981 |
155 | 100.000 | 100.000 | 99.690 | 98.036 | 95.073 |
156 | 100.000 | 100.000 | 99.710 | 98.090 | 95.166 |
157 | 100.000 | 100.000 | 99.731 | 98.139 | 95.246 |
158 | 100.000 | 100.000 | 99.752 | 98.193 | 95.338 |
159 | 100.000 | 100.000 | 99.772 | 98.247 | 95.431 |
160 | 100.000 | 100.000 | 99.793 | 98.300 | 95.523 |
161 | 100.000 | 100.000 | 99.814 | 98.362 | 95.639 |
162 | 100.000 | 100.000 | 99.834 | 98.416 | 95.731 |
163 | 100.000 | 100.000 | 99.855 | 98.470 | 95.823 |
164 | 100.000 | 100.000 | 99.876 | 98.524 | 95.916 |
165 | 100.000 | 100.000 | 99.897 | 98.573 | 95.996 |
166 | 100.000 | 100.000 | 99.917 | 98.627 | 96.089 |
167 | 100.000 | 100.000 | 99.938 | 98.681 | 96.181 |
168 | 100.000 | 100.000 | 99.959 | 98.743 | 96.296 |
169 | 100.000 | 100.000 | 99.979 | 98.797 | 96.389 |
170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |


sed
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 43.023 | 25.926 | 16.354 | 10.818 |
4 | 80.000 | 50.166 | 31.533 | 20.552 | 13.924 |
5 | 80.000 | 54.319 | 36.189 | 24.480 | 16.964 |
6 | 85.000 | 60.133 | 41.196 | 28.417 | 19.960 |
7 | 87.500 | 64.452 | 45.438 | 31.973 | 22.769 |
8 | 92.500 | 69.435 | 49.576 | 35.260 | 25.347 |
9 | 95.000 | 72.591 | 52.576 | 37.865 | 27.525 |
10 | 97.500 | 75.581 | 55.411 | 40.355 | 29.635 |
11 | 100.000 | 78.405 | 58.059 | 42.695 | 31.642 |
12 | 100.000 | 79.568 | 59.921 | 44.717 | 33.528 |
13 | 100.000 | 80.731 | 61.701 | 46.653 | 35.365 |
14 | 100.000 | 82.226 | 63.687 | 48.613 | 37.111 |
15 | 100.000 | 83.056 | 65.115 | 50.271 | 38.753 |
16 | 100.000 | 84.385 | 66.915 | 52.082 | 40.396 |
17 | 100.000 | 85.548 | 68.487 | 53.670 | 41.851 |
18 | 100.000 | 86.711 | 70.039 | 55.241 | 43.305 |
19 | 100.000 | 87.874 | 71.612 | 56.829 | 44.759 |
20 | 100.000 | 87.874 | 72.233 | 57.884 | 45.975 |
21 | 100.000 | 88.704 | 73.453 | 59.207 | 47.259 |
22 | 100.000 | 89.535 | 74.674 | 60.530 | 48.543 |
23 | 100.000 | 90.199 | 75.729 | 61.742 | 49.768 |
24 | 100.000 | 91.030 | 76.888 | 62.929 | 50.858 |
25 | 100.000 | 91.528 | 77.736 | 63.946 | 51.917 |
26 | 100.000 | 92.027 | 78.585 | 64.963 | 52.976 |
27 | 100.000 | 92.359 | 79.247 | 65.844 | 53.962 |
28 | 100.000 | 92.857 | 80.033 | 66.754 | 54.899 |
29 | 100.000 | 93.355 | 80.819 | 67.663 | 55.837 |
30 | 100.000 | 93.688 | 81.440 | 68.465 | 56.727 |
31 | 100.000 | 94.020 | 82.061 | 69.259 | 57.596 |
32 | 100.000 | 94.352 | 82.682 | 70.053 | 58.466 |
33 | 100.000 | 94.684 | 83.282 | 70.806 | 59.280 |
34 | 100.000 | 94.850 | 83.716 | 71.451 | 60.047 |
35 | 100.000 | 95.017 | 84.151 | 72.096 | 60.815 |
36 | 100.000 | 95.183 | 84.585 | 72.741 | 61.582 |
37 | 100.000 | 95.515 | 85.144 | 73.415 | 62.296 |
38 | 100.000 | 95.847 | 85.702 | 74.089 | 63.009 |
39 | 100.000 | 96.013 | 86.116 | 74.685 | 63.699 |
40 | 100.000 | 96.179 | 86.509 | 75.251 | 64.367 |
41 | 100.000 | 96.512 | 87.047 | 75.888 | 65.037 |
42 | 100.000 | 96.844 | 87.585 | 76.525 | 65.707 |
43 | 100.000 | 97.176 | 88.123 | 77.162 | 66.377 |
44 | 100.000 | 97.342 | 88.516 | 77.720 | 67.023 |
45 | 100.000 | 97.508 | 88.910 | 78.278 | 67.669 |
46 | 100.000 | 97.674 | 89.303 | 78.836 | 68.314 |
47 | 100.000 | 97.841 | 89.696 | 79.395 | 68.960 |
48 | 100.000 | 98.007 | 90.048 | 79.891 | 69.551 |
49 | 100.000 | 98.173 | 90.399 | 80.379 | 70.119 |
50 | 100.000 | 98.339 | 90.751 | 80.867 | 70.686 |
51 | 100.000 | 98.505 | 91.082 | 81.322 | 71.220 |
52 | 100.000 | 98.671 | 91.413 | 81.776 | 71.754 |
53 | 100.000 | 98.837 | 91.744 | 82.231 | 72.290 |
54 | 100.000 | 99.003 | 92.075 | 82.686 | 72.826 |
55 | 100.000 | 99.169 | 92.406 | 83.141 | 73.363 |
56 | 100.000 | 99.336 | 92.737 | 83.596 | 73.896 |
57 | 100.000 | 99.336 | 92.944 | 83.989 | 74.401 |
58 | 100.000 | 99.336 | 93.151 | 84.382 | 74.905 |
59 | 100.000 | 99.336 | 93.358 | 84.774 | 75.410 |
60 | 100.000 | 99.502 | 93.689 | 85.221 | 75.923 |
61 | 100.000 | 99.668 | 94.020 | 85.668 | 76.436 |
62 | 100.000 | 99.834 | 94.331 | 86.089 | 76.938 |
63 | 100.000 | 100.000 | 94.641 | 86.499 | 77.408 |
64 | 100.000 | 100.000 | 94.765 | 86.780 | 77.830 |
65 | 100.000 | 100.000 | 94.889 | 87.057 | 78.241 |
66 | 100.000 | 100.000 | 95.013 | 87.326 | 78.631 |
67 | 100.000 | 100.000 | 95.138 | 87.586 | 78.998 |
68 | 100.000 | 100.000 | 95.241 | 87.814 | 79.334 |
69 | 100.000 | 100.000 | 95.345 | 88.037 | 79.657 |
70 | 100.000 | 100.000 | 95.448 | 88.260 | 79.981 |
71 | 100.000 | 100.000 | 95.551 | 88.484 | 80.304 |
72 | 100.000 | 100.000 | 95.655 | 88.707 | 80.627 |
73 | 100.000 | 100.000 | 95.758 | 88.926 | 80.940 |
74 | 100.000 | 100.000 | 95.841 | 89.112 | 81.221 |
75 | 100.000 | 100.000 | 95.924 | 89.298 | 81.502 |
76 | 100.000 | 100.000 | 96.007 | 89.480 | 81.770 |
77 | 100.000 | 100.000 | 96.089 | 89.662 | 82.039 |
78 | 100.000 | 100.000 | 96.172 | 89.844 | 82.308 |
79 | 100.000 | 100.000 | 96.255 | 90.026 | 82.576 |
80 | 100.000 | 100.000 | 96.338 | 90.208 | 82.845 |
81 | 100.000 | 100.000 | 96.420 | 90.390 | 83.114 |
82 | 100.000 | 100.000 | 96.483 | 90.543 | 83.359 |
83 | 100.000 | 100.000 | 96.545 | 90.696 | 83.605 |
84 | 100.000 | 100.000 | 96.607 | 90.849 | 83.851 |
85 | 100.000 | 100.000 | 96.669 | 90.998 | 84.085 |
86 | 100.000 | 100.000 | 96.731 | 91.143 | 84.310 |
87 | 100.000 | 100.000 | 96.793 | 91.287 | 84.535 |
88 | 100.000 | 100.000 | 96.855 | 91.432 | 84.760 |
89 | 100.000 | 100.000 | 96.917 | 91.577 | 84.985 |
90 | 100.000 | 100.000 | 96.979 | 91.717 | 85.199 |
91 | 100.000 | 100.000 | 97.041 | 91.858 | 85.413 |
92 | 100.000 | 100.000 | 97.103 | 91.994 | 85.615 |
93 | 100.000 | 100.000 | 97.165 | 92.131 | 85.817 |
94 | 100.000 | 100.000 | 97.227 | 92.267 | 86.019 |
95 | 100.000 | 100.000 | 97.289 | 92.404 | 86.220 |
96 | 100.000 | 100.000 | 97.352 | 92.540 | 86.422 |
97 | 100.000 | 100.000 | 97.414 | 92.677 | 86.624 |
98 | 100.000 | 100.000 | 97.476 | 92.813 | 86.826 |
99 | 100.000 | 100.000 | 97.538 | 92.950 | 87.028 |
100 | 100.000 | 100.000 | 97.600 | 93.086 | 87.230 |
101 | 100.000 | 100.000 | 97.662 | 93.223 | 87.431 |
102 | 100.000 | 100.000 | 97.724 | 93.359 | 87.633 |
103 | 100.000 | 100.000 | 97.786 | 93.495 | 87.835 |
104 | 100.000 | 100.000 | 97.848 | 93.632 | 88.037 |
105 | 100.000 | 100.000 | 97.910 | 93.768 | 88.239 |
106 | 100.000 | 100.000 | 97.972 | 93.905 | 88.441 |
107 | 100.000 | 100.000 | 98.034 | 94.041 | 88.642 |
108 | 100.000 | 100.000 | 98.076 | 94.153 | 88.833 |
109 | 100.000 | 100.000 | 98.117 | 94.265 | 89.024 |
110 | 100.000 | 100.000 | 98.158 | 94.368 | 89.194 |
111 | 100.000 | 100.000 | 98.200 | 94.471 | 89.365 |
112 | 100.000 | 100.000 | 98.241 | 94.575 | 89.535 |
113 | 100.000 | 100.000 | 98.283 | 94.678 | 89.705 |
114 | 100.000 | 100.000 | 98.324 | 94.781 | 89.875 |
115 | 100.000 | 100.000 | 98.365 | 94.877 | 90.022 |
116 | 100.000 | 100.000 | 98.407 | 94.972 | 90.170 |
117 | 100.000 | 100.000 | 98.448 | 95.067 | 90.317 |
118 | 100.000 | 100.000 | 98.490 | 95.162 | 90.464 |
119 | 100.000 | 100.000 | 98.531 | 95.257 | 90.611 |
120 | 100.000 | 100.000 | 98.572 | 95.352 | 90.758 |
121 | 100.000 | 100.000 | 98.614 | 95.447 | 90.905 |
122 | 100.000 | 100.000 | 98.655 | 95.542 | 91.052 |
123 | 100.000 | 100.000 | 98.696 | 95.637 | 91.199 |
124 | 100.000 | 100.000 | 98.738 | 95.733 | 91.347 |
125 | 100.000 | 100.000 | 98.779 | 95.828 | 91.494 |
126 | 100.000 | 100.000 | 98.821 | 95.923 | 91.641 |
127 | 100.000 | 100.000 | 98.862 | 96.018 | 91.788 |
128 | 100.000 | 100.000 | 98.903 | 96.113 | 91.935 |
129 | 100.000 | 100.000 | 98.945 | 96.208 | 92.082 |
130 | 100.000 | 100.000 | 98.986 | 96.303 | 92.229 |
131 | 100.000 | 100.000 | 99.028 | 96.398 | 92.376 |
132 | 100.000 | 100.000 | 99.069 | 96.493 | 92.524 |
133 | 100.000 | 100.000 | 99.110 | 96.589 | 92.671 |
134 | 100.000 | 100.000 | 99.152 | 96.684 | 92.818 |
135 | 100.000 | 100.000 | 99.193 | 96.779 | 92.965 |
136 | 100.000 | 100.000 | 99.234 | 96.874 | 93.112 |
137 | 100.000 | 100.000 | 99.276 | 96.969 | 93.259 |
138 | 100.000 | 100.000 | 99.317 | 97.064 | 93.406 |
139 | 100.000 | 100.000 | 99.359 | 97.159 | 93.553 |
140 | 100.000 | 100.000 | 99.379 | 97.221 | 93.669 |
141 | 100.000 | 100.000 | 99.400 | 97.283 | 93.784 |
142 | 100.000 | 100.000 | 99.421 | 97.345 | 93.900 |
143 | 100.000 | 100.000 | 99.441 | 97.407 | 94.015 |
144 | 100.000 | 100.000 | 99.462 | 97.469 | 94.131 |
145 | 100.000 | 100.000 | 99.483 | 97.527 | 94.234 |
146 | 100.000 | 100.000 | 99.503 | 97.585 | 94.338 |
147 | 100.000 | 100.000 | 99.524 | 97.643 | 94.441 |
148 | 100.000 | 100.000 | 99.545 | 97.697 | 94.533 |
149 | 100.000 | 100.000 | 99.565 | 97.750 | 94.626 |
150 | 100.000 | 100.000 | 99.586 | 97.804 | 94.718 |
151 | 100.000 | 100.000 | 99.607 | 97.858 | 94.811 |
152 | 100.000 | 100.000 | 99.628 | 97.912 | 94.903 |
153 | 100.000 | 100.000 | 99.648 | 97.966 | 94.995 |
154 | 100.000 | 100.000 | 99.669 | 98.019 | 95.088 |
155 | 100.000 | 100.000 | 99.690 | 98.073 | 95.180 |
156 | 100.000 | 100.000 | 99.710 | 98.127 | 95.273 |
157 | 100.000 | 100.000 | 99.731 | 98.181 | 95.365 |
158 | 100.000 | 100.000 | 99.752 | 98.234 | 95.457 |
159 | 100.000 | 100.000 | 99.772 | 98.288 | 95.550 |
160 | 100.000 | 100.000 | 99.793 | 98.342 | 95.642 |
161 | 100.000 | 100.000 | 99.814 | 98.396 | 95.735 |
162 | 100.000 | 100.000 | 99.834 | 98.449 | 95.827 |
163 | 100.000 | 100.000 | 99.855 | 98.503 | 95.920 |
164 | 100.000 | 100.000 | 99.876 | 98.553 | 96.000 |
165 | 100.000 | 100.000 | 99.897 | 98.602 | 96.080 |
166 | 100.000 | 100.000 | 99.917 | 98.652 | 96.160 |
167 | 100.000 | 100.000 | 99.938 | 98.702 | 96.241 |
168 | 100.000 | 100.000 | 99.959 | 98.751 | 96.321 |
169 | 100.000 | 100.000 | 99.979 | 98.801 | 96.401 |
170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |


sed
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 55.000 | 30.233 | 17.670 | 10.999 | 7.234 |
3 | 70.000 | 41.694 | 25.347 | 16.123 | 10.733 |
4 | 77.500 | 49.169 | 31.264 | 20.523 | 13.950 |
5 | 80.000 | 54.817 | 36.665 | 24.798 | 17.143 |
6 | 82.500 | 59.302 | 41.092 | 28.503 | 20.053 |
7 | 85.000 | 63.455 | 45.169 | 31.944 | 22.794 |
8 | 92.500 | 69.934 | 50.052 | 35.579 | 25.526 |
9 | 95.000 | 72.924 | 52.886 | 38.068 | 27.636 |
10 | 97.500 | 75.748 | 55.535 | 40.409 | 29.643 |
11 | 97.500 | 77.243 | 57.604 | 42.517 | 31.565 |
12 | 100.000 | 79.734 | 59.963 | 44.684 | 33.488 |
13 | 100.000 | 80.565 | 61.556 | 46.557 | 35.307 |
14 | 100.000 | 81.561 | 63.170 | 48.365 | 37.050 |
15 | 100.000 | 82.724 | 64.825 | 50.114 | 38.706 |
16 | 100.000 | 84.053 | 66.646 | 51.958 | 40.384 |
17 | 100.000 | 85.216 | 68.260 | 53.624 | 41.937 |
18 | 100.000 | 86.545 | 70.039 | 55.398 | 43.537 |
19 | 100.000 | 87.708 | 71.633 | 57.024 | 45.035 |
20 | 100.000 | 88.538 | 72.874 | 58.388 | 46.374 |
21 | 100.000 | 88.538 | 73.495 | 59.443 | 47.590 |
22 | 100.000 | 89.203 | 74.550 | 60.650 | 48.805 |
23 | 100.000 | 89.701 | 75.378 | 61.655 | 49.875 |
24 | 100.000 | 90.698 | 76.723 | 63.015 | 51.129 |
25 | 100.000 | 91.196 | 77.550 | 64.012 | 52.186 |
26 | 100.000 | 91.694 | 78.399 | 65.029 | 53.245 |
27 | 100.000 | 92.193 | 79.205 | 65.976 | 54.226 |
28 | 100.000 | 92.359 | 79.681 | 66.691 | 55.071 |
29 | 100.000 | 92.525 | 80.157 | 67.407 | 55.916 |
30 | 100.000 | 93.355 | 81.295 | 68.565 | 56.980 |
31 | 100.000 | 93.688 | 81.895 | 69.330 | 57.824 |
32 | 100.000 | 93.854 | 82.330 | 69.975 | 58.591 |
33 | 100.000 | 94.020 | 82.764 | 70.620 | 59.358 |
34 | 100.000 | 94.352 | 83.364 | 71.377 | 60.184 |
35 | 100.000 | 94.850 | 84.109 | 72.208 | 61.023 |
36 | 100.000 | 95.183 | 84.668 | 72.898 | 61.780 |
37 | 100.000 | 95.349 | 85.082 | 73.502 | 62.493 |
38 | 100.000 | 95.681 | 85.682 | 74.246 | 63.284 |
39 | 100.000 | 96.013 | 86.282 | 74.991 | 64.076 |
40 | 100.000 | 96.346 | 86.840 | 75.673 | 64.812 |
41 | 100.000 | 96.346 | 87.068 | 76.128 | 65.423 |
42 | 100.000 | 96.512 | 87.441 | 76.661 | 66.058 |
43 | 100.000 | 96.678 | 87.834 | 77.220 | 66.703 |
44 | 100.000 | 96.844 | 88.227 | 77.778 | 67.349 |
45 | 100.000 | 97.010 | 88.620 | 78.336 | 67.994 |
46 | 100.000 | 97.176 | 89.013 | 78.894 | 68.640 |
47 | 100.000 | 97.342 | 89.385 | 79.419 | 69.254 |
48 | 100.000 | 97.342 | 89.551 | 79.775 | 69.764 |
49 | 100.000 | 97.674 | 90.089 | 80.412 | 70.433 |
50 | 100.000 | 97.841 | 90.461 | 80.937 | 71.045 |
51 | 100.000 | 98.007 | 90.813 | 81.429 | 71.625 |
52 | 100.000 | 98.339 | 91.330 | 82.033 | 72.261 |
53 | 100.000 | 98.671 | 91.848 | 82.628 | 72.876 |
54 | 100.000 | 98.837 | 92.179 | 83.083 | 73.410 |
55 | 100.000 | 99.003 | 92.510 | 83.538 | 73.944 |
56 | 100.000 | 99.003 | 92.717 | 83.931 | 74.448 |
57 | 100.000 | 99.003 | 92.924 | 84.324 | 74.953 |
58 | 100.000 | 99.003 | 93.131 | 84.717 | 75.457 |
59 | 100.000 | 99.169 | 93.462 | 85.163 | 75.971 |
60 | 100.000 | 99.336 | 93.772 | 85.581 | 76.461 |
61 | 100.000 | 99.502 | 94.082 | 85.994 | 76.942 |
62 | 100.000 | 99.502 | 94.227 | 86.296 | 77.364 |
63 | 100.000 | 99.668 | 94.538 | 86.706 | 77.833 |
64 | 100.000 | 99.834 | 94.848 | 87.115 | 78.303 |
65 | 100.000 | 99.834 | 94.972 | 87.380 | 78.682 |
66 | 100.000 | 99.834 | 95.117 | 87.673 | 79.083 |
67 | 100.000 | 99.834 | 95.241 | 87.934 | 79.450 |
68 | 100.000 | 99.834 | 95.345 | 88.157 | 79.774 |
69 | 100.000 | 100.000 | 95.613 | 88.488 | 80.145 |
70 | 100.000 | 100.000 | 95.717 | 88.707 | 80.457 |
71 | 100.000 | 100.000 | 95.820 | 88.926 | 80.770 |
72 | 100.000 | 100.000 | 95.903 | 89.108 | 81.038 |
73 | 100.000 | 100.000 | 95.965 | 89.261 | 81.285 |
74 | 100.000 | 100.000 | 96.048 | 89.443 | 81.554 |
75 | 100.000 | 100.000 | 96.131 | 89.625 | 81.823 |
76 | 100.000 | 100.000 | 96.193 | 89.778 | 82.068 |
77 | 100.000 | 100.000 | 96.255 | 89.931 | 82.314 |
78 | 100.000 | 100.000 | 96.317 | 90.084 | 82.559 |
79 | 100.000 | 100.000 | 96.379 | 90.237 | 82.805 |
80 | 100.000 | 100.000 | 96.441 | 90.382 | 83.030 |
81 | 100.000 | 100.000 | 96.503 | 90.526 | 83.255 |
82 | 100.000 | 100.000 | 96.565 | 90.671 | 83.480 |
83 | 100.000 | 100.000 | 96.627 | 90.816 | 83.705 |
84 | 100.000 | 100.000 | 96.689 | 90.961 | 83.930 |
85 | 100.000 | 100.000 | 96.752 | 91.105 | 84.155 |
86 | 100.000 | 100.000 | 96.814 | 91.250 | 84.380 |
87 | 100.000 | 100.000 | 96.876 | 91.391 | 84.594 |
88 | 100.000 | 100.000 | 96.917 | 91.502 | 84.784 |
89 | 100.000 | 100.000 | 96.958 | 91.614 | 84.975 |
90 | 100.000 | 100.000 | 97.000 | 91.726 | 85.166 |
91 | 100.000 | 100.000 | 97.041 | 91.837 | 85.357 |
92 | 100.000 | 100.000 | 97.083 | 91.949 | 85.548 |
93 | 100.000 | 100.000 | 97.124 | 92.061 | 85.739 |
94 | 100.000 | 100.000 | 97.165 | 92.172 | 85.930 |
95 | 100.000 | 100.000 | 97.227 | 92.309 | 86.132 |
96 | 100.000 | 100.000 | 97.289 | 92.445 | 86.334 |
97 | 100.000 | 100.000 | 97.352 | 92.582 | 86.535 |
98 | 100.000 | 100.000 | 97.414 | 92.718 | 86.737 |
99 | 100.000 | 100.000 | 97.476 | 92.854 | 86.939 |
100 | 100.000 | 100.000 | 97.538 | 92.991 | 87.141 |
101 | 100.000 | 100.000 | 97.600 | 93.127 | 87.343 |
102 | 100.000 | 100.000 | 97.662 | 93.264 | 87.545 |
103 | 100.000 | 100.000 | 97.724 | 93.400 | 87.746 |
104 | 100.000 | 100.000 | 97.786 | 93.537 | 87.948 |
105 | 100.000 | 100.000 | 97.848 | 93.673 | 88.150 |
106 | 100.000 | 100.000 | 97.910 | 93.810 | 88.352 |
107 | 100.000 | 100.000 | 97.972 | 93.946 | 88.554 |
108 | 100.000 | 100.000 | 98.034 | 94.083 | 88.756 |
109 | 100.000 | 100.000 | 98.096 | 94.219 | 88.957 |
110 | 100.000 | 100.000 | 98.138 | 94.322 | 89.128 |
111 | 100.000 | 100.000 | 98.179 | 94.426 | 89.298 |
112 | 100.000 | 100.000 | 98.221 | 94.529 | 89.468 |
113 | 100.000 | 100.000 | 98.262 | 94.633 | 89.638 |
114 | 100.000 | 100.000 | 98.303 | 94.736 | 89.808 |
115 | 100.000 | 100.000 | 98.345 | 94.839 | 89.979 |
116 | 100.000 | 100.000 | 98.386 | 94.934 | 90.126 |
117 | 100.000 | 100.000 | 98.427 | 95.030 | 90.273 |
118 | 100.000 | 100.000 | 98.469 | 95.125 | 90.420 |
119 | 100.000 | 100.000 | 98.510 | 95.220 | 90.567 |
120 | 100.000 | 100.000 | 98.552 | 95.315 | 90.714 |
121 | 100.000 | 100.000 | 98.593 | 95.410 | 90.861 |
122 | 100.000 | 100.000 | 98.634 | 95.505 | 91.009 |
123 | 100.000 | 100.000 | 98.676 | 95.600 | 91.156 |
124 | 100.000 | 100.000 | 98.717 | 95.695 | 91.303 |
125 | 100.000 | 100.000 | 98.759 | 95.790 | 91.450 |
126 | 100.000 | 100.000 | 98.800 | 95.886 | 91.597 |
127 | 100.000 | 100.000 | 98.841 | 95.981 | 91.744 |
128 | 100.000 | 100.000 | 98.883 | 96.076 | 91.891 |
129 | 100.000 | 100.000 | 98.924 | 96.171 | 92.038 |
130 | 100.000 | 100.000 | 98.965 | 96.266 | 92.186 |
131 | 100.000 | 100.000 | 99.007 | 96.361 | 92.333 |
132 | 100.000 | 100.000 | 99.048 | 96.456 | 92.480 |
133 | 100.000 | 100.000 | 99.090 | 96.551 | 92.627 |
134 | 100.000 | 100.000 | 99.131 | 96.646 | 92.774 |
135 | 100.000 | 100.000 | 99.172 | 96.742 | 92.921 |
136 | 100.000 | 100.000 | 99.214 | 96.837 | 93.068 |
137 | 100.000 | 100.000 | 99.255 | 96.932 | 93.215 |
138 | 100.000 | 100.000 | 99.297 | 97.027 | 93.363 |
139 | 100.000 | 100.000 | 99.338 | 97.122 | 93.510 |
140 | 100.000 | 100.000 | 99.379 | 97.217 | 93.657 |
141 | 100.000 | 100.000 | 99.400 | 97.279 | 93.772 |
142 | 100.000 | 100.000 | 99.421 | 97.341 | 93.888 |
143 | 100.000 | 100.000 | 99.441 | 97.403 | 94.003 |
144 | 100.000 | 100.000 | 99.462 | 97.465 | 94.119 |
145 | 100.000 | 100.000 | 99.483 | 97.527 | 94.234 |
146 | 100.000 | 100.000 | 99.503 | 97.589 | 94.350 |
147 | 100.000 | 100.000 | 99.524 | 97.647 | 94.453 |
148 | 100.000 | 100.000 | 99.545 | 97.701 | 94.546 |
149 | 100.000 | 100.000 | 99.565 | 97.755 | 94.638 |
150 | 100.000 | 100.000 | 99.586 | 97.808 | 94.730 |
151 | 100.000 | 100.000 | 99.607 | 97.862 | 94.823 |
152 | 100.000 | 100.000 | 99.628 | 97.916 | 94.915 |
153 | 100.000 | 100.000 | 99.648 | 97.970 | 95.008 |
154 | 100.000 | 100.000 | 99.669 | 98.023 | 95.100 |
155 | 100.000 | 100.000 | 99.690 | 98.077 | 95.192 |
156 | 100.000 | 100.000 | 99.710 | 98.131 | 95.285 |
157 | 100.000 | 100.000 | 99.731 | 98.185 | 95.377 |
158 | 100.000 | 100.000 | 99.752 | 98.238 | 95.470 |
159 | 100.000 | 100.000 | 99.772 | 98.292 | 95.562 |
160 | 100.000 | 100.000 | 99.793 | 98.346 | 95.654 |
161 | 100.000 | 100.000 | 99.814 | 98.400 | 95.747 |
162 | 100.000 | 100.000 | 99.834 | 98.453 | 95.839 |
163 | 100.000 | 100.000 | 99.855 | 98.503 | 95.920 |
164 | 100.000 | 100.000 | 99.876 | 98.553 | 96.000 |
165 | 100.000 | 100.000 | 99.897 | 98.602 | 96.080 |
166 | 100.000 | 100.000 | 99.917 | 98.652 | 96.160 |
167 | 100.000 | 100.000 | 99.938 | 98.702 | 96.241 |
168 | 100.000 | 100.000 | 99.959 | 98.751 | 96.321 |
169 | 100.000 | 100.000 | 99.979 | 98.801 | 96.401 |
170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |


sed
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
5 | 77.500 | 54.153 | 36.582 | 24.819 | 17.161 |
6 | 85.000 | 60.631 | 41.486 | 28.487 | 19.927 |
7 | 90.000 | 65.781 | 45.810 | 31.923 | 22.608 |
8 | 92.500 | 69.767 | 49.638 | 35.128 | 25.169 |
9 | 95.000 | 72.924 | 52.638 | 37.733 | 27.346 |
10 | 97.500 | 75.748 | 55.535 | 40.421 | 29.669 |
11 | 100.000 | 78.571 | 58.101 | 42.613 | 31.500 |
12 | 100.000 | 80.066 | 60.170 | 44.721 | 33.422 |
13 | 100.000 | 81.561 | 62.239 | 46.822 | 35.326 |
14 | 100.000 | 83.056 | 64.329 | 48.948 | 37.248 |
15 | 100.000 | 84.219 | 65.901 | 50.536 | 38.703 |
16 | 100.000 | 85.382 | 67.474 | 52.123 | 40.157 |
17 | 100.000 | 86.379 | 68.777 | 53.397 | 41.289 |
18 | 100.000 | 87.375 | 70.122 | 54.757 | 42.542 |
19 | 100.000 | 88.372 | 71.591 | 56.329 | 44.032 |
20 | 100.000 | 89.203 | 72.771 | 57.574 | 45.217 |
21 | 100.000 | 90.033 | 73.909 | 58.731 | 46.281 |
22 | 100.000 | 90.864 | 75.460 | 60.551 | 48.056 |
23 | 100.000 | 91.362 | 76.309 | 61.572 | 49.126 |
24 | 100.000 | 91.860 | 77.136 | 62.552 | 50.142 |
25 | 100.000 | 92.359 | 77.985 | 63.569 | 51.201 |
26 | 100.000 | 92.857 | 78.854 | 64.628 | 52.314 |
27 | 100.000 | 93.355 | 79.764 | 65.769 | 53.536 |
28 | 100.000 | 93.688 | 80.364 | 66.522 | 54.351 |
29 | 100.000 | 94.020 | 81.006 | 67.357 | 55.275 |
30 | 100.000 | 94.352 | 81.523 | 67.953 | 55.890 |
31 | 100.000 | 94.684 | 82.164 | 68.792 | 56.824 |
32 | 100.000 | 95.017 | 82.744 | 69.499 | 57.572 |
33 | 100.000 | 95.349 | 83.344 | 70.252 | 58.387 |
34 | 100.000 | 95.681 | 83.944 | 71.004 | 59.201 |
35 | 100.000 | 95.847 | 84.378 | 71.650 | 59.968 |
36 | 100.000 | 96.013 | 84.813 | 72.295 | 60.736 |
37 | 100.000 | 96.179 | 85.082 | 72.617 | 61.083 |
38 | 100.000 | 96.346 | 85.433 | 73.122 | 61.695 |
39 | 100.000 | 96.512 | 85.806 | 73.655 | 62.330 |
40 | 100.000 | 96.678 | 86.220 | 74.259 | 63.042 |
41 | 100.000 | 96.844 | 86.551 | 74.705 | 63.555 |
42 | 100.000 | 97.010 | 86.903 | 75.193 | 64.123 |
43 | 100.000 | 97.176 | 87.172 | 75.516 | 64.471 |
44 | 100.000 | 97.342 | 87.544 | 76.041 | 65.082 |
45 | 100.000 | 97.508 | 87.896 | 76.521 | 65.630 |
46 | 100.000 | 97.674 | 88.206 | 76.930 | 66.099 |
47 | 100.000 | 97.841 | 88.496 | 77.298 | 66.513 |
48 | 100.000 | 98.007 | 88.785 | 77.666 | 66.928 |
49 | 100.000 | 98.173 | 89.137 | 78.154 | 67.496 |
50 | 100.000 | 98.339 | 89.406 | 78.477 | 67.844 |
51 | 100.000 | 98.505 | 89.799 | 79.051 | 68.533 |
52 | 100.000 | 98.671 | 90.130 | 79.498 | 69.046 |
53 | 100.000 | 98.837 | 90.399 | 79.821 | 69.394 |
54 | 100.000 | 99.003 | 90.710 | 80.230 | 69.863 |
55 | 100.000 | 99.169 | 91.020 | 80.639 | 70.333 |
56 | 100.000 | 99.336 | 91.372 | 81.127 | 70.900 |
57 | 100.000 | 99.502 | 91.641 | 81.450 | 71.248 |
58 | 100.000 | 99.668 | 91.930 | 81.818 | 71.663 |
59 | 100.000 | 99.834 | 92.220 | 82.186 | 72.077 |
60 | 100.000 | 100.000 | 92.489 | 82.508 | 72.425 |
61 | 100.000 | 100.000 | 92.551 | 82.649 | 72.639 |
62 | 100.000 | 100.000 | 92.572 | 82.707 | 72.742 |
63 | 100.000 | 100.000 | 92.593 | 82.756 | 72.823 |
64 | 100.000 | 100.000 | 92.717 | 83.025 | 73.213 |
65 | 100.000 | 100.000 | 92.800 | 83.199 | 73.459 |
66 | 100.000 | 100.000 | 92.903 | 83.418 | 73.771 |
67 | 100.000 | 100.000 | 93.048 | 83.720 | 74.193 |
68 | 100.000 | 100.000 | 93.131 | 83.910 | 74.485 |
69 | 100.000 | 100.000 | 93.172 | 84.005 | 74.632 |
70 | 100.000 | 100.000 | 93.213 | 84.100 | 74.779 |
71 | 100.000 | 100.000 | 93.358 | 84.402 | 75.201 |
72 | 100.000 | 100.000 | 93.462 | 84.626 | 75.524 |
73 | 100.000 | 100.000 | 93.524 | 84.779 | 75.770 |
74 | 100.000 | 100.000 | 93.524 | 84.787 | 75.795 |
75 | 100.000 | 100.000 | 93.544 | 84.845 | 75.899 |
76 | 100.000 | 100.000 | 93.606 | 84.985 | 76.113 |
77 | 100.000 | 100.000 | 93.627 | 85.035 | 76.193 |
78 | 100.000 | 100.000 | 93.669 | 85.138 | 76.361 |
79 | 100.000 | 100.000 | 93.731 | 85.279 | 76.575 |
80 | 100.000 | 100.000 | 93.855 | 85.539 | 76.942 |
81 | 100.000 | 100.000 | 93.938 | 85.721 | 77.211 |
82 | 100.000 | 100.000 | 94.041 | 85.945 | 77.534 |
83 | 100.000 | 100.000 | 94.082 | 86.040 | 77.681 |
84 | 100.000 | 100.000 | 94.165 | 86.222 | 77.950 |
85 | 100.000 | 100.000 | 94.248 | 86.404 | 78.219 |
86 | 100.000 | 100.000 | 94.289 | 86.507 | 78.389 |
87 | 100.000 | 100.000 | 94.331 | 86.602 | 78.536 |
88 | 100.000 | 100.000 | 94.351 | 86.664 | 78.652 |
89 | 100.000 | 100.000 | 94.455 | 86.887 | 78.975 |
90 | 100.000 | 100.000 | 94.475 | 86.945 | 79.078 |
91 | 100.000 | 100.000 | 94.558 | 87.127 | 79.347 |
92 | 100.000 | 100.000 | 94.558 | 87.131 | 79.360 |
93 | 100.000 | 100.000 | 94.641 | 87.313 | 79.629 |
94 | 100.000 | 100.000 | 94.641 | 87.318 | 79.643 |
95 | 100.000 | 100.000 | 94.682 | 87.429 | 79.833 |
96 | 100.000 | 100.000 | 94.703 | 87.479 | 79.914 |
97 | 100.000 | 100.000 | 94.786 | 87.669 | 80.203 |
98 | 100.000 | 100.000 | 94.848 | 87.805 | 80.405 |
99 | 100.000 | 100.000 | 94.889 | 87.901 | 80.552 |
100 | 100.000 | 100.000 | 94.931 | 87.996 | 80.699 |
101 | 100.000 | 100.000 | 95.055 | 88.256 | 81.066 |
102 | 100.000 | 100.000 | 95.096 | 88.360 | 81.237 |
103 | 100.000 | 100.000 | 95.096 | 88.364 | 81.250 |
104 | 100.000 | 100.000 | 95.158 | 88.500 | 81.452 |
105 | 100.000 | 100.000 | 95.158 | 88.525 | 81.521 |
106 | 100.000 | 100.000 | 95.220 | 88.661 | 81.723 |
107 | 100.000 | 100.000 | 95.282 | 88.798 | 81.925 |
108 | 100.000 | 100.000 | 95.427 | 89.100 | 82.347 |
109 | 100.000 | 100.000 | 95.427 | 89.116 | 82.395 |
110 | 100.000 | 100.000 | 95.551 | 89.377 | 82.762 |
111 | 100.000 | 100.000 | 95.613 | 89.513 | 82.964 |
112 | 100.000 | 100.000 | 95.634 | 89.563 | 83.045 |
113 | 100.000 | 100.000 | 95.676 | 89.658 | 83.192 |
114 | 100.000 | 100.000 | 95.820 | 89.960 | 83.614 |
115 | 100.000 | 100.000 | 95.841 | 90.010 | 83.694 |
116 | 100.000 | 100.000 | 95.882 | 90.121 | 83.885 |
117 | 100.000 | 100.000 | 95.924 | 90.216 | 84.032 |
118 | 100.000 | 100.000 | 95.945 | 90.266 | 84.112 |
119 | 100.000 | 100.000 | 96.007 | 90.402 | 84.314 |
120 | 100.000 | 100.000 | 96.027 | 90.469 | 84.438 |
121 | 100.000 | 100.000 | 96.048 | 90.518 | 84.518 |
122 | 100.000 | 100.000 | 96.089 | 90.613 | 84.665 |
123 | 100.000 | 100.000 | 96.172 | 90.812 | 84.978 |
124 | 100.000 | 100.000 | 96.214 | 90.907 | 85.125 |
125 | 100.000 | 100.000 | 96.214 | 90.915 | 85.150 |
126 | 100.000 | 100.000 | 96.234 | 90.973 | 85.254 |
127 | 100.000 | 100.000 | 96.255 | 91.035 | 85.369 |
128 | 100.000 | 100.000 | 96.255 | 91.043 | 85.395 |
129 | 100.000 | 100.000 | 96.296 | 91.138 | 85.542 |
130 | 100.000 | 100.000 | 96.317 | 91.188 | 85.622 |
131 | 100.000 | 100.000 | 96.379 | 91.329 | 85.836 |
132 | 100.000 | 100.000 | 96.379 | 91.341 | 85.873 |
133 | 100.000 | 100.000 | 96.379 | 91.349 | 85.898 |
134 | 100.000 | 100.000 | 96.400 | 91.399 | 85.978 |
135 | 100.000 | 100.000 | 96.441 | 91.494 | 86.126 |
136 | 100.000 | 100.000 | 96.483 | 91.593 | 86.285 |
137 | 100.000 | 100.000 | 96.483 | 91.597 | 86.298 |
138 | 100.000 | 100.000 | 96.503 | 91.647 | 86.379 |
139 | 100.000 | 100.000 | 96.545 | 91.742 | 86.526 |
140 | 100.000 | 100.000 | 96.586 | 91.837 | 86.673 |
141 | 100.000 | 100.000 | 96.607 | 91.887 | 86.753 |
142 | 100.000 | 100.000 | 96.648 | 91.982 | 86.900 |
143 | 100.000 | 100.000 | 96.648 | 91.986 | 86.913 |
144 | 100.000 | 100.000 | 96.648 | 91.994 | 86.939 |
145 | 100.000 | 100.000 | 96.689 | 92.089 | 87.086 |
146 | 100.000 | 100.000 | 96.689 | 92.098 | 87.112 |
147 | 100.000 | 100.000 | 96.689 | 92.106 | 87.137 |
148 | 100.000 | 100.000 | 96.689 | 92.114 | 87.163 |
149 | 100.000 | 100.000 | 96.731 | 92.209 | 87.310 |
150 | 100.000 | 100.000 | 96.772 | 92.305 | 87.457 |
151 | 100.000 | 100.000 | 96.814 | 92.416 | 87.648 |
152 | 100.000 | 100.000 | 96.855 | 92.511 | 87.795 |
153 | 100.000 | 100.000 | 96.855 | 92.520 | 87.821 |
154 | 100.000 | 100.000 | 96.876 | 92.586 | 87.945 |
155 | 100.000 | 100.000 | 96.896 | 92.635 | 88.025 |
156 | 100.000 | 100.000 | 96.896 | 92.644 | 88.050 |
157 | 100.000 | 100.000 | 96.896 | 92.648 | 88.064 |
158 | 100.000 | 100.000 | 96.938 | 92.751 | 88.234 |
159 | 100.000 | 100.000 | 97.000 | 92.888 | 88.436 |
160 | 100.000 | 100.000 | 97.621 | 93.942 | 89.652 |
161 | 100.000 | 100.000 | 97.683 | 94.078 | 89.853 |
162 | 100.000 | 100.000 | 97.683 | 94.083 | 89.867 |
163 | 100.000 | 100.000 | 97.683 | 94.087 | 89.880 |
164 | 100.000 | 100.000 | 97.724 | 94.182 | 90.027 |
165 | 100.000 | 100.000 | 97.724 | 94.190 | 90.053 |
166 | 100.000 | 100.000 | 97.724 | 94.203 | 90.089 |
167 | 100.000 | 100.000 | 97.724 | 94.211 | 90.115 |
168 | 100.000 | 100.000 | 97.724 | 94.219 | 90.140 |
169 | 100.000 | 100.000 | 97.765 | 94.314 | 90.288 |
170 | 100.000 | 100.000 | 97.786 | 94.364 | 90.368 |
171 | 100.000 | 100.000 | 97.786 | 94.372 | 90.393 |
172 | 100.000 | 100.000 | 97.827 | 94.467 | 90.540 |
173 | 100.000 | 100.000 | 97.848 | 94.521 | 90.633 |
174 | 100.000 | 100.000 | 97.848 | 94.525 | 90.646 |
175 | 100.000 | 100.000 | 97.931 | 94.707 | 90.915 |
176 | 100.000 | 100.000 | 97.952 | 94.757 | 90.995 |
177 | 100.000 | 100.000 | 97.972 | 94.806 | 91.075 |
178 | 100.000 | 100.000 | 97.993 | 94.856 | 91.156 |
179 | 100.000 | 100.000 | 97.993 | 94.872 | 91.204 |
180 | 100.000 | 100.000 | 98.034 | 94.968 | 91.351 |
181 | 100.000 | 100.000 | 98.034 | 94.980 | 91.388 |
182 | 100.000 | 100.000 | 98.055 | 95.030 | 91.468 |
183 | 100.000 | 100.000 | 98.076 | 95.083 | 91.561 |
184 | 100.000 | 100.000 | 98.117 | 95.178 | 91.708 |
185 | 100.000 | 100.000 | 98.158 | 95.274 | 91.855 |
186 | 100.000 | 100.000 | 98.200 | 95.369 | 92.002 |
187 | 100.000 | 100.000 | 98.200 | 95.373 | 92.015 |
188 | 100.000 | 100.000 | 98.221 | 95.439 | 92.139 |
189 | 100.000 | 100.000 | 98.241 | 95.489 | 92.220 |
190 | 100.000 | 100.000 | 98.241 | 95.505 | 92.268 |
191 | 100.000 | 100.000 | 98.262 | 95.555 | 92.348 |
192 | 100.000 | 100.000 | 98.262 | 95.559 | 92.362 |
193 | 100.000 | 100.000 | 98.262 | 95.563 | 92.375 |
194 | 100.000 | 100.000 | 98.303 | 95.658 | 92.522 |
195 | 100.000 | 100.000 | 98.303 | 95.671 | 92.559 |
196 | 100.000 | 100.000 | 98.303 | 95.675 | 92.572 |
197 | 100.000 | 100.000 | 98.365 | 95.811 | 92.774 |
198 | 100.000 | 100.000 | 98.386 | 95.861 | 92.854 |
199 | 100.000 | 100.000 | 98.448 | 95.997 | 93.056 |
200 | 100.000 | 100.000 | 98.448 | 96.001 | 93.069 |
201 | 100.000 | 100.000 | 98.448 | 96.010 | 93.095 |
202 | 100.000 | 100.000 | 98.448 | 96.014 | 93.108 |
203 | 100.000 | 100.000 | 98.448 | 96.039 | 93.178 |
204 | 100.000 | 100.000 | 98.490 | 96.134 | 93.325 |
205 | 100.000 | 100.000 | 98.531 | 96.229 | 93.472 |
206 | 100.000 | 100.000 | 98.531 | 96.237 | 93.497 |
207 | 100.000 | 100.000 | 98.531 | 96.245 | 93.523 |
208 | 100.000 | 100.000 | 98.552 | 96.295 | 93.603 |
209 | 100.000 | 100.000 | 98.572 | 96.345 | 93.684 |
210 | 100.000 | 100.000 | 98.572 | 96.357 | 93.720 |
211 | 100.000 | 100.000 | 98.572 | 96.365 | 93.746 |
212 | 100.000 | 100.000 | 98.572 | 96.373 | 93.771 |
213 | 100.000 | 100.000 | 98.634 | 96.510 | 93.973 |
214 | 100.000 | 100.000 | 98.634 | 96.522 | 94.009 |
215 | 100.000 | 100.000 | 98.634 | 96.531 | 94.035 |
216 | 100.000 | 100.000 | 98.655 | 96.580 | 94.115 |
217 | 100.000 | 100.000 | 98.676 | 96.630 | 94.195 |
218 | 100.000 | 100.000 | 98.696 | 96.679 | 94.276 |
219 | 100.000 | 100.000 | 98.696 | 96.688 | 94.301 |
220 | 100.000 | 100.000 | 98.696 | 96.692 | 94.315 |
221 | 100.000 | 100.000 | 98.696 | 96.696 | 94.328 |
222 | 100.000 | 100.000 | 98.717 | 96.750 | 94.420 |
223 | 100.000 | 100.000 | 98.738 | 96.804 | 94.513 |
224 | 100.000 | 100.000 | 98.738 | 96.808 | 94.526 |
225 | 100.000 | 100.000 | 98.759 | 96.857 | 94.606 |
226 | 100.000 | 100.000 | 98.759 | 96.866 | 94.632 |
227 | 100.000 | 100.000 | 98.759 | 96.870 | 94.645 |
228 | 100.000 | 100.000 | 98.800 | 96.965 | 94.792 |
229 | 100.000 | 100.000 | 98.821 | 97.014 | 94.873 |
230 | 100.000 | 100.000 | 98.821 | 97.019 | 94.886 |
231 | 100.000 | 100.000 | 98.821 | 97.027 | 94.912 |
232 | 100.000 | 100.000 | 98.821 | 97.039 | 94.948 |
233 | 100.000 | 100.000 | 98.821 | 97.043 | 94.961 |
234 | 100.000 | 100.000 | 98.862 | 97.138 | 95.109 |
235 | 100.000 | 100.000 | 98.862 | 97.143 | 95.122 |
236 | 100.000 | 100.000 | 98.903 | 97.238 | 95.269 |
237 | 100.000 | 100.000 | 98.945 | 97.333 | 95.416 |
238 | 100.000 | 100.000 | 98.986 | 97.428 | 95.563 |
239 | 100.000 | 100.000 | 98.986 | 97.436 | 95.589 |
240 | 100.000 | 100.000 | 99.028 | 97.531 | 95.736 |
241 | 100.000 | 100.000 | 99.069 | 97.626 | 95.883 |
242 | 100.000 | 100.000 | 99.069 | 97.635 | 95.909 |
243 | 100.000 | 100.000 | 99.069 | 97.643 | 95.934 |
244 | 100.000 | 100.000 | 99.110 | 97.738 | 96.081 |
245 | 100.000 | 100.000 | 99.110 | 97.746 | 96.107 |
246 | 100.000 | 100.000 | 99.110 | 97.755 | 96.132 |
247 | 100.000 | 100.000 | 99.110 | 97.763 | 96.158 |
248 | 100.000 | 100.000 | 99.110 | 97.767 | 96.171 |
249 | 100.000 | 100.000 | 99.110 | 97.771 | 96.185 |
250 | 100.000 | 100.000 | 99.110 | 97.775 | 96.198 |
251 | 100.000 | 100.000 | 99.152 | 97.870 | 96.345 |
252 | 100.000 | 100.000 | 99.152 | 97.875 | 96.358 |
253 | 100.000 | 100.000 | 99.152 | 97.879 | 96.372 |
254 | 100.000 | 100.000 | 99.152 | 97.887 | 96.397 |
255 | 100.000 | 100.000 | 99.152 | 97.891 | 96.411 |
256 | 100.000 | 100.000 | 99.152 | 97.899 | 96.436 |
257 | 100.000 | 100.000 | 99.152 | 97.908 | 96.462 |
258 | 100.000 | 100.000 | 99.172 | 97.957 | 96.542 |
259 | 100.000 | 100.000 | 99.379 | 98.350 | 97.047 |
260 | 100.000 | 100.000 | 99.379 | 98.358 | 97.072 |
261 | 100.000 | 100.000 | 99.379 | 98.367 | 97.098 |
262 | 100.000 | 100.000 | 99.379 | 98.371 | 97.111 |
263 | 100.000 | 100.000 | 99.379 | 98.375 | 97.124 |
264 | 100.000 | 100.000 | 99.379 | 98.391 | 97.173 |
265 | 100.000 | 100.000 | 99.379 | 98.400 | 97.199 |
266 | 100.000 | 100.000 | 99.421 | 98.495 | 97.346 |
267 | 100.000 | 100.000 | 99.441 | 98.549 | 97.438 |
268 | 100.000 | 100.000 | 99.441 | 98.553 | 97.452 |
269 | 100.000 | 100.000 | 99.462 | 98.602 | 97.532 |
270 | 100.000 | 100.000 | 99.483 | 98.652 | 97.612 |
271 | 100.000 | 100.000 | 99.483 | 98.656 | 97.625 |
272 | 100.000 | 100.000 | 99.483 | 98.664 | 97.651 |
273 | 100.000 | 100.000 | 99.483 | 98.668 | 97.664 |
274 | 100.000 | 100.000 | 99.483 | 98.677 | 97.690 |
275 | 100.000 | 100.000 | 99.483 | 98.693 | 97.738 |
276 | 100.000 | 100.000 | 99.483 | 98.697 | 97.752 |
277 | 100.000 | 100.000 | 99.483 | 98.702 | 97.765 |
278 | 100.000 | 100.000 | 99.483 | 98.710 | 97.791 |
279 | 100.000 | 100.000 | 99.483 | 98.726 | 97.839 |
280 | 100.000 | 100.000 | 99.483 | 98.735 | 97.865 |
281 | 100.000 | 100.000 | 99.483 | 98.739 | 97.878 |