Higher Strength Combinatorial Interaction Testing
This page is supplementary to the paper entitled "Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing" ([PDF] [BIB]), which is currently to appear in the proceedings of the ESEC/FSE 2013.
Abstract
Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints and should, ideally, report a test priority order that maximises earliest fault detection. We show that we can achieve the highest strength CIT in 5.65 minutes on average. This was previously thought to be too computationally expensive to be feasible. Furthermore, we show that higher strength suites find more faults, while prioritisations using lower strengths are no worse at achieving early fault revelation.
Experimental Setup
Subject Programs
We use five Unix utilities from the Software Infrastructure Repository (SIR), maintained by University of Nebraska Lincoln: flex, grep, gzip, sed, and make.
Test Suites
The test suites we use are subsets of those provided by SIR. It can be downloaded from here. For each of the subjects we provide:
-
An annotated TSL file (from SIR) :
_v0_#_small.tsl
Description: The v0_# matches the associated TSL name from SIR for that subject. The leftmost column in this file is a mapping to the covering array values given as sequential numbers starting at 0. For instance, the first parameter if binary will be 0,1 and the next parameter's values will start at 2. The single and error parameters (and any other parameters that we did not include in our model) are not numbered and not used in our experiments. -
An input model file for CASA:
_v0_small.model
Description: The first row is the strength (t) and can be changed to 3,4 etc. to generate higher strength arrays. The second row is the number of factors (TSL parameters in the model). The last row lists the number of values (TSL choices) for each of the parameters. This model matches the associated TSL file. -
The constraint file in CASA input format:
_v0_small.constraints
Description: The first row of this file is the number of constraints. For each of the constraints there are two rows. The first row gives the arity of the constraint (i.e. how many values will be included in this constraint). The next row is list of the constraints between values. The - signs indicate excludes. For instance, a row with a 2, followed by a row that has -10 -12 encodes the binary constraint that values 10 and 12 cannot appear together in a test. Please see the CASA tool for full documentation.
Combinatorial Interaction Testing Tool
We use CASA (Covering Arrays by Simulated Annealing) tool to generate combinatorial interaction test suites from the TSL files.
Full Results
This section contains the full APCC data for all experiments and plots for APCC and fault detection.
| Subject | Generation Strength | Prioritsation Strengths | Subject | Generation Strength | Prioritsation Strengths |
| flex | 2 | 2, 3, 4, 5 | gzip | 2 | 2, 3, 4, 5 |
| flex | 3 | 2, 3, 4, 5 | gzip | 3 | 2, 3, 4, 5 |
| flex | 4 | 2, 3, 4, 5 | gzip | 4 | 2, 3, 4, 5 |
| flex | 5 | 2, 3, 4, 5 | gzip | 5 | 2, 3, 4, 5 |
| grep | 2 | 2, 3, 4, 5 | sed | 2 | 2, 3, 4, 5 |
| grep | 3 | 2, 3, 4, 5 | sed | 3 | 2, 3, 4, 5 |
| grep | 4 | 2, 3, 4, 5 | sed | 4 | 2, 3, 4, 5 |
| grep | 5 | 2, 3, 4, 5 | sed | 5 | 2, 3, 4, 5 |
flex
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
| 4 | 93.023 | 82.851 | 71.707 | 60.886 | 51.033 |
| 5 | 97.674 | 89.340 | 78.722 | 67.791 | 57.518 |
| 6 | 100.000 | 93.163 | 83.403 | 72.852 | 62.623 |
| 7 | 100.000 | 94.206 | 85.516 | 75.762 | 66.013 |
| 8 | 100.000 | 95.017 | 87.113 | 77.908 | 68.466 |
| 9 | 100.000 | 95.829 | 88.849 | 80.407 | 71.487 |
| 10 | 100.000 | 96.408 | 90.086 | 82.192 | 73.659 |
| 11 | 100.000 | 96.987 | 91.240 | 83.759 | 75.468 |
| 12 | 100.000 | 97.451 | 92.394 | 85.598 | 77.854 |
| 13 | 100.000 | 97.798 | 93.160 | 86.739 | 79.287 |
| 14 | 100.000 | 98.030 | 93.760 | 87.730 | 80.618 |
| 15 | 100.000 | 98.262 | 94.268 | 88.491 | 81.588 |
| 16 | 100.000 | 98.494 | 94.831 | 89.396 | 82.795 |
| 17 | 100.000 | 98.725 | 95.458 | 90.470 | 84.280 |
| 18 | 100.000 | 98.957 | 95.994 | 91.305 | 85.375 |
| 19 | 100.000 | 99.189 | 96.594 | 92.307 | 86.742 |
| 20 | 100.000 | 99.305 | 96.843 | 92.677 | 87.212 |
| 21 | 100.000 | 99.421 | 97.175 | 93.258 | 88.023 |
| 22 | 100.000 | 99.537 | 97.434 | 93.656 | 88.545 |
| 23 | 100.000 | 99.652 | 97.711 | 94.097 | 89.130 |
| 24 | 100.000 | 99.768 | 98.006 | 94.592 | 89.818 |
| 25 | 100.000 | 99.884 | 98.274 | 95.012 | 90.372 |
| 26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |
flex
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 93.023 | 82.271 | 70.747 | 59.812 | 50.024 |
| 5 | 97.674 | 89.108 | 78.418 | 67.553 | 57.408 |
| 6 | 100.000 | 93.048 | 83.310 | 72.881 | 62.799 |
| 7 | 100.000 | 93.975 | 85.240 | 75.594 | 66.014 |
| 8 | 100.000 | 94.670 | 86.753 | 77.807 | 68.727 |
| 9 | 100.000 | 95.481 | 88.433 | 80.170 | 71.538 |
| 10 | 100.000 | 96.176 | 89.901 | 82.268 | 74.067 |
| 11 | 100.000 | 96.871 | 91.341 | 84.300 | 76.498 |
| 12 | 100.000 | 97.219 | 92.218 | 85.721 | 78.377 |
| 13 | 100.000 | 97.567 | 93.021 | 86.959 | 79.971 |
| 14 | 100.000 | 97.914 | 93.806 | 88.142 | 81.462 |
| 15 | 100.000 | 98.262 | 94.581 | 89.308 | 82.934 |
| 16 | 100.000 | 98.610 | 95.320 | 90.379 | 84.256 |
| 17 | 100.000 | 98.841 | 95.855 | 91.219 | 85.365 |
| 18 | 100.000 | 99.073 | 96.381 | 92.021 | 86.389 |
| 19 | 100.000 | 99.189 | 96.705 | 92.589 | 87.195 |
| 20 | 100.000 | 99.305 | 97.009 | 93.109 | 87.925 |
| 21 | 100.000 | 99.421 | 97.268 | 93.507 | 88.448 |
| 22 | 100.000 | 99.537 | 97.526 | 93.899 | 88.950 |
| 23 | 100.000 | 99.652 | 97.775 | 94.271 | 89.426 |
| 24 | 100.000 | 99.768 | 98.025 | 94.640 | 89.896 |
| 25 | 100.000 | 99.884 | 98.274 | 95.010 | 90.366 |
| 26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |
flex
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 93.023 | 81.808 | 69.934 | 58.848 | 49.062 |
| 5 | 97.674 | 88.528 | 77.375 | 66.284 | 56.108 |
| 6 | 97.674 | 90.151 | 80.504 | 70.398 | 60.699 |
| 7 | 97.674 | 91.078 | 82.424 | 73.093 | 63.894 |
| 8 | 97.674 | 91.889 | 84.187 | 75.652 | 66.997 |
| 9 | 97.674 | 92.352 | 85.267 | 77.317 | 69.125 |
| 10 | 100.000 | 96.060 | 89.670 | 81.958 | 73.716 |
| 11 | 100.000 | 96.408 | 90.538 | 83.356 | 75.562 |
| 12 | 100.000 | 97.103 | 91.978 | 85.384 | 77.980 |
| 13 | 100.000 | 97.451 | 92.791 | 86.642 | 79.600 |
| 14 | 100.000 | 97.798 | 93.557 | 87.793 | 81.063 |
| 15 | 100.000 | 98.146 | 94.341 | 88.982 | 82.571 |
| 16 | 100.000 | 98.494 | 95.126 | 90.168 | 84.073 |
| 17 | 100.000 | 98.725 | 95.671 | 91.019 | 85.181 |
| 18 | 100.000 | 98.957 | 96.178 | 91.780 | 86.151 |
| 19 | 100.000 | 99.189 | 96.705 | 92.582 | 87.175 |
| 20 | 100.000 | 99.305 | 97.000 | 93.078 | 87.863 |
| 21 | 100.000 | 99.421 | 97.268 | 93.498 | 88.418 |
| 22 | 100.000 | 99.537 | 97.526 | 93.896 | 88.940 |
| 23 | 100.000 | 99.652 | 97.775 | 94.271 | 89.426 |
| 24 | 100.000 | 99.768 | 98.025 | 94.642 | 89.902 |
| 25 | 100.000 | 99.884 | 98.274 | 95.012 | 90.372 |
| 26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |
flex
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 90.698 | 79.954 | 68.993 | 58.624 | 49.270 |
| 5 | 93.023 | 84.125 | 74.328 | 64.519 | 55.269 |
| 6 | 95.349 | 87.717 | 78.685 | 69.298 | 60.196 |
| 7 | 97.674 | 91.194 | 82.766 | 73.665 | 64.635 |
| 8 | 97.674 | 92.236 | 84.870 | 76.556 | 68.001 |
| 9 | 97.674 | 92.816 | 86.135 | 78.421 | 70.313 |
| 10 | 97.674 | 93.395 | 87.409 | 80.302 | 72.642 |
| 11 | 100.000 | 96.640 | 90.972 | 83.914 | 76.172 |
| 12 | 100.000 | 97.335 | 92.394 | 85.900 | 78.531 |
| 13 | 100.000 | 97.683 | 93.178 | 87.092 | 80.054 |
| 14 | 100.000 | 98.030 | 93.963 | 88.282 | 81.565 |
| 15 | 100.000 | 98.262 | 94.508 | 89.137 | 82.686 |
| 16 | 100.000 | 98.378 | 94.858 | 89.773 | 83.600 |
| 17 | 100.000 | 98.610 | 95.403 | 90.628 | 84.721 |
| 18 | 100.000 | 98.725 | 95.708 | 91.150 | 85.456 |
| 19 | 100.000 | 98.957 | 96.197 | 91.860 | 86.337 |
| 20 | 100.000 | 99.189 | 96.741 | 92.705 | 87.423 |
| 21 | 100.000 | 99.421 | 97.268 | 93.507 | 88.448 |
| 22 | 100.000 | 99.537 | 97.517 | 93.879 | 88.924 |
| 23 | 100.000 | 99.652 | 97.766 | 94.251 | 89.400 |
| 24 | 100.000 | 99.768 | 98.015 | 94.621 | 89.870 |
| 25 | 100.000 | 99.884 | 98.265 | 94.990 | 90.340 |
| 26 | 100.000 | 100.000 | 98.523 | 95.382 | 90.842 |
flex
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 86.047 | 69.757 | 55.183 | 43.241 | 33.810 |
| 3 | 90.698 | 77.520 | 64.405 | 52.735 | 42.856 |
| 4 | 95.349 | 84.820 | 72.851 | 61.352 | 51.071 |
| 5 | 100.000 | 91.309 | 79.886 | 68.304 | 57.631 |
| 6 | 100.000 | 93.048 | 83.283 | 72.814 | 62.698 |
| 7 | 100.000 | 94.322 | 85.821 | 76.250 | 66.635 |
| 8 | 100.000 | 95.249 | 87.723 | 78.901 | 69.762 |
| 9 | 100.000 | 96.060 | 89.357 | 81.147 | 72.387 |
| 10 | 100.000 | 96.756 | 90.843 | 83.296 | 75.004 |
| 11 | 100.000 | 97.103 | 91.609 | 84.434 | 76.427 |
| 12 | 100.000 | 97.451 | 92.458 | 85.785 | 78.194 |
| 13 | 100.000 | 97.798 | 93.252 | 86.997 | 79.738 |
| 14 | 100.000 | 98.146 | 93.981 | 88.041 | 81.010 |
| 15 | 100.000 | 98.378 | 94.609 | 89.104 | 82.459 |
| 16 | 100.000 | 98.610 | 95.255 | 90.212 | 83.980 |
| 17 | 100.000 | 98.841 | 95.818 | 91.119 | 85.194 |
| 18 | 100.000 | 98.957 | 96.114 | 91.621 | 85.902 |
| 19 | 100.000 | 99.073 | 96.372 | 92.019 | 86.424 |
| 20 | 100.000 | 99.189 | 96.649 | 92.459 | 87.009 |
| 21 | 100.000 | 99.305 | 96.917 | 92.877 | 87.558 |
| 22 | 100.000 | 99.421 | 97.203 | 93.347 | 88.198 |
| 23 | 100.000 | 99.537 | 97.471 | 93.772 | 88.770 |
| 24 | 100.000 | 99.652 | 97.738 | 94.191 | 89.325 |
| 25 | 100.000 | 99.768 | 98.052 | 94.746 | 90.125 |
| 26 | 100.000 | 99.884 | 98.301 | 95.117 | 90.601 |
| 27 | 100.000 | 100.000 | 98.569 | 95.537 | 91.156 |
| 28 | 100.000 | 100.000 | 98.643 | 95.745 | 91.529 |
| 29 | 100.000 | 100.000 | 98.726 | 95.972 | 91.924 |
| 30 | 100.000 | 100.000 | 98.735 | 96.003 | 91.985 |
| 31 | 100.000 | 100.000 | 98.800 | 96.180 | 92.294 |
| 32 | 100.000 | 100.000 | 98.837 | 96.282 | 92.474 |
| 33 | 100.000 | 100.000 | 98.911 | 96.494 | 92.860 |
| 34 | 100.000 | 100.000 | 98.994 | 96.726 | 93.273 |
| 35 | 100.000 | 100.000 | 99.086 | 96.980 | 93.715 |
| 36 | 100.000 | 100.000 | 99.169 | 97.223 | 94.164 |
| 37 | 100.000 | 100.000 | 99.178 | 97.250 | 94.215 |
| 38 | 100.000 | 100.000 | 99.188 | 97.277 | 94.266 |
| 39 | 100.000 | 100.000 | 99.197 | 97.306 | 94.320 |
| 40 | 100.000 | 100.000 | 99.289 | 97.560 | 94.767 |
| 41 | 100.000 | 100.000 | 99.400 | 97.875 | 95.334 |
| 42 | 100.000 | 100.000 | 99.409 | 97.905 | 95.395 |
| 43 | 100.000 | 100.000 | 99.492 | 98.136 | 95.799 |
| 44 | 100.000 | 100.000 | 99.502 | 98.164 | 95.854 |
| 45 | 100.000 | 100.000 | 99.511 | 98.191 | 95.905 |
| 46 | 100.000 | 100.000 | 99.585 | 98.403 | 96.292 |
| 47 | 100.000 | 100.000 | 99.631 | 98.538 | 96.538 |
| 48 | 100.000 | 100.000 | 99.640 | 98.565 | 96.589 |
| 49 | 100.000 | 100.000 | 99.732 | 98.817 | 97.027 |
| 50 | 100.000 | 100.000 | 99.788 | 98.979 | 97.326 |
| 51 | 100.000 | 100.000 | 99.852 | 99.159 | 97.647 |
| 52 | 100.000 | 100.000 | 99.862 | 99.187 | 97.698 |
| 53 | 100.000 | 100.000 | 99.898 | 99.289 | 97.877 |
| 54 | 100.000 | 100.000 | 99.954 | 99.447 | 98.166 |
| 55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |
flex
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
| 4 | 95.349 | 84.936 | 73.064 | 61.614 | 51.341 |
| 5 | 97.674 | 88.992 | 78.187 | 67.247 | 57.070 |
| 6 | 100.000 | 92.816 | 82.895 | 72.377 | 62.284 |
| 7 | 100.000 | 94.090 | 85.452 | 75.862 | 66.303 |
| 8 | 100.000 | 95.017 | 87.390 | 78.600 | 69.560 |
| 9 | 100.000 | 95.829 | 89.098 | 81.031 | 72.476 |
| 10 | 100.000 | 96.524 | 90.566 | 83.128 | 75.005 |
| 11 | 100.000 | 97.219 | 91.987 | 85.116 | 77.367 |
| 12 | 100.000 | 97.451 | 92.624 | 86.204 | 78.862 |
| 13 | 100.000 | 97.683 | 93.261 | 87.292 | 80.357 |
| 14 | 100.000 | 98.030 | 94.046 | 88.476 | 81.849 |
| 15 | 100.000 | 98.378 | 94.821 | 89.644 | 83.331 |
| 16 | 100.000 | 98.494 | 95.172 | 90.290 | 84.276 |
| 17 | 100.000 | 98.610 | 95.523 | 90.935 | 85.217 |
| 18 | 100.000 | 98.841 | 96.040 | 91.718 | 86.220 |
| 19 | 100.000 | 98.957 | 96.372 | 92.312 | 87.076 |
| 20 | 100.000 | 98.957 | 96.511 | 92.687 | 87.721 |
| 21 | 100.000 | 99.073 | 96.825 | 93.226 | 88.469 |
| 22 | 100.000 | 99.073 | 96.945 | 93.559 | 89.056 |
| 23 | 100.000 | 99.189 | 97.231 | 94.045 | 89.748 |
| 24 | 100.000 | 99.189 | 97.332 | 94.324 | 90.237 |
| 25 | 100.000 | 99.189 | 97.425 | 94.577 | 90.677 |
| 26 | 100.000 | 99.189 | 97.517 | 94.836 | 91.139 |
| 27 | 100.000 | 99.305 | 97.775 | 95.236 | 91.668 |
| 28 | 100.000 | 99.305 | 97.849 | 95.451 | 92.061 |
| 29 | 100.000 | 99.305 | 97.923 | 95.668 | 92.464 |
| 30 | 100.000 | 99.421 | 98.181 | 96.066 | 92.987 |
| 31 | 100.000 | 99.537 | 98.431 | 96.443 | 93.480 |
| 32 | 100.000 | 99.652 | 98.680 | 96.813 | 93.950 |
| 33 | 100.000 | 99.768 | 98.929 | 97.181 | 94.413 |
| 34 | 100.000 | 99.884 | 99.178 | 97.549 | 94.879 |
| 35 | 100.000 | 100.000 | 99.428 | 97.915 | 95.335 |
| 36 | 100.000 | 100.000 | 99.483 | 98.071 | 95.617 |
| 37 | 100.000 | 100.000 | 99.538 | 98.228 | 95.902 |
| 38 | 100.000 | 100.000 | 99.594 | 98.385 | 96.183 |
| 39 | 100.000 | 100.000 | 99.649 | 98.548 | 96.485 |
| 40 | 100.000 | 100.000 | 99.695 | 98.682 | 96.735 |
| 41 | 100.000 | 100.000 | 99.742 | 98.824 | 97.005 |
| 42 | 100.000 | 100.000 | 99.788 | 98.957 | 97.250 |
| 43 | 100.000 | 100.000 | 99.834 | 99.093 | 97.500 |
| 44 | 100.000 | 100.000 | 99.871 | 99.199 | 97.694 |
| 45 | 100.000 | 100.000 | 99.908 | 99.303 | 97.881 |
| 46 | 100.000 | 100.000 | 99.917 | 99.330 | 97.932 |
| 47 | 100.000 | 100.000 | 99.926 | 99.357 | 97.982 |
| 48 | 100.000 | 100.000 | 99.935 | 99.385 | 98.037 |
| 49 | 100.000 | 100.000 | 99.945 | 99.415 | 98.095 |
| 50 | 100.000 | 100.000 | 99.954 | 99.442 | 98.146 |
| 51 | 100.000 | 100.000 | 99.963 | 99.470 | 98.200 |
| 52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
| 53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.305 |
| 54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
| 55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |
flex
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 88.372 | 74.739 | 61.885 | 50.692 | 41.300 |
| 4 | 93.023 | 82.039 | 70.331 | 59.310 | 49.515 |
| 5 | 97.674 | 88.644 | 77.605 | 66.592 | 56.452 |
| 6 | 97.674 | 90.267 | 80.809 | 70.883 | 61.312 |
| 7 | 100.000 | 93.975 | 85.276 | 75.688 | 66.165 |
| 8 | 100.000 | 94.670 | 86.781 | 77.877 | 68.838 |
| 9 | 100.000 | 95.249 | 88.064 | 79.782 | 71.205 |
| 10 | 100.000 | 95.944 | 89.523 | 81.856 | 73.695 |
| 11 | 100.000 | 96.524 | 90.741 | 83.602 | 75.818 |
| 12 | 100.000 | 97.219 | 92.191 | 85.649 | 78.258 |
| 13 | 100.000 | 97.451 | 92.800 | 86.673 | 79.658 |
| 14 | 100.000 | 97.567 | 93.151 | 87.325 | 80.621 |
| 15 | 100.000 | 98.030 | 94.138 | 88.755 | 82.373 |
| 16 | 100.000 | 98.262 | 94.720 | 89.705 | 83.649 |
| 17 | 100.000 | 98.262 | 94.849 | 90.065 | 84.285 |
| 18 | 100.000 | 98.378 | 95.200 | 90.703 | 85.206 |
| 19 | 100.000 | 98.610 | 95.744 | 91.558 | 86.327 |
| 20 | 100.000 | 98.725 | 96.077 | 92.152 | 87.179 |
| 21 | 100.000 | 98.841 | 96.381 | 92.677 | 87.923 |
| 22 | 100.000 | 98.957 | 96.695 | 93.223 | 88.698 |
| 23 | 100.000 | 99.189 | 97.194 | 93.962 | 89.635 |
| 24 | 100.000 | 99.189 | 97.286 | 94.222 | 90.098 |
| 25 | 100.000 | 99.189 | 97.388 | 94.502 | 90.589 |
| 26 | 100.000 | 99.305 | 97.692 | 95.016 | 91.299 |
| 27 | 100.000 | 99.305 | 97.785 | 95.269 | 91.739 |
| 28 | 100.000 | 99.305 | 97.840 | 95.435 | 92.048 |
| 29 | 100.000 | 99.305 | 97.905 | 95.621 | 92.389 |
| 30 | 100.000 | 99.305 | 97.960 | 95.787 | 92.698 |
| 31 | 100.000 | 99.305 | 98.015 | 95.952 | 93.004 |
| 32 | 100.000 | 99.421 | 98.274 | 96.355 | 93.543 |
| 33 | 100.000 | 99.421 | 98.329 | 96.518 | 93.848 |
| 34 | 100.000 | 99.421 | 98.385 | 96.678 | 94.137 |
| 35 | 100.000 | 99.421 | 98.440 | 96.836 | 94.426 |
| 36 | 100.000 | 99.421 | 98.495 | 96.991 | 94.705 |
| 37 | 100.000 | 99.537 | 98.745 | 97.365 | 95.188 |
| 38 | 100.000 | 99.537 | 98.791 | 97.497 | 95.430 |
| 39 | 100.000 | 99.537 | 98.837 | 97.628 | 95.669 |
| 40 | 100.000 | 99.652 | 99.086 | 97.996 | 96.132 |
| 41 | 100.000 | 99.768 | 99.335 | 98.364 | 96.595 |
| 42 | 100.000 | 99.884 | 99.585 | 98.731 | 97.058 |
| 43 | 100.000 | 100.000 | 99.834 | 99.099 | 97.521 |
| 44 | 100.000 | 100.000 | 99.871 | 99.203 | 97.708 |
| 45 | 100.000 | 100.000 | 99.908 | 99.305 | 97.888 |
| 46 | 100.000 | 100.000 | 99.917 | 99.333 | 97.943 |
| 47 | 100.000 | 100.000 | 99.926 | 99.362 | 97.997 |
| 48 | 100.000 | 100.000 | 99.935 | 99.390 | 98.052 |
| 49 | 100.000 | 100.000 | 99.945 | 99.417 | 98.102 |
| 50 | 100.000 | 100.000 | 99.954 | 99.444 | 98.153 |
| 51 | 100.000 | 100.000 | 99.963 | 99.471 | 98.204 |
| 52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
| 53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.306 |
| 54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
| 55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |
flex
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 79.070 | 62.109 | 48.777 | 38.407 | 30.360 |
| 3 | 86.047 | 71.958 | 59.356 | 48.629 | 39.714 |
| 4 | 90.698 | 79.374 | 68.014 | 57.508 | 48.199 |
| 5 | 95.349 | 86.327 | 75.898 | 65.511 | 55.852 |
| 6 | 95.349 | 87.370 | 78.095 | 68.622 | 59.549 |
| 7 | 95.349 | 88.181 | 79.830 | 71.124 | 62.579 |
| 8 | 95.349 | 88.760 | 81.169 | 73.159 | 65.140 |
| 9 | 100.000 | 95.133 | 87.898 | 79.628 | 71.093 |
| 10 | 100.000 | 95.829 | 89.338 | 81.663 | 73.534 |
| 11 | 100.000 | 96.408 | 90.557 | 83.410 | 75.656 |
| 12 | 100.000 | 96.987 | 91.775 | 85.156 | 77.779 |
| 13 | 100.000 | 97.451 | 92.781 | 86.637 | 79.619 |
| 14 | 100.000 | 97.683 | 93.363 | 87.595 | 80.921 |
| 15 | 100.000 | 97.798 | 93.714 | 88.240 | 81.862 |
| 16 | 100.000 | 98.030 | 94.277 | 89.147 | 83.075 |
| 17 | 100.000 | 98.030 | 94.397 | 89.481 | 83.662 |
| 18 | 100.000 | 98.146 | 94.738 | 90.099 | 84.558 |
| 19 | 100.000 | 98.262 | 95.071 | 90.692 | 85.407 |
| 20 | 100.000 | 98.262 | 95.191 | 91.023 | 85.987 |
| 21 | 100.000 | 98.494 | 95.754 | 91.921 | 87.171 |
| 22 | 100.000 | 98.494 | 95.864 | 92.227 | 87.708 |
| 23 | 100.000 | 98.494 | 95.975 | 92.533 | 88.246 |
| 24 | 100.000 | 98.725 | 96.474 | 93.276 | 89.193 |
| 25 | 100.000 | 98.957 | 96.972 | 94.014 | 90.127 |
| 26 | 100.000 | 98.957 | 97.028 | 94.178 | 90.432 |
| 27 | 100.000 | 98.957 | 97.083 | 94.339 | 90.730 |
| 28 | 100.000 | 99.073 | 97.360 | 94.789 | 91.346 |
| 29 | 100.000 | 99.073 | 97.415 | 94.947 | 91.631 |
| 30 | 100.000 | 99.073 | 97.480 | 95.127 | 91.951 |
| 31 | 100.000 | 99.073 | 97.591 | 95.422 | 92.454 |
| 32 | 100.000 | 99.073 | 97.637 | 95.556 | 92.699 |
| 33 | 100.000 | 99.189 | 97.905 | 95.982 | 93.275 |
| 34 | 100.000 | 99.189 | 97.960 | 96.144 | 93.574 |
| 35 | 100.000 | 99.305 | 98.228 | 96.567 | 94.140 |
| 36 | 100.000 | 99.305 | 98.283 | 96.726 | 94.430 |
| 37 | 100.000 | 99.421 | 98.532 | 97.100 | 94.913 |
| 38 | 100.000 | 99.537 | 98.791 | 97.494 | 95.423 |
| 39 | 100.000 | 99.537 | 98.837 | 97.628 | 95.669 |
| 40 | 100.000 | 99.652 | 99.086 | 97.998 | 96.139 |
| 41 | 100.000 | 99.768 | 99.335 | 98.366 | 96.603 |
| 42 | 100.000 | 99.884 | 99.585 | 98.734 | 97.065 |
| 43 | 100.000 | 99.884 | 99.622 | 98.838 | 97.253 |
| 44 | 100.000 | 100.000 | 99.871 | 99.204 | 97.712 |
| 45 | 100.000 | 100.000 | 99.908 | 99.306 | 97.891 |
| 46 | 100.000 | 100.000 | 99.917 | 99.334 | 97.946 |
| 47 | 100.000 | 100.000 | 99.926 | 99.363 | 98.001 |
| 48 | 100.000 | 100.000 | 99.935 | 99.390 | 98.052 |
| 49 | 100.000 | 100.000 | 99.945 | 99.417 | 98.103 |
| 50 | 100.000 | 100.000 | 99.954 | 99.444 | 98.154 |
| 51 | 100.000 | 100.000 | 99.963 | 99.471 | 98.204 |
| 52 | 100.000 | 100.000 | 99.972 | 99.498 | 98.255 |
| 53 | 100.000 | 100.000 | 99.982 | 99.525 | 98.306 |
| 54 | 100.000 | 100.000 | 99.991 | 99.552 | 98.356 |
| 55 | 100.000 | 100.000 | 100.000 | 99.579 | 98.407 |
flex
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 86.047 | 69.757 | 55.183 | 43.241 | 33.810 |
| 3 | 90.698 | 77.520 | 64.405 | 52.735 | 42.856 |
| 4 | 95.349 | 84.820 | 72.833 | 61.307 | 51.003 |
| 5 | 100.000 | 91.425 | 80.070 | 68.500 | 57.802 |
| 6 | 100.000 | 93.048 | 83.264 | 72.769 | 62.629 |
| 7 | 100.000 | 94.206 | 85.544 | 75.832 | 66.127 |
| 8 | 100.000 | 95.133 | 87.437 | 78.456 | 69.205 |
| 9 | 100.000 | 95.944 | 89.117 | 80.815 | 72.003 |
| 10 | 100.000 | 96.756 | 90.778 | 83.131 | 74.739 |
| 11 | 100.000 | 97.219 | 91.849 | 84.772 | 76.828 |
| 12 | 100.000 | 97.567 | 92.615 | 85.911 | 78.255 |
| 13 | 100.000 | 97.914 | 93.400 | 87.101 | 79.766 |
| 14 | 100.000 | 98.146 | 93.972 | 88.031 | 81.015 |
| 15 | 100.000 | 98.378 | 94.517 | 88.875 | 82.102 |
| 16 | 100.000 | 98.610 | 95.108 | 89.841 | 83.390 |
| 17 | 100.000 | 98.841 | 95.671 | 90.744 | 84.591 |
| 18 | 100.000 | 98.957 | 96.031 | 91.392 | 85.504 |
| 19 | 100.000 | 99.073 | 96.354 | 91.962 | 86.316 |
| 20 | 100.000 | 99.189 | 96.695 | 92.574 | 87.191 |
| 21 | 100.000 | 99.305 | 96.954 | 92.968 | 87.700 |
| 22 | 100.000 | 99.421 | 97.314 | 93.636 | 88.681 |
| 23 | 100.000 | 99.537 | 97.655 | 94.240 | 89.528 |
| 24 | 100.000 | 99.652 | 97.905 | 94.610 | 89.998 |
| 25 | 100.000 | 99.768 | 98.154 | 94.982 | 90.474 |
| 26 | 100.000 | 99.884 | 98.403 | 95.360 | 90.971 |
| 27 | 100.000 | 100.000 | 98.661 | 95.756 | 91.486 |
| 28 | 100.000 | 100.000 | 98.698 | 95.858 | 91.666 |
| 29 | 100.000 | 100.000 | 98.782 | 96.089 | 92.074 |
| 30 | 100.000 | 100.000 | 98.883 | 96.367 | 92.558 |
| 31 | 100.000 | 100.000 | 98.975 | 96.635 | 93.047 |
| 32 | 100.000 | 100.000 | 99.040 | 96.816 | 93.370 |
| 33 | 100.000 | 100.000 | 99.105 | 97.007 | 93.724 |
| 34 | 100.000 | 100.000 | 99.151 | 97.147 | 93.992 |
| 35 | 100.000 | 100.000 | 99.169 | 97.211 | 94.126 |
| 36 | 100.000 | 100.000 | 99.188 | 97.276 | 94.267 |
| 37 | 100.000 | 100.000 | 99.206 | 97.337 | 94.389 |
| 38 | 100.000 | 100.000 | 99.225 | 97.398 | 94.515 |
| 39 | 100.000 | 100.000 | 99.271 | 97.533 | 94.762 |
| 40 | 100.000 | 100.000 | 99.308 | 97.646 | 94.980 |
| 41 | 100.000 | 100.000 | 99.326 | 97.707 | 95.102 |
| 42 | 100.000 | 100.000 | 99.335 | 97.735 | 95.157 |
| 43 | 100.000 | 100.000 | 99.409 | 97.949 | 95.550 |
| 44 | 100.000 | 100.000 | 99.446 | 98.053 | 95.737 |
| 45 | 100.000 | 100.000 | 99.465 | 98.112 | 95.853 |
| 46 | 100.000 | 100.000 | 99.483 | 98.170 | 95.969 |
| 47 | 100.000 | 100.000 | 99.502 | 98.230 | 96.092 |
| 48 | 100.000 | 100.000 | 99.511 | 98.263 | 96.161 |
| 49 | 100.000 | 100.000 | 99.520 | 98.291 | 96.216 |
| 50 | 100.000 | 100.000 | 99.548 | 98.376 | 96.381 |
| 51 | 100.000 | 100.000 | 99.594 | 98.519 | 96.656 |
| 52 | 100.000 | 100.000 | 99.612 | 98.575 | 96.764 |
| 53 | 100.000 | 100.000 | 99.668 | 98.734 | 97.051 |
| 54 | 100.000 | 100.000 | 99.677 | 98.767 | 97.124 |
| 55 | 100.000 | 100.000 | 99.686 | 98.799 | 97.194 |
| 56 | 100.000 | 100.000 | 99.686 | 98.810 | 97.231 |
| 57 | 100.000 | 100.000 | 99.695 | 98.841 | 97.298 |
| 58 | 100.000 | 100.000 | 99.723 | 98.927 | 97.466 |
| 59 | 100.000 | 100.000 | 99.742 | 98.983 | 97.575 |
| 60 | 100.000 | 100.000 | 99.760 | 99.047 | 97.708 |
| 61 | 100.000 | 100.000 | 99.778 | 99.107 | 97.832 |
| 62 | 100.000 | 100.000 | 99.788 | 99.139 | 97.897 |
| 63 | 100.000 | 100.000 | 99.797 | 99.170 | 97.963 |
| 64 | 100.000 | 100.000 | 99.806 | 99.204 | 98.041 |
| 65 | 100.000 | 100.000 | 99.825 | 99.262 | 98.153 |
| 66 | 100.000 | 100.000 | 99.834 | 99.299 | 98.240 |
| 67 | 100.000 | 100.000 | 99.843 | 99.332 | 98.312 |
| 68 | 100.000 | 100.000 | 99.852 | 99.359 | 98.364 |
| 69 | 100.000 | 100.000 | 99.852 | 99.366 | 98.386 |
| 70 | 100.000 | 100.000 | 99.852 | 99.372 | 98.410 |
| 71 | 100.000 | 100.000 | 99.871 | 99.427 | 98.515 |
| 72 | 100.000 | 100.000 | 99.880 | 99.457 | 98.577 |
| 73 | 100.000 | 100.000 | 99.880 | 99.473 | 98.632 |
| 74 | 100.000 | 100.000 | 99.889 | 99.501 | 98.689 |
| 75 | 100.000 | 100.000 | 99.908 | 99.554 | 98.786 |
| 76 | 100.000 | 100.000 | 99.917 | 99.583 | 98.845 |
| 77 | 100.000 | 100.000 | 99.917 | 99.588 | 98.860 |
| 78 | 100.000 | 100.000 | 99.917 | 99.599 | 98.901 |
| 79 | 100.000 | 100.000 | 99.926 | 99.633 | 98.978 |
| 80 | 100.000 | 100.000 | 99.935 | 99.660 | 99.029 |
| 81 | 100.000 | 100.000 | 99.945 | 99.689 | 99.083 |
| 82 | 100.000 | 100.000 | 99.945 | 99.700 | 99.125 |
| 83 | 100.000 | 100.000 | 99.945 | 99.704 | 99.140 |
| 84 | 100.000 | 100.000 | 99.945 | 99.710 | 99.163 |
| 85 | 100.000 | 100.000 | 99.945 | 99.718 | 99.189 |
| 86 | 100.000 | 100.000 | 99.954 | 99.745 | 99.240 |
| 87 | 100.000 | 100.000 | 99.954 | 99.748 | 99.253 |
| 88 | 100.000 | 100.000 | 99.954 | 99.754 | 99.275 |
| 89 | 100.000 | 100.000 | 99.954 | 99.760 | 99.298 |
| 90 | 100.000 | 100.000 | 99.963 | 99.790 | 99.357 |
| 91 | 100.000 | 100.000 | 99.963 | 99.796 | 99.379 |
| 92 | 100.000 | 100.000 | 99.963 | 99.804 | 99.409 |
| 93 | 100.000 | 100.000 | 99.963 | 99.810 | 99.432 |
| 94 | 100.000 | 100.000 | 99.963 | 99.816 | 99.451 |
| 95 | 100.000 | 100.000 | 99.963 | 99.821 | 99.470 |
| 96 | 100.000 | 100.000 | 99.972 | 99.848 | 99.521 |
| 97 | 100.000 | 100.000 | 99.982 | 99.875 | 99.572 |
| 98 | 100.000 | 100.000 | 99.982 | 99.883 | 99.603 |
| 99 | 100.000 | 100.000 | 99.982 | 99.890 | 99.626 |
| 100 | 100.000 | 100.000 | 99.982 | 99.895 | 99.645 |
| 101 | 100.000 | 100.000 | 99.982 | 99.900 | 99.664 |
| 102 | 100.000 | 100.000 | 99.991 | 99.930 | 99.726 |
| 103 | 100.000 | 100.000 | 99.991 | 99.934 | 99.742 |
| 104 | 100.000 | 100.000 | 99.991 | 99.944 | 99.775 |
| 105 | 100.000 | 100.000 | 99.991 | 99.949 | 99.794 |
| 106 | 100.000 | 100.000 | 100.000 | 99.976 | 99.844 |
| 107 | 100.000 | 100.000 | 100.000 | 99.979 | 99.856 |
| 108 | 100.000 | 100.000 | 100.000 | 99.985 | 99.879 |
| 109 | 100.000 | 100.000 | 100.000 | 99.993 | 99.906 |
| 110 | 100.000 | 100.000 | 100.000 | 99.997 | 99.922 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
flex
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
| 4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
| 5 | 97.674 | 89.224 | 78.593 | 67.727 | 57.546 |
| 6 | 97.674 | 90.498 | 81.215 | 71.361 | 61.782 |
| 7 | 100.000 | 94.206 | 85.646 | 76.075 | 66.494 |
| 8 | 100.000 | 95.249 | 87.760 | 78.985 | 69.883 |
| 9 | 100.000 | 95.944 | 89.255 | 81.158 | 72.540 |
| 10 | 100.000 | 96.756 | 90.907 | 83.454 | 75.250 |
| 11 | 100.000 | 97.219 | 91.969 | 85.072 | 77.303 |
| 12 | 100.000 | 97.567 | 92.809 | 86.401 | 79.038 |
| 13 | 100.000 | 97.798 | 93.427 | 87.444 | 80.464 |
| 14 | 100.000 | 98.146 | 94.194 | 88.587 | 81.900 |
| 15 | 100.000 | 98.262 | 94.581 | 89.324 | 82.989 |
| 16 | 100.000 | 98.494 | 95.135 | 90.196 | 84.126 |
| 17 | 100.000 | 98.610 | 95.486 | 90.829 | 85.030 |
| 18 | 100.000 | 98.841 | 96.012 | 91.632 | 86.059 |
| 19 | 100.000 | 98.957 | 96.354 | 92.246 | 86.937 |
| 20 | 100.000 | 99.073 | 96.695 | 92.851 | 87.788 |
| 21 | 100.000 | 99.073 | 96.815 | 93.184 | 88.375 |
| 22 | 100.000 | 99.073 | 96.926 | 93.495 | 88.926 |
| 23 | 100.000 | 99.189 | 97.212 | 93.963 | 89.563 |
| 24 | 100.000 | 99.305 | 97.489 | 94.413 | 90.179 |
| 25 | 100.000 | 99.421 | 97.766 | 94.868 | 90.812 |
| 26 | 100.000 | 99.421 | 97.849 | 95.114 | 91.268 |
| 27 | 100.000 | 99.421 | 97.932 | 95.351 | 91.693 |
| 28 | 100.000 | 99.537 | 98.191 | 95.748 | 92.216 |
| 29 | 100.000 | 99.652 | 98.449 | 96.147 | 92.742 |
| 30 | 100.000 | 99.652 | 98.514 | 96.330 | 93.069 |
| 31 | 100.000 | 99.768 | 98.763 | 96.699 | 93.539 |
| 32 | 100.000 | 99.884 | 99.012 | 97.082 | 94.049 |
| 33 | 100.000 | 100.000 | 99.262 | 97.448 | 94.508 |
| 34 | 100.000 | 100.000 | 99.326 | 97.635 | 94.849 |
| 35 | 100.000 | 100.000 | 99.372 | 97.773 | 95.109 |
| 36 | 100.000 | 100.000 | 99.409 | 97.880 | 95.308 |
| 37 | 100.000 | 100.000 | 99.446 | 97.986 | 95.500 |
| 38 | 100.000 | 100.000 | 99.483 | 98.103 | 95.733 |
| 39 | 100.000 | 100.000 | 99.520 | 98.211 | 95.931 |
| 40 | 100.000 | 100.000 | 99.548 | 98.304 | 96.123 |
| 41 | 100.000 | 100.000 | 99.575 | 98.385 | 96.270 |
| 42 | 100.000 | 100.000 | 99.603 | 98.472 | 96.442 |
| 43 | 100.000 | 100.000 | 99.631 | 98.551 | 96.586 |
| 44 | 100.000 | 100.000 | 99.658 | 98.631 | 96.734 |
| 45 | 100.000 | 100.000 | 99.686 | 98.720 | 96.911 |
| 46 | 100.000 | 100.000 | 99.705 | 98.781 | 97.037 |
| 47 | 100.000 | 100.000 | 99.723 | 98.846 | 97.172 |
| 48 | 100.000 | 100.000 | 99.742 | 98.907 | 97.299 |
| 49 | 100.000 | 100.000 | 99.760 | 98.970 | 97.428 |
| 50 | 100.000 | 100.000 | 99.778 | 99.027 | 97.540 |
| 51 | 100.000 | 100.000 | 99.797 | 99.087 | 97.660 |
| 52 | 100.000 | 100.000 | 99.815 | 99.147 | 97.783 |
| 53 | 100.000 | 100.000 | 99.825 | 99.180 | 97.856 |
| 54 | 100.000 | 100.000 | 99.834 | 99.218 | 97.944 |
| 55 | 100.000 | 100.000 | 99.843 | 99.247 | 98.002 |
| 56 | 100.000 | 100.000 | 99.852 | 99.278 | 98.068 |
| 57 | 100.000 | 100.000 | 99.862 | 99.314 | 98.148 |
| 58 | 100.000 | 100.000 | 99.871 | 99.343 | 98.206 |
| 59 | 100.000 | 100.000 | 99.880 | 99.370 | 98.257 |
| 60 | 100.000 | 100.000 | 99.889 | 99.403 | 98.330 |
| 61 | 100.000 | 100.000 | 99.898 | 99.442 | 98.420 |
| 62 | 100.000 | 100.000 | 99.908 | 99.469 | 98.471 |
| 63 | 100.000 | 100.000 | 99.917 | 99.501 | 98.540 |
| 64 | 100.000 | 100.000 | 99.926 | 99.528 | 98.591 |
| 65 | 100.000 | 100.000 | 99.935 | 99.555 | 98.642 |
| 66 | 100.000 | 100.000 | 99.945 | 99.583 | 98.697 |
| 67 | 100.000 | 100.000 | 99.954 | 99.619 | 98.776 |
| 68 | 100.000 | 100.000 | 99.963 | 99.653 | 98.853 |
| 69 | 100.000 | 100.000 | 99.972 | 99.681 | 98.908 |
| 70 | 100.000 | 100.000 | 99.982 | 99.708 | 98.958 |
| 71 | 100.000 | 100.000 | 99.991 | 99.735 | 99.009 |
| 72 | 100.000 | 100.000 | 100.000 | 99.763 | 99.060 |
| 73 | 100.000 | 100.000 | 100.000 | 99.771 | 99.090 |
| 74 | 100.000 | 100.000 | 100.000 | 99.780 | 99.124 |
| 75 | 100.000 | 100.000 | 100.000 | 99.791 | 99.161 |
| 76 | 100.000 | 100.000 | 100.000 | 99.797 | 99.184 |
| 77 | 100.000 | 100.000 | 100.000 | 99.804 | 99.211 |
| 78 | 100.000 | 100.000 | 100.000 | 99.811 | 99.237 |
| 79 | 100.000 | 100.000 | 100.000 | 99.818 | 99.260 |
| 80 | 100.000 | 100.000 | 100.000 | 99.824 | 99.284 |
| 81 | 100.000 | 100.000 | 100.000 | 99.836 | 99.328 |
| 82 | 100.000 | 100.000 | 100.000 | 99.844 | 99.355 |
| 83 | 100.000 | 100.000 | 100.000 | 99.850 | 99.378 |
| 84 | 100.000 | 100.000 | 100.000 | 99.860 | 99.415 |
| 85 | 100.000 | 100.000 | 100.000 | 99.869 | 99.444 |
| 86 | 100.000 | 100.000 | 100.000 | 99.874 | 99.465 |
| 87 | 100.000 | 100.000 | 100.000 | 99.881 | 99.491 |
| 88 | 100.000 | 100.000 | 100.000 | 99.885 | 99.507 |
| 89 | 100.000 | 100.000 | 100.000 | 99.896 | 99.546 |
| 90 | 100.000 | 100.000 | 100.000 | 99.900 | 99.561 |
| 91 | 100.000 | 100.000 | 100.000 | 99.903 | 99.574 |
| 92 | 100.000 | 100.000 | 100.000 | 99.906 | 99.586 |
| 93 | 100.000 | 100.000 | 100.000 | 99.913 | 99.609 |
| 94 | 100.000 | 100.000 | 100.000 | 99.917 | 99.625 |
| 95 | 100.000 | 100.000 | 100.000 | 99.920 | 99.637 |
| 96 | 100.000 | 100.000 | 100.000 | 99.927 | 99.664 |
| 97 | 100.000 | 100.000 | 100.000 | 99.932 | 99.683 |
| 98 | 100.000 | 100.000 | 100.000 | 99.938 | 99.703 |
| 99 | 100.000 | 100.000 | 100.000 | 99.943 | 99.723 |
| 100 | 100.000 | 100.000 | 100.000 | 99.950 | 99.749 |
| 101 | 100.000 | 100.000 | 100.000 | 99.956 | 99.772 |
| 102 | 100.000 | 100.000 | 100.000 | 99.960 | 99.788 |
| 103 | 100.000 | 100.000 | 100.000 | 99.966 | 99.807 |
| 104 | 100.000 | 100.000 | 100.000 | 99.969 | 99.818 |
| 105 | 100.000 | 100.000 | 100.000 | 99.976 | 99.845 |
| 106 | 100.000 | 100.000 | 100.000 | 99.982 | 99.869 |
| 107 | 100.000 | 100.000 | 100.000 | 99.986 | 99.884 |
| 108 | 100.000 | 100.000 | 100.000 | 99.988 | 99.889 |
| 109 | 100.000 | 100.000 | 100.000 | 99.994 | 99.912 |
| 110 | 100.000 | 100.000 | 100.000 | 99.997 | 99.923 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
flex
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
| 5 | 97.674 | 89.108 | 78.399 | 67.511 | 57.346 |
| 6 | 97.674 | 90.498 | 81.169 | 71.255 | 61.628 |
| 7 | 97.674 | 91.425 | 83.135 | 74.060 | 64.987 |
| 8 | 100.000 | 95.133 | 87.584 | 78.814 | 69.754 |
| 9 | 100.000 | 95.713 | 88.877 | 80.744 | 72.165 |
| 10 | 100.000 | 96.524 | 90.538 | 83.065 | 74.914 |
| 11 | 100.000 | 96.987 | 91.600 | 84.684 | 76.973 |
| 12 | 100.000 | 97.219 | 92.227 | 85.751 | 78.435 |
| 13 | 100.000 | 97.683 | 93.261 | 87.292 | 80.357 |
| 14 | 100.000 | 98.030 | 94.046 | 88.486 | 81.883 |
| 15 | 100.000 | 98.378 | 94.831 | 89.674 | 83.391 |
| 16 | 100.000 | 98.378 | 94.978 | 90.079 | 84.096 |
| 17 | 100.000 | 98.378 | 95.108 | 90.439 | 84.728 |
| 18 | 100.000 | 98.494 | 95.468 | 91.099 | 85.683 |
| 19 | 100.000 | 98.610 | 95.800 | 91.690 | 86.525 |
| 20 | 100.000 | 98.725 | 96.114 | 92.236 | 87.302 |
| 21 | 100.000 | 98.725 | 96.215 | 92.520 | 87.804 |
| 22 | 100.000 | 98.957 | 96.723 | 93.281 | 88.774 |
| 23 | 100.000 | 99.073 | 97.000 | 93.736 | 89.407 |
| 24 | 100.000 | 99.073 | 97.083 | 93.972 | 89.829 |
| 25 | 100.000 | 99.189 | 97.360 | 94.424 | 90.452 |
| 26 | 100.000 | 99.305 | 97.628 | 94.853 | 91.038 |
| 27 | 100.000 | 99.421 | 97.895 | 95.280 | 91.617 |
| 28 | 100.000 | 99.421 | 97.941 | 95.422 | 91.891 |
| 29 | 100.000 | 99.537 | 98.218 | 95.866 | 92.486 |
| 30 | 100.000 | 99.537 | 98.265 | 96.008 | 92.759 |
| 31 | 100.000 | 99.537 | 98.311 | 96.147 | 93.024 |
| 32 | 100.000 | 99.537 | 98.348 | 96.264 | 93.251 |
| 33 | 100.000 | 99.537 | 98.403 | 96.422 | 93.539 |
| 34 | 100.000 | 99.537 | 98.440 | 96.537 | 93.761 |
| 35 | 100.000 | 99.537 | 98.477 | 96.649 | 93.975 |
| 36 | 100.000 | 99.652 | 98.726 | 97.021 | 94.452 |
| 37 | 100.000 | 99.652 | 98.763 | 97.133 | 94.663 |
| 38 | 100.000 | 99.652 | 98.809 | 97.264 | 94.900 |
| 39 | 100.000 | 99.768 | 99.058 | 97.634 | 95.371 |
| 40 | 100.000 | 99.768 | 99.095 | 97.742 | 95.572 |
| 41 | 100.000 | 99.884 | 99.345 | 98.110 | 96.036 |
| 42 | 100.000 | 100.000 | 99.594 | 98.476 | 96.496 |
| 43 | 100.000 | 100.000 | 99.612 | 98.541 | 96.633 |
| 44 | 100.000 | 100.000 | 99.631 | 98.604 | 96.765 |
| 45 | 100.000 | 100.000 | 99.658 | 98.689 | 96.926 |
| 46 | 100.000 | 100.000 | 99.686 | 98.771 | 97.081 |
| 47 | 100.000 | 100.000 | 99.695 | 98.812 | 97.178 |
| 48 | 100.000 | 100.000 | 99.714 | 98.872 | 97.301 |
| 49 | 100.000 | 100.000 | 99.723 | 98.912 | 97.395 |
| 50 | 100.000 | 100.000 | 99.751 | 98.993 | 97.547 |
| 51 | 100.000 | 100.000 | 99.760 | 99.030 | 97.634 |
| 52 | 100.000 | 100.000 | 99.778 | 99.089 | 97.749 |
| 53 | 100.000 | 100.000 | 99.797 | 99.147 | 97.866 |
| 54 | 100.000 | 100.000 | 99.815 | 99.205 | 97.983 |
| 55 | 100.000 | 100.000 | 99.825 | 99.242 | 98.066 |
| 56 | 100.000 | 100.000 | 99.843 | 99.298 | 98.175 |
| 57 | 100.000 | 100.000 | 99.852 | 99.333 | 98.255 |
| 58 | 100.000 | 100.000 | 99.871 | 99.388 | 98.356 |
| 59 | 100.000 | 100.000 | 99.871 | 99.399 | 98.397 |
| 60 | 100.000 | 100.000 | 99.880 | 99.431 | 98.467 |
| 61 | 100.000 | 100.000 | 99.880 | 99.443 | 98.509 |
| 62 | 100.000 | 100.000 | 99.889 | 99.474 | 98.575 |
| 63 | 100.000 | 100.000 | 99.898 | 99.505 | 98.642 |
| 64 | 100.000 | 100.000 | 99.898 | 99.516 | 98.681 |
| 65 | 100.000 | 100.000 | 99.908 | 99.546 | 98.744 |
| 66 | 100.000 | 100.000 | 99.908 | 99.555 | 98.778 |
| 67 | 100.000 | 100.000 | 99.908 | 99.564 | 98.808 |
| 68 | 100.000 | 100.000 | 99.908 | 99.572 | 98.839 |
| 69 | 100.000 | 100.000 | 99.908 | 99.580 | 98.870 |
| 70 | 100.000 | 100.000 | 99.908 | 99.589 | 98.900 |
| 71 | 100.000 | 100.000 | 99.908 | 99.596 | 98.927 |
| 72 | 100.000 | 100.000 | 99.917 | 99.624 | 98.982 |
| 73 | 100.000 | 100.000 | 99.917 | 99.631 | 99.008 |
| 74 | 100.000 | 100.000 | 99.926 | 99.659 | 99.062 |
| 75 | 100.000 | 100.000 | 99.935 | 99.687 | 99.113 |
| 76 | 100.000 | 100.000 | 99.935 | 99.693 | 99.137 |
| 77 | 100.000 | 100.000 | 99.945 | 99.720 | 99.188 |
| 78 | 100.000 | 100.000 | 99.954 | 99.747 | 99.239 |
| 79 | 100.000 | 100.000 | 99.954 | 99.753 | 99.262 |
| 80 | 100.000 | 100.000 | 99.954 | 99.759 | 99.285 |
| 81 | 100.000 | 100.000 | 99.954 | 99.766 | 99.308 |
| 82 | 100.000 | 100.000 | 99.963 | 99.793 | 99.358 |
| 83 | 100.000 | 100.000 | 99.972 | 99.820 | 99.409 |
| 84 | 100.000 | 100.000 | 99.982 | 99.847 | 99.460 |
| 85 | 100.000 | 100.000 | 99.991 | 99.874 | 99.511 |
| 86 | 100.000 | 100.000 | 100.000 | 99.901 | 99.561 |
| 87 | 100.000 | 100.000 | 100.000 | 99.906 | 99.581 |
| 88 | 100.000 | 100.000 | 100.000 | 99.911 | 99.600 |
| 89 | 100.000 | 100.000 | 100.000 | 99.917 | 99.620 |
| 90 | 100.000 | 100.000 | 100.000 | 99.922 | 99.639 |
| 91 | 100.000 | 100.000 | 100.000 | 99.927 | 99.659 |
| 92 | 100.000 | 100.000 | 100.000 | 99.932 | 99.679 |
| 93 | 100.000 | 100.000 | 100.000 | 99.938 | 99.699 |
| 94 | 100.000 | 100.000 | 100.000 | 99.942 | 99.714 |
| 95 | 100.000 | 100.000 | 100.000 | 99.946 | 99.730 |
| 96 | 100.000 | 100.000 | 100.000 | 99.950 | 99.747 |
| 97 | 100.000 | 100.000 | 100.000 | 99.954 | 99.762 |
| 98 | 100.000 | 100.000 | 100.000 | 99.958 | 99.777 |
| 99 | 100.000 | 100.000 | 100.000 | 99.963 | 99.793 |
| 100 | 100.000 | 100.000 | 100.000 | 99.967 | 99.808 |
| 101 | 100.000 | 100.000 | 100.000 | 99.971 | 99.825 |
| 102 | 100.000 | 100.000 | 100.000 | 99.974 | 99.836 |
| 103 | 100.000 | 100.000 | 100.000 | 99.977 | 99.848 |
| 104 | 100.000 | 100.000 | 100.000 | 99.980 | 99.860 |
| 105 | 100.000 | 100.000 | 100.000 | 99.983 | 99.872 |
| 106 | 100.000 | 100.000 | 100.000 | 99.986 | 99.884 |
| 107 | 100.000 | 100.000 | 100.000 | 99.990 | 99.895 |
| 108 | 100.000 | 100.000 | 100.000 | 99.993 | 99.907 |
| 109 | 100.000 | 100.000 | 100.000 | 99.996 | 99.918 |
| 110 | 100.000 | 100.000 | 100.000 | 99.999 | 99.930 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
flex
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 79.070 | 62.109 | 48.777 | 38.407 | 30.360 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 90.698 | 79.374 | 68.014 | 57.508 | 48.199 |
| 5 | 93.023 | 83.314 | 72.925 | 62.880 | 53.659 |
| 6 | 95.349 | 87.486 | 78.252 | 68.754 | 59.628 |
| 7 | 97.674 | 91.194 | 82.756 | 73.646 | 64.612 |
| 8 | 97.674 | 91.889 | 84.252 | 75.820 | 67.271 |
| 9 | 100.000 | 95.597 | 88.655 | 80.467 | 71.882 |
| 10 | 100.000 | 96.524 | 90.529 | 83.049 | 74.900 |
| 11 | 100.000 | 96.987 | 91.554 | 84.578 | 76.819 |
| 12 | 100.000 | 97.219 | 92.209 | 85.710 | 78.380 |
| 13 | 100.000 | 97.683 | 93.215 | 87.193 | 80.227 |
| 14 | 100.000 | 98.030 | 94.018 | 88.434 | 81.828 |
| 15 | 100.000 | 98.262 | 94.581 | 89.341 | 83.042 |
| 16 | 100.000 | 98.262 | 94.729 | 89.742 | 83.734 |
| 17 | 100.000 | 98.262 | 94.877 | 90.140 | 84.416 |
| 18 | 100.000 | 98.262 | 94.969 | 90.404 | 84.895 |
| 19 | 100.000 | 98.262 | 95.052 | 90.645 | 85.335 |
| 20 | 100.000 | 98.262 | 95.154 | 90.926 | 85.830 |
| 21 | 100.000 | 98.494 | 95.717 | 91.823 | 87.010 |
| 22 | 100.000 | 98.725 | 96.252 | 92.653 | 88.089 |
| 23 | 100.000 | 98.841 | 96.520 | 93.084 | 88.683 |
| 24 | 100.000 | 98.957 | 96.815 | 93.584 | 89.384 |
| 25 | 100.000 | 99.073 | 97.083 | 94.015 | 89.978 |
| 26 | 100.000 | 99.073 | 97.175 | 94.272 | 90.429 |
| 27 | 100.000 | 99.073 | 97.231 | 94.436 | 90.737 |
| 28 | 100.000 | 99.073 | 97.295 | 94.622 | 91.074 |
| 29 | 100.000 | 99.073 | 97.341 | 94.763 | 91.346 |
| 30 | 100.000 | 99.189 | 97.600 | 95.163 | 91.877 |
| 31 | 100.000 | 99.189 | 97.665 | 95.350 | 92.216 |
| 32 | 100.000 | 99.305 | 97.923 | 95.750 | 92.747 |
| 33 | 100.000 | 99.305 | 97.969 | 95.889 | 93.011 |
| 34 | 100.000 | 99.305 | 98.006 | 96.003 | 93.230 |
| 35 | 100.000 | 99.305 | 98.043 | 96.112 | 93.436 |
| 36 | 100.000 | 99.305 | 98.080 | 96.222 | 93.645 |
| 37 | 100.000 | 99.305 | 98.089 | 96.265 | 93.749 |
| 38 | 100.000 | 99.305 | 98.117 | 96.354 | 93.925 |
| 39 | 100.000 | 99.421 | 98.366 | 96.726 | 94.405 |
| 40 | 100.000 | 99.537 | 98.615 | 97.098 | 94.883 |
| 41 | 100.000 | 99.652 | 98.865 | 97.469 | 95.357 |
| 42 | 100.000 | 99.652 | 98.883 | 97.533 | 95.490 |
| 43 | 100.000 | 99.652 | 98.902 | 97.596 | 95.622 |
| 44 | 100.000 | 99.652 | 98.911 | 97.636 | 95.716 |
| 45 | 100.000 | 99.652 | 98.920 | 97.677 | 95.815 |
| 46 | 100.000 | 99.652 | 98.948 | 97.760 | 95.971 |
| 47 | 100.000 | 99.652 | 98.966 | 97.820 | 96.093 |
| 48 | 100.000 | 99.652 | 98.994 | 97.900 | 96.242 |
| 49 | 100.000 | 99.652 | 99.012 | 97.958 | 96.355 |
| 50 | 100.000 | 99.652 | 99.012 | 97.968 | 96.393 |
| 51 | 100.000 | 99.652 | 99.012 | 97.976 | 96.425 |
| 52 | 100.000 | 99.652 | 99.022 | 98.011 | 96.501 |
| 53 | 100.000 | 99.652 | 99.040 | 98.067 | 96.611 |
| 54 | 100.000 | 99.768 | 99.289 | 98.434 | 97.070 |
| 55 | 100.000 | 99.884 | 99.538 | 98.799 | 97.527 |
| 56 | 100.000 | 99.884 | 99.548 | 98.836 | 97.609 |
| 57 | 100.000 | 99.884 | 99.557 | 98.871 | 97.689 |
| 58 | 100.000 | 100.000 | 99.806 | 99.237 | 98.145 |
| 59 | 100.000 | 100.000 | 99.825 | 99.292 | 98.251 |
| 60 | 100.000 | 100.000 | 99.834 | 99.324 | 98.320 |
| 61 | 100.000 | 100.000 | 99.843 | 99.355 | 98.387 |
| 62 | 100.000 | 100.000 | 99.852 | 99.385 | 98.450 |
| 63 | 100.000 | 100.000 | 99.862 | 99.416 | 98.513 |
| 64 | 100.000 | 100.000 | 99.862 | 99.425 | 98.547 |
| 65 | 100.000 | 100.000 | 99.862 | 99.432 | 98.574 |
| 66 | 100.000 | 100.000 | 99.862 | 99.442 | 98.607 |
| 67 | 100.000 | 100.000 | 99.862 | 99.448 | 98.631 |
| 68 | 100.000 | 100.000 | 99.880 | 99.502 | 98.733 |
| 69 | 100.000 | 100.000 | 99.889 | 99.534 | 98.802 |
| 70 | 100.000 | 100.000 | 99.889 | 99.542 | 98.829 |
| 71 | 100.000 | 100.000 | 99.889 | 99.549 | 98.855 |
| 72 | 100.000 | 100.000 | 99.889 | 99.554 | 98.875 |
| 73 | 100.000 | 100.000 | 99.889 | 99.560 | 98.898 |
| 74 | 100.000 | 100.000 | 99.889 | 99.565 | 98.915 |
| 75 | 100.000 | 100.000 | 99.889 | 99.571 | 98.939 |
| 76 | 100.000 | 100.000 | 99.908 | 99.625 | 99.040 |
| 77 | 100.000 | 100.000 | 99.908 | 99.631 | 99.063 |
| 78 | 100.000 | 100.000 | 99.908 | 99.639 | 99.089 |
| 79 | 100.000 | 100.000 | 99.908 | 99.644 | 99.108 |
| 80 | 100.000 | 100.000 | 99.908 | 99.649 | 99.128 |
| 81 | 100.000 | 100.000 | 99.908 | 99.654 | 99.147 |
| 82 | 100.000 | 100.000 | 99.908 | 99.660 | 99.170 |
| 83 | 100.000 | 100.000 | 99.908 | 99.666 | 99.189 |
| 84 | 100.000 | 100.000 | 99.908 | 99.671 | 99.209 |
| 85 | 100.000 | 100.000 | 99.908 | 99.675 | 99.225 |
| 86 | 100.000 | 100.000 | 99.908 | 99.679 | 99.240 |
| 87 | 100.000 | 100.000 | 99.908 | 99.683 | 99.256 |
| 88 | 100.000 | 100.000 | 99.908 | 99.688 | 99.272 |
| 89 | 100.000 | 100.000 | 99.908 | 99.692 | 99.287 |
| 90 | 100.000 | 100.000 | 99.908 | 99.696 | 99.302 |
| 91 | 100.000 | 100.000 | 99.908 | 99.700 | 99.318 |
| 92 | 100.000 | 100.000 | 99.908 | 99.703 | 99.330 |
| 93 | 100.000 | 100.000 | 99.908 | 99.706 | 99.342 |
| 94 | 100.000 | 100.000 | 99.917 | 99.733 | 99.393 |
| 95 | 100.000 | 100.000 | 99.917 | 99.736 | 99.404 |
| 96 | 100.000 | 100.000 | 99.926 | 99.764 | 99.455 |
| 97 | 100.000 | 100.000 | 99.935 | 99.791 | 99.506 |
| 98 | 100.000 | 100.000 | 99.945 | 99.818 | 99.557 |
| 99 | 100.000 | 100.000 | 99.954 | 99.845 | 99.608 |
| 100 | 100.000 | 100.000 | 99.954 | 99.848 | 99.619 |
| 101 | 100.000 | 100.000 | 99.963 | 99.875 | 99.670 |
| 102 | 100.000 | 100.000 | 99.972 | 99.902 | 99.721 |
| 103 | 100.000 | 100.000 | 99.972 | 99.905 | 99.732 |
| 104 | 100.000 | 100.000 | 99.972 | 99.908 | 99.744 |
| 105 | 100.000 | 100.000 | 99.982 | 99.935 | 99.795 |
| 106 | 100.000 | 100.000 | 99.991 | 99.963 | 99.845 |
| 107 | 100.000 | 100.000 | 100.000 | 99.990 | 99.896 |
| 108 | 100.000 | 100.000 | 100.000 | 99.993 | 99.907 |
| 109 | 100.000 | 100.000 | 100.000 | 99.996 | 99.919 |
| 110 | 100.000 | 100.000 | 100.000 | 99.999 | 99.930 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
flex
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
| 4 | 90.698 | 80.185 | 69.371 | 59.041 | 49.654 |
| 5 | 95.349 | 86.790 | 76.645 | 66.321 | 56.585 |
| 6 | 100.000 | 93.163 | 83.486 | 73.051 | 62.925 |
| 7 | 100.000 | 94.322 | 85.876 | 76.378 | 66.822 |
| 8 | 100.000 | 95.365 | 87.935 | 79.158 | 70.018 |
| 9 | 100.000 | 96.176 | 89.597 | 81.480 | 72.770 |
| 10 | 100.000 | 96.871 | 90.954 | 83.288 | 74.819 |
| 11 | 100.000 | 97.335 | 91.997 | 84.858 | 76.797 |
| 12 | 100.000 | 97.683 | 92.754 | 85.966 | 78.159 |
| 13 | 100.000 | 97.914 | 93.298 | 86.817 | 79.267 |
| 14 | 100.000 | 98.146 | 93.908 | 87.839 | 80.660 |
| 15 | 100.000 | 98.378 | 94.498 | 88.802 | 81.938 |
| 16 | 100.000 | 98.610 | 95.080 | 89.746 | 83.196 |
| 17 | 100.000 | 98.841 | 95.661 | 90.688 | 84.445 |
| 18 | 100.000 | 98.957 | 95.920 | 91.085 | 84.964 |
| 19 | 100.000 | 99.073 | 96.243 | 91.648 | 85.757 |
| 20 | 100.000 | 99.189 | 96.612 | 92.332 | 86.750 |
| 21 | 100.000 | 99.305 | 96.889 | 92.779 | 87.355 |
| 22 | 100.000 | 99.421 | 97.166 | 93.223 | 87.950 |
| 23 | 100.000 | 99.537 | 97.415 | 93.588 | 88.405 |
| 24 | 100.000 | 99.652 | 97.665 | 93.963 | 88.892 |
| 25 | 100.000 | 99.768 | 97.960 | 94.456 | 89.569 |
| 26 | 100.000 | 99.884 | 98.218 | 94.854 | 90.091 |
| 27 | 100.000 | 100.000 | 98.468 | 95.225 | 90.564 |
| 28 | 100.000 | 100.000 | 98.495 | 95.305 | 90.711 |
| 29 | 100.000 | 100.000 | 98.615 | 95.634 | 91.284 |
| 30 | 100.000 | 100.000 | 98.745 | 95.990 | 91.906 |
| 31 | 100.000 | 100.000 | 98.818 | 96.202 | 92.289 |
| 32 | 100.000 | 100.000 | 98.883 | 96.382 | 92.608 |
| 33 | 100.000 | 100.000 | 98.929 | 96.520 | 92.869 |
| 34 | 100.000 | 100.000 | 98.975 | 96.655 | 93.116 |
| 35 | 100.000 | 100.000 | 99.012 | 96.766 | 93.327 |
| 36 | 100.000 | 100.000 | 99.058 | 96.899 | 93.571 |
| 37 | 100.000 | 100.000 | 99.105 | 97.033 | 93.815 |
| 38 | 100.000 | 100.000 | 99.142 | 97.147 | 94.037 |
| 39 | 100.000 | 100.000 | 99.151 | 97.180 | 94.108 |
| 40 | 100.000 | 100.000 | 99.188 | 97.286 | 94.302 |
| 41 | 100.000 | 100.000 | 99.262 | 97.501 | 94.699 |
| 42 | 100.000 | 100.000 | 99.280 | 97.569 | 94.845 |
| 43 | 100.000 | 100.000 | 99.298 | 97.633 | 94.978 |
| 44 | 100.000 | 100.000 | 99.326 | 97.718 | 95.144 |
| 45 | 100.000 | 100.000 | 99.326 | 97.733 | 95.195 |
| 46 | 100.000 | 100.000 | 99.354 | 97.816 | 95.355 |
| 47 | 100.000 | 100.000 | 99.382 | 97.898 | 95.511 |
| 48 | 100.000 | 100.000 | 99.428 | 98.026 | 95.737 |
| 49 | 100.000 | 100.000 | 99.465 | 98.137 | 95.946 |
| 50 | 100.000 | 100.000 | 99.483 | 98.194 | 96.058 |
| 51 | 100.000 | 100.000 | 99.511 | 98.276 | 96.213 |
| 52 | 100.000 | 100.000 | 99.511 | 98.285 | 96.243 |
| 53 | 100.000 | 100.000 | 99.520 | 98.325 | 96.340 |
| 54 | 100.000 | 100.000 | 99.529 | 98.354 | 96.399 |
| 55 | 100.000 | 100.000 | 99.548 | 98.409 | 96.500 |
| 56 | 100.000 | 100.000 | 99.594 | 98.543 | 96.747 |
| 57 | 100.000 | 100.000 | 99.594 | 98.548 | 96.768 |
| 58 | 100.000 | 100.000 | 99.603 | 98.577 | 96.827 |
| 59 | 100.000 | 100.000 | 99.631 | 98.664 | 96.996 |
| 60 | 100.000 | 100.000 | 99.658 | 98.748 | 97.158 |
| 61 | 100.000 | 100.000 | 99.686 | 98.828 | 97.305 |
| 62 | 100.000 | 100.000 | 99.695 | 98.864 | 97.386 |
| 63 | 100.000 | 100.000 | 99.695 | 98.868 | 97.404 |
| 64 | 100.000 | 100.000 | 99.705 | 98.898 | 97.467 |
| 65 | 100.000 | 100.000 | 99.714 | 98.926 | 97.521 |
| 66 | 100.000 | 100.000 | 99.714 | 98.929 | 97.533 |
| 67 | 100.000 | 100.000 | 99.732 | 98.990 | 97.656 |
| 68 | 100.000 | 100.000 | 99.751 | 99.053 | 97.789 |
| 69 | 100.000 | 100.000 | 99.760 | 99.086 | 97.858 |
| 70 | 100.000 | 100.000 | 99.760 | 99.089 | 97.869 |
| 71 | 100.000 | 100.000 | 99.760 | 99.098 | 97.903 |
| 72 | 100.000 | 100.000 | 99.778 | 99.153 | 98.008 |
| 73 | 100.000 | 100.000 | 99.788 | 99.180 | 98.059 |
| 74 | 100.000 | 100.000 | 99.797 | 99.213 | 98.129 |
| 75 | 100.000 | 100.000 | 99.806 | 99.242 | 98.188 |
| 76 | 100.000 | 100.000 | 99.825 | 99.295 | 98.286 |
| 77 | 100.000 | 100.000 | 99.834 | 99.322 | 98.336 |
| 78 | 100.000 | 100.000 | 99.843 | 99.350 | 98.391 |
| 79 | 100.000 | 100.000 | 99.843 | 99.354 | 98.407 |
| 80 | 100.000 | 100.000 | 99.852 | 99.381 | 98.457 |
| 81 | 100.000 | 100.000 | 99.862 | 99.410 | 98.516 |
| 82 | 100.000 | 100.000 | 99.862 | 99.413 | 98.524 |
| 83 | 100.000 | 100.000 | 99.871 | 99.448 | 98.604 |
| 84 | 100.000 | 100.000 | 99.871 | 99.454 | 98.627 |
| 85 | 100.000 | 100.000 | 99.880 | 99.481 | 98.677 |
| 86 | 100.000 | 100.000 | 99.880 | 99.483 | 98.686 |
| 87 | 100.000 | 100.000 | 99.880 | 99.488 | 98.702 |
| 88 | 100.000 | 100.000 | 99.880 | 99.488 | 98.702 |
| 89 | 100.000 | 100.000 | 99.889 | 99.518 | 98.764 |
| 90 | 100.000 | 100.000 | 99.908 | 99.579 | 98.891 |
| 91 | 100.000 | 100.000 | 99.908 | 99.583 | 98.907 |
| 92 | 100.000 | 100.000 | 99.908 | 99.587 | 98.918 |
| 93 | 100.000 | 100.000 | 99.908 | 99.591 | 98.934 |
| 94 | 100.000 | 100.000 | 99.917 | 99.622 | 99.000 |
| 95 | 100.000 | 100.000 | 99.935 | 99.677 | 99.106 |
| 96 | 100.000 | 100.000 | 99.935 | 99.677 | 99.106 |
| 97 | 100.000 | 100.000 | 99.935 | 99.688 | 99.143 |
| 98 | 100.000 | 100.000 | 99.935 | 99.691 | 99.155 |
| 99 | 100.000 | 100.000 | 99.945 | 99.718 | 99.206 |
| 100 | 100.000 | 100.000 | 99.945 | 99.721 | 99.218 |
| 101 | 100.000 | 100.000 | 99.945 | 99.724 | 99.229 |
| 102 | 100.000 | 100.000 | 99.945 | 99.724 | 99.230 |
| 103 | 100.000 | 100.000 | 99.945 | 99.729 | 99.249 |
| 104 | 100.000 | 100.000 | 99.945 | 99.733 | 99.265 |
| 105 | 100.000 | 100.000 | 99.954 | 99.761 | 99.321 |
| 106 | 100.000 | 100.000 | 99.954 | 99.766 | 99.336 |
| 107 | 100.000 | 100.000 | 99.954 | 99.768 | 99.344 |
| 108 | 100.000 | 100.000 | 99.954 | 99.769 | 99.348 |
| 109 | 100.000 | 100.000 | 99.954 | 99.773 | 99.364 |
| 110 | 100.000 | 100.000 | 99.963 | 99.801 | 99.420 |
| 111 | 100.000 | 100.000 | 99.963 | 99.805 | 99.435 |
| 112 | 100.000 | 100.000 | 99.963 | 99.806 | 99.440 |
| 113 | 100.000 | 100.000 | 99.963 | 99.809 | 99.452 |
| 114 | 100.000 | 100.000 | 99.963 | 99.813 | 99.464 |
| 115 | 100.000 | 100.000 | 99.963 | 99.815 | 99.473 |
| 116 | 100.000 | 100.000 | 99.963 | 99.819 | 99.488 |
| 117 | 100.000 | 100.000 | 99.963 | 99.821 | 99.496 |
| 118 | 100.000 | 100.000 | 99.963 | 99.822 | 99.500 |
| 119 | 100.000 | 100.000 | 99.963 | 99.826 | 99.516 |
| 120 | 100.000 | 100.000 | 99.963 | 99.828 | 99.525 |
| 121 | 100.000 | 100.000 | 99.963 | 99.829 | 99.530 |
| 122 | 100.000 | 100.000 | 99.963 | 99.833 | 99.545 |
| 123 | 100.000 | 100.000 | 99.963 | 99.834 | 99.550 |
| 124 | 100.000 | 100.000 | 99.972 | 99.861 | 99.601 |
| 125 | 100.000 | 100.000 | 99.972 | 99.861 | 99.603 |
| 126 | 100.000 | 100.000 | 99.972 | 99.861 | 99.603 |
| 127 | 100.000 | 100.000 | 99.972 | 99.863 | 99.608 |
| 128 | 100.000 | 100.000 | 99.982 | 99.891 | 99.662 |
| 129 | 100.000 | 100.000 | 99.982 | 99.894 | 99.675 |
| 130 | 100.000 | 100.000 | 99.982 | 99.897 | 99.687 |
| 131 | 100.000 | 100.000 | 99.982 | 99.899 | 99.695 |
| 132 | 100.000 | 100.000 | 99.982 | 99.899 | 99.696 |
| 133 | 100.000 | 100.000 | 99.982 | 99.903 | 99.712 |
| 134 | 100.000 | 100.000 | 99.982 | 99.905 | 99.720 |
| 135 | 100.000 | 100.000 | 99.982 | 99.907 | 99.728 |
| 136 | 100.000 | 100.000 | 99.982 | 99.908 | 99.733 |
| 137 | 100.000 | 100.000 | 99.982 | 99.911 | 99.745 |
| 138 | 100.000 | 100.000 | 99.982 | 99.914 | 99.753 |
| 139 | 100.000 | 100.000 | 99.982 | 99.916 | 99.761 |
| 140 | 100.000 | 100.000 | 99.982 | 99.919 | 99.772 |
| 141 | 100.000 | 100.000 | 99.982 | 99.919 | 99.773 |
| 142 | 100.000 | 100.000 | 99.982 | 99.920 | 99.777 |
| 143 | 100.000 | 100.000 | 99.982 | 99.922 | 99.785 |
| 144 | 100.000 | 100.000 | 99.982 | 99.924 | 99.794 |
| 145 | 100.000 | 100.000 | 99.982 | 99.924 | 99.796 |
| 146 | 100.000 | 100.000 | 99.982 | 99.926 | 99.804 |
| 147 | 100.000 | 100.000 | 99.991 | 99.953 | 99.854 |
| 148 | 100.000 | 100.000 | 99.991 | 99.953 | 99.855 |
| 149 | 100.000 | 100.000 | 99.991 | 99.955 | 99.863 |
| 150 | 100.000 | 100.000 | 99.991 | 99.956 | 99.867 |
| 151 | 100.000 | 100.000 | 99.991 | 99.956 | 99.867 |
| 152 | 100.000 | 100.000 | 99.991 | 99.958 | 99.875 |
| 153 | 100.000 | 100.000 | 99.991 | 99.959 | 99.880 |
| 154 | 100.000 | 100.000 | 99.991 | 99.959 | 99.881 |
| 155 | 100.000 | 100.000 | 99.991 | 99.961 | 99.889 |
| 156 | 100.000 | 100.000 | 99.991 | 99.964 | 99.896 |
| 157 | 100.000 | 100.000 | 99.991 | 99.964 | 99.898 |
| 158 | 100.000 | 100.000 | 99.991 | 99.965 | 99.902 |
| 159 | 100.000 | 100.000 | 99.991 | 99.965 | 99.903 |
| 160 | 100.000 | 100.000 | 99.991 | 99.965 | 99.904 |
| 161 | 100.000 | 100.000 | 99.991 | 99.965 | 99.905 |
| 162 | 100.000 | 100.000 | 99.991 | 99.967 | 99.913 |
| 163 | 100.000 | 100.000 | 99.991 | 99.968 | 99.918 |
| 164 | 100.000 | 100.000 | 99.991 | 99.969 | 99.922 |
| 165 | 100.000 | 100.000 | 99.991 | 99.969 | 99.923 |
| 166 | 100.000 | 100.000 | 99.991 | 99.969 | 99.924 |
| 167 | 100.000 | 100.000 | 100.000 | 99.996 | 99.975 |
| 168 | 100.000 | 100.000 | 100.000 | 99.997 | 99.979 |
| 169 | 100.000 | 100.000 | 100.000 | 99.998 | 99.984 |
| 170 | 100.000 | 100.000 | 100.000 | 99.998 | 99.984 |
| 171 | 100.000 | 100.000 | 100.000 | 99.998 | 99.985 |
| 172 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
| 173 | 100.000 | 100.000 | 100.000 | 99.998 | 99.987 |
| 174 | 100.000 | 100.000 | 100.000 | 99.998 | 99.988 |
| 175 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
| 176 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
| 177 | 100.000 | 100.000 | 100.000 | 99.999 | 99.994 |
| 178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
flex
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 83.721 | 67.323 | 53.245 | 41.856 | 32.878 |
| 3 | 90.698 | 77.752 | 64.802 | 53.194 | 43.303 |
| 4 | 93.023 | 82.619 | 71.291 | 60.381 | 50.518 |
| 5 | 95.349 | 86.559 | 76.230 | 65.817 | 56.069 |
| 6 | 97.674 | 90.382 | 80.947 | 70.967 | 61.313 |
| 7 | 97.674 | 91.425 | 83.153 | 74.099 | 65.039 |
| 8 | 100.000 | 95.249 | 87.778 | 79.027 | 69.942 |
| 9 | 100.000 | 96.176 | 89.661 | 81.632 | 72.997 |
| 10 | 100.000 | 96.871 | 91.138 | 83.755 | 75.571 |
| 11 | 100.000 | 97.219 | 92.015 | 85.177 | 77.453 |
| 12 | 100.000 | 97.567 | 92.837 | 86.465 | 79.132 |
| 13 | 100.000 | 97.914 | 93.658 | 87.745 | 80.785 |
| 14 | 100.000 | 98.146 | 94.258 | 88.733 | 82.106 |
| 15 | 100.000 | 98.378 | 94.849 | 89.712 | 83.437 |
| 16 | 100.000 | 98.378 | 95.034 | 90.202 | 84.266 |
| 17 | 100.000 | 98.610 | 95.588 | 91.081 | 85.426 |
| 18 | 100.000 | 98.725 | 95.938 | 91.726 | 86.367 |
| 19 | 100.000 | 98.841 | 96.280 | 92.330 | 87.214 |
| 20 | 100.000 | 98.957 | 96.612 | 92.923 | 88.063 |
| 21 | 100.000 | 99.189 | 97.111 | 93.656 | 88.981 |
| 22 | 100.000 | 99.305 | 97.406 | 94.158 | 89.689 |
| 23 | 100.000 | 99.305 | 97.498 | 94.418 | 90.154 |
| 24 | 100.000 | 99.305 | 97.581 | 94.650 | 90.562 |
| 25 | 100.000 | 99.421 | 97.840 | 95.051 | 91.095 |
| 26 | 100.000 | 99.537 | 98.098 | 95.446 | 91.611 |
| 27 | 100.000 | 99.537 | 98.172 | 95.659 | 91.996 |
| 28 | 100.000 | 99.652 | 98.421 | 96.026 | 92.456 |
| 29 | 100.000 | 99.768 | 98.671 | 96.391 | 92.911 |
| 30 | 100.000 | 99.884 | 98.920 | 96.768 | 93.404 |
| 31 | 100.000 | 99.884 | 98.985 | 96.955 | 93.743 |
| 32 | 100.000 | 100.000 | 99.234 | 97.322 | 94.206 |
| 33 | 100.000 | 100.000 | 99.289 | 97.480 | 94.488 |
| 34 | 100.000 | 100.000 | 99.335 | 97.616 | 94.742 |
| 35 | 100.000 | 100.000 | 99.382 | 97.758 | 95.013 |
| 36 | 100.000 | 100.000 | 99.428 | 97.900 | 95.288 |
| 37 | 100.000 | 100.000 | 99.465 | 98.020 | 95.527 |
| 38 | 100.000 | 100.000 | 99.502 | 98.136 | 95.753 |
| 39 | 100.000 | 100.000 | 99.538 | 98.240 | 95.940 |
| 40 | 100.000 | 100.000 | 99.575 | 98.355 | 96.167 |
| 41 | 100.000 | 100.000 | 99.612 | 98.460 | 96.354 |
| 42 | 100.000 | 100.000 | 99.649 | 98.570 | 96.563 |
| 43 | 100.000 | 100.000 | 99.686 | 98.684 | 96.782 |
| 44 | 100.000 | 100.000 | 99.723 | 98.791 | 96.980 |
| 45 | 100.000 | 100.000 | 99.751 | 98.878 | 97.151 |
| 46 | 100.000 | 100.000 | 99.778 | 98.961 | 97.307 |
| 47 | 100.000 | 100.000 | 99.806 | 99.051 | 97.489 |
| 48 | 100.000 | 100.000 | 99.834 | 99.139 | 97.663 |
| 49 | 100.000 | 100.000 | 99.852 | 99.196 | 97.775 |
| 50 | 100.000 | 100.000 | 99.871 | 99.254 | 97.892 |
| 51 | 100.000 | 100.000 | 99.880 | 99.281 | 97.943 |
| 52 | 100.000 | 100.000 | 99.889 | 99.316 | 98.019 |
| 53 | 100.000 | 100.000 | 99.898 | 99.351 | 98.100 |
| 54 | 100.000 | 100.000 | 99.908 | 99.382 | 98.167 |
| 55 | 100.000 | 100.000 | 99.917 | 99.409 | 98.218 |
| 56 | 100.000 | 100.000 | 99.926 | 99.438 | 98.273 |
| 57 | 100.000 | 100.000 | 99.935 | 99.475 | 98.360 |
| 58 | 100.000 | 100.000 | 99.945 | 99.505 | 98.421 |
| 59 | 100.000 | 100.000 | 99.954 | 99.532 | 98.473 |
| 60 | 100.000 | 100.000 | 99.963 | 99.560 | 98.527 |
| 61 | 100.000 | 100.000 | 99.972 | 99.588 | 98.578 |
| 62 | 100.000 | 100.000 | 99.982 | 99.616 | 98.632 |
| 63 | 100.000 | 100.000 | 99.991 | 99.643 | 98.683 |
| 64 | 100.000 | 100.000 | 100.000 | 99.671 | 98.737 |
| 65 | 100.000 | 100.000 | 100.000 | 99.675 | 98.752 |
| 66 | 100.000 | 100.000 | 100.000 | 99.685 | 98.789 |
| 67 | 100.000 | 100.000 | 100.000 | 99.694 | 98.819 |
| 68 | 100.000 | 100.000 | 100.000 | 99.699 | 98.839 |
| 69 | 100.000 | 100.000 | 100.000 | 99.708 | 98.873 |
| 70 | 100.000 | 100.000 | 100.000 | 99.717 | 98.904 |
| 71 | 100.000 | 100.000 | 100.000 | 99.726 | 98.938 |
| 72 | 100.000 | 100.000 | 100.000 | 99.736 | 98.975 |
| 73 | 100.000 | 100.000 | 100.000 | 99.743 | 98.997 |
| 74 | 100.000 | 100.000 | 100.000 | 99.748 | 99.017 |
| 75 | 100.000 | 100.000 | 100.000 | 99.753 | 99.036 |
| 76 | 100.000 | 100.000 | 100.000 | 99.759 | 99.059 |
| 77 | 100.000 | 100.000 | 100.000 | 99.761 | 99.067 |
| 78 | 100.000 | 100.000 | 100.000 | 99.767 | 99.087 |
| 79 | 100.000 | 100.000 | 100.000 | 99.772 | 99.106 |
| 80 | 100.000 | 100.000 | 100.000 | 99.779 | 99.132 |
| 81 | 100.000 | 100.000 | 100.000 | 99.788 | 99.162 |
| 82 | 100.000 | 100.000 | 100.000 | 99.793 | 99.182 |
| 83 | 100.000 | 100.000 | 100.000 | 99.799 | 99.205 |
| 84 | 100.000 | 100.000 | 100.000 | 99.803 | 99.222 |
| 85 | 100.000 | 100.000 | 100.000 | 99.806 | 99.233 |
| 86 | 100.000 | 100.000 | 100.000 | 99.816 | 99.267 |
| 87 | 100.000 | 100.000 | 100.000 | 99.823 | 99.293 |
| 88 | 100.000 | 100.000 | 100.000 | 99.828 | 99.313 |
| 89 | 100.000 | 100.000 | 100.000 | 99.832 | 99.329 |
| 90 | 100.000 | 100.000 | 100.000 | 99.838 | 99.348 |
| 91 | 100.000 | 100.000 | 100.000 | 99.841 | 99.360 |
| 92 | 100.000 | 100.000 | 100.000 | 99.846 | 99.380 |
| 93 | 100.000 | 100.000 | 100.000 | 99.851 | 99.399 |
| 94 | 100.000 | 100.000 | 100.000 | 99.855 | 99.415 |
| 95 | 100.000 | 100.000 | 100.000 | 99.856 | 99.419 |
| 96 | 100.000 | 100.000 | 100.000 | 99.860 | 99.435 |
| 97 | 100.000 | 100.000 | 100.000 | 99.865 | 99.450 |
| 98 | 100.000 | 100.000 | 100.000 | 99.869 | 99.466 |
| 99 | 100.000 | 100.000 | 100.000 | 99.871 | 99.474 |
| 100 | 100.000 | 100.000 | 100.000 | 99.877 | 99.497 |
| 101 | 100.000 | 100.000 | 100.000 | 99.880 | 99.508 |
| 102 | 100.000 | 100.000 | 100.000 | 99.885 | 99.528 |
| 103 | 100.000 | 100.000 | 100.000 | 99.890 | 99.544 |
| 104 | 100.000 | 100.000 | 100.000 | 99.894 | 99.559 |
| 105 | 100.000 | 100.000 | 100.000 | 99.897 | 99.571 |
| 106 | 100.000 | 100.000 | 100.000 | 99.900 | 99.583 |
| 107 | 100.000 | 100.000 | 100.000 | 99.901 | 99.587 |
| 108 | 100.000 | 100.000 | 100.000 | 99.902 | 99.591 |
| 109 | 100.000 | 100.000 | 100.000 | 99.905 | 99.604 |
| 110 | 100.000 | 100.000 | 100.000 | 99.908 | 99.616 |
| 111 | 100.000 | 100.000 | 100.000 | 99.913 | 99.632 |
| 112 | 100.000 | 100.000 | 100.000 | 99.915 | 99.641 |
| 113 | 100.000 | 100.000 | 100.000 | 99.916 | 99.646 |
| 114 | 100.000 | 100.000 | 100.000 | 99.916 | 99.648 |
| 115 | 100.000 | 100.000 | 100.000 | 99.919 | 99.660 |
| 116 | 100.000 | 100.000 | 100.000 | 99.922 | 99.671 |
| 117 | 100.000 | 100.000 | 100.000 | 99.924 | 99.679 |
| 118 | 100.000 | 100.000 | 100.000 | 99.927 | 99.691 |
| 119 | 100.000 | 100.000 | 100.000 | 99.928 | 99.696 |
| 120 | 100.000 | 100.000 | 100.000 | 99.934 | 99.719 |
| 121 | 100.000 | 100.000 | 100.000 | 99.938 | 99.731 |
| 122 | 100.000 | 100.000 | 100.000 | 99.940 | 99.739 |
| 123 | 100.000 | 100.000 | 100.000 | 99.941 | 99.744 |
| 124 | 100.000 | 100.000 | 100.000 | 99.944 | 99.756 |
| 125 | 100.000 | 100.000 | 100.000 | 99.944 | 99.757 |
| 126 | 100.000 | 100.000 | 100.000 | 99.946 | 99.765 |
| 127 | 100.000 | 100.000 | 100.000 | 99.947 | 99.769 |
| 128 | 100.000 | 100.000 | 100.000 | 99.947 | 99.770 |
| 129 | 100.000 | 100.000 | 100.000 | 99.948 | 99.775 |
| 130 | 100.000 | 100.000 | 100.000 | 99.948 | 99.775 |
| 131 | 100.000 | 100.000 | 100.000 | 99.950 | 99.783 |
| 132 | 100.000 | 100.000 | 100.000 | 99.951 | 99.788 |
| 133 | 100.000 | 100.000 | 100.000 | 99.955 | 99.803 |
| 134 | 100.000 | 100.000 | 100.000 | 99.959 | 99.819 |
| 135 | 100.000 | 100.000 | 100.000 | 99.960 | 99.824 |
| 136 | 100.000 | 100.000 | 100.000 | 99.963 | 99.832 |
| 137 | 100.000 | 100.000 | 100.000 | 99.963 | 99.833 |
| 138 | 100.000 | 100.000 | 100.000 | 99.964 | 99.837 |
| 139 | 100.000 | 100.000 | 100.000 | 99.966 | 99.846 |
| 140 | 100.000 | 100.000 | 100.000 | 99.967 | 99.851 |
| 141 | 100.000 | 100.000 | 100.000 | 99.969 | 99.859 |
| 142 | 100.000 | 100.000 | 100.000 | 99.970 | 99.864 |
| 143 | 100.000 | 100.000 | 100.000 | 99.970 | 99.864 |
| 144 | 100.000 | 100.000 | 100.000 | 99.971 | 99.869 |
| 145 | 100.000 | 100.000 | 100.000 | 99.973 | 99.877 |
| 146 | 100.000 | 100.000 | 100.000 | 99.974 | 99.881 |
| 147 | 100.000 | 100.000 | 100.000 | 99.977 | 99.893 |
| 148 | 100.000 | 100.000 | 100.000 | 99.980 | 99.904 |
| 149 | 100.000 | 100.000 | 100.000 | 99.982 | 99.912 |
| 150 | 100.000 | 100.000 | 100.000 | 99.982 | 99.913 |
| 151 | 100.000 | 100.000 | 100.000 | 99.983 | 99.918 |
| 152 | 100.000 | 100.000 | 100.000 | 99.985 | 99.926 |
| 153 | 100.000 | 100.000 | 100.000 | 99.988 | 99.934 |
| 154 | 100.000 | 100.000 | 100.000 | 99.991 | 99.945 |
| 155 | 100.000 | 100.000 | 100.000 | 99.991 | 99.946 |
| 156 | 100.000 | 100.000 | 100.000 | 99.992 | 99.952 |
| 157 | 100.000 | 100.000 | 100.000 | 99.994 | 99.959 |
| 158 | 100.000 | 100.000 | 100.000 | 99.996 | 99.967 |
| 159 | 100.000 | 100.000 | 100.000 | 99.997 | 99.972 |
| 160 | 100.000 | 100.000 | 100.000 | 99.997 | 99.972 |
| 161 | 100.000 | 100.000 | 100.000 | 99.997 | 99.973 |
| 162 | 100.000 | 100.000 | 100.000 | 99.997 | 99.974 |
| 163 | 100.000 | 100.000 | 100.000 | 99.997 | 99.975 |
| 164 | 100.000 | 100.000 | 100.000 | 99.997 | 99.977 |
| 165 | 100.000 | 100.000 | 100.000 | 99.997 | 99.978 |
| 166 | 100.000 | 100.000 | 100.000 | 99.997 | 99.979 |
| 167 | 100.000 | 100.000 | 100.000 | 99.997 | 99.980 |
| 168 | 100.000 | 100.000 | 100.000 | 99.998 | 99.985 |
| 169 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
| 170 | 100.000 | 100.000 | 100.000 | 99.999 | 99.990 |
| 171 | 100.000 | 100.000 | 100.000 | 99.999 | 99.991 |
| 172 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
| 173 | 100.000 | 100.000 | 100.000 | 99.999 | 99.992 |
| 174 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
| 175 | 100.000 | 100.000 | 100.000 | 99.999 | 99.993 |
| 176 | 100.000 | 100.000 | 100.000 | 99.999 | 99.994 |
| 177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
flex
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 90.698 | 79.722 | 68.596 | 58.165 | 48.823 |
| 5 | 95.349 | 86.327 | 75.870 | 65.445 | 55.754 |
| 6 | 97.674 | 90.267 | 80.809 | 70.881 | 61.305 |
| 7 | 97.674 | 90.846 | 82.110 | 72.833 | 63.749 |
| 8 | 100.000 | 94.322 | 86.135 | 77.062 | 67.971 |
| 9 | 100.000 | 95.017 | 87.630 | 79.227 | 70.602 |
| 10 | 100.000 | 96.060 | 89.689 | 82.005 | 73.792 |
| 11 | 100.000 | 96.640 | 90.907 | 83.753 | 75.921 |
| 12 | 100.000 | 97.103 | 91.951 | 85.325 | 77.902 |
| 13 | 100.000 | 97.567 | 92.966 | 86.827 | 79.771 |
| 14 | 100.000 | 97.914 | 93.788 | 88.109 | 81.431 |
| 15 | 100.000 | 98.030 | 94.175 | 88.844 | 82.513 |
| 16 | 100.000 | 98.146 | 94.517 | 89.467 | 83.422 |
| 17 | 100.000 | 98.262 | 94.849 | 90.067 | 84.294 |
| 18 | 100.000 | 98.494 | 95.384 | 90.898 | 85.376 |
| 19 | 100.000 | 98.725 | 95.920 | 91.727 | 86.451 |
| 20 | 100.000 | 98.725 | 96.012 | 91.993 | 86.934 |
| 21 | 100.000 | 98.957 | 96.529 | 92.778 | 87.944 |
| 22 | 100.000 | 99.073 | 96.852 | 93.343 | 88.740 |
| 23 | 100.000 | 99.189 | 97.157 | 93.864 | 89.474 |
| 24 | 100.000 | 99.189 | 97.240 | 94.099 | 89.892 |
| 25 | 100.000 | 99.305 | 97.526 | 94.572 | 90.544 |
| 26 | 100.000 | 99.305 | 97.609 | 94.805 | 90.960 |
| 27 | 100.000 | 99.305 | 97.692 | 95.036 | 91.368 |
| 28 | 100.000 | 99.305 | 97.757 | 95.226 | 91.718 |
| 29 | 100.000 | 99.305 | 97.821 | 95.413 | 92.062 |
| 30 | 100.000 | 99.305 | 97.886 | 95.599 | 92.399 |
| 31 | 100.000 | 99.305 | 97.932 | 95.737 | 92.661 |
| 32 | 100.000 | 99.305 | 97.988 | 95.894 | 92.945 |
| 33 | 100.000 | 99.305 | 98.034 | 96.031 | 93.199 |
| 34 | 100.000 | 99.421 | 98.292 | 96.427 | 93.715 |
| 35 | 100.000 | 99.421 | 98.320 | 96.519 | 93.904 |
| 36 | 100.000 | 99.421 | 98.348 | 96.610 | 94.087 |
| 37 | 100.000 | 99.537 | 98.597 | 96.982 | 94.564 |
| 38 | 100.000 | 99.537 | 98.615 | 97.050 | 94.713 |
| 39 | 100.000 | 99.537 | 98.652 | 97.161 | 94.923 |
| 40 | 100.000 | 99.652 | 98.902 | 97.531 | 95.394 |
| 41 | 100.000 | 99.652 | 98.938 | 97.639 | 95.596 |
| 42 | 100.000 | 99.768 | 99.188 | 98.008 | 96.062 |
| 43 | 100.000 | 99.884 | 99.437 | 98.376 | 96.529 |
| 44 | 100.000 | 99.884 | 99.474 | 98.482 | 96.724 |
| 45 | 100.000 | 99.884 | 99.502 | 98.568 | 96.891 |
| 46 | 100.000 | 100.000 | 99.751 | 98.933 | 97.347 |
| 47 | 100.000 | 100.000 | 99.769 | 98.997 | 97.480 |
| 48 | 100.000 | 100.000 | 99.806 | 99.101 | 97.667 |
| 49 | 100.000 | 100.000 | 99.825 | 99.163 | 97.794 |
| 50 | 100.000 | 100.000 | 99.843 | 99.224 | 97.920 |
| 51 | 100.000 | 100.000 | 99.852 | 99.262 | 98.007 |
| 52 | 100.000 | 100.000 | 99.871 | 99.319 | 98.120 |
| 53 | 100.000 | 100.000 | 99.880 | 99.354 | 98.200 |
| 54 | 100.000 | 100.000 | 99.898 | 99.408 | 98.302 |
| 55 | 100.000 | 100.000 | 99.898 | 99.421 | 98.348 |
| 56 | 100.000 | 100.000 | 99.898 | 99.432 | 98.388 |
| 57 | 100.000 | 100.000 | 99.908 | 99.465 | 98.457 |
| 58 | 100.000 | 100.000 | 99.908 | 99.476 | 98.497 |
| 59 | 100.000 | 100.000 | 99.908 | 99.487 | 98.533 |
| 60 | 100.000 | 100.000 | 99.908 | 99.496 | 98.567 |
| 61 | 100.000 | 100.000 | 99.908 | 99.504 | 98.597 |
| 62 | 100.000 | 100.000 | 99.908 | 99.513 | 98.627 |
| 63 | 100.000 | 100.000 | 99.908 | 99.520 | 98.654 |
| 64 | 100.000 | 100.000 | 99.908 | 99.527 | 98.680 |
| 65 | 100.000 | 100.000 | 99.908 | 99.534 | 98.707 |
| 66 | 100.000 | 100.000 | 99.908 | 99.541 | 98.730 |
| 67 | 100.000 | 100.000 | 99.917 | 99.568 | 98.780 |
| 68 | 100.000 | 100.000 | 99.917 | 99.574 | 98.803 |
| 69 | 100.000 | 100.000 | 99.926 | 99.601 | 98.854 |
| 70 | 100.000 | 100.000 | 99.926 | 99.607 | 98.877 |
| 71 | 100.000 | 100.000 | 99.935 | 99.634 | 98.928 |
| 72 | 100.000 | 100.000 | 99.945 | 99.662 | 98.979 |
| 73 | 100.000 | 100.000 | 99.945 | 99.668 | 99.002 |
| 74 | 100.000 | 100.000 | 99.954 | 99.695 | 99.053 |
| 75 | 100.000 | 100.000 | 99.954 | 99.701 | 99.076 |
| 76 | 100.000 | 100.000 | 99.963 | 99.728 | 99.127 |
| 77 | 100.000 | 100.000 | 99.972 | 99.755 | 99.177 |
| 78 | 100.000 | 100.000 | 99.982 | 99.782 | 99.229 |
| 79 | 100.000 | 100.000 | 99.991 | 99.809 | 99.280 |
| 80 | 100.000 | 100.000 | 99.991 | 99.816 | 99.303 |
| 81 | 100.000 | 100.000 | 100.000 | 99.843 | 99.354 |
| 82 | 100.000 | 100.000 | 100.000 | 99.848 | 99.374 |
| 83 | 100.000 | 100.000 | 100.000 | 99.853 | 99.393 |
| 84 | 100.000 | 100.000 | 100.000 | 99.858 | 99.413 |
| 85 | 100.000 | 100.000 | 100.000 | 99.864 | 99.432 |
| 86 | 100.000 | 100.000 | 100.000 | 99.869 | 99.451 |
| 87 | 100.000 | 100.000 | 100.000 | 99.873 | 99.467 |
| 88 | 100.000 | 100.000 | 100.000 | 99.877 | 99.482 |
| 89 | 100.000 | 100.000 | 100.000 | 99.881 | 99.498 |
| 90 | 100.000 | 100.000 | 100.000 | 99.885 | 99.514 |
| 91 | 100.000 | 100.000 | 100.000 | 99.890 | 99.530 |
| 92 | 100.000 | 100.000 | 100.000 | 99.894 | 99.547 |
| 93 | 100.000 | 100.000 | 100.000 | 99.898 | 99.562 |
| 94 | 100.000 | 100.000 | 100.000 | 99.902 | 99.578 |
| 95 | 100.000 | 100.000 | 100.000 | 99.906 | 99.593 |
| 96 | 100.000 | 100.000 | 100.000 | 99.910 | 99.609 |
| 97 | 100.000 | 100.000 | 100.000 | 99.915 | 99.624 |
| 98 | 100.000 | 100.000 | 100.000 | 99.918 | 99.636 |
| 99 | 100.000 | 100.000 | 100.000 | 99.921 | 99.648 |
| 100 | 100.000 | 100.000 | 100.000 | 99.924 | 99.660 |
| 101 | 100.000 | 100.000 | 100.000 | 99.927 | 99.671 |
| 102 | 100.000 | 100.000 | 100.000 | 99.930 | 99.684 |
| 103 | 100.000 | 100.000 | 100.000 | 99.933 | 99.696 |
| 104 | 100.000 | 100.000 | 100.000 | 99.936 | 99.708 |
| 105 | 100.000 | 100.000 | 100.000 | 99.940 | 99.721 |
| 106 | 100.000 | 100.000 | 100.000 | 99.943 | 99.733 |
| 107 | 100.000 | 100.000 | 100.000 | 99.945 | 99.742 |
| 108 | 100.000 | 100.000 | 100.000 | 99.947 | 99.750 |
| 109 | 100.000 | 100.000 | 100.000 | 99.949 | 99.758 |
| 110 | 100.000 | 100.000 | 100.000 | 99.951 | 99.766 |
| 111 | 100.000 | 100.000 | 100.000 | 99.953 | 99.774 |
| 112 | 100.000 | 100.000 | 100.000 | 99.955 | 99.782 |
| 113 | 100.000 | 100.000 | 100.000 | 99.957 | 99.790 |
| 114 | 100.000 | 100.000 | 100.000 | 99.959 | 99.799 |
| 115 | 100.000 | 100.000 | 100.000 | 99.961 | 99.807 |
| 116 | 100.000 | 100.000 | 100.000 | 99.964 | 99.815 |
| 117 | 100.000 | 100.000 | 100.000 | 99.966 | 99.823 |
| 118 | 100.000 | 100.000 | 100.000 | 99.968 | 99.831 |
| 119 | 100.000 | 100.000 | 100.000 | 99.970 | 99.840 |
| 120 | 100.000 | 100.000 | 100.000 | 99.972 | 99.848 |
| 121 | 100.000 | 100.000 | 100.000 | 99.974 | 99.857 |
| 122 | 100.000 | 100.000 | 100.000 | 99.976 | 99.865 |
| 123 | 100.000 | 100.000 | 100.000 | 99.978 | 99.873 |
| 124 | 100.000 | 100.000 | 100.000 | 99.980 | 99.881 |
| 125 | 100.000 | 100.000 | 100.000 | 99.982 | 99.889 |
| 126 | 100.000 | 100.000 | 100.000 | 99.984 | 99.897 |
| 127 | 100.000 | 100.000 | 100.000 | 99.986 | 99.905 |
| 128 | 100.000 | 100.000 | 100.000 | 99.989 | 99.913 |
| 129 | 100.000 | 100.000 | 100.000 | 99.991 | 99.921 |
| 130 | 100.000 | 100.000 | 100.000 | 99.993 | 99.929 |
| 131 | 100.000 | 100.000 | 100.000 | 99.994 | 99.934 |
| 132 | 100.000 | 100.000 | 100.000 | 99.995 | 99.939 |
| 133 | 100.000 | 100.000 | 100.000 | 99.996 | 99.944 |
| 134 | 100.000 | 100.000 | 100.000 | 99.997 | 99.949 |
| 135 | 100.000 | 100.000 | 100.000 | 99.998 | 99.954 |
| 136 | 100.000 | 100.000 | 100.000 | 99.999 | 99.959 |
| 137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
| 140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.966 |
| 141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
| 144 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
| 145 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
| 146 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
| 147 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
| 148 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
| 149 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 150 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 151 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
| 152 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 153 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 154 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
| 155 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
| 156 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
| 157 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
| 158 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
| 159 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
| 160 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 161 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
| 162 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
| 163 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 164 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 165 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 166 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 167 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
| 168 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
| 169 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 170 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 171 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 172 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 173 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 174 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 175 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 176 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 178 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 179 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
flex
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 67.442 | 47.045 | 33.730 | 24.737 | 18.489 |
| 2 | 81.395 | 64.774 | 51.112 | 40.252 | 31.739 |
| 3 | 88.372 | 74.855 | 62.088 | 50.933 | 41.540 |
| 4 | 93.023 | 82.503 | 71.125 | 60.228 | 50.409 |
| 5 | 95.349 | 86.327 | 75.852 | 65.403 | 55.691 |
| 6 | 95.349 | 87.601 | 78.445 | 68.973 | 59.835 |
| 7 | 95.349 | 88.528 | 80.384 | 71.714 | 63.102 |
| 8 | 95.349 | 89.108 | 81.723 | 73.754 | 65.676 |
| 9 | 97.674 | 92.932 | 86.357 | 78.710 | 70.634 |
| 10 | 97.674 | 93.163 | 87.012 | 79.849 | 72.214 |
| 11 | 97.674 | 93.859 | 88.461 | 81.905 | 74.684 |
| 12 | 97.674 | 94.670 | 90.067 | 84.089 | 77.221 |
| 13 | 100.000 | 97.914 | 93.640 | 87.720 | 80.778 |
| 14 | 100.000 | 98.030 | 94.018 | 88.436 | 81.834 |
| 15 | 100.000 | 98.262 | 94.628 | 89.451 | 83.208 |
| 16 | 100.000 | 98.378 | 95.006 | 90.161 | 84.244 |
| 17 | 100.000 | 98.494 | 95.357 | 90.801 | 85.172 |
| 18 | 100.000 | 98.494 | 95.486 | 91.157 | 85.795 |
| 19 | 100.000 | 98.494 | 95.597 | 91.468 | 86.346 |
| 20 | 100.000 | 98.610 | 95.901 | 91.991 | 87.084 |
| 21 | 100.000 | 98.610 | 95.994 | 92.247 | 87.536 |
| 22 | 100.000 | 98.841 | 96.492 | 92.989 | 88.483 |
| 23 | 100.000 | 98.841 | 96.594 | 93.272 | 88.980 |
| 24 | 100.000 | 99.073 | 97.101 | 94.033 | 89.950 |
| 25 | 100.000 | 99.073 | 97.185 | 94.264 | 90.360 |
| 26 | 100.000 | 99.073 | 97.240 | 94.434 | 90.684 |
| 27 | 100.000 | 99.073 | 97.286 | 94.576 | 90.956 |
| 28 | 100.000 | 99.073 | 97.323 | 94.691 | 91.183 |
| 29 | 100.000 | 99.073 | 97.406 | 94.925 | 91.597 |
| 30 | 100.000 | 99.073 | 97.434 | 95.017 | 91.787 |
| 31 | 100.000 | 99.073 | 97.461 | 95.109 | 91.974 |
| 32 | 100.000 | 99.189 | 97.729 | 95.533 | 92.543 |
| 33 | 100.000 | 99.189 | 97.775 | 95.672 | 92.807 |
| 34 | 100.000 | 99.189 | 97.794 | 95.736 | 92.942 |
| 35 | 100.000 | 99.305 | 98.043 | 96.111 | 93.430 |
| 36 | 100.000 | 99.305 | 98.052 | 96.151 | 93.525 |
| 37 | 100.000 | 99.305 | 98.098 | 96.284 | 93.769 |
| 38 | 100.000 | 99.305 | 98.145 | 96.416 | 94.010 |
| 39 | 100.000 | 99.305 | 98.154 | 96.454 | 94.098 |
| 40 | 100.000 | 99.305 | 98.181 | 96.540 | 94.267 |
| 41 | 100.000 | 99.305 | 98.218 | 96.646 | 94.462 |
| 42 | 100.000 | 99.421 | 98.477 | 97.038 | 94.965 |
| 43 | 100.000 | 99.421 | 98.514 | 97.144 | 95.161 |
| 44 | 100.000 | 99.421 | 98.532 | 97.205 | 95.284 |
| 45 | 100.000 | 99.421 | 98.551 | 97.264 | 95.403 |
| 46 | 100.000 | 99.421 | 98.560 | 97.299 | 95.484 |
| 47 | 100.000 | 99.537 | 98.809 | 97.666 | 95.944 |
| 48 | 100.000 | 99.537 | 98.818 | 97.703 | 96.031 |
| 49 | 100.000 | 99.652 | 99.068 | 98.071 | 96.494 |
| 50 | 100.000 | 99.652 | 99.077 | 98.105 | 96.571 |
| 51 | 100.000 | 99.652 | 99.086 | 98.141 | 96.650 |
| 52 | 100.000 | 99.768 | 99.335 | 98.507 | 97.110 |
| 53 | 100.000 | 99.884 | 99.585 | 98.874 | 97.569 |
| 54 | 100.000 | 99.884 | 99.603 | 98.931 | 97.681 |
| 55 | 100.000 | 99.884 | 99.612 | 98.963 | 97.748 |
| 56 | 100.000 | 99.884 | 99.612 | 98.972 | 97.783 |
| 57 | 100.000 | 99.884 | 99.622 | 99.006 | 97.858 |
| 58 | 100.000 | 100.000 | 99.871 | 99.372 | 98.314 |
| 59 | 100.000 | 100.000 | 99.889 | 99.429 | 98.426 |
| 60 | 100.000 | 100.000 | 99.889 | 99.439 | 98.460 |
| 61 | 100.000 | 100.000 | 99.889 | 99.450 | 98.500 |
| 62 | 100.000 | 100.000 | 99.889 | 99.457 | 98.527 |
| 63 | 100.000 | 100.000 | 99.889 | 99.466 | 98.557 |
| 64 | 100.000 | 100.000 | 99.889 | 99.473 | 98.584 |
| 65 | 100.000 | 100.000 | 99.889 | 99.480 | 98.611 |
| 66 | 100.000 | 100.000 | 99.908 | 99.533 | 98.709 |
| 67 | 100.000 | 100.000 | 99.908 | 99.542 | 98.739 |
| 68 | 100.000 | 100.000 | 99.908 | 99.548 | 98.762 |
| 69 | 100.000 | 100.000 | 99.908 | 99.556 | 98.792 |
| 70 | 100.000 | 100.000 | 99.908 | 99.563 | 98.815 |
| 71 | 100.000 | 100.000 | 99.908 | 99.569 | 98.838 |
| 72 | 100.000 | 100.000 | 99.908 | 99.573 | 98.855 |
| 73 | 100.000 | 100.000 | 99.908 | 99.578 | 98.875 |
| 74 | 100.000 | 100.000 | 99.908 | 99.584 | 98.897 |
| 75 | 100.000 | 100.000 | 99.908 | 99.590 | 98.917 |
| 76 | 100.000 | 100.000 | 99.908 | 99.595 | 98.936 |
| 77 | 100.000 | 100.000 | 99.908 | 99.599 | 98.953 |
| 78 | 100.000 | 100.000 | 99.908 | 99.603 | 98.969 |
| 79 | 100.000 | 100.000 | 99.908 | 99.608 | 98.988 |
| 80 | 100.000 | 100.000 | 99.908 | 99.614 | 99.008 |
| 81 | 100.000 | 100.000 | 99.908 | 99.617 | 99.021 |
| 82 | 100.000 | 100.000 | 99.908 | 99.621 | 99.037 |
| 83 | 100.000 | 100.000 | 99.908 | 99.625 | 99.052 |
| 84 | 100.000 | 100.000 | 99.908 | 99.629 | 99.068 |
| 85 | 100.000 | 100.000 | 99.908 | 99.633 | 99.084 |
| 86 | 100.000 | 100.000 | 99.908 | 99.637 | 99.097 |
| 87 | 100.000 | 100.000 | 99.908 | 99.640 | 99.109 |
| 88 | 100.000 | 100.000 | 99.908 | 99.644 | 99.124 |
| 89 | 100.000 | 100.000 | 99.908 | 99.647 | 99.137 |
| 90 | 100.000 | 100.000 | 99.908 | 99.651 | 99.152 |
| 91 | 100.000 | 100.000 | 99.908 | 99.653 | 99.161 |
| 92 | 100.000 | 100.000 | 99.917 | 99.680 | 99.212 |
| 93 | 100.000 | 100.000 | 99.917 | 99.683 | 99.224 |
| 94 | 100.000 | 100.000 | 99.917 | 99.687 | 99.236 |
| 95 | 100.000 | 100.000 | 99.917 | 99.689 | 99.245 |
| 96 | 100.000 | 100.000 | 99.917 | 99.691 | 99.254 |
| 97 | 100.000 | 100.000 | 99.917 | 99.695 | 99.269 |
| 98 | 100.000 | 100.000 | 99.917 | 99.697 | 99.278 |
| 99 | 100.000 | 100.000 | 99.917 | 99.700 | 99.290 |
| 100 | 100.000 | 100.000 | 99.926 | 99.727 | 99.341 |
| 101 | 100.000 | 100.000 | 99.926 | 99.729 | 99.349 |
| 102 | 100.000 | 100.000 | 99.935 | 99.756 | 99.400 |
| 103 | 100.000 | 100.000 | 99.945 | 99.783 | 99.450 |
| 104 | 100.000 | 100.000 | 99.945 | 99.785 | 99.459 |
| 105 | 100.000 | 100.000 | 99.954 | 99.813 | 99.510 |
| 106 | 100.000 | 100.000 | 99.963 | 99.840 | 99.560 |
| 107 | 100.000 | 100.000 | 99.972 | 99.867 | 99.611 |
| 108 | 100.000 | 100.000 | 99.982 | 99.894 | 99.661 |
| 109 | 100.000 | 100.000 | 99.991 | 99.921 | 99.712 |
| 110 | 100.000 | 100.000 | 100.000 | 99.948 | 99.763 |
| 111 | 100.000 | 100.000 | 100.000 | 99.950 | 99.771 |
| 112 | 100.000 | 100.000 | 100.000 | 99.952 | 99.779 |
| 113 | 100.000 | 100.000 | 100.000 | 99.954 | 99.788 |
| 114 | 100.000 | 100.000 | 100.000 | 99.956 | 99.796 |
| 115 | 100.000 | 100.000 | 100.000 | 99.958 | 99.804 |
| 116 | 100.000 | 100.000 | 100.000 | 99.959 | 99.809 |
| 117 | 100.000 | 100.000 | 100.000 | 99.963 | 99.821 |
| 118 | 100.000 | 100.000 | 100.000 | 99.965 | 99.829 |
| 119 | 100.000 | 100.000 | 100.000 | 99.967 | 99.837 |
| 120 | 100.000 | 100.000 | 100.000 | 99.968 | 99.842 |
| 121 | 100.000 | 100.000 | 100.000 | 99.970 | 99.850 |
| 122 | 100.000 | 100.000 | 100.000 | 99.972 | 99.858 |
| 123 | 100.000 | 100.000 | 100.000 | 99.974 | 99.866 |
| 124 | 100.000 | 100.000 | 100.000 | 99.976 | 99.874 |
| 125 | 100.000 | 100.000 | 100.000 | 99.978 | 99.882 |
| 126 | 100.000 | 100.000 | 100.000 | 99.980 | 99.890 |
| 127 | 100.000 | 100.000 | 100.000 | 99.982 | 99.898 |
| 128 | 100.000 | 100.000 | 100.000 | 99.984 | 99.906 |
| 129 | 100.000 | 100.000 | 100.000 | 99.986 | 99.914 |
| 130 | 100.000 | 100.000 | 100.000 | 99.989 | 99.922 |
| 131 | 100.000 | 100.000 | 100.000 | 99.990 | 99.926 |
| 132 | 100.000 | 100.000 | 100.000 | 99.992 | 99.934 |
| 133 | 100.000 | 100.000 | 100.000 | 99.994 | 99.942 |
| 134 | 100.000 | 100.000 | 100.000 | 99.996 | 99.950 |
| 135 | 100.000 | 100.000 | 100.000 | 99.997 | 99.954 |
| 136 | 100.000 | 100.000 | 100.000 | 99.999 | 99.962 |
| 137 | 100.000 | 100.000 | 100.000 | 99.999 | 99.963 |
| 138 | 100.000 | 100.000 | 100.000 | 99.999 | 99.965 |
| 139 | 100.000 | 100.000 | 100.000 | 99.999 | 99.966 |
| 140 | 100.000 | 100.000 | 100.000 | 99.999 | 99.967 |
| 141 | 100.000 | 100.000 | 100.000 | 99.999 | 99.968 |
| 142 | 100.000 | 100.000 | 100.000 | 99.999 | 99.969 |
| 143 | 100.000 | 100.000 | 100.000 | 99.999 | 99.970 |
| 144 | 100.000 | 100.000 | 100.000 | 99.999 | 99.972 |
| 145 | 100.000 | 100.000 | 100.000 | 99.999 | 99.973 |
| 146 | 100.000 | 100.000 | 100.000 | 99.999 | 99.974 |
| 147 | 100.000 | 100.000 | 100.000 | 99.999 | 99.974 |
| 148 | 100.000 | 100.000 | 100.000 | 99.999 | 99.975 |
| 149 | 100.000 | 100.000 | 100.000 | 99.999 | 99.976 |
| 150 | 100.000 | 100.000 | 100.000 | 99.999 | 99.977 |
| 151 | 100.000 | 100.000 | 100.000 | 99.999 | 99.978 |
| 152 | 100.000 | 100.000 | 100.000 | 99.999 | 99.979 |
| 153 | 100.000 | 100.000 | 100.000 | 99.999 | 99.980 |
| 154 | 100.000 | 100.000 | 100.000 | 99.999 | 99.981 |
| 155 | 100.000 | 100.000 | 100.000 | 99.999 | 99.982 |
| 156 | 100.000 | 100.000 | 100.000 | 99.999 | 99.983 |
| 157 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
| 158 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 159 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 160 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 161 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 162 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
| 163 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 164 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 165 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 166 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 167 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 168 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 169 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 170 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 171 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 172 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 173 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 174 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 175 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 176 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 177 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 178 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
| 179 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
| 180 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
grep
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 64.286 | 34.687 | 19.287 | 11.372 | 7.125 |
| 4 | 71.429 | 41.794 | 24.402 | 14.787 | 9.398 |
| 5 | 80.952 | 48.900 | 29.141 | 17.945 | 11.545 |
| 6 | 85.714 | 53.807 | 32.895 | 20.632 | 13.457 |
| 7 | 90.476 | 58.545 | 36.696 | 23.463 | 15.504 |
| 8 | 95.238 | 63.283 | 40.497 | 26.293 | 17.551 |
| 9 | 97.619 | 67.174 | 44.064 | 29.071 | 19.589 |
| 10 | 100.000 | 70.558 | 47.208 | 31.602 | 21.506 |
| 11 | 100.000 | 72.927 | 50.023 | 34.046 | 23.407 |
| 12 | 100.000 | 75.296 | 52.816 | 36.481 | 25.314 |
| 13 | 100.000 | 77.327 | 55.256 | 38.663 | 27.073 |
| 14 | 100.000 | 79.188 | 57.602 | 40.802 | 28.803 |
| 15 | 100.000 | 80.880 | 59.784 | 42.840 | 30.490 |
| 16 | 100.000 | 82.572 | 61.872 | 44.759 | 32.080 |
| 17 | 100.000 | 83.926 | 63.749 | 46.603 | 33.661 |
| 18 | 100.000 | 85.110 | 65.415 | 48.287 | 35.148 |
| 19 | 100.000 | 86.125 | 66.753 | 49.601 | 36.311 |
| 20 | 100.000 | 87.140 | 68.207 | 51.102 | 37.667 |
| 21 | 100.000 | 87.986 | 69.521 | 52.517 | 38.978 |
| 22 | 100.000 | 88.832 | 70.695 | 53.718 | 40.076 |
| 23 | 100.000 | 89.679 | 72.032 | 55.171 | 41.426 |
| 24 | 100.000 | 90.525 | 73.393 | 56.662 | 42.811 |
| 25 | 100.000 | 91.371 | 74.801 | 58.227 | 44.272 |
| 26 | 100.000 | 92.047 | 75.692 | 59.107 | 45.058 |
| 27 | 100.000 | 92.724 | 76.771 | 60.308 | 46.210 |
| 28 | 100.000 | 93.401 | 78.038 | 61.782 | 47.617 |
| 29 | 100.000 | 94.078 | 79.024 | 62.833 | 48.612 |
| 30 | 100.000 | 94.755 | 80.009 | 63.884 | 49.607 |
| 31 | 100.000 | 95.262 | 80.784 | 64.753 | 50.471 |
| 32 | 100.000 | 95.770 | 81.628 | 65.723 | 51.429 |
| 33 | 100.000 | 96.277 | 82.543 | 66.822 | 52.536 |
| 34 | 100.000 | 96.785 | 83.388 | 67.803 | 53.514 |
| 35 | 100.000 | 97.293 | 84.233 | 68.785 | 54.496 |
| 36 | 100.000 | 97.800 | 85.031 | 69.680 | 55.375 |
| 37 | 100.000 | 98.308 | 85.734 | 70.409 | 56.057 |
| 38 | 100.000 | 98.646 | 86.415 | 71.272 | 56.963 |
| 39 | 100.000 | 98.985 | 87.025 | 72.034 | 57.773 |
| 40 | 100.000 | 99.323 | 87.541 | 72.613 | 58.352 |
| 41 | 100.000 | 99.662 | 88.104 | 73.294 | 59.073 |
| 42 | 100.000 | 99.831 | 88.433 | 73.717 | 59.535 |
| 43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |
grep
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
| 3 | 64.286 | 34.518 | 19.193 | 11.340 | 7.118 |
| 4 | 71.429 | 41.455 | 24.214 | 14.723 | 9.383 |
| 5 | 76.190 | 46.870 | 28.578 | 17.881 | 11.587 |
| 6 | 83.333 | 52.961 | 33.060 | 21.039 | 13.782 |
| 7 | 85.714 | 57.191 | 36.931 | 24.009 | 15.922 |
| 8 | 88.095 | 61.421 | 40.779 | 26.958 | 18.050 |
| 9 | 90.476 | 65.313 | 44.392 | 29.795 | 20.134 |
| 10 | 92.857 | 68.697 | 47.489 | 32.277 | 22.022 |
| 11 | 95.238 | 71.912 | 50.493 | 34.738 | 23.920 |
| 12 | 97.619 | 74.958 | 53.332 | 37.086 | 25.752 |
| 13 | 97.619 | 76.819 | 55.655 | 39.215 | 27.492 |
| 14 | 97.619 | 78.511 | 57.837 | 41.263 | 29.195 |
| 15 | 97.619 | 80.034 | 59.808 | 43.129 | 30.767 |
| 16 | 97.619 | 81.049 | 61.427 | 44.850 | 32.313 |
| 17 | 97.619 | 82.064 | 62.928 | 46.432 | 33.757 |
| 18 | 97.619 | 83.249 | 64.524 | 48.008 | 35.142 |
| 19 | 97.619 | 84.264 | 66.025 | 49.579 | 36.565 |
| 20 | 97.619 | 85.279 | 67.480 | 51.091 | 37.940 |
| 21 | 97.619 | 86.294 | 68.888 | 52.523 | 39.233 |
| 22 | 97.619 | 87.310 | 70.296 | 53.954 | 40.526 |
| 23 | 97.619 | 88.325 | 71.680 | 55.338 | 41.755 |
| 24 | 97.619 | 88.832 | 72.712 | 56.603 | 43.013 |
| 25 | 97.619 | 89.509 | 73.792 | 57.815 | 44.185 |
| 26 | 97.619 | 90.186 | 74.871 | 59.016 | 45.337 |
| 27 | 100.000 | 92.217 | 76.420 | 60.195 | 46.253 |
| 28 | 100.000 | 92.893 | 77.452 | 61.326 | 47.338 |
| 29 | 100.000 | 93.570 | 78.461 | 62.410 | 48.361 |
| 30 | 100.000 | 94.247 | 79.446 | 63.460 | 49.356 |
| 31 | 100.000 | 94.924 | 80.432 | 64.511 | 50.350 |
| 32 | 100.000 | 95.431 | 81.300 | 65.530 | 51.364 |
| 33 | 100.000 | 95.770 | 82.074 | 66.527 | 52.390 |
| 34 | 100.000 | 96.108 | 82.802 | 67.460 | 53.363 |
| 35 | 100.000 | 96.785 | 83.693 | 68.339 | 54.149 |
| 36 | 100.000 | 97.293 | 84.491 | 69.251 | 55.064 |
| 37 | 100.000 | 97.800 | 85.265 | 70.120 | 55.928 |
| 38 | 100.000 | 98.308 | 86.016 | 70.945 | 56.739 |
| 39 | 100.000 | 98.646 | 86.649 | 71.733 | 57.566 |
| 40 | 100.000 | 99.154 | 87.353 | 72.463 | 58.248 |
| 41 | 100.000 | 99.492 | 87.916 | 73.138 | 58.957 |
| 42 | 100.000 | 99.831 | 88.433 | 73.717 | 59.535 |
| 43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |
grep
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
| 3 | 61.905 | 33.503 | 18.841 | 11.233 | 7.090 |
| 4 | 71.429 | 41.455 | 24.261 | 14.771 | 9.413 |
| 5 | 80.952 | 49.239 | 29.517 | 18.208 | 11.687 |
| 6 | 85.714 | 54.653 | 33.881 | 21.366 | 13.891 |
| 7 | 88.095 | 59.391 | 38.104 | 24.503 | 16.094 |
| 8 | 90.476 | 63.621 | 41.952 | 27.452 | 18.222 |
| 9 | 90.476 | 65.990 | 44.721 | 29.864 | 20.118 |
| 10 | 92.857 | 68.866 | 47.583 | 32.309 | 22.030 |
| 11 | 95.238 | 71.912 | 50.446 | 34.684 | 23.881 |
| 12 | 100.000 | 76.142 | 53.778 | 37.221 | 25.778 |
| 13 | 100.000 | 77.834 | 55.913 | 39.210 | 27.435 |
| 14 | 100.000 | 79.357 | 57.907 | 41.113 | 29.044 |
| 15 | 100.000 | 80.372 | 59.573 | 42.866 | 30.592 |
| 16 | 100.000 | 81.387 | 61.098 | 44.480 | 32.062 |
| 17 | 100.000 | 82.741 | 62.905 | 46.239 | 33.578 |
| 18 | 100.000 | 83.926 | 64.547 | 47.896 | 35.046 |
| 19 | 100.000 | 84.941 | 66.002 | 49.408 | 36.422 |
| 20 | 100.000 | 85.787 | 67.316 | 50.839 | 37.761 |
| 21 | 100.000 | 86.802 | 68.771 | 52.340 | 39.116 |
| 22 | 100.000 | 87.986 | 70.366 | 53.917 | 40.502 |
| 23 | 100.000 | 88.832 | 71.633 | 55.289 | 41.794 |
| 24 | 100.000 | 89.848 | 73.041 | 56.715 | 43.079 |
| 25 | 100.000 | 90.863 | 74.449 | 58.142 | 44.365 |
| 26 | 100.000 | 91.709 | 75.669 | 59.423 | 45.554 |
| 27 | 100.000 | 92.724 | 77.030 | 60.774 | 46.756 |
| 28 | 100.000 | 93.401 | 78.109 | 61.975 | 47.908 |
| 29 | 100.000 | 94.078 | 79.141 | 63.107 | 48.993 |
| 30 | 100.000 | 94.416 | 79.939 | 64.131 | 50.038 |
| 31 | 100.000 | 95.093 | 80.924 | 65.181 | 51.033 |
| 32 | 100.000 | 95.601 | 81.746 | 66.120 | 51.963 |
| 33 | 100.000 | 95.939 | 82.426 | 66.983 | 52.869 |
| 34 | 100.000 | 96.447 | 83.247 | 67.921 | 53.799 |
| 35 | 100.000 | 96.954 | 84.045 | 68.822 | 54.691 |
| 36 | 100.000 | 97.462 | 84.819 | 69.691 | 55.554 |
| 37 | 100.000 | 98.139 | 85.711 | 70.570 | 56.340 |
| 38 | 100.000 | 98.477 | 86.274 | 71.251 | 57.061 |
| 39 | 100.000 | 98.816 | 86.861 | 71.948 | 57.773 |
| 40 | 100.000 | 99.323 | 87.565 | 72.677 | 58.456 |
| 41 | 100.000 | 99.662 | 88.081 | 73.256 | 59.034 |
| 42 | 100.000 | 99.831 | 88.409 | 73.680 | 59.497 |
| 43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |
grep
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 59.524 | 32.318 | 18.348 | 11.045 | 7.025 |
| 4 | 66.667 | 39.594 | 23.627 | 14.562 | 9.346 |
| 5 | 73.810 | 46.701 | 28.789 | 18.026 | 11.648 |
| 6 | 78.571 | 51.777 | 32.895 | 21.050 | 13.797 |
| 7 | 80.952 | 54.992 | 36.039 | 23.666 | 15.800 |
| 8 | 83.333 | 58.376 | 39.207 | 26.235 | 17.752 |
| 9 | 88.095 | 62.944 | 42.891 | 29.012 | 19.781 |
| 10 | 88.095 | 65.144 | 45.519 | 31.333 | 21.623 |
| 11 | 88.095 | 67.343 | 48.170 | 33.676 | 23.476 |
| 12 | 90.476 | 70.051 | 50.634 | 35.757 | 25.159 |
| 13 | 92.857 | 73.096 | 53.426 | 38.046 | 26.945 |
| 14 | 92.857 | 74.450 | 55.256 | 39.842 | 28.500 |
| 15 | 97.619 | 78.680 | 58.517 | 42.282 | 30.312 |
| 16 | 97.619 | 80.034 | 60.347 | 44.078 | 31.864 |
| 17 | 100.000 | 82.910 | 62.905 | 46.164 | 33.510 |
| 18 | 100.000 | 84.095 | 64.547 | 47.810 | 34.961 |
| 19 | 100.000 | 84.941 | 65.931 | 49.322 | 36.357 |
| 20 | 100.000 | 86.125 | 67.597 | 50.995 | 37.828 |
| 21 | 100.000 | 86.971 | 68.935 | 52.448 | 39.177 |
| 22 | 100.000 | 87.817 | 70.202 | 53.809 | 40.453 |
| 23 | 100.000 | 88.832 | 71.656 | 55.311 | 41.809 |
| 24 | 100.000 | 90.017 | 73.229 | 56.855 | 43.166 |
| 25 | 100.000 | 90.863 | 74.519 | 58.238 | 44.450 |
| 26 | 100.000 | 91.878 | 75.880 | 59.611 | 45.696 |
| 27 | 100.000 | 92.555 | 76.983 | 60.844 | 46.876 |
| 28 | 100.000 | 93.401 | 78.179 | 62.088 | 48.026 |
| 29 | 100.000 | 94.078 | 79.165 | 63.155 | 49.056 |
| 30 | 100.000 | 94.416 | 79.962 | 64.179 | 50.101 |
| 31 | 100.000 | 94.755 | 80.666 | 65.069 | 51.022 |
| 32 | 100.000 | 95.262 | 81.488 | 66.007 | 51.952 |
| 33 | 100.000 | 95.770 | 82.309 | 66.945 | 52.882 |
| 34 | 100.000 | 96.447 | 83.271 | 67.959 | 53.838 |
| 35 | 100.000 | 97.124 | 84.233 | 68.972 | 54.794 |
| 36 | 100.000 | 97.631 | 85.007 | 69.841 | 55.658 |
| 37 | 100.000 | 97.970 | 85.570 | 70.522 | 56.379 |
| 38 | 100.000 | 98.308 | 86.157 | 71.219 | 57.091 |
| 39 | 100.000 | 98.816 | 86.861 | 71.948 | 57.773 |
| 40 | 100.000 | 99.323 | 87.565 | 72.677 | 58.456 |
| 41 | 100.000 | 99.662 | 88.081 | 73.256 | 59.034 |
| 42 | 100.000 | 99.831 | 88.409 | 73.680 | 59.497 |
| 43 | 100.000 | 100.000 | 88.714 | 74.066 | 59.920 |
grep
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 66.667 | 35.533 | 19.521 | 11.426 | 7.134 |
| 4 | 76.190 | 43.486 | 24.941 | 14.964 | 9.457 |
| 5 | 85.714 | 51.100 | 30.103 | 18.369 | 11.724 |
| 6 | 90.476 | 57.191 | 34.937 | 21.747 | 14.010 |
| 7 | 92.857 | 61.929 | 39.160 | 24.883 | 16.212 |
| 8 | 95.238 | 66.328 | 43.149 | 27.913 | 18.377 |
| 9 | 97.619 | 70.051 | 46.574 | 30.615 | 20.386 |
| 10 | 100.000 | 73.435 | 49.789 | 33.242 | 22.385 |
| 11 | 100.000 | 75.296 | 52.112 | 35.376 | 24.134 |
| 12 | 100.000 | 77.157 | 54.575 | 37.671 | 25.996 |
| 13 | 100.000 | 79.019 | 56.875 | 39.778 | 27.727 |
| 14 | 100.000 | 80.541 | 59.033 | 41.880 | 29.488 |
| 15 | 100.000 | 81.895 | 61.004 | 43.842 | 31.165 |
| 16 | 100.000 | 83.249 | 62.764 | 45.536 | 32.626 |
| 17 | 100.000 | 84.433 | 64.406 | 47.182 | 34.077 |
| 18 | 100.000 | 85.448 | 65.814 | 48.619 | 35.379 |
| 19 | 100.000 | 86.464 | 67.222 | 50.051 | 36.672 |
| 20 | 100.000 | 87.479 | 68.583 | 51.402 | 37.874 |
| 21 | 100.000 | 88.494 | 70.014 | 52.866 | 39.194 |
| 22 | 100.000 | 89.340 | 71.258 | 54.179 | 40.411 |
| 23 | 100.000 | 90.186 | 72.642 | 55.713 | 41.844 |
| 24 | 100.000 | 91.032 | 74.050 | 57.257 | 43.268 |
| 25 | 100.000 | 91.709 | 75.364 | 58.774 | 44.692 |
| 26 | 100.000 | 92.386 | 76.396 | 59.895 | 45.753 |
| 27 | 100.000 | 93.063 | 77.288 | 60.774 | 46.539 |
| 28 | 100.000 | 93.570 | 78.179 | 61.847 | 47.628 |
| 29 | 100.000 | 94.078 | 78.954 | 62.715 | 48.492 |
| 30 | 100.000 | 94.585 | 79.798 | 63.707 | 49.494 |
| 31 | 100.000 | 95.093 | 80.526 | 64.474 | 50.215 |
| 32 | 100.000 | 95.601 | 81.253 | 65.240 | 50.937 |
| 33 | 100.000 | 96.108 | 82.074 | 66.179 | 51.867 |
| 34 | 100.000 | 96.447 | 82.708 | 66.967 | 52.693 |
| 35 | 100.000 | 96.785 | 83.388 | 67.825 | 53.587 |
| 36 | 100.000 | 97.124 | 83.881 | 68.361 | 54.114 |
| 37 | 100.000 | 97.462 | 84.420 | 69.004 | 54.796 |
| 38 | 100.000 | 97.800 | 85.031 | 69.771 | 55.615 |
| 39 | 100.000 | 97.970 | 85.617 | 70.629 | 56.555 |
| 40 | 100.000 | 98.139 | 86.133 | 71.374 | 57.383 |
| 41 | 100.000 | 98.308 | 86.509 | 71.894 | 57.977 |
| 42 | 100.000 | 98.477 | 86.908 | 72.452 | 58.609 |
| 43 | 100.000 | 98.646 | 87.353 | 73.101 | 59.352 |
| 44 | 100.000 | 98.816 | 87.682 | 73.540 | 59.848 |
| 45 | 100.000 | 98.985 | 88.034 | 74.007 | 60.362 |
| 46 | 100.000 | 99.154 | 88.409 | 74.532 | 60.967 |
| 47 | 100.000 | 99.323 | 88.761 | 75.031 | 61.553 |
| 48 | 100.000 | 99.492 | 89.254 | 75.739 | 62.343 |
| 49 | 100.000 | 99.662 | 89.653 | 76.296 | 62.975 |
| 50 | 100.000 | 99.831 | 89.911 | 76.591 | 63.280 |
| 51 | 100.000 | 100.000 | 90.216 | 76.977 | 63.704 |
| 52 | 100.000 | 100.000 | 90.357 | 77.261 | 64.090 |
| 53 | 100.000 | 100.000 | 90.802 | 78.055 | 65.031 |
| 54 | 100.000 | 100.000 | 90.990 | 78.441 | 65.553 |
| 55 | 100.000 | 100.000 | 91.061 | 78.586 | 65.754 |
| 56 | 100.000 | 100.000 | 91.107 | 78.687 | 65.904 |
| 57 | 100.000 | 100.000 | 91.248 | 78.961 | 66.263 |
| 58 | 100.000 | 100.000 | 91.436 | 79.336 | 66.764 |
| 59 | 100.000 | 100.000 | 91.741 | 79.888 | 67.437 |
| 60 | 100.000 | 100.000 | 91.929 | 80.264 | 67.938 |
| 61 | 100.000 | 100.000 | 92.046 | 80.505 | 68.269 |
| 62 | 100.000 | 100.000 | 92.281 | 80.961 | 68.861 |
| 63 | 100.000 | 100.000 | 92.374 | 81.165 | 69.153 |
| 64 | 100.000 | 100.000 | 92.562 | 81.545 | 69.667 |
| 65 | 100.000 | 100.000 | 92.679 | 81.781 | 69.987 |
| 66 | 100.000 | 100.000 | 92.726 | 81.883 | 70.137 |
| 67 | 100.000 | 100.000 | 92.844 | 82.130 | 70.485 |
| 68 | 100.000 | 100.000 | 92.867 | 82.189 | 70.583 |
| 69 | 100.000 | 100.000 | 92.914 | 82.290 | 70.732 |
| 70 | 100.000 | 100.000 | 92.985 | 82.441 | 70.949 |
| 71 | 100.000 | 100.000 | 93.102 | 82.682 | 71.284 |
| 72 | 100.000 | 100.000 | 93.172 | 82.843 | 71.524 |
| 73 | 100.000 | 100.000 | 93.290 | 83.084 | 71.855 |
| 74 | 100.000 | 100.000 | 93.407 | 83.320 | 72.175 |
| 75 | 100.000 | 100.000 | 93.454 | 83.427 | 72.339 |
| 76 | 100.000 | 100.000 | 93.595 | 83.722 | 72.750 |
| 77 | 100.000 | 100.000 | 93.688 | 83.915 | 73.018 |
| 78 | 100.000 | 100.000 | 93.759 | 84.065 | 73.234 |
| 79 | 100.000 | 100.000 | 93.782 | 84.119 | 73.318 |
| 80 | 100.000 | 100.000 | 94.017 | 84.521 | 73.789 |
| 81 | 100.000 | 100.000 | 94.111 | 84.714 | 74.057 |
| 82 | 100.000 | 100.000 | 94.392 | 85.245 | 74.727 |
| 83 | 100.000 | 100.000 | 94.674 | 85.781 | 75.398 |
| 84 | 100.000 | 100.000 | 94.721 | 85.904 | 75.601 |
| 85 | 100.000 | 100.000 | 94.791 | 86.076 | 75.873 |
| 86 | 100.000 | 100.000 | 94.979 | 86.462 | 76.395 |
| 87 | 100.000 | 100.000 | 95.190 | 86.880 | 76.940 |
| 88 | 100.000 | 100.000 | 95.237 | 87.014 | 77.171 |
| 89 | 100.000 | 100.000 | 95.331 | 87.229 | 77.491 |
| 90 | 100.000 | 100.000 | 95.448 | 87.464 | 77.813 |
| 91 | 100.000 | 100.000 | 95.565 | 87.711 | 78.161 |
| 92 | 100.000 | 100.000 | 95.636 | 87.856 | 78.362 |
| 93 | 100.000 | 100.000 | 95.753 | 88.113 | 78.734 |
| 94 | 100.000 | 100.000 | 95.847 | 88.306 | 79.002 |
| 95 | 100.000 | 100.000 | 95.870 | 88.381 | 79.141 |
| 96 | 100.000 | 100.000 | 95.964 | 88.590 | 79.448 |
| 97 | 100.000 | 100.000 | 95.988 | 88.671 | 79.599 |
| 98 | 100.000 | 100.000 | 96.058 | 88.832 | 79.844 |
| 99 | 100.000 | 100.000 | 96.293 | 89.277 | 80.411 |
| 100 | 100.000 | 100.000 | 96.387 | 89.491 | 80.724 |
| 101 | 100.000 | 100.000 | 96.457 | 89.647 | 80.953 |
| 102 | 100.000 | 100.000 | 96.621 | 89.990 | 81.423 |
| 103 | 100.000 | 100.000 | 96.809 | 90.370 | 81.937 |
| 104 | 100.000 | 100.000 | 96.832 | 90.424 | 82.020 |
| 105 | 100.000 | 100.000 | 96.973 | 90.698 | 82.379 |
| 106 | 100.000 | 100.000 | 97.255 | 91.212 | 83.013 |
| 107 | 100.000 | 100.000 | 97.349 | 91.405 | 83.281 |
| 108 | 100.000 | 100.000 | 97.396 | 91.507 | 83.431 |
| 109 | 100.000 | 100.000 | 97.489 | 91.722 | 83.751 |
| 110 | 100.000 | 100.000 | 97.630 | 92.022 | 84.174 |
| 111 | 100.000 | 100.000 | 97.724 | 92.220 | 84.455 |
| 112 | 100.000 | 100.000 | 97.748 | 92.285 | 84.566 |
| 113 | 100.000 | 100.000 | 97.794 | 92.386 | 84.716 |
| 114 | 100.000 | 100.000 | 97.888 | 92.590 | 85.012 |
| 115 | 100.000 | 100.000 | 97.912 | 92.649 | 85.112 |
| 116 | 100.000 | 100.000 | 97.959 | 92.762 | 85.289 |
| 117 | 100.000 | 100.000 | 98.053 | 92.955 | 85.558 |
| 118 | 100.000 | 100.000 | 98.076 | 93.025 | 85.682 |
| 119 | 100.000 | 100.000 | 98.170 | 93.228 | 85.977 |
| 120 | 100.000 | 100.000 | 98.264 | 93.421 | 86.246 |
| 121 | 100.000 | 100.000 | 98.358 | 93.625 | 86.534 |
| 122 | 100.000 | 100.000 | 98.381 | 93.738 | 86.761 |
| 123 | 100.000 | 100.000 | 98.405 | 93.797 | 86.859 |
| 124 | 100.000 | 100.000 | 98.451 | 93.898 | 87.009 |
| 125 | 100.000 | 100.000 | 98.522 | 94.054 | 87.239 |
| 126 | 100.000 | 100.000 | 98.710 | 94.429 | 87.740 |
| 127 | 100.000 | 100.000 | 98.756 | 94.531 | 87.889 |
| 128 | 100.000 | 100.000 | 98.850 | 94.724 | 88.158 |
| 129 | 100.000 | 100.000 | 98.921 | 94.874 | 88.374 |
| 130 | 100.000 | 100.000 | 98.968 | 94.976 | 88.524 |
| 131 | 100.000 | 100.000 | 99.085 | 95.233 | 88.895 |
| 132 | 100.000 | 100.000 | 99.108 | 95.303 | 89.019 |
| 133 | 100.000 | 100.000 | 99.179 | 95.459 | 89.249 |
| 134 | 100.000 | 100.000 | 99.202 | 95.512 | 89.332 |
| 135 | 100.000 | 100.000 | 99.249 | 95.646 | 89.563 |
| 136 | 100.000 | 100.000 | 99.273 | 95.711 | 89.674 |
| 137 | 100.000 | 100.000 | 99.320 | 95.823 | 89.852 |
| 138 | 100.000 | 100.000 | 99.413 | 96.016 | 90.120 |
| 139 | 100.000 | 100.000 | 99.460 | 96.129 | 90.297 |
| 140 | 100.000 | 100.000 | 99.484 | 96.193 | 90.410 |
| 141 | 100.000 | 100.000 | 99.578 | 96.397 | 90.706 |
| 142 | 100.000 | 100.000 | 99.648 | 96.552 | 90.935 |
| 143 | 100.000 | 100.000 | 99.695 | 96.708 | 91.213 |
| 144 | 100.000 | 100.000 | 99.742 | 96.810 | 91.362 |
| 145 | 100.000 | 100.000 | 99.812 | 96.981 | 91.632 |
| 146 | 100.000 | 100.000 | 99.883 | 97.132 | 91.849 |
| 147 | 100.000 | 100.000 | 99.906 | 97.201 | 91.974 |
| 148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |
grep
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
| 3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
| 4 | 73.810 | 42.640 | 24.707 | 14.911 | 9.448 |
| 5 | 80.952 | 49.915 | 29.986 | 18.428 | 11.768 |
| 6 | 83.333 | 54.146 | 33.904 | 21.447 | 13.938 |
| 7 | 85.714 | 57.868 | 37.424 | 24.251 | 16.014 |
| 8 | 92.857 | 63.621 | 41.600 | 27.216 | 18.108 |
| 9 | 95.238 | 67.343 | 45.002 | 29.891 | 20.097 |
| 10 | 95.238 | 69.543 | 47.701 | 32.293 | 21.996 |
| 11 | 100.000 | 73.942 | 51.103 | 34.813 | 23.846 |
| 12 | 100.000 | 76.142 | 53.754 | 37.156 | 25.699 |
| 13 | 100.000 | 78.173 | 56.288 | 39.456 | 27.550 |
| 14 | 100.000 | 80.203 | 58.799 | 41.735 | 29.390 |
| 15 | 100.000 | 81.557 | 60.770 | 43.697 | 31.067 |
| 16 | 100.000 | 82.910 | 62.694 | 45.611 | 32.714 |
| 17 | 100.000 | 84.095 | 64.313 | 47.236 | 34.155 |
| 18 | 100.000 | 85.448 | 66.002 | 48.812 | 35.486 |
| 19 | 100.000 | 86.464 | 67.480 | 50.362 | 36.899 |
| 20 | 100.000 | 87.479 | 68.935 | 51.863 | 38.255 |
| 21 | 100.000 | 88.325 | 70.272 | 53.316 | 39.605 |
| 22 | 100.000 | 89.171 | 71.563 | 54.699 | 40.888 |
| 23 | 100.000 | 90.017 | 72.853 | 56.104 | 42.208 |
| 24 | 100.000 | 90.863 | 74.097 | 57.439 | 43.462 |
| 25 | 100.000 | 91.709 | 75.340 | 58.758 | 44.688 |
| 26 | 100.000 | 92.386 | 76.466 | 60.008 | 45.862 |
| 27 | 100.000 | 93.063 | 77.546 | 61.209 | 47.014 |
| 28 | 100.000 | 93.739 | 78.531 | 62.286 | 48.065 |
| 29 | 100.000 | 94.247 | 79.423 | 63.326 | 49.089 |
| 30 | 100.000 | 94.247 | 80.033 | 64.350 | 50.234 |
| 31 | 100.000 | 94.755 | 80.924 | 65.412 | 51.303 |
| 32 | 100.000 | 95.262 | 81.793 | 66.420 | 52.299 |
| 33 | 100.000 | 95.431 | 82.450 | 67.374 | 53.331 |
| 34 | 100.000 | 95.939 | 83.294 | 68.361 | 54.324 |
| 35 | 100.000 | 96.447 | 84.092 | 69.283 | 55.260 |
| 36 | 100.000 | 96.954 | 84.866 | 70.146 | 56.113 |
| 37 | 100.000 | 97.293 | 85.547 | 71.004 | 57.006 |
| 38 | 100.000 | 97.462 | 86.110 | 71.830 | 57.921 |
| 39 | 100.000 | 97.800 | 86.720 | 72.581 | 58.709 |
| 40 | 100.000 | 97.800 | 87.142 | 73.315 | 59.567 |
| 41 | 100.000 | 98.139 | 87.752 | 74.066 | 60.355 |
| 42 | 100.000 | 98.139 | 88.104 | 74.704 | 61.115 |
| 43 | 100.000 | 98.139 | 88.456 | 75.331 | 61.867 |
| 44 | 100.000 | 98.308 | 88.878 | 75.932 | 62.550 |
| 45 | 100.000 | 98.646 | 89.371 | 76.500 | 63.153 |
| 46 | 100.000 | 98.816 | 89.770 | 77.063 | 63.796 |
| 47 | 100.000 | 98.985 | 90.145 | 77.583 | 64.390 |
| 48 | 100.000 | 99.154 | 90.521 | 78.103 | 64.983 |
| 49 | 100.000 | 99.323 | 90.896 | 78.623 | 65.577 |
| 50 | 100.000 | 99.323 | 91.154 | 79.116 | 66.207 |
| 51 | 100.000 | 99.492 | 91.506 | 79.615 | 66.794 |
| 52 | 100.000 | 99.492 | 91.741 | 80.065 | 67.372 |
| 53 | 100.000 | 99.492 | 91.976 | 80.500 | 67.916 |
| 54 | 100.000 | 99.662 | 92.304 | 80.950 | 68.439 |
| 55 | 100.000 | 99.831 | 92.633 | 81.374 | 68.902 |
| 56 | 100.000 | 99.831 | 92.867 | 81.835 | 69.505 |
| 57 | 100.000 | 99.831 | 93.078 | 82.253 | 70.050 |
| 58 | 100.000 | 99.831 | 93.266 | 82.628 | 70.551 |
| 59 | 100.000 | 99.831 | 93.454 | 82.998 | 71.041 |
| 60 | 100.000 | 99.831 | 93.641 | 83.395 | 71.591 |
| 61 | 100.000 | 99.831 | 93.829 | 83.770 | 72.092 |
| 62 | 100.000 | 99.831 | 94.017 | 84.151 | 72.606 |
| 63 | 100.000 | 100.000 | 94.275 | 84.446 | 72.911 |
| 64 | 100.000 | 100.000 | 94.439 | 84.784 | 73.373 |
| 65 | 100.000 | 100.000 | 94.603 | 85.100 | 73.784 |
| 66 | 100.000 | 100.000 | 94.744 | 85.384 | 74.170 |
| 67 | 100.000 | 100.000 | 94.885 | 85.700 | 74.625 |
| 68 | 100.000 | 100.000 | 95.026 | 85.974 | 74.984 |
| 69 | 100.000 | 100.000 | 95.167 | 86.253 | 75.356 |
| 70 | 100.000 | 100.000 | 95.307 | 86.553 | 75.777 |
| 71 | 100.000 | 100.000 | 95.448 | 86.837 | 76.156 |
| 72 | 100.000 | 100.000 | 95.589 | 87.121 | 76.535 |
| 73 | 100.000 | 100.000 | 95.730 | 87.400 | 76.907 |
| 74 | 100.000 | 100.000 | 95.870 | 87.695 | 77.318 |
| 75 | 100.000 | 100.000 | 96.011 | 88.001 | 77.752 |
| 76 | 100.000 | 100.000 | 96.129 | 88.253 | 78.111 |
| 77 | 100.000 | 100.000 | 96.246 | 88.483 | 78.418 |
| 78 | 100.000 | 100.000 | 96.363 | 88.719 | 78.738 |
| 79 | 100.000 | 100.000 | 96.481 | 88.976 | 79.111 |
| 80 | 100.000 | 100.000 | 96.574 | 89.180 | 79.405 |
| 81 | 100.000 | 100.000 | 96.668 | 89.373 | 79.673 |
| 82 | 100.000 | 100.000 | 96.762 | 89.641 | 80.110 |
| 83 | 100.000 | 100.000 | 96.856 | 89.872 | 80.467 |
| 84 | 100.000 | 100.000 | 96.950 | 90.070 | 80.750 |
| 85 | 100.000 | 100.000 | 97.044 | 90.269 | 81.031 |
| 86 | 100.000 | 100.000 | 97.137 | 90.499 | 81.388 |
| 87 | 100.000 | 100.000 | 97.231 | 90.714 | 81.708 |
| 88 | 100.000 | 100.000 | 97.302 | 90.864 | 81.924 |
| 89 | 100.000 | 100.000 | 97.372 | 91.009 | 82.125 |
| 90 | 100.000 | 100.000 | 97.443 | 91.175 | 82.379 |
| 91 | 100.000 | 100.000 | 97.513 | 91.357 | 82.671 |
| 92 | 100.000 | 100.000 | 97.583 | 91.534 | 82.952 |
| 93 | 100.000 | 100.000 | 97.654 | 91.700 | 83.205 |
| 94 | 100.000 | 100.000 | 97.724 | 91.856 | 83.435 |
| 95 | 100.000 | 100.000 | 97.794 | 92.017 | 83.679 |
| 96 | 100.000 | 100.000 | 97.865 | 92.188 | 83.949 |
| 97 | 100.000 | 100.000 | 97.935 | 92.360 | 84.219 |
| 98 | 100.000 | 100.000 | 98.006 | 92.510 | 84.435 |
| 99 | 100.000 | 100.000 | 98.076 | 92.660 | 84.652 |
| 100 | 100.000 | 100.000 | 98.146 | 92.815 | 84.881 |
| 101 | 100.000 | 100.000 | 98.217 | 92.966 | 85.097 |
| 102 | 100.000 | 100.000 | 98.287 | 93.148 | 85.391 |
| 103 | 100.000 | 100.000 | 98.358 | 93.346 | 85.722 |
| 104 | 100.000 | 100.000 | 98.428 | 93.502 | 85.952 |
| 105 | 100.000 | 100.000 | 98.475 | 93.604 | 86.101 |
| 106 | 100.000 | 100.000 | 98.522 | 93.722 | 86.294 |
| 107 | 100.000 | 100.000 | 98.569 | 93.829 | 86.458 |
| 108 | 100.000 | 100.000 | 98.616 | 93.931 | 86.608 |
| 109 | 100.000 | 100.000 | 98.663 | 94.049 | 86.798 |
| 110 | 100.000 | 100.000 | 98.710 | 94.156 | 86.963 |
| 111 | 100.000 | 100.000 | 98.756 | 94.268 | 87.141 |
| 112 | 100.000 | 100.000 | 98.803 | 94.381 | 87.322 |
| 113 | 100.000 | 100.000 | 98.850 | 94.504 | 87.527 |
| 114 | 100.000 | 100.000 | 98.897 | 94.606 | 87.677 |
| 115 | 100.000 | 100.000 | 98.944 | 94.708 | 87.827 |
| 116 | 100.000 | 100.000 | 98.991 | 94.810 | 87.976 |
| 117 | 100.000 | 100.000 | 99.038 | 94.923 | 88.154 |
| 118 | 100.000 | 100.000 | 99.085 | 95.030 | 88.317 |
| 119 | 100.000 | 100.000 | 99.132 | 95.137 | 88.481 |
| 120 | 100.000 | 100.000 | 99.179 | 95.239 | 88.631 |
| 121 | 100.000 | 100.000 | 99.226 | 95.394 | 88.908 |
| 122 | 100.000 | 100.000 | 99.273 | 95.518 | 89.110 |
| 123 | 100.000 | 100.000 | 99.320 | 95.636 | 89.302 |
| 124 | 100.000 | 100.000 | 99.366 | 95.737 | 89.452 |
| 125 | 100.000 | 100.000 | 99.413 | 95.861 | 89.654 |
| 126 | 100.000 | 100.000 | 99.460 | 95.968 | 89.818 |
| 127 | 100.000 | 100.000 | 99.507 | 96.086 | 90.007 |
| 128 | 100.000 | 100.000 | 99.531 | 96.156 | 90.131 |
| 129 | 100.000 | 100.000 | 99.554 | 96.225 | 90.256 |
| 130 | 100.000 | 100.000 | 99.578 | 96.279 | 90.340 |
| 131 | 100.000 | 100.000 | 99.601 | 96.343 | 90.454 |
| 132 | 100.000 | 100.000 | 99.625 | 96.429 | 90.621 |
| 133 | 100.000 | 100.000 | 99.648 | 96.488 | 90.719 |
| 134 | 100.000 | 100.000 | 99.672 | 96.542 | 90.802 |
| 135 | 100.000 | 100.000 | 99.695 | 96.601 | 90.900 |
| 136 | 100.000 | 100.000 | 99.718 | 96.660 | 90.998 |
| 137 | 100.000 | 100.000 | 99.742 | 96.719 | 91.096 |
| 138 | 100.000 | 100.000 | 99.765 | 96.772 | 91.179 |
| 139 | 100.000 | 100.000 | 99.789 | 96.837 | 91.290 |
| 140 | 100.000 | 100.000 | 99.812 | 96.890 | 91.373 |
| 141 | 100.000 | 100.000 | 99.836 | 96.955 | 91.486 |
| 142 | 100.000 | 100.000 | 99.859 | 97.014 | 91.586 |
| 143 | 100.000 | 100.000 | 99.883 | 97.110 | 91.778 |
| 144 | 100.000 | 100.000 | 99.906 | 97.164 | 91.862 |
| 145 | 100.000 | 100.000 | 99.930 | 97.223 | 91.960 |
| 146 | 100.000 | 100.000 | 99.953 | 97.282 | 92.058 |
| 147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
| 148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |
grep
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
| 3 | 57.143 | 31.641 | 18.254 | 11.072 | 7.053 |
| 4 | 66.667 | 39.425 | 23.557 | 14.557 | 9.357 |
| 5 | 76.190 | 47.208 | 28.860 | 18.042 | 11.661 |
| 6 | 80.952 | 52.623 | 33.224 | 21.200 | 13.865 |
| 7 | 85.714 | 57.530 | 37.166 | 24.122 | 15.966 |
| 8 | 88.095 | 61.083 | 40.544 | 26.851 | 18.013 |
| 9 | 90.476 | 64.129 | 43.548 | 29.382 | 19.970 |
| 10 | 90.476 | 66.328 | 46.293 | 31.832 | 21.898 |
| 11 | 95.238 | 71.066 | 50.047 | 34.615 | 23.916 |
| 12 | 97.619 | 74.281 | 53.050 | 37.076 | 25.813 |
| 13 | 97.619 | 76.142 | 55.397 | 39.237 | 27.581 |
| 14 | 97.619 | 77.834 | 57.626 | 41.344 | 29.330 |
| 15 | 100.000 | 80.711 | 60.300 | 43.590 | 31.115 |
| 16 | 100.000 | 82.403 | 62.412 | 45.542 | 32.733 |
| 17 | 100.000 | 83.249 | 63.820 | 47.113 | 34.201 |
| 18 | 100.000 | 84.772 | 65.767 | 48.952 | 35.755 |
| 19 | 100.000 | 85.956 | 67.433 | 50.630 | 37.234 |
| 20 | 100.000 | 87.310 | 69.193 | 52.319 | 38.684 |
| 21 | 100.000 | 88.494 | 70.765 | 53.868 | 40.050 |
| 22 | 100.000 | 89.171 | 71.938 | 55.209 | 41.336 |
| 23 | 100.000 | 90.017 | 73.205 | 56.565 | 42.600 |
| 24 | 100.000 | 90.863 | 74.449 | 57.884 | 43.828 |
| 25 | 100.000 | 91.540 | 75.575 | 59.134 | 45.002 |
| 26 | 100.000 | 92.047 | 76.513 | 60.265 | 46.138 |
| 27 | 100.000 | 92.893 | 77.710 | 61.498 | 47.264 |
| 28 | 100.000 | 93.232 | 78.461 | 62.474 | 48.287 |
| 29 | 100.000 | 93.570 | 79.235 | 63.471 | 49.309 |
| 30 | 100.000 | 94.078 | 80.103 | 64.490 | 50.323 |
| 31 | 100.000 | 94.416 | 80.807 | 65.412 | 51.310 |
| 32 | 100.000 | 94.755 | 81.511 | 66.334 | 52.283 |
| 33 | 100.000 | 95.093 | 82.168 | 67.192 | 53.213 |
| 34 | 100.000 | 95.262 | 82.708 | 67.986 | 54.102 |
| 35 | 100.000 | 95.431 | 83.200 | 68.709 | 54.927 |
| 36 | 100.000 | 95.939 | 83.951 | 69.551 | 55.772 |
| 37 | 100.000 | 96.447 | 84.702 | 70.388 | 56.608 |
| 38 | 100.000 | 96.447 | 85.031 | 70.994 | 57.353 |
| 39 | 100.000 | 96.616 | 85.500 | 71.664 | 58.104 |
| 40 | 100.000 | 96.785 | 85.969 | 72.334 | 58.853 |
| 41 | 100.000 | 97.124 | 86.579 | 73.079 | 59.628 |
| 42 | 100.000 | 97.293 | 87.025 | 73.728 | 60.371 |
| 43 | 100.000 | 97.462 | 87.471 | 74.366 | 61.094 |
| 44 | 100.000 | 97.800 | 88.034 | 75.058 | 61.840 |
| 45 | 100.000 | 97.800 | 88.315 | 75.594 | 62.511 |
| 46 | 100.000 | 97.970 | 88.738 | 76.200 | 63.206 |
| 47 | 100.000 | 98.139 | 89.137 | 76.779 | 63.879 |
| 48 | 100.000 | 98.646 | 89.864 | 77.545 | 64.600 |
| 49 | 100.000 | 98.816 | 90.263 | 78.114 | 65.257 |
| 50 | 100.000 | 98.985 | 90.662 | 78.671 | 65.889 |
| 51 | 100.000 | 99.154 | 91.061 | 79.229 | 66.522 |
| 52 | 100.000 | 99.154 | 91.295 | 79.685 | 67.114 |
| 53 | 100.000 | 99.154 | 91.530 | 80.140 | 67.705 |
| 54 | 100.000 | 99.154 | 91.741 | 80.559 | 68.251 |
| 55 | 100.000 | 99.154 | 91.952 | 80.972 | 68.791 |
| 56 | 100.000 | 99.154 | 92.140 | 81.363 | 69.327 |
| 57 | 100.000 | 99.154 | 92.281 | 81.701 | 69.830 |
| 58 | 100.000 | 99.154 | 92.468 | 82.076 | 70.331 |
| 59 | 100.000 | 99.154 | 92.656 | 82.451 | 70.832 |
| 60 | 100.000 | 99.154 | 92.820 | 82.800 | 71.319 |
| 61 | 100.000 | 99.154 | 93.031 | 83.191 | 71.807 |
| 62 | 100.000 | 99.323 | 93.360 | 83.631 | 72.302 |
| 63 | 100.000 | 99.492 | 93.688 | 84.071 | 72.805 |
| 64 | 100.000 | 99.492 | 93.853 | 84.414 | 73.279 |
| 65 | 100.000 | 99.492 | 94.017 | 84.751 | 73.741 |
| 66 | 100.000 | 99.492 | 94.181 | 85.089 | 74.203 |
| 67 | 100.000 | 99.662 | 94.486 | 85.497 | 74.671 |
| 68 | 100.000 | 99.662 | 94.627 | 85.808 | 75.119 |
| 69 | 100.000 | 99.662 | 94.791 | 86.135 | 75.557 |
| 70 | 100.000 | 99.662 | 94.908 | 86.408 | 75.964 |
| 71 | 100.000 | 99.662 | 95.073 | 86.725 | 76.374 |
| 72 | 100.000 | 99.662 | 95.214 | 87.019 | 76.785 |
| 73 | 100.000 | 99.662 | 95.354 | 87.314 | 77.197 |
| 74 | 100.000 | 99.662 | 95.495 | 87.604 | 77.595 |
| 75 | 100.000 | 99.831 | 95.800 | 87.990 | 78.018 |
| 76 | 100.000 | 100.000 | 96.105 | 88.376 | 78.442 |
| 77 | 100.000 | 100.000 | 96.246 | 88.660 | 78.828 |
| 78 | 100.000 | 100.000 | 96.340 | 88.896 | 79.193 |
| 79 | 100.000 | 100.000 | 96.457 | 89.148 | 79.553 |
| 80 | 100.000 | 100.000 | 96.551 | 89.368 | 79.888 |
| 81 | 100.000 | 100.000 | 96.668 | 89.609 | 80.221 |
| 82 | 100.000 | 100.000 | 96.786 | 89.850 | 80.556 |
| 83 | 100.000 | 100.000 | 96.856 | 90.038 | 80.861 |
| 84 | 100.000 | 100.000 | 96.950 | 90.247 | 81.169 |
| 85 | 100.000 | 100.000 | 97.044 | 90.451 | 81.458 |
| 86 | 100.000 | 100.000 | 97.137 | 90.655 | 81.750 |
| 87 | 100.000 | 100.000 | 97.231 | 90.858 | 82.048 |
| 88 | 100.000 | 100.000 | 97.325 | 91.057 | 82.329 |
| 89 | 100.000 | 100.000 | 97.419 | 91.250 | 82.597 |
| 90 | 100.000 | 100.000 | 97.489 | 91.421 | 82.865 |
| 91 | 100.000 | 100.000 | 97.560 | 91.593 | 83.133 |
| 92 | 100.000 | 100.000 | 97.630 | 91.759 | 83.390 |
| 93 | 100.000 | 100.000 | 97.701 | 91.925 | 83.647 |
| 94 | 100.000 | 100.000 | 97.771 | 92.086 | 83.890 |
| 95 | 100.000 | 100.000 | 97.841 | 92.247 | 84.130 |
| 96 | 100.000 | 100.000 | 97.912 | 92.403 | 84.359 |
| 97 | 100.000 | 100.000 | 97.982 | 92.558 | 84.592 |
| 98 | 100.000 | 100.000 | 98.053 | 92.714 | 84.822 |
| 99 | 100.000 | 100.000 | 98.099 | 92.842 | 85.040 |
| 100 | 100.000 | 100.000 | 98.170 | 92.992 | 85.258 |
| 101 | 100.000 | 100.000 | 98.240 | 93.142 | 85.474 |
| 102 | 100.000 | 100.000 | 98.311 | 93.293 | 85.691 |
| 103 | 100.000 | 100.000 | 98.381 | 93.443 | 85.907 |
| 104 | 100.000 | 100.000 | 98.451 | 93.587 | 86.109 |
| 105 | 100.000 | 100.000 | 98.498 | 93.711 | 86.310 |
| 106 | 100.000 | 100.000 | 98.545 | 93.834 | 86.516 |
| 107 | 100.000 | 100.000 | 98.592 | 93.957 | 86.717 |
| 108 | 100.000 | 100.000 | 98.639 | 94.075 | 86.906 |
| 109 | 100.000 | 100.000 | 98.663 | 94.172 | 87.098 |
| 110 | 100.000 | 100.000 | 98.710 | 94.284 | 87.276 |
| 111 | 100.000 | 100.000 | 98.756 | 94.397 | 87.455 |
| 112 | 100.000 | 100.000 | 98.803 | 94.510 | 87.632 |
| 113 | 100.000 | 100.000 | 98.850 | 94.622 | 87.810 |
| 114 | 100.000 | 100.000 | 98.897 | 94.735 | 87.989 |
| 115 | 100.000 | 100.000 | 98.944 | 94.847 | 88.167 |
| 116 | 100.000 | 100.000 | 98.991 | 94.960 | 88.344 |
| 117 | 100.000 | 100.000 | 99.015 | 95.046 | 88.513 |
| 118 | 100.000 | 100.000 | 99.061 | 95.153 | 88.679 |
| 119 | 100.000 | 100.000 | 99.108 | 95.260 | 88.844 |
| 120 | 100.000 | 100.000 | 99.155 | 95.362 | 88.993 |
| 121 | 100.000 | 100.000 | 99.202 | 95.464 | 89.143 |
| 122 | 100.000 | 100.000 | 99.249 | 95.566 | 89.293 |
| 123 | 100.000 | 100.000 | 99.296 | 95.668 | 89.443 |
| 124 | 100.000 | 100.000 | 99.343 | 95.770 | 89.593 |
| 125 | 100.000 | 100.000 | 99.390 | 95.872 | 89.742 |
| 126 | 100.000 | 100.000 | 99.437 | 95.973 | 89.892 |
| 127 | 100.000 | 100.000 | 99.484 | 96.075 | 90.042 |
| 128 | 100.000 | 100.000 | 99.531 | 96.177 | 90.194 |
| 129 | 100.000 | 100.000 | 99.554 | 96.252 | 90.332 |
| 130 | 100.000 | 100.000 | 99.578 | 96.327 | 90.471 |
| 131 | 100.000 | 100.000 | 99.601 | 96.397 | 90.597 |
| 132 | 100.000 | 100.000 | 99.625 | 96.467 | 90.722 |
| 133 | 100.000 | 100.000 | 99.648 | 96.536 | 90.846 |
| 134 | 100.000 | 100.000 | 99.672 | 96.601 | 90.957 |
| 135 | 100.000 | 100.000 | 99.695 | 96.665 | 91.068 |
| 136 | 100.000 | 100.000 | 99.718 | 96.729 | 91.179 |
| 137 | 100.000 | 100.000 | 99.742 | 96.794 | 91.292 |
| 138 | 100.000 | 100.000 | 99.765 | 96.858 | 91.405 |
| 139 | 100.000 | 100.000 | 99.789 | 96.922 | 91.516 |
| 140 | 100.000 | 100.000 | 99.812 | 96.981 | 91.614 |
| 141 | 100.000 | 100.000 | 99.836 | 97.035 | 91.697 |
| 142 | 100.000 | 100.000 | 99.859 | 97.089 | 91.780 |
| 143 | 100.000 | 100.000 | 99.883 | 97.142 | 91.863 |
| 144 | 100.000 | 100.000 | 99.906 | 97.196 | 91.947 |
| 145 | 100.000 | 100.000 | 99.930 | 97.249 | 92.030 |
| 146 | 100.000 | 100.000 | 99.953 | 97.303 | 92.113 |
| 147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
| 148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |
grep
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 42.857 | 21.658 | 12.107 | 7.292 | 4.656 |
| 3 | 52.381 | 29.272 | 17.269 | 10.696 | 6.923 |
| 4 | 59.524 | 36.041 | 22.220 | 14.074 | 9.200 |
| 5 | 71.429 | 44.501 | 27.616 | 17.533 | 11.476 |
| 6 | 78.571 | 51.100 | 32.426 | 20.830 | 13.716 |
| 7 | 80.952 | 56.176 | 36.861 | 24.052 | 15.944 |
| 8 | 85.714 | 61.421 | 41.107 | 27.157 | 18.130 |
| 9 | 90.476 | 66.159 | 44.932 | 30.025 | 20.212 |
| 10 | 95.238 | 70.728 | 48.545 | 32.727 | 22.192 |
| 11 | 95.238 | 73.096 | 51.337 | 35.162 | 24.099 |
| 12 | 97.619 | 76.311 | 54.411 | 37.698 | 26.044 |
| 13 | 100.000 | 79.188 | 57.109 | 39.966 | 27.840 |
| 14 | 100.000 | 80.372 | 58.893 | 41.810 | 29.467 |
| 15 | 100.000 | 81.726 | 60.863 | 43.772 | 31.144 |
| 16 | 100.000 | 83.080 | 62.740 | 45.633 | 32.755 |
| 17 | 100.000 | 84.095 | 64.336 | 47.327 | 34.283 |
| 18 | 100.000 | 85.618 | 66.283 | 49.166 | 35.836 |
| 19 | 100.000 | 86.633 | 67.785 | 50.732 | 37.251 |
| 20 | 100.000 | 87.648 | 69.310 | 52.324 | 38.684 |
| 21 | 100.000 | 88.325 | 70.507 | 53.697 | 39.997 |
| 22 | 100.000 | 89.509 | 72.055 | 55.214 | 41.336 |
| 23 | 100.000 | 90.186 | 73.158 | 56.453 | 42.526 |
| 24 | 100.000 | 90.694 | 74.097 | 57.579 | 43.653 |
| 25 | 100.000 | 91.371 | 75.153 | 58.748 | 44.777 |
| 26 | 100.000 | 92.047 | 76.208 | 59.916 | 45.901 |
| 27 | 100.000 | 92.555 | 77.100 | 60.978 | 46.970 |
| 28 | 100.000 | 92.724 | 77.687 | 61.852 | 47.946 |
| 29 | 100.000 | 93.232 | 78.578 | 62.903 | 48.995 |
| 30 | 100.000 | 93.739 | 79.446 | 63.922 | 50.016 |
| 31 | 100.000 | 94.078 | 80.150 | 64.828 | 50.972 |
| 32 | 100.000 | 94.585 | 80.948 | 65.750 | 51.907 |
| 33 | 100.000 | 94.755 | 81.534 | 66.608 | 52.847 |
| 34 | 100.000 | 94.755 | 81.863 | 67.230 | 53.627 |
| 35 | 100.000 | 95.431 | 82.872 | 68.313 | 54.650 |
| 36 | 100.000 | 95.601 | 83.365 | 69.037 | 55.471 |
| 37 | 100.000 | 95.939 | 84.045 | 69.894 | 56.364 |
| 38 | 100.000 | 95.939 | 84.374 | 70.500 | 57.109 |
| 39 | 100.000 | 96.277 | 85.007 | 71.294 | 57.947 |
| 40 | 100.000 | 96.785 | 85.805 | 72.195 | 58.838 |
| 41 | 100.000 | 97.124 | 86.415 | 72.951 | 59.637 |
| 42 | 100.000 | 97.631 | 87.213 | 73.851 | 60.525 |
| 43 | 100.000 | 97.800 | 87.635 | 74.468 | 61.240 |
| 44 | 100.000 | 97.970 | 88.081 | 75.111 | 61.975 |
| 45 | 100.000 | 97.970 | 88.339 | 75.610 | 62.616 |
| 46 | 100.000 | 97.970 | 88.620 | 76.135 | 63.275 |
| 47 | 100.000 | 97.970 | 88.902 | 76.671 | 63.946 |
| 48 | 100.000 | 98.139 | 89.301 | 77.245 | 64.613 |
| 49 | 100.000 | 98.308 | 89.700 | 77.814 | 65.270 |
| 50 | 100.000 | 98.477 | 90.122 | 78.409 | 65.941 |
| 51 | 100.000 | 98.646 | 90.521 | 78.972 | 66.585 |
| 52 | 100.000 | 98.646 | 90.756 | 79.422 | 67.163 |
| 53 | 100.000 | 98.646 | 90.967 | 79.840 | 67.716 |
| 54 | 100.000 | 98.816 | 91.366 | 80.398 | 68.349 |
| 55 | 100.000 | 98.816 | 91.577 | 80.811 | 68.891 |
| 56 | 100.000 | 98.816 | 91.788 | 81.224 | 69.432 |
| 57 | 100.000 | 98.816 | 91.976 | 81.610 | 69.954 |
| 58 | 100.000 | 98.816 | 92.163 | 81.985 | 70.455 |
| 59 | 100.000 | 98.816 | 92.328 | 82.328 | 70.928 |
| 60 | 100.000 | 98.985 | 92.679 | 82.811 | 71.483 |
| 61 | 100.000 | 98.985 | 92.867 | 83.181 | 71.973 |
| 62 | 100.000 | 98.985 | 93.055 | 83.545 | 72.454 |
| 63 | 100.000 | 99.154 | 93.360 | 83.958 | 72.942 |
| 64 | 100.000 | 99.154 | 93.501 | 84.258 | 73.366 |
| 65 | 100.000 | 99.323 | 93.829 | 84.698 | 73.869 |
| 66 | 100.000 | 99.323 | 93.993 | 85.025 | 74.307 |
| 67 | 100.000 | 99.492 | 94.298 | 85.432 | 74.782 |
| 68 | 100.000 | 99.492 | 94.439 | 85.727 | 75.193 |
| 69 | 100.000 | 99.492 | 94.580 | 86.022 | 75.603 |
| 70 | 100.000 | 99.492 | 94.721 | 86.317 | 76.014 |
| 71 | 100.000 | 99.492 | 94.862 | 86.607 | 76.413 |
| 72 | 100.000 | 99.662 | 95.190 | 87.046 | 76.909 |
| 73 | 100.000 | 99.662 | 95.331 | 87.330 | 77.297 |
| 74 | 100.000 | 99.662 | 95.495 | 87.652 | 77.719 |
| 75 | 100.000 | 99.662 | 95.612 | 87.904 | 78.079 |
| 76 | 100.000 | 99.662 | 95.730 | 88.151 | 78.427 |
| 77 | 100.000 | 99.662 | 95.824 | 88.365 | 78.747 |
| 78 | 100.000 | 99.662 | 95.941 | 88.612 | 79.093 |
| 79 | 100.000 | 99.662 | 96.035 | 88.826 | 79.413 |
| 80 | 100.000 | 99.662 | 96.176 | 89.111 | 79.792 |
| 81 | 100.000 | 99.662 | 96.269 | 89.320 | 80.102 |
| 82 | 100.000 | 99.662 | 96.387 | 89.561 | 80.437 |
| 83 | 100.000 | 99.662 | 96.481 | 89.770 | 80.746 |
| 84 | 100.000 | 99.662 | 96.574 | 89.979 | 81.053 |
| 85 | 100.000 | 99.662 | 96.668 | 90.183 | 81.349 |
| 86 | 100.000 | 99.662 | 96.762 | 90.387 | 81.645 |
| 87 | 100.000 | 99.662 | 96.856 | 90.596 | 81.948 |
| 88 | 100.000 | 99.662 | 96.950 | 90.799 | 82.244 |
| 89 | 100.000 | 99.662 | 97.044 | 91.003 | 82.540 |
| 90 | 100.000 | 99.662 | 97.091 | 91.143 | 82.780 |
| 91 | 100.000 | 99.831 | 97.372 | 91.486 | 83.152 |
| 92 | 100.000 | 99.831 | 97.466 | 91.689 | 83.440 |
| 93 | 100.000 | 99.831 | 97.489 | 91.786 | 83.634 |
| 94 | 100.000 | 99.831 | 97.536 | 91.915 | 83.853 |
| 95 | 100.000 | 99.831 | 97.560 | 92.011 | 84.045 |
| 96 | 100.000 | 99.831 | 97.630 | 92.172 | 84.289 |
| 97 | 100.000 | 99.831 | 97.724 | 92.365 | 84.557 |
| 98 | 100.000 | 99.831 | 97.794 | 92.521 | 84.790 |
| 99 | 100.000 | 99.831 | 97.841 | 92.644 | 84.995 |
| 100 | 100.000 | 99.831 | 97.912 | 92.799 | 85.225 |
| 101 | 100.000 | 99.831 | 97.982 | 92.955 | 85.454 |
| 102 | 100.000 | 99.831 | 98.053 | 93.110 | 85.683 |
| 103 | 100.000 | 99.831 | 98.099 | 93.234 | 85.885 |
| 104 | 100.000 | 99.831 | 98.146 | 93.357 | 86.087 |
| 105 | 100.000 | 99.831 | 98.193 | 93.475 | 86.279 |
| 106 | 100.000 | 99.831 | 98.217 | 93.561 | 86.445 |
| 107 | 100.000 | 99.831 | 98.287 | 93.711 | 86.662 |
| 108 | 100.000 | 99.831 | 98.358 | 93.861 | 86.878 |
| 109 | 100.000 | 99.831 | 98.428 | 94.011 | 87.094 |
| 110 | 100.000 | 100.000 | 98.686 | 94.306 | 87.399 |
| 111 | 100.000 | 100.000 | 98.756 | 94.456 | 87.616 |
| 112 | 100.000 | 100.000 | 98.803 | 94.574 | 87.804 |
| 113 | 100.000 | 100.000 | 98.827 | 94.654 | 87.958 |
| 114 | 100.000 | 100.000 | 98.874 | 94.767 | 88.135 |
| 115 | 100.000 | 100.000 | 98.921 | 94.880 | 88.313 |
| 116 | 100.000 | 100.000 | 98.968 | 94.992 | 88.490 |
| 117 | 100.000 | 100.000 | 99.038 | 95.137 | 88.692 |
| 118 | 100.000 | 100.000 | 99.085 | 95.244 | 88.858 |
| 119 | 100.000 | 100.000 | 99.132 | 95.351 | 89.023 |
| 120 | 100.000 | 100.000 | 99.179 | 95.453 | 89.175 |
| 121 | 100.000 | 100.000 | 99.202 | 95.523 | 89.300 |
| 122 | 100.000 | 100.000 | 99.249 | 95.625 | 89.450 |
| 123 | 100.000 | 100.000 | 99.273 | 95.695 | 89.574 |
| 124 | 100.000 | 100.000 | 99.320 | 95.796 | 89.724 |
| 125 | 100.000 | 100.000 | 99.366 | 95.898 | 89.874 |
| 126 | 100.000 | 100.000 | 99.413 | 96.000 | 90.023 |
| 127 | 100.000 | 100.000 | 99.460 | 96.102 | 90.173 |
| 128 | 100.000 | 100.000 | 99.507 | 96.204 | 90.323 |
| 129 | 100.000 | 100.000 | 99.531 | 96.274 | 90.447 |
| 130 | 100.000 | 100.000 | 99.578 | 96.376 | 90.597 |
| 131 | 100.000 | 100.000 | 99.601 | 96.440 | 90.710 |
| 132 | 100.000 | 100.000 | 99.625 | 96.504 | 90.820 |
| 133 | 100.000 | 100.000 | 99.648 | 96.569 | 90.931 |
| 134 | 100.000 | 100.000 | 99.672 | 96.633 | 91.042 |
| 135 | 100.000 | 100.000 | 99.695 | 96.697 | 91.153 |
| 136 | 100.000 | 100.000 | 99.718 | 96.756 | 91.251 |
| 137 | 100.000 | 100.000 | 99.742 | 96.815 | 91.349 |
| 138 | 100.000 | 100.000 | 99.765 | 96.874 | 91.447 |
| 139 | 100.000 | 100.000 | 99.789 | 96.928 | 91.531 |
| 140 | 100.000 | 100.000 | 99.812 | 96.981 | 91.614 |
| 141 | 100.000 | 100.000 | 99.836 | 97.035 | 91.697 |
| 142 | 100.000 | 100.000 | 99.859 | 97.089 | 91.780 |
| 143 | 100.000 | 100.000 | 99.883 | 97.142 | 91.863 |
| 144 | 100.000 | 100.000 | 99.906 | 97.196 | 91.947 |
| 145 | 100.000 | 100.000 | 99.930 | 97.249 | 92.030 |
| 146 | 100.000 | 100.000 | 99.953 | 97.303 | 92.113 |
| 147 | 100.000 | 100.000 | 99.977 | 97.357 | 92.196 |
| 148 | 100.000 | 100.000 | 100.000 | 97.410 | 92.280 |
grep
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 66.667 | 35.533 | 19.521 | 11.426 | 7.134 |
| 4 | 76.190 | 43.486 | 24.941 | 14.964 | 9.457 |
| 5 | 80.952 | 49.577 | 29.775 | 18.342 | 11.743 |
| 6 | 85.714 | 55.668 | 34.608 | 21.720 | 14.028 |
| 7 | 92.857 | 62.098 | 39.301 | 24.964 | 16.249 |
| 8 | 95.238 | 66.836 | 43.524 | 28.100 | 18.451 |
| 9 | 97.619 | 70.389 | 46.715 | 30.594 | 20.310 |
| 10 | 100.000 | 73.435 | 49.648 | 33.049 | 22.218 |
| 11 | 100.000 | 75.635 | 52.229 | 35.301 | 23.994 |
| 12 | 100.000 | 77.496 | 54.364 | 37.151 | 25.464 |
| 13 | 100.000 | 79.357 | 56.617 | 39.183 | 27.119 |
| 14 | 100.000 | 81.049 | 58.611 | 40.952 | 28.552 |
| 15 | 100.000 | 82.572 | 60.676 | 42.947 | 30.237 |
| 16 | 100.000 | 83.926 | 62.694 | 44.957 | 31.943 |
| 17 | 100.000 | 85.279 | 64.735 | 46.989 | 33.661 |
| 18 | 100.000 | 86.464 | 66.448 | 48.737 | 35.207 |
| 19 | 100.000 | 87.648 | 68.067 | 50.319 | 36.566 |
| 20 | 100.000 | 88.663 | 69.451 | 51.702 | 37.796 |
| 21 | 100.000 | 89.509 | 70.695 | 53.016 | 39.013 |
| 22 | 100.000 | 90.355 | 72.102 | 54.560 | 40.437 |
| 23 | 100.000 | 91.201 | 73.322 | 55.836 | 41.615 |
| 24 | 100.000 | 92.047 | 74.542 | 57.107 | 42.780 |
| 25 | 100.000 | 92.893 | 75.809 | 58.426 | 43.974 |
| 26 | 100.000 | 93.570 | 76.795 | 59.466 | 44.945 |
| 27 | 100.000 | 94.247 | 77.945 | 60.742 | 46.138 |
| 28 | 100.000 | 94.755 | 78.672 | 61.509 | 46.859 |
| 29 | 100.000 | 95.262 | 79.587 | 62.619 | 47.983 |
| 30 | 100.000 | 95.770 | 80.432 | 63.616 | 48.986 |
| 31 | 100.000 | 96.108 | 80.995 | 64.286 | 49.683 |
| 32 | 100.000 | 96.447 | 81.699 | 65.181 | 50.613 |
| 33 | 100.000 | 96.785 | 82.426 | 66.131 | 51.617 |
| 34 | 100.000 | 97.124 | 82.942 | 66.715 | 52.207 |
| 35 | 100.000 | 97.462 | 83.458 | 67.294 | 52.786 |
| 36 | 100.000 | 97.800 | 84.186 | 68.222 | 53.746 |
| 37 | 100.000 | 98.139 | 85.007 | 69.267 | 54.798 |
| 38 | 100.000 | 98.477 | 85.687 | 70.141 | 55.726 |
| 39 | 100.000 | 98.816 | 86.204 | 70.720 | 56.305 |
| 40 | 100.000 | 99.154 | 86.861 | 71.546 | 57.163 |
| 41 | 100.000 | 99.323 | 87.166 | 71.932 | 57.586 |
| 42 | 100.000 | 99.492 | 87.494 | 72.361 | 58.062 |
| 43 | 100.000 | 99.662 | 88.175 | 73.379 | 59.179 |
| 44 | 100.000 | 99.831 | 88.503 | 73.803 | 59.641 |
| 45 | 100.000 | 100.000 | 88.832 | 74.243 | 60.136 |
| 46 | 100.000 | 100.000 | 89.043 | 74.650 | 60.665 |
| 47 | 100.000 | 100.000 | 89.230 | 75.015 | 61.142 |
| 48 | 100.000 | 100.000 | 89.559 | 75.631 | 61.904 |
| 49 | 100.000 | 100.000 | 89.747 | 76.007 | 62.405 |
| 50 | 100.000 | 100.000 | 89.840 | 76.205 | 62.687 |
| 51 | 100.000 | 100.000 | 89.958 | 76.436 | 62.994 |
| 52 | 100.000 | 100.000 | 90.075 | 76.671 | 63.313 |
| 53 | 100.000 | 100.000 | 90.521 | 77.449 | 64.223 |
| 54 | 100.000 | 100.000 | 90.756 | 77.948 | 64.900 |
| 55 | 100.000 | 100.000 | 90.943 | 78.312 | 65.377 |
| 56 | 100.000 | 100.000 | 91.389 | 79.090 | 66.287 |
| 57 | 100.000 | 100.000 | 91.436 | 79.197 | 66.452 |
| 58 | 100.000 | 100.000 | 91.600 | 79.524 | 66.890 |
| 59 | 100.000 | 100.000 | 91.835 | 79.969 | 67.458 |
| 60 | 100.000 | 100.000 | 92.046 | 80.355 | 67.935 |
| 61 | 100.000 | 100.000 | 92.140 | 80.569 | 68.251 |
| 62 | 100.000 | 100.000 | 92.210 | 80.714 | 68.452 |
| 63 | 100.000 | 100.000 | 92.351 | 81.020 | 68.887 |
| 64 | 100.000 | 100.000 | 92.398 | 81.127 | 69.052 |
| 65 | 100.000 | 100.000 | 92.445 | 81.250 | 69.253 |
| 66 | 100.000 | 100.000 | 92.445 | 81.283 | 69.336 |
| 67 | 100.000 | 100.000 | 92.586 | 81.577 | 69.747 |
| 68 | 100.000 | 100.000 | 92.633 | 81.690 | 69.924 |
| 69 | 100.000 | 100.000 | 92.679 | 81.803 | 70.102 |
| 70 | 100.000 | 100.000 | 92.726 | 81.910 | 70.265 |
| 71 | 100.000 | 100.000 | 92.773 | 82.012 | 70.414 |
| 72 | 100.000 | 100.000 | 92.820 | 82.114 | 70.564 |
| 73 | 100.000 | 100.000 | 92.938 | 82.371 | 70.934 |
| 74 | 100.000 | 100.000 | 92.985 | 82.489 | 71.125 |
| 75 | 100.000 | 100.000 | 93.055 | 82.660 | 71.393 |
| 76 | 100.000 | 100.000 | 93.196 | 82.955 | 71.788 |
| 77 | 100.000 | 100.000 | 93.407 | 83.374 | 72.334 |
| 78 | 100.000 | 100.000 | 93.430 | 83.438 | 72.445 |
| 79 | 100.000 | 100.000 | 93.618 | 83.813 | 72.942 |
| 80 | 100.000 | 100.000 | 93.712 | 84.006 | 73.210 |
| 81 | 100.000 | 100.000 | 93.993 | 84.553 | 73.917 |
| 82 | 100.000 | 100.000 | 94.040 | 84.687 | 74.142 |
| 83 | 100.000 | 100.000 | 94.064 | 84.757 | 74.268 |
| 84 | 100.000 | 100.000 | 94.087 | 84.821 | 74.379 |
| 85 | 100.000 | 100.000 | 94.158 | 84.966 | 74.581 |
| 86 | 100.000 | 100.000 | 94.181 | 85.030 | 74.692 |
| 87 | 100.000 | 100.000 | 94.369 | 85.373 | 75.117 |
| 88 | 100.000 | 100.000 | 94.369 | 85.406 | 75.200 |
| 89 | 100.000 | 100.000 | 94.510 | 85.690 | 75.587 |
| 90 | 100.000 | 100.000 | 94.627 | 85.920 | 75.894 |
| 91 | 100.000 | 100.000 | 94.674 | 86.070 | 76.160 |
| 92 | 100.000 | 100.000 | 94.721 | 86.172 | 76.310 |
| 93 | 100.000 | 100.000 | 94.862 | 86.456 | 76.696 |
| 94 | 100.000 | 100.000 | 94.932 | 86.633 | 76.970 |
| 95 | 100.000 | 100.000 | 95.026 | 86.826 | 77.238 |
| 96 | 100.000 | 100.000 | 95.096 | 86.977 | 77.454 |
| 97 | 100.000 | 100.000 | 95.214 | 87.245 | 77.850 |
| 98 | 100.000 | 100.000 | 95.354 | 87.550 | 78.285 |
| 99 | 100.000 | 100.000 | 95.354 | 87.582 | 78.372 |
| 100 | 100.000 | 100.000 | 95.378 | 87.641 | 78.470 |
| 101 | 100.000 | 100.000 | 95.495 | 87.883 | 78.801 |
| 102 | 100.000 | 100.000 | 95.589 | 88.086 | 79.096 |
| 103 | 100.000 | 100.000 | 95.659 | 88.231 | 79.298 |
| 104 | 100.000 | 100.000 | 95.706 | 88.333 | 79.448 |
| 105 | 100.000 | 100.000 | 95.753 | 88.446 | 79.625 |
| 106 | 100.000 | 100.000 | 95.894 | 88.719 | 79.984 |
| 107 | 100.000 | 100.000 | 96.176 | 89.255 | 80.655 |
| 108 | 100.000 | 100.000 | 96.269 | 89.448 | 80.927 |
| 109 | 100.000 | 100.000 | 96.340 | 89.604 | 81.156 |
| 110 | 100.000 | 100.000 | 96.340 | 89.636 | 81.247 |
| 111 | 100.000 | 100.000 | 96.340 | 89.652 | 81.293 |
| 112 | 100.000 | 100.000 | 96.387 | 89.754 | 81.443 |
| 113 | 100.000 | 100.000 | 96.481 | 89.958 | 81.735 |
| 114 | 100.000 | 100.000 | 96.481 | 89.968 | 81.769 |
| 115 | 100.000 | 100.000 | 96.527 | 90.076 | 81.933 |
| 116 | 100.000 | 100.000 | 96.692 | 90.397 | 82.355 |
| 117 | 100.000 | 100.000 | 96.762 | 90.569 | 82.621 |
| 118 | 100.000 | 100.000 | 96.786 | 90.622 | 82.704 |
| 119 | 100.000 | 100.000 | 96.809 | 90.676 | 82.788 |
| 120 | 100.000 | 100.000 | 96.809 | 90.698 | 82.847 |
| 121 | 100.000 | 100.000 | 96.856 | 90.799 | 82.996 |
| 122 | 100.000 | 100.000 | 96.903 | 90.901 | 83.146 |
| 123 | 100.000 | 100.000 | 96.950 | 91.014 | 83.324 |
| 124 | 100.000 | 100.000 | 96.997 | 91.121 | 83.488 |
| 125 | 100.000 | 100.000 | 97.067 | 91.293 | 83.756 |
| 126 | 100.000 | 100.000 | 97.114 | 91.400 | 83.921 |
| 127 | 100.000 | 100.000 | 97.114 | 91.421 | 83.984 |
| 128 | 100.000 | 100.000 | 97.137 | 91.507 | 84.145 |
| 129 | 100.000 | 100.000 | 97.161 | 91.598 | 84.319 |
| 130 | 100.000 | 100.000 | 97.161 | 91.609 | 84.350 |
| 131 | 100.000 | 100.000 | 97.161 | 91.641 | 84.433 |
| 132 | 100.000 | 100.000 | 97.255 | 91.850 | 84.731 |
| 133 | 100.000 | 100.000 | 97.255 | 91.893 | 84.838 |
| 134 | 100.000 | 100.000 | 97.302 | 91.995 | 84.988 |
| 135 | 100.000 | 100.000 | 97.325 | 92.054 | 85.086 |
| 136 | 100.000 | 100.000 | 97.396 | 92.204 | 85.302 |
| 137 | 100.000 | 100.000 | 97.396 | 92.210 | 85.319 |
| 138 | 100.000 | 100.000 | 97.396 | 92.285 | 85.502 |
| 139 | 100.000 | 100.000 | 97.396 | 92.295 | 85.534 |
| 140 | 100.000 | 100.000 | 97.419 | 92.376 | 85.687 |
| 141 | 100.000 | 100.000 | 97.419 | 92.392 | 85.731 |
| 142 | 100.000 | 100.000 | 97.513 | 92.596 | 86.024 |
| 143 | 100.000 | 100.000 | 97.536 | 92.660 | 86.136 |
| 144 | 100.000 | 100.000 | 97.583 | 92.762 | 86.286 |
| 145 | 100.000 | 100.000 | 97.607 | 92.815 | 86.369 |
| 146 | 100.000 | 100.000 | 97.630 | 92.869 | 86.453 |
| 147 | 100.000 | 100.000 | 97.654 | 92.923 | 86.536 |
| 148 | 100.000 | 100.000 | 97.654 | 92.923 | 86.540 |
| 149 | 100.000 | 100.000 | 97.654 | 92.928 | 86.556 |
| 150 | 100.000 | 100.000 | 97.701 | 93.030 | 86.706 |
| 151 | 100.000 | 100.000 | 97.748 | 93.142 | 86.884 |
| 152 | 100.000 | 100.000 | 97.771 | 93.207 | 86.998 |
| 153 | 100.000 | 100.000 | 97.818 | 93.330 | 87.200 |
| 154 | 100.000 | 100.000 | 97.818 | 93.346 | 87.246 |
| 155 | 100.000 | 100.000 | 97.888 | 93.496 | 87.462 |
| 156 | 100.000 | 100.000 | 97.888 | 93.496 | 87.466 |
| 157 | 100.000 | 100.000 | 97.912 | 93.577 | 87.616 |
| 158 | 100.000 | 100.000 | 97.912 | 93.582 | 87.632 |
| 159 | 100.000 | 100.000 | 97.912 | 93.614 | 87.721 |
| 160 | 100.000 | 100.000 | 97.912 | 93.641 | 87.797 |
| 161 | 100.000 | 100.000 | 97.912 | 93.646 | 87.814 |
| 162 | 100.000 | 100.000 | 97.959 | 93.748 | 87.963 |
| 163 | 100.000 | 100.000 | 97.959 | 93.759 | 87.995 |
| 164 | 100.000 | 100.000 | 98.006 | 93.861 | 88.145 |
| 165 | 100.000 | 100.000 | 98.029 | 93.920 | 88.243 |
| 166 | 100.000 | 100.000 | 98.053 | 93.995 | 88.383 |
| 167 | 100.000 | 100.000 | 98.099 | 94.102 | 88.552 |
| 168 | 100.000 | 100.000 | 98.123 | 94.172 | 88.681 |
| 169 | 100.000 | 100.000 | 98.123 | 94.193 | 88.740 |
| 170 | 100.000 | 100.000 | 98.170 | 94.295 | 88.890 |
| 171 | 100.000 | 100.000 | 98.170 | 94.376 | 89.090 |
| 172 | 100.000 | 100.000 | 98.193 | 94.429 | 89.173 |
| 173 | 100.000 | 100.000 | 98.287 | 94.633 | 89.465 |
| 174 | 100.000 | 100.000 | 98.358 | 94.778 | 89.667 |
| 175 | 100.000 | 100.000 | 98.498 | 95.073 | 90.077 |
| 176 | 100.000 | 100.000 | 98.592 | 95.260 | 90.332 |
| 177 | 100.000 | 100.000 | 98.592 | 95.266 | 90.349 |
| 178 | 100.000 | 100.000 | 98.639 | 95.368 | 90.499 |
| 179 | 100.000 | 100.000 | 98.639 | 95.373 | 90.515 |
| 180 | 100.000 | 100.000 | 98.686 | 95.528 | 90.789 |
| 181 | 100.000 | 100.000 | 98.733 | 95.630 | 90.939 |
| 182 | 100.000 | 100.000 | 98.756 | 95.684 | 91.022 |
| 183 | 100.000 | 100.000 | 98.780 | 95.737 | 91.105 |
| 184 | 100.000 | 100.000 | 98.780 | 95.754 | 91.155 |
| 185 | 100.000 | 100.000 | 98.803 | 95.813 | 91.253 |
| 186 | 100.000 | 100.000 | 98.827 | 95.866 | 91.336 |
| 187 | 100.000 | 100.000 | 98.850 | 95.952 | 91.499 |
| 188 | 100.000 | 100.000 | 98.850 | 95.963 | 91.531 |
| 189 | 100.000 | 100.000 | 98.874 | 96.032 | 91.658 |
| 190 | 100.000 | 100.000 | 98.874 | 96.048 | 91.704 |
| 191 | 100.000 | 100.000 | 98.921 | 96.150 | 91.856 |
| 192 | 100.000 | 100.000 | 98.921 | 96.161 | 91.887 |
| 193 | 100.000 | 100.000 | 98.921 | 96.172 | 91.919 |
| 194 | 100.000 | 100.000 | 98.921 | 96.193 | 91.980 |
| 195 | 100.000 | 100.000 | 98.921 | 96.220 | 92.056 |
| 196 | 100.000 | 100.000 | 98.944 | 96.274 | 92.139 |
| 197 | 100.000 | 100.000 | 98.944 | 96.290 | 92.185 |
| 198 | 100.000 | 100.000 | 98.968 | 96.349 | 92.283 |
| 199 | 100.000 | 100.000 | 99.015 | 96.451 | 92.433 |
| 200 | 100.000 | 100.000 | 99.015 | 96.472 | 92.492 |
| 201 | 100.000 | 100.000 | 99.061 | 96.574 | 92.642 |
| 202 | 100.000 | 100.000 | 99.085 | 96.628 | 92.725 |
| 203 | 100.000 | 100.000 | 99.108 | 96.681 | 92.808 |
| 204 | 100.000 | 100.000 | 99.108 | 96.729 | 92.932 |
| 205 | 100.000 | 100.000 | 99.108 | 96.740 | 92.964 |
| 206 | 100.000 | 100.000 | 99.108 | 96.756 | 93.008 |
| 207 | 100.000 | 100.000 | 99.108 | 96.767 | 93.040 |
| 208 | 100.000 | 100.000 | 99.108 | 96.772 | 93.056 |
| 209 | 100.000 | 100.000 | 99.108 | 96.778 | 93.073 |
| 210 | 100.000 | 100.000 | 99.108 | 96.794 | 93.119 |
| 211 | 100.000 | 100.000 | 99.132 | 96.847 | 93.204 |
| 212 | 100.000 | 100.000 | 99.155 | 96.901 | 93.287 |
| 213 | 100.000 | 100.000 | 99.155 | 96.906 | 93.306 |
| 214 | 100.000 | 100.000 | 99.155 | 96.928 | 93.365 |
| 215 | 100.000 | 100.000 | 99.155 | 96.933 | 93.384 |
| 216 | 100.000 | 100.000 | 99.179 | 96.992 | 93.482 |
| 217 | 100.000 | 100.000 | 99.179 | 96.997 | 93.498 |
| 218 | 100.000 | 100.000 | 99.226 | 97.121 | 93.700 |
| 219 | 100.000 | 100.000 | 99.226 | 97.121 | 93.707 |
| 220 | 100.000 | 100.000 | 99.226 | 97.126 | 93.724 |
| 221 | 100.000 | 100.000 | 99.320 | 97.341 | 94.044 |
| 222 | 100.000 | 100.000 | 99.320 | 97.357 | 94.088 |
| 223 | 100.000 | 100.000 | 99.343 | 97.410 | 94.171 |
| 224 | 100.000 | 100.000 | 99.343 | 97.416 | 94.188 |
| 225 | 100.000 | 100.000 | 99.343 | 97.432 | 94.238 |
| 226 | 100.000 | 100.000 | 99.343 | 97.432 | 94.242 |
| 227 | 100.000 | 100.000 | 99.390 | 97.560 | 94.460 |
| 228 | 100.000 | 100.000 | 99.413 | 97.614 | 94.543 |
| 229 | 100.000 | 100.000 | 99.460 | 97.721 | 94.706 |
| 230 | 100.000 | 100.000 | 99.460 | 97.732 | 94.737 |
| 231 | 100.000 | 100.000 | 99.460 | 97.737 | 94.756 |
| 232 | 100.000 | 100.000 | 99.460 | 97.743 | 94.772 |
| 233 | 100.000 | 100.000 | 99.554 | 97.946 | 95.068 |
| 234 | 100.000 | 100.000 | 99.554 | 97.963 | 95.114 |
| 235 | 100.000 | 100.000 | 99.578 | 98.016 | 95.198 |
| 236 | 100.000 | 100.000 | 99.578 | 98.038 | 95.257 |
| 237 | 100.000 | 100.000 | 99.578 | 98.043 | 95.273 |
| 238 | 100.000 | 100.000 | 99.578 | 98.043 | 95.275 |
| 239 | 100.000 | 100.000 | 99.578 | 98.064 | 95.334 |
| 240 | 100.000 | 100.000 | 99.578 | 98.086 | 95.394 |
| 241 | 100.000 | 100.000 | 99.578 | 98.091 | 95.410 |
| 242 | 100.000 | 100.000 | 99.578 | 98.097 | 95.427 |
| 243 | 100.000 | 100.000 | 99.578 | 98.107 | 95.458 |
| 244 | 100.000 | 100.000 | 99.601 | 98.161 | 95.542 |
| 245 | 100.000 | 100.000 | 99.625 | 98.215 | 95.625 |
| 246 | 100.000 | 100.000 | 99.625 | 98.215 | 95.627 |
| 247 | 100.000 | 100.000 | 99.672 | 98.327 | 95.804 |
| 248 | 100.000 | 100.000 | 99.695 | 98.397 | 95.926 |
| 249 | 100.000 | 100.000 | 99.695 | 98.408 | 95.961 |
| 250 | 100.000 | 100.000 | 99.695 | 98.418 | 95.993 |
| 251 | 100.000 | 100.000 | 99.695 | 98.429 | 96.024 |
| 252 | 100.000 | 100.000 | 99.695 | 98.450 | 96.087 |
| 253 | 100.000 | 100.000 | 99.742 | 98.552 | 96.237 |
| 254 | 100.000 | 100.000 | 99.742 | 98.558 | 96.253 |
| 255 | 100.000 | 100.000 | 99.742 | 98.568 | 96.289 |
| 256 | 100.000 | 100.000 | 99.742 | 98.601 | 96.376 |
| 257 | 100.000 | 100.000 | 99.742 | 98.606 | 96.392 |
| 258 | 100.000 | 100.000 | 99.765 | 98.660 | 96.475 |
| 259 | 100.000 | 100.000 | 99.765 | 98.665 | 96.492 |
| 260 | 100.000 | 100.000 | 99.812 | 98.778 | 96.670 |
| 261 | 100.000 | 100.000 | 99.812 | 98.788 | 96.701 |
| 262 | 100.000 | 100.000 | 99.836 | 98.853 | 96.812 |
| 263 | 100.000 | 100.000 | 99.836 | 98.858 | 96.829 |
| 264 | 100.000 | 100.000 | 99.836 | 98.858 | 96.836 |
| 265 | 100.000 | 100.000 | 99.859 | 98.912 | 96.919 |
| 266 | 100.000 | 100.000 | 99.859 | 98.917 | 96.936 |
| 267 | 100.000 | 100.000 | 99.859 | 98.944 | 97.010 |
| 268 | 100.000 | 100.000 | 99.859 | 98.944 | 97.012 |
| 269 | 100.000 | 100.000 | 99.859 | 98.954 | 97.043 |
| 270 | 100.000 | 100.000 | 99.859 | 98.965 | 97.075 |
| 271 | 100.000 | 100.000 | 99.906 | 99.067 | 97.224 |
| 272 | 100.000 | 100.000 | 99.930 | 99.121 | 97.308 |
| 273 | 100.000 | 100.000 | 99.930 | 99.185 | 97.470 |
| 274 | 100.000 | 100.000 | 99.930 | 99.185 | 97.474 |
| 275 | 100.000 | 100.000 | 99.930 | 99.190 | 97.491 |
| 276 | 100.000 | 100.000 | 99.930 | 99.196 | 97.507 |
| 277 | 100.000 | 100.000 | 99.930 | 99.217 | 97.568 |
| 278 | 100.000 | 100.000 | 99.930 | 99.223 | 97.585 |
| 279 | 100.000 | 100.000 | 99.930 | 99.233 | 97.616 |
| 280 | 100.000 | 100.000 | 99.930 | 99.239 | 97.633 |
| 281 | 100.000 | 100.000 | 99.930 | 99.239 | 97.635 |
| 282 | 100.000 | 100.000 | 99.930 | 99.244 | 97.653 |
| 283 | 100.000 | 100.000 | 99.930 | 99.260 | 97.698 |
| 284 | 100.000 | 100.000 | 99.930 | 99.287 | 97.770 |
| 285 | 100.000 | 100.000 | 99.930 | 99.308 | 97.829 |
| 286 | 100.000 | 100.000 | 99.930 | 99.319 | 97.860 |
| 287 | 100.000 | 100.000 | 99.930 | 99.324 | 97.877 |
| 288 | 100.000 | 100.000 | 99.930 | 99.335 | 97.909 |
| 289 | 100.000 | 100.000 | 99.930 | 99.335 | 97.910 |
| 290 | 100.000 | 100.000 | 99.930 | 99.341 | 97.927 |
| 291 | 100.000 | 100.000 | 99.930 | 99.351 | 97.962 |
| 292 | 100.000 | 100.000 | 99.930 | 99.351 | 97.964 |
| 293 | 100.000 | 100.000 | 99.930 | 99.362 | 97.995 |
| 294 | 100.000 | 100.000 | 99.930 | 99.367 | 98.012 |
| 295 | 100.000 | 100.000 | 99.930 | 99.367 | 98.014 |
| 296 | 100.000 | 100.000 | 99.930 | 99.373 | 98.031 |
| 297 | 100.000 | 100.000 | 99.930 | 99.378 | 98.047 |
| 298 | 100.000 | 100.000 | 99.930 | 99.383 | 98.064 |
| 299 | 100.000 | 100.000 | 99.930 | 99.389 | 98.081 |
| 300 | 100.000 | 100.000 | 99.930 | 99.394 | 98.097 |
| 301 | 100.000 | 100.000 | 99.930 | 99.399 | 98.119 |
| 302 | 100.000 | 100.000 | 99.930 | 99.410 | 98.154 |
| 303 | 100.000 | 100.000 | 99.930 | 99.421 | 98.186 |
| 304 | 100.000 | 100.000 | 99.930 | 99.442 | 98.245 |
| 305 | 100.000 | 100.000 | 99.930 | 99.464 | 98.308 |
| 306 | 100.000 | 100.000 | 99.930 | 99.469 | 98.325 |
| 307 | 100.000 | 100.000 | 99.930 | 99.485 | 98.371 |
| 308 | 100.000 | 100.000 | 99.930 | 99.496 | 98.402 |
| 309 | 100.000 | 100.000 | 99.953 | 99.550 | 98.485 |
| 310 | 100.000 | 100.000 | 99.953 | 99.555 | 98.502 |
| 311 | 100.000 | 100.000 | 99.953 | 99.560 | 98.519 |
| 312 | 100.000 | 100.000 | 99.953 | 99.566 | 98.535 |
| 313 | 100.000 | 100.000 | 99.953 | 99.566 | 98.537 |
| 314 | 100.000 | 100.000 | 99.953 | 99.571 | 98.554 |
| 315 | 100.000 | 100.000 | 99.953 | 99.582 | 98.589 |
| 316 | 100.000 | 100.000 | 100.000 | 99.694 | 98.767 |
| 317 | 100.000 | 100.000 | 100.000 | 99.716 | 98.826 |
| 318 | 100.000 | 100.000 | 100.000 | 99.727 | 98.857 |
| 319 | 100.000 | 100.000 | 100.000 | 99.732 | 98.874 |
| 320 | 100.000 | 100.000 | 100.000 | 99.743 | 98.905 |
| 321 | 100.000 | 100.000 | 100.000 | 99.748 | 98.922 |
| 322 | 100.000 | 100.000 | 100.000 | 99.748 | 98.931 |
| 323 | 100.000 | 100.000 | 100.000 | 99.748 | 98.935 |
| 324 | 100.000 | 100.000 | 100.000 | 99.753 | 98.951 |
| 325 | 100.000 | 100.000 | 100.000 | 99.759 | 98.968 |
| 326 | 100.000 | 100.000 | 100.000 | 99.759 | 98.970 |
| 327 | 100.000 | 100.000 | 100.000 | 99.759 | 98.977 |
| 328 | 100.000 | 100.000 | 100.000 | 99.759 | 98.981 |
| 329 | 100.000 | 100.000 | 100.000 | 99.769 | 99.016 |
| 330 | 100.000 | 100.000 | 100.000 | 99.780 | 99.048 |
| 331 | 100.000 | 100.000 | 100.000 | 99.786 | 99.064 |
| 332 | 100.000 | 100.000 | 100.000 | 99.791 | 99.081 |
| 333 | 100.000 | 100.000 | 100.000 | 99.812 | 99.140 |
| 334 | 100.000 | 100.000 | 100.000 | 99.823 | 99.172 |
| 335 | 100.000 | 100.000 | 100.000 | 99.828 | 99.188 |
| 336 | 100.000 | 100.000 | 100.000 | 99.834 | 99.205 |
| 337 | 100.000 | 100.000 | 100.000 | 99.845 | 99.236 |
| 338 | 100.000 | 100.000 | 100.000 | 99.866 | 99.295 |
| 339 | 100.000 | 100.000 | 100.000 | 99.866 | 99.299 |
| 340 | 100.000 | 100.000 | 100.000 | 99.866 | 99.301 |
| 341 | 100.000 | 100.000 | 100.000 | 99.877 | 99.336 |
| 342 | 100.000 | 100.000 | 100.000 | 99.877 | 99.338 |
| 343 | 100.000 | 100.000 | 100.000 | 99.887 | 99.369 |
| 344 | 100.000 | 100.000 | 100.000 | 99.893 | 99.386 |
| 345 | 100.000 | 100.000 | 100.000 | 99.903 | 99.417 |
| 346 | 100.000 | 100.000 | 100.000 | 99.914 | 99.449 |
| 347 | 100.000 | 100.000 | 100.000 | 99.920 | 99.466 |
| 348 | 100.000 | 100.000 | 100.000 | 99.920 | 99.469 |
| 349 | 100.000 | 100.000 | 100.000 | 99.930 | 99.504 |
| 350 | 100.000 | 100.000 | 100.000 | 99.936 | 99.521 |
| 351 | 100.000 | 100.000 | 100.000 | 99.941 | 99.538 |
| 352 | 100.000 | 100.000 | 100.000 | 99.946 | 99.554 |
| 353 | 100.000 | 100.000 | 100.000 | 99.957 | 99.586 |
| 354 | 100.000 | 100.000 | 100.000 | 99.973 | 99.630 |
| 355 | 100.000 | 100.000 | 100.000 | 99.979 | 99.647 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
grep
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
| 4 | 71.429 | 41.794 | 24.472 | 14.857 | 9.438 |
| 5 | 78.571 | 48.900 | 29.634 | 18.321 | 11.741 |
| 6 | 85.714 | 55.330 | 34.374 | 21.613 | 13.991 |
| 7 | 88.095 | 59.391 | 38.128 | 24.530 | 16.112 |
| 8 | 92.857 | 63.959 | 41.835 | 27.334 | 18.159 |
| 9 | 97.619 | 68.866 | 45.730 | 30.197 | 20.214 |
| 10 | 100.000 | 72.420 | 49.061 | 32.867 | 22.215 |
| 11 | 100.000 | 74.958 | 51.971 | 35.355 | 24.140 |
| 12 | 100.000 | 77.157 | 54.622 | 37.708 | 26.009 |
| 13 | 100.000 | 79.188 | 57.156 | 39.998 | 27.844 |
| 14 | 100.000 | 81.049 | 59.456 | 42.110 | 29.582 |
| 15 | 100.000 | 82.572 | 61.520 | 44.116 | 31.283 |
| 16 | 100.000 | 84.095 | 63.491 | 45.987 | 32.864 |
| 17 | 100.000 | 85.279 | 65.275 | 47.820 | 34.475 |
| 18 | 100.000 | 86.464 | 66.917 | 49.467 | 35.927 |
| 19 | 100.000 | 87.479 | 68.466 | 51.102 | 37.408 |
| 20 | 100.000 | 88.663 | 70.108 | 52.748 | 38.859 |
| 21 | 100.000 | 89.679 | 71.563 | 54.254 | 40.228 |
| 22 | 100.000 | 90.694 | 72.994 | 55.718 | 41.548 |
| 23 | 100.000 | 91.371 | 74.214 | 57.128 | 42.896 |
| 24 | 100.000 | 92.047 | 75.340 | 58.394 | 44.104 |
| 25 | 100.000 | 92.724 | 76.420 | 59.605 | 45.276 |
| 26 | 100.000 | 93.401 | 77.475 | 60.790 | 46.430 |
| 27 | 100.000 | 93.739 | 78.297 | 61.879 | 47.567 |
| 28 | 100.000 | 94.416 | 79.282 | 62.930 | 48.562 |
| 29 | 100.000 | 94.585 | 79.962 | 63.932 | 49.653 |
| 30 | 100.000 | 95.093 | 80.831 | 64.951 | 50.667 |
| 31 | 100.000 | 95.601 | 81.699 | 65.964 | 51.674 |
| 32 | 100.000 | 96.108 | 82.543 | 66.962 | 52.688 |
| 33 | 100.000 | 96.616 | 83.365 | 67.894 | 53.607 |
| 34 | 100.000 | 96.954 | 84.069 | 68.801 | 54.563 |
| 35 | 100.000 | 97.124 | 84.679 | 69.696 | 55.537 |
| 36 | 100.000 | 97.631 | 85.453 | 70.565 | 56.401 |
| 37 | 100.000 | 97.970 | 86.133 | 71.428 | 57.309 |
| 38 | 100.000 | 98.308 | 86.767 | 72.205 | 58.112 |
| 39 | 100.000 | 98.646 | 87.400 | 72.988 | 58.927 |
| 40 | 100.000 | 98.816 | 87.870 | 73.701 | 59.767 |
| 41 | 100.000 | 98.985 | 88.339 | 74.377 | 60.529 |
| 42 | 100.000 | 99.323 | 88.855 | 74.956 | 61.107 |
| 43 | 100.000 | 99.323 | 89.183 | 75.551 | 61.832 |
| 44 | 100.000 | 99.492 | 89.606 | 76.157 | 62.528 |
| 45 | 100.000 | 99.492 | 89.934 | 76.752 | 63.252 |
| 46 | 100.000 | 99.492 | 90.239 | 77.320 | 63.959 |
| 47 | 100.000 | 99.492 | 90.544 | 77.883 | 64.656 |
| 48 | 100.000 | 99.492 | 90.826 | 78.414 | 65.325 |
| 49 | 100.000 | 99.492 | 91.084 | 78.907 | 65.956 |
| 50 | 100.000 | 99.492 | 91.342 | 79.406 | 66.599 |
| 51 | 100.000 | 99.492 | 91.577 | 79.862 | 67.191 |
| 52 | 100.000 | 99.492 | 91.811 | 80.307 | 67.759 |
| 53 | 100.000 | 99.662 | 92.140 | 80.746 | 68.255 |
| 54 | 100.000 | 99.831 | 92.468 | 81.207 | 68.802 |
| 55 | 100.000 | 99.831 | 92.679 | 81.626 | 69.347 |
| 56 | 100.000 | 99.831 | 92.891 | 82.028 | 69.863 |
| 57 | 100.000 | 99.831 | 93.102 | 82.441 | 70.403 |
| 58 | 100.000 | 99.831 | 93.313 | 82.859 | 70.956 |
| 59 | 100.000 | 99.831 | 93.501 | 83.213 | 71.406 |
| 60 | 100.000 | 99.831 | 93.688 | 83.583 | 71.894 |
| 61 | 100.000 | 100.000 | 93.970 | 83.926 | 72.265 |
| 62 | 100.000 | 100.000 | 94.158 | 84.296 | 72.756 |
| 63 | 100.000 | 100.000 | 94.345 | 84.671 | 73.257 |
| 64 | 100.000 | 100.000 | 94.533 | 85.025 | 73.706 |
| 65 | 100.000 | 100.000 | 94.697 | 85.352 | 74.144 |
| 66 | 100.000 | 100.000 | 94.862 | 85.700 | 74.631 |
| 67 | 100.000 | 100.000 | 95.002 | 85.985 | 75.010 |
| 68 | 100.000 | 100.000 | 95.143 | 86.290 | 75.444 |
| 69 | 100.000 | 100.000 | 95.284 | 86.585 | 75.857 |
| 70 | 100.000 | 100.000 | 95.425 | 86.875 | 76.258 |
| 71 | 100.000 | 100.000 | 95.565 | 87.148 | 76.617 |
| 72 | 100.000 | 100.000 | 95.683 | 87.389 | 76.953 |
| 73 | 100.000 | 100.000 | 95.800 | 87.636 | 77.301 |
| 74 | 100.000 | 100.000 | 95.917 | 87.883 | 77.649 |
| 75 | 100.000 | 100.000 | 96.035 | 88.129 | 77.996 |
| 76 | 100.000 | 100.000 | 96.152 | 88.387 | 78.368 |
| 77 | 100.000 | 100.000 | 96.269 | 88.623 | 78.688 |
| 78 | 100.000 | 100.000 | 96.387 | 88.864 | 79.023 |
| 79 | 100.000 | 100.000 | 96.481 | 89.057 | 79.291 |
| 80 | 100.000 | 100.000 | 96.574 | 89.271 | 79.614 |
| 81 | 100.000 | 100.000 | 96.668 | 89.480 | 79.919 |
| 82 | 100.000 | 100.000 | 96.762 | 89.673 | 80.188 |
| 83 | 100.000 | 100.000 | 96.856 | 89.888 | 80.507 |
| 84 | 100.000 | 100.000 | 96.950 | 90.092 | 80.796 |
| 85 | 100.000 | 100.000 | 97.044 | 90.285 | 81.064 |
| 86 | 100.000 | 100.000 | 97.137 | 90.478 | 81.332 |
| 87 | 100.000 | 100.000 | 97.231 | 90.703 | 81.678 |
| 88 | 100.000 | 100.000 | 97.325 | 90.917 | 81.998 |
| 89 | 100.000 | 100.000 | 97.419 | 91.110 | 82.266 |
| 90 | 100.000 | 100.000 | 97.489 | 91.261 | 82.482 |
| 91 | 100.000 | 100.000 | 97.560 | 91.432 | 82.754 |
| 92 | 100.000 | 100.000 | 97.630 | 91.588 | 82.984 |
| 93 | 100.000 | 100.000 | 97.701 | 91.732 | 83.185 |
| 94 | 100.000 | 100.000 | 97.771 | 91.893 | 83.429 |
| 95 | 100.000 | 100.000 | 97.841 | 92.054 | 83.671 |
| 96 | 100.000 | 100.000 | 97.912 | 92.204 | 83.888 |
| 97 | 100.000 | 100.000 | 97.982 | 92.392 | 84.193 |
| 98 | 100.000 | 100.000 | 98.053 | 92.547 | 84.422 |
| 99 | 100.000 | 100.000 | 98.123 | 92.697 | 84.639 |
| 100 | 100.000 | 100.000 | 98.193 | 92.848 | 84.855 |
| 101 | 100.000 | 100.000 | 98.264 | 93.030 | 85.147 |
| 102 | 100.000 | 100.000 | 98.334 | 93.207 | 85.428 |
| 103 | 100.000 | 100.000 | 98.381 | 93.319 | 85.606 |
| 104 | 100.000 | 100.000 | 98.428 | 93.432 | 85.783 |
| 105 | 100.000 | 100.000 | 98.475 | 93.561 | 85.998 |
| 106 | 100.000 | 100.000 | 98.522 | 93.684 | 86.201 |
| 107 | 100.000 | 100.000 | 98.569 | 93.813 | 86.419 |
| 108 | 100.000 | 100.000 | 98.616 | 93.915 | 86.569 |
| 109 | 100.000 | 100.000 | 98.663 | 94.038 | 86.771 |
| 110 | 100.000 | 100.000 | 98.710 | 94.167 | 86.989 |
| 111 | 100.000 | 100.000 | 98.756 | 94.279 | 87.168 |
| 112 | 100.000 | 100.000 | 98.803 | 94.402 | 87.370 |
| 113 | 100.000 | 100.000 | 98.850 | 94.504 | 87.520 |
| 114 | 100.000 | 100.000 | 98.897 | 94.606 | 87.669 |
| 115 | 100.000 | 100.000 | 98.944 | 94.708 | 87.819 |
| 116 | 100.000 | 100.000 | 98.991 | 94.831 | 88.021 |
| 117 | 100.000 | 100.000 | 99.038 | 94.933 | 88.171 |
| 118 | 100.000 | 100.000 | 99.085 | 95.046 | 88.350 |
| 119 | 100.000 | 100.000 | 99.132 | 95.158 | 88.527 |
| 120 | 100.000 | 100.000 | 99.179 | 95.271 | 88.705 |
| 121 | 100.000 | 100.000 | 99.226 | 95.373 | 88.855 |
| 122 | 100.000 | 100.000 | 99.249 | 95.427 | 88.938 |
| 123 | 100.000 | 100.000 | 99.273 | 95.485 | 89.036 |
| 124 | 100.000 | 100.000 | 99.296 | 95.550 | 89.149 |
| 125 | 100.000 | 100.000 | 99.320 | 95.609 | 89.247 |
| 126 | 100.000 | 100.000 | 99.343 | 95.679 | 89.373 |
| 127 | 100.000 | 100.000 | 99.366 | 95.743 | 89.485 |
| 128 | 100.000 | 100.000 | 99.390 | 95.813 | 89.611 |
| 129 | 100.000 | 100.000 | 99.413 | 95.866 | 89.694 |
| 130 | 100.000 | 100.000 | 99.437 | 95.941 | 89.833 |
| 131 | 100.000 | 100.000 | 99.460 | 96.038 | 90.025 |
| 132 | 100.000 | 100.000 | 99.484 | 96.113 | 90.164 |
| 133 | 100.000 | 100.000 | 99.507 | 96.172 | 90.262 |
| 134 | 100.000 | 100.000 | 99.531 | 96.225 | 90.345 |
| 135 | 100.000 | 100.000 | 99.554 | 96.290 | 90.456 |
| 136 | 100.000 | 100.000 | 99.578 | 96.365 | 90.595 |
| 137 | 100.000 | 100.000 | 99.601 | 96.445 | 90.747 |
| 138 | 100.000 | 100.000 | 99.625 | 96.510 | 90.857 |
| 139 | 100.000 | 100.000 | 99.648 | 96.585 | 90.996 |
| 140 | 100.000 | 100.000 | 99.672 | 96.638 | 91.079 |
| 141 | 100.000 | 100.000 | 99.695 | 96.708 | 91.205 |
| 142 | 100.000 | 100.000 | 99.718 | 96.767 | 91.303 |
| 143 | 100.000 | 100.000 | 99.742 | 96.821 | 91.386 |
| 144 | 100.000 | 100.000 | 99.765 | 96.906 | 91.549 |
| 145 | 100.000 | 100.000 | 99.789 | 96.976 | 91.673 |
| 146 | 100.000 | 100.000 | 99.812 | 97.030 | 91.756 |
| 147 | 100.000 | 100.000 | 99.836 | 97.089 | 91.858 |
| 148 | 100.000 | 100.000 | 99.859 | 97.142 | 91.941 |
| 149 | 100.000 | 100.000 | 99.883 | 97.201 | 92.039 |
| 150 | 100.000 | 100.000 | 99.906 | 97.266 | 92.150 |
| 151 | 100.000 | 100.000 | 99.930 | 97.325 | 92.248 |
| 152 | 100.000 | 100.000 | 99.953 | 97.378 | 92.331 |
| 153 | 100.000 | 100.000 | 99.977 | 97.432 | 92.415 |
| 154 | 100.000 | 100.000 | 100.000 | 97.501 | 92.540 |
| 155 | 100.000 | 100.000 | 100.000 | 97.512 | 92.572 |
| 156 | 100.000 | 100.000 | 100.000 | 97.518 | 92.588 |
| 157 | 100.000 | 100.000 | 100.000 | 97.534 | 92.633 |
| 158 | 100.000 | 100.000 | 100.000 | 97.544 | 92.666 |
| 159 | 100.000 | 100.000 | 100.000 | 97.560 | 92.712 |
| 160 | 100.000 | 100.000 | 100.000 | 97.566 | 92.731 |
| 161 | 100.000 | 100.000 | 100.000 | 97.593 | 92.805 |
| 162 | 100.000 | 100.000 | 100.000 | 97.614 | 92.864 |
| 163 | 100.000 | 100.000 | 100.000 | 97.630 | 92.910 |
| 164 | 100.000 | 100.000 | 100.000 | 97.668 | 93.006 |
| 165 | 100.000 | 100.000 | 100.000 | 97.700 | 93.093 |
| 166 | 100.000 | 100.000 | 100.000 | 97.716 | 93.138 |
| 167 | 100.000 | 100.000 | 100.000 | 97.727 | 93.169 |
| 168 | 100.000 | 100.000 | 100.000 | 97.748 | 93.228 |
| 169 | 100.000 | 100.000 | 100.000 | 97.759 | 93.260 |
| 170 | 100.000 | 100.000 | 100.000 | 97.775 | 93.306 |
| 171 | 100.000 | 100.000 | 100.000 | 97.780 | 93.326 |
| 172 | 100.000 | 100.000 | 100.000 | 97.786 | 93.345 |
| 173 | 100.000 | 100.000 | 100.000 | 97.807 | 93.404 |
| 174 | 100.000 | 100.000 | 100.000 | 97.812 | 93.420 |
| 175 | 100.000 | 100.000 | 100.000 | 97.823 | 93.452 |
| 176 | 100.000 | 100.000 | 100.000 | 97.845 | 93.511 |
| 177 | 100.000 | 100.000 | 100.000 | 97.855 | 93.543 |
| 178 | 100.000 | 100.000 | 100.000 | 97.877 | 93.605 |
| 179 | 100.000 | 100.000 | 100.000 | 97.946 | 93.777 |
| 180 | 100.000 | 100.000 | 100.000 | 97.963 | 93.822 |
| 181 | 100.000 | 100.000 | 100.000 | 97.979 | 93.866 |
| 182 | 100.000 | 100.000 | 100.000 | 97.984 | 93.885 |
| 183 | 100.000 | 100.000 | 100.000 | 98.005 | 93.944 |
| 184 | 100.000 | 100.000 | 100.000 | 98.016 | 93.975 |
| 185 | 100.000 | 100.000 | 100.000 | 98.027 | 94.010 |
| 186 | 100.000 | 100.000 | 100.000 | 98.059 | 94.094 |
| 187 | 100.000 | 100.000 | 100.000 | 98.075 | 94.140 |
| 188 | 100.000 | 100.000 | 100.000 | 98.102 | 94.212 |
| 189 | 100.000 | 100.000 | 100.000 | 98.113 | 94.243 |
| 190 | 100.000 | 100.000 | 100.000 | 98.123 | 94.279 |
| 191 | 100.000 | 100.000 | 100.000 | 98.198 | 94.458 |
| 192 | 100.000 | 100.000 | 100.000 | 98.204 | 94.475 |
| 193 | 100.000 | 100.000 | 100.000 | 98.247 | 94.585 |
| 194 | 100.000 | 100.000 | 100.000 | 98.257 | 94.621 |
| 195 | 100.000 | 100.000 | 100.000 | 98.322 | 94.780 |
| 196 | 100.000 | 100.000 | 100.000 | 98.333 | 94.811 |
| 197 | 100.000 | 100.000 | 100.000 | 98.365 | 94.898 |
| 198 | 100.000 | 100.000 | 100.000 | 98.370 | 94.915 |
| 199 | 100.000 | 100.000 | 100.000 | 98.386 | 94.961 |
| 200 | 100.000 | 100.000 | 100.000 | 98.402 | 95.007 |
| 201 | 100.000 | 100.000 | 100.000 | 98.429 | 95.079 |
| 202 | 100.000 | 100.000 | 100.000 | 98.445 | 95.125 |
| 203 | 100.000 | 100.000 | 100.000 | 98.467 | 95.185 |
| 204 | 100.000 | 100.000 | 100.000 | 98.483 | 95.233 |
| 205 | 100.000 | 100.000 | 100.000 | 98.499 | 95.277 |
| 206 | 100.000 | 100.000 | 100.000 | 98.515 | 95.325 |
| 207 | 100.000 | 100.000 | 100.000 | 98.531 | 95.370 |
| 208 | 100.000 | 100.000 | 100.000 | 98.542 | 95.401 |
| 209 | 100.000 | 100.000 | 100.000 | 98.552 | 95.447 |
| 210 | 100.000 | 100.000 | 100.000 | 98.568 | 95.493 |
| 211 | 100.000 | 100.000 | 100.000 | 98.601 | 95.577 |
| 212 | 100.000 | 100.000 | 100.000 | 98.606 | 95.593 |
| 213 | 100.000 | 100.000 | 100.000 | 98.649 | 95.704 |
| 214 | 100.000 | 100.000 | 100.000 | 98.660 | 95.736 |
| 215 | 100.000 | 100.000 | 100.000 | 98.665 | 95.754 |
| 216 | 100.000 | 100.000 | 100.000 | 98.670 | 95.771 |
| 217 | 100.000 | 100.000 | 100.000 | 98.676 | 95.787 |
| 218 | 100.000 | 100.000 | 100.000 | 98.697 | 95.847 |
| 219 | 100.000 | 100.000 | 100.000 | 98.713 | 95.891 |
| 220 | 100.000 | 100.000 | 100.000 | 98.719 | 95.908 |
| 221 | 100.000 | 100.000 | 100.000 | 98.724 | 95.924 |
| 222 | 100.000 | 100.000 | 100.000 | 98.745 | 95.984 |
| 223 | 100.000 | 100.000 | 100.000 | 98.751 | 96.000 |
| 224 | 100.000 | 100.000 | 100.000 | 98.756 | 96.017 |
| 225 | 100.000 | 100.000 | 100.000 | 98.761 | 96.033 |
| 226 | 100.000 | 100.000 | 100.000 | 98.778 | 96.080 |
| 227 | 100.000 | 100.000 | 100.000 | 98.778 | 96.089 |
| 228 | 100.000 | 100.000 | 100.000 | 98.788 | 96.120 |
| 229 | 100.000 | 100.000 | 100.000 | 98.831 | 96.231 |
| 230 | 100.000 | 100.000 | 100.000 | 98.837 | 96.248 |
| 231 | 100.000 | 100.000 | 100.000 | 98.842 | 96.266 |
| 232 | 100.000 | 100.000 | 100.000 | 98.853 | 96.298 |
| 233 | 100.000 | 100.000 | 100.000 | 98.879 | 96.370 |
| 234 | 100.000 | 100.000 | 100.000 | 98.912 | 96.457 |
| 235 | 100.000 | 100.000 | 100.000 | 98.917 | 96.474 |
| 236 | 100.000 | 100.000 | 100.000 | 98.922 | 96.490 |
| 237 | 100.000 | 100.000 | 100.000 | 98.944 | 96.549 |
| 238 | 100.000 | 100.000 | 100.000 | 98.949 | 96.568 |
| 239 | 100.000 | 100.000 | 100.000 | 98.971 | 96.627 |
| 240 | 100.000 | 100.000 | 100.000 | 99.003 | 96.714 |
| 241 | 100.000 | 100.000 | 100.000 | 99.013 | 96.745 |
| 242 | 100.000 | 100.000 | 100.000 | 99.019 | 96.762 |
| 243 | 100.000 | 100.000 | 100.000 | 99.035 | 96.808 |
| 244 | 100.000 | 100.000 | 100.000 | 99.040 | 96.825 |
| 245 | 100.000 | 100.000 | 100.000 | 99.056 | 96.869 |
| 246 | 100.000 | 100.000 | 100.000 | 99.056 | 96.873 |
| 247 | 100.000 | 100.000 | 100.000 | 99.078 | 96.932 |
| 248 | 100.000 | 100.000 | 100.000 | 99.089 | 96.967 |
| 249 | 100.000 | 100.000 | 100.000 | 99.094 | 96.986 |
| 250 | 100.000 | 100.000 | 100.000 | 99.105 | 97.017 |
| 251 | 100.000 | 100.000 | 100.000 | 99.121 | 97.065 |
| 252 | 100.000 | 100.000 | 100.000 | 99.131 | 97.097 |
| 253 | 100.000 | 100.000 | 100.000 | 99.137 | 97.115 |
| 254 | 100.000 | 100.000 | 100.000 | 99.147 | 97.150 |
| 255 | 100.000 | 100.000 | 100.000 | 99.164 | 97.197 |
| 256 | 100.000 | 100.000 | 100.000 | 99.169 | 97.213 |
| 257 | 100.000 | 100.000 | 100.000 | 99.180 | 97.245 |
| 258 | 100.000 | 100.000 | 100.000 | 99.190 | 97.276 |
| 259 | 100.000 | 100.000 | 100.000 | 99.212 | 97.339 |
| 260 | 100.000 | 100.000 | 100.000 | 99.217 | 97.356 |
| 261 | 100.000 | 100.000 | 100.000 | 99.282 | 97.518 |
| 262 | 100.000 | 100.000 | 100.000 | 99.292 | 97.550 |
| 263 | 100.000 | 100.000 | 100.000 | 99.303 | 97.581 |
| 264 | 100.000 | 100.000 | 100.000 | 99.303 | 97.585 |
| 265 | 100.000 | 100.000 | 100.000 | 99.330 | 97.661 |
| 266 | 100.000 | 100.000 | 100.000 | 99.330 | 97.664 |
| 267 | 100.000 | 100.000 | 100.000 | 99.335 | 97.681 |
| 268 | 100.000 | 100.000 | 100.000 | 99.341 | 97.701 |
| 269 | 100.000 | 100.000 | 100.000 | 99.351 | 97.733 |
| 270 | 100.000 | 100.000 | 100.000 | 99.357 | 97.750 |
| 271 | 100.000 | 100.000 | 100.000 | 99.378 | 97.809 |
| 272 | 100.000 | 100.000 | 100.000 | 99.389 | 97.840 |
| 273 | 100.000 | 100.000 | 100.000 | 99.394 | 97.857 |
| 274 | 100.000 | 100.000 | 100.000 | 99.405 | 97.888 |
| 275 | 100.000 | 100.000 | 100.000 | 99.410 | 97.905 |
| 276 | 100.000 | 100.000 | 100.000 | 99.416 | 97.923 |
| 277 | 100.000 | 100.000 | 100.000 | 99.421 | 97.942 |
| 278 | 100.000 | 100.000 | 100.000 | 99.426 | 97.958 |
| 279 | 100.000 | 100.000 | 100.000 | 99.432 | 97.975 |
| 280 | 100.000 | 100.000 | 100.000 | 99.442 | 98.007 |
| 281 | 100.000 | 100.000 | 100.000 | 99.448 | 98.023 |
| 282 | 100.000 | 100.000 | 100.000 | 99.469 | 98.082 |
| 283 | 100.000 | 100.000 | 100.000 | 99.501 | 98.169 |
| 284 | 100.000 | 100.000 | 100.000 | 99.501 | 98.173 |
| 285 | 100.000 | 100.000 | 100.000 | 99.501 | 98.175 |
| 286 | 100.000 | 100.000 | 100.000 | 99.512 | 98.206 |
| 287 | 100.000 | 100.000 | 100.000 | 99.534 | 98.265 |
| 288 | 100.000 | 100.000 | 100.000 | 99.544 | 98.297 |
| 289 | 100.000 | 100.000 | 100.000 | 99.555 | 98.328 |
| 290 | 100.000 | 100.000 | 100.000 | 99.555 | 98.330 |
| 291 | 100.000 | 100.000 | 100.000 | 99.566 | 98.362 |
| 292 | 100.000 | 100.000 | 100.000 | 99.571 | 98.378 |
| 293 | 100.000 | 100.000 | 100.000 | 99.576 | 98.395 |
| 294 | 100.000 | 100.000 | 100.000 | 99.587 | 98.426 |
| 295 | 100.000 | 100.000 | 100.000 | 99.593 | 98.443 |
| 296 | 100.000 | 100.000 | 100.000 | 99.598 | 98.460 |
| 297 | 100.000 | 100.000 | 100.000 | 99.598 | 98.463 |
| 298 | 100.000 | 100.000 | 100.000 | 99.609 | 98.495 |
| 299 | 100.000 | 100.000 | 100.000 | 99.609 | 98.498 |
| 300 | 100.000 | 100.000 | 100.000 | 99.619 | 98.530 |
| 301 | 100.000 | 100.000 | 100.000 | 99.625 | 98.547 |
| 302 | 100.000 | 100.000 | 100.000 | 99.635 | 98.582 |
| 303 | 100.000 | 100.000 | 100.000 | 99.646 | 98.613 |
| 304 | 100.000 | 100.000 | 100.000 | 99.651 | 98.630 |
| 305 | 100.000 | 100.000 | 100.000 | 99.662 | 98.661 |
| 306 | 100.000 | 100.000 | 100.000 | 99.668 | 98.678 |
| 307 | 100.000 | 100.000 | 100.000 | 99.668 | 98.685 |
| 308 | 100.000 | 100.000 | 100.000 | 99.700 | 98.772 |
| 309 | 100.000 | 100.000 | 100.000 | 99.721 | 98.831 |
| 310 | 100.000 | 100.000 | 100.000 | 99.727 | 98.848 |
| 311 | 100.000 | 100.000 | 100.000 | 99.732 | 98.866 |
| 312 | 100.000 | 100.000 | 100.000 | 99.753 | 98.926 |
| 313 | 100.000 | 100.000 | 100.000 | 99.764 | 98.957 |
| 314 | 100.000 | 100.000 | 100.000 | 99.769 | 98.974 |
| 315 | 100.000 | 100.000 | 100.000 | 99.780 | 99.005 |
| 316 | 100.000 | 100.000 | 100.000 | 99.786 | 99.022 |
| 317 | 100.000 | 100.000 | 100.000 | 99.818 | 99.109 |
| 318 | 100.000 | 100.000 | 100.000 | 99.823 | 99.125 |
| 319 | 100.000 | 100.000 | 100.000 | 99.823 | 99.129 |
| 320 | 100.000 | 100.000 | 100.000 | 99.823 | 99.133 |
| 321 | 100.000 | 100.000 | 100.000 | 99.823 | 99.135 |
| 322 | 100.000 | 100.000 | 100.000 | 99.828 | 99.151 |
| 323 | 100.000 | 100.000 | 100.000 | 99.828 | 99.153 |
| 324 | 100.000 | 100.000 | 100.000 | 99.828 | 99.160 |
| 325 | 100.000 | 100.000 | 100.000 | 99.828 | 99.162 |
| 326 | 100.000 | 100.000 | 100.000 | 99.828 | 99.164 |
| 327 | 100.000 | 100.000 | 100.000 | 99.839 | 99.196 |
| 328 | 100.000 | 100.000 | 100.000 | 99.861 | 99.255 |
| 329 | 100.000 | 100.000 | 100.000 | 99.861 | 99.257 |
| 330 | 100.000 | 100.000 | 100.000 | 99.866 | 99.273 |
| 331 | 100.000 | 100.000 | 100.000 | 99.871 | 99.295 |
| 332 | 100.000 | 100.000 | 100.000 | 99.877 | 99.312 |
| 333 | 100.000 | 100.000 | 100.000 | 99.887 | 99.344 |
| 334 | 100.000 | 100.000 | 100.000 | 99.887 | 99.345 |
| 335 | 100.000 | 100.000 | 100.000 | 99.898 | 99.377 |
| 336 | 100.000 | 100.000 | 100.000 | 99.909 | 99.408 |
| 337 | 100.000 | 100.000 | 100.000 | 99.920 | 99.440 |
| 338 | 100.000 | 100.000 | 100.000 | 99.925 | 99.456 |
| 339 | 100.000 | 100.000 | 100.000 | 99.925 | 99.458 |
| 340 | 100.000 | 100.000 | 100.000 | 99.930 | 99.475 |
| 341 | 100.000 | 100.000 | 100.000 | 99.941 | 99.510 |
| 342 | 100.000 | 100.000 | 100.000 | 99.946 | 99.527 |
| 343 | 100.000 | 100.000 | 100.000 | 99.946 | 99.530 |
| 344 | 100.000 | 100.000 | 100.000 | 99.946 | 99.532 |
| 345 | 100.000 | 100.000 | 100.000 | 99.952 | 99.549 |
| 346 | 100.000 | 100.000 | 100.000 | 99.962 | 99.580 |
| 347 | 100.000 | 100.000 | 100.000 | 99.968 | 99.597 |
| 348 | 100.000 | 100.000 | 100.000 | 99.973 | 99.614 |
| 349 | 100.000 | 100.000 | 100.000 | 99.979 | 99.630 |
| 350 | 100.000 | 100.000 | 100.000 | 99.984 | 99.647 |
| 351 | 100.000 | 100.000 | 100.000 | 99.984 | 99.650 |
| 352 | 100.000 | 100.000 | 100.000 | 99.984 | 99.654 |
| 353 | 100.000 | 100.000 | 100.000 | 99.989 | 99.671 |
| 354 | 100.000 | 100.000 | 100.000 | 99.989 | 99.673 |
| 355 | 100.000 | 100.000 | 100.000 | 99.995 | 99.689 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
grep
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 45.238 | 22.843 | 12.600 | 7.479 | 4.721 |
| 3 | 59.524 | 32.826 | 18.700 | 11.211 | 7.088 |
| 4 | 69.048 | 40.778 | 24.097 | 14.728 | 9.400 |
| 5 | 76.190 | 47.547 | 29.071 | 18.128 | 11.687 |
| 6 | 80.952 | 53.299 | 33.693 | 21.420 | 13.947 |
| 7 | 83.333 | 57.868 | 37.799 | 24.503 | 16.131 |
| 8 | 88.095 | 62.944 | 41.952 | 27.575 | 18.309 |
| 9 | 90.476 | 66.328 | 45.096 | 30.122 | 20.251 |
| 10 | 95.238 | 70.897 | 48.733 | 32.851 | 22.250 |
| 11 | 97.619 | 74.112 | 51.807 | 35.387 | 24.195 |
| 12 | 97.619 | 75.973 | 54.223 | 37.633 | 26.028 |
| 13 | 100.000 | 79.019 | 57.039 | 39.955 | 27.842 |
| 14 | 100.000 | 80.541 | 59.057 | 41.896 | 29.489 |
| 15 | 100.000 | 81.387 | 60.535 | 43.553 | 31.022 |
| 16 | 100.000 | 82.741 | 62.459 | 45.456 | 32.653 |
| 17 | 100.000 | 84.095 | 64.313 | 47.284 | 34.236 |
| 18 | 100.000 | 84.772 | 65.580 | 48.753 | 35.634 |
| 19 | 100.000 | 85.787 | 67.081 | 50.319 | 37.049 |
| 20 | 100.000 | 86.633 | 68.419 | 51.772 | 38.399 |
| 21 | 100.000 | 87.310 | 69.568 | 53.107 | 39.703 |
| 22 | 100.000 | 88.494 | 71.164 | 54.683 | 41.088 |
| 23 | 100.000 | 89.340 | 72.478 | 56.104 | 42.410 |
| 24 | 100.000 | 90.017 | 73.627 | 57.407 | 43.656 |
| 25 | 100.000 | 90.186 | 74.402 | 58.522 | 44.829 |
| 26 | 100.000 | 90.863 | 75.504 | 59.756 | 46.008 |
| 27 | 100.000 | 91.709 | 76.725 | 61.037 | 47.198 |
| 28 | 100.000 | 92.386 | 77.757 | 62.168 | 48.287 |
| 29 | 100.000 | 92.555 | 78.390 | 63.096 | 49.287 |
| 30 | 100.000 | 93.232 | 79.399 | 64.200 | 50.356 |
| 31 | 100.000 | 93.570 | 80.103 | 65.101 | 51.297 |
| 32 | 100.000 | 93.909 | 80.784 | 65.975 | 52.226 |
| 33 | 100.000 | 94.585 | 81.746 | 66.994 | 53.193 |
| 34 | 100.000 | 95.093 | 82.543 | 67.916 | 54.128 |
| 35 | 100.000 | 95.431 | 83.200 | 68.752 | 55.020 |
| 36 | 100.000 | 95.601 | 83.717 | 69.508 | 55.872 |
| 37 | 100.000 | 95.939 | 84.374 | 70.334 | 56.738 |
| 38 | 100.000 | 96.616 | 85.312 | 71.304 | 57.642 |
| 39 | 100.000 | 96.954 | 85.946 | 72.103 | 58.489 |
| 40 | 100.000 | 97.293 | 86.579 | 72.902 | 59.339 |
| 41 | 100.000 | 97.293 | 86.884 | 73.497 | 60.101 |
| 42 | 100.000 | 97.293 | 87.189 | 74.066 | 60.811 |
| 43 | 100.000 | 97.800 | 87.893 | 74.838 | 61.590 |
| 44 | 100.000 | 97.800 | 88.198 | 75.401 | 62.287 |
| 45 | 100.000 | 97.800 | 88.480 | 75.937 | 62.958 |
| 46 | 100.000 | 97.970 | 88.878 | 76.521 | 63.646 |
| 47 | 100.000 | 97.970 | 89.160 | 77.047 | 64.305 |
| 48 | 100.000 | 98.477 | 89.887 | 77.814 | 65.026 |
| 49 | 100.000 | 98.816 | 90.427 | 78.446 | 65.684 |
| 50 | 100.000 | 98.816 | 90.662 | 78.902 | 66.276 |
| 51 | 100.000 | 98.985 | 91.037 | 79.433 | 66.894 |
| 52 | 100.000 | 98.985 | 91.272 | 79.888 | 67.485 |
| 53 | 100.000 | 99.154 | 91.647 | 80.409 | 68.079 |
| 54 | 100.000 | 99.154 | 91.811 | 80.784 | 68.621 |
| 55 | 100.000 | 99.154 | 91.999 | 81.181 | 69.166 |
| 56 | 100.000 | 99.154 | 92.210 | 81.599 | 69.712 |
| 57 | 100.000 | 99.323 | 92.539 | 82.065 | 70.272 |
| 58 | 100.000 | 99.492 | 92.891 | 82.553 | 70.838 |
| 59 | 100.000 | 99.492 | 93.078 | 82.939 | 71.363 |
| 60 | 100.000 | 99.492 | 93.290 | 83.341 | 71.879 |
| 61 | 100.000 | 99.492 | 93.477 | 83.717 | 72.380 |
| 62 | 100.000 | 99.492 | 93.665 | 84.092 | 72.881 |
| 63 | 100.000 | 99.492 | 93.853 | 84.462 | 73.371 |
| 64 | 100.000 | 99.492 | 94.017 | 84.805 | 73.847 |
| 65 | 100.000 | 99.662 | 94.345 | 85.245 | 74.342 |
| 66 | 100.000 | 99.831 | 94.650 | 85.647 | 74.804 |
| 67 | 100.000 | 99.831 | 94.791 | 85.947 | 75.228 |
| 68 | 100.000 | 99.831 | 94.932 | 86.247 | 75.651 |
| 69 | 100.000 | 99.831 | 95.073 | 86.548 | 76.075 |
| 70 | 100.000 | 99.831 | 95.214 | 86.843 | 76.485 |
| 71 | 100.000 | 99.831 | 95.307 | 87.089 | 76.874 |
| 72 | 100.000 | 99.831 | 95.425 | 87.352 | 77.260 |
| 73 | 100.000 | 99.831 | 95.542 | 87.609 | 77.632 |
| 74 | 100.000 | 99.831 | 95.636 | 87.845 | 77.996 |
| 75 | 100.000 | 99.831 | 95.753 | 88.103 | 78.368 |
| 76 | 100.000 | 99.831 | 95.894 | 88.381 | 78.741 |
| 77 | 100.000 | 99.831 | 96.011 | 88.633 | 79.102 |
| 78 | 100.000 | 99.831 | 96.105 | 88.859 | 79.446 |
| 79 | 100.000 | 99.831 | 96.222 | 89.105 | 79.794 |
| 80 | 100.000 | 99.831 | 96.340 | 89.352 | 80.141 |
| 81 | 100.000 | 99.831 | 96.457 | 89.598 | 80.489 |
| 82 | 100.000 | 99.831 | 96.527 | 89.797 | 80.818 |
| 83 | 100.000 | 99.831 | 96.621 | 90.017 | 81.151 |
| 84 | 100.000 | 99.831 | 96.715 | 90.226 | 81.460 |
| 85 | 100.000 | 99.831 | 96.809 | 90.429 | 81.748 |
| 86 | 100.000 | 99.831 | 96.879 | 90.612 | 82.044 |
| 87 | 100.000 | 99.831 | 96.973 | 90.810 | 82.329 |
| 88 | 100.000 | 99.831 | 97.067 | 91.009 | 82.608 |
| 89 | 100.000 | 99.831 | 97.114 | 91.164 | 82.886 |
| 90 | 100.000 | 100.000 | 97.372 | 91.459 | 83.191 |
| 91 | 100.000 | 100.000 | 97.466 | 91.657 | 83.477 |
| 92 | 100.000 | 100.000 | 97.560 | 91.850 | 83.745 |
| 93 | 100.000 | 100.000 | 97.607 | 92.000 | 84.010 |
| 94 | 100.000 | 100.000 | 97.677 | 92.172 | 84.280 |
| 95 | 100.000 | 100.000 | 97.748 | 92.338 | 84.533 |
| 96 | 100.000 | 100.000 | 97.818 | 92.504 | 84.790 |
| 97 | 100.000 | 100.000 | 97.888 | 92.671 | 85.047 |
| 98 | 100.000 | 100.000 | 97.959 | 92.831 | 85.288 |
| 99 | 100.000 | 100.000 | 98.029 | 92.992 | 85.534 |
| 100 | 100.000 | 100.000 | 98.099 | 93.148 | 85.763 |
| 101 | 100.000 | 100.000 | 98.170 | 93.298 | 85.979 |
| 102 | 100.000 | 100.000 | 98.217 | 93.427 | 86.197 |
| 103 | 100.000 | 100.000 | 98.287 | 93.577 | 86.414 |
| 104 | 100.000 | 100.000 | 98.358 | 93.727 | 86.630 |
| 105 | 100.000 | 100.000 | 98.428 | 93.877 | 86.847 |
| 106 | 100.000 | 100.000 | 98.498 | 94.027 | 87.063 |
| 107 | 100.000 | 100.000 | 98.569 | 94.177 | 87.279 |
| 108 | 100.000 | 100.000 | 98.639 | 94.327 | 87.496 |
| 109 | 100.000 | 100.000 | 98.686 | 94.451 | 87.701 |
| 110 | 100.000 | 100.000 | 98.733 | 94.574 | 87.906 |
| 111 | 100.000 | 100.000 | 98.780 | 94.697 | 88.111 |
| 112 | 100.000 | 100.000 | 98.827 | 94.821 | 88.319 |
| 113 | 100.000 | 100.000 | 98.874 | 94.944 | 88.520 |
| 114 | 100.000 | 100.000 | 98.921 | 95.057 | 88.705 |
| 115 | 100.000 | 100.000 | 98.968 | 95.169 | 88.883 |
| 116 | 100.000 | 100.000 | 99.015 | 95.282 | 89.060 |
| 117 | 100.000 | 100.000 | 99.061 | 95.389 | 89.225 |
| 118 | 100.000 | 100.000 | 99.061 | 95.453 | 89.387 |
| 119 | 100.000 | 100.000 | 99.108 | 95.561 | 89.552 |
| 120 | 100.000 | 100.000 | 99.155 | 95.668 | 89.717 |
| 121 | 100.000 | 100.000 | 99.179 | 95.754 | 89.887 |
| 122 | 100.000 | 100.000 | 99.226 | 95.861 | 90.055 |
| 123 | 100.000 | 100.000 | 99.249 | 95.947 | 90.221 |
| 124 | 100.000 | 100.000 | 99.249 | 96.011 | 90.377 |
| 125 | 100.000 | 100.000 | 99.273 | 96.091 | 90.532 |
| 126 | 100.000 | 100.000 | 99.320 | 96.193 | 90.682 |
| 127 | 100.000 | 100.000 | 99.343 | 96.274 | 90.837 |
| 128 | 100.000 | 100.000 | 99.366 | 96.354 | 90.992 |
| 129 | 100.000 | 100.000 | 99.413 | 96.456 | 91.142 |
| 130 | 100.000 | 100.000 | 99.437 | 96.536 | 91.296 |
| 131 | 100.000 | 100.000 | 99.484 | 96.638 | 91.446 |
| 132 | 100.000 | 100.000 | 99.484 | 96.692 | 91.581 |
| 133 | 100.000 | 100.000 | 99.507 | 96.767 | 91.719 |
| 134 | 100.000 | 100.000 | 99.531 | 96.842 | 91.860 |
| 135 | 100.000 | 100.000 | 99.554 | 96.917 | 92.000 |
| 136 | 100.000 | 100.000 | 99.578 | 96.987 | 92.126 |
| 137 | 100.000 | 100.000 | 99.601 | 97.056 | 92.252 |
| 138 | 100.000 | 100.000 | 99.625 | 97.126 | 92.378 |
| 139 | 100.000 | 100.000 | 99.648 | 97.190 | 92.488 |
| 140 | 100.000 | 100.000 | 99.672 | 97.255 | 92.599 |
| 141 | 100.000 | 100.000 | 99.695 | 97.319 | 92.712 |
| 142 | 100.000 | 100.000 | 99.718 | 97.384 | 92.825 |
| 143 | 100.000 | 100.000 | 99.742 | 97.448 | 92.938 |
| 144 | 100.000 | 100.000 | 99.765 | 97.512 | 93.052 |
| 145 | 100.000 | 100.000 | 99.789 | 97.577 | 93.163 |
| 146 | 100.000 | 100.000 | 99.789 | 97.614 | 93.263 |
| 147 | 100.000 | 100.000 | 99.812 | 97.673 | 93.361 |
| 148 | 100.000 | 100.000 | 99.836 | 97.732 | 93.459 |
| 149 | 100.000 | 100.000 | 99.859 | 97.791 | 93.557 |
| 150 | 100.000 | 100.000 | 99.883 | 97.850 | 93.655 |
| 151 | 100.000 | 100.000 | 99.883 | 97.888 | 93.752 |
| 152 | 100.000 | 100.000 | 99.883 | 97.925 | 93.851 |
| 153 | 100.000 | 100.000 | 99.883 | 97.963 | 93.955 |
| 154 | 100.000 | 100.000 | 99.906 | 98.022 | 94.057 |
| 155 | 100.000 | 100.000 | 99.906 | 98.054 | 94.144 |
| 156 | 100.000 | 100.000 | 99.906 | 98.086 | 94.230 |
| 157 | 100.000 | 100.000 | 99.930 | 98.140 | 94.314 |
| 158 | 100.000 | 100.000 | 99.953 | 98.193 | 94.397 |
| 159 | 100.000 | 100.000 | 99.953 | 98.225 | 94.484 |
| 160 | 100.000 | 100.000 | 99.977 | 98.279 | 94.567 |
| 161 | 100.000 | 100.000 | 99.977 | 98.311 | 94.654 |
| 162 | 100.000 | 100.000 | 99.977 | 98.343 | 94.745 |
| 163 | 100.000 | 100.000 | 99.977 | 98.375 | 94.835 |
| 164 | 100.000 | 100.000 | 100.000 | 98.429 | 94.918 |
| 165 | 100.000 | 100.000 | 100.000 | 98.456 | 94.990 |
| 166 | 100.000 | 100.000 | 100.000 | 98.483 | 95.063 |
| 167 | 100.000 | 100.000 | 100.000 | 98.509 | 95.135 |
| 168 | 100.000 | 100.000 | 100.000 | 98.536 | 95.211 |
| 169 | 100.000 | 100.000 | 100.000 | 98.563 | 95.283 |
| 170 | 100.000 | 100.000 | 100.000 | 98.590 | 95.355 |
| 171 | 100.000 | 100.000 | 100.000 | 98.617 | 95.427 |
| 172 | 100.000 | 100.000 | 100.000 | 98.638 | 95.490 |
| 173 | 100.000 | 100.000 | 100.000 | 98.660 | 95.549 |
| 174 | 100.000 | 100.000 | 100.000 | 98.681 | 95.608 |
| 175 | 100.000 | 100.000 | 100.000 | 98.702 | 95.667 |
| 176 | 100.000 | 100.000 | 100.000 | 98.724 | 95.728 |
| 177 | 100.000 | 100.000 | 100.000 | 98.745 | 95.791 |
| 178 | 100.000 | 100.000 | 100.000 | 98.767 | 95.850 |
| 179 | 100.000 | 100.000 | 100.000 | 98.788 | 95.911 |
| 180 | 100.000 | 100.000 | 100.000 | 98.810 | 95.971 |
| 181 | 100.000 | 100.000 | 100.000 | 98.831 | 96.033 |
| 182 | 100.000 | 100.000 | 100.000 | 98.853 | 96.093 |
| 183 | 100.000 | 100.000 | 100.000 | 98.874 | 96.152 |
| 184 | 100.000 | 100.000 | 100.000 | 98.896 | 96.211 |
| 185 | 100.000 | 100.000 | 100.000 | 98.917 | 96.274 |
| 186 | 100.000 | 100.000 | 100.000 | 98.938 | 96.335 |
| 187 | 100.000 | 100.000 | 100.000 | 98.960 | 96.394 |
| 188 | 100.000 | 100.000 | 100.000 | 98.981 | 96.453 |
| 189 | 100.000 | 100.000 | 100.000 | 99.003 | 96.512 |
| 190 | 100.000 | 100.000 | 100.000 | 99.024 | 96.572 |
| 191 | 100.000 | 100.000 | 100.000 | 99.046 | 96.631 |
| 192 | 100.000 | 100.000 | 100.000 | 99.067 | 96.690 |
| 193 | 100.000 | 100.000 | 100.000 | 99.089 | 96.751 |
| 194 | 100.000 | 100.000 | 100.000 | 99.105 | 96.797 |
| 195 | 100.000 | 100.000 | 100.000 | 99.121 | 96.843 |
| 196 | 100.000 | 100.000 | 100.000 | 99.137 | 96.893 |
| 197 | 100.000 | 100.000 | 100.000 | 99.153 | 96.945 |
| 198 | 100.000 | 100.000 | 100.000 | 99.169 | 96.991 |
| 199 | 100.000 | 100.000 | 100.000 | 99.185 | 97.038 |
| 200 | 100.000 | 100.000 | 100.000 | 99.201 | 97.086 |
| 201 | 100.000 | 100.000 | 100.000 | 99.217 | 97.132 |
| 202 | 100.000 | 100.000 | 100.000 | 99.233 | 97.178 |
| 203 | 100.000 | 100.000 | 100.000 | 99.249 | 97.224 |
| 204 | 100.000 | 100.000 | 100.000 | 99.265 | 97.271 |
| 205 | 100.000 | 100.000 | 100.000 | 99.282 | 97.317 |
| 206 | 100.000 | 100.000 | 100.000 | 99.298 | 97.365 |
| 207 | 100.000 | 100.000 | 100.000 | 99.314 | 97.409 |
| 208 | 100.000 | 100.000 | 100.000 | 99.324 | 97.441 |
| 209 | 100.000 | 100.000 | 100.000 | 99.335 | 97.472 |
| 210 | 100.000 | 100.000 | 100.000 | 99.346 | 97.504 |
| 211 | 100.000 | 100.000 | 100.000 | 99.357 | 97.535 |
| 212 | 100.000 | 100.000 | 100.000 | 99.367 | 97.566 |
| 213 | 100.000 | 100.000 | 100.000 | 99.378 | 97.598 |
| 214 | 100.000 | 100.000 | 100.000 | 99.389 | 97.629 |
| 215 | 100.000 | 100.000 | 100.000 | 99.399 | 97.661 |
| 216 | 100.000 | 100.000 | 100.000 | 99.410 | 97.694 |
| 217 | 100.000 | 100.000 | 100.000 | 99.421 | 97.725 |
| 218 | 100.000 | 100.000 | 100.000 | 99.432 | 97.761 |
| 219 | 100.000 | 100.000 | 100.000 | 99.442 | 97.792 |
| 220 | 100.000 | 100.000 | 100.000 | 99.453 | 97.823 |
| 221 | 100.000 | 100.000 | 100.000 | 99.464 | 97.855 |
| 222 | 100.000 | 100.000 | 100.000 | 99.475 | 97.886 |
| 223 | 100.000 | 100.000 | 100.000 | 99.485 | 97.918 |
| 224 | 100.000 | 100.000 | 100.000 | 99.496 | 97.949 |
| 225 | 100.000 | 100.000 | 100.000 | 99.507 | 97.981 |
| 226 | 100.000 | 100.000 | 100.000 | 99.517 | 98.012 |
| 227 | 100.000 | 100.000 | 100.000 | 99.528 | 98.044 |
| 228 | 100.000 | 100.000 | 100.000 | 99.539 | 98.075 |
| 229 | 100.000 | 100.000 | 100.000 | 99.550 | 98.106 |
| 230 | 100.000 | 100.000 | 100.000 | 99.560 | 98.138 |
| 231 | 100.000 | 100.000 | 100.000 | 99.571 | 98.171 |
| 232 | 100.000 | 100.000 | 100.000 | 99.582 | 98.204 |
| 233 | 100.000 | 100.000 | 100.000 | 99.593 | 98.236 |
| 234 | 100.000 | 100.000 | 100.000 | 99.603 | 98.267 |
| 235 | 100.000 | 100.000 | 100.000 | 99.614 | 98.299 |
| 236 | 100.000 | 100.000 | 100.000 | 99.625 | 98.334 |
| 237 | 100.000 | 100.000 | 100.000 | 99.635 | 98.365 |
| 238 | 100.000 | 100.000 | 100.000 | 99.646 | 98.397 |
| 239 | 100.000 | 100.000 | 100.000 | 99.657 | 98.428 |
| 240 | 100.000 | 100.000 | 100.000 | 99.668 | 98.463 |
| 241 | 100.000 | 100.000 | 100.000 | 99.678 | 98.495 |
| 242 | 100.000 | 100.000 | 100.000 | 99.684 | 98.513 |
| 243 | 100.000 | 100.000 | 100.000 | 99.689 | 98.530 |
| 244 | 100.000 | 100.000 | 100.000 | 99.694 | 98.558 |
| 245 | 100.000 | 100.000 | 100.000 | 99.700 | 98.574 |
| 246 | 100.000 | 100.000 | 100.000 | 99.705 | 98.591 |
| 247 | 100.000 | 100.000 | 100.000 | 99.710 | 98.608 |
| 248 | 100.000 | 100.000 | 100.000 | 99.716 | 98.624 |
| 249 | 100.000 | 100.000 | 100.000 | 99.721 | 98.641 |
| 250 | 100.000 | 100.000 | 100.000 | 99.727 | 98.657 |
| 251 | 100.000 | 100.000 | 100.000 | 99.732 | 98.676 |
| 252 | 100.000 | 100.000 | 100.000 | 99.737 | 98.694 |
| 253 | 100.000 | 100.000 | 100.000 | 99.743 | 98.711 |
| 254 | 100.000 | 100.000 | 100.000 | 99.748 | 98.730 |
| 255 | 100.000 | 100.000 | 100.000 | 99.753 | 98.746 |
| 256 | 100.000 | 100.000 | 100.000 | 99.759 | 98.763 |
| 257 | 100.000 | 100.000 | 100.000 | 99.764 | 98.780 |
| 258 | 100.000 | 100.000 | 100.000 | 99.769 | 98.796 |
| 259 | 100.000 | 100.000 | 100.000 | 99.775 | 98.813 |
| 260 | 100.000 | 100.000 | 100.000 | 99.780 | 98.831 |
| 261 | 100.000 | 100.000 | 100.000 | 99.786 | 98.848 |
| 262 | 100.000 | 100.000 | 100.000 | 99.791 | 98.865 |
| 263 | 100.000 | 100.000 | 100.000 | 99.796 | 98.881 |
| 264 | 100.000 | 100.000 | 100.000 | 99.802 | 98.898 |
| 265 | 100.000 | 100.000 | 100.000 | 99.807 | 98.915 |
| 266 | 100.000 | 100.000 | 100.000 | 99.812 | 98.931 |
| 267 | 100.000 | 100.000 | 100.000 | 99.818 | 98.948 |
| 268 | 100.000 | 100.000 | 100.000 | 99.823 | 98.964 |
| 269 | 100.000 | 100.000 | 100.000 | 99.828 | 98.981 |
| 270 | 100.000 | 100.000 | 100.000 | 99.834 | 98.998 |
| 271 | 100.000 | 100.000 | 100.000 | 99.839 | 99.014 |
| 272 | 100.000 | 100.000 | 100.000 | 99.845 | 99.031 |
| 273 | 100.000 | 100.000 | 100.000 | 99.850 | 99.050 |
| 274 | 100.000 | 100.000 | 100.000 | 99.855 | 99.066 |
| 275 | 100.000 | 100.000 | 100.000 | 99.861 | 99.083 |
| 276 | 100.000 | 100.000 | 100.000 | 99.866 | 99.099 |
| 277 | 100.000 | 100.000 | 100.000 | 99.871 | 99.116 |
| 278 | 100.000 | 100.000 | 100.000 | 99.877 | 99.133 |
| 279 | 100.000 | 100.000 | 100.000 | 99.882 | 99.151 |
| 280 | 100.000 | 100.000 | 100.000 | 99.887 | 99.168 |
| 281 | 100.000 | 100.000 | 100.000 | 99.893 | 99.184 |
| 282 | 100.000 | 100.000 | 100.000 | 99.898 | 99.201 |
| 283 | 100.000 | 100.000 | 100.000 | 99.903 | 99.221 |
| 284 | 100.000 | 100.000 | 100.000 | 99.909 | 99.240 |
| 285 | 100.000 | 100.000 | 100.000 | 99.914 | 99.257 |
| 286 | 100.000 | 100.000 | 100.000 | 99.920 | 99.275 |
| 287 | 100.000 | 100.000 | 100.000 | 99.925 | 99.292 |
| 288 | 100.000 | 100.000 | 100.000 | 99.930 | 99.308 |
| 289 | 100.000 | 100.000 | 100.000 | 99.936 | 99.325 |
| 290 | 100.000 | 100.000 | 100.000 | 99.941 | 99.342 |
| 291 | 100.000 | 100.000 | 100.000 | 99.946 | 99.358 |
| 292 | 100.000 | 100.000 | 100.000 | 99.952 | 99.377 |
| 293 | 100.000 | 100.000 | 100.000 | 99.957 | 99.393 |
| 294 | 100.000 | 100.000 | 100.000 | 99.962 | 99.410 |
| 295 | 100.000 | 100.000 | 100.000 | 99.968 | 99.429 |
| 296 | 100.000 | 100.000 | 100.000 | 99.973 | 99.447 |
| 297 | 100.000 | 100.000 | 100.000 | 99.979 | 99.464 |
| 298 | 100.000 | 100.000 | 100.000 | 99.984 | 99.480 |
| 299 | 100.000 | 100.000 | 100.000 | 99.989 | 99.497 |
| 300 | 100.000 | 100.000 | 100.000 | 99.995 | 99.514 |
| 301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.530 |
| 302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.532 |
| 303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.540 |
| 304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.543 |
| 305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.545 |
| 306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
| 307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.552 |
| 308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.554 |
| 309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.556 |
| 310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.560 |
| 311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
| 312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.565 |
| 313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.569 |
| 314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.573 |
| 315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.577 |
| 316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.580 |
| 317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.584 |
| 318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.588 |
| 319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.591 |
| 320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.599 |
| 321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
| 322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.604 |
| 323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.606 |
| 324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.610 |
| 325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.614 |
| 326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
| 327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.617 |
| 328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.619 |
| 329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.621 |
| 330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
| 331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.625 |
| 332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
| 333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.636 |
| 334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
| 335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.639 |
| 336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
| 337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.645 |
| 338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
| 339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.650 |
| 340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
| 341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
| 342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
| 343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
| 344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.675 |
| 345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
| 346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
| 347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
| 348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
| 349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
| 350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.689 |
| 351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
| 352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
| 353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.697 |
| 354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
| 355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
grep
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 59.524 | 32.487 | 18.442 | 11.077 | 7.033 |
| 4 | 64.286 | 37.563 | 22.548 | 14.101 | 9.181 |
| 5 | 71.429 | 43.993 | 27.241 | 17.345 | 11.402 |
| 6 | 76.190 | 49.577 | 31.699 | 20.535 | 13.614 |
| 7 | 85.714 | 56.853 | 36.603 | 23.806 | 15.826 |
| 8 | 88.095 | 60.406 | 39.934 | 26.487 | 17.843 |
| 9 | 90.476 | 64.129 | 43.383 | 29.221 | 19.879 |
| 10 | 92.857 | 66.836 | 46.011 | 31.494 | 21.702 |
| 11 | 92.857 | 69.205 | 48.780 | 33.902 | 23.590 |
| 12 | 92.857 | 71.743 | 51.783 | 36.486 | 25.575 |
| 13 | 92.857 | 74.112 | 54.528 | 38.867 | 27.444 |
| 14 | 95.238 | 76.819 | 57.062 | 41.033 | 29.192 |
| 15 | 95.238 | 78.680 | 59.362 | 43.134 | 30.913 |
| 16 | 95.238 | 79.526 | 60.770 | 44.689 | 32.358 |
| 17 | 95.238 | 80.372 | 62.224 | 46.292 | 33.828 |
| 18 | 97.619 | 83.418 | 64.969 | 48.512 | 35.547 |
| 19 | 97.619 | 84.772 | 66.729 | 50.212 | 37.018 |
| 20 | 100.000 | 86.971 | 68.724 | 51.965 | 38.497 |
| 21 | 100.000 | 87.986 | 70.202 | 53.498 | 39.884 |
| 22 | 100.000 | 89.002 | 71.633 | 54.962 | 41.204 |
| 23 | 100.000 | 89.679 | 72.736 | 56.206 | 42.404 |
| 24 | 100.000 | 90.525 | 74.003 | 57.557 | 43.660 |
| 25 | 100.000 | 91.032 | 74.988 | 58.748 | 44.842 |
| 26 | 100.000 | 91.878 | 76.255 | 60.099 | 46.097 |
| 27 | 100.000 | 92.217 | 77.030 | 61.117 | 47.168 |
| 28 | 100.000 | 92.724 | 77.921 | 62.174 | 48.228 |
| 29 | 100.000 | 93.232 | 78.789 | 63.198 | 49.259 |
| 30 | 100.000 | 93.570 | 79.493 | 64.115 | 50.236 |
| 31 | 100.000 | 94.078 | 80.338 | 65.112 | 51.247 |
| 32 | 100.000 | 94.247 | 80.878 | 65.911 | 52.148 |
| 33 | 100.000 | 94.247 | 81.300 | 66.656 | 53.030 |
| 34 | 100.000 | 94.247 | 81.628 | 67.289 | 53.831 |
| 35 | 100.000 | 94.755 | 82.567 | 68.393 | 54.909 |
| 36 | 100.000 | 94.924 | 83.153 | 69.251 | 55.848 |
| 37 | 100.000 | 95.431 | 83.951 | 70.168 | 56.775 |
| 38 | 100.000 | 95.601 | 84.444 | 70.897 | 57.605 |
| 39 | 100.000 | 95.939 | 85.054 | 71.664 | 58.428 |
| 40 | 100.000 | 96.277 | 85.687 | 72.457 | 59.265 |
| 41 | 100.000 | 96.447 | 86.086 | 73.052 | 59.977 |
| 42 | 100.000 | 96.616 | 86.532 | 73.701 | 60.725 |
| 43 | 100.000 | 96.616 | 86.861 | 74.296 | 61.449 |
| 44 | 100.000 | 97.293 | 87.823 | 75.304 | 62.393 |
| 45 | 100.000 | 97.293 | 88.104 | 75.830 | 63.051 |
| 46 | 100.000 | 97.293 | 88.339 | 76.296 | 63.663 |
| 47 | 100.000 | 97.462 | 88.738 | 76.870 | 64.330 |
| 48 | 100.000 | 97.800 | 89.301 | 77.551 | 65.052 |
| 49 | 100.000 | 97.970 | 89.700 | 78.119 | 65.708 |
| 50 | 100.000 | 98.646 | 90.591 | 79.020 | 66.546 |
| 51 | 100.000 | 98.816 | 90.967 | 79.556 | 67.175 |
| 52 | 100.000 | 98.816 | 91.178 | 79.980 | 67.739 |
| 53 | 100.000 | 98.816 | 91.389 | 80.403 | 68.303 |
| 54 | 100.000 | 98.816 | 91.553 | 80.757 | 68.800 |
| 55 | 100.000 | 98.816 | 91.741 | 81.138 | 69.314 |
| 56 | 100.000 | 98.816 | 91.882 | 81.454 | 69.775 |
| 57 | 100.000 | 98.816 | 92.093 | 81.872 | 70.320 |
| 58 | 100.000 | 98.985 | 92.398 | 82.301 | 70.840 |
| 59 | 100.000 | 99.154 | 92.726 | 82.757 | 71.378 |
| 60 | 100.000 | 99.154 | 92.891 | 83.100 | 71.851 |
| 61 | 100.000 | 99.154 | 93.055 | 83.443 | 72.325 |
| 62 | 100.000 | 99.323 | 93.407 | 83.926 | 72.879 |
| 63 | 100.000 | 99.323 | 93.595 | 84.296 | 73.369 |
| 64 | 100.000 | 99.323 | 93.759 | 84.634 | 73.832 |
| 65 | 100.000 | 99.492 | 94.111 | 85.111 | 74.374 |
| 66 | 100.000 | 99.492 | 94.181 | 85.320 | 74.725 |
| 67 | 100.000 | 99.492 | 94.275 | 85.556 | 75.093 |
| 68 | 100.000 | 99.492 | 94.416 | 85.851 | 75.503 |
| 69 | 100.000 | 99.492 | 94.533 | 86.113 | 75.888 |
| 70 | 100.000 | 99.492 | 94.627 | 86.349 | 76.252 |
| 71 | 100.000 | 99.492 | 94.744 | 86.612 | 76.635 |
| 72 | 100.000 | 99.662 | 95.073 | 87.052 | 77.131 |
| 73 | 100.000 | 99.662 | 95.167 | 87.277 | 77.477 |
| 74 | 100.000 | 99.662 | 95.260 | 87.502 | 77.821 |
| 75 | 100.000 | 99.662 | 95.401 | 87.786 | 78.207 |
| 76 | 100.000 | 99.662 | 95.542 | 88.065 | 78.582 |
| 77 | 100.000 | 99.662 | 95.683 | 88.349 | 78.961 |
| 78 | 100.000 | 99.662 | 95.777 | 88.564 | 79.281 |
| 79 | 100.000 | 99.662 | 95.870 | 88.773 | 79.592 |
| 80 | 100.000 | 99.662 | 95.988 | 89.014 | 79.929 |
| 81 | 100.000 | 99.662 | 96.082 | 89.223 | 80.237 |
| 82 | 100.000 | 99.662 | 96.199 | 89.464 | 80.572 |
| 83 | 100.000 | 99.831 | 96.504 | 89.850 | 80.996 |
| 84 | 100.000 | 99.831 | 96.598 | 90.060 | 81.304 |
| 85 | 100.000 | 99.831 | 96.668 | 90.236 | 81.586 |
| 86 | 100.000 | 99.831 | 96.739 | 90.419 | 81.874 |
| 87 | 100.000 | 99.831 | 96.832 | 90.622 | 82.172 |
| 88 | 100.000 | 99.831 | 96.926 | 90.826 | 82.468 |
| 89 | 100.000 | 99.831 | 96.997 | 90.992 | 82.726 |
| 90 | 100.000 | 99.831 | 97.067 | 91.159 | 82.984 |
| 91 | 100.000 | 99.831 | 97.161 | 91.357 | 83.266 |
| 92 | 100.000 | 99.831 | 97.231 | 91.529 | 83.529 |
| 93 | 100.000 | 99.831 | 97.278 | 91.657 | 83.749 |
| 94 | 100.000 | 99.831 | 97.349 | 91.823 | 84.002 |
| 95 | 100.000 | 99.831 | 97.396 | 91.952 | 84.221 |
| 96 | 100.000 | 99.831 | 97.443 | 92.081 | 84.439 |
| 97 | 100.000 | 99.831 | 97.536 | 92.274 | 84.707 |
| 98 | 100.000 | 99.831 | 97.630 | 92.467 | 84.975 |
| 99 | 100.000 | 99.831 | 97.677 | 92.596 | 85.190 |
| 100 | 100.000 | 99.831 | 97.677 | 92.660 | 85.352 |
| 101 | 100.000 | 99.831 | 97.724 | 92.783 | 85.558 |
| 102 | 100.000 | 99.831 | 97.771 | 92.907 | 85.763 |
| 103 | 100.000 | 99.831 | 97.841 | 93.062 | 85.994 |
| 104 | 100.000 | 99.831 | 97.888 | 93.185 | 86.199 |
| 105 | 100.000 | 99.831 | 97.959 | 93.341 | 86.429 |
| 106 | 100.000 | 99.831 | 98.029 | 93.496 | 86.658 |
| 107 | 100.000 | 99.831 | 98.099 | 93.652 | 86.887 |
| 108 | 100.000 | 99.831 | 98.146 | 93.775 | 87.089 |
| 109 | 100.000 | 99.831 | 98.146 | 93.839 | 87.244 |
| 110 | 100.000 | 99.831 | 98.170 | 93.925 | 87.411 |
| 111 | 100.000 | 100.000 | 98.428 | 94.220 | 87.716 |
| 112 | 100.000 | 100.000 | 98.475 | 94.338 | 87.906 |
| 113 | 100.000 | 100.000 | 98.522 | 94.456 | 88.097 |
| 114 | 100.000 | 100.000 | 98.592 | 94.606 | 88.313 |
| 115 | 100.000 | 100.000 | 98.663 | 94.756 | 88.529 |
| 116 | 100.000 | 100.000 | 98.710 | 94.874 | 88.720 |
| 117 | 100.000 | 100.000 | 98.780 | 95.024 | 88.936 |
| 118 | 100.000 | 100.000 | 98.780 | 95.078 | 89.075 |
| 119 | 100.000 | 100.000 | 98.850 | 95.228 | 89.291 |
| 120 | 100.000 | 100.000 | 98.850 | 95.282 | 89.430 |
| 121 | 100.000 | 100.000 | 98.874 | 95.362 | 89.583 |
| 122 | 100.000 | 100.000 | 98.921 | 95.475 | 89.763 |
| 123 | 100.000 | 100.000 | 98.968 | 95.587 | 89.940 |
| 124 | 100.000 | 100.000 | 99.015 | 95.700 | 90.118 |
| 125 | 100.000 | 100.000 | 99.061 | 95.813 | 90.295 |
| 126 | 100.000 | 100.000 | 99.085 | 95.888 | 90.438 |
| 127 | 100.000 | 100.000 | 99.085 | 95.931 | 90.552 |
| 128 | 100.000 | 100.000 | 99.132 | 96.038 | 90.719 |
| 129 | 100.000 | 100.000 | 99.179 | 96.145 | 90.883 |
| 130 | 100.000 | 100.000 | 99.226 | 96.252 | 91.048 |
| 131 | 100.000 | 100.000 | 99.273 | 96.359 | 91.213 |
| 132 | 100.000 | 100.000 | 99.273 | 96.402 | 91.323 |
| 133 | 100.000 | 100.000 | 99.296 | 96.472 | 91.453 |
| 134 | 100.000 | 100.000 | 99.320 | 96.542 | 91.581 |
| 135 | 100.000 | 100.000 | 99.343 | 96.611 | 91.708 |
| 136 | 100.000 | 100.000 | 99.366 | 96.681 | 91.834 |
| 137 | 100.000 | 100.000 | 99.413 | 96.783 | 91.986 |
| 138 | 100.000 | 100.000 | 99.437 | 96.853 | 92.111 |
| 139 | 100.000 | 100.000 | 99.460 | 96.922 | 92.237 |
| 140 | 100.000 | 100.000 | 99.484 | 96.992 | 92.363 |
| 141 | 100.000 | 100.000 | 99.531 | 97.094 | 92.513 |
| 142 | 100.000 | 100.000 | 99.554 | 97.158 | 92.629 |
| 143 | 100.000 | 100.000 | 99.554 | 97.190 | 92.720 |
| 144 | 100.000 | 100.000 | 99.601 | 97.292 | 92.869 |
| 145 | 100.000 | 100.000 | 99.648 | 97.394 | 93.019 |
| 146 | 100.000 | 100.000 | 99.648 | 97.421 | 93.102 |
| 147 | 100.000 | 100.000 | 99.648 | 97.453 | 93.189 |
| 148 | 100.000 | 100.000 | 99.672 | 97.518 | 93.302 |
| 149 | 100.000 | 100.000 | 99.672 | 97.550 | 93.389 |
| 150 | 100.000 | 100.000 | 99.672 | 97.577 | 93.465 |
| 151 | 100.000 | 100.000 | 99.672 | 97.603 | 93.541 |
| 152 | 100.000 | 100.000 | 99.672 | 97.630 | 93.615 |
| 153 | 100.000 | 100.000 | 99.695 | 97.689 | 93.715 |
| 154 | 100.000 | 100.000 | 99.718 | 97.748 | 93.814 |
| 155 | 100.000 | 100.000 | 99.718 | 97.775 | 93.888 |
| 156 | 100.000 | 100.000 | 99.742 | 97.834 | 93.986 |
| 157 | 100.000 | 100.000 | 99.742 | 97.861 | 94.058 |
| 158 | 100.000 | 100.000 | 99.765 | 97.920 | 94.156 |
| 159 | 100.000 | 100.000 | 99.789 | 97.979 | 94.254 |
| 160 | 100.000 | 100.000 | 99.812 | 98.038 | 94.352 |
| 161 | 100.000 | 100.000 | 99.836 | 98.097 | 94.451 |
| 162 | 100.000 | 100.000 | 99.859 | 98.156 | 94.549 |
| 163 | 100.000 | 100.000 | 99.859 | 98.182 | 94.621 |
| 164 | 100.000 | 100.000 | 99.859 | 98.209 | 94.693 |
| 165 | 100.000 | 100.000 | 99.883 | 98.268 | 94.791 |
| 166 | 100.000 | 100.000 | 99.883 | 98.295 | 94.863 |
| 167 | 100.000 | 100.000 | 99.883 | 98.322 | 94.935 |
| 168 | 100.000 | 100.000 | 99.883 | 98.343 | 94.998 |
| 169 | 100.000 | 100.000 | 99.883 | 98.365 | 95.061 |
| 170 | 100.000 | 100.000 | 99.883 | 98.386 | 95.124 |
| 171 | 100.000 | 100.000 | 99.883 | 98.408 | 95.186 |
| 172 | 100.000 | 100.000 | 99.883 | 98.429 | 95.249 |
| 173 | 100.000 | 100.000 | 99.906 | 98.483 | 95.336 |
| 174 | 100.000 | 100.000 | 99.906 | 98.504 | 95.397 |
| 175 | 100.000 | 100.000 | 99.906 | 98.526 | 95.458 |
| 176 | 100.000 | 100.000 | 99.906 | 98.536 | 95.505 |
| 177 | 100.000 | 100.000 | 99.906 | 98.558 | 95.564 |
| 178 | 100.000 | 100.000 | 99.906 | 98.579 | 95.623 |
| 179 | 100.000 | 100.000 | 99.906 | 98.601 | 95.682 |
| 180 | 100.000 | 100.000 | 99.906 | 98.622 | 95.741 |
| 181 | 100.000 | 100.000 | 99.906 | 98.644 | 95.800 |
| 182 | 100.000 | 100.000 | 99.906 | 98.665 | 95.860 |
| 183 | 100.000 | 100.000 | 99.930 | 98.719 | 95.945 |
| 184 | 100.000 | 100.000 | 99.930 | 98.740 | 96.004 |
| 185 | 100.000 | 100.000 | 99.953 | 98.794 | 96.089 |
| 186 | 100.000 | 100.000 | 99.953 | 98.810 | 96.141 |
| 187 | 100.000 | 100.000 | 99.953 | 98.831 | 96.200 |
| 188 | 100.000 | 100.000 | 99.953 | 98.853 | 96.259 |
| 189 | 100.000 | 100.000 | 99.953 | 98.874 | 96.318 |
| 190 | 100.000 | 100.000 | 99.953 | 98.896 | 96.377 |
| 191 | 100.000 | 100.000 | 99.953 | 98.917 | 96.437 |
| 192 | 100.000 | 100.000 | 99.977 | 98.971 | 96.520 |
| 193 | 100.000 | 100.000 | 100.000 | 99.024 | 96.603 |
| 194 | 100.000 | 100.000 | 100.000 | 99.040 | 96.653 |
| 195 | 100.000 | 100.000 | 100.000 | 99.056 | 96.701 |
| 196 | 100.000 | 100.000 | 100.000 | 99.072 | 96.749 |
| 197 | 100.000 | 100.000 | 100.000 | 99.089 | 96.797 |
| 198 | 100.000 | 100.000 | 100.000 | 99.105 | 96.845 |
| 199 | 100.000 | 100.000 | 100.000 | 99.121 | 96.893 |
| 200 | 100.000 | 100.000 | 100.000 | 99.137 | 96.940 |
| 201 | 100.000 | 100.000 | 100.000 | 99.153 | 96.986 |
| 202 | 100.000 | 100.000 | 100.000 | 99.169 | 97.032 |
| 203 | 100.000 | 100.000 | 100.000 | 99.185 | 97.078 |
| 204 | 100.000 | 100.000 | 100.000 | 99.201 | 97.124 |
| 205 | 100.000 | 100.000 | 100.000 | 99.217 | 97.171 |
| 206 | 100.000 | 100.000 | 100.000 | 99.233 | 97.217 |
| 207 | 100.000 | 100.000 | 100.000 | 99.249 | 97.263 |
| 208 | 100.000 | 100.000 | 100.000 | 99.265 | 97.309 |
| 209 | 100.000 | 100.000 | 100.000 | 99.282 | 97.354 |
| 210 | 100.000 | 100.000 | 100.000 | 99.292 | 97.389 |
| 211 | 100.000 | 100.000 | 100.000 | 99.303 | 97.424 |
| 212 | 100.000 | 100.000 | 100.000 | 99.314 | 97.459 |
| 213 | 100.000 | 100.000 | 100.000 | 99.324 | 97.492 |
| 214 | 100.000 | 100.000 | 100.000 | 99.335 | 97.526 |
| 215 | 100.000 | 100.000 | 100.000 | 99.346 | 97.559 |
| 216 | 100.000 | 100.000 | 100.000 | 99.357 | 97.592 |
| 217 | 100.000 | 100.000 | 100.000 | 99.367 | 97.626 |
| 218 | 100.000 | 100.000 | 100.000 | 99.378 | 97.659 |
| 219 | 100.000 | 100.000 | 100.000 | 99.389 | 97.690 |
| 220 | 100.000 | 100.000 | 100.000 | 99.399 | 97.722 |
| 221 | 100.000 | 100.000 | 100.000 | 99.410 | 97.753 |
| 222 | 100.000 | 100.000 | 100.000 | 99.421 | 97.785 |
| 223 | 100.000 | 100.000 | 100.000 | 99.432 | 97.816 |
| 224 | 100.000 | 100.000 | 100.000 | 99.442 | 97.848 |
| 225 | 100.000 | 100.000 | 100.000 | 99.453 | 97.879 |
| 226 | 100.000 | 100.000 | 100.000 | 99.464 | 97.910 |
| 227 | 100.000 | 100.000 | 100.000 | 99.475 | 97.942 |
| 228 | 100.000 | 100.000 | 100.000 | 99.485 | 97.973 |
| 229 | 100.000 | 100.000 | 100.000 | 99.496 | 98.005 |
| 230 | 100.000 | 100.000 | 100.000 | 99.507 | 98.036 |
| 231 | 100.000 | 100.000 | 100.000 | 99.517 | 98.068 |
| 232 | 100.000 | 100.000 | 100.000 | 99.528 | 98.099 |
| 233 | 100.000 | 100.000 | 100.000 | 99.539 | 98.130 |
| 234 | 100.000 | 100.000 | 100.000 | 99.550 | 98.162 |
| 235 | 100.000 | 100.000 | 100.000 | 99.560 | 98.193 |
| 236 | 100.000 | 100.000 | 100.000 | 99.571 | 98.225 |
| 237 | 100.000 | 100.000 | 100.000 | 99.582 | 98.256 |
| 238 | 100.000 | 100.000 | 100.000 | 99.593 | 98.288 |
| 239 | 100.000 | 100.000 | 100.000 | 99.603 | 98.319 |
| 240 | 100.000 | 100.000 | 100.000 | 99.614 | 98.351 |
| 241 | 100.000 | 100.000 | 100.000 | 99.625 | 98.382 |
| 242 | 100.000 | 100.000 | 100.000 | 99.635 | 98.413 |
| 243 | 100.000 | 100.000 | 100.000 | 99.646 | 98.445 |
| 244 | 100.000 | 100.000 | 100.000 | 99.657 | 98.476 |
| 245 | 100.000 | 100.000 | 100.000 | 99.662 | 98.497 |
| 246 | 100.000 | 100.000 | 100.000 | 99.668 | 98.517 |
| 247 | 100.000 | 100.000 | 100.000 | 99.673 | 98.535 |
| 248 | 100.000 | 100.000 | 100.000 | 99.678 | 98.554 |
| 249 | 100.000 | 100.000 | 100.000 | 99.684 | 98.572 |
| 250 | 100.000 | 100.000 | 100.000 | 99.689 | 98.591 |
| 251 | 100.000 | 100.000 | 100.000 | 99.694 | 98.609 |
| 252 | 100.000 | 100.000 | 100.000 | 99.700 | 98.628 |
| 253 | 100.000 | 100.000 | 100.000 | 99.705 | 98.645 |
| 254 | 100.000 | 100.000 | 100.000 | 99.710 | 98.661 |
| 255 | 100.000 | 100.000 | 100.000 | 99.710 | 98.670 |
| 256 | 100.000 | 100.000 | 100.000 | 99.716 | 98.687 |
| 257 | 100.000 | 100.000 | 100.000 | 99.721 | 98.704 |
| 258 | 100.000 | 100.000 | 100.000 | 99.727 | 98.720 |
| 259 | 100.000 | 100.000 | 100.000 | 99.732 | 98.737 |
| 260 | 100.000 | 100.000 | 100.000 | 99.737 | 98.754 |
| 261 | 100.000 | 100.000 | 100.000 | 99.743 | 98.770 |
| 262 | 100.000 | 100.000 | 100.000 | 99.748 | 98.787 |
| 263 | 100.000 | 100.000 | 100.000 | 99.753 | 98.804 |
| 264 | 100.000 | 100.000 | 100.000 | 99.759 | 98.820 |
| 265 | 100.000 | 100.000 | 100.000 | 99.764 | 98.837 |
| 266 | 100.000 | 100.000 | 100.000 | 99.769 | 98.853 |
| 267 | 100.000 | 100.000 | 100.000 | 99.775 | 98.870 |
| 268 | 100.000 | 100.000 | 100.000 | 99.780 | 98.887 |
| 269 | 100.000 | 100.000 | 100.000 | 99.786 | 98.903 |
| 270 | 100.000 | 100.000 | 100.000 | 99.791 | 98.920 |
| 271 | 100.000 | 100.000 | 100.000 | 99.796 | 98.937 |
| 272 | 100.000 | 100.000 | 100.000 | 99.802 | 98.953 |
| 273 | 100.000 | 100.000 | 100.000 | 99.807 | 98.970 |
| 274 | 100.000 | 100.000 | 100.000 | 99.812 | 98.987 |
| 275 | 100.000 | 100.000 | 100.000 | 99.818 | 99.003 |
| 276 | 100.000 | 100.000 | 100.000 | 99.823 | 99.020 |
| 277 | 100.000 | 100.000 | 100.000 | 99.828 | 99.037 |
| 278 | 100.000 | 100.000 | 100.000 | 99.834 | 99.053 |
| 279 | 100.000 | 100.000 | 100.000 | 99.839 | 99.070 |
| 280 | 100.000 | 100.000 | 100.000 | 99.845 | 99.086 |
| 281 | 100.000 | 100.000 | 100.000 | 99.850 | 99.103 |
| 282 | 100.000 | 100.000 | 100.000 | 99.855 | 99.120 |
| 283 | 100.000 | 100.000 | 100.000 | 99.861 | 99.136 |
| 284 | 100.000 | 100.000 | 100.000 | 99.866 | 99.153 |
| 285 | 100.000 | 100.000 | 100.000 | 99.871 | 99.170 |
| 286 | 100.000 | 100.000 | 100.000 | 99.877 | 99.186 |
| 287 | 100.000 | 100.000 | 100.000 | 99.882 | 99.203 |
| 288 | 100.000 | 100.000 | 100.000 | 99.887 | 99.220 |
| 289 | 100.000 | 100.000 | 100.000 | 99.893 | 99.236 |
| 290 | 100.000 | 100.000 | 100.000 | 99.898 | 99.253 |
| 291 | 100.000 | 100.000 | 100.000 | 99.903 | 99.270 |
| 292 | 100.000 | 100.000 | 100.000 | 99.909 | 99.286 |
| 293 | 100.000 | 100.000 | 100.000 | 99.914 | 99.303 |
| 294 | 100.000 | 100.000 | 100.000 | 99.920 | 99.319 |
| 295 | 100.000 | 100.000 | 100.000 | 99.925 | 99.336 |
| 296 | 100.000 | 100.000 | 100.000 | 99.930 | 99.353 |
| 297 | 100.000 | 100.000 | 100.000 | 99.936 | 99.369 |
| 298 | 100.000 | 100.000 | 100.000 | 99.941 | 99.386 |
| 299 | 100.000 | 100.000 | 100.000 | 99.946 | 99.403 |
| 300 | 100.000 | 100.000 | 100.000 | 99.952 | 99.419 |
| 301 | 100.000 | 100.000 | 100.000 | 99.957 | 99.436 |
| 302 | 100.000 | 100.000 | 100.000 | 99.962 | 99.453 |
| 303 | 100.000 | 100.000 | 100.000 | 99.968 | 99.469 |
| 304 | 100.000 | 100.000 | 100.000 | 99.973 | 99.486 |
| 305 | 100.000 | 100.000 | 100.000 | 99.979 | 99.503 |
| 306 | 100.000 | 100.000 | 100.000 | 99.984 | 99.519 |
| 307 | 100.000 | 100.000 | 100.000 | 99.989 | 99.536 |
| 308 | 100.000 | 100.000 | 100.000 | 99.995 | 99.552 |
| 309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.569 |
| 310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.577 |
| 311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.584 |
| 312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.591 |
| 313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.597 |
| 314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
| 315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.604 |
| 316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.608 |
| 317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.612 |
| 318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
| 319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.619 |
| 320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
| 321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
| 322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
| 323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.634 |
| 324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
| 325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
| 326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.645 |
| 327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.649 |
| 328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.652 |
| 329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.656 |
| 330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
| 331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.660 |
| 332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
| 333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.663 |
| 334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
| 335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.667 |
| 336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.669 |
| 337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.671 |
| 338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
| 339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.675 |
| 340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.676 |
| 341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
| 342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
| 343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.682 |
| 344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
| 345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
| 346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
| 347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.689 |
| 348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
| 349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.693 |
| 350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
| 351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.697 |
| 352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
| 353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
| 354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.702 |
| 355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
grep
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 50.000 | 24.704 | 13.186 | 7.640 | 4.758 |
| 3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
| 4 | 73.810 | 42.809 | 24.801 | 14.943 | 9.455 |
| 5 | 83.333 | 50.761 | 30.221 | 18.482 | 11.778 |
| 6 | 85.714 | 55.499 | 34.444 | 21.618 | 13.980 |
| 7 | 92.857 | 61.421 | 38.691 | 24.588 | 16.062 |
| 8 | 95.238 | 65.990 | 42.820 | 27.693 | 18.257 |
| 9 | 97.619 | 70.051 | 46.574 | 30.610 | 20.378 |
| 10 | 100.000 | 73.773 | 49.883 | 33.156 | 22.255 |
| 11 | 100.000 | 75.804 | 52.346 | 35.365 | 24.032 |
| 12 | 100.000 | 77.834 | 54.974 | 37.773 | 25.959 |
| 13 | 100.000 | 79.865 | 57.461 | 40.025 | 27.781 |
| 14 | 100.000 | 81.557 | 59.596 | 42.008 | 29.427 |
| 15 | 100.000 | 82.910 | 61.427 | 43.805 | 30.982 |
| 16 | 100.000 | 84.264 | 63.257 | 45.590 | 32.517 |
| 17 | 100.000 | 85.618 | 65.275 | 47.601 | 34.223 |
| 18 | 100.000 | 86.802 | 66.964 | 49.327 | 35.758 |
| 19 | 100.000 | 87.817 | 68.419 | 50.818 | 37.097 |
| 20 | 100.000 | 88.832 | 69.826 | 52.228 | 38.345 |
| 21 | 100.000 | 89.679 | 71.187 | 53.708 | 39.710 |
| 22 | 100.000 | 90.525 | 72.478 | 55.096 | 41.004 |
| 23 | 100.000 | 91.371 | 73.698 | 56.367 | 42.169 |
| 24 | 100.000 | 92.217 | 74.918 | 57.648 | 43.359 |
| 25 | 100.000 | 92.893 | 75.903 | 58.699 | 44.353 |
| 26 | 100.000 | 93.570 | 76.983 | 59.906 | 45.515 |
| 27 | 100.000 | 94.078 | 77.992 | 61.123 | 46.715 |
| 28 | 100.000 | 94.585 | 78.930 | 62.243 | 47.830 |
| 29 | 100.000 | 95.093 | 79.775 | 63.214 | 48.788 |
| 30 | 100.000 | 95.601 | 80.690 | 64.308 | 49.888 |
| 31 | 100.000 | 96.108 | 81.417 | 65.074 | 50.609 |
| 32 | 100.000 | 96.447 | 82.309 | 66.243 | 51.800 |
| 33 | 100.000 | 96.785 | 82.802 | 66.779 | 52.327 |
| 34 | 100.000 | 97.124 | 83.623 | 67.836 | 53.400 |
| 35 | 100.000 | 97.462 | 84.280 | 68.656 | 54.254 |
| 36 | 100.000 | 97.800 | 84.984 | 69.557 | 55.199 |
| 37 | 100.000 | 98.139 | 85.476 | 70.103 | 55.750 |
| 38 | 100.000 | 98.477 | 85.969 | 70.640 | 56.277 |
| 39 | 100.000 | 98.646 | 86.368 | 71.192 | 56.897 |
| 40 | 100.000 | 98.816 | 86.649 | 71.540 | 57.281 |
| 41 | 100.000 | 98.985 | 87.236 | 72.398 | 58.221 |
| 42 | 100.000 | 99.154 | 87.565 | 72.838 | 58.716 |
| 43 | 100.000 | 99.323 | 88.104 | 73.626 | 59.593 |
| 44 | 100.000 | 99.492 | 88.456 | 74.114 | 60.159 |
| 45 | 100.000 | 99.662 | 88.761 | 74.500 | 60.582 |
| 46 | 100.000 | 99.831 | 89.019 | 74.795 | 60.887 |
| 47 | 100.000 | 100.000 | 89.277 | 75.090 | 61.192 |
| 48 | 100.000 | 100.000 | 89.465 | 75.433 | 61.618 |
| 49 | 100.000 | 100.000 | 89.676 | 75.830 | 62.123 |
| 50 | 100.000 | 100.000 | 89.887 | 76.248 | 62.672 |
| 51 | 100.000 | 100.000 | 90.099 | 76.661 | 63.212 |
| 52 | 100.000 | 100.000 | 90.216 | 76.897 | 63.532 |
| 53 | 100.000 | 100.000 | 90.333 | 77.133 | 63.852 |
| 54 | 100.000 | 100.000 | 90.544 | 77.556 | 64.416 |
| 55 | 100.000 | 100.000 | 90.591 | 77.679 | 64.617 |
| 56 | 100.000 | 100.000 | 90.779 | 78.044 | 65.094 |
| 57 | 100.000 | 100.000 | 90.967 | 78.409 | 65.571 |
| 58 | 100.000 | 100.000 | 91.154 | 78.763 | 66.021 |
| 59 | 100.000 | 100.000 | 91.248 | 78.966 | 66.313 |
| 60 | 100.000 | 100.000 | 91.412 | 79.342 | 66.851 |
| 61 | 100.000 | 100.000 | 91.483 | 79.502 | 67.091 |
| 62 | 100.000 | 100.000 | 91.835 | 80.157 | 67.899 |
| 63 | 100.000 | 100.000 | 92.163 | 80.752 | 68.624 |
| 64 | 100.000 | 100.000 | 92.210 | 80.870 | 68.815 |
| 65 | 100.000 | 100.000 | 92.374 | 81.207 | 69.277 |
| 66 | 100.000 | 100.000 | 92.445 | 81.363 | 69.506 |
| 67 | 100.000 | 100.000 | 92.656 | 81.781 | 70.058 |
| 68 | 100.000 | 100.000 | 92.750 | 81.974 | 70.327 |
| 69 | 100.000 | 100.000 | 92.961 | 82.382 | 70.856 |
| 70 | 100.000 | 100.000 | 93.008 | 82.484 | 71.006 |
| 71 | 100.000 | 100.000 | 93.219 | 82.929 | 71.611 |
| 72 | 100.000 | 100.000 | 93.360 | 83.202 | 71.970 |
| 73 | 100.000 | 100.000 | 93.430 | 83.368 | 72.227 |
| 74 | 100.000 | 100.000 | 93.430 | 83.433 | 72.382 |
| 75 | 100.000 | 100.000 | 93.454 | 83.497 | 72.493 |
| 76 | 100.000 | 100.000 | 93.524 | 83.642 | 72.694 |
| 77 | 100.000 | 100.000 | 93.618 | 83.845 | 72.990 |
| 78 | 100.000 | 100.000 | 93.665 | 83.979 | 73.216 |
| 79 | 100.000 | 100.000 | 93.853 | 84.333 | 73.665 |
| 80 | 100.000 | 100.000 | 94.134 | 84.848 | 74.300 |
| 81 | 100.000 | 100.000 | 94.228 | 85.062 | 74.620 |
| 82 | 100.000 | 100.000 | 94.298 | 85.234 | 74.886 |
| 83 | 100.000 | 100.000 | 94.345 | 85.347 | 75.063 |
| 84 | 100.000 | 100.000 | 94.369 | 85.406 | 75.161 |
| 85 | 100.000 | 100.000 | 94.416 | 85.518 | 75.339 |
| 86 | 100.000 | 100.000 | 94.510 | 85.717 | 75.618 |
| 87 | 100.000 | 100.000 | 94.603 | 85.899 | 75.858 |
| 88 | 100.000 | 100.000 | 94.603 | 85.942 | 75.969 |
| 89 | 100.000 | 100.000 | 94.603 | 85.958 | 76.016 |
| 90 | 100.000 | 100.000 | 94.627 | 86.033 | 76.154 |
| 91 | 100.000 | 100.000 | 94.791 | 86.371 | 76.617 |
| 92 | 100.000 | 100.000 | 94.791 | 86.398 | 76.689 |
| 93 | 100.000 | 100.000 | 94.838 | 86.521 | 76.894 |
| 94 | 100.000 | 100.000 | 94.908 | 86.676 | 77.123 |
| 95 | 100.000 | 100.000 | 94.955 | 86.789 | 77.301 |
| 96 | 100.000 | 100.000 | 95.049 | 86.982 | 77.569 |
| 97 | 100.000 | 100.000 | 95.096 | 87.116 | 77.795 |
| 98 | 100.000 | 100.000 | 95.143 | 87.223 | 77.957 |
| 99 | 100.000 | 100.000 | 95.167 | 87.282 | 78.055 |
| 100 | 100.000 | 100.000 | 95.260 | 87.475 | 78.324 |
| 101 | 100.000 | 100.000 | 95.448 | 87.845 | 78.814 |
| 102 | 100.000 | 100.000 | 95.495 | 87.974 | 79.026 |
| 103 | 100.000 | 100.000 | 95.612 | 88.226 | 79.385 |
| 104 | 100.000 | 100.000 | 95.659 | 88.349 | 79.587 |
| 105 | 100.000 | 100.000 | 95.753 | 88.564 | 79.899 |
| 106 | 100.000 | 100.000 | 95.753 | 88.596 | 79.990 |
| 107 | 100.000 | 100.000 | 95.777 | 88.665 | 80.115 |
| 108 | 100.000 | 100.000 | 95.824 | 88.778 | 80.293 |
| 109 | 100.000 | 100.000 | 95.824 | 88.853 | 80.472 |
| 110 | 100.000 | 100.000 | 95.847 | 88.917 | 80.583 |
| 111 | 100.000 | 100.000 | 95.894 | 89.030 | 80.761 |
| 112 | 100.000 | 100.000 | 95.964 | 89.202 | 81.025 |
| 113 | 100.000 | 100.000 | 96.082 | 89.443 | 81.360 |
| 114 | 100.000 | 100.000 | 96.082 | 89.486 | 81.471 |
| 115 | 100.000 | 100.000 | 96.129 | 89.598 | 81.648 |
| 116 | 100.000 | 100.000 | 96.129 | 89.614 | 81.693 |
| 117 | 100.000 | 100.000 | 96.152 | 89.673 | 81.791 |
| 118 | 100.000 | 100.000 | 96.152 | 89.684 | 81.822 |
| 119 | 100.000 | 100.000 | 96.222 | 89.829 | 82.024 |
| 120 | 100.000 | 100.000 | 96.246 | 89.888 | 82.125 |
| 121 | 100.000 | 100.000 | 96.340 | 90.092 | 82.418 |
| 122 | 100.000 | 100.000 | 96.410 | 90.242 | 82.634 |
| 123 | 100.000 | 100.000 | 96.410 | 90.269 | 82.708 |
| 124 | 100.000 | 100.000 | 96.457 | 90.370 | 82.858 |
| 125 | 100.000 | 100.000 | 96.527 | 90.521 | 83.076 |
| 126 | 100.000 | 100.000 | 96.574 | 90.622 | 83.229 |
| 127 | 100.000 | 100.000 | 96.598 | 90.676 | 83.313 |
| 128 | 100.000 | 100.000 | 96.598 | 90.730 | 83.448 |
| 129 | 100.000 | 100.000 | 96.668 | 90.891 | 83.692 |
| 130 | 100.000 | 100.000 | 96.715 | 90.998 | 83.855 |
| 131 | 100.000 | 100.000 | 96.786 | 91.143 | 84.058 |
| 132 | 100.000 | 100.000 | 96.809 | 91.202 | 84.158 |
| 133 | 100.000 | 100.000 | 96.832 | 91.261 | 84.256 |
| 134 | 100.000 | 100.000 | 96.832 | 91.271 | 84.287 |
| 135 | 100.000 | 100.000 | 96.856 | 91.325 | 84.370 |
| 136 | 100.000 | 100.000 | 96.856 | 91.346 | 84.430 |
| 137 | 100.000 | 100.000 | 96.856 | 91.357 | 84.461 |
| 138 | 100.000 | 100.000 | 96.879 | 91.421 | 84.572 |
| 139 | 100.000 | 100.000 | 96.903 | 91.502 | 84.722 |
| 140 | 100.000 | 100.000 | 96.950 | 91.614 | 84.905 |
| 141 | 100.000 | 100.000 | 96.950 | 91.625 | 84.936 |
| 142 | 100.000 | 100.000 | 96.997 | 91.727 | 85.086 |
| 143 | 100.000 | 100.000 | 97.020 | 91.791 | 85.197 |
| 144 | 100.000 | 100.000 | 97.044 | 91.845 | 85.280 |
| 145 | 100.000 | 100.000 | 97.067 | 91.899 | 85.363 |
| 146 | 100.000 | 100.000 | 97.114 | 92.011 | 85.541 |
| 147 | 100.000 | 100.000 | 97.231 | 92.242 | 85.848 |
| 148 | 100.000 | 100.000 | 97.278 | 92.365 | 86.050 |
| 149 | 100.000 | 100.000 | 97.278 | 92.381 | 86.094 |
| 150 | 100.000 | 100.000 | 97.325 | 92.499 | 86.286 |
| 151 | 100.000 | 100.000 | 97.372 | 92.606 | 86.449 |
| 152 | 100.000 | 100.000 | 97.443 | 92.767 | 86.689 |
| 153 | 100.000 | 100.000 | 97.443 | 92.815 | 86.813 |
| 154 | 100.000 | 100.000 | 97.466 | 92.869 | 86.898 |
| 155 | 100.000 | 100.000 | 97.466 | 92.885 | 86.945 |
| 156 | 100.000 | 100.000 | 97.466 | 92.896 | 86.976 |
| 157 | 100.000 | 100.000 | 97.489 | 92.949 | 87.059 |
| 158 | 100.000 | 100.000 | 97.489 | 92.960 | 87.091 |
| 159 | 100.000 | 100.000 | 97.513 | 93.041 | 87.242 |
| 160 | 100.000 | 100.000 | 97.536 | 93.100 | 87.340 |
| 161 | 100.000 | 100.000 | 97.536 | 93.121 | 87.399 |
| 162 | 100.000 | 100.000 | 97.536 | 93.142 | 87.459 |
| 163 | 100.000 | 100.000 | 97.607 | 93.293 | 87.675 |
| 164 | 100.000 | 100.000 | 97.607 | 93.298 | 87.692 |
| 165 | 100.000 | 100.000 | 97.630 | 93.352 | 87.775 |
| 166 | 100.000 | 100.000 | 97.724 | 93.550 | 88.056 |
| 167 | 100.000 | 100.000 | 97.724 | 93.604 | 88.191 |
| 168 | 100.000 | 100.000 | 97.724 | 93.614 | 88.222 |
| 169 | 100.000 | 100.000 | 97.748 | 93.689 | 88.361 |
| 170 | 100.000 | 100.000 | 97.771 | 93.743 | 88.444 |
| 171 | 100.000 | 100.000 | 97.818 | 93.845 | 88.594 |
| 172 | 100.000 | 100.000 | 97.818 | 93.877 | 88.681 |
| 173 | 100.000 | 100.000 | 97.818 | 93.898 | 88.740 |
| 174 | 100.000 | 100.000 | 97.841 | 93.952 | 88.823 |
| 175 | 100.000 | 100.000 | 97.888 | 94.059 | 88.988 |
| 176 | 100.000 | 100.000 | 97.888 | 94.070 | 89.019 |
| 177 | 100.000 | 100.000 | 97.888 | 94.070 | 89.023 |
| 178 | 100.000 | 100.000 | 97.888 | 94.081 | 89.054 |
| 179 | 100.000 | 100.000 | 97.888 | 94.102 | 89.117 |
| 180 | 100.000 | 100.000 | 97.888 | 94.129 | 89.189 |
| 181 | 100.000 | 100.000 | 97.912 | 94.188 | 89.289 |
| 182 | 100.000 | 100.000 | 97.912 | 94.188 | 89.291 |
| 183 | 100.000 | 100.000 | 97.912 | 94.204 | 89.339 |
| 184 | 100.000 | 100.000 | 97.912 | 94.226 | 89.400 |
| 185 | 100.000 | 100.000 | 97.912 | 94.236 | 89.432 |
| 186 | 100.000 | 100.000 | 97.912 | 94.236 | 89.434 |
| 187 | 100.000 | 100.000 | 97.912 | 94.242 | 89.450 |
| 188 | 100.000 | 100.000 | 97.912 | 94.258 | 89.496 |
| 189 | 100.000 | 100.000 | 97.912 | 94.258 | 89.504 |
| 190 | 100.000 | 100.000 | 97.935 | 94.311 | 89.591 |
| 191 | 100.000 | 100.000 | 97.935 | 94.322 | 89.622 |
| 192 | 100.000 | 100.000 | 97.935 | 94.333 | 89.654 |
| 193 | 100.000 | 100.000 | 98.006 | 94.483 | 89.870 |
| 194 | 100.000 | 100.000 | 98.099 | 94.676 | 90.138 |
| 195 | 100.000 | 100.000 | 98.099 | 94.687 | 90.170 |
| 196 | 100.000 | 100.000 | 98.099 | 94.719 | 90.260 |
| 197 | 100.000 | 100.000 | 98.123 | 94.778 | 90.358 |
| 198 | 100.000 | 100.000 | 98.170 | 94.880 | 90.508 |
| 199 | 100.000 | 100.000 | 98.193 | 94.933 | 90.591 |
| 200 | 100.000 | 100.000 | 98.264 | 95.078 | 90.793 |
| 201 | 100.000 | 100.000 | 98.358 | 95.271 | 91.061 |
| 202 | 100.000 | 100.000 | 98.358 | 95.287 | 91.109 |
| 203 | 100.000 | 100.000 | 98.381 | 95.351 | 91.220 |
| 204 | 100.000 | 100.000 | 98.428 | 95.480 | 91.434 |
| 205 | 100.000 | 100.000 | 98.451 | 95.534 | 91.518 |
| 206 | 100.000 | 100.000 | 98.451 | 95.555 | 91.577 |
| 207 | 100.000 | 100.000 | 98.451 | 95.566 | 91.612 |
| 208 | 100.000 | 100.000 | 98.451 | 95.587 | 91.671 |
| 209 | 100.000 | 100.000 | 98.451 | 95.587 | 91.679 |
| 210 | 100.000 | 100.000 | 98.451 | 95.587 | 91.680 |
| 211 | 100.000 | 100.000 | 98.451 | 95.603 | 91.727 |
| 212 | 100.000 | 100.000 | 98.522 | 95.754 | 91.943 |
| 213 | 100.000 | 100.000 | 98.522 | 95.759 | 91.961 |
| 214 | 100.000 | 100.000 | 98.592 | 95.904 | 92.163 |
| 215 | 100.000 | 100.000 | 98.592 | 95.925 | 92.222 |
| 216 | 100.000 | 100.000 | 98.592 | 95.936 | 92.254 |
| 217 | 100.000 | 100.000 | 98.592 | 95.941 | 92.272 |
| 218 | 100.000 | 100.000 | 98.592 | 95.947 | 92.289 |
| 219 | 100.000 | 100.000 | 98.592 | 95.952 | 92.307 |
| 220 | 100.000 | 100.000 | 98.592 | 95.952 | 92.315 |
| 221 | 100.000 | 100.000 | 98.592 | 95.963 | 92.350 |
| 222 | 100.000 | 100.000 | 98.616 | 96.016 | 92.433 |
| 223 | 100.000 | 100.000 | 98.616 | 96.022 | 92.450 |
| 224 | 100.000 | 100.000 | 98.663 | 96.124 | 92.599 |
| 225 | 100.000 | 100.000 | 98.663 | 96.140 | 92.646 |
| 226 | 100.000 | 100.000 | 98.663 | 96.156 | 92.692 |
| 227 | 100.000 | 100.000 | 98.710 | 96.258 | 92.842 |
| 228 | 100.000 | 100.000 | 98.710 | 96.258 | 92.849 |
| 229 | 100.000 | 100.000 | 98.710 | 96.263 | 92.866 |
| 230 | 100.000 | 100.000 | 98.710 | 96.274 | 92.897 |
| 231 | 100.000 | 100.000 | 98.710 | 96.295 | 92.956 |
| 232 | 100.000 | 100.000 | 98.733 | 96.359 | 93.069 |
| 233 | 100.000 | 100.000 | 98.733 | 96.365 | 93.086 |
| 234 | 100.000 | 100.000 | 98.733 | 96.376 | 93.117 |
| 235 | 100.000 | 100.000 | 98.733 | 96.381 | 93.134 |
| 236 | 100.000 | 100.000 | 98.756 | 96.435 | 93.219 |
| 237 | 100.000 | 100.000 | 98.756 | 96.445 | 93.250 |
| 238 | 100.000 | 100.000 | 98.756 | 96.461 | 93.295 |
| 239 | 100.000 | 100.000 | 98.780 | 96.520 | 93.393 |
| 240 | 100.000 | 100.000 | 98.780 | 96.526 | 93.411 |
| 241 | 100.000 | 100.000 | 98.803 | 96.585 | 93.509 |
| 242 | 100.000 | 100.000 | 98.803 | 96.611 | 93.581 |
| 243 | 100.000 | 100.000 | 98.827 | 96.670 | 93.679 |
| 244 | 100.000 | 100.000 | 98.827 | 96.670 | 93.683 |
| 245 | 100.000 | 100.000 | 98.827 | 96.692 | 93.742 |
| 246 | 100.000 | 100.000 | 98.827 | 96.703 | 93.774 |
| 247 | 100.000 | 100.000 | 98.827 | 96.713 | 93.805 |
| 248 | 100.000 | 100.000 | 98.827 | 96.724 | 93.837 |
| 249 | 100.000 | 100.000 | 98.827 | 96.729 | 93.853 |
| 250 | 100.000 | 100.000 | 98.827 | 96.745 | 93.898 |
| 251 | 100.000 | 100.000 | 98.850 | 96.799 | 93.981 |
| 252 | 100.000 | 100.000 | 98.850 | 96.831 | 94.071 |
| 253 | 100.000 | 100.000 | 98.850 | 96.837 | 94.092 |
| 254 | 100.000 | 100.000 | 98.874 | 96.890 | 94.175 |
| 255 | 100.000 | 100.000 | 98.874 | 96.912 | 94.236 |
| 256 | 100.000 | 100.000 | 98.874 | 96.922 | 94.267 |
| 257 | 100.000 | 100.000 | 98.874 | 96.928 | 94.284 |
| 258 | 100.000 | 100.000 | 98.874 | 96.939 | 94.327 |
| 259 | 100.000 | 100.000 | 98.874 | 96.949 | 94.358 |
| 260 | 100.000 | 100.000 | 98.874 | 96.960 | 94.389 |
| 261 | 100.000 | 100.000 | 98.874 | 96.965 | 94.408 |
| 262 | 100.000 | 100.000 | 98.874 | 96.971 | 94.425 |
| 263 | 100.000 | 100.000 | 98.874 | 96.976 | 94.441 |
| 264 | 100.000 | 100.000 | 98.874 | 96.987 | 94.473 |
| 265 | 100.000 | 100.000 | 98.874 | 96.997 | 94.504 |
| 266 | 100.000 | 100.000 | 98.897 | 97.062 | 94.615 |
| 267 | 100.000 | 100.000 | 98.897 | 97.083 | 94.674 |
| 268 | 100.000 | 100.000 | 98.897 | 97.083 | 94.678 |
| 269 | 100.000 | 100.000 | 98.897 | 97.094 | 94.709 |
| 270 | 100.000 | 100.000 | 98.897 | 97.094 | 94.711 |
| 271 | 100.000 | 100.000 | 98.897 | 97.099 | 94.728 |
| 272 | 100.000 | 100.000 | 98.897 | 97.105 | 94.745 |
| 273 | 100.000 | 100.000 | 98.944 | 97.207 | 94.894 |
| 274 | 100.000 | 100.000 | 98.944 | 97.212 | 94.911 |
| 275 | 100.000 | 100.000 | 98.944 | 97.217 | 94.929 |
| 276 | 100.000 | 100.000 | 98.944 | 97.217 | 94.933 |
| 277 | 100.000 | 100.000 | 98.968 | 97.271 | 95.016 |
| 278 | 100.000 | 100.000 | 98.968 | 97.271 | 95.018 |
| 279 | 100.000 | 100.000 | 98.968 | 97.271 | 95.026 |
| 280 | 100.000 | 100.000 | 98.991 | 97.325 | 95.109 |
| 281 | 100.000 | 100.000 | 98.991 | 97.335 | 95.140 |
| 282 | 100.000 | 100.000 | 98.991 | 97.351 | 95.185 |
| 283 | 100.000 | 100.000 | 98.991 | 97.351 | 95.192 |
| 284 | 100.000 | 100.000 | 98.991 | 97.351 | 95.198 |
| 285 | 100.000 | 100.000 | 98.991 | 97.357 | 95.214 |
| 286 | 100.000 | 100.000 | 98.991 | 97.357 | 95.222 |
| 287 | 100.000 | 100.000 | 98.991 | 97.362 | 95.238 |
| 288 | 100.000 | 100.000 | 98.991 | 97.362 | 95.240 |
| 289 | 100.000 | 100.000 | 99.015 | 97.416 | 95.323 |
| 290 | 100.000 | 100.000 | 99.015 | 97.426 | 95.355 |
| 291 | 100.000 | 100.000 | 99.015 | 97.432 | 95.371 |
| 292 | 100.000 | 100.000 | 99.015 | 97.442 | 95.405 |
| 293 | 100.000 | 100.000 | 99.015 | 97.448 | 95.421 |
| 294 | 100.000 | 100.000 | 99.038 | 97.507 | 95.519 |
| 295 | 100.000 | 100.000 | 99.273 | 97.957 | 96.091 |
| 296 | 100.000 | 100.000 | 99.273 | 97.968 | 96.122 |
| 297 | 100.000 | 100.000 | 99.296 | 98.022 | 96.205 |
| 298 | 100.000 | 100.000 | 99.296 | 98.032 | 96.239 |
| 299 | 100.000 | 100.000 | 99.296 | 98.038 | 96.259 |
| 300 | 100.000 | 100.000 | 99.296 | 98.048 | 96.290 |
| 301 | 100.000 | 100.000 | 99.296 | 98.054 | 96.307 |
| 302 | 100.000 | 100.000 | 99.296 | 98.054 | 96.311 |
| 303 | 100.000 | 100.000 | 99.296 | 98.054 | 96.313 |
| 304 | 100.000 | 100.000 | 99.296 | 98.064 | 96.344 |
| 305 | 100.000 | 100.000 | 99.296 | 98.081 | 96.388 |
| 306 | 100.000 | 100.000 | 99.296 | 98.091 | 96.420 |
| 307 | 100.000 | 100.000 | 99.296 | 98.091 | 96.422 |
| 308 | 100.000 | 100.000 | 99.296 | 98.097 | 96.438 |
| 309 | 100.000 | 100.000 | 99.320 | 98.150 | 96.522 |
| 310 | 100.000 | 100.000 | 99.320 | 98.156 | 96.538 |
| 311 | 100.000 | 100.000 | 99.320 | 98.161 | 96.555 |
| 312 | 100.000 | 100.000 | 99.320 | 98.161 | 96.557 |
| 313 | 100.000 | 100.000 | 99.320 | 98.166 | 96.573 |
| 314 | 100.000 | 100.000 | 99.320 | 98.166 | 96.577 |
| 315 | 100.000 | 100.000 | 99.320 | 98.177 | 96.609 |
| 316 | 100.000 | 100.000 | 99.320 | 98.188 | 96.644 |
| 317 | 100.000 | 100.000 | 99.320 | 98.188 | 96.646 |
| 318 | 100.000 | 100.000 | 99.320 | 98.188 | 96.653 |
| 319 | 100.000 | 100.000 | 99.320 | 98.193 | 96.671 |
| 320 | 100.000 | 100.000 | 99.320 | 98.193 | 96.673 |
| 321 | 100.000 | 100.000 | 99.320 | 98.209 | 96.718 |
| 322 | 100.000 | 100.000 | 99.531 | 98.627 | 97.263 |
| 323 | 100.000 | 100.000 | 99.531 | 98.627 | 97.271 |
| 324 | 100.000 | 100.000 | 99.554 | 98.681 | 97.354 |
| 325 | 100.000 | 100.000 | 99.554 | 98.692 | 97.385 |
| 326 | 100.000 | 100.000 | 99.578 | 98.745 | 97.470 |
| 327 | 100.000 | 100.000 | 99.578 | 98.745 | 97.476 |
| 328 | 100.000 | 100.000 | 99.578 | 98.756 | 97.507 |
| 329 | 100.000 | 100.000 | 99.578 | 98.756 | 97.511 |
| 330 | 100.000 | 100.000 | 99.578 | 98.767 | 97.542 |
| 331 | 100.000 | 100.000 | 99.601 | 98.820 | 97.626 |
| 332 | 100.000 | 100.000 | 99.601 | 98.820 | 97.627 |
| 333 | 100.000 | 100.000 | 99.601 | 98.826 | 97.644 |
| 334 | 100.000 | 100.000 | 99.601 | 98.826 | 97.646 |
| 335 | 100.000 | 100.000 | 99.601 | 98.842 | 97.692 |
| 336 | 100.000 | 100.000 | 99.601 | 98.847 | 97.709 |
| 337 | 100.000 | 100.000 | 99.601 | 98.847 | 97.714 |
| 338 | 100.000 | 100.000 | 99.601 | 98.858 | 97.750 |
| 339 | 100.000 | 100.000 | 99.601 | 98.863 | 97.766 |
| 340 | 100.000 | 100.000 | 99.601 | 98.863 | 97.774 |
| 341 | 100.000 | 100.000 | 99.601 | 98.869 | 97.790 |
| 342 | 100.000 | 100.000 | 99.601 | 98.869 | 97.794 |
| 343 | 100.000 | 100.000 | 99.601 | 98.874 | 97.811 |
| 344 | 100.000 | 100.000 | 99.601 | 98.879 | 97.827 |
| 345 | 100.000 | 100.000 | 99.601 | 98.879 | 97.831 |
| 346 | 100.000 | 100.000 | 99.601 | 98.890 | 97.862 |
| 347 | 100.000 | 100.000 | 99.601 | 98.901 | 97.894 |
| 348 | 100.000 | 100.000 | 99.601 | 98.906 | 97.910 |
| 349 | 100.000 | 100.000 | 99.601 | 98.912 | 97.927 |
| 350 | 100.000 | 100.000 | 99.601 | 98.917 | 97.944 |
| 351 | 100.000 | 100.000 | 99.601 | 98.917 | 97.947 |
| 352 | 100.000 | 100.000 | 99.601 | 98.928 | 97.979 |
| 353 | 100.000 | 100.000 | 99.601 | 98.933 | 97.995 |
| 354 | 100.000 | 100.000 | 99.601 | 98.944 | 98.027 |
| 355 | 100.000 | 100.000 | 99.601 | 98.949 | 98.044 |
| 356 | 100.000 | 100.000 | 99.601 | 98.954 | 98.060 |
| 357 | 100.000 | 100.000 | 99.648 | 99.056 | 98.210 |
| 358 | 100.000 | 100.000 | 99.789 | 99.341 | 98.589 |
| 359 | 100.000 | 100.000 | 99.789 | 99.346 | 98.606 |
| 360 | 100.000 | 100.000 | 99.789 | 99.346 | 98.609 |
| 361 | 100.000 | 100.000 | 99.789 | 99.351 | 98.626 |
| 362 | 100.000 | 100.000 | 99.789 | 99.351 | 98.630 |
| 363 | 100.000 | 100.000 | 99.789 | 99.351 | 98.633 |
| 364 | 100.000 | 100.000 | 99.789 | 99.357 | 98.654 |
| 365 | 100.000 | 100.000 | 99.789 | 99.362 | 98.672 |
| 366 | 100.000 | 100.000 | 99.789 | 99.362 | 98.674 |
| 367 | 100.000 | 100.000 | 99.789 | 99.362 | 98.676 |
| 368 | 100.000 | 100.000 | 99.789 | 99.362 | 98.680 |
| 369 | 100.000 | 100.000 | 99.789 | 99.373 | 98.715 |
| 370 | 100.000 | 100.000 | 99.789 | 99.378 | 98.731 |
| 371 | 100.000 | 100.000 | 99.789 | 99.383 | 98.748 |
| 372 | 100.000 | 100.000 | 99.789 | 99.389 | 98.765 |
| 373 | 100.000 | 100.000 | 99.789 | 99.389 | 98.768 |
| 374 | 100.000 | 100.000 | 99.789 | 99.394 | 98.785 |
| 375 | 100.000 | 100.000 | 99.812 | 99.448 | 98.868 |
| 376 | 100.000 | 100.000 | 99.812 | 99.448 | 98.872 |
| 377 | 100.000 | 100.000 | 99.812 | 99.448 | 98.874 |
| 378 | 100.000 | 100.000 | 99.812 | 99.485 | 98.970 |
| 379 | 100.000 | 100.000 | 99.812 | 99.485 | 98.974 |
| 380 | 100.000 | 100.000 | 99.812 | 99.485 | 98.976 |
| 381 | 100.000 | 100.000 | 99.812 | 99.496 | 99.007 |
| 382 | 100.000 | 100.000 | 99.812 | 99.496 | 99.011 |
| 383 | 100.000 | 100.000 | 99.812 | 99.496 | 99.013 |
| 384 | 100.000 | 100.000 | 99.812 | 99.501 | 99.029 |
| 385 | 100.000 | 100.000 | 99.812 | 99.501 | 99.033 |
| 386 | 100.000 | 100.000 | 99.812 | 99.501 | 99.037 |
| 387 | 100.000 | 100.000 | 99.812 | 99.501 | 99.040 |
| 388 | 100.000 | 100.000 | 99.812 | 99.501 | 99.044 |
| 389 | 100.000 | 100.000 | 99.812 | 99.507 | 99.061 |
| 390 | 100.000 | 100.000 | 99.812 | 99.512 | 99.077 |
| 391 | 100.000 | 100.000 | 99.883 | 99.662 | 99.294 |
| 392 | 100.000 | 100.000 | 99.883 | 99.668 | 99.310 |
| 393 | 100.000 | 100.000 | 99.883 | 99.668 | 99.314 |
| 394 | 100.000 | 100.000 | 99.930 | 99.769 | 99.464 |
| 395 | 100.000 | 100.000 | 99.930 | 99.769 | 99.466 |
| 396 | 100.000 | 100.000 | 99.930 | 99.769 | 99.467 |
| 397 | 100.000 | 100.000 | 99.930 | 99.780 | 99.499 |
| 398 | 100.000 | 100.000 | 99.930 | 99.780 | 99.501 |
| 399 | 100.000 | 100.000 | 99.930 | 99.780 | 99.504 |
| 400 | 100.000 | 100.000 | 99.930 | 99.780 | 99.508 |
| 401 | 100.000 | 100.000 | 99.930 | 99.780 | 99.510 |
| 402 | 100.000 | 100.000 | 99.930 | 99.786 | 99.527 |
| 403 | 100.000 | 100.000 | 99.930 | 99.791 | 99.543 |
| 404 | 100.000 | 100.000 | 99.930 | 99.791 | 99.545 |
| 405 | 100.000 | 100.000 | 99.930 | 99.796 | 99.562 |
| 406 | 100.000 | 100.000 | 99.930 | 99.796 | 99.565 |
| 407 | 100.000 | 100.000 | 99.930 | 99.796 | 99.567 |
| 408 | 100.000 | 100.000 | 99.930 | 99.796 | 99.571 |
| 409 | 100.000 | 100.000 | 99.930 | 99.802 | 99.588 |
| 410 | 100.000 | 100.000 | 99.930 | 99.802 | 99.589 |
| 411 | 100.000 | 100.000 | 99.930 | 99.802 | 99.593 |
| 412 | 100.000 | 100.000 | 99.930 | 99.802 | 99.597 |
| 413 | 100.000 | 100.000 | 99.930 | 99.802 | 99.601 |
| 414 | 100.000 | 100.000 | 99.930 | 99.812 | 99.632 |
| 415 | 100.000 | 100.000 | 99.930 | 99.818 | 99.649 |
| 416 | 100.000 | 100.000 | 99.930 | 99.818 | 99.650 |
| 417 | 100.000 | 100.000 | 99.930 | 99.818 | 99.652 |
| 418 | 100.000 | 100.000 | 99.930 | 99.823 | 99.669 |
| 419 | 100.000 | 100.000 | 99.930 | 99.834 | 99.700 |
| 420 | 100.000 | 100.000 | 99.930 | 99.839 | 99.717 |
| 421 | 100.000 | 100.000 | 99.930 | 99.845 | 99.734 |
| 422 | 100.000 | 100.000 | 99.930 | 99.845 | 99.736 |
| 423 | 100.000 | 100.000 | 99.930 | 99.845 | 99.739 |
| 424 | 100.000 | 100.000 | 99.930 | 99.845 | 99.741 |
| 425 | 100.000 | 100.000 | 99.930 | 99.845 | 99.745 |
| 426 | 100.000 | 100.000 | 99.930 | 99.845 | 99.747 |
| 427 | 100.000 | 100.000 | 99.930 | 99.845 | 99.750 |
| 428 | 100.000 | 100.000 | 99.930 | 99.845 | 99.752 |
| 429 | 100.000 | 100.000 | 99.930 | 99.845 | 99.756 |
| 430 | 100.000 | 100.000 | 99.930 | 99.845 | 99.758 |
| 431 | 100.000 | 100.000 | 99.930 | 99.845 | 99.760 |
| 432 | 100.000 | 100.000 | 99.930 | 99.845 | 99.761 |
| 433 | 100.000 | 100.000 | 99.930 | 99.845 | 99.765 |
| 434 | 100.000 | 100.000 | 99.930 | 99.845 | 99.767 |
| 435 | 100.000 | 100.000 | 100.000 | 99.995 | 99.983 |
| 436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
grep
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
| 3 | 61.905 | 33.841 | 19.052 | 11.318 | 7.116 |
| 4 | 71.429 | 41.794 | 24.472 | 14.857 | 9.438 |
| 5 | 78.571 | 48.393 | 29.329 | 18.203 | 11.707 |
| 6 | 80.952 | 52.623 | 33.247 | 21.221 | 13.877 |
| 7 | 85.714 | 58.037 | 37.588 | 24.358 | 16.070 |
| 8 | 90.476 | 63.283 | 41.741 | 27.355 | 18.180 |
| 9 | 95.238 | 68.020 | 45.519 | 30.165 | 20.216 |
| 10 | 95.238 | 70.728 | 48.592 | 32.760 | 22.196 |
| 11 | 97.619 | 74.112 | 51.760 | 35.328 | 24.149 |
| 12 | 100.000 | 76.481 | 54.223 | 37.564 | 25.976 |
| 13 | 100.000 | 78.511 | 56.710 | 39.805 | 27.781 |
| 14 | 100.000 | 79.865 | 58.775 | 41.874 | 29.534 |
| 15 | 100.000 | 81.557 | 60.934 | 43.906 | 31.235 |
| 16 | 100.000 | 83.249 | 63.069 | 45.906 | 32.909 |
| 17 | 100.000 | 84.772 | 65.063 | 47.820 | 34.538 |
| 18 | 100.000 | 86.125 | 66.823 | 49.509 | 35.988 |
| 19 | 100.000 | 87.140 | 68.325 | 51.096 | 37.439 |
| 20 | 100.000 | 88.156 | 69.803 | 52.608 | 38.785 |
| 21 | 100.000 | 88.832 | 71.023 | 54.013 | 40.126 |
| 22 | 100.000 | 89.679 | 72.313 | 55.418 | 41.450 |
| 23 | 100.000 | 90.694 | 73.698 | 56.833 | 42.741 |
| 24 | 100.000 | 91.540 | 74.965 | 58.185 | 43.997 |
| 25 | 100.000 | 92.047 | 75.997 | 59.455 | 45.265 |
| 26 | 100.000 | 92.724 | 77.053 | 60.629 | 46.399 |
| 27 | 100.000 | 93.570 | 78.203 | 61.793 | 47.458 |
| 28 | 100.000 | 94.078 | 79.141 | 62.924 | 48.594 |
| 29 | 100.000 | 94.585 | 79.986 | 63.895 | 49.552 |
| 30 | 100.000 | 94.755 | 80.643 | 64.854 | 50.593 |
| 31 | 100.000 | 94.924 | 81.276 | 65.777 | 51.586 |
| 32 | 100.000 | 95.093 | 81.910 | 66.726 | 52.636 |
| 33 | 100.000 | 95.601 | 82.708 | 67.626 | 53.527 |
| 34 | 100.000 | 96.108 | 83.505 | 68.533 | 54.430 |
| 35 | 100.000 | 96.616 | 84.280 | 69.406 | 55.303 |
| 36 | 100.000 | 96.954 | 84.960 | 70.270 | 56.207 |
| 37 | 100.000 | 97.462 | 85.687 | 71.085 | 57.026 |
| 38 | 100.000 | 97.970 | 86.415 | 71.851 | 57.747 |
| 39 | 100.000 | 98.308 | 87.048 | 72.640 | 58.574 |
| 40 | 100.000 | 98.308 | 87.471 | 73.374 | 59.432 |
| 41 | 100.000 | 98.646 | 88.081 | 74.157 | 60.286 |
| 42 | 100.000 | 98.816 | 88.550 | 74.865 | 61.113 |
| 43 | 100.000 | 98.985 | 89.019 | 75.540 | 61.875 |
| 44 | 100.000 | 99.154 | 89.489 | 76.221 | 62.648 |
| 45 | 100.000 | 99.154 | 89.840 | 76.848 | 63.400 |
| 46 | 100.000 | 99.154 | 90.169 | 77.444 | 64.133 |
| 47 | 100.000 | 99.323 | 90.568 | 78.001 | 64.765 |
| 48 | 100.000 | 99.323 | 90.873 | 78.564 | 65.462 |
| 49 | 100.000 | 99.323 | 91.178 | 79.127 | 66.159 |
| 50 | 100.000 | 99.662 | 91.671 | 79.663 | 66.686 |
| 51 | 100.000 | 99.662 | 91.952 | 80.199 | 67.358 |
| 52 | 100.000 | 99.662 | 92.187 | 80.666 | 67.972 |
| 53 | 100.000 | 99.831 | 92.515 | 81.106 | 68.467 |
| 54 | 100.000 | 99.831 | 92.726 | 81.524 | 69.020 |
| 55 | 100.000 | 99.831 | 92.938 | 81.947 | 69.584 |
| 56 | 100.000 | 99.831 | 93.149 | 82.366 | 70.130 |
| 57 | 100.000 | 100.000 | 93.454 | 82.752 | 70.553 |
| 58 | 100.000 | 100.000 | 93.641 | 83.138 | 71.078 |
| 59 | 100.000 | 100.000 | 93.829 | 83.513 | 71.579 |
| 60 | 100.000 | 100.000 | 93.993 | 83.840 | 72.018 |
| 61 | 100.000 | 100.000 | 94.158 | 84.178 | 72.480 |
| 62 | 100.000 | 100.000 | 94.322 | 84.521 | 72.955 |
| 63 | 100.000 | 100.000 | 94.486 | 84.843 | 73.381 |
| 64 | 100.000 | 100.000 | 94.650 | 85.170 | 73.819 |
| 65 | 100.000 | 100.000 | 94.815 | 85.502 | 74.268 |
| 66 | 100.000 | 100.000 | 94.955 | 85.786 | 74.647 |
| 67 | 100.000 | 100.000 | 95.096 | 86.092 | 75.082 |
| 68 | 100.000 | 100.000 | 95.237 | 86.381 | 75.481 |
| 69 | 100.000 | 100.000 | 95.378 | 86.666 | 75.864 |
| 70 | 100.000 | 100.000 | 95.519 | 86.971 | 76.299 |
| 71 | 100.000 | 100.000 | 95.636 | 87.218 | 76.643 |
| 72 | 100.000 | 100.000 | 95.753 | 87.481 | 77.027 |
| 73 | 100.000 | 100.000 | 95.870 | 87.738 | 77.399 |
| 74 | 100.000 | 100.000 | 95.988 | 87.990 | 77.759 |
| 75 | 100.000 | 100.000 | 96.105 | 88.247 | 78.131 |
| 76 | 100.000 | 100.000 | 96.222 | 88.489 | 78.462 |
| 77 | 100.000 | 100.000 | 96.340 | 88.730 | 78.793 |
| 78 | 100.000 | 100.000 | 96.457 | 88.982 | 79.154 |
| 79 | 100.000 | 100.000 | 96.551 | 89.191 | 79.459 |
| 80 | 100.000 | 100.000 | 96.645 | 89.373 | 79.699 |
| 81 | 100.000 | 100.000 | 96.739 | 89.620 | 80.091 |
| 82 | 100.000 | 100.000 | 96.832 | 89.824 | 80.389 |
| 83 | 100.000 | 100.000 | 96.926 | 90.017 | 80.657 |
| 84 | 100.000 | 100.000 | 97.020 | 90.215 | 80.936 |
| 85 | 100.000 | 100.000 | 97.114 | 90.440 | 81.280 |
| 86 | 100.000 | 100.000 | 97.208 | 90.655 | 81.598 |
| 87 | 100.000 | 100.000 | 97.302 | 90.864 | 81.905 |
| 88 | 100.000 | 100.000 | 97.396 | 91.062 | 82.187 |
| 89 | 100.000 | 100.000 | 97.466 | 91.228 | 82.440 |
| 90 | 100.000 | 100.000 | 97.536 | 91.384 | 82.669 |
| 91 | 100.000 | 100.000 | 97.607 | 91.539 | 82.900 |
| 92 | 100.000 | 100.000 | 97.677 | 91.716 | 83.181 |
| 93 | 100.000 | 100.000 | 97.748 | 91.893 | 83.464 |
| 94 | 100.000 | 100.000 | 97.818 | 92.059 | 83.721 |
| 95 | 100.000 | 100.000 | 97.888 | 92.226 | 83.975 |
| 96 | 100.000 | 100.000 | 97.959 | 92.376 | 84.191 |
| 97 | 100.000 | 100.000 | 98.029 | 92.531 | 84.420 |
| 98 | 100.000 | 100.000 | 98.099 | 92.681 | 84.637 |
| 99 | 100.000 | 100.000 | 98.146 | 92.794 | 84.814 |
| 100 | 100.000 | 100.000 | 98.193 | 92.912 | 85.005 |
| 101 | 100.000 | 100.000 | 98.240 | 93.067 | 85.284 |
| 102 | 100.000 | 100.000 | 98.287 | 93.169 | 85.434 |
| 103 | 100.000 | 100.000 | 98.334 | 93.293 | 85.635 |
| 104 | 100.000 | 100.000 | 98.381 | 93.394 | 85.787 |
| 105 | 100.000 | 100.000 | 98.428 | 93.523 | 86.001 |
| 106 | 100.000 | 100.000 | 98.475 | 93.636 | 86.179 |
| 107 | 100.000 | 100.000 | 98.522 | 93.738 | 86.329 |
| 108 | 100.000 | 100.000 | 98.569 | 93.866 | 86.547 |
| 109 | 100.000 | 100.000 | 98.616 | 93.995 | 86.761 |
| 110 | 100.000 | 100.000 | 98.663 | 94.108 | 86.939 |
| 111 | 100.000 | 100.000 | 98.710 | 94.209 | 87.089 |
| 112 | 100.000 | 100.000 | 98.756 | 94.333 | 87.292 |
| 113 | 100.000 | 100.000 | 98.803 | 94.435 | 87.442 |
| 114 | 100.000 | 100.000 | 98.850 | 94.536 | 87.592 |
| 115 | 100.000 | 100.000 | 98.897 | 94.649 | 87.769 |
| 116 | 100.000 | 100.000 | 98.944 | 94.778 | 87.987 |
| 117 | 100.000 | 100.000 | 98.991 | 94.896 | 88.178 |
| 118 | 100.000 | 100.000 | 99.038 | 94.998 | 88.328 |
| 119 | 100.000 | 100.000 | 99.085 | 95.105 | 88.492 |
| 120 | 100.000 | 100.000 | 99.132 | 95.228 | 88.694 |
| 121 | 100.000 | 100.000 | 99.179 | 95.330 | 88.844 |
| 122 | 100.000 | 100.000 | 99.226 | 95.448 | 89.034 |
| 123 | 100.000 | 100.000 | 99.273 | 95.630 | 89.369 |
| 124 | 100.000 | 100.000 | 99.320 | 95.732 | 89.519 |
| 125 | 100.000 | 100.000 | 99.343 | 95.786 | 89.602 |
| 126 | 100.000 | 100.000 | 99.366 | 95.839 | 89.685 |
| 127 | 100.000 | 100.000 | 99.390 | 95.914 | 89.824 |
| 128 | 100.000 | 100.000 | 99.413 | 96.006 | 90.001 |
| 129 | 100.000 | 100.000 | 99.437 | 96.081 | 90.140 |
| 130 | 100.000 | 100.000 | 99.460 | 96.145 | 90.253 |
| 131 | 100.000 | 100.000 | 99.484 | 96.204 | 90.351 |
| 132 | 100.000 | 100.000 | 99.507 | 96.268 | 90.462 |
| 133 | 100.000 | 100.000 | 99.531 | 96.354 | 90.624 |
| 134 | 100.000 | 100.000 | 99.554 | 96.413 | 90.722 |
| 135 | 100.000 | 100.000 | 99.578 | 96.483 | 90.848 |
| 136 | 100.000 | 100.000 | 99.601 | 96.552 | 90.974 |
| 137 | 100.000 | 100.000 | 99.625 | 96.606 | 91.057 |
| 138 | 100.000 | 100.000 | 99.648 | 96.665 | 91.155 |
| 139 | 100.000 | 100.000 | 99.672 | 96.735 | 91.290 |
| 140 | 100.000 | 100.000 | 99.695 | 96.794 | 91.388 |
| 141 | 100.000 | 100.000 | 99.718 | 96.869 | 91.531 |
| 142 | 100.000 | 100.000 | 99.742 | 96.922 | 91.614 |
| 143 | 100.000 | 100.000 | 99.765 | 96.981 | 91.712 |
| 144 | 100.000 | 100.000 | 99.789 | 97.046 | 91.823 |
| 145 | 100.000 | 100.000 | 99.812 | 97.121 | 91.965 |
| 146 | 100.000 | 100.000 | 99.836 | 97.180 | 92.063 |
| 147 | 100.000 | 100.000 | 99.859 | 97.239 | 92.161 |
| 148 | 100.000 | 100.000 | 99.883 | 97.298 | 92.259 |
| 149 | 100.000 | 100.000 | 99.906 | 97.351 | 92.342 |
| 150 | 100.000 | 100.000 | 99.930 | 97.405 | 92.426 |
| 151 | 100.000 | 100.000 | 99.953 | 97.459 | 92.509 |
| 152 | 100.000 | 100.000 | 99.977 | 97.523 | 92.622 |
| 153 | 100.000 | 100.000 | 100.000 | 97.577 | 92.705 |
| 154 | 100.000 | 100.000 | 100.000 | 97.582 | 92.723 |
| 155 | 100.000 | 100.000 | 100.000 | 97.587 | 92.744 |
| 156 | 100.000 | 100.000 | 100.000 | 97.614 | 92.818 |
| 157 | 100.000 | 100.000 | 100.000 | 97.641 | 92.892 |
| 158 | 100.000 | 100.000 | 100.000 | 97.684 | 93.003 |
| 159 | 100.000 | 100.000 | 100.000 | 97.705 | 93.062 |
| 160 | 100.000 | 100.000 | 100.000 | 97.716 | 93.093 |
| 161 | 100.000 | 100.000 | 100.000 | 97.721 | 93.112 |
| 162 | 100.000 | 100.000 | 100.000 | 97.743 | 93.173 |
| 163 | 100.000 | 100.000 | 100.000 | 97.764 | 93.232 |
| 164 | 100.000 | 100.000 | 100.000 | 97.807 | 93.345 |
| 165 | 100.000 | 100.000 | 100.000 | 97.818 | 93.376 |
| 166 | 100.000 | 100.000 | 100.000 | 97.829 | 93.408 |
| 167 | 100.000 | 100.000 | 100.000 | 97.850 | 93.467 |
| 168 | 100.000 | 100.000 | 100.000 | 97.866 | 93.511 |
| 169 | 100.000 | 100.000 | 100.000 | 97.877 | 93.543 |
| 170 | 100.000 | 100.000 | 100.000 | 97.888 | 93.574 |
| 171 | 100.000 | 100.000 | 100.000 | 97.914 | 93.648 |
| 172 | 100.000 | 100.000 | 100.000 | 97.920 | 93.666 |
| 173 | 100.000 | 100.000 | 100.000 | 97.925 | 93.683 |
| 174 | 100.000 | 100.000 | 100.000 | 97.957 | 93.770 |
| 175 | 100.000 | 100.000 | 100.000 | 97.968 | 93.801 |
| 176 | 100.000 | 100.000 | 100.000 | 97.989 | 93.861 |
| 177 | 100.000 | 100.000 | 100.000 | 98.022 | 93.948 |
| 178 | 100.000 | 100.000 | 100.000 | 98.064 | 94.058 |
| 179 | 100.000 | 100.000 | 100.000 | 98.113 | 94.186 |
| 180 | 100.000 | 100.000 | 100.000 | 98.118 | 94.205 |
| 181 | 100.000 | 100.000 | 100.000 | 98.150 | 94.291 |
| 182 | 100.000 | 100.000 | 100.000 | 98.161 | 94.323 |
| 183 | 100.000 | 100.000 | 100.000 | 98.193 | 94.406 |
| 184 | 100.000 | 100.000 | 100.000 | 98.209 | 94.452 |
| 185 | 100.000 | 100.000 | 100.000 | 98.231 | 94.515 |
| 186 | 100.000 | 100.000 | 100.000 | 98.241 | 94.550 |
| 187 | 100.000 | 100.000 | 100.000 | 98.247 | 94.567 |
| 188 | 100.000 | 100.000 | 100.000 | 98.257 | 94.598 |
| 189 | 100.000 | 100.000 | 100.000 | 98.279 | 94.658 |
| 190 | 100.000 | 100.000 | 100.000 | 98.295 | 94.704 |
| 191 | 100.000 | 100.000 | 100.000 | 98.300 | 94.722 |
| 192 | 100.000 | 100.000 | 100.000 | 98.311 | 94.754 |
| 193 | 100.000 | 100.000 | 100.000 | 98.327 | 94.800 |
| 194 | 100.000 | 100.000 | 100.000 | 98.343 | 94.848 |
| 195 | 100.000 | 100.000 | 100.000 | 98.343 | 94.855 |
| 196 | 100.000 | 100.000 | 100.000 | 98.365 | 94.915 |
| 197 | 100.000 | 100.000 | 100.000 | 98.413 | 95.042 |
| 198 | 100.000 | 100.000 | 100.000 | 98.434 | 95.101 |
| 199 | 100.000 | 100.000 | 100.000 | 98.440 | 95.118 |
| 200 | 100.000 | 100.000 | 100.000 | 98.445 | 95.137 |
| 201 | 100.000 | 100.000 | 100.000 | 98.450 | 95.153 |
| 202 | 100.000 | 100.000 | 100.000 | 98.456 | 95.172 |
| 203 | 100.000 | 100.000 | 100.000 | 98.467 | 95.207 |
| 204 | 100.000 | 100.000 | 100.000 | 98.488 | 95.266 |
| 205 | 100.000 | 100.000 | 100.000 | 98.520 | 95.353 |
| 206 | 100.000 | 100.000 | 100.000 | 98.526 | 95.371 |
| 207 | 100.000 | 100.000 | 100.000 | 98.531 | 95.388 |
| 208 | 100.000 | 100.000 | 100.000 | 98.536 | 95.407 |
| 209 | 100.000 | 100.000 | 100.000 | 98.552 | 95.453 |
| 210 | 100.000 | 100.000 | 100.000 | 98.563 | 95.488 |
| 211 | 100.000 | 100.000 | 100.000 | 98.574 | 95.519 |
| 212 | 100.000 | 100.000 | 100.000 | 98.585 | 95.554 |
| 213 | 100.000 | 100.000 | 100.000 | 98.590 | 95.571 |
| 214 | 100.000 | 100.000 | 100.000 | 98.606 | 95.616 |
| 215 | 100.000 | 100.000 | 100.000 | 98.611 | 95.632 |
| 216 | 100.000 | 100.000 | 100.000 | 98.638 | 95.704 |
| 217 | 100.000 | 100.000 | 100.000 | 98.660 | 95.767 |
| 218 | 100.000 | 100.000 | 100.000 | 98.670 | 95.799 |
| 219 | 100.000 | 100.000 | 100.000 | 98.676 | 95.815 |
| 220 | 100.000 | 100.000 | 100.000 | 98.676 | 95.823 |
| 221 | 100.000 | 100.000 | 100.000 | 98.697 | 95.885 |
| 222 | 100.000 | 100.000 | 100.000 | 98.708 | 95.919 |
| 223 | 100.000 | 100.000 | 100.000 | 98.724 | 95.963 |
| 224 | 100.000 | 100.000 | 100.000 | 98.729 | 95.980 |
| 225 | 100.000 | 100.000 | 100.000 | 98.740 | 96.011 |
| 226 | 100.000 | 100.000 | 100.000 | 98.740 | 96.013 |
| 227 | 100.000 | 100.000 | 100.000 | 98.745 | 96.032 |
| 228 | 100.000 | 100.000 | 100.000 | 98.745 | 96.033 |
| 229 | 100.000 | 100.000 | 100.000 | 98.756 | 96.065 |
| 230 | 100.000 | 100.000 | 100.000 | 98.761 | 96.082 |
| 231 | 100.000 | 100.000 | 100.000 | 98.767 | 96.102 |
| 232 | 100.000 | 100.000 | 100.000 | 98.778 | 96.137 |
| 233 | 100.000 | 100.000 | 100.000 | 98.788 | 96.172 |
| 234 | 100.000 | 100.000 | 100.000 | 98.788 | 96.176 |
| 235 | 100.000 | 100.000 | 100.000 | 98.799 | 96.207 |
| 236 | 100.000 | 100.000 | 100.000 | 98.810 | 96.239 |
| 237 | 100.000 | 100.000 | 100.000 | 98.815 | 96.255 |
| 238 | 100.000 | 100.000 | 100.000 | 98.820 | 96.272 |
| 239 | 100.000 | 100.000 | 100.000 | 98.826 | 96.289 |
| 240 | 100.000 | 100.000 | 100.000 | 98.831 | 96.305 |
| 241 | 100.000 | 100.000 | 100.000 | 98.869 | 96.401 |
| 242 | 100.000 | 100.000 | 100.000 | 98.879 | 96.433 |
| 243 | 100.000 | 100.000 | 100.000 | 98.944 | 96.596 |
| 244 | 100.000 | 100.000 | 100.000 | 98.954 | 96.629 |
| 245 | 100.000 | 100.000 | 100.000 | 98.965 | 96.660 |
| 246 | 100.000 | 100.000 | 100.000 | 98.981 | 96.707 |
| 247 | 100.000 | 100.000 | 100.000 | 98.987 | 96.723 |
| 248 | 100.000 | 100.000 | 100.000 | 98.997 | 96.758 |
| 249 | 100.000 | 100.000 | 100.000 | 99.003 | 96.775 |
| 250 | 100.000 | 100.000 | 100.000 | 99.013 | 96.808 |
| 251 | 100.000 | 100.000 | 100.000 | 99.024 | 96.843 |
| 252 | 100.000 | 100.000 | 100.000 | 99.024 | 96.845 |
| 253 | 100.000 | 100.000 | 100.000 | 99.035 | 96.877 |
| 254 | 100.000 | 100.000 | 100.000 | 99.040 | 96.893 |
| 255 | 100.000 | 100.000 | 100.000 | 99.046 | 96.910 |
| 256 | 100.000 | 100.000 | 100.000 | 99.051 | 96.927 |
| 257 | 100.000 | 100.000 | 100.000 | 99.072 | 96.986 |
| 258 | 100.000 | 100.000 | 100.000 | 99.083 | 97.017 |
| 259 | 100.000 | 100.000 | 100.000 | 99.083 | 97.019 |
| 260 | 100.000 | 100.000 | 100.000 | 99.099 | 97.069 |
| 261 | 100.000 | 100.000 | 100.000 | 99.105 | 97.086 |
| 262 | 100.000 | 100.000 | 100.000 | 99.126 | 97.149 |
| 263 | 100.000 | 100.000 | 100.000 | 99.137 | 97.180 |
| 264 | 100.000 | 100.000 | 100.000 | 99.147 | 97.211 |
| 265 | 100.000 | 100.000 | 100.000 | 99.153 | 97.228 |
| 266 | 100.000 | 100.000 | 100.000 | 99.153 | 97.232 |
| 267 | 100.000 | 100.000 | 100.000 | 99.158 | 97.248 |
| 268 | 100.000 | 100.000 | 100.000 | 99.164 | 97.265 |
| 269 | 100.000 | 100.000 | 100.000 | 99.174 | 97.296 |
| 270 | 100.000 | 100.000 | 100.000 | 99.180 | 97.313 |
| 271 | 100.000 | 100.000 | 100.000 | 99.180 | 97.315 |
| 272 | 100.000 | 100.000 | 100.000 | 99.190 | 97.346 |
| 273 | 100.000 | 100.000 | 100.000 | 99.196 | 97.365 |
| 274 | 100.000 | 100.000 | 100.000 | 99.196 | 97.367 |
| 275 | 100.000 | 100.000 | 100.000 | 99.217 | 97.426 |
| 276 | 100.000 | 100.000 | 100.000 | 99.228 | 97.457 |
| 277 | 100.000 | 100.000 | 100.000 | 99.233 | 97.476 |
| 278 | 100.000 | 100.000 | 100.000 | 99.255 | 97.535 |
| 279 | 100.000 | 100.000 | 100.000 | 99.260 | 97.552 |
| 280 | 100.000 | 100.000 | 100.000 | 99.271 | 97.583 |
| 281 | 100.000 | 100.000 | 100.000 | 99.276 | 97.600 |
| 282 | 100.000 | 100.000 | 100.000 | 99.287 | 97.631 |
| 283 | 100.000 | 100.000 | 100.000 | 99.292 | 97.653 |
| 284 | 100.000 | 100.000 | 100.000 | 99.357 | 97.816 |
| 285 | 100.000 | 100.000 | 100.000 | 99.378 | 97.875 |
| 286 | 100.000 | 100.000 | 100.000 | 99.378 | 97.879 |
| 287 | 100.000 | 100.000 | 100.000 | 99.389 | 97.910 |
| 288 | 100.000 | 100.000 | 100.000 | 99.399 | 97.942 |
| 289 | 100.000 | 100.000 | 100.000 | 99.416 | 97.986 |
| 290 | 100.000 | 100.000 | 100.000 | 99.421 | 98.003 |
| 291 | 100.000 | 100.000 | 100.000 | 99.421 | 98.005 |
| 292 | 100.000 | 100.000 | 100.000 | 99.421 | 98.007 |
| 293 | 100.000 | 100.000 | 100.000 | 99.426 | 98.023 |
| 294 | 100.000 | 100.000 | 100.000 | 99.437 | 98.062 |
| 295 | 100.000 | 100.000 | 100.000 | 99.448 | 98.093 |
| 296 | 100.000 | 100.000 | 100.000 | 99.448 | 98.101 |
| 297 | 100.000 | 100.000 | 100.000 | 99.453 | 98.117 |
| 298 | 100.000 | 100.000 | 100.000 | 99.453 | 98.119 |
| 299 | 100.000 | 100.000 | 100.000 | 99.464 | 98.151 |
| 300 | 100.000 | 100.000 | 100.000 | 99.464 | 98.154 |
| 301 | 100.000 | 100.000 | 100.000 | 99.469 | 98.171 |
| 302 | 100.000 | 100.000 | 100.000 | 99.480 | 98.203 |
| 303 | 100.000 | 100.000 | 100.000 | 99.501 | 98.265 |
| 304 | 100.000 | 100.000 | 100.000 | 99.507 | 98.282 |
| 305 | 100.000 | 100.000 | 100.000 | 99.507 | 98.286 |
| 306 | 100.000 | 100.000 | 100.000 | 99.512 | 98.302 |
| 307 | 100.000 | 100.000 | 100.000 | 99.517 | 98.319 |
| 308 | 100.000 | 100.000 | 100.000 | 99.517 | 98.321 |
| 309 | 100.000 | 100.000 | 100.000 | 99.528 | 98.352 |
| 310 | 100.000 | 100.000 | 100.000 | 99.539 | 98.384 |
| 311 | 100.000 | 100.000 | 100.000 | 99.539 | 98.387 |
| 312 | 100.000 | 100.000 | 100.000 | 99.544 | 98.408 |
| 313 | 100.000 | 100.000 | 100.000 | 99.566 | 98.467 |
| 314 | 100.000 | 100.000 | 100.000 | 99.576 | 98.498 |
| 315 | 100.000 | 100.000 | 100.000 | 99.587 | 98.530 |
| 316 | 100.000 | 100.000 | 100.000 | 99.593 | 98.547 |
| 317 | 100.000 | 100.000 | 100.000 | 99.603 | 98.578 |
| 318 | 100.000 | 100.000 | 100.000 | 99.603 | 98.582 |
| 319 | 100.000 | 100.000 | 100.000 | 99.609 | 98.598 |
| 320 | 100.000 | 100.000 | 100.000 | 99.614 | 98.615 |
| 321 | 100.000 | 100.000 | 100.000 | 99.614 | 98.619 |
| 322 | 100.000 | 100.000 | 100.000 | 99.619 | 98.635 |
| 323 | 100.000 | 100.000 | 100.000 | 99.625 | 98.652 |
| 324 | 100.000 | 100.000 | 100.000 | 99.625 | 98.656 |
| 325 | 100.000 | 100.000 | 100.000 | 99.635 | 98.687 |
| 326 | 100.000 | 100.000 | 100.000 | 99.641 | 98.704 |
| 327 | 100.000 | 100.000 | 100.000 | 99.646 | 98.720 |
| 328 | 100.000 | 100.000 | 100.000 | 99.646 | 98.724 |
| 329 | 100.000 | 100.000 | 100.000 | 99.646 | 98.726 |
| 330 | 100.000 | 100.000 | 100.000 | 99.657 | 98.757 |
| 331 | 100.000 | 100.000 | 100.000 | 99.662 | 98.778 |
| 332 | 100.000 | 100.000 | 100.000 | 99.668 | 98.794 |
| 333 | 100.000 | 100.000 | 100.000 | 99.673 | 98.811 |
| 334 | 100.000 | 100.000 | 100.000 | 99.673 | 98.813 |
| 335 | 100.000 | 100.000 | 100.000 | 99.678 | 98.829 |
| 336 | 100.000 | 100.000 | 100.000 | 99.689 | 98.861 |
| 337 | 100.000 | 100.000 | 100.000 | 99.694 | 98.878 |
| 338 | 100.000 | 100.000 | 100.000 | 99.700 | 98.894 |
| 339 | 100.000 | 100.000 | 100.000 | 99.705 | 98.911 |
| 340 | 100.000 | 100.000 | 100.000 | 99.710 | 98.927 |
| 341 | 100.000 | 100.000 | 100.000 | 99.710 | 98.929 |
| 342 | 100.000 | 100.000 | 100.000 | 99.710 | 98.933 |
| 343 | 100.000 | 100.000 | 100.000 | 99.721 | 98.968 |
| 344 | 100.000 | 100.000 | 100.000 | 99.732 | 99.000 |
| 345 | 100.000 | 100.000 | 100.000 | 99.732 | 99.003 |
| 346 | 100.000 | 100.000 | 100.000 | 99.732 | 99.005 |
| 347 | 100.000 | 100.000 | 100.000 | 99.748 | 99.051 |
| 348 | 100.000 | 100.000 | 100.000 | 99.759 | 99.083 |
| 349 | 100.000 | 100.000 | 100.000 | 99.764 | 99.099 |
| 350 | 100.000 | 100.000 | 100.000 | 99.764 | 99.101 |
| 351 | 100.000 | 100.000 | 100.000 | 99.775 | 99.133 |
| 352 | 100.000 | 100.000 | 100.000 | 99.775 | 99.140 |
| 353 | 100.000 | 100.000 | 100.000 | 99.775 | 99.144 |
| 354 | 100.000 | 100.000 | 100.000 | 99.775 | 99.148 |
| 355 | 100.000 | 100.000 | 100.000 | 99.780 | 99.164 |
| 356 | 100.000 | 100.000 | 100.000 | 99.786 | 99.181 |
| 357 | 100.000 | 100.000 | 100.000 | 99.786 | 99.184 |
| 358 | 100.000 | 100.000 | 100.000 | 99.786 | 99.188 |
| 359 | 100.000 | 100.000 | 100.000 | 99.786 | 99.190 |
| 360 | 100.000 | 100.000 | 100.000 | 99.786 | 99.194 |
| 361 | 100.000 | 100.000 | 100.000 | 99.791 | 99.210 |
| 362 | 100.000 | 100.000 | 100.000 | 99.791 | 99.212 |
| 363 | 100.000 | 100.000 | 100.000 | 99.802 | 99.244 |
| 364 | 100.000 | 100.000 | 100.000 | 99.807 | 99.260 |
| 365 | 100.000 | 100.000 | 100.000 | 99.812 | 99.277 |
| 366 | 100.000 | 100.000 | 100.000 | 99.818 | 99.295 |
| 367 | 100.000 | 100.000 | 100.000 | 99.828 | 99.327 |
| 368 | 100.000 | 100.000 | 100.000 | 99.828 | 99.329 |
| 369 | 100.000 | 100.000 | 100.000 | 99.834 | 99.345 |
| 370 | 100.000 | 100.000 | 100.000 | 99.834 | 99.347 |
| 371 | 100.000 | 100.000 | 100.000 | 99.834 | 99.351 |
| 372 | 100.000 | 100.000 | 100.000 | 99.834 | 99.353 |
| 373 | 100.000 | 100.000 | 100.000 | 99.850 | 99.399 |
| 374 | 100.000 | 100.000 | 100.000 | 99.850 | 99.403 |
| 375 | 100.000 | 100.000 | 100.000 | 99.850 | 99.406 |
| 376 | 100.000 | 100.000 | 100.000 | 99.850 | 99.412 |
| 377 | 100.000 | 100.000 | 100.000 | 99.855 | 99.429 |
| 378 | 100.000 | 100.000 | 100.000 | 99.861 | 99.445 |
| 379 | 100.000 | 100.000 | 100.000 | 99.866 | 99.462 |
| 380 | 100.000 | 100.000 | 100.000 | 99.871 | 99.479 |
| 381 | 100.000 | 100.000 | 100.000 | 99.871 | 99.482 |
| 382 | 100.000 | 100.000 | 100.000 | 99.882 | 99.514 |
| 383 | 100.000 | 100.000 | 100.000 | 99.882 | 99.517 |
| 384 | 100.000 | 100.000 | 100.000 | 99.882 | 99.525 |
| 385 | 100.000 | 100.000 | 100.000 | 99.893 | 99.556 |
| 386 | 100.000 | 100.000 | 100.000 | 99.893 | 99.558 |
| 387 | 100.000 | 100.000 | 100.000 | 99.903 | 99.593 |
| 388 | 100.000 | 100.000 | 100.000 | 99.903 | 99.597 |
| 389 | 100.000 | 100.000 | 100.000 | 99.903 | 99.601 |
| 390 | 100.000 | 100.000 | 100.000 | 99.903 | 99.604 |
| 391 | 100.000 | 100.000 | 100.000 | 99.909 | 99.621 |
| 392 | 100.000 | 100.000 | 100.000 | 99.909 | 99.623 |
| 393 | 100.000 | 100.000 | 100.000 | 99.909 | 99.625 |
| 394 | 100.000 | 100.000 | 100.000 | 99.914 | 99.641 |
| 395 | 100.000 | 100.000 | 100.000 | 99.925 | 99.673 |
| 396 | 100.000 | 100.000 | 100.000 | 99.925 | 99.676 |
| 397 | 100.000 | 100.000 | 100.000 | 99.930 | 99.693 |
| 398 | 100.000 | 100.000 | 100.000 | 99.941 | 99.724 |
| 399 | 100.000 | 100.000 | 100.000 | 99.946 | 99.741 |
| 400 | 100.000 | 100.000 | 100.000 | 99.946 | 99.745 |
| 401 | 100.000 | 100.000 | 100.000 | 99.946 | 99.749 |
| 402 | 100.000 | 100.000 | 100.000 | 99.952 | 99.765 |
| 403 | 100.000 | 100.000 | 100.000 | 99.952 | 99.769 |
| 404 | 100.000 | 100.000 | 100.000 | 99.952 | 99.771 |
| 405 | 100.000 | 100.000 | 100.000 | 99.952 | 99.782 |
| 406 | 100.000 | 100.000 | 100.000 | 99.952 | 99.784 |
| 407 | 100.000 | 100.000 | 100.000 | 99.952 | 99.787 |
| 408 | 100.000 | 100.000 | 100.000 | 99.952 | 99.791 |
| 409 | 100.000 | 100.000 | 100.000 | 99.952 | 99.795 |
| 410 | 100.000 | 100.000 | 100.000 | 99.952 | 99.798 |
| 411 | 100.000 | 100.000 | 100.000 | 99.957 | 99.815 |
| 412 | 100.000 | 100.000 | 100.000 | 99.962 | 99.832 |
| 413 | 100.000 | 100.000 | 100.000 | 99.962 | 99.839 |
| 414 | 100.000 | 100.000 | 100.000 | 99.962 | 99.843 |
| 415 | 100.000 | 100.000 | 100.000 | 99.968 | 99.859 |
| 416 | 100.000 | 100.000 | 100.000 | 99.973 | 99.876 |
| 417 | 100.000 | 100.000 | 100.000 | 99.973 | 99.880 |
| 418 | 100.000 | 100.000 | 100.000 | 99.979 | 99.896 |
| 419 | 100.000 | 100.000 | 100.000 | 99.979 | 99.900 |
| 420 | 100.000 | 100.000 | 100.000 | 99.979 | 99.904 |
| 421 | 100.000 | 100.000 | 100.000 | 99.979 | 99.906 |
| 422 | 100.000 | 100.000 | 100.000 | 99.979 | 99.908 |
| 423 | 100.000 | 100.000 | 100.000 | 99.979 | 99.909 |
| 424 | 100.000 | 100.000 | 100.000 | 99.979 | 99.911 |
| 425 | 100.000 | 100.000 | 100.000 | 99.984 | 99.928 |
| 426 | 100.000 | 100.000 | 100.000 | 99.984 | 99.932 |
| 427 | 100.000 | 100.000 | 100.000 | 99.995 | 99.963 |
| 428 | 100.000 | 100.000 | 100.000 | 99.995 | 99.967 |
| 429 | 100.000 | 100.000 | 100.000 | 99.995 | 99.970 |
| 430 | 100.000 | 100.000 | 100.000 | 99.995 | 99.974 |
| 431 | 100.000 | 100.000 | 100.000 | 99.995 | 99.976 |
| 432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
grep
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 47.619 | 23.858 | 12.952 | 7.587 | 4.749 |
| 3 | 57.143 | 31.641 | 18.254 | 11.072 | 7.053 |
| 4 | 64.286 | 38.579 | 23.322 | 14.503 | 9.348 |
| 5 | 73.810 | 46.024 | 28.414 | 17.903 | 11.626 |
| 6 | 83.333 | 53.807 | 33.740 | 21.409 | 13.941 |
| 7 | 88.095 | 59.222 | 38.081 | 24.546 | 16.134 |
| 8 | 90.476 | 63.113 | 41.764 | 27.457 | 18.267 |
| 9 | 95.238 | 68.697 | 46.222 | 30.647 | 20.478 |
| 10 | 100.000 | 73.435 | 50.000 | 33.457 | 22.514 |
| 11 | 100.000 | 75.465 | 52.557 | 35.784 | 24.384 |
| 12 | 100.000 | 76.988 | 54.716 | 37.896 | 26.161 |
| 13 | 100.000 | 79.019 | 57.227 | 40.164 | 27.984 |
| 14 | 100.000 | 80.372 | 59.338 | 42.282 | 29.767 |
| 15 | 100.000 | 82.234 | 61.614 | 44.368 | 31.487 |
| 16 | 100.000 | 83.249 | 63.210 | 46.078 | 33.038 |
| 17 | 100.000 | 84.772 | 65.204 | 47.992 | 34.667 |
| 18 | 100.000 | 86.125 | 67.058 | 49.810 | 36.230 |
| 19 | 100.000 | 87.310 | 68.700 | 51.466 | 37.702 |
| 20 | 100.000 | 88.325 | 70.155 | 52.994 | 39.107 |
| 21 | 100.000 | 89.171 | 71.422 | 54.356 | 40.383 |
| 22 | 100.000 | 89.679 | 72.407 | 55.568 | 41.602 |
| 23 | 100.000 | 90.355 | 73.534 | 56.849 | 42.841 |
| 24 | 100.000 | 91.201 | 74.847 | 58.238 | 44.098 |
| 25 | 100.000 | 91.878 | 75.950 | 59.477 | 45.289 |
| 26 | 100.000 | 92.555 | 77.053 | 60.715 | 46.480 |
| 27 | 100.000 | 92.893 | 77.851 | 61.771 | 47.590 |
| 28 | 100.000 | 93.401 | 78.719 | 62.822 | 48.664 |
| 29 | 100.000 | 93.909 | 79.634 | 63.911 | 49.751 |
| 30 | 100.000 | 94.416 | 80.526 | 64.967 | 50.811 |
| 31 | 100.000 | 94.755 | 81.276 | 65.937 | 51.822 |
| 32 | 100.000 | 95.262 | 82.145 | 66.956 | 52.836 |
| 33 | 100.000 | 95.431 | 82.708 | 67.798 | 53.784 |
| 34 | 100.000 | 95.601 | 83.294 | 68.656 | 54.724 |
| 35 | 100.000 | 95.939 | 83.975 | 69.535 | 55.665 |
| 36 | 100.000 | 96.108 | 84.514 | 70.334 | 56.566 |
| 37 | 100.000 | 96.616 | 85.312 | 71.256 | 57.501 |
| 38 | 100.000 | 96.954 | 85.946 | 72.050 | 58.339 |
| 39 | 100.000 | 97.293 | 86.579 | 72.838 | 59.166 |
| 40 | 100.000 | 97.462 | 87.001 | 73.471 | 59.911 |
| 41 | 100.000 | 97.631 | 87.447 | 74.125 | 60.669 |
| 42 | 100.000 | 97.800 | 87.893 | 74.768 | 61.403 |
| 43 | 100.000 | 97.970 | 88.339 | 75.412 | 62.137 |
| 44 | 100.000 | 97.970 | 88.620 | 75.958 | 62.840 |
| 45 | 100.000 | 98.139 | 89.066 | 76.602 | 63.574 |
| 46 | 100.000 | 98.308 | 89.465 | 77.192 | 64.277 |
| 47 | 100.000 | 98.308 | 89.747 | 77.722 | 64.946 |
| 48 | 100.000 | 98.477 | 90.145 | 78.291 | 65.599 |
| 49 | 100.000 | 98.646 | 90.544 | 78.854 | 66.244 |
| 50 | 100.000 | 98.816 | 90.896 | 79.358 | 66.845 |
| 51 | 100.000 | 98.816 | 91.107 | 79.781 | 67.409 |
| 52 | 100.000 | 99.154 | 91.600 | 80.350 | 68.012 |
| 53 | 100.000 | 99.323 | 91.976 | 80.864 | 68.593 |
| 54 | 100.000 | 99.323 | 92.187 | 81.283 | 69.138 |
| 55 | 100.000 | 99.323 | 92.374 | 81.669 | 69.664 |
| 56 | 100.000 | 99.323 | 92.562 | 82.055 | 70.189 |
| 57 | 100.000 | 99.492 | 92.914 | 82.532 | 70.731 |
| 58 | 100.000 | 99.492 | 93.102 | 82.907 | 71.232 |
| 59 | 100.000 | 99.492 | 93.290 | 83.272 | 71.709 |
| 60 | 100.000 | 99.492 | 93.454 | 83.615 | 72.182 |
| 61 | 100.000 | 99.662 | 93.782 | 84.054 | 72.678 |
| 62 | 100.000 | 99.662 | 93.947 | 84.392 | 73.140 |
| 63 | 100.000 | 99.831 | 94.252 | 84.800 | 73.615 |
| 64 | 100.000 | 99.831 | 94.416 | 85.132 | 74.067 |
| 65 | 100.000 | 99.831 | 94.580 | 85.465 | 74.518 |
| 66 | 100.000 | 99.831 | 94.721 | 85.770 | 74.954 |
| 67 | 100.000 | 99.831 | 94.838 | 86.054 | 75.385 |
| 68 | 100.000 | 99.831 | 94.979 | 86.355 | 75.809 |
| 69 | 100.000 | 99.831 | 95.120 | 86.650 | 76.221 |
| 70 | 100.000 | 99.831 | 95.260 | 86.944 | 76.631 |
| 71 | 100.000 | 99.831 | 95.401 | 87.239 | 77.042 |
| 72 | 100.000 | 99.831 | 95.542 | 87.534 | 77.456 |
| 73 | 100.000 | 99.831 | 95.683 | 87.824 | 77.856 |
| 74 | 100.000 | 99.831 | 95.800 | 88.092 | 78.251 |
| 75 | 100.000 | 99.831 | 95.941 | 88.376 | 78.638 |
| 76 | 100.000 | 99.831 | 96.082 | 88.660 | 79.017 |
| 77 | 100.000 | 99.831 | 96.222 | 88.939 | 79.389 |
| 78 | 100.000 | 99.831 | 96.363 | 89.212 | 79.747 |
| 79 | 100.000 | 99.831 | 96.481 | 89.459 | 80.095 |
| 80 | 100.000 | 99.831 | 96.574 | 89.684 | 80.439 |
| 81 | 100.000 | 99.831 | 96.692 | 89.931 | 80.787 |
| 82 | 100.000 | 99.831 | 96.786 | 90.151 | 81.118 |
| 83 | 100.000 | 99.831 | 96.879 | 90.360 | 81.426 |
| 84 | 100.000 | 99.831 | 96.973 | 90.569 | 81.735 |
| 85 | 100.000 | 99.831 | 97.020 | 90.735 | 82.037 |
| 86 | 100.000 | 99.831 | 97.067 | 90.901 | 82.334 |
| 87 | 100.000 | 99.831 | 97.137 | 91.084 | 82.630 |
| 88 | 100.000 | 99.831 | 97.184 | 91.244 | 82.921 |
| 89 | 100.000 | 100.000 | 97.443 | 91.539 | 83.226 |
| 90 | 100.000 | 100.000 | 97.489 | 91.695 | 83.503 |
| 91 | 100.000 | 100.000 | 97.536 | 91.850 | 83.781 |
| 92 | 100.000 | 100.000 | 97.630 | 92.049 | 84.060 |
| 93 | 100.000 | 100.000 | 97.724 | 92.242 | 84.328 |
| 94 | 100.000 | 100.000 | 97.818 | 92.435 | 84.596 |
| 95 | 100.000 | 100.000 | 97.888 | 92.606 | 84.862 |
| 96 | 100.000 | 100.000 | 97.959 | 92.773 | 85.119 |
| 97 | 100.000 | 100.000 | 98.029 | 92.939 | 85.376 |
| 98 | 100.000 | 100.000 | 98.099 | 93.100 | 85.621 |
| 99 | 100.000 | 100.000 | 98.170 | 93.260 | 85.866 |
| 100 | 100.000 | 100.000 | 98.217 | 93.400 | 86.109 |
| 101 | 100.000 | 100.000 | 98.287 | 93.561 | 86.355 |
| 102 | 100.000 | 100.000 | 98.334 | 93.695 | 86.584 |
| 103 | 100.000 | 100.000 | 98.405 | 93.845 | 86.800 |
| 104 | 100.000 | 100.000 | 98.475 | 93.995 | 87.017 |
| 105 | 100.000 | 100.000 | 98.522 | 94.124 | 87.231 |
| 106 | 100.000 | 100.000 | 98.569 | 94.247 | 87.436 |
| 107 | 100.000 | 100.000 | 98.616 | 94.370 | 87.642 |
| 108 | 100.000 | 100.000 | 98.663 | 94.494 | 87.849 |
| 109 | 100.000 | 100.000 | 98.710 | 94.612 | 88.039 |
| 110 | 100.000 | 100.000 | 98.756 | 94.730 | 88.230 |
| 111 | 100.000 | 100.000 | 98.803 | 94.842 | 88.411 |
| 112 | 100.000 | 100.000 | 98.850 | 94.955 | 88.588 |
| 113 | 100.000 | 100.000 | 98.897 | 95.067 | 88.770 |
| 114 | 100.000 | 100.000 | 98.944 | 95.180 | 88.949 |
| 115 | 100.000 | 100.000 | 98.944 | 95.244 | 89.104 |
| 116 | 100.000 | 100.000 | 98.991 | 95.351 | 89.269 |
| 117 | 100.000 | 100.000 | 99.038 | 95.459 | 89.434 |
| 118 | 100.000 | 100.000 | 99.061 | 95.544 | 89.600 |
| 119 | 100.000 | 100.000 | 99.061 | 95.609 | 89.763 |
| 120 | 100.000 | 100.000 | 99.108 | 95.716 | 89.931 |
| 121 | 100.000 | 100.000 | 99.155 | 95.818 | 90.081 |
| 122 | 100.000 | 100.000 | 99.179 | 95.898 | 90.234 |
| 123 | 100.000 | 100.000 | 99.226 | 96.000 | 90.384 |
| 124 | 100.000 | 100.000 | 99.249 | 96.081 | 90.538 |
| 125 | 100.000 | 100.000 | 99.296 | 96.183 | 90.687 |
| 126 | 100.000 | 100.000 | 99.343 | 96.284 | 90.837 |
| 127 | 100.000 | 100.000 | 99.390 | 96.386 | 90.987 |
| 128 | 100.000 | 100.000 | 99.413 | 96.461 | 91.126 |
| 129 | 100.000 | 100.000 | 99.437 | 96.536 | 91.266 |
| 130 | 100.000 | 100.000 | 99.437 | 96.590 | 91.405 |
| 131 | 100.000 | 100.000 | 99.460 | 96.665 | 91.545 |
| 132 | 100.000 | 100.000 | 99.484 | 96.735 | 91.671 |
| 133 | 100.000 | 100.000 | 99.507 | 96.804 | 91.797 |
| 134 | 100.000 | 100.000 | 99.531 | 96.869 | 91.908 |
| 135 | 100.000 | 100.000 | 99.531 | 96.912 | 92.022 |
| 136 | 100.000 | 100.000 | 99.554 | 96.976 | 92.135 |
| 137 | 100.000 | 100.000 | 99.554 | 97.019 | 92.246 |
| 138 | 100.000 | 100.000 | 99.578 | 97.083 | 92.357 |
| 139 | 100.000 | 100.000 | 99.601 | 97.148 | 92.468 |
| 140 | 100.000 | 100.000 | 99.625 | 97.212 | 92.581 |
| 141 | 100.000 | 100.000 | 99.648 | 97.276 | 92.696 |
| 142 | 100.000 | 100.000 | 99.672 | 97.341 | 92.807 |
| 143 | 100.000 | 100.000 | 99.695 | 97.405 | 92.918 |
| 144 | 100.000 | 100.000 | 99.718 | 97.469 | 93.028 |
| 145 | 100.000 | 100.000 | 99.742 | 97.528 | 93.126 |
| 146 | 100.000 | 100.000 | 99.765 | 97.587 | 93.224 |
| 147 | 100.000 | 100.000 | 99.789 | 97.646 | 93.322 |
| 148 | 100.000 | 100.000 | 99.812 | 97.705 | 93.422 |
| 149 | 100.000 | 100.000 | 99.812 | 97.743 | 93.522 |
| 150 | 100.000 | 100.000 | 99.836 | 97.802 | 93.620 |
| 151 | 100.000 | 100.000 | 99.836 | 97.839 | 93.720 |
| 152 | 100.000 | 100.000 | 99.859 | 97.898 | 93.818 |
| 153 | 100.000 | 100.000 | 99.883 | 97.957 | 93.918 |
| 154 | 100.000 | 100.000 | 99.883 | 97.995 | 94.014 |
| 155 | 100.000 | 100.000 | 99.906 | 98.054 | 94.112 |
| 156 | 100.000 | 100.000 | 99.906 | 98.086 | 94.199 |
| 157 | 100.000 | 100.000 | 99.906 | 98.118 | 94.286 |
| 158 | 100.000 | 100.000 | 99.930 | 98.172 | 94.369 |
| 159 | 100.000 | 100.000 | 99.953 | 98.225 | 94.452 |
| 160 | 100.000 | 100.000 | 99.953 | 98.257 | 94.539 |
| 161 | 100.000 | 100.000 | 99.953 | 98.290 | 94.626 |
| 162 | 100.000 | 100.000 | 99.977 | 98.343 | 94.709 |
| 163 | 100.000 | 100.000 | 99.977 | 98.375 | 94.796 |
| 164 | 100.000 | 100.000 | 100.000 | 98.429 | 94.880 |
| 165 | 100.000 | 100.000 | 100.000 | 98.461 | 94.966 |
| 166 | 100.000 | 100.000 | 100.000 | 98.493 | 95.053 |
| 167 | 100.000 | 100.000 | 100.000 | 98.526 | 95.140 |
| 168 | 100.000 | 100.000 | 100.000 | 98.552 | 95.216 |
| 169 | 100.000 | 100.000 | 100.000 | 98.579 | 95.288 |
| 170 | 100.000 | 100.000 | 100.000 | 98.606 | 95.368 |
| 171 | 100.000 | 100.000 | 100.000 | 98.633 | 95.440 |
| 172 | 100.000 | 100.000 | 100.000 | 98.660 | 95.512 |
| 173 | 100.000 | 100.000 | 100.000 | 98.686 | 95.588 |
| 174 | 100.000 | 100.000 | 100.000 | 98.713 | 95.664 |
| 175 | 100.000 | 100.000 | 100.000 | 98.735 | 95.723 |
| 176 | 100.000 | 100.000 | 100.000 | 98.756 | 95.793 |
| 177 | 100.000 | 100.000 | 100.000 | 98.778 | 95.860 |
| 178 | 100.000 | 100.000 | 100.000 | 98.799 | 95.922 |
| 179 | 100.000 | 100.000 | 100.000 | 98.820 | 95.984 |
| 180 | 100.000 | 100.000 | 100.000 | 98.842 | 96.045 |
| 181 | 100.000 | 100.000 | 100.000 | 98.863 | 96.104 |
| 182 | 100.000 | 100.000 | 100.000 | 98.885 | 96.167 |
| 183 | 100.000 | 100.000 | 100.000 | 98.906 | 96.226 |
| 184 | 100.000 | 100.000 | 100.000 | 98.928 | 96.289 |
| 185 | 100.000 | 100.000 | 100.000 | 98.949 | 96.348 |
| 186 | 100.000 | 100.000 | 100.000 | 98.971 | 96.411 |
| 187 | 100.000 | 100.000 | 100.000 | 98.992 | 96.474 |
| 188 | 100.000 | 100.000 | 100.000 | 99.013 | 96.533 |
| 189 | 100.000 | 100.000 | 100.000 | 99.035 | 96.592 |
| 190 | 100.000 | 100.000 | 100.000 | 99.056 | 96.655 |
| 191 | 100.000 | 100.000 | 100.000 | 99.078 | 96.714 |
| 192 | 100.000 | 100.000 | 100.000 | 99.099 | 96.777 |
| 193 | 100.000 | 100.000 | 100.000 | 99.121 | 96.836 |
| 194 | 100.000 | 100.000 | 100.000 | 99.142 | 96.895 |
| 195 | 100.000 | 100.000 | 100.000 | 99.164 | 96.958 |
| 196 | 100.000 | 100.000 | 100.000 | 99.185 | 97.021 |
| 197 | 100.000 | 100.000 | 100.000 | 99.201 | 97.067 |
| 198 | 100.000 | 100.000 | 100.000 | 99.217 | 97.113 |
| 199 | 100.000 | 100.000 | 100.000 | 99.233 | 97.163 |
| 200 | 100.000 | 100.000 | 100.000 | 99.249 | 97.213 |
| 201 | 100.000 | 100.000 | 100.000 | 99.265 | 97.259 |
| 202 | 100.000 | 100.000 | 100.000 | 99.282 | 97.306 |
| 203 | 100.000 | 100.000 | 100.000 | 99.298 | 97.354 |
| 204 | 100.000 | 100.000 | 100.000 | 99.314 | 97.400 |
| 205 | 100.000 | 100.000 | 100.000 | 99.330 | 97.444 |
| 206 | 100.000 | 100.000 | 100.000 | 99.346 | 97.491 |
| 207 | 100.000 | 100.000 | 100.000 | 99.362 | 97.537 |
| 208 | 100.000 | 100.000 | 100.000 | 99.373 | 97.570 |
| 209 | 100.000 | 100.000 | 100.000 | 99.383 | 97.603 |
| 210 | 100.000 | 100.000 | 100.000 | 99.394 | 97.635 |
| 211 | 100.000 | 100.000 | 100.000 | 99.405 | 97.666 |
| 212 | 100.000 | 100.000 | 100.000 | 99.416 | 97.698 |
| 213 | 100.000 | 100.000 | 100.000 | 99.426 | 97.729 |
| 214 | 100.000 | 100.000 | 100.000 | 99.437 | 97.762 |
| 215 | 100.000 | 100.000 | 100.000 | 99.448 | 97.794 |
| 216 | 100.000 | 100.000 | 100.000 | 99.458 | 97.825 |
| 217 | 100.000 | 100.000 | 100.000 | 99.469 | 97.859 |
| 218 | 100.000 | 100.000 | 100.000 | 99.480 | 97.890 |
| 219 | 100.000 | 100.000 | 100.000 | 99.491 | 97.925 |
| 220 | 100.000 | 100.000 | 100.000 | 99.501 | 97.958 |
| 221 | 100.000 | 100.000 | 100.000 | 99.512 | 97.994 |
| 222 | 100.000 | 100.000 | 100.000 | 99.523 | 98.025 |
| 223 | 100.000 | 100.000 | 100.000 | 99.534 | 98.056 |
| 224 | 100.000 | 100.000 | 100.000 | 99.544 | 98.090 |
| 225 | 100.000 | 100.000 | 100.000 | 99.555 | 98.125 |
| 226 | 100.000 | 100.000 | 100.000 | 99.566 | 98.156 |
| 227 | 100.000 | 100.000 | 100.000 | 99.576 | 98.188 |
| 228 | 100.000 | 100.000 | 100.000 | 99.587 | 98.223 |
| 229 | 100.000 | 100.000 | 100.000 | 99.598 | 98.254 |
| 230 | 100.000 | 100.000 | 100.000 | 99.609 | 98.288 |
| 231 | 100.000 | 100.000 | 100.000 | 99.619 | 98.325 |
| 232 | 100.000 | 100.000 | 100.000 | 99.630 | 98.362 |
| 233 | 100.000 | 100.000 | 100.000 | 99.641 | 98.393 |
| 234 | 100.000 | 100.000 | 100.000 | 99.651 | 98.424 |
| 235 | 100.000 | 100.000 | 100.000 | 99.662 | 98.458 |
| 236 | 100.000 | 100.000 | 100.000 | 99.673 | 98.489 |
| 237 | 100.000 | 100.000 | 100.000 | 99.684 | 98.521 |
| 238 | 100.000 | 100.000 | 100.000 | 99.694 | 98.552 |
| 239 | 100.000 | 100.000 | 100.000 | 99.705 | 98.584 |
| 240 | 100.000 | 100.000 | 100.000 | 99.716 | 98.615 |
| 241 | 100.000 | 100.000 | 100.000 | 99.721 | 98.632 |
| 242 | 100.000 | 100.000 | 100.000 | 99.727 | 98.648 |
| 243 | 100.000 | 100.000 | 100.000 | 99.732 | 98.665 |
| 244 | 100.000 | 100.000 | 100.000 | 99.737 | 98.683 |
| 245 | 100.000 | 100.000 | 100.000 | 99.743 | 98.700 |
| 246 | 100.000 | 100.000 | 100.000 | 99.748 | 98.717 |
| 247 | 100.000 | 100.000 | 100.000 | 99.753 | 98.733 |
| 248 | 100.000 | 100.000 | 100.000 | 99.759 | 98.750 |
| 249 | 100.000 | 100.000 | 100.000 | 99.764 | 98.767 |
| 250 | 100.000 | 100.000 | 100.000 | 99.769 | 98.783 |
| 251 | 100.000 | 100.000 | 100.000 | 99.775 | 98.802 |
| 252 | 100.000 | 100.000 | 100.000 | 99.780 | 98.818 |
| 253 | 100.000 | 100.000 | 100.000 | 99.786 | 98.835 |
| 254 | 100.000 | 100.000 | 100.000 | 99.791 | 98.852 |
| 255 | 100.000 | 100.000 | 100.000 | 99.796 | 98.868 |
| 256 | 100.000 | 100.000 | 100.000 | 99.802 | 98.885 |
| 257 | 100.000 | 100.000 | 100.000 | 99.807 | 98.902 |
| 258 | 100.000 | 100.000 | 100.000 | 99.812 | 98.918 |
| 259 | 100.000 | 100.000 | 100.000 | 99.818 | 98.937 |
| 260 | 100.000 | 100.000 | 100.000 | 99.823 | 98.953 |
| 261 | 100.000 | 100.000 | 100.000 | 99.828 | 98.970 |
| 262 | 100.000 | 100.000 | 100.000 | 99.834 | 98.987 |
| 263 | 100.000 | 100.000 | 100.000 | 99.839 | 99.003 |
| 264 | 100.000 | 100.000 | 100.000 | 99.845 | 99.020 |
| 265 | 100.000 | 100.000 | 100.000 | 99.850 | 99.037 |
| 266 | 100.000 | 100.000 | 100.000 | 99.855 | 99.053 |
| 267 | 100.000 | 100.000 | 100.000 | 99.861 | 99.070 |
| 268 | 100.000 | 100.000 | 100.000 | 99.866 | 99.088 |
| 269 | 100.000 | 100.000 | 100.000 | 99.871 | 99.107 |
| 270 | 100.000 | 100.000 | 100.000 | 99.877 | 99.123 |
| 271 | 100.000 | 100.000 | 100.000 | 99.882 | 99.144 |
| 272 | 100.000 | 100.000 | 100.000 | 99.887 | 99.160 |
| 273 | 100.000 | 100.000 | 100.000 | 99.893 | 99.177 |
| 274 | 100.000 | 100.000 | 100.000 | 99.898 | 99.196 |
| 275 | 100.000 | 100.000 | 100.000 | 99.903 | 99.216 |
| 276 | 100.000 | 100.000 | 100.000 | 99.909 | 99.233 |
| 277 | 100.000 | 100.000 | 100.000 | 99.914 | 99.249 |
| 278 | 100.000 | 100.000 | 100.000 | 99.920 | 99.266 |
| 279 | 100.000 | 100.000 | 100.000 | 99.925 | 99.283 |
| 280 | 100.000 | 100.000 | 100.000 | 99.930 | 99.299 |
| 281 | 100.000 | 100.000 | 100.000 | 99.936 | 99.316 |
| 282 | 100.000 | 100.000 | 100.000 | 99.941 | 99.332 |
| 283 | 100.000 | 100.000 | 100.000 | 99.946 | 99.351 |
| 284 | 100.000 | 100.000 | 100.000 | 99.952 | 99.368 |
| 285 | 100.000 | 100.000 | 100.000 | 99.957 | 99.384 |
| 286 | 100.000 | 100.000 | 100.000 | 99.962 | 99.405 |
| 287 | 100.000 | 100.000 | 100.000 | 99.968 | 99.421 |
| 288 | 100.000 | 100.000 | 100.000 | 99.973 | 99.438 |
| 289 | 100.000 | 100.000 | 100.000 | 99.979 | 99.454 |
| 290 | 100.000 | 100.000 | 100.000 | 99.984 | 99.471 |
| 291 | 100.000 | 100.000 | 100.000 | 99.989 | 99.488 |
| 292 | 100.000 | 100.000 | 100.000 | 99.995 | 99.504 |
| 293 | 100.000 | 100.000 | 100.000 | 100.000 | 99.521 |
| 294 | 100.000 | 100.000 | 100.000 | 100.000 | 99.525 |
| 295 | 100.000 | 100.000 | 100.000 | 100.000 | 99.532 |
| 296 | 100.000 | 100.000 | 100.000 | 100.000 | 99.536 |
| 297 | 100.000 | 100.000 | 100.000 | 100.000 | 99.538 |
| 298 | 100.000 | 100.000 | 100.000 | 100.000 | 99.540 |
| 299 | 100.000 | 100.000 | 100.000 | 100.000 | 99.541 |
| 300 | 100.000 | 100.000 | 100.000 | 100.000 | 99.543 |
| 301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.545 |
| 302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.547 |
| 303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
| 304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.551 |
| 305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.554 |
| 306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.562 |
| 307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
| 308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.567 |
| 309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.571 |
| 310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.575 |
| 311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.578 |
| 312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.582 |
| 313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.586 |
| 314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.589 |
| 315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.593 |
| 316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.597 |
| 317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
| 318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.602 |
| 319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.606 |
| 320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.610 |
| 321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.612 |
| 322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
| 323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
| 324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.625 |
| 325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.626 |
| 326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
| 327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.632 |
| 328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.634 |
| 329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.641 |
| 330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.643 |
| 331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
| 332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.649 |
| 333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.652 |
| 334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
| 335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
| 336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.660 |
| 337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
| 338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
| 339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.671 |
| 340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.678 |
| 341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
| 342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.682 |
| 343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.686 |
| 344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
| 345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
| 346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
| 347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
| 348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.700 |
| 349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.704 |
| 350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.708 |
| 351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.712 |
| 352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.713 |
| 353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.715 |
| 354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.723 |
| 355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.724 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.726 |
| 357 | 100.000 | 100.000 | 100.000 | 100.000 | 99.728 |
| 358 | 100.000 | 100.000 | 100.000 | 100.000 | 99.730 |
| 359 | 100.000 | 100.000 | 100.000 | 100.000 | 99.734 |
| 360 | 100.000 | 100.000 | 100.000 | 100.000 | 99.737 |
| 361 | 100.000 | 100.000 | 100.000 | 100.000 | 99.745 |
| 362 | 100.000 | 100.000 | 100.000 | 100.000 | 99.749 |
| 363 | 100.000 | 100.000 | 100.000 | 100.000 | 99.752 |
| 364 | 100.000 | 100.000 | 100.000 | 100.000 | 99.754 |
| 365 | 100.000 | 100.000 | 100.000 | 100.000 | 99.756 |
| 366 | 100.000 | 100.000 | 100.000 | 100.000 | 99.760 |
| 367 | 100.000 | 100.000 | 100.000 | 100.000 | 99.761 |
| 368 | 100.000 | 100.000 | 100.000 | 100.000 | 99.776 |
| 369 | 100.000 | 100.000 | 100.000 | 100.000 | 99.778 |
| 370 | 100.000 | 100.000 | 100.000 | 100.000 | 99.782 |
| 371 | 100.000 | 100.000 | 100.000 | 100.000 | 99.789 |
| 372 | 100.000 | 100.000 | 100.000 | 100.000 | 99.793 |
| 373 | 100.000 | 100.000 | 100.000 | 100.000 | 99.797 |
| 374 | 100.000 | 100.000 | 100.000 | 100.000 | 99.800 |
| 375 | 100.000 | 100.000 | 100.000 | 100.000 | 99.802 |
| 376 | 100.000 | 100.000 | 100.000 | 100.000 | 99.806 |
| 377 | 100.000 | 100.000 | 100.000 | 100.000 | 99.810 |
| 378 | 100.000 | 100.000 | 100.000 | 100.000 | 99.813 |
| 379 | 100.000 | 100.000 | 100.000 | 100.000 | 99.817 |
| 380 | 100.000 | 100.000 | 100.000 | 100.000 | 99.821 |
| 381 | 100.000 | 100.000 | 100.000 | 100.000 | 99.822 |
| 382 | 100.000 | 100.000 | 100.000 | 100.000 | 99.826 |
| 383 | 100.000 | 100.000 | 100.000 | 100.000 | 99.830 |
| 384 | 100.000 | 100.000 | 100.000 | 100.000 | 99.835 |
| 385 | 100.000 | 100.000 | 100.000 | 100.000 | 99.837 |
| 386 | 100.000 | 100.000 | 100.000 | 100.000 | 99.841 |
| 387 | 100.000 | 100.000 | 100.000 | 100.000 | 99.843 |
| 388 | 100.000 | 100.000 | 100.000 | 100.000 | 99.847 |
| 389 | 100.000 | 100.000 | 100.000 | 100.000 | 99.850 |
| 390 | 100.000 | 100.000 | 100.000 | 100.000 | 99.852 |
| 391 | 100.000 | 100.000 | 100.000 | 100.000 | 99.854 |
| 392 | 100.000 | 100.000 | 100.000 | 100.000 | 99.858 |
| 393 | 100.000 | 100.000 | 100.000 | 100.000 | 99.861 |
| 394 | 100.000 | 100.000 | 100.000 | 100.000 | 99.863 |
| 395 | 100.000 | 100.000 | 100.000 | 100.000 | 99.865 |
| 396 | 100.000 | 100.000 | 100.000 | 100.000 | 99.869 |
| 397 | 100.000 | 100.000 | 100.000 | 100.000 | 99.872 |
| 398 | 100.000 | 100.000 | 100.000 | 100.000 | 99.874 |
| 399 | 100.000 | 100.000 | 100.000 | 100.000 | 99.876 |
| 400 | 100.000 | 100.000 | 100.000 | 100.000 | 99.880 |
| 401 | 100.000 | 100.000 | 100.000 | 100.000 | 99.883 |
| 402 | 100.000 | 100.000 | 100.000 | 100.000 | 99.887 |
| 403 | 100.000 | 100.000 | 100.000 | 100.000 | 99.889 |
| 404 | 100.000 | 100.000 | 100.000 | 100.000 | 99.893 |
| 405 | 100.000 | 100.000 | 100.000 | 100.000 | 99.895 |
| 406 | 100.000 | 100.000 | 100.000 | 100.000 | 99.896 |
| 407 | 100.000 | 100.000 | 100.000 | 100.000 | 99.900 |
| 408 | 100.000 | 100.000 | 100.000 | 100.000 | 99.908 |
| 409 | 100.000 | 100.000 | 100.000 | 100.000 | 99.911 |
| 410 | 100.000 | 100.000 | 100.000 | 100.000 | 99.913 |
| 411 | 100.000 | 100.000 | 100.000 | 100.000 | 99.917 |
| 412 | 100.000 | 100.000 | 100.000 | 100.000 | 99.920 |
| 413 | 100.000 | 100.000 | 100.000 | 100.000 | 99.922 |
| 414 | 100.000 | 100.000 | 100.000 | 100.000 | 99.924 |
| 415 | 100.000 | 100.000 | 100.000 | 100.000 | 99.928 |
| 416 | 100.000 | 100.000 | 100.000 | 100.000 | 99.930 |
| 417 | 100.000 | 100.000 | 100.000 | 100.000 | 99.932 |
| 418 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
| 419 | 100.000 | 100.000 | 100.000 | 100.000 | 99.937 |
| 420 | 100.000 | 100.000 | 100.000 | 100.000 | 99.941 |
| 421 | 100.000 | 100.000 | 100.000 | 100.000 | 99.945 |
| 422 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
| 423 | 100.000 | 100.000 | 100.000 | 100.000 | 99.948 |
| 424 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
| 425 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
| 426 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 427 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
| 428 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
| 429 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
| 430 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
| 431 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
| 433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
grep
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 30.952 | 13.198 | 6.710 | 3.834 | 2.380 |
| 2 | 42.857 | 21.658 | 12.107 | 7.292 | 4.656 |
| 3 | 54.762 | 30.288 | 17.644 | 10.831 | 6.970 |
| 4 | 61.905 | 36.717 | 22.337 | 14.074 | 9.191 |
| 5 | 69.048 | 42.978 | 26.889 | 17.238 | 11.375 |
| 6 | 73.810 | 48.562 | 31.347 | 20.428 | 13.586 |
| 7 | 80.952 | 54.992 | 36.039 | 23.672 | 15.807 |
| 8 | 80.952 | 57.360 | 38.878 | 26.165 | 17.760 |
| 9 | 80.952 | 59.729 | 41.671 | 28.610 | 19.683 |
| 10 | 83.333 | 62.944 | 44.721 | 31.124 | 21.617 |
| 11 | 85.714 | 66.497 | 48.076 | 33.816 | 23.629 |
| 12 | 90.476 | 71.066 | 51.689 | 36.518 | 25.610 |
| 13 | 92.857 | 73.773 | 54.247 | 38.711 | 27.376 |
| 14 | 92.857 | 75.635 | 56.546 | 40.813 | 29.097 |
| 15 | 95.238 | 78.173 | 58.916 | 42.866 | 30.780 |
| 16 | 95.238 | 79.357 | 60.605 | 44.587 | 32.308 |
| 17 | 95.238 | 80.880 | 62.529 | 46.405 | 33.854 |
| 18 | 100.000 | 85.110 | 65.791 | 48.845 | 35.666 |
| 19 | 100.000 | 85.787 | 67.034 | 50.276 | 37.025 |
| 20 | 100.000 | 86.802 | 68.536 | 51.842 | 38.440 |
| 21 | 100.000 | 87.648 | 69.873 | 53.295 | 39.790 |
| 22 | 100.000 | 88.663 | 71.305 | 54.775 | 41.139 |
| 23 | 100.000 | 89.679 | 72.877 | 56.421 | 42.606 |
| 24 | 100.000 | 90.355 | 73.956 | 57.648 | 43.808 |
| 25 | 100.000 | 91.371 | 75.411 | 59.144 | 45.156 |
| 26 | 100.000 | 91.878 | 76.396 | 60.335 | 46.338 |
| 27 | 100.000 | 92.386 | 77.311 | 61.434 | 47.449 |
| 28 | 100.000 | 93.232 | 78.555 | 62.753 | 48.677 |
| 29 | 100.000 | 93.739 | 79.423 | 63.787 | 49.729 |
| 30 | 100.000 | 93.909 | 80.033 | 64.683 | 50.715 |
| 31 | 100.000 | 94.247 | 80.760 | 65.621 | 51.699 |
| 32 | 100.000 | 94.247 | 81.206 | 66.399 | 52.605 |
| 33 | 100.000 | 94.416 | 81.793 | 67.251 | 53.538 |
| 34 | 100.000 | 94.585 | 82.309 | 68.012 | 54.400 |
| 35 | 100.000 | 95.431 | 83.482 | 69.213 | 55.499 |
| 36 | 100.000 | 95.939 | 84.280 | 70.136 | 56.434 |
| 37 | 100.000 | 96.277 | 84.913 | 70.945 | 57.305 |
| 38 | 100.000 | 96.785 | 85.758 | 71.921 | 58.274 |
| 39 | 100.000 | 97.124 | 86.368 | 72.693 | 59.108 |
| 40 | 100.000 | 97.462 | 86.978 | 73.455 | 59.920 |
| 41 | 100.000 | 97.462 | 87.236 | 73.964 | 60.586 |
| 42 | 100.000 | 97.631 | 87.705 | 74.639 | 61.348 |
| 43 | 100.000 | 97.631 | 87.940 | 75.106 | 61.967 |
| 44 | 100.000 | 97.800 | 88.386 | 75.749 | 62.701 |
| 45 | 100.000 | 97.800 | 88.644 | 76.248 | 63.345 |
| 46 | 100.000 | 98.308 | 89.395 | 77.079 | 64.170 |
| 47 | 100.000 | 98.308 | 89.747 | 77.712 | 64.922 |
| 48 | 100.000 | 98.477 | 90.169 | 78.312 | 65.606 |
| 49 | 100.000 | 98.477 | 90.286 | 78.623 | 66.091 |
| 50 | 100.000 | 98.646 | 90.685 | 79.186 | 66.738 |
| 51 | 100.000 | 98.646 | 90.873 | 79.583 | 67.284 |
| 52 | 100.000 | 98.646 | 91.084 | 80.006 | 67.848 |
| 53 | 100.000 | 98.646 | 91.295 | 80.430 | 68.412 |
| 54 | 100.000 | 98.646 | 91.506 | 80.848 | 68.965 |
| 55 | 100.000 | 98.985 | 92.046 | 81.486 | 69.634 |
| 56 | 100.000 | 98.985 | 92.234 | 81.862 | 70.139 |
| 57 | 100.000 | 98.985 | 92.445 | 82.280 | 70.684 |
| 58 | 100.000 | 98.985 | 92.586 | 82.596 | 71.143 |
| 59 | 100.000 | 99.154 | 92.938 | 83.084 | 71.709 |
| 60 | 100.000 | 99.323 | 93.243 | 83.513 | 72.228 |
| 61 | 100.000 | 99.323 | 93.430 | 83.888 | 72.730 |
| 62 | 100.000 | 99.323 | 93.571 | 84.199 | 73.175 |
| 63 | 100.000 | 99.323 | 93.759 | 84.569 | 73.665 |
| 64 | 100.000 | 99.323 | 93.876 | 84.848 | 74.085 |
| 65 | 100.000 | 99.323 | 94.040 | 85.186 | 74.547 |
| 66 | 100.000 | 99.323 | 94.134 | 85.432 | 74.939 |
| 67 | 100.000 | 99.323 | 94.252 | 85.700 | 75.335 |
| 68 | 100.000 | 99.323 | 94.345 | 85.931 | 75.694 |
| 69 | 100.000 | 99.323 | 94.416 | 86.129 | 76.023 |
| 70 | 100.000 | 99.323 | 94.557 | 86.419 | 76.423 |
| 71 | 100.000 | 99.323 | 94.697 | 86.708 | 76.822 |
| 72 | 100.000 | 99.323 | 94.838 | 86.998 | 77.221 |
| 73 | 100.000 | 99.323 | 94.955 | 87.255 | 77.593 |
| 74 | 100.000 | 99.323 | 95.026 | 87.448 | 77.909 |
| 75 | 100.000 | 99.323 | 95.073 | 87.609 | 78.200 |
| 76 | 100.000 | 99.323 | 95.143 | 87.802 | 78.516 |
| 77 | 100.000 | 99.492 | 95.448 | 88.194 | 78.958 |
| 78 | 100.000 | 99.662 | 95.753 | 88.585 | 79.398 |
| 79 | 100.000 | 99.662 | 95.847 | 88.805 | 79.731 |
| 80 | 100.000 | 99.831 | 96.176 | 89.228 | 80.193 |
| 81 | 100.000 | 99.831 | 96.269 | 89.443 | 80.515 |
| 82 | 100.000 | 99.831 | 96.340 | 89.625 | 80.809 |
| 83 | 100.000 | 99.831 | 96.481 | 89.909 | 81.188 |
| 84 | 100.000 | 99.831 | 96.574 | 90.124 | 81.508 |
| 85 | 100.000 | 99.831 | 96.715 | 90.408 | 81.887 |
| 86 | 100.000 | 99.831 | 96.809 | 90.612 | 82.187 |
| 87 | 100.000 | 99.831 | 96.809 | 90.698 | 82.394 |
| 88 | 100.000 | 99.831 | 96.903 | 90.901 | 82.689 |
| 89 | 100.000 | 99.831 | 96.973 | 91.073 | 82.958 |
| 90 | 100.000 | 99.831 | 97.067 | 91.271 | 83.242 |
| 91 | 100.000 | 99.831 | 97.137 | 91.437 | 83.499 |
| 92 | 100.000 | 99.831 | 97.208 | 91.604 | 83.756 |
| 93 | 100.000 | 99.831 | 97.255 | 91.738 | 83.986 |
| 94 | 100.000 | 99.831 | 97.302 | 91.872 | 84.215 |
| 95 | 100.000 | 99.831 | 97.396 | 92.070 | 84.496 |
| 96 | 100.000 | 99.831 | 97.466 | 92.231 | 84.742 |
| 97 | 100.000 | 99.831 | 97.536 | 92.392 | 84.986 |
| 98 | 100.000 | 99.831 | 97.583 | 92.521 | 85.204 |
| 99 | 100.000 | 99.831 | 97.677 | 92.714 | 85.473 |
| 100 | 100.000 | 99.831 | 97.771 | 92.907 | 85.741 |
| 101 | 100.000 | 99.831 | 97.818 | 93.030 | 85.948 |
| 102 | 100.000 | 99.831 | 97.888 | 93.185 | 86.181 |
| 103 | 100.000 | 99.831 | 97.935 | 93.309 | 86.386 |
| 104 | 100.000 | 99.831 | 98.006 | 93.464 | 86.617 |
| 105 | 100.000 | 99.831 | 98.076 | 93.620 | 86.848 |
| 106 | 100.000 | 99.831 | 98.146 | 93.775 | 87.078 |
| 107 | 100.000 | 99.831 | 98.217 | 93.931 | 87.307 |
| 108 | 100.000 | 99.831 | 98.264 | 94.049 | 87.501 |
| 109 | 100.000 | 99.831 | 98.264 | 94.113 | 87.656 |
| 110 | 100.000 | 99.831 | 98.334 | 94.263 | 87.873 |
| 111 | 100.000 | 100.000 | 98.592 | 94.558 | 88.178 |
| 112 | 100.000 | 100.000 | 98.639 | 94.676 | 88.368 |
| 113 | 100.000 | 100.000 | 98.710 | 94.826 | 88.585 |
| 114 | 100.000 | 100.000 | 98.756 | 94.939 | 88.768 |
| 115 | 100.000 | 100.000 | 98.780 | 95.019 | 88.923 |
| 116 | 100.000 | 100.000 | 98.827 | 95.132 | 89.103 |
| 117 | 100.000 | 100.000 | 98.897 | 95.276 | 89.308 |
| 118 | 100.000 | 100.000 | 98.921 | 95.357 | 89.461 |
| 119 | 100.000 | 100.000 | 98.944 | 95.437 | 89.615 |
| 120 | 100.000 | 100.000 | 99.015 | 95.582 | 89.820 |
| 121 | 100.000 | 100.000 | 99.061 | 95.695 | 89.998 |
| 122 | 100.000 | 100.000 | 99.108 | 95.807 | 90.175 |
| 123 | 100.000 | 100.000 | 99.155 | 95.920 | 90.353 |
| 124 | 100.000 | 100.000 | 99.202 | 96.032 | 90.530 |
| 125 | 100.000 | 100.000 | 99.226 | 96.107 | 90.673 |
| 126 | 100.000 | 100.000 | 99.273 | 96.215 | 90.841 |
| 127 | 100.000 | 100.000 | 99.273 | 96.258 | 90.955 |
| 128 | 100.000 | 100.000 | 99.296 | 96.333 | 91.094 |
| 129 | 100.000 | 100.000 | 99.320 | 96.408 | 91.233 |
| 130 | 100.000 | 100.000 | 99.366 | 96.515 | 91.397 |
| 131 | 100.000 | 100.000 | 99.413 | 96.622 | 91.562 |
| 132 | 100.000 | 100.000 | 99.460 | 96.729 | 91.727 |
| 133 | 100.000 | 100.000 | 99.460 | 96.762 | 91.821 |
| 134 | 100.000 | 100.000 | 99.484 | 96.831 | 91.947 |
| 135 | 100.000 | 100.000 | 99.484 | 96.869 | 92.047 |
| 136 | 100.000 | 100.000 | 99.507 | 96.939 | 92.172 |
| 137 | 100.000 | 100.000 | 99.507 | 96.976 | 92.272 |
| 138 | 100.000 | 100.000 | 99.531 | 97.046 | 92.398 |
| 139 | 100.000 | 100.000 | 99.554 | 97.115 | 92.524 |
| 140 | 100.000 | 100.000 | 99.601 | 97.217 | 92.673 |
| 141 | 100.000 | 100.000 | 99.648 | 97.319 | 92.823 |
| 142 | 100.000 | 100.000 | 99.672 | 97.384 | 92.938 |
| 143 | 100.000 | 100.000 | 99.672 | 97.421 | 93.034 |
| 144 | 100.000 | 100.000 | 99.672 | 97.453 | 93.121 |
| 145 | 100.000 | 100.000 | 99.672 | 97.485 | 93.208 |
| 146 | 100.000 | 100.000 | 99.672 | 97.512 | 93.287 |
| 147 | 100.000 | 100.000 | 99.672 | 97.544 | 93.374 |
| 148 | 100.000 | 100.000 | 99.672 | 97.577 | 93.461 |
| 149 | 100.000 | 100.000 | 99.672 | 97.609 | 93.548 |
| 150 | 100.000 | 100.000 | 99.672 | 97.641 | 93.635 |
| 151 | 100.000 | 100.000 | 99.695 | 97.705 | 93.746 |
| 152 | 100.000 | 100.000 | 99.695 | 97.732 | 93.822 |
| 153 | 100.000 | 100.000 | 99.695 | 97.759 | 93.898 |
| 154 | 100.000 | 100.000 | 99.695 | 97.786 | 93.973 |
| 155 | 100.000 | 100.000 | 99.695 | 97.812 | 94.049 |
| 156 | 100.000 | 100.000 | 99.718 | 97.871 | 94.149 |
| 157 | 100.000 | 100.000 | 99.718 | 97.898 | 94.223 |
| 158 | 100.000 | 100.000 | 99.742 | 97.957 | 94.323 |
| 159 | 100.000 | 100.000 | 99.765 | 98.016 | 94.421 |
| 160 | 100.000 | 100.000 | 99.765 | 98.043 | 94.493 |
| 161 | 100.000 | 100.000 | 99.789 | 98.102 | 94.591 |
| 162 | 100.000 | 100.000 | 99.812 | 98.161 | 94.689 |
| 163 | 100.000 | 100.000 | 99.836 | 98.220 | 94.787 |
| 164 | 100.000 | 100.000 | 99.836 | 98.247 | 94.859 |
| 165 | 100.000 | 100.000 | 99.836 | 98.274 | 94.931 |
| 166 | 100.000 | 100.000 | 99.836 | 98.300 | 95.003 |
| 167 | 100.000 | 100.000 | 99.859 | 98.359 | 95.101 |
| 168 | 100.000 | 100.000 | 99.883 | 98.418 | 95.199 |
| 169 | 100.000 | 100.000 | 99.906 | 98.477 | 95.297 |
| 170 | 100.000 | 100.000 | 99.906 | 98.504 | 95.370 |
| 171 | 100.000 | 100.000 | 99.906 | 98.526 | 95.432 |
| 172 | 100.000 | 100.000 | 99.906 | 98.547 | 95.495 |
| 173 | 100.000 | 100.000 | 99.906 | 98.568 | 95.558 |
| 174 | 100.000 | 100.000 | 99.906 | 98.590 | 95.621 |
| 175 | 100.000 | 100.000 | 99.906 | 98.611 | 95.684 |
| 176 | 100.000 | 100.000 | 99.906 | 98.633 | 95.745 |
| 177 | 100.000 | 100.000 | 99.906 | 98.654 | 95.806 |
| 178 | 100.000 | 100.000 | 99.906 | 98.676 | 95.865 |
| 179 | 100.000 | 100.000 | 99.906 | 98.697 | 95.924 |
| 180 | 100.000 | 100.000 | 99.906 | 98.719 | 95.984 |
| 181 | 100.000 | 100.000 | 99.906 | 98.740 | 96.043 |
| 182 | 100.000 | 100.000 | 99.906 | 98.761 | 96.102 |
| 183 | 100.000 | 100.000 | 99.906 | 98.783 | 96.161 |
| 184 | 100.000 | 100.000 | 99.906 | 98.804 | 96.220 |
| 185 | 100.000 | 100.000 | 99.906 | 98.826 | 96.279 |
| 186 | 100.000 | 100.000 | 99.930 | 98.879 | 96.364 |
| 187 | 100.000 | 100.000 | 99.930 | 98.901 | 96.424 |
| 188 | 100.000 | 100.000 | 99.930 | 98.922 | 96.483 |
| 189 | 100.000 | 100.000 | 99.930 | 98.944 | 96.542 |
| 190 | 100.000 | 100.000 | 99.930 | 98.965 | 96.601 |
| 191 | 100.000 | 100.000 | 99.930 | 98.987 | 96.660 |
| 192 | 100.000 | 100.000 | 99.930 | 99.008 | 96.719 |
| 193 | 100.000 | 100.000 | 99.953 | 99.062 | 96.805 |
| 194 | 100.000 | 100.000 | 99.977 | 99.115 | 96.888 |
| 195 | 100.000 | 100.000 | 100.000 | 99.169 | 96.971 |
| 196 | 100.000 | 100.000 | 100.000 | 99.185 | 97.019 |
| 197 | 100.000 | 100.000 | 100.000 | 99.201 | 97.067 |
| 198 | 100.000 | 100.000 | 100.000 | 99.217 | 97.115 |
| 199 | 100.000 | 100.000 | 100.000 | 99.233 | 97.163 |
| 200 | 100.000 | 100.000 | 100.000 | 99.249 | 97.211 |
| 201 | 100.000 | 100.000 | 100.000 | 99.265 | 97.259 |
| 202 | 100.000 | 100.000 | 100.000 | 99.282 | 97.306 |
| 203 | 100.000 | 100.000 | 100.000 | 99.298 | 97.352 |
| 204 | 100.000 | 100.000 | 100.000 | 99.314 | 97.398 |
| 205 | 100.000 | 100.000 | 100.000 | 99.330 | 97.444 |
| 206 | 100.000 | 100.000 | 100.000 | 99.346 | 97.491 |
| 207 | 100.000 | 100.000 | 100.000 | 99.362 | 97.537 |
| 208 | 100.000 | 100.000 | 100.000 | 99.373 | 97.572 |
| 209 | 100.000 | 100.000 | 100.000 | 99.383 | 97.607 |
| 210 | 100.000 | 100.000 | 100.000 | 99.394 | 97.642 |
| 211 | 100.000 | 100.000 | 100.000 | 99.405 | 97.676 |
| 212 | 100.000 | 100.000 | 100.000 | 99.416 | 97.709 |
| 213 | 100.000 | 100.000 | 100.000 | 99.426 | 97.742 |
| 214 | 100.000 | 100.000 | 100.000 | 99.437 | 97.775 |
| 215 | 100.000 | 100.000 | 100.000 | 99.448 | 97.809 |
| 216 | 100.000 | 100.000 | 100.000 | 99.458 | 97.842 |
| 217 | 100.000 | 100.000 | 100.000 | 99.469 | 97.875 |
| 218 | 100.000 | 100.000 | 100.000 | 99.480 | 97.909 |
| 219 | 100.000 | 100.000 | 100.000 | 99.491 | 97.942 |
| 220 | 100.000 | 100.000 | 100.000 | 99.501 | 97.975 |
| 221 | 100.000 | 100.000 | 100.000 | 99.512 | 98.007 |
| 222 | 100.000 | 100.000 | 100.000 | 99.523 | 98.038 |
| 223 | 100.000 | 100.000 | 100.000 | 99.534 | 98.069 |
| 224 | 100.000 | 100.000 | 100.000 | 99.544 | 98.101 |
| 225 | 100.000 | 100.000 | 100.000 | 99.555 | 98.132 |
| 226 | 100.000 | 100.000 | 100.000 | 99.566 | 98.164 |
| 227 | 100.000 | 100.000 | 100.000 | 99.576 | 98.195 |
| 228 | 100.000 | 100.000 | 100.000 | 99.587 | 98.227 |
| 229 | 100.000 | 100.000 | 100.000 | 99.598 | 98.258 |
| 230 | 100.000 | 100.000 | 100.000 | 99.609 | 98.289 |
| 231 | 100.000 | 100.000 | 100.000 | 99.619 | 98.321 |
| 232 | 100.000 | 100.000 | 100.000 | 99.630 | 98.352 |
| 233 | 100.000 | 100.000 | 100.000 | 99.641 | 98.384 |
| 234 | 100.000 | 100.000 | 100.000 | 99.651 | 98.415 |
| 235 | 100.000 | 100.000 | 100.000 | 99.662 | 98.447 |
| 236 | 100.000 | 100.000 | 100.000 | 99.673 | 98.478 |
| 237 | 100.000 | 100.000 | 100.000 | 99.684 | 98.510 |
| 238 | 100.000 | 100.000 | 100.000 | 99.684 | 98.524 |
| 239 | 100.000 | 100.000 | 100.000 | 99.689 | 98.547 |
| 240 | 100.000 | 100.000 | 100.000 | 99.694 | 98.567 |
| 241 | 100.000 | 100.000 | 100.000 | 99.700 | 98.587 |
| 242 | 100.000 | 100.000 | 100.000 | 99.705 | 98.608 |
| 243 | 100.000 | 100.000 | 100.000 | 99.710 | 98.628 |
| 244 | 100.000 | 100.000 | 100.000 | 99.716 | 98.648 |
| 245 | 100.000 | 100.000 | 100.000 | 99.716 | 98.659 |
| 246 | 100.000 | 100.000 | 100.000 | 99.721 | 98.678 |
| 247 | 100.000 | 100.000 | 100.000 | 99.727 | 98.696 |
| 248 | 100.000 | 100.000 | 100.000 | 99.732 | 98.715 |
| 249 | 100.000 | 100.000 | 100.000 | 99.737 | 98.733 |
| 250 | 100.000 | 100.000 | 100.000 | 99.743 | 98.750 |
| 251 | 100.000 | 100.000 | 100.000 | 99.748 | 98.767 |
| 252 | 100.000 | 100.000 | 100.000 | 99.753 | 98.783 |
| 253 | 100.000 | 100.000 | 100.000 | 99.759 | 98.800 |
| 254 | 100.000 | 100.000 | 100.000 | 99.764 | 98.817 |
| 255 | 100.000 | 100.000 | 100.000 | 99.769 | 98.833 |
| 256 | 100.000 | 100.000 | 100.000 | 99.775 | 98.850 |
| 257 | 100.000 | 100.000 | 100.000 | 99.780 | 98.866 |
| 258 | 100.000 | 100.000 | 100.000 | 99.786 | 98.883 |
| 259 | 100.000 | 100.000 | 100.000 | 99.791 | 98.900 |
| 260 | 100.000 | 100.000 | 100.000 | 99.796 | 98.916 |
| 261 | 100.000 | 100.000 | 100.000 | 99.802 | 98.933 |
| 262 | 100.000 | 100.000 | 100.000 | 99.807 | 98.950 |
| 263 | 100.000 | 100.000 | 100.000 | 99.812 | 98.966 |
| 264 | 100.000 | 100.000 | 100.000 | 99.818 | 98.983 |
| 265 | 100.000 | 100.000 | 100.000 | 99.823 | 99.000 |
| 266 | 100.000 | 100.000 | 100.000 | 99.828 | 99.016 |
| 267 | 100.000 | 100.000 | 100.000 | 99.834 | 99.033 |
| 268 | 100.000 | 100.000 | 100.000 | 99.839 | 99.050 |
| 269 | 100.000 | 100.000 | 100.000 | 99.845 | 99.066 |
| 270 | 100.000 | 100.000 | 100.000 | 99.850 | 99.083 |
| 271 | 100.000 | 100.000 | 100.000 | 99.855 | 99.099 |
| 272 | 100.000 | 100.000 | 100.000 | 99.861 | 99.116 |
| 273 | 100.000 | 100.000 | 100.000 | 99.866 | 99.133 |
| 274 | 100.000 | 100.000 | 100.000 | 99.871 | 99.149 |
| 275 | 100.000 | 100.000 | 100.000 | 99.877 | 99.166 |
| 276 | 100.000 | 100.000 | 100.000 | 99.882 | 99.183 |
| 277 | 100.000 | 100.000 | 100.000 | 99.887 | 99.199 |
| 278 | 100.000 | 100.000 | 100.000 | 99.893 | 99.216 |
| 279 | 100.000 | 100.000 | 100.000 | 99.898 | 99.233 |
| 280 | 100.000 | 100.000 | 100.000 | 99.903 | 99.249 |
| 281 | 100.000 | 100.000 | 100.000 | 99.909 | 99.266 |
| 282 | 100.000 | 100.000 | 100.000 | 99.914 | 99.283 |
| 283 | 100.000 | 100.000 | 100.000 | 99.920 | 99.299 |
| 284 | 100.000 | 100.000 | 100.000 | 99.925 | 99.316 |
| 285 | 100.000 | 100.000 | 100.000 | 99.930 | 99.332 |
| 286 | 100.000 | 100.000 | 100.000 | 99.936 | 99.349 |
| 287 | 100.000 | 100.000 | 100.000 | 99.941 | 99.366 |
| 288 | 100.000 | 100.000 | 100.000 | 99.946 | 99.382 |
| 289 | 100.000 | 100.000 | 100.000 | 99.952 | 99.399 |
| 290 | 100.000 | 100.000 | 100.000 | 99.957 | 99.416 |
| 291 | 100.000 | 100.000 | 100.000 | 99.962 | 99.432 |
| 292 | 100.000 | 100.000 | 100.000 | 99.968 | 99.449 |
| 293 | 100.000 | 100.000 | 100.000 | 99.973 | 99.466 |
| 294 | 100.000 | 100.000 | 100.000 | 99.979 | 99.482 |
| 295 | 100.000 | 100.000 | 100.000 | 99.984 | 99.499 |
| 296 | 100.000 | 100.000 | 100.000 | 99.989 | 99.516 |
| 297 | 100.000 | 100.000 | 100.000 | 99.995 | 99.532 |
| 298 | 100.000 | 100.000 | 100.000 | 100.000 | 99.549 |
| 299 | 100.000 | 100.000 | 100.000 | 100.000 | 99.556 |
| 300 | 100.000 | 100.000 | 100.000 | 100.000 | 99.564 |
| 301 | 100.000 | 100.000 | 100.000 | 100.000 | 99.571 |
| 302 | 100.000 | 100.000 | 100.000 | 100.000 | 99.578 |
| 303 | 100.000 | 100.000 | 100.000 | 100.000 | 99.586 |
| 304 | 100.000 | 100.000 | 100.000 | 100.000 | 99.593 |
| 305 | 100.000 | 100.000 | 100.000 | 100.000 | 99.601 |
| 306 | 100.000 | 100.000 | 100.000 | 100.000 | 99.608 |
| 307 | 100.000 | 100.000 | 100.000 | 100.000 | 99.615 |
| 308 | 100.000 | 100.000 | 100.000 | 100.000 | 99.623 |
| 309 | 100.000 | 100.000 | 100.000 | 100.000 | 99.630 |
| 310 | 100.000 | 100.000 | 100.000 | 100.000 | 99.638 |
| 311 | 100.000 | 100.000 | 100.000 | 100.000 | 99.643 |
| 312 | 100.000 | 100.000 | 100.000 | 100.000 | 99.647 |
| 313 | 100.000 | 100.000 | 100.000 | 100.000 | 99.650 |
| 314 | 100.000 | 100.000 | 100.000 | 100.000 | 99.654 |
| 315 | 100.000 | 100.000 | 100.000 | 100.000 | 99.658 |
| 316 | 100.000 | 100.000 | 100.000 | 100.000 | 99.662 |
| 317 | 100.000 | 100.000 | 100.000 | 100.000 | 99.665 |
| 318 | 100.000 | 100.000 | 100.000 | 100.000 | 99.669 |
| 319 | 100.000 | 100.000 | 100.000 | 100.000 | 99.673 |
| 320 | 100.000 | 100.000 | 100.000 | 100.000 | 99.676 |
| 321 | 100.000 | 100.000 | 100.000 | 100.000 | 99.680 |
| 322 | 100.000 | 100.000 | 100.000 | 100.000 | 99.684 |
| 323 | 100.000 | 100.000 | 100.000 | 100.000 | 99.687 |
| 324 | 100.000 | 100.000 | 100.000 | 100.000 | 99.691 |
| 325 | 100.000 | 100.000 | 100.000 | 100.000 | 99.695 |
| 326 | 100.000 | 100.000 | 100.000 | 100.000 | 99.699 |
| 327 | 100.000 | 100.000 | 100.000 | 100.000 | 99.702 |
| 328 | 100.000 | 100.000 | 100.000 | 100.000 | 99.706 |
| 329 | 100.000 | 100.000 | 100.000 | 100.000 | 99.710 |
| 330 | 100.000 | 100.000 | 100.000 | 100.000 | 99.713 |
| 331 | 100.000 | 100.000 | 100.000 | 100.000 | 99.717 |
| 332 | 100.000 | 100.000 | 100.000 | 100.000 | 99.721 |
| 333 | 100.000 | 100.000 | 100.000 | 100.000 | 99.724 |
| 334 | 100.000 | 100.000 | 100.000 | 100.000 | 99.728 |
| 335 | 100.000 | 100.000 | 100.000 | 100.000 | 99.732 |
| 336 | 100.000 | 100.000 | 100.000 | 100.000 | 99.736 |
| 337 | 100.000 | 100.000 | 100.000 | 100.000 | 99.739 |
| 338 | 100.000 | 100.000 | 100.000 | 100.000 | 99.743 |
| 339 | 100.000 | 100.000 | 100.000 | 100.000 | 99.747 |
| 340 | 100.000 | 100.000 | 100.000 | 100.000 | 99.750 |
| 341 | 100.000 | 100.000 | 100.000 | 100.000 | 99.754 |
| 342 | 100.000 | 100.000 | 100.000 | 100.000 | 99.758 |
| 343 | 100.000 | 100.000 | 100.000 | 100.000 | 99.761 |
| 344 | 100.000 | 100.000 | 100.000 | 100.000 | 99.765 |
| 345 | 100.000 | 100.000 | 100.000 | 100.000 | 99.769 |
| 346 | 100.000 | 100.000 | 100.000 | 100.000 | 99.773 |
| 347 | 100.000 | 100.000 | 100.000 | 100.000 | 99.776 |
| 348 | 100.000 | 100.000 | 100.000 | 100.000 | 99.780 |
| 349 | 100.000 | 100.000 | 100.000 | 100.000 | 99.784 |
| 350 | 100.000 | 100.000 | 100.000 | 100.000 | 99.787 |
| 351 | 100.000 | 100.000 | 100.000 | 100.000 | 99.791 |
| 352 | 100.000 | 100.000 | 100.000 | 100.000 | 99.795 |
| 353 | 100.000 | 100.000 | 100.000 | 100.000 | 99.798 |
| 354 | 100.000 | 100.000 | 100.000 | 100.000 | 99.802 |
| 355 | 100.000 | 100.000 | 100.000 | 100.000 | 99.806 |
| 356 | 100.000 | 100.000 | 100.000 | 100.000 | 99.810 |
| 357 | 100.000 | 100.000 | 100.000 | 100.000 | 99.813 |
| 358 | 100.000 | 100.000 | 100.000 | 100.000 | 99.817 |
| 359 | 100.000 | 100.000 | 100.000 | 100.000 | 99.821 |
| 360 | 100.000 | 100.000 | 100.000 | 100.000 | 99.824 |
| 361 | 100.000 | 100.000 | 100.000 | 100.000 | 99.828 |
| 362 | 100.000 | 100.000 | 100.000 | 100.000 | 99.832 |
| 363 | 100.000 | 100.000 | 100.000 | 100.000 | 99.835 |
| 364 | 100.000 | 100.000 | 100.000 | 100.000 | 99.839 |
| 365 | 100.000 | 100.000 | 100.000 | 100.000 | 99.843 |
| 366 | 100.000 | 100.000 | 100.000 | 100.000 | 99.847 |
| 367 | 100.000 | 100.000 | 100.000 | 100.000 | 99.850 |
| 368 | 100.000 | 100.000 | 100.000 | 100.000 | 99.854 |
| 369 | 100.000 | 100.000 | 100.000 | 100.000 | 99.858 |
| 370 | 100.000 | 100.000 | 100.000 | 100.000 | 99.861 |
| 371 | 100.000 | 100.000 | 100.000 | 100.000 | 99.865 |
| 372 | 100.000 | 100.000 | 100.000 | 100.000 | 99.869 |
| 373 | 100.000 | 100.000 | 100.000 | 100.000 | 99.872 |
| 374 | 100.000 | 100.000 | 100.000 | 100.000 | 99.876 |
| 375 | 100.000 | 100.000 | 100.000 | 100.000 | 99.880 |
| 376 | 100.000 | 100.000 | 100.000 | 100.000 | 99.883 |
| 377 | 100.000 | 100.000 | 100.000 | 100.000 | 99.887 |
| 378 | 100.000 | 100.000 | 100.000 | 100.000 | 99.891 |
| 379 | 100.000 | 100.000 | 100.000 | 100.000 | 99.895 |
| 380 | 100.000 | 100.000 | 100.000 | 100.000 | 99.896 |
| 381 | 100.000 | 100.000 | 100.000 | 100.000 | 99.898 |
| 382 | 100.000 | 100.000 | 100.000 | 100.000 | 99.900 |
| 383 | 100.000 | 100.000 | 100.000 | 100.000 | 99.902 |
| 384 | 100.000 | 100.000 | 100.000 | 100.000 | 99.904 |
| 385 | 100.000 | 100.000 | 100.000 | 100.000 | 99.906 |
| 386 | 100.000 | 100.000 | 100.000 | 100.000 | 99.908 |
| 387 | 100.000 | 100.000 | 100.000 | 100.000 | 99.909 |
| 388 | 100.000 | 100.000 | 100.000 | 100.000 | 99.911 |
| 389 | 100.000 | 100.000 | 100.000 | 100.000 | 99.913 |
| 390 | 100.000 | 100.000 | 100.000 | 100.000 | 99.915 |
| 391 | 100.000 | 100.000 | 100.000 | 100.000 | 99.917 |
| 392 | 100.000 | 100.000 | 100.000 | 100.000 | 99.919 |
| 393 | 100.000 | 100.000 | 100.000 | 100.000 | 99.920 |
| 394 | 100.000 | 100.000 | 100.000 | 100.000 | 99.922 |
| 395 | 100.000 | 100.000 | 100.000 | 100.000 | 99.924 |
| 396 | 100.000 | 100.000 | 100.000 | 100.000 | 99.926 |
| 397 | 100.000 | 100.000 | 100.000 | 100.000 | 99.928 |
| 398 | 100.000 | 100.000 | 100.000 | 100.000 | 99.930 |
| 399 | 100.000 | 100.000 | 100.000 | 100.000 | 99.932 |
| 400 | 100.000 | 100.000 | 100.000 | 100.000 | 99.933 |
| 401 | 100.000 | 100.000 | 100.000 | 100.000 | 99.935 |
| 402 | 100.000 | 100.000 | 100.000 | 100.000 | 99.937 |
| 403 | 100.000 | 100.000 | 100.000 | 100.000 | 99.939 |
| 404 | 100.000 | 100.000 | 100.000 | 100.000 | 99.941 |
| 405 | 100.000 | 100.000 | 100.000 | 100.000 | 99.943 |
| 406 | 100.000 | 100.000 | 100.000 | 100.000 | 99.945 |
| 407 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
| 408 | 100.000 | 100.000 | 100.000 | 100.000 | 99.948 |
| 409 | 100.000 | 100.000 | 100.000 | 100.000 | 99.950 |
| 410 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
| 411 | 100.000 | 100.000 | 100.000 | 100.000 | 99.954 |
| 412 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
| 413 | 100.000 | 100.000 | 100.000 | 100.000 | 99.957 |
| 414 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
| 415 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
| 416 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
| 417 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
| 418 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 419 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
| 420 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
| 421 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
| 422 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 423 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
| 424 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
| 425 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
| 426 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
| 427 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 428 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
| 429 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
| 430 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 431 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 432 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 433 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 434 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 435 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 436 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
gzip
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
| 7 | 94.737 | 86.022 | 76.557 | 67.361 | 58.835 |
| 8 | 97.368 | 89.555 | 80.347 | 71.159 | 62.542 |
| 9 | 100.000 | 93.088 | 84.092 | 74.853 | 66.091 |
| 10 | 100.000 | 93.856 | 85.664 | 77.056 | 68.721 |
| 11 | 100.000 | 94.624 | 87.147 | 79.050 | 71.033 |
| 12 | 100.000 | 95.392 | 88.616 | 81.004 | 73.276 |
| 13 | 100.000 | 96.160 | 90.100 | 82.990 | 75.565 |
| 14 | 100.000 | 96.928 | 91.569 | 84.927 | 77.757 |
| 15 | 100.000 | 97.696 | 93.038 | 86.869 | 79.963 |
| 16 | 100.000 | 98.464 | 94.506 | 88.806 | 82.154 |
| 17 | 100.000 | 99.232 | 96.034 | 90.906 | 84.627 |
| 18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |
gzip
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.347 | 62.970 | 52.942 | 44.301 |
| 5 | 89.474 | 78.648 | 68.243 | 58.664 | 50.095 |
| 6 | 92.105 | 82.488 | 72.738 | 63.484 | 54.998 |
| 7 | 94.737 | 86.022 | 76.601 | 67.465 | 58.989 |
| 8 | 97.368 | 89.555 | 80.376 | 71.230 | 62.647 |
| 9 | 100.000 | 93.088 | 84.122 | 74.918 | 66.180 |
| 10 | 100.000 | 93.856 | 85.679 | 77.095 | 68.783 |
| 11 | 100.000 | 94.624 | 87.206 | 79.194 | 71.258 |
| 12 | 100.000 | 95.392 | 88.704 | 81.219 | 73.611 |
| 13 | 100.000 | 96.160 | 90.203 | 83.249 | 75.983 |
| 14 | 100.000 | 96.928 | 91.686 | 85.236 | 78.272 |
| 15 | 100.000 | 97.696 | 93.155 | 87.177 | 80.477 |
| 16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
| 17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
| 18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |
gzip
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 78.947 | 61.290 | 47.474 | 36.905 | 28.873 |
| 3 | 84.211 | 69.585 | 56.962 | 46.428 | 37.793 |
| 4 | 86.842 | 75.115 | 64.233 | 54.358 | 45.643 |
| 5 | 89.474 | 78.802 | 68.478 | 58.904 | 50.294 |
| 6 | 92.105 | 82.488 | 72.723 | 63.448 | 54.940 |
| 7 | 94.737 | 86.022 | 76.572 | 67.396 | 58.886 |
| 8 | 97.368 | 89.555 | 80.405 | 71.303 | 62.763 |
| 9 | 100.000 | 93.088 | 84.151 | 74.989 | 66.290 |
| 10 | 100.000 | 93.856 | 85.679 | 77.093 | 68.778 |
| 11 | 100.000 | 94.624 | 87.206 | 79.194 | 71.260 |
| 12 | 100.000 | 95.392 | 88.719 | 81.261 | 73.688 |
| 13 | 100.000 | 96.160 | 90.217 | 83.294 | 76.066 |
| 14 | 100.000 | 96.928 | 91.686 | 85.240 | 78.285 |
| 15 | 100.000 | 97.696 | 93.155 | 87.181 | 80.490 |
| 16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
| 17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
| 18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |
gzip
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 89.474 | 79.724 | 70.535 | 61.935 | 54.002 |
| 7 | 92.105 | 83.257 | 74.354 | 65.811 | 57.839 |
| 8 | 94.737 | 86.790 | 78.173 | 69.682 | 61.660 |
| 9 | 97.368 | 90.323 | 81.933 | 73.411 | 65.262 |
| 10 | 100.000 | 93.856 | 85.693 | 77.135 | 68.854 |
| 11 | 100.000 | 94.624 | 87.221 | 79.237 | 71.335 |
| 12 | 100.000 | 95.392 | 88.719 | 81.272 | 73.719 |
| 13 | 100.000 | 96.160 | 90.217 | 83.294 | 76.066 |
| 14 | 100.000 | 96.928 | 91.686 | 85.240 | 78.285 |
| 15 | 100.000 | 97.696 | 93.155 | 87.181 | 80.490 |
| 16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.696 |
| 17 | 100.000 | 99.232 | 96.093 | 91.060 | 84.886 |
| 18 | 100.000 | 100.000 | 97.562 | 92.997 | 87.077 |
gzip
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.347 | 62.970 | 52.942 | 44.301 |
| 5 | 89.474 | 78.648 | 68.199 | 58.561 | 49.940 |
| 6 | 92.105 | 82.335 | 72.429 | 63.075 | 54.547 |
| 7 | 94.737 | 86.022 | 76.601 | 67.463 | 58.983 |
| 8 | 97.368 | 89.555 | 80.479 | 71.477 | 63.025 |
| 9 | 100.000 | 93.088 | 84.269 | 75.277 | 66.739 |
| 10 | 100.000 | 93.702 | 85.517 | 77.042 | 68.885 |
| 11 | 100.000 | 94.316 | 86.722 | 78.683 | 70.814 |
| 12 | 100.000 | 94.931 | 87.985 | 80.480 | 73.005 |
| 13 | 100.000 | 95.545 | 89.189 | 82.120 | 74.929 |
| 14 | 100.000 | 96.006 | 90.115 | 83.420 | 76.510 |
| 15 | 100.000 | 96.467 | 91.040 | 84.700 | 78.027 |
| 16 | 100.000 | 96.928 | 91.965 | 85.980 | 79.541 |
| 17 | 100.000 | 97.389 | 92.891 | 87.260 | 81.056 |
| 18 | 100.000 | 97.849 | 93.816 | 88.544 | 82.587 |
| 19 | 100.000 | 98.157 | 94.477 | 89.530 | 83.849 |
| 20 | 100.000 | 98.464 | 95.138 | 90.500 | 85.057 |
| 21 | 100.000 | 98.771 | 95.799 | 91.470 | 86.268 |
| 22 | 100.000 | 99.078 | 96.460 | 92.443 | 87.489 |
| 23 | 100.000 | 99.386 | 97.121 | 93.419 | 88.717 |
| 24 | 100.000 | 99.539 | 97.488 | 94.005 | 89.501 |
| 25 | 100.000 | 99.693 | 97.870 | 94.628 | 90.342 |
| 26 | 100.000 | 99.846 | 98.237 | 95.212 | 91.119 |
| 27 | 100.000 | 100.000 | 98.605 | 95.797 | 91.896 |
| 28 | 100.000 | 100.000 | 98.707 | 96.073 | 92.373 |
| 29 | 100.000 | 100.000 | 98.796 | 96.310 | 92.782 |
| 30 | 100.000 | 100.000 | 98.884 | 96.555 | 93.220 |
| 31 | 100.000 | 100.000 | 98.972 | 96.793 | 93.630 |
| 32 | 100.000 | 100.000 | 99.060 | 97.034 | 94.054 |
| 33 | 100.000 | 100.000 | 99.163 | 97.318 | 94.556 |
| 34 | 100.000 | 100.000 | 99.222 | 97.489 | 94.871 |
| 35 | 100.000 | 100.000 | 99.324 | 97.764 | 95.346 |
| 36 | 100.000 | 100.000 | 99.412 | 98.002 | 95.757 |
| 37 | 100.000 | 100.000 | 99.471 | 98.164 | 96.046 |
| 38 | 100.000 | 100.000 | 99.559 | 98.401 | 96.456 |
| 39 | 100.000 | 100.000 | 99.618 | 98.564 | 96.743 |
| 40 | 100.000 | 100.000 | 99.706 | 98.801 | 97.153 |
| 41 | 100.000 | 100.000 | 99.765 | 98.963 | 97.441 |
| 42 | 100.000 | 100.000 | 99.824 | 99.126 | 97.728 |
| 43 | 100.000 | 100.000 | 99.882 | 99.288 | 98.015 |
| 44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
| 45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |
gzip
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
| 7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
| 8 | 97.368 | 89.555 | 80.552 | 71.650 | 63.284 |
| 9 | 100.000 | 93.088 | 84.342 | 75.451 | 67.002 |
| 10 | 100.000 | 93.702 | 85.590 | 77.218 | 69.154 |
| 11 | 100.000 | 94.316 | 86.810 | 78.916 | 71.203 |
| 12 | 100.000 | 94.931 | 88.014 | 80.568 | 73.164 |
| 13 | 100.000 | 95.545 | 89.219 | 82.207 | 75.088 |
| 14 | 100.000 | 96.160 | 90.423 | 83.852 | 77.030 |
| 15 | 100.000 | 96.774 | 91.627 | 85.483 | 78.929 |
| 16 | 100.000 | 97.389 | 92.832 | 87.122 | 80.852 |
| 17 | 100.000 | 98.003 | 94.036 | 88.749 | 82.737 |
| 18 | 100.000 | 98.464 | 94.962 | 90.037 | 84.281 |
| 19 | 100.000 | 98.771 | 95.623 | 91.007 | 85.489 |
| 20 | 100.000 | 98.925 | 96.005 | 91.630 | 86.331 |
| 21 | 100.000 | 99.078 | 96.387 | 92.267 | 87.219 |
| 22 | 100.000 | 99.232 | 96.769 | 92.900 | 88.095 |
| 23 | 100.000 | 99.386 | 97.150 | 93.523 | 88.938 |
| 24 | 100.000 | 99.539 | 97.532 | 94.147 | 89.786 |
| 25 | 100.000 | 99.693 | 97.914 | 94.770 | 90.627 |
| 26 | 100.000 | 99.846 | 98.296 | 95.393 | 91.468 |
| 27 | 100.000 | 100.000 | 98.678 | 96.016 | 92.310 |
| 28 | 100.000 | 100.000 | 98.766 | 96.257 | 92.736 |
| 29 | 100.000 | 100.000 | 98.854 | 96.495 | 93.146 |
| 30 | 100.000 | 100.000 | 98.942 | 96.732 | 93.556 |
| 31 | 100.000 | 100.000 | 99.031 | 96.969 | 93.965 |
| 32 | 100.000 | 100.000 | 99.119 | 97.207 | 94.377 |
| 33 | 100.000 | 100.000 | 99.207 | 97.444 | 94.787 |
| 34 | 100.000 | 100.000 | 99.295 | 97.681 | 95.197 |
| 35 | 100.000 | 100.000 | 99.383 | 97.919 | 95.607 |
| 36 | 100.000 | 100.000 | 99.471 | 98.156 | 96.018 |
| 37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.305 |
| 38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.593 |
| 39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.880 |
| 40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.168 |
| 41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.455 |
| 42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
| 43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
| 44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
| 45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |
gzip
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
| 7 | 94.737 | 86.022 | 76.675 | 67.637 | 59.246 |
| 8 | 97.368 | 89.555 | 80.567 | 71.684 | 63.334 |
| 9 | 100.000 | 93.088 | 84.327 | 75.417 | 66.951 |
| 10 | 100.000 | 93.702 | 85.576 | 77.180 | 69.091 |
| 11 | 100.000 | 94.316 | 86.795 | 78.878 | 71.139 |
| 12 | 100.000 | 94.931 | 87.999 | 80.531 | 73.107 |
| 13 | 100.000 | 95.545 | 89.204 | 82.172 | 75.036 |
| 14 | 100.000 | 96.160 | 90.408 | 83.813 | 76.966 |
| 15 | 100.000 | 96.774 | 91.613 | 85.451 | 78.884 |
| 16 | 100.000 | 97.389 | 92.817 | 87.082 | 80.782 |
| 17 | 100.000 | 98.003 | 94.022 | 88.709 | 82.665 |
| 18 | 100.000 | 98.618 | 95.226 | 90.336 | 84.548 |
| 19 | 100.000 | 98.771 | 95.623 | 91.007 | 85.488 |
| 20 | 100.000 | 98.925 | 96.005 | 91.634 | 86.344 |
| 21 | 100.000 | 99.078 | 96.387 | 92.259 | 87.192 |
| 22 | 100.000 | 99.232 | 96.769 | 92.884 | 88.041 |
| 23 | 100.000 | 99.386 | 97.150 | 93.506 | 88.883 |
| 24 | 100.000 | 99.539 | 97.532 | 94.129 | 89.724 |
| 25 | 100.000 | 99.693 | 97.914 | 94.752 | 90.565 |
| 26 | 100.000 | 99.846 | 98.296 | 95.375 | 91.408 |
| 27 | 100.000 | 100.000 | 98.678 | 95.997 | 92.248 |
| 28 | 100.000 | 100.000 | 98.766 | 96.245 | 92.692 |
| 29 | 100.000 | 100.000 | 98.854 | 96.486 | 93.116 |
| 30 | 100.000 | 100.000 | 98.942 | 96.728 | 93.541 |
| 31 | 100.000 | 100.000 | 99.031 | 96.965 | 93.951 |
| 32 | 100.000 | 100.000 | 99.119 | 97.202 | 94.361 |
| 33 | 100.000 | 100.000 | 99.207 | 97.440 | 94.770 |
| 34 | 100.000 | 100.000 | 99.295 | 97.677 | 95.180 |
| 35 | 100.000 | 100.000 | 99.383 | 97.915 | 95.590 |
| 36 | 100.000 | 100.000 | 99.471 | 98.152 | 96.002 |
| 37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.304 |
| 38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.592 |
| 39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.879 |
| 40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.167 |
| 41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.454 |
| 42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
| 43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
| 44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
| 45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |
gzip
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
| 6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
| 7 | 94.737 | 85.868 | 76.293 | 67.053 | 58.532 |
| 8 | 97.368 | 89.401 | 80.185 | 71.096 | 62.610 |
| 9 | 100.000 | 93.088 | 84.224 | 75.173 | 66.585 |
| 10 | 100.000 | 93.702 | 85.532 | 77.076 | 68.937 |
| 11 | 100.000 | 94.316 | 86.780 | 78.837 | 71.071 |
| 12 | 100.000 | 94.931 | 87.985 | 80.493 | 73.042 |
| 13 | 100.000 | 95.545 | 89.189 | 82.136 | 74.978 |
| 14 | 100.000 | 96.160 | 90.394 | 83.777 | 76.909 |
| 15 | 100.000 | 96.774 | 91.598 | 85.410 | 78.812 |
| 16 | 100.000 | 97.389 | 92.803 | 87.037 | 80.696 |
| 17 | 100.000 | 98.003 | 94.007 | 88.664 | 82.580 |
| 18 | 100.000 | 98.464 | 94.932 | 89.950 | 84.117 |
| 19 | 100.000 | 98.771 | 95.593 | 90.920 | 85.327 |
| 20 | 100.000 | 98.925 | 95.990 | 91.591 | 86.267 |
| 21 | 100.000 | 99.078 | 96.387 | 92.263 | 87.206 |
| 22 | 100.000 | 99.232 | 96.769 | 92.890 | 88.062 |
| 23 | 100.000 | 99.386 | 97.150 | 93.514 | 88.911 |
| 24 | 100.000 | 99.539 | 97.532 | 94.137 | 89.753 |
| 25 | 100.000 | 99.693 | 97.914 | 94.760 | 90.594 |
| 26 | 100.000 | 99.846 | 98.296 | 95.383 | 91.434 |
| 27 | 100.000 | 100.000 | 98.678 | 96.006 | 92.274 |
| 28 | 100.000 | 100.000 | 98.766 | 96.253 | 92.719 |
| 29 | 100.000 | 100.000 | 98.854 | 96.495 | 93.144 |
| 30 | 100.000 | 100.000 | 98.942 | 96.732 | 93.556 |
| 31 | 100.000 | 100.000 | 99.031 | 96.969 | 93.968 |
| 32 | 100.000 | 100.000 | 99.119 | 97.207 | 94.378 |
| 33 | 100.000 | 100.000 | 99.207 | 97.444 | 94.787 |
| 34 | 100.000 | 100.000 | 99.295 | 97.681 | 95.197 |
| 35 | 100.000 | 100.000 | 99.383 | 97.919 | 95.607 |
| 36 | 100.000 | 100.000 | 99.471 | 98.156 | 96.017 |
| 37 | 100.000 | 100.000 | 99.530 | 98.318 | 96.305 |
| 38 | 100.000 | 100.000 | 99.589 | 98.481 | 96.593 |
| 39 | 100.000 | 100.000 | 99.647 | 98.643 | 96.880 |
| 40 | 100.000 | 100.000 | 99.706 | 98.805 | 97.168 |
| 41 | 100.000 | 100.000 | 99.765 | 98.967 | 97.455 |
| 42 | 100.000 | 100.000 | 99.824 | 99.130 | 97.743 |
| 43 | 100.000 | 100.000 | 99.882 | 99.292 | 98.030 |
| 44 | 100.000 | 100.000 | 99.941 | 99.454 | 98.318 |
| 45 | 100.000 | 100.000 | 100.000 | 99.617 | 98.605 |
gzip
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
| 6 | 92.105 | 82.181 | 72.150 | 62.732 | 54.193 |
| 7 | 94.737 | 86.022 | 76.542 | 67.329 | 58.789 |
| 8 | 97.368 | 89.555 | 80.288 | 71.013 | 62.310 |
| 9 | 100.000 | 93.088 | 84.122 | 74.908 | 66.153 |
| 10 | 100.000 | 93.856 | 85.693 | 77.119 | 68.805 |
| 11 | 100.000 | 94.624 | 87.192 | 79.152 | 71.182 |
| 12 | 100.000 | 95.392 | 88.690 | 81.182 | 73.554 |
| 13 | 100.000 | 96.160 | 90.159 | 83.120 | 75.744 |
| 14 | 100.000 | 96.928 | 91.657 | 85.142 | 78.092 |
| 15 | 100.000 | 97.696 | 93.126 | 87.088 | 80.310 |
| 16 | 100.000 | 98.464 | 94.624 | 89.122 | 82.693 |
| 17 | 100.000 | 99.232 | 96.093 | 91.068 | 84.910 |
| 18 | 100.000 | 100.000 | 97.562 | 93.005 | 87.101 |
| 19 | 100.000 | 100.000 | 97.679 | 93.330 | 87.673 |
| 20 | 100.000 | 100.000 | 97.767 | 93.581 | 88.132 |
| 21 | 100.000 | 100.000 | 97.855 | 93.819 | 88.545 |
| 22 | 100.000 | 100.000 | 97.944 | 94.060 | 88.969 |
| 23 | 100.000 | 100.000 | 98.061 | 94.383 | 89.536 |
| 24 | 100.000 | 100.000 | 98.135 | 94.588 | 89.903 |
| 25 | 100.000 | 100.000 | 98.223 | 94.825 | 90.316 |
| 26 | 100.000 | 100.000 | 98.281 | 94.987 | 90.606 |
| 27 | 100.000 | 100.000 | 98.370 | 95.225 | 91.015 |
| 28 | 100.000 | 100.000 | 98.443 | 95.434 | 91.397 |
| 29 | 100.000 | 100.000 | 98.502 | 95.596 | 91.684 |
| 30 | 100.000 | 100.000 | 98.575 | 95.797 | 92.038 |
| 31 | 100.000 | 100.000 | 98.663 | 96.034 | 92.449 |
| 32 | 100.000 | 100.000 | 98.751 | 96.271 | 92.858 |
| 33 | 100.000 | 100.000 | 98.840 | 96.509 | 93.269 |
| 34 | 100.000 | 100.000 | 98.913 | 96.710 | 93.621 |
| 35 | 100.000 | 100.000 | 98.957 | 96.835 | 93.851 |
| 36 | 100.000 | 100.000 | 99.031 | 97.036 | 94.204 |
| 37 | 100.000 | 100.000 | 99.119 | 97.274 | 94.616 |
| 38 | 100.000 | 100.000 | 99.177 | 97.436 | 94.904 |
| 39 | 100.000 | 100.000 | 99.236 | 97.598 | 95.191 |
| 40 | 100.000 | 100.000 | 99.266 | 97.687 | 95.363 |
| 41 | 100.000 | 100.000 | 99.295 | 97.777 | 95.535 |
| 42 | 100.000 | 100.000 | 99.310 | 97.827 | 95.642 |
| 43 | 100.000 | 100.000 | 99.324 | 97.878 | 95.750 |
| 44 | 100.000 | 100.000 | 99.368 | 98.004 | 95.980 |
| 45 | 100.000 | 100.000 | 99.383 | 98.055 | 96.087 |
| 46 | 100.000 | 100.000 | 99.383 | 98.069 | 96.137 |
| 47 | 100.000 | 100.000 | 99.398 | 98.119 | 96.244 |
| 48 | 100.000 | 100.000 | 99.486 | 98.357 | 96.654 |
| 49 | 100.000 | 100.000 | 99.486 | 98.371 | 96.703 |
| 50 | 100.000 | 100.000 | 99.545 | 98.533 | 96.991 |
| 51 | 100.000 | 100.000 | 99.574 | 98.623 | 97.163 |
| 52 | 100.000 | 100.000 | 99.633 | 98.785 | 97.450 |
| 53 | 100.000 | 100.000 | 99.662 | 98.874 | 97.622 |
| 54 | 100.000 | 100.000 | 99.706 | 99.000 | 97.852 |
| 55 | 100.000 | 100.000 | 99.780 | 99.201 | 98.204 |
| 56 | 100.000 | 100.000 | 99.794 | 99.251 | 98.311 |
| 57 | 100.000 | 100.000 | 99.838 | 99.377 | 98.540 |
| 58 | 100.000 | 100.000 | 99.853 | 99.428 | 98.648 |
| 59 | 100.000 | 100.000 | 99.853 | 99.442 | 98.697 |
| 60 | 100.000 | 100.000 | 99.882 | 99.531 | 98.869 |
| 61 | 100.000 | 100.000 | 99.882 | 99.546 | 98.919 |
| 62 | 100.000 | 100.000 | 99.927 | 99.671 | 99.148 |
| 63 | 100.000 | 100.000 | 99.956 | 99.761 | 99.320 |
| 64 | 100.000 | 100.000 | 99.956 | 99.775 | 99.370 |
| 65 | 100.000 | 100.000 | 99.971 | 99.826 | 99.477 |
| 66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
| 67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
| 68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
| 69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
| 70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
| 71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
| 72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
gzip
Generation Strength: 4, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.335 | 72.459 | 63.146 | 54.656 |
| 7 | 94.737 | 86.022 | 76.572 | 67.396 | 58.886 |
| 8 | 97.368 | 89.555 | 80.449 | 71.408 | 62.923 |
| 9 | 100.000 | 93.088 | 84.254 | 75.242 | 66.687 |
| 10 | 100.000 | 93.856 | 85.781 | 77.352 | 69.194 |
| 11 | 100.000 | 94.624 | 87.294 | 79.423 | 71.636 |
| 12 | 100.000 | 95.392 | 88.793 | 81.464 | 74.038 |
| 13 | 100.000 | 96.160 | 90.276 | 83.460 | 76.359 |
| 14 | 100.000 | 96.928 | 91.745 | 85.422 | 78.628 |
| 15 | 100.000 | 97.696 | 93.214 | 87.360 | 80.822 |
| 16 | 100.000 | 98.464 | 94.683 | 89.297 | 83.015 |
| 17 | 100.000 | 99.232 | 96.152 | 91.234 | 85.209 |
| 18 | 100.000 | 100.000 | 97.620 | 93.172 | 87.402 |
| 19 | 100.000 | 100.000 | 97.709 | 93.409 | 87.817 |
| 20 | 100.000 | 100.000 | 97.797 | 93.665 | 88.286 |
| 21 | 100.000 | 100.000 | 97.885 | 93.902 | 88.699 |
| 22 | 100.000 | 100.000 | 97.973 | 94.139 | 89.112 |
| 23 | 100.000 | 100.000 | 98.061 | 94.381 | 89.536 |
| 24 | 100.000 | 100.000 | 98.149 | 94.618 | 89.946 |
| 25 | 100.000 | 100.000 | 98.237 | 94.855 | 90.357 |
| 26 | 100.000 | 100.000 | 98.325 | 95.097 | 90.781 |
| 27 | 100.000 | 100.000 | 98.414 | 95.334 | 91.191 |
| 28 | 100.000 | 100.000 | 98.502 | 95.571 | 91.601 |
| 29 | 100.000 | 100.000 | 98.590 | 95.809 | 92.012 |
| 30 | 100.000 | 100.000 | 98.678 | 96.046 | 92.422 |
| 31 | 100.000 | 100.000 | 98.766 | 96.288 | 92.846 |
| 32 | 100.000 | 100.000 | 98.854 | 96.525 | 93.255 |
| 33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.688 |
| 34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.098 |
| 35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.508 |
| 36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.917 |
| 37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.147 |
| 38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.377 |
| 39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.606 |
| 40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.836 |
| 41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.066 |
| 42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.295 |
| 43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.525 |
| 44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.756 |
| 45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.985 |
| 46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.216 |
| 47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
| 48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.676 |
| 49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.906 |
| 50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
| 51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.365 |
| 52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.595 |
| 53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
| 54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
| 55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
| 56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
| 57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
| 58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
| 59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
| 60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
| 61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
| 62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
| 63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
| 64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
| 65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
| 66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
| 67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
| 68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
| 69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
| 70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
| 71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
| 72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
gzip
Generation Strength: 4, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
| 6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
| 7 | 94.737 | 85.714 | 76.058 | 66.807 | 58.315 |
| 8 | 97.368 | 89.247 | 79.950 | 70.852 | 62.398 |
| 9 | 100.000 | 92.934 | 83.975 | 74.896 | 66.323 |
| 10 | 100.000 | 93.856 | 85.752 | 77.283 | 69.092 |
| 11 | 100.000 | 94.624 | 87.221 | 79.239 | 71.341 |
| 12 | 100.000 | 95.392 | 88.690 | 81.194 | 73.590 |
| 13 | 100.000 | 96.160 | 90.159 | 83.146 | 75.828 |
| 14 | 100.000 | 96.928 | 91.627 | 85.098 | 78.066 |
| 15 | 100.000 | 97.696 | 93.096 | 87.039 | 80.271 |
| 16 | 100.000 | 98.464 | 94.565 | 88.980 | 82.477 |
| 17 | 100.000 | 99.232 | 96.034 | 90.922 | 84.681 |
| 18 | 100.000 | 100.000 | 97.503 | 92.863 | 86.886 |
| 19 | 100.000 | 100.000 | 97.650 | 93.251 | 87.543 |
| 20 | 100.000 | 100.000 | 97.797 | 93.638 | 88.200 |
| 21 | 100.000 | 100.000 | 97.885 | 93.888 | 88.651 |
| 22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.102 |
| 23 | 100.000 | 100.000 | 98.061 | 94.385 | 89.547 |
| 24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.993 |
| 25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.404 |
| 26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
| 27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.228 |
| 28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.638 |
| 29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.049 |
| 30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.460 |
| 31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.870 |
| 32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.281 |
| 33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.692 |
| 34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
| 35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
| 36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
| 37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.151 |
| 38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.380 |
| 39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.610 |
| 40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.840 |
| 41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.069 |
| 42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.299 |
| 43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.529 |
| 44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.758 |
| 45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.988 |
| 46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.218 |
| 47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
| 48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.677 |
| 49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.907 |
| 50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
| 51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.366 |
| 52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.596 |
| 53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
| 54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
| 55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
| 56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
| 57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
| 58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
| 59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
| 60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
| 61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
| 62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
| 63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
| 64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
| 65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
| 66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
| 67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
| 68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
| 69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
| 70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
| 71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
| 72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
gzip
Generation Strength: 4, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.341 | 67.685 | 57.975 | 49.375 |
| 6 | 92.105 | 82.181 | 72.165 | 62.762 | 54.233 |
| 7 | 94.737 | 85.714 | 76.058 | 66.807 | 58.315 |
| 8 | 97.368 | 89.247 | 79.950 | 70.852 | 62.398 |
| 9 | 100.000 | 92.934 | 83.975 | 74.896 | 66.323 |
| 10 | 100.000 | 93.856 | 85.752 | 77.283 | 69.092 |
| 11 | 100.000 | 94.624 | 87.221 | 79.239 | 71.341 |
| 12 | 100.000 | 95.392 | 88.690 | 81.194 | 73.590 |
| 13 | 100.000 | 96.160 | 90.159 | 83.146 | 75.828 |
| 14 | 100.000 | 96.928 | 91.627 | 85.098 | 78.066 |
| 15 | 100.000 | 97.696 | 93.096 | 87.039 | 80.271 |
| 16 | 100.000 | 98.464 | 94.565 | 88.980 | 82.477 |
| 17 | 100.000 | 99.232 | 96.034 | 90.922 | 84.681 |
| 18 | 100.000 | 100.000 | 97.503 | 92.863 | 86.886 |
| 19 | 100.000 | 100.000 | 97.650 | 93.251 | 87.543 |
| 20 | 100.000 | 100.000 | 97.797 | 93.638 | 88.200 |
| 21 | 100.000 | 100.000 | 97.885 | 93.888 | 88.651 |
| 22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.102 |
| 23 | 100.000 | 100.000 | 98.061 | 94.385 | 89.547 |
| 24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.993 |
| 25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.404 |
| 26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
| 27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.228 |
| 28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.639 |
| 29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.050 |
| 30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.461 |
| 31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.871 |
| 32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.282 |
| 33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.692 |
| 34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
| 35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
| 36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
| 37 | 100.000 | 100.000 | 99.251 | 97.606 | 95.151 |
| 38 | 100.000 | 100.000 | 99.295 | 97.732 | 95.380 |
| 39 | 100.000 | 100.000 | 99.339 | 97.858 | 95.610 |
| 40 | 100.000 | 100.000 | 99.383 | 97.984 | 95.840 |
| 41 | 100.000 | 100.000 | 99.427 | 98.109 | 96.069 |
| 42 | 100.000 | 100.000 | 99.471 | 98.235 | 96.299 |
| 43 | 100.000 | 100.000 | 99.515 | 98.361 | 96.529 |
| 44 | 100.000 | 100.000 | 99.559 | 98.487 | 96.758 |
| 45 | 100.000 | 100.000 | 99.603 | 98.612 | 96.988 |
| 46 | 100.000 | 100.000 | 99.647 | 98.738 | 97.218 |
| 47 | 100.000 | 100.000 | 99.692 | 98.864 | 97.447 |
| 48 | 100.000 | 100.000 | 99.736 | 98.990 | 97.677 |
| 49 | 100.000 | 100.000 | 99.780 | 99.116 | 97.907 |
| 50 | 100.000 | 100.000 | 99.824 | 99.241 | 98.136 |
| 51 | 100.000 | 100.000 | 99.868 | 99.367 | 98.366 |
| 52 | 100.000 | 100.000 | 99.912 | 99.493 | 98.596 |
| 53 | 100.000 | 100.000 | 99.956 | 99.619 | 98.825 |
| 54 | 100.000 | 100.000 | 100.000 | 99.744 | 99.055 |
| 55 | 100.000 | 100.000 | 100.000 | 99.759 | 99.104 |
| 56 | 100.000 | 100.000 | 100.000 | 99.773 | 99.154 |
| 57 | 100.000 | 100.000 | 100.000 | 99.787 | 99.203 |
| 58 | 100.000 | 100.000 | 100.000 | 99.801 | 99.253 |
| 59 | 100.000 | 100.000 | 100.000 | 99.815 | 99.302 |
| 60 | 100.000 | 100.000 | 100.000 | 99.830 | 99.352 |
| 61 | 100.000 | 100.000 | 100.000 | 99.844 | 99.401 |
| 62 | 100.000 | 100.000 | 100.000 | 99.858 | 99.451 |
| 63 | 100.000 | 100.000 | 100.000 | 99.872 | 99.500 |
| 64 | 100.000 | 100.000 | 100.000 | 99.886 | 99.550 |
| 65 | 100.000 | 100.000 | 100.000 | 99.901 | 99.599 |
| 66 | 100.000 | 100.000 | 100.000 | 99.915 | 99.649 |
| 67 | 100.000 | 100.000 | 100.000 | 99.929 | 99.698 |
| 68 | 100.000 | 100.000 | 100.000 | 99.943 | 99.748 |
| 69 | 100.000 | 100.000 | 100.000 | 99.957 | 99.797 |
| 70 | 100.000 | 100.000 | 100.000 | 99.972 | 99.847 |
| 71 | 100.000 | 100.000 | 100.000 | 99.986 | 99.896 |
| 72 | 100.000 | 100.000 | 100.000 | 100.000 | 99.946 |
gzip
Generation Strength: 5, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
| 7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
| 8 | 97.368 | 89.555 | 80.449 | 71.408 | 62.925 |
| 9 | 100.000 | 93.088 | 84.195 | 75.088 | 66.434 |
| 10 | 100.000 | 93.856 | 85.664 | 77.030 | 68.638 |
| 11 | 100.000 | 94.624 | 87.147 | 79.036 | 70.987 |
| 12 | 100.000 | 95.392 | 88.646 | 81.057 | 73.327 |
| 13 | 100.000 | 96.160 | 90.129 | 83.051 | 75.641 |
| 14 | 100.000 | 96.928 | 91.627 | 85.071 | 77.980 |
| 15 | 100.000 | 97.696 | 93.111 | 87.061 | 80.282 |
| 16 | 100.000 | 98.464 | 94.683 | 89.279 | 82.953 |
| 17 | 100.000 | 99.232 | 96.152 | 91.222 | 85.166 |
| 18 | 100.000 | 100.000 | 97.620 | 93.159 | 87.358 |
| 19 | 100.000 | 100.000 | 97.679 | 93.318 | 87.632 |
| 20 | 100.000 | 100.000 | 97.767 | 93.555 | 88.045 |
| 21 | 100.000 | 100.000 | 97.855 | 93.792 | 88.455 |
| 22 | 100.000 | 100.000 | 97.914 | 93.963 | 88.771 |
| 23 | 100.000 | 100.000 | 97.973 | 94.121 | 89.044 |
| 24 | 100.000 | 100.000 | 98.002 | 94.212 | 89.223 |
| 25 | 100.000 | 100.000 | 98.090 | 94.458 | 89.662 |
| 26 | 100.000 | 100.000 | 98.179 | 94.695 | 90.072 |
| 27 | 100.000 | 100.000 | 98.193 | 94.738 | 90.152 |
| 28 | 100.000 | 100.000 | 98.281 | 94.975 | 90.565 |
| 29 | 100.000 | 100.000 | 98.340 | 95.133 | 90.838 |
| 30 | 100.000 | 100.000 | 98.428 | 95.375 | 91.264 |
| 31 | 100.000 | 100.000 | 98.472 | 95.505 | 91.507 |
| 32 | 100.000 | 100.000 | 98.531 | 95.663 | 91.781 |
| 33 | 100.000 | 100.000 | 98.561 | 95.746 | 91.933 |
| 34 | 100.000 | 100.000 | 98.649 | 95.983 | 92.344 |
| 35 | 100.000 | 100.000 | 98.678 | 96.066 | 92.496 |
| 36 | 100.000 | 100.000 | 98.766 | 96.304 | 92.906 |
| 37 | 100.000 | 100.000 | 98.781 | 96.355 | 93.012 |
| 38 | 100.000 | 100.000 | 98.810 | 96.438 | 93.163 |
| 39 | 100.000 | 100.000 | 98.854 | 96.559 | 93.381 |
| 40 | 100.000 | 100.000 | 98.913 | 96.722 | 93.669 |
| 41 | 100.000 | 100.000 | 99.001 | 96.959 | 94.079 |
| 42 | 100.000 | 100.000 | 99.031 | 97.042 | 94.230 |
| 43 | 100.000 | 100.000 | 99.045 | 97.085 | 94.309 |
| 44 | 100.000 | 100.000 | 99.104 | 97.247 | 94.597 |
| 45 | 100.000 | 100.000 | 99.148 | 97.369 | 94.812 |
| 46 | 100.000 | 100.000 | 99.163 | 97.411 | 94.892 |
| 47 | 100.000 | 100.000 | 99.207 | 97.533 | 95.108 |
| 48 | 100.000 | 100.000 | 99.236 | 97.616 | 95.261 |
| 49 | 100.000 | 100.000 | 99.266 | 97.700 | 95.412 |
| 50 | 100.000 | 100.000 | 99.354 | 97.937 | 95.822 |
| 51 | 100.000 | 100.000 | 99.354 | 97.945 | 95.850 |
| 52 | 100.000 | 100.000 | 99.354 | 97.949 | 95.865 |
| 53 | 100.000 | 100.000 | 99.354 | 97.957 | 95.894 |
| 54 | 100.000 | 100.000 | 99.354 | 97.959 | 95.901 |
| 55 | 100.000 | 100.000 | 99.412 | 98.117 | 96.176 |
| 56 | 100.000 | 100.000 | 99.412 | 98.124 | 96.197 |
| 57 | 100.000 | 100.000 | 99.471 | 98.286 | 96.485 |
| 58 | 100.000 | 100.000 | 99.530 | 98.448 | 96.772 |
| 59 | 100.000 | 100.000 | 99.589 | 98.610 | 97.059 |
| 60 | 100.000 | 100.000 | 99.618 | 98.698 | 97.224 |
| 61 | 100.000 | 100.000 | 99.633 | 98.740 | 97.304 |
| 62 | 100.000 | 100.000 | 99.633 | 98.746 | 97.325 |
| 63 | 100.000 | 100.000 | 99.692 | 98.911 | 97.620 |
| 64 | 100.000 | 100.000 | 99.692 | 98.915 | 97.634 |
| 65 | 100.000 | 100.000 | 99.692 | 98.919 | 97.649 |
| 66 | 100.000 | 100.000 | 99.721 | 99.002 | 97.800 |
| 67 | 100.000 | 100.000 | 99.750 | 99.085 | 97.951 |
| 68 | 100.000 | 100.000 | 99.780 | 99.168 | 98.102 |
| 69 | 100.000 | 100.000 | 99.794 | 99.211 | 98.182 |
| 70 | 100.000 | 100.000 | 99.794 | 99.215 | 98.196 |
| 71 | 100.000 | 100.000 | 99.794 | 99.221 | 98.218 |
| 72 | 100.000 | 100.000 | 99.838 | 99.343 | 98.434 |
| 73 | 100.000 | 100.000 | 99.838 | 99.347 | 98.448 |
| 74 | 100.000 | 100.000 | 99.868 | 99.430 | 98.599 |
| 75 | 100.000 | 100.000 | 99.868 | 99.434 | 98.614 |
| 76 | 100.000 | 100.000 | 99.868 | 99.434 | 98.615 |
| 77 | 100.000 | 100.000 | 99.882 | 99.477 | 98.694 |
| 78 | 100.000 | 100.000 | 99.882 | 99.487 | 98.730 |
| 79 | 100.000 | 100.000 | 99.882 | 99.493 | 98.751 |
| 80 | 100.000 | 100.000 | 99.882 | 99.499 | 98.773 |
| 81 | 100.000 | 100.000 | 99.882 | 99.507 | 98.801 |
| 82 | 100.000 | 100.000 | 99.912 | 99.590 | 98.953 |
| 83 | 100.000 | 100.000 | 99.912 | 99.594 | 98.967 |
| 84 | 100.000 | 100.000 | 99.912 | 99.594 | 98.968 |
| 85 | 100.000 | 100.000 | 99.912 | 99.598 | 98.983 |
| 86 | 100.000 | 100.000 | 99.912 | 99.606 | 99.011 |
| 87 | 100.000 | 100.000 | 99.912 | 99.613 | 99.033 |
| 88 | 100.000 | 100.000 | 99.912 | 99.613 | 99.034 |
| 89 | 100.000 | 100.000 | 99.912 | 99.623 | 99.069 |
| 90 | 100.000 | 100.000 | 99.941 | 99.706 | 99.220 |
| 91 | 100.000 | 100.000 | 99.956 | 99.753 | 99.314 |
| 92 | 100.000 | 100.000 | 99.956 | 99.753 | 99.314 |
| 93 | 100.000 | 100.000 | 99.956 | 99.757 | 99.329 |
| 94 | 100.000 | 100.000 | 99.956 | 99.761 | 99.344 |
| 95 | 100.000 | 100.000 | 99.956 | 99.769 | 99.373 |
| 96 | 100.000 | 100.000 | 99.956 | 99.775 | 99.394 |
| 97 | 100.000 | 100.000 | 99.985 | 99.858 | 99.545 |
| 98 | 100.000 | 100.000 | 99.985 | 99.858 | 99.546 |
| 99 | 100.000 | 100.000 | 99.985 | 99.858 | 99.547 |
| 100 | 100.000 | 100.000 | 99.985 | 99.864 | 99.568 |
| 101 | 100.000 | 100.000 | 99.985 | 99.870 | 99.590 |
| 102 | 100.000 | 100.000 | 100.000 | 99.913 | 99.669 |
| 103 | 100.000 | 100.000 | 100.000 | 99.919 | 99.691 |
| 104 | 100.000 | 100.000 | 100.000 | 99.919 | 99.691 |
| 105 | 100.000 | 100.000 | 100.000 | 99.919 | 99.692 |
| 106 | 100.000 | 100.000 | 100.000 | 99.921 | 99.700 |
| 107 | 100.000 | 100.000 | 100.000 | 99.925 | 99.714 |
| 108 | 100.000 | 100.000 | 100.000 | 99.927 | 99.722 |
| 109 | 100.000 | 100.000 | 100.000 | 99.931 | 99.737 |
| 110 | 100.000 | 100.000 | 100.000 | 99.935 | 99.751 |
| 111 | 100.000 | 100.000 | 100.000 | 99.943 | 99.780 |
| 112 | 100.000 | 100.000 | 100.000 | 99.945 | 99.788 |
| 113 | 100.000 | 100.000 | 100.000 | 99.949 | 99.802 |
| 114 | 100.000 | 100.000 | 100.000 | 99.951 | 99.810 |
| 115 | 100.000 | 100.000 | 100.000 | 99.955 | 99.825 |
| 116 | 100.000 | 100.000 | 100.000 | 99.955 | 99.825 |
| 117 | 100.000 | 100.000 | 100.000 | 99.959 | 99.840 |
| 118 | 100.000 | 100.000 | 100.000 | 99.959 | 99.841 |
| 119 | 100.000 | 100.000 | 100.000 | 99.963 | 99.856 |
| 120 | 100.000 | 100.000 | 100.000 | 99.968 | 99.870 |
| 121 | 100.000 | 100.000 | 100.000 | 99.976 | 99.899 |
| 122 | 100.000 | 100.000 | 100.000 | 99.980 | 99.914 |
| 123 | 100.000 | 100.000 | 100.000 | 99.984 | 99.929 |
| 124 | 100.000 | 100.000 | 100.000 | 99.984 | 99.929 |
| 125 | 100.000 | 100.000 | 100.000 | 99.988 | 99.944 |
| 126 | 100.000 | 100.000 | 100.000 | 99.988 | 99.945 |
| 127 | 100.000 | 100.000 | 100.000 | 99.988 | 99.946 |
| 128 | 100.000 | 100.000 | 100.000 | 99.992 | 99.960 |
| 129 | 100.000 | 100.000 | 100.000 | 99.992 | 99.961 |
| 130 | 100.000 | 100.000 | 100.000 | 99.992 | 99.962 |
| 131 | 100.000 | 100.000 | 100.000 | 99.992 | 99.963 |
| 132 | 100.000 | 100.000 | 100.000 | 99.994 | 99.970 |
| 133 | 100.000 | 100.000 | 100.000 | 99.994 | 99.971 |
| 134 | 100.000 | 100.000 | 100.000 | 99.994 | 99.972 |
| 135 | 100.000 | 100.000 | 100.000 | 99.998 | 99.986 |
| 136 | 100.000 | 100.000 | 100.000 | 99.998 | 99.987 |
| 137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
gzip
Generation Strength: 5, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
| 7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
| 8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
| 9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
| 10 | 100.000 | 93.856 | 85.870 | 77.559 | 69.501 |
| 11 | 100.000 | 94.624 | 87.338 | 79.511 | 71.739 |
| 12 | 100.000 | 95.392 | 88.807 | 81.478 | 74.025 |
| 13 | 100.000 | 96.160 | 90.276 | 83.428 | 76.257 |
| 14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.525 |
| 15 | 100.000 | 97.696 | 93.214 | 87.335 | 80.745 |
| 16 | 100.000 | 98.464 | 94.683 | 89.283 | 82.970 |
| 17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
| 18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
| 19 | 100.000 | 100.000 | 97.709 | 93.421 | 87.851 |
| 20 | 100.000 | 100.000 | 97.797 | 93.663 | 88.276 |
| 21 | 100.000 | 100.000 | 97.885 | 93.900 | 88.688 |
| 22 | 100.000 | 100.000 | 97.973 | 94.137 | 89.098 |
| 23 | 100.000 | 100.000 | 98.061 | 94.379 | 89.523 |
| 24 | 100.000 | 100.000 | 98.149 | 94.616 | 89.934 |
| 25 | 100.000 | 100.000 | 98.237 | 94.853 | 90.346 |
| 26 | 100.000 | 100.000 | 98.325 | 95.091 | 90.757 |
| 27 | 100.000 | 100.000 | 98.414 | 95.328 | 91.167 |
| 28 | 100.000 | 100.000 | 98.502 | 95.569 | 91.592 |
| 29 | 100.000 | 100.000 | 98.590 | 95.807 | 92.002 |
| 30 | 100.000 | 100.000 | 98.678 | 96.044 | 92.413 |
| 31 | 100.000 | 100.000 | 98.766 | 96.286 | 92.838 |
| 32 | 100.000 | 100.000 | 98.854 | 96.523 | 93.248 |
| 33 | 100.000 | 100.000 | 98.942 | 96.764 | 93.672 |
| 34 | 100.000 | 100.000 | 99.031 | 97.002 | 94.082 |
| 35 | 100.000 | 100.000 | 99.119 | 97.239 | 94.492 |
| 36 | 100.000 | 100.000 | 99.207 | 97.476 | 94.902 |
| 37 | 100.000 | 100.000 | 99.266 | 97.639 | 95.190 |
| 38 | 100.000 | 100.000 | 99.324 | 97.797 | 95.464 |
| 39 | 100.000 | 100.000 | 99.383 | 97.959 | 95.752 |
| 40 | 100.000 | 100.000 | 99.442 | 98.121 | 96.039 |
| 41 | 100.000 | 100.000 | 99.486 | 98.247 | 96.269 |
| 42 | 100.000 | 100.000 | 99.530 | 98.373 | 96.499 |
| 43 | 100.000 | 100.000 | 99.574 | 98.499 | 96.728 |
| 44 | 100.000 | 100.000 | 99.618 | 98.625 | 96.958 |
| 45 | 100.000 | 100.000 | 99.662 | 98.750 | 97.187 |
| 46 | 100.000 | 100.000 | 99.706 | 98.876 | 97.417 |
| 47 | 100.000 | 100.000 | 99.750 | 99.002 | 97.647 |
| 48 | 100.000 | 100.000 | 99.794 | 99.128 | 97.876 |
| 49 | 100.000 | 100.000 | 99.824 | 99.211 | 98.028 |
| 50 | 100.000 | 100.000 | 99.853 | 99.298 | 98.193 |
| 51 | 100.000 | 100.000 | 99.882 | 99.385 | 98.359 |
| 52 | 100.000 | 100.000 | 99.912 | 99.468 | 98.510 |
| 53 | 100.000 | 100.000 | 99.941 | 99.556 | 98.675 |
| 54 | 100.000 | 100.000 | 99.971 | 99.639 | 98.826 |
| 55 | 100.000 | 100.000 | 99.985 | 99.686 | 98.920 |
| 56 | 100.000 | 100.000 | 100.000 | 99.732 | 99.013 |
| 57 | 100.000 | 100.000 | 100.000 | 99.736 | 99.028 |
| 58 | 100.000 | 100.000 | 100.000 | 99.738 | 99.036 |
| 59 | 100.000 | 100.000 | 100.000 | 99.740 | 99.044 |
| 60 | 100.000 | 100.000 | 100.000 | 99.742 | 99.052 |
| 61 | 100.000 | 100.000 | 100.000 | 99.757 | 99.101 |
| 62 | 100.000 | 100.000 | 100.000 | 99.767 | 99.137 |
| 63 | 100.000 | 100.000 | 100.000 | 99.767 | 99.138 |
| 64 | 100.000 | 100.000 | 100.000 | 99.771 | 99.152 |
| 65 | 100.000 | 100.000 | 100.000 | 99.773 | 99.160 |
| 66 | 100.000 | 100.000 | 100.000 | 99.777 | 99.175 |
| 67 | 100.000 | 100.000 | 100.000 | 99.783 | 99.197 |
| 68 | 100.000 | 100.000 | 100.000 | 99.797 | 99.246 |
| 69 | 100.000 | 100.000 | 100.000 | 99.801 | 99.261 |
| 70 | 100.000 | 100.000 | 100.000 | 99.805 | 99.276 |
| 71 | 100.000 | 100.000 | 100.000 | 99.805 | 99.277 |
| 72 | 100.000 | 100.000 | 100.000 | 99.813 | 99.305 |
| 73 | 100.000 | 100.000 | 100.000 | 99.817 | 99.320 |
| 74 | 100.000 | 100.000 | 100.000 | 99.821 | 99.335 |
| 75 | 100.000 | 100.000 | 100.000 | 99.821 | 99.336 |
| 76 | 100.000 | 100.000 | 100.000 | 99.826 | 99.350 |
| 77 | 100.000 | 100.000 | 100.000 | 99.832 | 99.372 |
| 78 | 100.000 | 100.000 | 100.000 | 99.834 | 99.379 |
| 79 | 100.000 | 100.000 | 100.000 | 99.838 | 99.394 |
| 80 | 100.000 | 100.000 | 100.000 | 99.840 | 99.402 |
| 81 | 100.000 | 100.000 | 100.000 | 99.840 | 99.402 |
| 82 | 100.000 | 100.000 | 100.000 | 99.844 | 99.417 |
| 83 | 100.000 | 100.000 | 100.000 | 99.844 | 99.418 |
| 84 | 100.000 | 100.000 | 100.000 | 99.848 | 99.433 |
| 85 | 100.000 | 100.000 | 100.000 | 99.856 | 99.461 |
| 86 | 100.000 | 100.000 | 100.000 | 99.860 | 99.476 |
| 87 | 100.000 | 100.000 | 100.000 | 99.862 | 99.484 |
| 88 | 100.000 | 100.000 | 100.000 | 99.870 | 99.512 |
| 89 | 100.000 | 100.000 | 100.000 | 99.878 | 99.541 |
| 90 | 100.000 | 100.000 | 100.000 | 99.878 | 99.542 |
| 91 | 100.000 | 100.000 | 100.000 | 99.886 | 99.571 |
| 92 | 100.000 | 100.000 | 100.000 | 99.890 | 99.585 |
| 93 | 100.000 | 100.000 | 100.000 | 99.899 | 99.614 |
| 94 | 100.000 | 100.000 | 100.000 | 99.903 | 99.629 |
| 95 | 100.000 | 100.000 | 100.000 | 99.907 | 99.643 |
| 96 | 100.000 | 100.000 | 100.000 | 99.907 | 99.644 |
| 97 | 100.000 | 100.000 | 100.000 | 99.907 | 99.645 |
| 98 | 100.000 | 100.000 | 100.000 | 99.913 | 99.666 |
| 99 | 100.000 | 100.000 | 100.000 | 99.913 | 99.667 |
| 100 | 100.000 | 100.000 | 100.000 | 99.923 | 99.703 |
| 101 | 100.000 | 100.000 | 100.000 | 99.929 | 99.724 |
| 102 | 100.000 | 100.000 | 100.000 | 99.935 | 99.746 |
| 103 | 100.000 | 100.000 | 100.000 | 99.941 | 99.767 |
| 104 | 100.000 | 100.000 | 100.000 | 99.947 | 99.789 |
| 105 | 100.000 | 100.000 | 100.000 | 99.955 | 99.818 |
| 106 | 100.000 | 100.000 | 100.000 | 99.955 | 99.818 |
| 107 | 100.000 | 100.000 | 100.000 | 99.963 | 99.847 |
| 108 | 100.000 | 100.000 | 100.000 | 99.963 | 99.848 |
| 109 | 100.000 | 100.000 | 100.000 | 99.963 | 99.849 |
| 110 | 100.000 | 100.000 | 100.000 | 99.966 | 99.856 |
| 111 | 100.000 | 100.000 | 100.000 | 99.968 | 99.864 |
| 112 | 100.000 | 100.000 | 100.000 | 99.970 | 99.871 |
| 113 | 100.000 | 100.000 | 100.000 | 99.970 | 99.872 |
| 114 | 100.000 | 100.000 | 100.000 | 99.972 | 99.880 |
| 115 | 100.000 | 100.000 | 100.000 | 99.972 | 99.880 |
| 116 | 100.000 | 100.000 | 100.000 | 99.976 | 99.895 |
| 117 | 100.000 | 100.000 | 100.000 | 99.980 | 99.910 |
| 118 | 100.000 | 100.000 | 100.000 | 99.980 | 99.910 |
| 119 | 100.000 | 100.000 | 100.000 | 99.980 | 99.911 |
| 120 | 100.000 | 100.000 | 100.000 | 99.980 | 99.912 |
| 121 | 100.000 | 100.000 | 100.000 | 99.980 | 99.913 |
| 122 | 100.000 | 100.000 | 100.000 | 99.988 | 99.941 |
| 123 | 100.000 | 100.000 | 100.000 | 99.988 | 99.942 |
| 124 | 100.000 | 100.000 | 100.000 | 99.988 | 99.943 |
| 125 | 100.000 | 100.000 | 100.000 | 99.992 | 99.958 |
| 126 | 100.000 | 100.000 | 100.000 | 99.992 | 99.958 |
| 127 | 100.000 | 100.000 | 100.000 | 99.992 | 99.959 |
| 128 | 100.000 | 100.000 | 100.000 | 99.992 | 99.960 |
| 129 | 100.000 | 100.000 | 100.000 | 99.992 | 99.961 |
| 130 | 100.000 | 100.000 | 100.000 | 99.996 | 99.975 |
| 131 | 100.000 | 100.000 | 100.000 | 99.996 | 99.976 |
| 132 | 100.000 | 100.000 | 100.000 | 99.996 | 99.977 |
| 133 | 100.000 | 100.000 | 100.000 | 99.996 | 99.978 |
| 134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
gzip
Generation Strength: 5, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
| 7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
| 8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
| 9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
| 10 | 100.000 | 93.856 | 85.870 | 77.563 | 69.515 |
| 11 | 100.000 | 94.624 | 87.338 | 79.525 | 71.783 |
| 12 | 100.000 | 95.392 | 88.807 | 81.487 | 74.051 |
| 13 | 100.000 | 96.160 | 90.276 | 83.438 | 76.289 |
| 14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.526 |
| 15 | 100.000 | 97.696 | 93.214 | 87.341 | 80.762 |
| 16 | 100.000 | 98.464 | 94.683 | 89.289 | 82.988 |
| 17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
| 18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
| 19 | 100.000 | 100.000 | 97.709 | 93.425 | 87.865 |
| 20 | 100.000 | 100.000 | 97.797 | 93.667 | 88.291 |
| 21 | 100.000 | 100.000 | 97.885 | 93.908 | 88.717 |
| 22 | 100.000 | 100.000 | 97.973 | 94.149 | 89.143 |
| 23 | 100.000 | 100.000 | 98.061 | 94.391 | 89.569 |
| 24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.995 |
| 25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.406 |
| 26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.816 |
| 27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.226 |
| 28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.636 |
| 29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.046 |
| 30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.456 |
| 31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.866 |
| 32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.277 |
| 33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.687 |
| 34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.097 |
| 35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.507 |
| 36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.917 |
| 37 | 100.000 | 100.000 | 99.266 | 97.643 | 95.205 |
| 38 | 100.000 | 100.000 | 99.324 | 97.805 | 95.492 |
| 39 | 100.000 | 100.000 | 99.383 | 97.967 | 95.780 |
| 40 | 100.000 | 100.000 | 99.442 | 98.130 | 96.067 |
| 41 | 100.000 | 100.000 | 99.501 | 98.292 | 96.355 |
| 42 | 100.000 | 100.000 | 99.545 | 98.418 | 96.584 |
| 43 | 100.000 | 100.000 | 99.589 | 98.543 | 96.814 |
| 44 | 100.000 | 100.000 | 99.633 | 98.669 | 97.044 |
| 45 | 100.000 | 100.000 | 99.677 | 98.795 | 97.273 |
| 46 | 100.000 | 100.000 | 99.706 | 98.882 | 97.438 |
| 47 | 100.000 | 100.000 | 99.736 | 98.969 | 97.603 |
| 48 | 100.000 | 100.000 | 99.765 | 99.057 | 97.768 |
| 49 | 100.000 | 100.000 | 99.794 | 99.144 | 97.934 |
| 50 | 100.000 | 100.000 | 99.824 | 99.231 | 98.099 |
| 51 | 100.000 | 100.000 | 99.853 | 99.314 | 98.250 |
| 52 | 100.000 | 100.000 | 99.882 | 99.397 | 98.401 |
| 53 | 100.000 | 100.000 | 99.912 | 99.481 | 98.553 |
| 54 | 100.000 | 100.000 | 99.941 | 99.564 | 98.704 |
| 55 | 100.000 | 100.000 | 99.971 | 99.647 | 98.855 |
| 56 | 100.000 | 100.000 | 99.985 | 99.694 | 98.948 |
| 57 | 100.000 | 100.000 | 100.000 | 99.740 | 99.041 |
| 58 | 100.000 | 100.000 | 100.000 | 99.755 | 99.091 |
| 59 | 100.000 | 100.000 | 100.000 | 99.769 | 99.140 |
| 60 | 100.000 | 100.000 | 100.000 | 99.783 | 99.190 |
| 61 | 100.000 | 100.000 | 100.000 | 99.797 | 99.239 |
| 62 | 100.000 | 100.000 | 100.000 | 99.807 | 99.275 |
| 63 | 100.000 | 100.000 | 100.000 | 99.817 | 99.310 |
| 64 | 100.000 | 100.000 | 100.000 | 99.828 | 99.346 |
| 65 | 100.000 | 100.000 | 100.000 | 99.838 | 99.381 |
| 66 | 100.000 | 100.000 | 100.000 | 99.848 | 99.417 |
| 67 | 100.000 | 100.000 | 100.000 | 99.858 | 99.452 |
| 68 | 100.000 | 100.000 | 100.000 | 99.868 | 99.488 |
| 69 | 100.000 | 100.000 | 100.000 | 99.878 | 99.523 |
| 70 | 100.000 | 100.000 | 100.000 | 99.888 | 99.559 |
| 71 | 100.000 | 100.000 | 100.000 | 99.899 | 99.594 |
| 72 | 100.000 | 100.000 | 100.000 | 99.909 | 99.630 |
| 73 | 100.000 | 100.000 | 100.000 | 99.919 | 99.665 |
| 74 | 100.000 | 100.000 | 100.000 | 99.929 | 99.701 |
| 75 | 100.000 | 100.000 | 100.000 | 99.939 | 99.736 |
| 76 | 100.000 | 100.000 | 100.000 | 99.949 | 99.773 |
| 77 | 100.000 | 100.000 | 100.000 | 99.959 | 99.808 |
| 78 | 100.000 | 100.000 | 100.000 | 99.970 | 99.844 |
| 79 | 100.000 | 100.000 | 100.000 | 99.980 | 99.879 |
| 80 | 100.000 | 100.000 | 100.000 | 99.988 | 99.908 |
| 81 | 100.000 | 100.000 | 100.000 | 99.994 | 99.929 |
| 82 | 100.000 | 100.000 | 100.000 | 100.000 | 99.951 |
| 83 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
| 84 | 100.000 | 100.000 | 100.000 | 100.000 | 99.952 |
| 85 | 100.000 | 100.000 | 100.000 | 100.000 | 99.954 |
| 86 | 100.000 | 100.000 | 100.000 | 100.000 | 99.955 |
| 87 | 100.000 | 100.000 | 100.000 | 100.000 | 99.955 |
| 88 | 100.000 | 100.000 | 100.000 | 100.000 | 99.956 |
| 89 | 100.000 | 100.000 | 100.000 | 100.000 | 99.957 |
| 90 | 100.000 | 100.000 | 100.000 | 100.000 | 99.958 |
| 91 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
| 92 | 100.000 | 100.000 | 100.000 | 100.000 | 99.960 |
| 93 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
| 94 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
| 95 | 100.000 | 100.000 | 100.000 | 100.000 | 99.962 |
| 96 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
| 97 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 98 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 99 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
| 100 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 101 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
| 102 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
| 103 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
| 104 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
| 105 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
| 106 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
| 107 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
| 108 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
| 109 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 110 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
| 112 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
| 113 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 114 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 115 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
| 116 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
| 117 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
| 118 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
| 119 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
| 120 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
| 121 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 122 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 123 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
| 124 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
| 125 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 126 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 127 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
| 128 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 129 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 130 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 131 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
| 132 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 133 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
gzip
Generation Strength: 5, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 63.158 | 42.396 | 29.730 | 21.556 | 16.055 |
| 2 | 81.579 | 63.902 | 49.471 | 38.285 | 29.772 |
| 3 | 84.211 | 69.278 | 56.448 | 45.841 | 37.228 |
| 4 | 86.842 | 74.501 | 63.205 | 53.185 | 44.513 |
| 5 | 89.474 | 78.495 | 67.994 | 58.388 | 49.838 |
| 6 | 92.105 | 82.488 | 72.782 | 63.590 | 55.158 |
| 7 | 94.737 | 86.022 | 76.733 | 67.775 | 59.452 |
| 8 | 97.368 | 89.555 | 80.684 | 71.960 | 63.746 |
| 9 | 100.000 | 93.088 | 84.401 | 75.589 | 67.208 |
| 10 | 100.000 | 93.856 | 85.870 | 77.563 | 69.515 |
| 11 | 100.000 | 94.624 | 87.338 | 79.525 | 71.783 |
| 12 | 100.000 | 95.392 | 88.807 | 81.487 | 74.051 |
| 13 | 100.000 | 96.160 | 90.276 | 83.438 | 76.289 |
| 14 | 100.000 | 96.928 | 91.745 | 85.390 | 78.526 |
| 15 | 100.000 | 97.696 | 93.214 | 87.339 | 80.758 |
| 16 | 100.000 | 98.464 | 94.683 | 89.289 | 82.989 |
| 17 | 100.000 | 99.232 | 96.152 | 91.236 | 85.214 |
| 18 | 100.000 | 100.000 | 97.620 | 93.184 | 87.438 |
| 19 | 100.000 | 100.000 | 97.709 | 93.425 | 87.865 |
| 20 | 100.000 | 100.000 | 97.797 | 93.667 | 88.291 |
| 21 | 100.000 | 100.000 | 97.885 | 93.908 | 88.717 |
| 22 | 100.000 | 100.000 | 97.973 | 94.149 | 89.143 |
| 23 | 100.000 | 100.000 | 98.061 | 94.391 | 89.570 |
| 24 | 100.000 | 100.000 | 98.149 | 94.632 | 89.996 |
| 25 | 100.000 | 100.000 | 98.237 | 94.870 | 90.406 |
| 26 | 100.000 | 100.000 | 98.325 | 95.107 | 90.817 |
| 27 | 100.000 | 100.000 | 98.414 | 95.344 | 91.227 |
| 28 | 100.000 | 100.000 | 98.502 | 95.582 | 91.638 |
| 29 | 100.000 | 100.000 | 98.590 | 95.819 | 92.049 |
| 30 | 100.000 | 100.000 | 98.678 | 96.056 | 92.459 |
| 31 | 100.000 | 100.000 | 98.766 | 96.294 | 92.870 |
| 32 | 100.000 | 100.000 | 98.854 | 96.531 | 93.280 |
| 33 | 100.000 | 100.000 | 98.942 | 96.768 | 93.691 |
| 34 | 100.000 | 100.000 | 99.031 | 97.006 | 94.101 |
| 35 | 100.000 | 100.000 | 99.119 | 97.243 | 94.511 |
| 36 | 100.000 | 100.000 | 99.207 | 97.480 | 94.921 |
| 37 | 100.000 | 100.000 | 99.266 | 97.643 | 95.209 |
| 38 | 100.000 | 100.000 | 99.324 | 97.805 | 95.496 |
| 39 | 100.000 | 100.000 | 99.383 | 97.967 | 95.783 |
| 40 | 100.000 | 100.000 | 99.442 | 98.130 | 96.071 |
| 41 | 100.000 | 100.000 | 99.486 | 98.255 | 96.301 |
| 42 | 100.000 | 100.000 | 99.530 | 98.381 | 96.530 |
| 43 | 100.000 | 100.000 | 99.559 | 98.468 | 96.695 |
| 44 | 100.000 | 100.000 | 99.589 | 98.556 | 96.860 |
| 45 | 100.000 | 100.000 | 99.618 | 98.643 | 97.025 |
| 46 | 100.000 | 100.000 | 99.647 | 98.730 | 97.190 |
| 47 | 100.000 | 100.000 | 99.677 | 98.813 | 97.342 |
| 48 | 100.000 | 100.000 | 99.706 | 98.896 | 97.493 |
| 49 | 100.000 | 100.000 | 99.736 | 98.980 | 97.644 |
| 50 | 100.000 | 100.000 | 99.765 | 99.063 | 97.795 |
| 51 | 100.000 | 100.000 | 99.794 | 99.146 | 97.946 |
| 52 | 100.000 | 100.000 | 99.824 | 99.229 | 98.097 |
| 53 | 100.000 | 100.000 | 99.853 | 99.312 | 98.248 |
| 54 | 100.000 | 100.000 | 99.882 | 99.395 | 98.399 |
| 55 | 100.000 | 100.000 | 99.912 | 99.479 | 98.550 |
| 56 | 100.000 | 100.000 | 99.927 | 99.525 | 98.644 |
| 57 | 100.000 | 100.000 | 99.941 | 99.572 | 98.737 |
| 58 | 100.000 | 100.000 | 99.956 | 99.619 | 98.830 |
| 59 | 100.000 | 100.000 | 99.971 | 99.665 | 98.923 |
| 60 | 100.000 | 100.000 | 99.985 | 99.712 | 99.017 |
| 61 | 100.000 | 100.000 | 100.000 | 99.759 | 99.110 |
| 62 | 100.000 | 100.000 | 100.000 | 99.773 | 99.160 |
| 63 | 100.000 | 100.000 | 100.000 | 99.787 | 99.209 |
| 64 | 100.000 | 100.000 | 100.000 | 99.797 | 99.245 |
| 65 | 100.000 | 100.000 | 100.000 | 99.807 | 99.280 |
| 66 | 100.000 | 100.000 | 100.000 | 99.817 | 99.316 |
| 67 | 100.000 | 100.000 | 100.000 | 99.828 | 99.351 |
| 68 | 100.000 | 100.000 | 100.000 | 99.838 | 99.387 |
| 69 | 100.000 | 100.000 | 100.000 | 99.848 | 99.422 |
| 70 | 100.000 | 100.000 | 100.000 | 99.858 | 99.458 |
| 71 | 100.000 | 100.000 | 100.000 | 99.868 | 99.493 |
| 72 | 100.000 | 100.000 | 100.000 | 99.878 | 99.529 |
| 73 | 100.000 | 100.000 | 100.000 | 99.888 | 99.564 |
| 74 | 100.000 | 100.000 | 100.000 | 99.899 | 99.600 |
| 75 | 100.000 | 100.000 | 100.000 | 99.909 | 99.635 |
| 76 | 100.000 | 100.000 | 100.000 | 99.919 | 99.671 |
| 77 | 100.000 | 100.000 | 100.000 | 99.929 | 99.706 |
| 78 | 100.000 | 100.000 | 100.000 | 99.939 | 99.742 |
| 79 | 100.000 | 100.000 | 100.000 | 99.947 | 99.770 |
| 80 | 100.000 | 100.000 | 100.000 | 99.953 | 99.792 |
| 81 | 100.000 | 100.000 | 100.000 | 99.959 | 99.813 |
| 82 | 100.000 | 100.000 | 100.000 | 99.966 | 99.835 |
| 83 | 100.000 | 100.000 | 100.000 | 99.972 | 99.856 |
| 84 | 100.000 | 100.000 | 100.000 | 99.978 | 99.878 |
| 85 | 100.000 | 100.000 | 100.000 | 99.984 | 99.900 |
| 86 | 100.000 | 100.000 | 100.000 | 99.988 | 99.914 |
| 87 | 100.000 | 100.000 | 100.000 | 99.992 | 99.929 |
| 88 | 100.000 | 100.000 | 100.000 | 99.996 | 99.944 |
| 89 | 100.000 | 100.000 | 100.000 | 100.000 | 99.958 |
| 90 | 100.000 | 100.000 | 100.000 | 100.000 | 99.959 |
| 91 | 100.000 | 100.000 | 100.000 | 100.000 | 99.960 |
| 92 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
| 93 | 100.000 | 100.000 | 100.000 | 100.000 | 99.961 |
| 94 | 100.000 | 100.000 | 100.000 | 100.000 | 99.962 |
| 95 | 100.000 | 100.000 | 100.000 | 100.000 | 99.963 |
| 96 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 97 | 100.000 | 100.000 | 100.000 | 100.000 | 99.964 |
| 98 | 100.000 | 100.000 | 100.000 | 100.000 | 99.965 |
| 99 | 100.000 | 100.000 | 100.000 | 100.000 | 99.966 |
| 100 | 100.000 | 100.000 | 100.000 | 100.000 | 99.967 |
| 101 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
| 102 | 100.000 | 100.000 | 100.000 | 100.000 | 99.968 |
| 103 | 100.000 | 100.000 | 100.000 | 100.000 | 99.969 |
| 104 | 100.000 | 100.000 | 100.000 | 100.000 | 99.970 |
| 105 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
| 106 | 100.000 | 100.000 | 100.000 | 100.000 | 99.971 |
| 107 | 100.000 | 100.000 | 100.000 | 100.000 | 99.972 |
| 108 | 100.000 | 100.000 | 100.000 | 100.000 | 99.973 |
| 109 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 110 | 100.000 | 100.000 | 100.000 | 100.000 | 99.974 |
| 111 | 100.000 | 100.000 | 100.000 | 100.000 | 99.975 |
| 112 | 100.000 | 100.000 | 100.000 | 100.000 | 99.976 |
| 113 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 114 | 100.000 | 100.000 | 100.000 | 100.000 | 99.977 |
| 115 | 100.000 | 100.000 | 100.000 | 100.000 | 99.978 |
| 116 | 100.000 | 100.000 | 100.000 | 100.000 | 99.979 |
| 117 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
| 118 | 100.000 | 100.000 | 100.000 | 100.000 | 99.980 |
| 119 | 100.000 | 100.000 | 100.000 | 100.000 | 99.981 |
| 120 | 100.000 | 100.000 | 100.000 | 100.000 | 99.982 |
| 121 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 122 | 100.000 | 100.000 | 100.000 | 100.000 | 99.983 |
| 123 | 100.000 | 100.000 | 100.000 | 100.000 | 99.984 |
| 124 | 100.000 | 100.000 | 100.000 | 100.000 | 99.985 |
| 125 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 126 | 100.000 | 100.000 | 100.000 | 100.000 | 99.986 |
| 127 | 100.000 | 100.000 | 100.000 | 100.000 | 99.987 |
| 128 | 100.000 | 100.000 | 100.000 | 100.000 | 99.988 |
| 129 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 130 | 100.000 | 100.000 | 100.000 | 100.000 | 99.989 |
| 131 | 100.000 | 100.000 | 100.000 | 100.000 | 99.990 |
| 132 | 100.000 | 100.000 | 100.000 | 100.000 | 99.991 |
| 133 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 134 | 100.000 | 100.000 | 100.000 | 100.000 | 99.992 |
| 135 | 100.000 | 100.000 | 100.000 | 100.000 | 99.993 |
| 136 | 100.000 | 100.000 | 100.000 | 100.000 | 99.994 |
| 137 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 138 | 100.000 | 100.000 | 100.000 | 100.000 | 99.995 |
| 139 | 100.000 | 100.000 | 100.000 | 100.000 | 99.996 |
| 140 | 100.000 | 100.000 | 100.000 | 100.000 | 99.997 |
| 141 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 142 | 100.000 | 100.000 | 100.000 | 100.000 | 99.998 |
| 143 | 100.000 | 100.000 | 100.000 | 100.000 | 99.999 |
| 144 | 100.000 | 100.000 | 100.000 | 100.000 | 100.000 |
sed
Generation Strength: 2, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 57.500 | 31.395 | 18.105 | 11.144 | 7.277 |
| 3 | 70.000 | 42.359 | 25.843 | 16.375 | 10.836 |
| 4 | 75.000 | 49.668 | 32.050 | 20.986 | 14.150 |
| 5 | 82.500 | 56.478 | 37.430 | 25.080 | 17.220 |
| 6 | 87.500 | 61.960 | 41.982 | 28.594 | 19.891 |
| 7 | 90.000 | 66.279 | 46.182 | 32.097 | 22.655 |
| 8 | 92.500 | 69.601 | 49.348 | 34.818 | 24.901 |
| 9 | 95.000 | 72.591 | 52.142 | 37.237 | 26.933 |
| 10 | 97.500 | 75.415 | 54.852 | 39.656 | 29.000 |
| 11 | 100.000 | 78.239 | 57.438 | 41.889 | 30.886 |
| 12 | 100.000 | 79.734 | 59.756 | 44.345 | 33.123 |
| 13 | 100.000 | 81.063 | 61.577 | 46.189 | 34.801 |
| 14 | 100.000 | 82.392 | 63.439 | 48.112 | 36.577 |
| 15 | 100.000 | 83.555 | 65.053 | 49.770 | 38.109 |
| 16 | 100.000 | 84.551 | 66.377 | 51.090 | 39.308 |
| 17 | 100.000 | 85.548 | 67.763 | 52.520 | 40.640 |
| 18 | 100.000 | 86.545 | 69.150 | 53.951 | 41.971 |
| 19 | 100.000 | 87.375 | 70.536 | 55.531 | 43.514 |
| 20 | 100.000 | 88.206 | 71.674 | 56.689 | 44.578 |
| 21 | 100.000 | 89.037 | 72.812 | 57.846 | 45.642 |
| 22 | 100.000 | 89.701 | 73.846 | 59.013 | 46.803 |
| 23 | 100.000 | 90.365 | 74.798 | 60.022 | 47.765 |
| 24 | 100.000 | 91.030 | 75.874 | 61.266 | 49.024 |
| 25 | 100.000 | 91.528 | 76.619 | 62.089 | 49.840 |
| 26 | 100.000 | 92.027 | 77.488 | 63.152 | 50.964 |
| 27 | 100.000 | 92.525 | 78.233 | 63.975 | 51.779 |
| 28 | 100.000 | 93.023 | 79.019 | 64.868 | 52.673 |
| 29 | 100.000 | 93.522 | 79.806 | 65.761 | 53.567 |
| 30 | 100.000 | 94.020 | 80.633 | 66.741 | 54.582 |
| 31 | 100.000 | 94.352 | 81.233 | 67.498 | 55.408 |
| 32 | 100.000 | 94.684 | 81.833 | 68.242 | 56.199 |
| 33 | 100.000 | 95.017 | 82.351 | 68.838 | 56.814 |
| 34 | 100.000 | 95.349 | 82.909 | 69.512 | 57.528 |
| 35 | 100.000 | 95.681 | 83.468 | 70.194 | 58.265 |
| 36 | 100.000 | 96.013 | 84.026 | 70.868 | 58.979 |
| 37 | 100.000 | 96.346 | 84.606 | 71.592 | 59.770 |
| 38 | 100.000 | 96.678 | 85.164 | 72.266 | 60.484 |
| 39 | 100.000 | 96.844 | 85.599 | 72.894 | 61.207 |
| 40 | 100.000 | 97.010 | 85.992 | 73.452 | 61.853 |
| 41 | 100.000 | 97.176 | 86.261 | 73.792 | 62.245 |
| 42 | 100.000 | 97.342 | 86.530 | 74.114 | 62.592 |
| 43 | 100.000 | 97.508 | 86.944 | 74.722 | 63.316 |
| 44 | 100.000 | 97.674 | 87.378 | 75.350 | 64.039 |
| 45 | 100.000 | 97.841 | 87.772 | 75.921 | 64.718 |
| 46 | 100.000 | 98.007 | 88.041 | 76.244 | 65.065 |
| 47 | 100.000 | 98.173 | 88.434 | 76.802 | 65.711 |
| 48 | 100.000 | 98.339 | 88.744 | 77.224 | 66.213 |
| 49 | 100.000 | 98.505 | 89.075 | 77.670 | 66.726 |
| 50 | 100.000 | 98.671 | 89.427 | 78.158 | 67.294 |
| 51 | 100.000 | 98.837 | 89.758 | 78.605 | 67.807 |
| 52 | 100.000 | 99.003 | 90.068 | 79.014 | 68.276 |
| 53 | 100.000 | 99.169 | 90.379 | 79.424 | 68.746 |
| 54 | 100.000 | 99.336 | 90.648 | 79.763 | 69.137 |
| 55 | 100.000 | 99.502 | 91.082 | 80.408 | 69.905 |
| 56 | 100.000 | 99.668 | 91.393 | 80.817 | 70.374 |
| 57 | 100.000 | 99.834 | 91.724 | 81.264 | 70.887 |
| 58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |
sed
Generation Strength: 2, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
| 4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
| 5 | 80.000 | 55.316 | 37.016 | 24.964 | 17.203 |
| 6 | 85.000 | 60.465 | 41.548 | 28.677 | 20.122 |
| 7 | 87.500 | 64.452 | 45.396 | 31.915 | 22.716 |
| 8 | 92.500 | 69.269 | 49.328 | 35.016 | 25.149 |
| 9 | 95.000 | 72.924 | 52.866 | 38.035 | 27.599 |
| 10 | 97.500 | 75.914 | 55.659 | 40.454 | 29.631 |
| 11 | 100.000 | 78.738 | 58.225 | 42.646 | 31.462 |
| 12 | 100.000 | 80.066 | 60.087 | 44.568 | 33.238 |
| 13 | 100.000 | 81.395 | 61.949 | 46.491 | 35.015 |
| 14 | 100.000 | 82.392 | 63.542 | 48.274 | 36.743 |
| 15 | 100.000 | 83.389 | 65.032 | 49.899 | 38.315 |
| 16 | 100.000 | 84.385 | 66.418 | 51.329 | 39.646 |
| 17 | 100.000 | 85.382 | 67.805 | 52.760 | 40.978 |
| 18 | 100.000 | 86.213 | 69.088 | 54.199 | 42.404 |
| 19 | 100.000 | 87.209 | 70.432 | 55.560 | 43.657 |
| 20 | 100.000 | 88.040 | 71.612 | 56.804 | 44.843 |
| 21 | 100.000 | 88.870 | 72.750 | 57.962 | 45.907 |
| 22 | 100.000 | 89.701 | 73.888 | 59.120 | 46.971 |
| 23 | 100.000 | 90.365 | 74.922 | 60.286 | 48.132 |
| 24 | 100.000 | 91.030 | 75.936 | 61.423 | 49.270 |
| 25 | 100.000 | 91.694 | 76.929 | 62.519 | 50.353 |
| 26 | 100.000 | 92.359 | 77.881 | 63.528 | 51.315 |
| 27 | 100.000 | 92.857 | 78.709 | 64.508 | 52.330 |
| 28 | 100.000 | 93.355 | 79.495 | 65.401 | 53.224 |
| 29 | 100.000 | 93.854 | 80.240 | 66.224 | 54.040 |
| 30 | 100.000 | 94.020 | 80.799 | 67.080 | 55.043 |
| 31 | 100.000 | 94.352 | 81.419 | 67.874 | 55.912 |
| 32 | 100.000 | 94.684 | 82.019 | 68.618 | 56.704 |
| 33 | 100.000 | 95.017 | 82.599 | 69.338 | 57.484 |
| 34 | 100.000 | 95.349 | 83.178 | 70.045 | 58.232 |
| 35 | 100.000 | 95.681 | 83.737 | 70.719 | 58.946 |
| 36 | 100.000 | 95.847 | 84.171 | 71.348 | 59.669 |
| 37 | 100.000 | 96.013 | 84.606 | 71.976 | 60.393 |
| 38 | 100.000 | 96.346 | 85.123 | 72.572 | 61.008 |
| 39 | 100.000 | 96.678 | 85.640 | 73.167 | 61.623 |
| 40 | 100.000 | 97.010 | 86.158 | 73.763 | 62.238 |
| 41 | 100.000 | 97.176 | 86.571 | 74.366 | 62.951 |
| 42 | 100.000 | 97.342 | 86.965 | 74.925 | 63.597 |
| 43 | 100.000 | 97.508 | 87.337 | 75.462 | 64.242 |
| 44 | 100.000 | 97.674 | 87.709 | 76.004 | 64.898 |
| 45 | 100.000 | 97.841 | 88.061 | 76.492 | 65.465 |
| 46 | 100.000 | 98.007 | 88.413 | 76.971 | 66.013 |
| 47 | 100.000 | 98.173 | 88.765 | 77.459 | 66.580 |
| 48 | 100.000 | 98.339 | 89.096 | 77.906 | 67.093 |
| 49 | 100.000 | 98.505 | 89.427 | 78.353 | 67.607 |
| 50 | 100.000 | 98.671 | 89.737 | 78.762 | 68.076 |
| 51 | 100.000 | 98.837 | 90.048 | 79.171 | 68.545 |
| 52 | 100.000 | 99.003 | 90.358 | 79.585 | 69.025 |
| 53 | 100.000 | 99.169 | 90.668 | 79.994 | 69.495 |
| 54 | 100.000 | 99.336 | 90.958 | 80.371 | 69.933 |
| 55 | 100.000 | 99.502 | 91.227 | 80.693 | 70.280 |
| 56 | 100.000 | 99.668 | 91.496 | 81.016 | 70.628 |
| 57 | 100.000 | 99.834 | 91.765 | 81.338 | 70.976 |
| 58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |
sed
Generation Strength: 2, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
| 4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
| 5 | 77.500 | 54.319 | 36.706 | 24.881 | 17.185 |
| 6 | 82.500 | 59.468 | 41.237 | 28.594 | 20.103 |
| 7 | 85.000 | 63.289 | 44.920 | 31.700 | 22.596 |
| 8 | 90.000 | 67.774 | 48.624 | 34.723 | 25.040 |
| 9 | 95.000 | 72.591 | 52.555 | 37.824 | 27.473 |
| 10 | 97.500 | 75.415 | 55.204 | 40.165 | 29.480 |
| 11 | 100.000 | 78.073 | 57.645 | 42.319 | 31.342 |
| 12 | 100.000 | 79.070 | 59.197 | 44.039 | 33.007 |
| 13 | 100.000 | 80.233 | 60.852 | 45.776 | 34.638 |
| 14 | 100.000 | 81.561 | 62.632 | 47.550 | 36.238 |
| 15 | 100.000 | 82.724 | 64.246 | 49.208 | 37.770 |
| 16 | 100.000 | 83.721 | 65.673 | 50.717 | 39.200 |
| 17 | 100.000 | 84.718 | 67.080 | 52.190 | 40.586 |
| 18 | 100.000 | 85.714 | 68.487 | 53.662 | 41.972 |
| 19 | 100.000 | 86.545 | 69.708 | 54.985 | 43.256 |
| 20 | 100.000 | 87.209 | 70.764 | 56.192 | 44.472 |
| 21 | 100.000 | 88.206 | 72.088 | 57.511 | 45.671 |
| 22 | 100.000 | 89.037 | 73.246 | 58.715 | 46.802 |
| 23 | 100.000 | 89.867 | 74.384 | 59.873 | 47.866 |
| 24 | 100.000 | 90.698 | 75.522 | 61.030 | 48.929 |
| 25 | 100.000 | 91.196 | 76.350 | 62.019 | 49.965 |
| 26 | 100.000 | 91.528 | 77.033 | 62.924 | 50.967 |
| 27 | 100.000 | 92.027 | 77.840 | 63.871 | 51.948 |
| 28 | 100.000 | 92.691 | 78.792 | 64.880 | 52.910 |
| 29 | 100.000 | 93.189 | 79.578 | 65.790 | 53.848 |
| 30 | 100.000 | 93.522 | 80.219 | 66.621 | 54.761 |
| 31 | 100.000 | 94.020 | 81.006 | 67.514 | 55.654 |
| 32 | 100.000 | 94.352 | 81.626 | 68.304 | 56.513 |
| 33 | 100.000 | 94.850 | 82.371 | 69.127 | 57.329 |
| 34 | 100.000 | 95.183 | 82.930 | 69.818 | 58.086 |
| 35 | 100.000 | 95.349 | 83.364 | 70.446 | 58.810 |
| 36 | 100.000 | 95.515 | 83.799 | 71.075 | 59.533 |
| 37 | 100.000 | 95.681 | 84.192 | 71.645 | 60.212 |
| 38 | 100.000 | 95.847 | 84.585 | 72.204 | 60.857 |
| 39 | 100.000 | 96.013 | 84.978 | 72.762 | 61.503 |
| 40 | 100.000 | 96.179 | 85.371 | 73.320 | 62.148 |
| 41 | 100.000 | 96.512 | 85.889 | 73.916 | 62.764 |
| 42 | 100.000 | 96.844 | 86.406 | 74.511 | 63.379 |
| 43 | 100.000 | 97.176 | 86.923 | 75.106 | 63.994 |
| 44 | 100.000 | 97.508 | 87.441 | 75.702 | 64.609 |
| 45 | 100.000 | 97.841 | 87.958 | 76.297 | 65.225 |
| 46 | 100.000 | 98.007 | 88.310 | 76.789 | 65.805 |
| 47 | 100.000 | 98.173 | 88.661 | 77.277 | 66.372 |
| 48 | 100.000 | 98.339 | 89.013 | 77.765 | 66.940 |
| 49 | 100.000 | 98.505 | 89.365 | 78.245 | 67.487 |
| 50 | 100.000 | 98.671 | 89.696 | 78.692 | 68.000 |
| 51 | 100.000 | 98.837 | 90.027 | 79.138 | 68.514 |
| 52 | 100.000 | 99.003 | 90.337 | 79.552 | 68.994 |
| 53 | 100.000 | 99.169 | 90.648 | 79.961 | 69.463 |
| 54 | 100.000 | 99.336 | 90.958 | 80.371 | 69.933 |
| 55 | 100.000 | 99.502 | 91.227 | 80.710 | 70.324 |
| 56 | 100.000 | 99.668 | 91.496 | 81.032 | 70.672 |
| 57 | 100.000 | 99.834 | 91.765 | 81.355 | 71.020 |
| 58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |
sed
Generation Strength: 2, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 70.000 | 41.694 | 25.347 | 16.123 | 10.733 |
| 4 | 75.000 | 48.505 | 31.181 | 20.543 | 13.968 |
| 5 | 75.000 | 52.658 | 35.837 | 24.472 | 17.008 |
| 6 | 77.500 | 56.312 | 39.623 | 27.838 | 19.773 |
| 7 | 85.000 | 62.791 | 44.527 | 31.506 | 22.539 |
| 8 | 90.000 | 67.608 | 48.500 | 34.677 | 25.050 |
| 9 | 92.500 | 70.764 | 51.500 | 37.282 | 27.227 |
| 10 | 95.000 | 73.588 | 54.149 | 39.623 | 29.235 |
| 11 | 95.000 | 75.083 | 56.218 | 41.732 | 31.157 |
| 12 | 97.500 | 77.741 | 58.680 | 43.923 | 33.062 |
| 13 | 97.500 | 79.070 | 60.501 | 45.776 | 34.761 |
| 14 | 100.000 | 81.728 | 62.880 | 47.823 | 36.501 |
| 15 | 100.000 | 82.724 | 64.349 | 49.411 | 38.029 |
| 16 | 100.000 | 83.721 | 65.798 | 50.961 | 39.514 |
| 17 | 100.000 | 84.551 | 67.101 | 52.417 | 40.935 |
| 18 | 100.000 | 85.548 | 68.529 | 53.926 | 42.365 |
| 19 | 100.000 | 86.545 | 69.915 | 55.357 | 43.696 |
| 20 | 100.000 | 87.209 | 70.929 | 56.494 | 44.834 |
| 21 | 100.000 | 88.040 | 72.108 | 57.739 | 46.020 |
| 22 | 100.000 | 88.704 | 73.102 | 58.835 | 47.103 |
| 23 | 100.000 | 89.037 | 73.743 | 59.666 | 48.016 |
| 24 | 100.000 | 89.535 | 74.529 | 60.576 | 48.954 |
| 25 | 100.000 | 90.365 | 75.667 | 61.733 | 50.018 |
| 26 | 100.000 | 91.196 | 76.805 | 62.891 | 51.082 |
| 27 | 100.000 | 91.694 | 77.571 | 63.760 | 51.964 |
| 28 | 100.000 | 92.193 | 78.336 | 64.628 | 52.847 |
| 29 | 100.000 | 92.857 | 79.288 | 65.637 | 53.809 |
| 30 | 100.000 | 93.355 | 80.074 | 66.530 | 54.702 |
| 31 | 100.000 | 93.854 | 80.861 | 67.423 | 55.596 |
| 32 | 100.000 | 94.186 | 81.440 | 68.143 | 56.377 |
| 33 | 100.000 | 94.518 | 82.019 | 68.862 | 57.157 |
| 34 | 100.000 | 94.850 | 82.619 | 69.607 | 57.949 |
| 35 | 100.000 | 95.017 | 83.054 | 70.235 | 58.672 |
| 36 | 100.000 | 95.183 | 83.426 | 70.773 | 59.318 |
| 37 | 100.000 | 95.515 | 84.006 | 71.480 | 60.066 |
| 38 | 100.000 | 95.847 | 84.564 | 72.154 | 60.779 |
| 39 | 100.000 | 96.013 | 84.958 | 72.712 | 61.425 |
| 40 | 100.000 | 96.179 | 85.351 | 73.270 | 62.071 |
| 41 | 100.000 | 96.346 | 85.744 | 73.829 | 62.716 |
| 42 | 100.000 | 96.512 | 86.096 | 74.317 | 63.284 |
| 43 | 100.000 | 96.844 | 86.613 | 74.912 | 63.899 |
| 44 | 100.000 | 97.176 | 87.130 | 75.508 | 64.515 |
| 45 | 100.000 | 97.508 | 87.647 | 76.103 | 65.130 |
| 46 | 100.000 | 97.841 | 88.165 | 76.699 | 65.745 |
| 47 | 100.000 | 98.173 | 88.682 | 77.294 | 66.360 |
| 48 | 100.000 | 98.339 | 89.013 | 77.749 | 66.894 |
| 49 | 100.000 | 98.505 | 89.365 | 78.229 | 67.441 |
| 50 | 100.000 | 98.671 | 89.696 | 78.675 | 67.954 |
| 51 | 100.000 | 98.837 | 90.006 | 79.089 | 68.435 |
| 52 | 100.000 | 99.003 | 90.317 | 79.498 | 68.904 |
| 53 | 100.000 | 99.169 | 90.627 | 79.907 | 69.373 |
| 54 | 100.000 | 99.336 | 90.937 | 80.317 | 69.843 |
| 55 | 100.000 | 99.502 | 91.227 | 80.693 | 70.280 |
| 56 | 100.000 | 99.668 | 91.496 | 81.032 | 70.672 |
| 57 | 100.000 | 99.834 | 91.765 | 81.355 | 71.020 |
| 58 | 100.000 | 100.000 | 92.034 | 81.677 | 71.367 |
sed
Generation Strength: 3, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
| 4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
| 5 | 85.000 | 56.146 | 36.478 | 24.286 | 16.699 |
| 6 | 85.000 | 60.299 | 41.134 | 28.214 | 19.739 |
| 7 | 90.000 | 65.449 | 45.665 | 31.927 | 22.657 |
| 8 | 92.500 | 69.435 | 49.514 | 35.165 | 25.251 |
| 9 | 95.000 | 72.757 | 52.700 | 37.919 | 27.531 |
| 10 | 97.500 | 75.748 | 55.493 | 40.338 | 29.563 |
| 11 | 100.000 | 78.405 | 57.914 | 42.460 | 31.390 |
| 12 | 100.000 | 79.734 | 59.735 | 44.304 | 33.068 |
| 13 | 100.000 | 81.063 | 61.556 | 46.148 | 34.746 |
| 14 | 100.000 | 82.392 | 63.418 | 48.071 | 36.523 |
| 15 | 100.000 | 83.555 | 65.156 | 49.915 | 38.236 |
| 16 | 100.000 | 84.718 | 66.770 | 51.573 | 39.768 |
| 17 | 100.000 | 85.714 | 68.156 | 53.004 | 41.099 |
| 18 | 100.000 | 86.711 | 69.584 | 54.514 | 42.529 |
| 19 | 100.000 | 87.542 | 70.722 | 55.671 | 43.593 |
| 20 | 100.000 | 88.372 | 72.026 | 57.143 | 45.053 |
| 21 | 100.000 | 89.203 | 73.164 | 58.301 | 46.117 |
| 22 | 100.000 | 90.033 | 74.302 | 59.459 | 47.181 |
| 23 | 100.000 | 90.698 | 75.336 | 60.634 | 48.363 |
| 24 | 100.000 | 91.362 | 76.371 | 61.800 | 49.524 |
| 25 | 100.000 | 92.027 | 77.364 | 62.895 | 50.607 |
| 26 | 100.000 | 92.525 | 78.274 | 64.024 | 51.799 |
| 27 | 100.000 | 93.023 | 79.040 | 64.893 | 52.682 |
| 28 | 100.000 | 93.522 | 79.826 | 65.802 | 53.619 |
| 29 | 100.000 | 94.020 | 80.612 | 66.696 | 54.513 |
| 30 | 100.000 | 94.352 | 81.212 | 67.448 | 55.327 |
| 31 | 100.000 | 94.684 | 81.730 | 68.044 | 55.943 |
| 32 | 100.000 | 95.017 | 82.288 | 68.718 | 56.656 |
| 33 | 100.000 | 95.349 | 82.888 | 69.462 | 57.448 |
| 34 | 100.000 | 95.681 | 83.489 | 70.215 | 58.263 |
| 35 | 100.000 | 96.013 | 84.047 | 70.897 | 58.999 |
| 36 | 100.000 | 96.346 | 84.564 | 71.492 | 59.615 |
| 37 | 100.000 | 96.512 | 84.833 | 71.823 | 59.985 |
| 38 | 100.000 | 96.678 | 85.206 | 72.357 | 60.620 |
| 39 | 100.000 | 96.844 | 85.496 | 72.725 | 61.035 |
| 40 | 100.000 | 97.010 | 85.930 | 73.370 | 61.802 |
| 41 | 100.000 | 97.176 | 86.323 | 73.945 | 62.491 |
| 42 | 100.000 | 97.342 | 86.716 | 74.503 | 63.137 |
| 43 | 100.000 | 97.508 | 87.109 | 75.061 | 63.783 |
| 44 | 100.000 | 97.674 | 87.399 | 75.429 | 64.197 |
| 45 | 100.000 | 97.841 | 87.813 | 76.041 | 64.930 |
| 46 | 100.000 | 98.007 | 88.185 | 76.566 | 65.542 |
| 47 | 100.000 | 98.173 | 88.496 | 76.984 | 66.032 |
| 48 | 100.000 | 98.339 | 88.827 | 77.430 | 66.545 |
| 49 | 100.000 | 98.505 | 89.179 | 77.918 | 67.113 |
| 50 | 100.000 | 98.671 | 89.448 | 78.257 | 67.504 |
| 51 | 100.000 | 98.837 | 89.758 | 78.667 | 67.974 |
| 52 | 100.000 | 99.003 | 90.068 | 79.076 | 68.443 |
| 53 | 100.000 | 99.169 | 90.420 | 79.564 | 69.011 |
| 54 | 100.000 | 99.336 | 90.772 | 80.052 | 69.579 |
| 55 | 100.000 | 99.502 | 91.165 | 80.623 | 70.257 |
| 56 | 100.000 | 99.668 | 91.517 | 81.111 | 70.825 |
| 57 | 100.000 | 99.834 | 91.786 | 81.433 | 71.173 |
| 58 | 100.000 | 100.000 | 92.055 | 81.764 | 71.544 |
| 59 | 100.000 | 100.000 | 92.158 | 81.987 | 71.867 |
| 60 | 100.000 | 100.000 | 92.220 | 82.124 | 72.069 |
| 61 | 100.000 | 100.000 | 92.262 | 82.219 | 72.216 |
| 62 | 100.000 | 100.000 | 92.882 | 83.273 | 73.432 |
| 63 | 100.000 | 100.000 | 93.503 | 84.328 | 74.648 |
| 64 | 100.000 | 100.000 | 93.710 | 84.745 | 75.214 |
| 65 | 100.000 | 100.000 | 93.834 | 85.006 | 75.581 |
| 66 | 100.000 | 100.000 | 93.896 | 85.142 | 75.783 |
| 67 | 100.000 | 100.000 | 94.103 | 85.535 | 76.288 |
| 68 | 100.000 | 100.000 | 94.269 | 85.883 | 76.777 |
| 69 | 100.000 | 100.000 | 94.413 | 86.185 | 77.199 |
| 70 | 100.000 | 100.000 | 94.455 | 86.280 | 77.346 |
| 71 | 100.000 | 100.000 | 94.682 | 86.722 | 77.923 |
| 72 | 100.000 | 100.000 | 94.765 | 86.904 | 78.192 |
| 73 | 100.000 | 100.000 | 94.807 | 86.999 | 78.339 |
| 74 | 100.000 | 100.000 | 94.972 | 87.346 | 78.828 |
| 75 | 100.000 | 100.000 | 95.034 | 87.483 | 79.030 |
| 76 | 100.000 | 100.000 | 95.076 | 87.578 | 79.177 |
| 77 | 100.000 | 100.000 | 95.241 | 87.909 | 79.622 |
| 78 | 100.000 | 100.000 | 95.303 | 88.045 | 79.824 |
| 79 | 100.000 | 100.000 | 95.345 | 88.140 | 79.971 |
| 80 | 100.000 | 100.000 | 95.448 | 88.368 | 80.304 |
| 81 | 100.000 | 100.000 | 95.593 | 88.670 | 80.726 |
| 82 | 100.000 | 100.000 | 95.634 | 88.773 | 80.896 |
| 83 | 100.000 | 100.000 | 95.717 | 88.947 | 81.142 |
| 84 | 100.000 | 100.000 | 95.820 | 89.187 | 81.509 |
| 85 | 100.000 | 100.000 | 95.903 | 89.369 | 81.778 |
| 86 | 100.000 | 100.000 | 95.924 | 89.422 | 81.870 |
| 87 | 100.000 | 100.000 | 95.986 | 89.567 | 82.095 |
| 88 | 100.000 | 100.000 | 96.027 | 89.662 | 82.242 |
| 89 | 100.000 | 100.000 | 96.069 | 89.757 | 82.389 |
| 90 | 100.000 | 100.000 | 96.131 | 89.894 | 82.591 |
| 91 | 100.000 | 100.000 | 96.172 | 89.993 | 82.750 |
| 92 | 100.000 | 100.000 | 96.234 | 90.134 | 82.964 |
| 93 | 100.000 | 100.000 | 96.338 | 90.353 | 83.277 |
| 94 | 100.000 | 100.000 | 96.379 | 90.448 | 83.424 |
| 95 | 100.000 | 100.000 | 96.420 | 90.543 | 83.571 |
| 96 | 100.000 | 100.000 | 96.462 | 90.638 | 83.718 |
| 97 | 100.000 | 100.000 | 96.524 | 90.787 | 83.955 |
| 98 | 100.000 | 100.000 | 96.586 | 90.923 | 84.157 |
| 99 | 100.000 | 100.000 | 96.648 | 91.060 | 84.359 |
| 100 | 100.000 | 100.000 | 96.689 | 91.155 | 84.506 |
| 101 | 100.000 | 100.000 | 96.710 | 91.205 | 84.586 |
| 102 | 100.000 | 100.000 | 96.814 | 91.428 | 84.910 |
| 103 | 100.000 | 100.000 | 96.855 | 91.523 | 85.057 |
| 104 | 100.000 | 100.000 | 96.917 | 91.668 | 85.282 |
| 105 | 100.000 | 100.000 | 96.938 | 91.721 | 85.374 |
| 106 | 100.000 | 100.000 | 96.979 | 91.817 | 85.521 |
| 107 | 100.000 | 100.000 | 97.103 | 92.077 | 85.889 |
| 108 | 100.000 | 100.000 | 97.145 | 92.172 | 86.036 |
| 109 | 100.000 | 100.000 | 97.186 | 92.267 | 86.183 |
| 110 | 100.000 | 100.000 | 97.289 | 92.491 | 86.506 |
| 111 | 100.000 | 100.000 | 97.352 | 92.631 | 86.720 |
| 112 | 100.000 | 100.000 | 97.414 | 92.784 | 86.966 |
| 113 | 100.000 | 100.000 | 97.434 | 92.838 | 87.058 |
| 114 | 100.000 | 100.000 | 97.496 | 92.999 | 87.327 |
| 115 | 100.000 | 100.000 | 97.538 | 93.103 | 87.497 |
| 116 | 100.000 | 100.000 | 97.579 | 93.198 | 87.644 |
| 117 | 100.000 | 100.000 | 97.600 | 93.251 | 87.737 |
| 118 | 100.000 | 100.000 | 97.662 | 93.388 | 87.938 |
| 119 | 100.000 | 100.000 | 97.724 | 93.524 | 88.140 |
| 120 | 100.000 | 100.000 | 97.765 | 93.636 | 88.331 |
| 121 | 100.000 | 100.000 | 97.827 | 93.772 | 88.533 |
| 122 | 100.000 | 100.000 | 97.848 | 93.822 | 88.613 |
| 123 | 100.000 | 100.000 | 97.910 | 93.959 | 88.815 |
| 124 | 100.000 | 100.000 | 97.931 | 94.021 | 88.931 |
| 125 | 100.000 | 100.000 | 98.034 | 94.240 | 89.243 |
| 126 | 100.000 | 100.000 | 98.076 | 94.335 | 89.390 |
| 127 | 100.000 | 100.000 | 98.283 | 94.728 | 89.895 |
| 128 | 100.000 | 100.000 | 98.324 | 94.823 | 90.042 |
| 129 | 100.000 | 100.000 | 98.345 | 94.881 | 90.145 |
| 130 | 100.000 | 100.000 | 98.365 | 94.951 | 90.281 |
| 131 | 100.000 | 100.000 | 98.407 | 95.046 | 90.429 |
| 132 | 100.000 | 100.000 | 98.448 | 95.141 | 90.576 |
| 133 | 100.000 | 100.000 | 98.510 | 95.294 | 90.821 |
| 134 | 100.000 | 100.000 | 98.572 | 95.431 | 91.023 |
| 135 | 100.000 | 100.000 | 98.614 | 95.526 | 91.170 |
| 136 | 100.000 | 100.000 | 98.655 | 95.621 | 91.317 |
| 137 | 100.000 | 100.000 | 98.696 | 95.716 | 91.465 |
| 138 | 100.000 | 100.000 | 98.738 | 95.828 | 91.655 |
| 139 | 100.000 | 100.000 | 98.800 | 95.964 | 91.857 |
| 140 | 100.000 | 100.000 | 98.821 | 96.018 | 91.950 |
| 141 | 100.000 | 100.000 | 98.862 | 96.130 | 92.141 |
| 142 | 100.000 | 100.000 | 98.883 | 96.183 | 92.233 |
| 143 | 100.000 | 100.000 | 98.903 | 96.237 | 92.325 |
| 144 | 100.000 | 100.000 | 98.965 | 96.373 | 92.527 |
| 145 | 100.000 | 100.000 | 99.007 | 96.477 | 92.697 |
| 146 | 100.000 | 100.000 | 99.028 | 96.526 | 92.778 |
| 147 | 100.000 | 100.000 | 99.069 | 96.622 | 92.925 |
| 148 | 100.000 | 100.000 | 99.110 | 96.717 | 93.072 |
| 149 | 100.000 | 100.000 | 99.152 | 96.828 | 93.263 |
| 150 | 100.000 | 100.000 | 99.234 | 97.027 | 93.575 |
| 151 | 100.000 | 100.000 | 99.255 | 97.081 | 93.668 |
| 152 | 100.000 | 100.000 | 99.297 | 97.176 | 93.815 |
| 153 | 100.000 | 100.000 | 99.317 | 97.238 | 93.930 |
| 154 | 100.000 | 100.000 | 99.359 | 97.333 | 94.077 |
| 155 | 100.000 | 100.000 | 99.379 | 97.387 | 94.170 |
| 156 | 100.000 | 100.000 | 99.421 | 97.490 | 94.340 |
| 157 | 100.000 | 100.000 | 99.462 | 97.585 | 94.487 |
| 158 | 100.000 | 100.000 | 99.545 | 97.767 | 94.756 |
| 159 | 100.000 | 100.000 | 99.565 | 97.821 | 94.848 |
| 160 | 100.000 | 100.000 | 99.586 | 97.875 | 94.941 |
| 161 | 100.000 | 100.000 | 99.607 | 97.924 | 95.021 |
| 162 | 100.000 | 100.000 | 99.628 | 97.978 | 95.113 |
| 163 | 100.000 | 100.000 | 99.669 | 98.073 | 95.261 |
| 164 | 100.000 | 100.000 | 99.690 | 98.135 | 95.376 |
| 165 | 100.000 | 100.000 | 99.710 | 98.193 | 95.479 |
| 166 | 100.000 | 100.000 | 99.752 | 98.288 | 95.626 |
| 167 | 100.000 | 100.000 | 99.855 | 98.511 | 95.950 |
| 168 | 100.000 | 100.000 | 99.897 | 98.615 | 96.120 |
| 169 | 100.000 | 100.000 | 99.959 | 98.755 | 96.334 |
| 170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |
sed
Generation Strength: 3, Prioritisation Strength: 3
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
| 4 | 75.000 | 48.837 | 31.450 | 20.692 | 14.035 |
| 5 | 80.000 | 55.648 | 37.285 | 25.113 | 17.270 |
| 6 | 87.500 | 62.458 | 42.541 | 29.045 | 20.207 |
| 7 | 90.000 | 65.781 | 45.976 | 32.147 | 22.801 |
| 8 | 92.500 | 69.435 | 49.555 | 35.235 | 25.329 |
| 9 | 95.000 | 72.425 | 52.473 | 37.841 | 27.542 |
| 10 | 97.500 | 75.249 | 55.121 | 40.181 | 29.550 |
| 11 | 97.500 | 76.910 | 57.397 | 42.472 | 31.608 |
| 12 | 100.000 | 79.568 | 59.797 | 44.556 | 33.392 |
| 13 | 100.000 | 81.063 | 61.825 | 46.595 | 35.236 |
| 14 | 100.000 | 82.558 | 63.811 | 48.555 | 36.982 |
| 15 | 100.000 | 83.555 | 65.529 | 50.490 | 38.825 |
| 16 | 100.000 | 84.884 | 67.308 | 52.264 | 40.426 |
| 17 | 100.000 | 86.047 | 68.881 | 53.852 | 41.880 |
| 18 | 100.000 | 87.043 | 70.308 | 55.361 | 43.310 |
| 19 | 100.000 | 88.040 | 71.695 | 56.800 | 44.662 |
| 20 | 100.000 | 89.037 | 73.081 | 58.231 | 45.993 |
| 21 | 100.000 | 89.867 | 74.281 | 59.513 | 47.222 |
| 22 | 100.000 | 90.698 | 75.419 | 60.671 | 48.286 |
| 23 | 100.000 | 91.196 | 76.329 | 61.808 | 49.498 |
| 24 | 100.000 | 91.860 | 77.323 | 62.904 | 50.582 |
| 25 | 100.000 | 91.860 | 77.943 | 63.958 | 51.798 |
| 26 | 100.000 | 92.359 | 78.771 | 64.938 | 52.813 |
| 27 | 100.000 | 92.857 | 79.599 | 65.918 | 53.828 |
| 28 | 100.000 | 93.355 | 80.426 | 66.898 | 54.843 |
| 29 | 100.000 | 93.854 | 81.233 | 67.853 | 55.845 |
| 30 | 100.000 | 94.186 | 81.937 | 68.796 | 56.891 |
| 31 | 100.000 | 94.518 | 82.599 | 69.661 | 57.838 |
| 32 | 100.000 | 94.850 | 83.220 | 70.454 | 58.708 |
| 33 | 100.000 | 95.183 | 83.840 | 71.253 | 59.587 |
| 34 | 100.000 | 95.515 | 84.440 | 72.009 | 60.412 |
| 35 | 100.000 | 95.847 | 84.999 | 72.683 | 61.126 |
| 36 | 100.000 | 96.013 | 85.433 | 73.328 | 61.893 |
| 37 | 100.000 | 96.179 | 85.868 | 73.957 | 62.617 |
| 38 | 100.000 | 96.346 | 86.303 | 74.585 | 63.340 |
| 39 | 100.000 | 96.512 | 86.737 | 75.231 | 64.107 |
| 40 | 100.000 | 96.678 | 87.172 | 75.871 | 64.864 |
| 41 | 100.000 | 96.844 | 87.565 | 76.430 | 65.509 |
| 42 | 100.000 | 97.010 | 87.958 | 76.988 | 66.155 |
| 43 | 100.000 | 97.176 | 88.351 | 77.546 | 66.800 |
| 44 | 100.000 | 97.342 | 88.703 | 78.034 | 67.368 |
| 45 | 100.000 | 97.508 | 89.054 | 78.522 | 67.936 |
| 46 | 100.000 | 97.674 | 89.406 | 79.010 | 68.504 |
| 47 | 100.000 | 97.841 | 89.758 | 79.502 | 69.084 |
| 48 | 100.000 | 98.007 | 90.110 | 79.990 | 69.652 |
| 49 | 100.000 | 98.173 | 90.461 | 80.478 | 70.219 |
| 50 | 100.000 | 98.339 | 90.813 | 80.966 | 70.787 |
| 51 | 100.000 | 98.505 | 91.165 | 81.454 | 71.355 |
| 52 | 100.000 | 98.671 | 91.496 | 81.901 | 71.868 |
| 53 | 100.000 | 98.837 | 91.827 | 82.347 | 72.381 |
| 54 | 100.000 | 99.003 | 92.158 | 82.810 | 72.938 |
| 55 | 100.000 | 99.169 | 92.489 | 83.265 | 73.474 |
| 56 | 100.000 | 99.336 | 92.820 | 83.716 | 73.998 |
| 57 | 100.000 | 99.502 | 93.131 | 84.125 | 74.468 |
| 58 | 100.000 | 99.668 | 93.441 | 84.543 | 74.958 |
| 59 | 100.000 | 99.834 | 93.751 | 84.952 | 75.427 |
| 60 | 100.000 | 100.000 | 94.062 | 85.370 | 75.917 |
| 61 | 100.000 | 100.000 | 94.269 | 85.763 | 76.422 |
| 62 | 100.000 | 100.000 | 94.475 | 86.156 | 76.926 |
| 63 | 100.000 | 100.000 | 94.682 | 86.548 | 77.431 |
| 64 | 100.000 | 100.000 | 94.827 | 86.859 | 77.876 |
| 65 | 100.000 | 100.000 | 94.972 | 87.169 | 78.321 |
| 66 | 100.000 | 100.000 | 95.096 | 87.429 | 78.688 |
| 67 | 100.000 | 100.000 | 95.220 | 87.690 | 79.055 |
| 68 | 100.000 | 100.000 | 95.345 | 87.950 | 79.422 |
| 69 | 100.000 | 100.000 | 95.448 | 88.178 | 79.758 |
| 70 | 100.000 | 100.000 | 95.551 | 88.401 | 80.081 |
| 71 | 100.000 | 100.000 | 95.655 | 88.624 | 80.405 |
| 72 | 100.000 | 100.000 | 95.758 | 88.848 | 80.728 |
| 73 | 100.000 | 100.000 | 95.841 | 89.042 | 81.030 |
| 74 | 100.000 | 100.000 | 95.924 | 89.224 | 81.299 |
| 75 | 100.000 | 100.000 | 96.007 | 89.406 | 81.567 |
| 76 | 100.000 | 100.000 | 96.089 | 89.604 | 81.880 |
| 77 | 100.000 | 100.000 | 96.172 | 89.786 | 82.148 |
| 78 | 100.000 | 100.000 | 96.255 | 89.968 | 82.417 |
| 79 | 100.000 | 100.000 | 96.317 | 90.105 | 82.619 |
| 80 | 100.000 | 100.000 | 96.379 | 90.245 | 82.833 |
| 81 | 100.000 | 100.000 | 96.441 | 90.382 | 83.035 |
| 82 | 100.000 | 100.000 | 96.503 | 90.522 | 83.249 |
| 83 | 100.000 | 100.000 | 96.565 | 90.667 | 83.474 |
| 84 | 100.000 | 100.000 | 96.627 | 90.820 | 83.719 |
| 85 | 100.000 | 100.000 | 96.689 | 90.965 | 83.944 |
| 86 | 100.000 | 100.000 | 96.752 | 91.114 | 84.179 |
| 87 | 100.000 | 100.000 | 96.814 | 91.267 | 84.425 |
| 88 | 100.000 | 100.000 | 96.876 | 91.403 | 84.626 |
| 89 | 100.000 | 100.000 | 96.938 | 91.540 | 84.828 |
| 90 | 100.000 | 100.000 | 97.000 | 91.676 | 85.030 |
| 91 | 100.000 | 100.000 | 97.062 | 91.812 | 85.232 |
| 92 | 100.000 | 100.000 | 97.124 | 91.949 | 85.434 |
| 93 | 100.000 | 100.000 | 97.186 | 92.085 | 85.636 |
| 94 | 100.000 | 100.000 | 97.248 | 92.222 | 85.837 |
| 95 | 100.000 | 100.000 | 97.310 | 92.358 | 86.039 |
| 96 | 100.000 | 100.000 | 97.372 | 92.495 | 86.241 |
| 97 | 100.000 | 100.000 | 97.434 | 92.631 | 86.443 |
| 98 | 100.000 | 100.000 | 97.496 | 92.768 | 86.645 |
| 99 | 100.000 | 100.000 | 97.558 | 92.904 | 86.847 |
| 100 | 100.000 | 100.000 | 97.621 | 93.041 | 87.048 |
| 101 | 100.000 | 100.000 | 97.683 | 93.177 | 87.250 |
| 102 | 100.000 | 100.000 | 97.745 | 93.313 | 87.452 |
| 103 | 100.000 | 100.000 | 97.807 | 93.466 | 87.698 |
| 104 | 100.000 | 100.000 | 97.869 | 93.603 | 87.900 |
| 105 | 100.000 | 100.000 | 97.931 | 93.739 | 88.101 |
| 106 | 100.000 | 100.000 | 97.993 | 93.876 | 88.303 |
| 107 | 100.000 | 100.000 | 98.055 | 94.012 | 88.505 |
| 108 | 100.000 | 100.000 | 98.117 | 94.149 | 88.707 |
| 109 | 100.000 | 100.000 | 98.158 | 94.244 | 88.854 |
| 110 | 100.000 | 100.000 | 98.200 | 94.347 | 89.024 |
| 111 | 100.000 | 100.000 | 98.241 | 94.447 | 89.184 |
| 112 | 100.000 | 100.000 | 98.283 | 94.542 | 89.331 |
| 113 | 100.000 | 100.000 | 98.324 | 94.653 | 89.522 |
| 114 | 100.000 | 100.000 | 98.365 | 94.765 | 89.712 |
| 115 | 100.000 | 100.000 | 98.407 | 94.868 | 89.883 |
| 116 | 100.000 | 100.000 | 98.448 | 94.963 | 90.030 |
| 117 | 100.000 | 100.000 | 98.490 | 95.075 | 90.221 |
| 118 | 100.000 | 100.000 | 98.531 | 95.170 | 90.368 |
| 119 | 100.000 | 100.000 | 98.572 | 95.265 | 90.515 |
| 120 | 100.000 | 100.000 | 98.614 | 95.360 | 90.662 |
| 121 | 100.000 | 100.000 | 98.655 | 95.455 | 90.809 |
| 122 | 100.000 | 100.000 | 98.696 | 95.559 | 90.979 |
| 123 | 100.000 | 100.000 | 98.738 | 95.654 | 91.127 |
| 124 | 100.000 | 100.000 | 98.779 | 95.749 | 91.274 |
| 125 | 100.000 | 100.000 | 98.821 | 95.844 | 91.421 |
| 126 | 100.000 | 100.000 | 98.862 | 95.943 | 91.580 |
| 127 | 100.000 | 100.000 | 98.903 | 96.047 | 91.750 |
| 128 | 100.000 | 100.000 | 98.945 | 96.142 | 91.897 |
| 129 | 100.000 | 100.000 | 98.986 | 96.241 | 92.057 |
| 130 | 100.000 | 100.000 | 99.028 | 96.336 | 92.204 |
| 131 | 100.000 | 100.000 | 99.069 | 96.431 | 92.351 |
| 132 | 100.000 | 100.000 | 99.110 | 96.526 | 92.498 |
| 133 | 100.000 | 100.000 | 99.152 | 96.622 | 92.645 |
| 134 | 100.000 | 100.000 | 99.193 | 96.717 | 92.792 |
| 135 | 100.000 | 100.000 | 99.234 | 96.812 | 92.939 |
| 136 | 100.000 | 100.000 | 99.276 | 96.907 | 93.087 |
| 137 | 100.000 | 100.000 | 99.317 | 97.019 | 93.277 |
| 138 | 100.000 | 100.000 | 99.338 | 97.089 | 93.414 |
| 139 | 100.000 | 100.000 | 99.359 | 97.151 | 93.529 |
| 140 | 100.000 | 100.000 | 99.379 | 97.213 | 93.645 |
| 141 | 100.000 | 100.000 | 99.400 | 97.275 | 93.760 |
| 142 | 100.000 | 100.000 | 99.421 | 97.337 | 93.876 |
| 143 | 100.000 | 100.000 | 99.441 | 97.391 | 93.968 |
| 144 | 100.000 | 100.000 | 99.462 | 97.440 | 94.048 |
| 145 | 100.000 | 100.000 | 99.483 | 97.498 | 94.153 |
| 146 | 100.000 | 100.000 | 99.503 | 97.556 | 94.256 |
| 147 | 100.000 | 100.000 | 99.524 | 97.614 | 94.360 |
| 148 | 100.000 | 100.000 | 99.545 | 97.664 | 94.440 |
| 149 | 100.000 | 100.000 | 99.565 | 97.726 | 94.555 |
| 150 | 100.000 | 100.000 | 99.586 | 97.775 | 94.636 |
| 151 | 100.000 | 100.000 | 99.607 | 97.829 | 94.728 |
| 152 | 100.000 | 100.000 | 99.628 | 97.879 | 94.808 |
| 153 | 100.000 | 100.000 | 99.648 | 97.932 | 94.901 |
| 154 | 100.000 | 100.000 | 99.669 | 97.982 | 94.981 |
| 155 | 100.000 | 100.000 | 99.690 | 98.036 | 95.073 |
| 156 | 100.000 | 100.000 | 99.710 | 98.090 | 95.166 |
| 157 | 100.000 | 100.000 | 99.731 | 98.139 | 95.246 |
| 158 | 100.000 | 100.000 | 99.752 | 98.193 | 95.338 |
| 159 | 100.000 | 100.000 | 99.772 | 98.247 | 95.431 |
| 160 | 100.000 | 100.000 | 99.793 | 98.300 | 95.523 |
| 161 | 100.000 | 100.000 | 99.814 | 98.362 | 95.639 |
| 162 | 100.000 | 100.000 | 99.834 | 98.416 | 95.731 |
| 163 | 100.000 | 100.000 | 99.855 | 98.470 | 95.823 |
| 164 | 100.000 | 100.000 | 99.876 | 98.524 | 95.916 |
| 165 | 100.000 | 100.000 | 99.897 | 98.573 | 95.996 |
| 166 | 100.000 | 100.000 | 99.917 | 98.627 | 96.089 |
| 167 | 100.000 | 100.000 | 99.938 | 98.681 | 96.181 |
| 168 | 100.000 | 100.000 | 99.959 | 98.743 | 96.296 |
| 169 | 100.000 | 100.000 | 99.979 | 98.797 | 96.389 |
| 170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |
sed
Generation Strength: 3, Prioritisation Strength: 4
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 43.023 | 25.926 | 16.354 | 10.818 |
| 4 | 80.000 | 50.166 | 31.533 | 20.552 | 13.924 |
| 5 | 80.000 | 54.319 | 36.189 | 24.480 | 16.964 |
| 6 | 85.000 | 60.133 | 41.196 | 28.417 | 19.960 |
| 7 | 87.500 | 64.452 | 45.438 | 31.973 | 22.769 |
| 8 | 92.500 | 69.435 | 49.576 | 35.260 | 25.347 |
| 9 | 95.000 | 72.591 | 52.576 | 37.865 | 27.525 |
| 10 | 97.500 | 75.581 | 55.411 | 40.355 | 29.635 |
| 11 | 100.000 | 78.405 | 58.059 | 42.695 | 31.642 |
| 12 | 100.000 | 79.568 | 59.921 | 44.717 | 33.528 |
| 13 | 100.000 | 80.731 | 61.701 | 46.653 | 35.365 |
| 14 | 100.000 | 82.226 | 63.687 | 48.613 | 37.111 |
| 15 | 100.000 | 83.056 | 65.115 | 50.271 | 38.753 |
| 16 | 100.000 | 84.385 | 66.915 | 52.082 | 40.396 |
| 17 | 100.000 | 85.548 | 68.487 | 53.670 | 41.851 |
| 18 | 100.000 | 86.711 | 70.039 | 55.241 | 43.305 |
| 19 | 100.000 | 87.874 | 71.612 | 56.829 | 44.759 |
| 20 | 100.000 | 87.874 | 72.233 | 57.884 | 45.975 |
| 21 | 100.000 | 88.704 | 73.453 | 59.207 | 47.259 |
| 22 | 100.000 | 89.535 | 74.674 | 60.530 | 48.543 |
| 23 | 100.000 | 90.199 | 75.729 | 61.742 | 49.768 |
| 24 | 100.000 | 91.030 | 76.888 | 62.929 | 50.858 |
| 25 | 100.000 | 91.528 | 77.736 | 63.946 | 51.917 |
| 26 | 100.000 | 92.027 | 78.585 | 64.963 | 52.976 |
| 27 | 100.000 | 92.359 | 79.247 | 65.844 | 53.962 |
| 28 | 100.000 | 92.857 | 80.033 | 66.754 | 54.899 |
| 29 | 100.000 | 93.355 | 80.819 | 67.663 | 55.837 |
| 30 | 100.000 | 93.688 | 81.440 | 68.465 | 56.727 |
| 31 | 100.000 | 94.020 | 82.061 | 69.259 | 57.596 |
| 32 | 100.000 | 94.352 | 82.682 | 70.053 | 58.466 |
| 33 | 100.000 | 94.684 | 83.282 | 70.806 | 59.280 |
| 34 | 100.000 | 94.850 | 83.716 | 71.451 | 60.047 |
| 35 | 100.000 | 95.017 | 84.151 | 72.096 | 60.815 |
| 36 | 100.000 | 95.183 | 84.585 | 72.741 | 61.582 |
| 37 | 100.000 | 95.515 | 85.144 | 73.415 | 62.296 |
| 38 | 100.000 | 95.847 | 85.702 | 74.089 | 63.009 |
| 39 | 100.000 | 96.013 | 86.116 | 74.685 | 63.699 |
| 40 | 100.000 | 96.179 | 86.509 | 75.251 | 64.367 |
| 41 | 100.000 | 96.512 | 87.047 | 75.888 | 65.037 |
| 42 | 100.000 | 96.844 | 87.585 | 76.525 | 65.707 |
| 43 | 100.000 | 97.176 | 88.123 | 77.162 | 66.377 |
| 44 | 100.000 | 97.342 | 88.516 | 77.720 | 67.023 |
| 45 | 100.000 | 97.508 | 88.910 | 78.278 | 67.669 |
| 46 | 100.000 | 97.674 | 89.303 | 78.836 | 68.314 |
| 47 | 100.000 | 97.841 | 89.696 | 79.395 | 68.960 |
| 48 | 100.000 | 98.007 | 90.048 | 79.891 | 69.551 |
| 49 | 100.000 | 98.173 | 90.399 | 80.379 | 70.119 |
| 50 | 100.000 | 98.339 | 90.751 | 80.867 | 70.686 |
| 51 | 100.000 | 98.505 | 91.082 | 81.322 | 71.220 |
| 52 | 100.000 | 98.671 | 91.413 | 81.776 | 71.754 |
| 53 | 100.000 | 98.837 | 91.744 | 82.231 | 72.290 |
| 54 | 100.000 | 99.003 | 92.075 | 82.686 | 72.826 |
| 55 | 100.000 | 99.169 | 92.406 | 83.141 | 73.363 |
| 56 | 100.000 | 99.336 | 92.737 | 83.596 | 73.896 |
| 57 | 100.000 | 99.336 | 92.944 | 83.989 | 74.401 |
| 58 | 100.000 | 99.336 | 93.151 | 84.382 | 74.905 |
| 59 | 100.000 | 99.336 | 93.358 | 84.774 | 75.410 |
| 60 | 100.000 | 99.502 | 93.689 | 85.221 | 75.923 |
| 61 | 100.000 | 99.668 | 94.020 | 85.668 | 76.436 |
| 62 | 100.000 | 99.834 | 94.331 | 86.089 | 76.938 |
| 63 | 100.000 | 100.000 | 94.641 | 86.499 | 77.408 |
| 64 | 100.000 | 100.000 | 94.765 | 86.780 | 77.830 |
| 65 | 100.000 | 100.000 | 94.889 | 87.057 | 78.241 |
| 66 | 100.000 | 100.000 | 95.013 | 87.326 | 78.631 |
| 67 | 100.000 | 100.000 | 95.138 | 87.586 | 78.998 |
| 68 | 100.000 | 100.000 | 95.241 | 87.814 | 79.334 |
| 69 | 100.000 | 100.000 | 95.345 | 88.037 | 79.657 |
| 70 | 100.000 | 100.000 | 95.448 | 88.260 | 79.981 |
| 71 | 100.000 | 100.000 | 95.551 | 88.484 | 80.304 |
| 72 | 100.000 | 100.000 | 95.655 | 88.707 | 80.627 |
| 73 | 100.000 | 100.000 | 95.758 | 88.926 | 80.940 |
| 74 | 100.000 | 100.000 | 95.841 | 89.112 | 81.221 |
| 75 | 100.000 | 100.000 | 95.924 | 89.298 | 81.502 |
| 76 | 100.000 | 100.000 | 96.007 | 89.480 | 81.770 |
| 77 | 100.000 | 100.000 | 96.089 | 89.662 | 82.039 |
| 78 | 100.000 | 100.000 | 96.172 | 89.844 | 82.308 |
| 79 | 100.000 | 100.000 | 96.255 | 90.026 | 82.576 |
| 80 | 100.000 | 100.000 | 96.338 | 90.208 | 82.845 |
| 81 | 100.000 | 100.000 | 96.420 | 90.390 | 83.114 |
| 82 | 100.000 | 100.000 | 96.483 | 90.543 | 83.359 |
| 83 | 100.000 | 100.000 | 96.545 | 90.696 | 83.605 |
| 84 | 100.000 | 100.000 | 96.607 | 90.849 | 83.851 |
| 85 | 100.000 | 100.000 | 96.669 | 90.998 | 84.085 |
| 86 | 100.000 | 100.000 | 96.731 | 91.143 | 84.310 |
| 87 | 100.000 | 100.000 | 96.793 | 91.287 | 84.535 |
| 88 | 100.000 | 100.000 | 96.855 | 91.432 | 84.760 |
| 89 | 100.000 | 100.000 | 96.917 | 91.577 | 84.985 |
| 90 | 100.000 | 100.000 | 96.979 | 91.717 | 85.199 |
| 91 | 100.000 | 100.000 | 97.041 | 91.858 | 85.413 |
| 92 | 100.000 | 100.000 | 97.103 | 91.994 | 85.615 |
| 93 | 100.000 | 100.000 | 97.165 | 92.131 | 85.817 |
| 94 | 100.000 | 100.000 | 97.227 | 92.267 | 86.019 |
| 95 | 100.000 | 100.000 | 97.289 | 92.404 | 86.220 |
| 96 | 100.000 | 100.000 | 97.352 | 92.540 | 86.422 |
| 97 | 100.000 | 100.000 | 97.414 | 92.677 | 86.624 |
| 98 | 100.000 | 100.000 | 97.476 | 92.813 | 86.826 |
| 99 | 100.000 | 100.000 | 97.538 | 92.950 | 87.028 |
| 100 | 100.000 | 100.000 | 97.600 | 93.086 | 87.230 |
| 101 | 100.000 | 100.000 | 97.662 | 93.223 | 87.431 |
| 102 | 100.000 | 100.000 | 97.724 | 93.359 | 87.633 |
| 103 | 100.000 | 100.000 | 97.786 | 93.495 | 87.835 |
| 104 | 100.000 | 100.000 | 97.848 | 93.632 | 88.037 |
| 105 | 100.000 | 100.000 | 97.910 | 93.768 | 88.239 |
| 106 | 100.000 | 100.000 | 97.972 | 93.905 | 88.441 |
| 107 | 100.000 | 100.000 | 98.034 | 94.041 | 88.642 |
| 108 | 100.000 | 100.000 | 98.076 | 94.153 | 88.833 |
| 109 | 100.000 | 100.000 | 98.117 | 94.265 | 89.024 |
| 110 | 100.000 | 100.000 | 98.158 | 94.368 | 89.194 |
| 111 | 100.000 | 100.000 | 98.200 | 94.471 | 89.365 |
| 112 | 100.000 | 100.000 | 98.241 | 94.575 | 89.535 |
| 113 | 100.000 | 100.000 | 98.283 | 94.678 | 89.705 |
| 114 | 100.000 | 100.000 | 98.324 | 94.781 | 89.875 |
| 115 | 100.000 | 100.000 | 98.365 | 94.877 | 90.022 |
| 116 | 100.000 | 100.000 | 98.407 | 94.972 | 90.170 |
| 117 | 100.000 | 100.000 | 98.448 | 95.067 | 90.317 |
| 118 | 100.000 | 100.000 | 98.490 | 95.162 | 90.464 |
| 119 | 100.000 | 100.000 | 98.531 | 95.257 | 90.611 |
| 120 | 100.000 | 100.000 | 98.572 | 95.352 | 90.758 |
| 121 | 100.000 | 100.000 | 98.614 | 95.447 | 90.905 |
| 122 | 100.000 | 100.000 | 98.655 | 95.542 | 91.052 |
| 123 | 100.000 | 100.000 | 98.696 | 95.637 | 91.199 |
| 124 | 100.000 | 100.000 | 98.738 | 95.733 | 91.347 |
| 125 | 100.000 | 100.000 | 98.779 | 95.828 | 91.494 |
| 126 | 100.000 | 100.000 | 98.821 | 95.923 | 91.641 |
| 127 | 100.000 | 100.000 | 98.862 | 96.018 | 91.788 |
| 128 | 100.000 | 100.000 | 98.903 | 96.113 | 91.935 |
| 129 | 100.000 | 100.000 | 98.945 | 96.208 | 92.082 |
| 130 | 100.000 | 100.000 | 98.986 | 96.303 | 92.229 |
| 131 | 100.000 | 100.000 | 99.028 | 96.398 | 92.376 |
| 132 | 100.000 | 100.000 | 99.069 | 96.493 | 92.524 |
| 133 | 100.000 | 100.000 | 99.110 | 96.589 | 92.671 |
| 134 | 100.000 | 100.000 | 99.152 | 96.684 | 92.818 |
| 135 | 100.000 | 100.000 | 99.193 | 96.779 | 92.965 |
| 136 | 100.000 | 100.000 | 99.234 | 96.874 | 93.112 |
| 137 | 100.000 | 100.000 | 99.276 | 96.969 | 93.259 |
| 138 | 100.000 | 100.000 | 99.317 | 97.064 | 93.406 |
| 139 | 100.000 | 100.000 | 99.359 | 97.159 | 93.553 |
| 140 | 100.000 | 100.000 | 99.379 | 97.221 | 93.669 |
| 141 | 100.000 | 100.000 | 99.400 | 97.283 | 93.784 |
| 142 | 100.000 | 100.000 | 99.421 | 97.345 | 93.900 |
| 143 | 100.000 | 100.000 | 99.441 | 97.407 | 94.015 |
| 144 | 100.000 | 100.000 | 99.462 | 97.469 | 94.131 |
| 145 | 100.000 | 100.000 | 99.483 | 97.527 | 94.234 |
| 146 | 100.000 | 100.000 | 99.503 | 97.585 | 94.338 |
| 147 | 100.000 | 100.000 | 99.524 | 97.643 | 94.441 |
| 148 | 100.000 | 100.000 | 99.545 | 97.697 | 94.533 |
| 149 | 100.000 | 100.000 | 99.565 | 97.750 | 94.626 |
| 150 | 100.000 | 100.000 | 99.586 | 97.804 | 94.718 |
| 151 | 100.000 | 100.000 | 99.607 | 97.858 | 94.811 |
| 152 | 100.000 | 100.000 | 99.628 | 97.912 | 94.903 |
| 153 | 100.000 | 100.000 | 99.648 | 97.966 | 94.995 |
| 154 | 100.000 | 100.000 | 99.669 | 98.019 | 95.088 |
| 155 | 100.000 | 100.000 | 99.690 | 98.073 | 95.180 |
| 156 | 100.000 | 100.000 | 99.710 | 98.127 | 95.273 |
| 157 | 100.000 | 100.000 | 99.731 | 98.181 | 95.365 |
| 158 | 100.000 | 100.000 | 99.752 | 98.234 | 95.457 |
| 159 | 100.000 | 100.000 | 99.772 | 98.288 | 95.550 |
| 160 | 100.000 | 100.000 | 99.793 | 98.342 | 95.642 |
| 161 | 100.000 | 100.000 | 99.814 | 98.396 | 95.735 |
| 162 | 100.000 | 100.000 | 99.834 | 98.449 | 95.827 |
| 163 | 100.000 | 100.000 | 99.855 | 98.503 | 95.920 |
| 164 | 100.000 | 100.000 | 99.876 | 98.553 | 96.000 |
| 165 | 100.000 | 100.000 | 99.897 | 98.602 | 96.080 |
| 166 | 100.000 | 100.000 | 99.917 | 98.652 | 96.160 |
| 167 | 100.000 | 100.000 | 99.938 | 98.702 | 96.241 |
| 168 | 100.000 | 100.000 | 99.959 | 98.751 | 96.321 |
| 169 | 100.000 | 100.000 | 99.979 | 98.801 | 96.401 |
| 170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |
sed
Generation Strength: 3, Prioritisation Strength: 5
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 55.000 | 30.233 | 17.670 | 10.999 | 7.234 |
| 3 | 70.000 | 41.694 | 25.347 | 16.123 | 10.733 |
| 4 | 77.500 | 49.169 | 31.264 | 20.523 | 13.950 |
| 5 | 80.000 | 54.817 | 36.665 | 24.798 | 17.143 |
| 6 | 82.500 | 59.302 | 41.092 | 28.503 | 20.053 |
| 7 | 85.000 | 63.455 | 45.169 | 31.944 | 22.794 |
| 8 | 92.500 | 69.934 | 50.052 | 35.579 | 25.526 |
| 9 | 95.000 | 72.924 | 52.886 | 38.068 | 27.636 |
| 10 | 97.500 | 75.748 | 55.535 | 40.409 | 29.643 |
| 11 | 97.500 | 77.243 | 57.604 | 42.517 | 31.565 |
| 12 | 100.000 | 79.734 | 59.963 | 44.684 | 33.488 |
| 13 | 100.000 | 80.565 | 61.556 | 46.557 | 35.307 |
| 14 | 100.000 | 81.561 | 63.170 | 48.365 | 37.050 |
| 15 | 100.000 | 82.724 | 64.825 | 50.114 | 38.706 |
| 16 | 100.000 | 84.053 | 66.646 | 51.958 | 40.384 |
| 17 | 100.000 | 85.216 | 68.260 | 53.624 | 41.937 |
| 18 | 100.000 | 86.545 | 70.039 | 55.398 | 43.537 |
| 19 | 100.000 | 87.708 | 71.633 | 57.024 | 45.035 |
| 20 | 100.000 | 88.538 | 72.874 | 58.388 | 46.374 |
| 21 | 100.000 | 88.538 | 73.495 | 59.443 | 47.590 |
| 22 | 100.000 | 89.203 | 74.550 | 60.650 | 48.805 |
| 23 | 100.000 | 89.701 | 75.378 | 61.655 | 49.875 |
| 24 | 100.000 | 90.698 | 76.723 | 63.015 | 51.129 |
| 25 | 100.000 | 91.196 | 77.550 | 64.012 | 52.186 |
| 26 | 100.000 | 91.694 | 78.399 | 65.029 | 53.245 |
| 27 | 100.000 | 92.193 | 79.205 | 65.976 | 54.226 |
| 28 | 100.000 | 92.359 | 79.681 | 66.691 | 55.071 |
| 29 | 100.000 | 92.525 | 80.157 | 67.407 | 55.916 |
| 30 | 100.000 | 93.355 | 81.295 | 68.565 | 56.980 |
| 31 | 100.000 | 93.688 | 81.895 | 69.330 | 57.824 |
| 32 | 100.000 | 93.854 | 82.330 | 69.975 | 58.591 |
| 33 | 100.000 | 94.020 | 82.764 | 70.620 | 59.358 |
| 34 | 100.000 | 94.352 | 83.364 | 71.377 | 60.184 |
| 35 | 100.000 | 94.850 | 84.109 | 72.208 | 61.023 |
| 36 | 100.000 | 95.183 | 84.668 | 72.898 | 61.780 |
| 37 | 100.000 | 95.349 | 85.082 | 73.502 | 62.493 |
| 38 | 100.000 | 95.681 | 85.682 | 74.246 | 63.284 |
| 39 | 100.000 | 96.013 | 86.282 | 74.991 | 64.076 |
| 40 | 100.000 | 96.346 | 86.840 | 75.673 | 64.812 |
| 41 | 100.000 | 96.346 | 87.068 | 76.128 | 65.423 |
| 42 | 100.000 | 96.512 | 87.441 | 76.661 | 66.058 |
| 43 | 100.000 | 96.678 | 87.834 | 77.220 | 66.703 |
| 44 | 100.000 | 96.844 | 88.227 | 77.778 | 67.349 |
| 45 | 100.000 | 97.010 | 88.620 | 78.336 | 67.994 |
| 46 | 100.000 | 97.176 | 89.013 | 78.894 | 68.640 |
| 47 | 100.000 | 97.342 | 89.385 | 79.419 | 69.254 |
| 48 | 100.000 | 97.342 | 89.551 | 79.775 | 69.764 |
| 49 | 100.000 | 97.674 | 90.089 | 80.412 | 70.433 |
| 50 | 100.000 | 97.841 | 90.461 | 80.937 | 71.045 |
| 51 | 100.000 | 98.007 | 90.813 | 81.429 | 71.625 |
| 52 | 100.000 | 98.339 | 91.330 | 82.033 | 72.261 |
| 53 | 100.000 | 98.671 | 91.848 | 82.628 | 72.876 |
| 54 | 100.000 | 98.837 | 92.179 | 83.083 | 73.410 |
| 55 | 100.000 | 99.003 | 92.510 | 83.538 | 73.944 |
| 56 | 100.000 | 99.003 | 92.717 | 83.931 | 74.448 |
| 57 | 100.000 | 99.003 | 92.924 | 84.324 | 74.953 |
| 58 | 100.000 | 99.003 | 93.131 | 84.717 | 75.457 |
| 59 | 100.000 | 99.169 | 93.462 | 85.163 | 75.971 |
| 60 | 100.000 | 99.336 | 93.772 | 85.581 | 76.461 |
| 61 | 100.000 | 99.502 | 94.082 | 85.994 | 76.942 |
| 62 | 100.000 | 99.502 | 94.227 | 86.296 | 77.364 |
| 63 | 100.000 | 99.668 | 94.538 | 86.706 | 77.833 |
| 64 | 100.000 | 99.834 | 94.848 | 87.115 | 78.303 |
| 65 | 100.000 | 99.834 | 94.972 | 87.380 | 78.682 |
| 66 | 100.000 | 99.834 | 95.117 | 87.673 | 79.083 |
| 67 | 100.000 | 99.834 | 95.241 | 87.934 | 79.450 |
| 68 | 100.000 | 99.834 | 95.345 | 88.157 | 79.774 |
| 69 | 100.000 | 100.000 | 95.613 | 88.488 | 80.145 |
| 70 | 100.000 | 100.000 | 95.717 | 88.707 | 80.457 |
| 71 | 100.000 | 100.000 | 95.820 | 88.926 | 80.770 |
| 72 | 100.000 | 100.000 | 95.903 | 89.108 | 81.038 |
| 73 | 100.000 | 100.000 | 95.965 | 89.261 | 81.285 |
| 74 | 100.000 | 100.000 | 96.048 | 89.443 | 81.554 |
| 75 | 100.000 | 100.000 | 96.131 | 89.625 | 81.823 |
| 76 | 100.000 | 100.000 | 96.193 | 89.778 | 82.068 |
| 77 | 100.000 | 100.000 | 96.255 | 89.931 | 82.314 |
| 78 | 100.000 | 100.000 | 96.317 | 90.084 | 82.559 |
| 79 | 100.000 | 100.000 | 96.379 | 90.237 | 82.805 |
| 80 | 100.000 | 100.000 | 96.441 | 90.382 | 83.030 |
| 81 | 100.000 | 100.000 | 96.503 | 90.526 | 83.255 |
| 82 | 100.000 | 100.000 | 96.565 | 90.671 | 83.480 |
| 83 | 100.000 | 100.000 | 96.627 | 90.816 | 83.705 |
| 84 | 100.000 | 100.000 | 96.689 | 90.961 | 83.930 |
| 85 | 100.000 | 100.000 | 96.752 | 91.105 | 84.155 |
| 86 | 100.000 | 100.000 | 96.814 | 91.250 | 84.380 |
| 87 | 100.000 | 100.000 | 96.876 | 91.391 | 84.594 |
| 88 | 100.000 | 100.000 | 96.917 | 91.502 | 84.784 |
| 89 | 100.000 | 100.000 | 96.958 | 91.614 | 84.975 |
| 90 | 100.000 | 100.000 | 97.000 | 91.726 | 85.166 |
| 91 | 100.000 | 100.000 | 97.041 | 91.837 | 85.357 |
| 92 | 100.000 | 100.000 | 97.083 | 91.949 | 85.548 |
| 93 | 100.000 | 100.000 | 97.124 | 92.061 | 85.739 |
| 94 | 100.000 | 100.000 | 97.165 | 92.172 | 85.930 |
| 95 | 100.000 | 100.000 | 97.227 | 92.309 | 86.132 |
| 96 | 100.000 | 100.000 | 97.289 | 92.445 | 86.334 |
| 97 | 100.000 | 100.000 | 97.352 | 92.582 | 86.535 |
| 98 | 100.000 | 100.000 | 97.414 | 92.718 | 86.737 |
| 99 | 100.000 | 100.000 | 97.476 | 92.854 | 86.939 |
| 100 | 100.000 | 100.000 | 97.538 | 92.991 | 87.141 |
| 101 | 100.000 | 100.000 | 97.600 | 93.127 | 87.343 |
| 102 | 100.000 | 100.000 | 97.662 | 93.264 | 87.545 |
| 103 | 100.000 | 100.000 | 97.724 | 93.400 | 87.746 |
| 104 | 100.000 | 100.000 | 97.786 | 93.537 | 87.948 |
| 105 | 100.000 | 100.000 | 97.848 | 93.673 | 88.150 |
| 106 | 100.000 | 100.000 | 97.910 | 93.810 | 88.352 |
| 107 | 100.000 | 100.000 | 97.972 | 93.946 | 88.554 |
| 108 | 100.000 | 100.000 | 98.034 | 94.083 | 88.756 |
| 109 | 100.000 | 100.000 | 98.096 | 94.219 | 88.957 |
| 110 | 100.000 | 100.000 | 98.138 | 94.322 | 89.128 |
| 111 | 100.000 | 100.000 | 98.179 | 94.426 | 89.298 |
| 112 | 100.000 | 100.000 | 98.221 | 94.529 | 89.468 |
| 113 | 100.000 | 100.000 | 98.262 | 94.633 | 89.638 |
| 114 | 100.000 | 100.000 | 98.303 | 94.736 | 89.808 |
| 115 | 100.000 | 100.000 | 98.345 | 94.839 | 89.979 |
| 116 | 100.000 | 100.000 | 98.386 | 94.934 | 90.126 |
| 117 | 100.000 | 100.000 | 98.427 | 95.030 | 90.273 |
| 118 | 100.000 | 100.000 | 98.469 | 95.125 | 90.420 |
| 119 | 100.000 | 100.000 | 98.510 | 95.220 | 90.567 |
| 120 | 100.000 | 100.000 | 98.552 | 95.315 | 90.714 |
| 121 | 100.000 | 100.000 | 98.593 | 95.410 | 90.861 |
| 122 | 100.000 | 100.000 | 98.634 | 95.505 | 91.009 |
| 123 | 100.000 | 100.000 | 98.676 | 95.600 | 91.156 |
| 124 | 100.000 | 100.000 | 98.717 | 95.695 | 91.303 |
| 125 | 100.000 | 100.000 | 98.759 | 95.790 | 91.450 |
| 126 | 100.000 | 100.000 | 98.800 | 95.886 | 91.597 |
| 127 | 100.000 | 100.000 | 98.841 | 95.981 | 91.744 |
| 128 | 100.000 | 100.000 | 98.883 | 96.076 | 91.891 |
| 129 | 100.000 | 100.000 | 98.924 | 96.171 | 92.038 |
| 130 | 100.000 | 100.000 | 98.965 | 96.266 | 92.186 |
| 131 | 100.000 | 100.000 | 99.007 | 96.361 | 92.333 |
| 132 | 100.000 | 100.000 | 99.048 | 96.456 | 92.480 |
| 133 | 100.000 | 100.000 | 99.090 | 96.551 | 92.627 |
| 134 | 100.000 | 100.000 | 99.131 | 96.646 | 92.774 |
| 135 | 100.000 | 100.000 | 99.172 | 96.742 | 92.921 |
| 136 | 100.000 | 100.000 | 99.214 | 96.837 | 93.068 |
| 137 | 100.000 | 100.000 | 99.255 | 96.932 | 93.215 |
| 138 | 100.000 | 100.000 | 99.297 | 97.027 | 93.363 |
| 139 | 100.000 | 100.000 | 99.338 | 97.122 | 93.510 |
| 140 | 100.000 | 100.000 | 99.379 | 97.217 | 93.657 |
| 141 | 100.000 | 100.000 | 99.400 | 97.279 | 93.772 |
| 142 | 100.000 | 100.000 | 99.421 | 97.341 | 93.888 |
| 143 | 100.000 | 100.000 | 99.441 | 97.403 | 94.003 |
| 144 | 100.000 | 100.000 | 99.462 | 97.465 | 94.119 |
| 145 | 100.000 | 100.000 | 99.483 | 97.527 | 94.234 |
| 146 | 100.000 | 100.000 | 99.503 | 97.589 | 94.350 |
| 147 | 100.000 | 100.000 | 99.524 | 97.647 | 94.453 |
| 148 | 100.000 | 100.000 | 99.545 | 97.701 | 94.546 |
| 149 | 100.000 | 100.000 | 99.565 | 97.755 | 94.638 |
| 150 | 100.000 | 100.000 | 99.586 | 97.808 | 94.730 |
| 151 | 100.000 | 100.000 | 99.607 | 97.862 | 94.823 |
| 152 | 100.000 | 100.000 | 99.628 | 97.916 | 94.915 |
| 153 | 100.000 | 100.000 | 99.648 | 97.970 | 95.008 |
| 154 | 100.000 | 100.000 | 99.669 | 98.023 | 95.100 |
| 155 | 100.000 | 100.000 | 99.690 | 98.077 | 95.192 |
| 156 | 100.000 | 100.000 | 99.710 | 98.131 | 95.285 |
| 157 | 100.000 | 100.000 | 99.731 | 98.185 | 95.377 |
| 158 | 100.000 | 100.000 | 99.752 | 98.238 | 95.470 |
| 159 | 100.000 | 100.000 | 99.772 | 98.292 | 95.562 |
| 160 | 100.000 | 100.000 | 99.793 | 98.346 | 95.654 |
| 161 | 100.000 | 100.000 | 99.814 | 98.400 | 95.747 |
| 162 | 100.000 | 100.000 | 99.834 | 98.453 | 95.839 |
| 163 | 100.000 | 100.000 | 99.855 | 98.503 | 95.920 |
| 164 | 100.000 | 100.000 | 99.876 | 98.553 | 96.000 |
| 165 | 100.000 | 100.000 | 99.897 | 98.602 | 96.080 |
| 166 | 100.000 | 100.000 | 99.917 | 98.652 | 96.160 |
| 167 | 100.000 | 100.000 | 99.938 | 98.702 | 96.241 |
| 168 | 100.000 | 100.000 | 99.959 | 98.751 | 96.321 |
| 169 | 100.000 | 100.000 | 99.979 | 98.801 | 96.401 |
| 170 | 100.000 | 100.000 | 100.000 | 98.850 | 96.481 |
sed
Generation Strength: 4, Prioritisation Strength: 2
Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$
The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.
Complete Results
| Num. of Tests | Cov. Str. t=1 | Cov. Str. t=2 | Cov. Str. t=3 | Cov. Str. t=4 | Cov. Str. t=5 |
| 1 | 37.500 | 17.442 | 9.414 | 5.644 | 3.651 |
| 2 | 62.500 | 33.223 | 18.622 | 11.268 | 7.301 |
| 3 | 72.500 | 42.857 | 25.781 | 16.268 | 10.775 |
| 4 | 77.500 | 49.668 | 31.616 | 20.688 | 14.011 |
| 5 | 77.500 | 54.153 | 36.582 | 24.819 | 17.161 |
| 6 | 85.000 | 60.631 | 41.486 | 28.487 | 19.927 |
| 7 | 90.000 | 65.781 | 45.810 | 31.923 | 22.608 |
| 8 | 92.500 | 69.767 | 49.638 | 35.128 | 25.169 |
| 9 | 95.000 | 72.924 | 52.638 | 37.733 | 27.346 |
| 10 | 97.500 | 75.748 | 55.535 | 40.421 | 29.669 |
| 11 | 100.000 | 78.571 | 58.101 | 42.613 | 31.500 |
| 12 | 100.000 | 80.066 | 60.170 | 44.721 | 33.422 |
| 13 | 100.000 | 81.561 | 62.239 | 46.822 | 35.326 |
| 14 | 100.000 | 83.056 | 64.329 | 48.948 | 37.248 |
| 15 | 100.000 | 84.219 | 65.901 | 50.536 | 38.703 |
| 16 | 100.000 | 85.382 | 67.474 | 52.123 | 40.157 |
| 17 | 100.000 | 86.379 | 68.777 | 53.397 | 41.289 |
| 18 | 100.000 | 87.375 | 70.122 | 54.757 | 42.542 |
| 19 | 100.000 | 88.372 | 71.591 | 56.329 | 44.032 |
| 20 | 100.000 | 89.203 | 72.771 | 57.574 | 45.217 |
| 21 | 100.000 | 90.033 | 73.909 | 58.731 | 46.281 |
| 22 | 100.000 | 90.864 | 75.460 | 60.551 | 48.056 |
| 23 | 100.000 | 91.362 | 76.309 | 61.572 | 49.126 |
| 24 | 100.000 | 91.860 | 77.136 | 62.552 | 50.142 |
| 25 | 100.000 | 92.359 | 77.985 | 63.569 | 51.201 |
| 26 | 100.000 | 92.857 | 78.854 | 64.628 | 52.314 |
| 27 | 100.000 | 93.355 | 79.764 | 65.769 | 53.536 |
| 28 | 100.000 | 93.688 | 80.364 | 66.522 | 54.351 |
| 29 | 100.000 | 94.020 | 81.006 | 67.357 | 55.275 |
| 30 | 100.000 | 94.352 | 81.523 | 67.953 | 55.890 |
| 31 | 100.000 | 94.684 | 82.164 | 68.792 | 56.824 |
| 32 | 100.000 | 95.017 | 82.744 | 69.499 | 57.572 |
| 33 | 100.000 | 95.349 | 83.344 | 70.252 | 58.387 |
| 34 | 100.000 | 95.681 | 83.944 | 71.004 | 59.201 |
| 35 | 100.000 | 95.847 | 84.378 | 71.650 | 59.968 |
| 36 | 100.000 | 96.013 | 84.813 | 72.295 | 60.736 |
| 37 | 100.000 | 96.179 | 85.082 | 72.617 | 61.083 |
| 38 | 100.000 | 96.346 | 85.433 | 73.122 | 61.695 |
| 39 | 100.000 | 96.512 | 85.806 | 73.655 | 62.330 |
| 40 | 100.000 | 96.678 | 86.220 | 74.259 | 63.042 |
| 41 | 100.000 | 96.844 | 86.551 | 74.705 | 63.555 |
| 42 | 100.000 | 97.010 | 86.903 | 75.193 | 64.123 |
| 43 | 100.000 | 97.176 | 87.172 | 75.516 | 64.471 |
| 44 | 100.000 | 97.342 | 87.544 | 76.041 | 65.082 |
| 45 | 100.000 | 97.508 | 87.896 | 76.521 | 65.630 |
| 46 | 100.000 | 97.674 | 88.206 | 76.930 | 66.099 |
| 47 | 100.000 | 97.841 | 88.496 | 77.298 | 66.513 |
| 48 | 100.000 | 98.007 | 88.785 | 77.666 | 66.928 |
| 49 | 100.000 | 98.173 | 89.137 | 78.154 | 67.496 |
| 50 | 100.000 | 98.339 | 89.406 | 78.477 | 67.844 |
| 51 | 100.000 | 98.505 | 89.799 | 79.051 | 68.533 |
| 52 | 100.000 | 98.671 | 90.130 | 79.498 | 69.046 |
| 53 | 100.000 | 98.837 | 90.399 | 79.821 | 69.394 |
| 54 | 100.000 | 99.003 | 90.710 | 80.230 | 69.863 |
| 55 | 100.000 | 99.169 | 91.020 | 80.639 | 70.333 |
| 56 | 100.000 | 99.336 | 91.372 | 81.127 | 70.900 |
| 57 | 100.000 | 99.502 | 91.641 | 81.450 | 71.248 |
| 58 | 100.000 | 99.668 | 91.930 | 81.818 | 71.663 |
| 59 | 100.000 | 99.834 | 92.220 | 82.186 | 72.077 |
| 60 | 100.000 | 100.000 | 92.489 | 82.508 | 72.425 |
| 61 | 100.000 | 100.000 | 92.551 | 82.649 | 72.639 |
| 62 | 100.000 | 100.000 | 92.572 | 82.707 | 72.742 |
| 63 | 100.000 | 100.000 | 92.593 | 82.756 | 72.823 |
| 64 | 100.000 | 100.000 | 92.717 | 83.025 | 73.213 |
| 65 | 100.000 | 100.000 | 92.800 | 83.199 | 73.459 |
| 66 | 100.000 | 100.000 | 92.903 | 83.418 | 73.771 |
| 67 | 100.000 | 100.000 | 93.048 | 83.720 | 74.193 |
| 68 | 100.000 | 100.000 | 93.131 | 83.910 | 74.485 |
| 69 | 100.000 | 100.000 | 93.172 | 84.005 | 74.632 |
| 70 | 100.000 | 100.000 | 93.213 | 84.100 | 74.779 |
| 71 | 100.000 | 100.000 | 93.358 | 84.402 | 75.201 |
| 72 | 100.000 | 100.000 | 93.462 | 84.626 | 75.524 |
| 73 | 100.000 | 100.000 | 93.524 | 84.779 | 75.770 |
| 74 | 100.000 | 100.000 | 93.524 | 84.787 | 75.795 |
| 75 | 100.000 | 100.000 | 93.544 | 84.845 | 75.899 |
| 76 | 100.000 | 100.000 | 93.606 | 84.985 | 76.113 |
| 77 | 100.000 | 100.000 | 93.627 | 85.035 | 76.193 |
| 78 | 100.000 | 100.000 | 93.669 | 85.138 | 76.361 |
| 79 | 100.000 | 100.000 | 93.731 | 85.279 | 76.575 |
| 80 | 100.000 | 100.000 | 93.855 | 85.539 | 76.942 |
| 81 | 100.000 | 100.000 | 93.938 | 85.721 | 77.211 |
| 82 | 100.000 | 100.000 | 94.041 | 85.945 | 77.534 |
| 83 | 100.000 | 100.000 | 94.082 | 86.040 | 77.681 |
| 84 | 100.000 | 100.000 | 94.165 | 86.222 | 77.950 |
| 85 | 100.000 | 100.000 | 94.248 | 86.404 | 78.219 |
| 86 | 100.000 | 100.000 | 94.289 | 86.507 | 78.389 |
| 87 | 100.000 | 100.000 | 94.331 | 86.602 | 78.536 |
| 88 | 100.000 | 100.000 | 94.351 | 86.664 | 78.652 |
| 89 | 100.000 | 100.000 | 94.455 | 86.887 | 78.975 |
| 90 | 100.000 | 100.000 | 94.475 | 86.945 | 79.078 |
| 91 | 100.000 | 100.000 | 94.558 | 87.127 | 79.347 |
| 92 | 100.000 | 100.000 | 94.558 | 87.131 | 79.360 |
| 93 | 100.000 | 100.000 | 94.641 | 87.313 | 79.629 |
| 94 | 100.000 | 100.000 | 94.641 | 87.318 | 79.643 |
| 95 | 100.000 | 100.000 | 94.682 | 87.429 | 79.833 |
| 96 | 100.000 | 100.000 | 94.703 | 87.479 | 79.914 |
| 97 | 100.000 | 100.000 | 94.786 | 87.669 | 80.203 |
| 98 | 100.000 | 100.000 | 94.848 | 87.805 | 80.405 |
| 99 | 100.000 | 100.000 | 94.889 | 87.901 | 80.552 |
| 100 | 100.000 | 100.000 | 94.931 | 87.996 | 80.699 |
| 101 | 100.000 | 100.000 | 95.055 | 88.256 | 81.066 |
| 102 | 100.000 | 100.000 | 95.096 | 88.360 | 81.237 |
| 103 | 100.000 | 100.000 | 95.096 | 88.364 | 81.250 |
| 104 | 100.000 | 100.000 | 95.158 | 88.500 | 81.452 |
| 105 | 100.000 | 100.000 | 95.158 | 88.525 | 81.521 |
| 106 | 100.000 | 100.000 | 95.220 | 88.661 | 81.723 |
| 107 | 100.000 | 100.000 | 95.282 | 88.798 | 81.925 |
| 108 | 100.000 | 100.000 | 95.427 | 89.100 | 82.347 |
| 109 | 100.000 | 100.000 | 95.427 | 89.116 | 82.395 |
| 110 | 100.000 | 100.000 | 95.551 | 89.377 | 82.762 |
| 111 | 100.000 | 100.000 | 95.613 | 89.513 | 82.964 |
| 112 | 100.000 | 100.000 | 95.634 | 89.563 | 83.045 |
| 113 | 100.000 | 100.000 | 95.676 | 89.658 | 83.192 |
| 114 | 100.000 | 100.000 | 95.820 | 89.960 | 83.614 |
| 115 | 100.000 | 100.000 | 95.841 | 90.010 | 83.694 |
| 116 | 100.000 | 100.000 | 95.882 | 90.121 | 83.885 |
| 117 | 100.000 | 100.000 | 95.924 | 90.216 | 84.032 |
| 118 | 100.000 | 100.000 | 95.945 | 90.266 | 84.112 |
| 119 | 100.000 | 100.000 | 96.007 | 90.402 | 84.314 |
| 120 | 100.000 | 100.000 | 96.027 | 90.469 | 84.438 |
| 121 | 100.000 | 100.000 | 96.048 | 90.518 | 84.518 |
| 122 | 100.000 | 100.000 | 96.089 | 90.613 | 84.665 |
| 123 | 100.000 | 100.000 | 96.172 | 90.812 | 84.978 |
| 124 | 100.000 | 100.000 | 96.214 | 90.907 | 85.125 |
| 125 | 100.000 | 100.000 | 96.214 | 90.915 | 85.150 |
| 126 | 100.000 | 100.000 | 96.234 | 90.973 | 85.254 |
| 127 | 100.000 | 100.000 | 96.255 | 91.035 | 85.369 |
| 128 | 100.000 | 100.000 | 96.255 | 91.043 | 85.395 |
| 129 | 100.000 | 100.000 | 96.296 | 91.138 | 85.542 |
| 130 | 100.000 | 100.000 | 96.317 | 91.188 | 85.622 |
| 131 | 100.000 | 100.000 | 96.379 | 91.329 | 85.836 |
| 132 | 100.000 | 100.000 | 96.379 | 91.341 | 85.873 |
| 133 | 100.000 | 100.000 | 96.379 | 91.349 | 85.898 |
| 134 | 100.000 | 100.000 | 96.400 | 91.399 | 85.978 |
| 135 | 100.000 | 100.000 | 96.441 | 91.494 | 86.126 |
| 136 | 100.000 | 100.000 | 96.483 | 91.593 | 86.285 |
| 137 | 100.000 | 100.000 | 96.483 | 91.597 | 86.298 |
| 138 | 100.000 | 100.000 | 96.503 | 91.647 | 86.379 |
| 139 | 100.000 | 100.000 | 96.545 | 91.742 | 86.526 |
| 140 | 100.000 | 100.000 | 96.586 | 91.837 | 86.673 |
| 141 | 100.000 | 100.000 | 96.607 | 91.887 | 86.753 |
| 142 | 100.000 | 100.000 | 96.648 | 91.982 | 86.900 |
| 143 | 100.000 | 100.000 | 96.648 | 91.986 | 86.913 |
| 144 | 100.000 | 100.000 | 96.648 | 91.994 | 86.939 |
| 145 | 100.000 | 100.000 | 96.689 | 92.089 | 87.086 |
| 146 | 100.000 | 100.000 | 96.689 | 92.098 | 87.112 |
| 147 | 100.000 | 100.000 | 96.689 | 92.106 | 87.137 |
| 148 | 100.000 | 100.000 | 96.689 | 92.114 | 87.163 |
| 149 | 100.000 | 100.000 | 96.731 | 92.209 | 87.310 |
| 150 | 100.000 | 100.000 | 96.772 | 92.305 | 87.457 |
| 151 | 100.000 | 100.000 | 96.814 | 92.416 | 87.648 |
| 152 | 100.000 | 100.000 | 96.855 | 92.511 | 87.795 |
| 153 | 100.000 | 100.000 | 96.855 | 92.520 | 87.821 |
| 154 | 100.000 | 100.000 | 96.876 | 92.586 | 87.945 |
| 155 | 100.000 | 100.000 | 96.896 | 92.635 | 88.025 |
| 156 | 100.000 | 100.000 | 96.896 | 92.644 | 88.050 |
| 157 | 100.000 | 100.000 | 96.896 | 92.648 | 88.064 |
| 158 | 100.000 | 100.000 | 96.938 | 92.751 | 88.234 |
| 159 | 100.000 | 100.000 | 97.000 | 92.888 | 88.436 |
| 160 | 100.000 | 100.000 | 97.621 | 93.942 | 89.652 |
| 161 | 100.000 | 100.000 | 97.683 | 94.078 | 89.853 |
| 162 | 100.000 | 100.000 | 97.683 | 94.083 | 89.867 |
| 163 | 100.000 | 100.000 | 97.683 | 94.087 | 89.880 |
| 164 | 100.000 | 100.000 | 97.724 | 94.182 | 90.027 |
| 165 | 100.000 | 100.000 | 97.724 | 94.190 | 90.053 |
| 166 | 100.000 | 100.000 | 97.724 | 94.203 | 90.089 |
| 167 | 100.000 | 100.000 | 97.724 | 94.211 | 90.115 |
| 168 | 100.000 | 100.000 | 97.724 | 94.219 | 90.140 |
| 169 | 100.000 | 100.000 | 97.765 | 94.314 | 90.288 |
| 170 | 100.000 | 100.000 | 97.786 | 94.364 | 90.368 |
| 171 | 100.000 | 100.000 | 97.786 | 94.372 | 90.393 |
| 172 | 100.000 | 100.000 | 97.827 | 94.467 | 90.540 |
| 173 | 100.000 | 100.000 | 97.848 | 94.521 | 90.633 |
| 174 | 100.000 | 100.000 | 97.848 | 94.525 | 90.646 |
| 175 | 100.000 | 100.000 | 97.931 | 94.707 | 90.915 |
| 176 | 100.000 | 100.000 | 97.952 | 94.757 | 90.995 |
| 177 | 100.000 | 100.000 | 97.972 | 94.806 | 91.075 |
| 178 | 100.000 | 100.000 | 97.993 | 94.856 | 91.156 |
| 179 | 100.000 | 100.000 | 97.993 | 94.872 | 91.204 |
| 180 | 100.000 | 100.000 | 98.034 | 94.968 | 91.351 |
| 181 | 100.000 | 100.000 | 98.034 | 94.980 | 91.388 |
| 182 | 100.000 | 100.000 | 98.055 | 95.030 | 91.468 |
| 183 | 100.000 | 100.000 | 98.076 | 95.083 | 91.561 |
| 184 | 100.000 | 100.000 | 98.117 | 95.178 | 91.708 |
| 185 | 100.000 | 100.000 | 98.158 | 95.274 | 91.855 |
| 186 | 100.000 | 100.000 | 98.200 | 95.369 | 92.002 |
| 187 | 100.000 | 100.000 | 98.200 | 95.373 | 92.015 |
| 188 | 100.000 | 100.000 | 98.221 | 95.439 | 92.139 |
| 189 | 100.000 | 100.000 | 98.241 | 95.489 | 92.220 |
| 190 | 100.000 | 100.000 | 98.241 | 95.505 | 92.268 |
| 191 | 100.000 | 100.000 | 98.262 | 95.555 | 92.348 |
| 192 | 100.000 | 100.000 | 98.262 | 95.559 | 92.362 |
| 193 | 100.000 | 100.000 | 98.262 | 95.563 | 92.375 |
| 194 | 100.000 | 100.000 | 98.303 | 95.658 | 92.522 |
| 195 | 100.000 | 100.000 | 98.303 | 95.671 | 92.559 |
| 196 | 100.000 | 100.000 | 98.303 | 95.675 | 92.572 |
| 197 | 100.000 | 100.000 | 98.365 | 95.811 | 92.774 |
| 198 | 100.000 | 100.000 | 98.386 | 95.861 | 92.854 |
| 199 | 100.000 | 100.000 | 98.448 | 95.997 | 93.056 |
| 200 | 100.000 | 100.000 | 98.448 | 96.001 | 93.069 |
| 201 | 100.000 | 100.000 | 98.448 | 96.010 | 93.095 |
| 202 | 100.000 | 100.000 | 98.448 | 96.014 | 93.108 |
| 203 | 100.000 | 100.000 | 98.448 | 96.039 | 93.178 |
| 204 | 100.000 | 100.000 | 98.490 | 96.134 | 93.325 |
| 205 | 100.000 | 100.000 | 98.531 | 96.229 | 93.472 |
| 206 | 100.000 | 100.000 | 98.531 | 96.237 | 93.497 |
| 207 | 100.000 | 100.000 | 98.531 | 96.245 | 93.523 |
| 208 | 100.000 | 100.000 | 98.552 | 96.295 | 93.603 |
| 209 | 100.000 | 100.000 | 98.572 | 96.345 | 93.684 |
| 210 | 100.000 | 100.000 | 98.572 | 96.357 | 93.720 |
| 211 | 100.000 | 100.000 | 98.572 | 96.365 | 93.746 |
| 212 | 100.000 | 100.000 | 98.572 | 96.373 | 93.771 |
| 213 | 100.000 | 100.000 | 98.634 | 96.510 | 93.973 |
| 214 | 100.000 | 100.000 | 98.634 | 96.522 | 94.009 |
| 215 | 100.000 | 100.000 | 98.634 | 96.531 | 94.035 |
| 216 | 100.000 | 100.000 | 98.655 | 96.580 | 94.115 |
| 217 | 100.000 | 100.000 | 98.676 | 96.630 | 94.195 |
| 218 | 100.000 | 100.000 | 98.696 | 96.679 | 94.276 |
| 219 | 100.000 | 100.000 | 98.696 | 96.688 | 94.301 |
| 220 | 100.000 | 100.000 | 98.696 | 96.692 | 94.315 |
| 221 | 100.000 | 100.000 | 98.696 | 96.696 | 94.328 |
| 222 | 100.000 | 100.000 | 98.717 | 96.750 | 94.420 |
| 223 | 100.000 | 100.000 | 98.738 | 96.804 | 94.513 |
| 224 | 100.000 | 100.000 | 98.738 | 96.808 | 94.526 |
| 225 | 100.000 | 100.000 | 98.759 | 96.857 | 94.606 |
| 226 | 100.000 | 100.000 | 98.759 | 96.866 | 94.632 |
| 227 | 100.000 | 100.000 | 98.759 | 96.870 | 94.645 |
| 228 | 100.000 | 100.000 | 98.800 | 96.965 | 94.792 |
| 229 | 100.000 | 100.000 | 98.821 | 97.014 | 94.873 |
| 230 | 100.000 | 100.000 | 98.821 | 97.019 | 94.886 |
| 231 | 100.000 | 100.000 | 98.821 | 97.027 | 94.912 |
| 232 | 100.000 | 100.000 | 98.821 | 97.039 | 94.948 |
| 233 | 100.000 | 100.000 | 98.821 | 97.043 | 94.961 |
| 234 | 100.000 | 100.000 | 98.862 | 97.138 | 95.109 |
| 235 | 100.000 | 100.000 | 98.862 | 97.143 | 95.122 |
| 236 | 100.000 | 100.000 | 98.903 | 97.238 | 95.269 |
| 237 | 100.000 | 100.000 | 98.945 | 97.333 | 95.416 |
| 238 | 100.000 | 100.000 | 98.986 | 97.428 | 95.563 |
| 239 | 100.000 | 100.000 | 98.986 | 97.436 | 95.589 |
| 240 | 100.000 | 100.000 | 99.028 | 97.531 | 95.736 |
| 241 | 100.000 | 100.000 | 99.069 | 97.626 | 95.883 |
| 242 | 100.000 | 100.000 | 99.069 | 97.635 | 95.909 |
| 243 | 100.000 | 100.000 | 99.069 | 97.643 | 95.934 |
| 244 | 100.000 | 100.000 | 99.110 | 97.738 | 96.081 |
| 245 | 100.000 | 100.000 | 99.110 | 97.746 | 96.107 |
| 246 | 100.000 | 100.000 | 99.110 | 97.755 | 96.132 |
| 247 | 100.000 | 100.000 | 99.110 | 97.763 | 96.158 |
| 248 | 100.000 | 100.000 | 99.110 | 97.767 | 96.171 |
| 249 | 100.000 | 100.000 | 99.110 | 97.771 | 96.185 |
| 250 | 100.000 | 100.000 | 99.110 | 97.775 | 96.198 |
| 251 | 100.000 | 100.000 | 99.152 | 97.870 | 96.345 |
| 252 | 100.000 | 100.000 | 99.152 | 97.875 | 96.358 |
| 253 | 100.000 | 100.000 | 99.152 | 97.879 | 96.372 |
| 254 | 100.000 | 100.000 | 99.152 | 97.887 | 96.397 |
| 255 | 100.000 | 100.000 | 99.152 | 97.891 | 96.411 |
| 256 | 100.000 | 100.000 | 99.152 | 97.899 | 96.436 |
| 257 | 100.000 | 100.000 | 99.152 | 97.908 | 96.462 |
| 258 | 100.000 | 100.000 | 99.172 | 97.957 | 96.542 |
| 259 | 100.000 | 100.000 | 99.379 | 98.350 | 97.047 |
| 260 | 100.000 | 100.000 | 99.379 | 98.358 | 97.072 |
| 261 | 100.000 | 100.000 | 99.379 | 98.367 | 97.098 |
| 262 | 100.000 | 100.000 | 99.379 | 98.371 | 97.111 |
| 263 | 100.000 | 100.000 | 99.379 | 98.375 | 97.124 |
| 264 | 100.000 | 100.000 | 99.379 | 98.391 | 97.173 |
| 265 | 100.000 | 100.000 | 99.379 | 98.400 | 97.199 |
| 266 | 100.000 | 100.000 | 99.421 | 98.495 | 97.346 |
| 267 | 100.000 | 100.000 | 99.441 | 98.549 | 97.438 |
| 268 | 100.000 | 100.000 | 99.441 | 98.553 | 97.452 |
| 269 | 100.000 | 100.000 | 99.462 | 98.602 | 97.532 |
| 270 | 100.000 | 100.000 | 99.483 | 98.652 | 97.612 |
| 271 | 100.000 | 100.000 | 99.483 | 98.656 | 97.625 |
| 272 | 100.000 | 100.000 | 99.483 | 98.664 | 97.651 |
| 273 | 100.000 | 100.000 | 99.483 | 98.668 | 97.664 |
| 274 | 100.000 | 100.000 | 99.483 | 98.677 | 97.690 |
| 275 | 100.000 | 100.000 | 99.483 | 98.693 | 97.738 |
| 276 | 100.000 | 100.000 | 99.483 | 98.697 | 97.752 |
| 277 | 100.000 | 100.000 | 99.483 | 98.702 | 97.765 |
| 278 | 100.000 | 100.000 | 99.483 | 98.710 | 97.791 |
| 279 | 100.000 | 100.000 | 99.483 | 98.726 | 97.839 |
| 280 | 100.000 | 100.000 | 99.483 | 98.735 | 97.865 |
| 281 | 100.000 | 100.000 | 99.483 |