Higher Strength Combinatorial Interaction Testing

This page is supplementary to the paper entitled "Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing" ([PDF] [BIB]), which is currently to appear in the proceedings of the ESEC/FSE 2013.

Abstract

Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints and should, ideally, report a test priority order that maximises earliest fault detection. We show that we can achieve the highest strength CIT in 5.65 minutes on average. This was previously thought to be too computationally expensive to be feasible. Furthermore, we show that higher strength suites find more faults, while prioritisations using lower strengths are no worse at achieving early fault revelation.

Experimental Setup

Subject Programs

We use five Unix utilities from the Software Infrastructure Repository (SIR), maintained by University of Nebraska Lincoln: flex, grep, gzip, sed, and make.

Test Suites

The test suites we use are subsets of those provided by SIR. It can be downloaded from here. For each of the subjects we provide:

  1. An annotated TSL file (from SIR) : _v0_#_small.tsl
    Description: The v0_# matches the associated TSL name from SIR for that subject. The leftmost column in this file is a mapping to the covering array values given as sequential numbers starting at 0. For instance, the first parameter if binary will be 0,1 and the next parameter's values will start at 2. The single and error parameters (and any other parameters that we did not include in our model) are not numbered and not used in our experiments.
  2. An input model file for CASA: _v0_small.model
    Description: The first row is the strength (t) and can be changed to 3,4 etc. to generate higher strength arrays. The second row is the number of factors (TSL parameters in the model). The last row lists the number of values (TSL choices) for each of the parameters. This model matches the associated TSL file.
  3. The constraint file in CASA input format: _v0_small.constraints
    Description: The first row of this file is the number of constraints. For each of the constraints there are two rows. The first row gives the arity of the constraint (i.e. how many values will be included in this constraint). The next row is list of the constraints between values. The - signs indicate excludes. For instance, a row with a 2, followed by a row that has -10 -12 encodes the binary constraint that values 10 and 12 cannot appear together in a test. Please see the CASA tool for full documentation.

Combinatorial Interaction Testing Tool

We use CASA (Covering Arrays by Simulated Annealing) tool to generate combinatorial interaction test suites from the TSL files.

Full Results

This section contains the full APCC data for all experiments and plots for APCC and fault detection.

Subject Generation Strength Prioritsation Strengths Subject Generation Strength Prioritsation Strengths
flex 2 2, 3, 4, 5 gzip 2 2, 3, 4, 5
flex 3 2, 3, 4, 5 gzip 3 2, 3, 4, 5
flex 4 2, 3, 4, 5 gzip 4 2, 3, 4, 5
flex 5 2, 3, 4, 5 gzip 5 2, 3, 4, 5
grep 2 2, 3, 4, 5 sed 2 2, 3, 4, 5
grep 3 2, 3, 4, 5 sed 3 2, 3, 4, 5
grep 4 2, 3, 4, 5 sed 4 2, 3, 4, 5
grep 5 2, 3, 4, 5 sed 5 2, 3, 4, 5

flex

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.851 71.707 60.886 51.033
5 97.674 89.340 78.722 67.791 57.518
6 100.000 93.163 83.403 72.852 62.623
7 100.000 94.206 85.516 75.762 66.013
8 100.000 95.017 87.113 77.908 68.466
9 100.000 95.829 88.849 80.407 71.487
10 100.000 96.408 90.086 82.192 73.659
11 100.000 96.987 91.240 83.759 75.468
12 100.000 97.451 92.394 85.598 77.854
13 100.000 97.798 93.160 86.739 79.287
14 100.000 98.030 93.760 87.730 80.618
15 100.000 98.262 94.268 88.491 81.588
16 100.000 98.494 94.831 89.396 82.795
17 100.000 98.725 95.458 90.470 84.280
18 100.000 98.957 95.994 91.305 85.375
19 100.000 99.189 96.594 92.307 86.742
20 100.000 99.305 96.843 92.677 87.212
21 100.000 99.421 97.175 93.258 88.023
22 100.000 99.537 97.434 93.656 88.545
23 100.000 99.652 97.711 94.097 89.130
24 100.000 99.768 98.006 94.592 89.818
25 100.000 99.884 98.274 95.012 90.372
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.271 70.747 59.812 50.024
5 97.674 89.108 78.418 67.553 57.408
6 100.000 93.048 83.310 72.881 62.799
7 100.000 93.975 85.240 75.594 66.014
8 100.000 94.670 86.753 77.807 68.727
9 100.000 95.481 88.433 80.170 71.538
10 100.000 96.176 89.901 82.268 74.067
11 100.000 96.871 91.341 84.300 76.498
12 100.000 97.219 92.218 85.721 78.377
13 100.000 97.567 93.021 86.959 79.971
14 100.000 97.914 93.806 88.142 81.462
15 100.000 98.262 94.581 89.308 82.934
16 100.000 98.610 95.320 90.379 84.256
17 100.000 98.841 95.855 91.219 85.365
18 100.000 99.073 96.381 92.021 86.389
19 100.000 99.189 96.705 92.589 87.195
20 100.000 99.305 97.009 93.109 87.925
21 100.000 99.421 97.268 93.507 88.448
22 100.000 99.537 97.526 93.899 88.950
23 100.000 99.652 97.775 94.271 89.426
24 100.000 99.768 98.025 94.640 89.896
25 100.000 99.884 98.274 95.010 90.366
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 81.808 69.934 58.848 49.062
5 97.674 88.528 77.375 66.284 56.108
6 97.674 90.151 80.504 70.398 60.699
7 97.674 91.078 82.424 73.093 63.894
8 97.674 91.889 84.187 75.652 66.997
9 97.674 92.352 85.267 77.317 69.125
10 100.000 96.060 89.670 81.958 73.716
11 100.000 96.408 90.538 83.356 75.562
12 100.000 97.103 91.978 85.384 77.980
13 100.000 97.451 92.791 86.642 79.600
14 100.000 97.798 93.557 87.793 81.063
15 100.000 98.146 94.341 88.982 82.571
16 100.000 98.494 95.126 90.168 84.073
17 100.000 98.725 95.671 91.019 85.181
18 100.000 98.957 96.178 91.780 86.151
19 100.000 99.189 96.705 92.582 87.175
20 100.000 99.305 97.000 93.078 87.863
21 100.000 99.421 97.268 93.498 88.418
22 100.000 99.537 97.526 93.896 88.940
23 100.000 99.652 97.775 94.271 89.426
24 100.000 99.768 98.025 94.642 89.902
25 100.000 99.884 98.274 95.012 90.372
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.954 68.993 58.624 49.270
5 93.023 84.125 74.328 64.519 55.269
6 95.349 87.717 78.685 69.298 60.196
7 97.674 91.194 82.766 73.665 64.635
8 97.674 92.236 84.870 76.556 68.001
9 97.674 92.816 86.135 78.421 70.313
10 97.674 93.395 87.409 80.302 72.642
11 100.000 96.640 90.972 83.914 76.172
12 100.000 97.335 92.394 85.900 78.531
13 100.000 97.683 93.178 87.092 80.054
14 100.000 98.030 93.963 88.282 81.565
15 100.000 98.262 94.508 89.137 82.686
16 100.000 98.378 94.858 89.773 83.600
17 100.000 98.610 95.403 90.628 84.721
18 100.000 98.725 95.708 91.150 85.456
19 100.000 98.957 96.197 91.860 86.337
20 100.000 99.189 96.741 92.705 87.423
21 100.000 99.421 97.268 93.507 88.448
22 100.000 99.537 97.517 93.879 88.924
23 100.000 99.652 97.766 94.251 89.400
24 100.000 99.768 98.015 94.621 89.870
25 100.000 99.884 98.265 94.990 90.340
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 86.047 69.757 55.183 43.241 33.810
3 90.698 77.520 64.405 52.735 42.856
4 95.349 84.820 72.851 61.352 51.071
5 100.000 91.309 79.886 68.304 57.631
6 100.000 93.048 83.283 72.814 62.698
7 100.000 94.322 85.821 76.250 66.635
8 100.000 95.249 87.723 78.901 69.762
9 100.000 96.060 89.357 81.147 72.387
10 100.000 96.756 90.843 83.296 75.004
11 100.000 97.103 91.609 84.434 76.427
12 100.000 97.451 92.458 85.785 78.194
13 100.000 97.798 93.252 86.997 79.738
14 100.000 98.146 93.981 88.041 81.010
15 100.000 98.378 94.609 89.104 82.459
16 100.000 98.610 95.255 90.212 83.980
17 100.000 98.841 95.818 91.119 85.194
18 100.000 98.957 96.114 91.621 85.902
19 100.000 99.073 96.372 92.019 86.424
20 100.000 99.189 96.649 92.459 87.009
21 100.000 99.305 96.917 92.877 87.558
22 100.000 99.421 97.203 93.347 88.198
23 100.000 99.537 97.471 93.772 88.770
24 100.000 99.652 97.738 94.191 89.325
25 100.000 99.768 98.052 94.746 90.125
26 100.000 99.884 98.301 95.117 90.601
27 100.000 100.000 98.569 95.537 91.156
28 100.000 100.000 98.643 95.745 91.529
29 100.000 100.000 98.726 95.972 91.924
30 100.000 100.000 98.735 96.003 91.985
31 100.000 100.000 98.800 96.180 92.294
32 100.000 100.000 98.837 96.282 92.474
33 100.000 100.000 98.911 96.494 92.860
34 100.000 100.000 98.994 96.726 93.273
35 100.000 100.000 99.086 96.980 93.715
36 100.000 100.000 99.169 97.223 94.164
37 100.000 100.000 99.178 97.250 94.215
38 100.000 100.000 99.188 97.277 94.266
39 100.000 100.000 99.197 97.306 94.320
40 100.000 100.000 99.289 97.560 94.767
41 100.000 100.000 99.400 97.875 95.334
42 100.000 100.000 99.409 97.905 95.395
43 100.000 100.000 99.492 98.136 95.799
44 100.000 100.000 99.502 98.164 95.854
45 100.000 100.000 99.511 98.191 95.905
46 100.000 100.000 99.585 98.403 96.292
47 100.000 100.000 99.631 98.538 96.538
48 100.000 100.000 99.640 98.565 96.589
49 100.000 100.000 99.732 98.817 97.027
50 100.000 100.000 99.788 98.979 97.326
51 100.000 100.000 99.852 99.159 97.647
52 100.000 100.000 99.862 99.187 97.698
53 100.000 100.000 99.898 99.289 97.877
54 100.000 100.000 99.954 99.447 98.166
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 95.349 84.936 73.064 61.614 51.341
5 97.674 88.992 78.187 67.247 57.070
6 100.000 92.816 82.895 72.377 62.284
7 100.000 94.090 85.452 75.862 66.303
8 100.000 95.017 87.390 78.600 69.560
9 100.000 95.829 89.098 81.031 72.476
10 100.000 96.524 90.566 83.128 75.005
11 100.000 97.219 91.987 85.116 77.367
12 100.000 97.451 92.624 86.204 78.862
13 100.000 97.683 93.261 87.292 80.357
14 100.000 98.030 94.046 88.476 81.849
15 100.000 98.378 94.821 89.644 83.331
16 100.000 98.494 95.172 90.290 84.276
17 100.000 98.610 95.523 90.935 85.217
18 100.000 98.841 96.040 91.718 86.220
19 100.000 98.957 96.372 92.312 87.076
20 100.000 98.957 96.511 92.687 87.721
21 100.000 99.073 96.825 93.226 88.469
22 100.000 99.073 96.945 93.559 89.056
23 100.000 99.189 97.231 94.045 89.748
24 100.000 99.189 97.332 94.324 90.237
25 100.000 99.189 97.425 94.577 90.677
26 100.000 99.189 97.517 94.836 91.139
27 100.000 99.305 97.775 95.236 91.668
28 100.000 99.305 97.849 95.451 92.061
29 100.000 99.305 97.923 95.668 92.464
30 100.000 99.421 98.181 96.066 92.987
31 100.000 99.537 98.431 96.443 93.480
32 100.000 99.652 98.680 96.813 93.950
33 100.000 99.768 98.929 97.181 94.413
34 100.000 99.884 99.178 97.549 94.879
35 100.000 100.000 99.428 97.915 95.335
36 100.000 100.000 99.483 98.071 95.617
37 100.000 100.000 99.538 98.228 95.902
38 100.000 100.000 99.594 98.385 96.183
39 100.000 100.000 99.649 98.548 96.485
40 100.000 100.000 99.695 98.682 96.735
41 100.000 100.000 99.742 98.824 97.005
42 100.000 100.000 99.788 98.957 97.250
43 100.000 100.000 99.834 99.093 97.500
44 100.000 100.000 99.871 99.199 97.694
45 100.000 100.000 99.908 99.303 97.881
46 100.000 100.000 99.917 99.330 97.932
47 100.000 100.000 99.926 99.357 97.982
48 100.000 100.000 99.935 99.385 98.037
49 100.000 100.000 99.945 99.415 98.095
50 100.000 100.000 99.954 99.442 98.146
51 100.000 100.000 99.963 99.470 98.200
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.305
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 88.372 74.739 61.885 50.692 41.300
4 93.023 82.039 70.331 59.310 49.515
5 97.674 88.644 77.605 66.592 56.452
6 97.674 90.267 80.809 70.883 61.312
7 100.000 93.975 85.276 75.688 66.165
8 100.000 94.670 86.781 77.877 68.838
9 100.000 95.249 88.064 79.782 71.205
10 100.000 95.944 89.523 81.856 73.695
11 100.000 96.524 90.741 83.602 75.818
12 100.000 97.219 92.191 85.649 78.258
13 100.000 97.451 92.800 86.673 79.658
14 100.000 97.567 93.151 87.325 80.621
15 100.000 98.030 94.138 88.755 82.373
16 100.000 98.262 94.720 89.705 83.649
17 100.000 98.262 94.849 90.065 84.285
18 100.000 98.378 95.200 90.703 85.206
19 100.000 98.610 95.744 91.558 86.327
20 100.000 98.725 96.077 92.152 87.179
21 100.000 98.841 96.381 92.677 87.923
22 100.000 98.957 96.695 93.223 88.698
23 100.000 99.189 97.194 93.962 89.635
24 100.000 99.189 97.286 94.222 90.098
25 100.000 99.189 97.388 94.502 90.589
26 100.000 99.305 97.692 95.016 91.299
27 100.000 99.305 97.785 95.269 91.739
28 100.000 99.305 97.840 95.435 92.048
29 100.000 99.305 97.905 95.621 92.389
30 100.000 99.305 97.960 95.787 92.698
31 100.000 99.305 98.015 95.952 93.004
32 100.000 99.421 98.274 96.355 93.543
33 100.000 99.421 98.329 96.518 93.848
34 100.000 99.421 98.385 96.678 94.137
35 100.000 99.421 98.440 96.836 94.426
36 100.000 99.421 98.495 96.991 94.705
37 100.000 99.537 98.745 97.365 95.188
38 100.000 99.537 98.791 97.497 95.430
39 100.000 99.537 98.837 97.628 95.669
40 100.000 99.652 99.086 97.996 96.132
41 100.000 99.768 99.335 98.364 96.595
42 100.000 99.884 99.585 98.731 97.058
43 100.000 100.000 99.834 99.099 97.521
44 100.000 100.000 99.871 99.203 97.708
45 100.000 100.000 99.908 99.305 97.888
46 100.000 100.000 99.917 99.333 97.943
47 100.000 100.000 99.926 99.362 97.997
48 100.000 100.000 99.935 99.390 98.052
49 100.000 100.000 99.945 99.417 98.102
50 100.000 100.000 99.954 99.444 98.153
51 100.000 100.000 99.963 99.471 98.204
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.306
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 79.070 62.109 48.777 38.407 30.360
3 86.047 71.958 59.356 48.629 39.714
4 90.698 79.374 68.014 57.508 48.199
5 95.349 86.327 75.898 65.511 55.852
6 95.349 87.370 78.095 68.622 59.549
7 95.349 88.181 79.830 71.124 62.579
8 95.349 88.760 81.169 73.159 65.140
9 100.000 95.133 87.898 79.628 71.093
10 100.000 95.829 89.338 81.663 73.534
11 100.000 96.408 90.557 83.410 75.656
12 100.000 96.987 91.775 85.156 77.779
13 100.000 97.451 92.781 86.637 79.619
14 100.000 97.683 93.363 87.595 80.921
15 100.000 97.798 93.714 88.240 81.862
16 100.000 98.030 94.277 89.147 83.075
17 100.000 98.030 94.397 89.481 83.662
18 100.000 98.146 94.738 90.099 84.558
19 100.000 98.262 95.071 90.692 85.407
20 100.000 98.262 95.191 91.023 85.987
21 100.000 98.494 95.754 91.921 87.171
22 100.000 98.494 95.864 92.227 87.708
23 100.000 98.494 95.975 92.533 88.246
24 100.000 98.725 96.474 93.276 89.193
25 100.000 98.957 96.972 94.014 90.127
26 100.000 98.957 97.028 94.178 90.432
27 100.000 98.957 97.083 94.339 90.730
28 100.000 99.073 97.360 94.789 91.346
29 100.000 99.073 97.415 94.947 91.631
30 100.000 99.073 97.480 95.127 91.951
31 100.000 99.073 97.591 95.422 92.454
32 100.000 99.073 97.637 95.556 92.699
33 100.000 99.189 97.905 95.982 93.275
34 100.000 99.189 97.960 96.144 93.574
35 100.000 99.305 98.228 96.567 94.140
36 100.000 99.305 98.283 96.726 94.430
37 100.000 99.421 98.532 97.100 94.913
38 100.000 99.537 98.791 97.494 95.423
39 100.000 99.537 98.837 97.628 95.669
40 100.000 99.652 99.086 97.998 96.139
41 100.000 99.768 99.335 98.366 96.603
42 100.000 99.884 99.585 98.734 97.065
43 100.000 99.884 99.622 98.838 97.253
44 100.000 100.000 99.871 99.204 97.712
45 100.000 100.000 99.908 99.306 97.891
46 100.000 100.000 99.917 99.334 97.946
47 100.000 100.000 99.926 99.363 98.001
48 100.000 100.000 99.935 99.390 98.052
49 100.000 100.000 99.945 99.417 98.103
50 100.000 100.000 99.954 99.444 98.154
51 100.000 100.000 99.963 99.471 98.204
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.306
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 86.047 69.757 55.183 43.241 33.810
3 90.698 77.520 64.405 52.735 42.856
4 95.349 84.820 72.833 61.307 51.003
5 100.000 91.425 80.070 68.500 57.802
6 100.000 93.048 83.264 72.769 62.629
7 100.000 94.206 85.544 75.832 66.127
8 100.000 95.133 87.437 78.456 69.205
9 100.000 95.944 89.117 80.815 72.003
10 100.000 96.756 90.778 83.131 74.739
11 100.000 97.219 91.849 84.772 76.828
12 100.000 97.567 92.615 85.911 78.255
13 100.000 97.914 93.400 87.101 79.766
14 100.000 98.146 93.972 88.031 81.015
15 100.000 98.378 94.517 88.875 82.102
16 100.000 98.610 95.108 89.841 83.390
17 100.000 98.841 95.671 90.744 84.591
18 100.000 98.957 96.031 91.392 85.504
19 100.000 99.073 96.354 91.962 86.316
20 100.000 99.189 96.695 92.574 87.191
21 100.000 99.305 96.954 92.968 87.700
22 100.000 99.421 97.314 93.636 88.681
23 100.000 99.537 97.655 94.240 89.528
24 100.000 99.652 97.905 94.610 89.998
25 100.000 99.768 98.154 94.982 90.474
26 100.000 99.884 98.403 95.360 90.971
27 100.000 100.000 98.661 95.756 91.486
28 100.000 100.000 98.698 95.858 91.666
29 100.000 100.000 98.782 96.089 92.074
30 100.000 100.000 98.883 96.367 92.558
31 100.000 100.000 98.975 96.635 93.047
32 100.000 100.000 99.040 96.816 93.370
33 100.000 100.000 99.105 97.007 93.724
34 100.000 100.000 99.151 97.147 93.992
35 100.000 100.000 99.169 97.211 94.126
36 100.000 100.000 99.188 97.276 94.267
37 100.000 100.000 99.206 97.337 94.389
38 100.000 100.000 99.225 97.398 94.515
39 100.000 100.000 99.271 97.533 94.762
40 100.000 100.000 99.308 97.646 94.980
41 100.000 100.000 99.326 97.707 95.102
42 100.000 100.000 99.335 97.735 95.157
43 100.000 100.000 99.409 97.949 95.550
44 100.000 100.000 99.446 98.053 95.737
45 100.000 100.000 99.465 98.112 95.853
46 100.000 100.000 99.483 98.170 95.969
47 100.000 100.000 99.502 98.230 96.092
48 100.000 100.000 99.511 98.263 96.161
49 100.000 100.000 99.520 98.291 96.216
50 100.000 100.000 99.548 98.376 96.381
51 100.000 100.000 99.594 98.519 96.656
52 100.000 100.000 99.612 98.575 96.764
53 100.000 100.000 99.668 98.734 97.051
54 100.000 100.000 99.677 98.767 97.124
55 100.000 100.000 99.686 98.799 97.194
56 100.000 100.000 99.686 98.810 97.231
57 100.000 100.000 99.695 98.841 97.298
58 100.000 100.000 99.723 98.927 97.466
59 100.000 100.000 99.742 98.983 97.575
60 100.000 100.000 99.760 99.047 97.708
61 100.000 100.000 99.778 99.107 97.832
62 100.000 100.000 99.788 99.139 97.897
63 100.000 100.000 99.797 99.170 97.963
64 100.000 100.000 99.806 99.204 98.041
65 100.000 100.000 99.825 99.262 98.153
66 100.000 100.000 99.834 99.299 98.240
67 100.000 100.000 99.843 99.332 98.312
68 100.000 100.000 99.852 99.359 98.364
69 100.000 100.000 99.852 99.366 98.386
70 100.000 100.000 99.852 99.372 98.410
71 100.000 100.000 99.871 99.427 98.515
72 100.000 100.000 99.880 99.457 98.577
73 100.000 100.000 99.880 99.473 98.632
74 100.000 100.000 99.889 99.501 98.689
75 100.000 100.000 99.908 99.554 98.786
76 100.000 100.000 99.917 99.583 98.845
77 100.000 100.000 99.917 99.588 98.860
78 100.000 100.000 99.917 99.599 98.901
79 100.000 100.000 99.926 99.633 98.978
80 100.000 100.000 99.935 99.660 99.029
81 100.000 100.000 99.945 99.689 99.083
82 100.000 100.000 99.945 99.700 99.125
83 100.000 100.000 99.945 99.704 99.140
84 100.000 100.000 99.945 99.710 99.163
85 100.000 100.000 99.945 99.718 99.189
86 100.000 100.000 99.954 99.745 99.240
87 100.000 100.000 99.954 99.748 99.253
88 100.000 100.000 99.954 99.754 99.275
89 100.000 100.000 99.954 99.760 99.298
90 100.000 100.000 99.963 99.790 99.357
91 100.000 100.000 99.963 99.796 99.379
92 100.000 100.000 99.963 99.804 99.409
93 100.000 100.000 99.963 99.810 99.432
94 100.000 100.000 99.963 99.816 99.451
95 100.000 100.000 99.963 99.821 99.470
96 100.000 100.000 99.972 99.848 99.521
97 100.000 100.000 99.982 99.875 99.572
98 100.000 100.000 99.982 99.883 99.603
99 100.000 100.000 99.982 99.890 99.626
100 100.000 100.000 99.982 99.895 99.645
101 100.000 100.000 99.982 99.900 99.664
102 100.000 100.000 99.991 99.930 99.726
103 100.000 100.000 99.991 99.934 99.742
104 100.000 100.000 99.991 99.944 99.775
105 100.000 100.000 99.991 99.949 99.794
106 100.000 100.000 100.000 99.976 99.844
107 100.000 100.000 100.000 99.979 99.856
108 100.000 100.000 100.000 99.985 99.879
109 100.000 100.000 100.000 99.993 99.906
110 100.000 100.000 100.000 99.997 99.922
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.503 71.125 60.228 50.409
5 97.674 89.224 78.593 67.727 57.546
6 97.674 90.498 81.215 71.361 61.782
7 100.000 94.206 85.646 76.075 66.494
8 100.000 95.249 87.760 78.985 69.883
9 100.000 95.944 89.255 81.158 72.540
10 100.000 96.756 90.907 83.454 75.250
11 100.000 97.219 91.969 85.072 77.303
12 100.000 97.567 92.809 86.401 79.038
13 100.000 97.798 93.427 87.444 80.464
14 100.000 98.146 94.194 88.587 81.900
15 100.000 98.262 94.581 89.324 82.989
16 100.000 98.494 95.135 90.196 84.126
17 100.000 98.610 95.486 90.829 85.030
18 100.000 98.841 96.012 91.632 86.059
19 100.000 98.957 96.354 92.246 86.937
20 100.000 99.073 96.695 92.851 87.788
21 100.000 99.073 96.815 93.184 88.375
22 100.000 99.073 96.926 93.495 88.926
23 100.000 99.189 97.212 93.963 89.563
24 100.000 99.305 97.489 94.413 90.179
25 100.000 99.421 97.766 94.868 90.812
26 100.000 99.421 97.849 95.114 91.268
27 100.000 99.421 97.932 95.351 91.693
28 100.000 99.537 98.191 95.748 92.216
29 100.000 99.652 98.449 96.147 92.742
30 100.000 99.652 98.514 96.330 93.069
31 100.000 99.768 98.763 96.699 93.539
32 100.000 99.884 99.012 97.082 94.049
33 100.000 100.000 99.262 97.448 94.508
34 100.000 100.000 99.326 97.635 94.849
35 100.000 100.000 99.372 97.773 95.109
36 100.000 100.000 99.409 97.880 95.308
37 100.000 100.000 99.446 97.986 95.500
38 100.000 100.000 99.483 98.103 95.733
39 100.000 100.000 99.520 98.211 95.931
40 100.000 100.000 99.548 98.304 96.123
41 100.000 100.000 99.575 98.385 96.270
42 100.000 100.000 99.603 98.472 96.442
43 100.000 100.000 99.631 98.551 96.586
44 100.000 100.000 99.658 98.631 96.734
45 100.000 100.000 99.686 98.720 96.911
46 100.000 100.000 99.705 98.781 97.037
47 100.000 100.000 99.723 98.846 97.172
48 100.000 100.000 99.742 98.907 97.299
49 100.000 100.000 99.760 98.970 97.428
50 100.000 100.000 99.778 99.027 97.540
51 100.000 100.000 99.797 99.087 97.660
52 100.000 100.000 99.815 99.147 97.783
53 100.000 100.000 99.825 99.180 97.856
54 100.000 100.000 99.834 99.218 97.944
55 100.000 100.000 99.843 99.247 98.002
56 100.000 100.000 99.852 99.278 98.068
57 100.000 100.000 99.862 99.314 98.148
58 100.000 100.000 99.871 99.343 98.206
59 100.000 100.000 99.880 99.370 98.257
60 100.000 100.000 99.889 99.403 98.330
61 100.000 100.000 99.898 99.442 98.420
62 100.000 100.000 99.908 99.469 98.471
63 100.000 100.000 99.917 99.501 98.540
64 100.000 100.000 99.926 99.528 98.591
65 100.000 100.000 99.935 99.555 98.642
66 100.000 100.000 99.945 99.583 98.697
67 100.000 100.000 99.954 99.619 98.776
68 100.000 100.000 99.963 99.653 98.853
69 100.000 100.000 99.972 99.681 98.908
70 100.000 100.000 99.982 99.708 98.958
71 100.000 100.000 99.991 99.735 99.009
72 100.000 100.000 100.000 99.763 99.060
73 100.000 100.000 100.000 99.771 99.090
74 100.000 100.000 100.000 99.780 99.124
75 100.000 100.000 100.000 99.791 99.161
76 100.000 100.000 100.000 99.797 99.184
77 100.000 100.000 100.000 99.804 99.211
78 100.000 100.000 100.000 99.811 99.237
79 100.000 100.000 100.000 99.818 99.260
80 100.000 100.000 100.000 99.824 99.284
81 100.000 100.000 100.000 99.836 99.328
82 100.000 100.000 100.000 99.844 99.355
83 100.000 100.000 100.000 99.850 99.378
84 100.000 100.000 100.000 99.860 99.415
85 100.000 100.000 100.000 99.869 99.444
86 100.000 100.000 100.000 99.874 99.465
87 100.000 100.000 100.000 99.881 99.491
88 100.000 100.000 100.000 99.885 99.507
89 100.000 100.000 100.000 99.896 99.546
90 100.000 100.000 100.000 99.900 99.561
91 100.000 100.000 100.000 99.903 99.574
92 100.000 100.000 100.000 99.906 99.586
93 100.000 100.000 100.000 99.913 99.609
94 100.000 100.000 100.000 99.917 99.625
95 100.000 100.000 100.000 99.920 99.637
96 100.000 100.000 100.000 99.927 99.664
97 100.000 100.000 100.000 99.932 99.683
98 100.000 100.000 100.000 99.938 99.703
99 100.000 100.000 100.000 99.943 99.723
100 100.000 100.000 100.000 99.950 99.749
101 100.000 100.000 100.000 99.956 99.772
102 100.000 100.000 100.000 99.960 99.788
103 100.000 100.000 100.000 99.966 99.807
104 100.000 100.000 100.000 99.969 99.818
105 100.000 100.000 100.000 99.976 99.845
106 100.000 100.000 100.000 99.982 99.869
107 100.000 100.000 100.000 99.986 99.884
108 100.000 100.000 100.000 99.988 99.889
109 100.000 100.000 100.000 99.994 99.912
110 100.000 100.000 100.000 99.997 99.923
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.503 71.125 60.228 50.409
5 97.674 89.108 78.399 67.511 57.346
6 97.674 90.498 81.169 71.255 61.628
7 97.674 91.425 83.135 74.060 64.987
8 100.000 95.133 87.584 78.814 69.754
9 100.000 95.713 88.877 80.744 72.165
10 100.000 96.524 90.538 83.065 74.914
11 100.000 96.987 91.600 84.684 76.973
12 100.000 97.219 92.227 85.751 78.435
13 100.000 97.683 93.261 87.292 80.357
14 100.000 98.030 94.046 88.486 81.883
15 100.000 98.378 94.831 89.674 83.391
16 100.000 98.378 94.978 90.079 84.096
17 100.000 98.378 95.108 90.439 84.728
18 100.000 98.494 95.468 91.099 85.683
19 100.000 98.610 95.800 91.690 86.525
20 100.000 98.725 96.114 92.236 87.302
21 100.000 98.725 96.215 92.520 87.804
22 100.000 98.957 96.723 93.281 88.774
23 100.000 99.073 97.000 93.736 89.407
24 100.000 99.073 97.083 93.972 89.829
25 100.000 99.189 97.360 94.424 90.452
26 100.000 99.305 97.628 94.853 91.038
27 100.000 99.421 97.895 95.280 91.617
28 100.000 99.421 97.941 95.422 91.891
29 100.000 99.537 98.218 95.866 92.486
30 100.000 99.537 98.265 96.008 92.759
31 100.000 99.537 98.311 96.147 93.024
32 100.000 99.537 98.348 96.264 93.251
33 100.000 99.537 98.403 96.422 93.539
34 100.000 99.537 98.440 96.537 93.761
35 100.000 99.537 98.477 96.649 93.975
36 100.000 99.652 98.726 97.021 94.452
37 100.000 99.652 98.763 97.133 94.663
38 100.000 99.652 98.809 97.264 94.900
39 100.000 99.768 99.058 97.634 95.371
40 100.000 99.768 99.095 97.742 95.572
41 100.000 99.884 99.345 98.110 96.036
42 100.000 100.000 99.594 98.476 96.496
43 100.000 100.000 99.612 98.541 96.633
44 100.000 100.000 99.631 98.604 96.765
45 100.000 100.000 99.658 98.689 96.926
46 100.000 100.000 99.686 98.771 97.081
47 100.000 100.000 99.695 98.812 97.178
48 100.000 100.000 99.714 98.872 97.301
49 100.000 100.000 99.723 98.912 97.395
50 100.000 100.000 99.751 98.993 97.547
51 100.000 100.000 99.760 99.030 97.634
52 100.000 100.000 99.778 99.089 97.749
53 100.000 100.000 99.797 99.147 97.866
54 100.000 100.000 99.815 99.205 97.983
55 100.000 100.000 99.825 99.242 98.066
56 100.000 100.000 99.843 99.298 98.175
57 100.000 100.000 99.852 99.333 98.255
58 100.000 100.000 99.871 99.388 98.356
59 100.000 100.000 99.871 99.399 98.397
60 100.000 100.000 99.880 99.431 98.467
61 100.000 100.000 99.880 99.443 98.509
62 100.000 100.000 99.889 99.474 98.575
63 100.000 100.000 99.898 99.505 98.642
64 100.000 100.000 99.898 99.516 98.681
65 100.000 100.000 99.908 99.546 98.744
66 100.000 100.000 99.908 99.555 98.778
67 100.000 100.000 99.908 99.564 98.808
68 100.000 100.000 99.908 99.572 98.839
69 100.000 100.000 99.908 99.580 98.870
70 100.000 100.000 99.908 99.589 98.900
71 100.000 100.000 99.908 99.596 98.927
72 100.000 100.000 99.917 99.624 98.982
73 100.000 100.000 99.917 99.631 99.008
74 100.000 100.000 99.926 99.659 99.062
75 100.000 100.000 99.935 99.687 99.113
76 100.000 100.000 99.935 99.693 99.137
77 100.000 100.000 99.945 99.720 99.188
78 100.000 100.000 99.954 99.747 99.239
79 100.000 100.000 99.954 99.753 99.262
80 100.000 100.000 99.954 99.759 99.285
81 100.000 100.000 99.954 99.766 99.308
82 100.000 100.000 99.963 99.793 99.358
83 100.000 100.000 99.972 99.820 99.409
84 100.000 100.000 99.982 99.847 99.460
85 100.000 100.000 99.991 99.874 99.511
86 100.000 100.000 100.000 99.901 99.561
87 100.000 100.000 100.000 99.906 99.581
88 100.000 100.000 100.000 99.911 99.600
89 100.000 100.000 100.000 99.917 99.620
90 100.000 100.000 100.000 99.922 99.639
91 100.000 100.000 100.000 99.927 99.659
92 100.000 100.000 100.000 99.932 99.679
93 100.000 100.000 100.000 99.938 99.699
94 100.000 100.000 100.000 99.942 99.714
95 100.000 100.000 100.000 99.946 99.730
96 100.000 100.000 100.000 99.950 99.747
97 100.000 100.000 100.000 99.954 99.762
98 100.000 100.000 100.000 99.958 99.777
99 100.000 100.000 100.000 99.963 99.793
100 100.000 100.000 100.000 99.967 99.808
101 100.000 100.000 100.000 99.971 99.825
102 100.000 100.000 100.000 99.974 99.836
103 100.000 100.000 100.000 99.977 99.848
104 100.000 100.000 100.000 99.980 99.860
105 100.000 100.000 100.000 99.983 99.872
106 100.000 100.000 100.000 99.986 99.884
107 100.000 100.000 100.000 99.990 99.895
108 100.000 100.000 100.000 99.993 99.907
109 100.000 100.000 100.000 99.996 99.918
110 100.000 100.000 100.000 99.999 99.930
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 79.070 62.109 48.777 38.407 30.360
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.374 68.014 57.508 48.199
5 93.023 83.314 72.925 62.880 53.659
6 95.349 87.486 78.252 68.754 59.628
7 97.674 91.194 82.756 73.646 64.612
8 97.674 91.889 84.252 75.820 67.271
9 100.000 95.597 88.655 80.467 71.882
10 100.000 96.524 90.529 83.049 74.900
11 100.000 96.987 91.554 84.578 76.819
12 100.000 97.219 92.209 85.710 78.380
13 100.000 97.683 93.215 87.193 80.227
14 100.000 98.030 94.018 88.434 81.828
15 100.000 98.262 94.581 89.341 83.042
16 100.000 98.262 94.729 89.742 83.734
17 100.000 98.262 94.877 90.140 84.416
18 100.000 98.262 94.969 90.404 84.895
19 100.000 98.262 95.052 90.645 85.335
20 100.000 98.262 95.154 90.926 85.830
21 100.000 98.494 95.717 91.823 87.010
22 100.000 98.725 96.252 92.653 88.089
23 100.000 98.841 96.520 93.084 88.683
24 100.000 98.957 96.815 93.584 89.384
25 100.000 99.073 97.083 94.015 89.978
26 100.000 99.073 97.175 94.272 90.429
27 100.000 99.073 97.231 94.436 90.737
28 100.000 99.073 97.295 94.622 91.074
29 100.000 99.073 97.341 94.763 91.346
30 100.000 99.189 97.600 95.163 91.877
31 100.000 99.189 97.665 95.350 92.216
32 100.000 99.305 97.923 95.750 92.747
33 100.000 99.305 97.969 95.889 93.011
34 100.000 99.305 98.006 96.003 93.230
35 100.000 99.305 98.043 96.112 93.436
36 100.000 99.305 98.080 96.222 93.645
37 100.000 99.305 98.089 96.265 93.749
38 100.000 99.305 98.117 96.354 93.925
39 100.000 99.421 98.366 96.726 94.405
40 100.000 99.537 98.615 97.098 94.883
41 100.000 99.652 98.865 97.469 95.357
42 100.000 99.652 98.883 97.533 95.490
43 100.000 99.652 98.902 97.596 95.622
44 100.000 99.652 98.911 97.636 95.716
45 100.000 99.652 98.920 97.677 95.815
46 100.000 99.652 98.948 97.760 95.971
47 100.000 99.652 98.966 97.820 96.093
48 100.000 99.652 98.994 97.900 96.242
49 100.000 99.652 99.012 97.958 96.355
50 100.000 99.652 99.012 97.968 96.393
51 100.000 99.652 99.012 97.976 96.425
52 100.000 99.652 99.022 98.011 96.501
53 100.000 99.652 99.040 98.067 96.611
54 100.000 99.768 99.289 98.434 97.070
55 100.000 99.884 99.538 98.799 97.527
56 100.000 99.884 99.548 98.836 97.609
57 100.000 99.884 99.557 98.871 97.689
58 100.000 100.000 99.806 99.237 98.145
59 100.000 100.000 99.825 99.292 98.251
60 100.000 100.000 99.834 99.324 98.320
61 100.000 100.000 99.843 99.355 98.387
62 100.000 100.000 99.852 99.385 98.450
63 100.000 100.000 99.862 99.416 98.513
64 100.000 100.000 99.862 99.425 98.547
65 100.000 100.000 99.862 99.432 98.574
66 100.000 100.000 99.862 99.442 98.607
67 100.000 100.000 99.862 99.448 98.631
68 100.000 100.000 99.880 99.502 98.733
69 100.000 100.000 99.889 99.534 98.802
70 100.000 100.000 99.889 99.542 98.829
71 100.000 100.000 99.889 99.549 98.855
72 100.000 100.000 99.889 99.554 98.875
73 100.000 100.000 99.889 99.560 98.898
74 100.000 100.000 99.889 99.565 98.915
75 100.000 100.000 99.889 99.571 98.939
76 100.000 100.000 99.908 99.625 99.040
77 100.000 100.000 99.908 99.631 99.063
78 100.000 100.000 99.908 99.639 99.089
79 100.000 100.000 99.908 99.644 99.108
80 100.000 100.000 99.908 99.649 99.128
81 100.000 100.000 99.908 99.654 99.147
82 100.000 100.000 99.908 99.660 99.170
83 100.000 100.000 99.908 99.666 99.189
84 100.000 100.000 99.908 99.671 99.209
85 100.000 100.000 99.908 99.675 99.225
86 100.000 100.000 99.908 99.679 99.240
87 100.000 100.000 99.908 99.683 99.256
88 100.000 100.000 99.908 99.688 99.272
89 100.000 100.000 99.908 99.692 99.287
90 100.000 100.000 99.908 99.696 99.302
91 100.000 100.000 99.908 99.700 99.318
92 100.000 100.000 99.908 99.703 99.330
93 100.000 100.000 99.908 99.706 99.342
94 100.000 100.000 99.917 99.733 99.393
95 100.000 100.000 99.917 99.736 99.404
96 100.000 100.000 99.926 99.764 99.455
97 100.000 100.000 99.935 99.791 99.506
98 100.000 100.000 99.945 99.818 99.557
99 100.000 100.000 99.954 99.845 99.608
100 100.000 100.000 99.954 99.848 99.619
101 100.000 100.000 99.963 99.875 99.670
102 100.000 100.000 99.972 99.902 99.721
103 100.000 100.000 99.972 99.905 99.732
104 100.000 100.000 99.972 99.908 99.744
105 100.000 100.000 99.982 99.935 99.795
106 100.000 100.000 99.991 99.963 99.845
107 100.000 100.000 100.000 99.990 99.896
108 100.000 100.000 100.000 99.993 99.907
109 100.000 100.000 100.000 99.996 99.919
110 100.000 100.000 100.000 99.999 99.930
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 90.698 80.185 69.371 59.041 49.654
5 95.349 86.790 76.645 66.321 56.585
6 100.000 93.163 83.486 73.051 62.925
7 100.000 94.322 85.876 76.378 66.822
8 100.000 95.365 87.935 79.158 70.018
9 100.000 96.176 89.597 81.480 72.770
10 100.000 96.871 90.954 83.288 74.819
11 100.000 97.335 91.997 84.858 76.797
12 100.000 97.683 92.754 85.966 78.159
13 100.000 97.914 93.298 86.817 79.267
14 100.000 98.146 93.908 87.839 80.660
15 100.000 98.378 94.498 88.802 81.938
16 100.000 98.610 95.080 89.746 83.196
17 100.000 98.841 95.661 90.688 84.445
18 100.000 98.957 95.920 91.085 84.964
19 100.000 99.073 96.243 91.648 85.757
20 100.000 99.189 96.612 92.332 86.750
21 100.000 99.305 96.889 92.779 87.355
22 100.000 99.421 97.166 93.223 87.950
23 100.000 99.537 97.415 93.588 88.405
24 100.000 99.652 97.665 93.963 88.892
25 100.000 99.768 97.960 94.456 89.569
26 100.000 99.884 98.218 94.854 90.091
27 100.000 100.000 98.468 95.225 90.564
28 100.000 100.000 98.495 95.305 90.711
29 100.000 100.000 98.615 95.634 91.284
30 100.000 100.000 98.745 95.990 91.906
31 100.000 100.000 98.818 96.202 92.289
32 100.000 100.000 98.883 96.382 92.608
33 100.000 100.000 98.929 96.520 92.869
34 100.000 100.000 98.975 96.655 93.116
35 100.000 100.000 99.012 96.766 93.327
36 100.000 100.000 99.058 96.899 93.571
37 100.000 100.000 99.105 97.033 93.815
38 100.000 100.000 99.142 97.147 94.037
39 100.000 100.000 99.151 97.180 94.108
40 100.000 100.000 99.188 97.286 94.302
41 100.000 100.000 99.262 97.501 94.699
42 100.000 100.000 99.280 97.569 94.845
43 100.000 100.000 99.298 97.633 94.978
44 100.000 100.000 99.326 97.718 95.144
45 100.000 100.000 99.326 97.733 95.195
46 100.000 100.000 99.354 97.816 95.355
47 100.000 100.000 99.382 97.898 95.511
48 100.000 100.000 99.428 98.026 95.737
49 100.000 100.000 99.465 98.137 95.946
50 100.000 100.000 99.483 98.194 96.058
51 100.000 100.000 99.511 98.276 96.213
52 100.000 100.000 99.511 98.285 96.243
53 100.000 100.000 99.520 98.325 96.340
54 100.000 100.000 99.529 98.354 96.399
55 100.000 100.000 99.548 98.409 96.500
56 100.000 100.000 99.594 98.543 96.747
57 100.000 100.000 99.594 98.548 96.768
58 100.000 100.000 99.603 98.577 96.827
59 100.000 100.000 99.631 98.664 96.996
60 100.000 100.000 99.658 98.748 97.158
61 100.000 100.000 99.686 98.828 97.305
62 100.000 100.000 99.695 98.864 97.386
63 100.000 100.000 99.695 98.868 97.404
64 100.000 100.000 99.705 98.898 97.467
65 100.000 100.000 99.714 98.926 97.521
66 100.000 100.000 99.714 98.929 97.533
67 100.000 100.000 99.732 98.990 97.656
68 100.000 100.000 99.751 99.053 97.789
69 100.000 100.000 99.760 99.086 97.858
70 100.000 100.000 99.760 99.089 97.869
71 100.000 100.000 99.760 99.098 97.903
72 100.000 100.000 99.778 99.153 98.008
73 100.000 100.000 99.788 99.180 98.059
74 100.000 100.000 99.797 99.213 98.129
75 100.000 100.000 99.806 99.242 98.188
76 100.000 100.000 99.825 99.295 98.286
77 100.000 100.000 99.834 99.322 98.336
78 100.000 100.000 99.843 99.350 98.391
79 100.000 100.000 99.843 99.354 98.407
80 100.000 100.000 99.852 99.381 98.457
81 100.000 100.000 99.862 99.410 98.516
82 100.000 100.000 99.862 99.413 98.524
83 100.000 100.000 99.871 99.448 98.604
84 100.000 100.000 99.871 99.454 98.627
85 100.000 100.000 99.880 99.481 98.677
86 100.000 100.000 99.880 99.483 98.686
87 100.000 100.000 99.880 99.488 98.702
88 100.000 100.000 99.880 99.488 98.702
89 100.000 100.000 99.889 99.518 98.764
90 100.000 100.000 99.908 99.579 98.891
91 100.000 100.000 99.908 99.583 98.907
92 100.000 100.000 99.908 99.587 98.918
93 100.000 100.000 99.908 99.591 98.934
94 100.000 100.000 99.917 99.622 99.000
95 100.000 100.000 99.935 99.677 99.106
96 100.000 100.000 99.935 99.677 99.106
97 100.000 100.000 99.935 99.688 99.143
98 100.000 100.000 99.935 99.691 99.155
99 100.000 100.000 99.945 99.718 99.206
100 100.000 100.000 99.945 99.721 99.218
101 100.000 100.000 99.945 99.724 99.229
102 100.000 100.000 99.945 99.724 99.230
103 100.000 100.000 99.945 99.729 99.249
104 100.000 100.000 99.945 99.733 99.265
105 100.000 100.000 99.954 99.761 99.321
106 100.000 100.000 99.954 99.766 99.336
107 100.000 100.000 99.954 99.768 99.344
108 100.000 100.000 99.954 99.769 99.348
109 100.000 100.000 99.954 99.773 99.364
110 100.000 100.000 99.963 99.801 99.420
111 100.000 100.000 99.963 99.805 99.435
112 100.000 100.000 99.963 99.806 99.440
113 100.000 100.000 99.963 99.809 99.452
114 100.000 100.000 99.963 99.813 99.464
115 100.000 100.000 99.963 99.815 99.473
116 100.000 100.000 99.963 99.819 99.488
117 100.000 100.000 99.963 99.821 99.496
118 100.000 100.000 99.963 99.822 99.500
119 100.000 100.000 99.963 99.826 99.516
120 100.000 100.000 99.963 99.828 99.525
121 100.000 100.000 99.963 99.829 99.530
122 100.000 100.000 99.963 99.833 99.545
123 100.000 100.000 99.963 99.834 99.550
124 100.000 100.000 99.972 99.861 99.601
125 100.000 100.000 99.972 99.861 99.603
126 100.000 100.000 99.972 99.861 99.603
127 100.000 100.000 99.972 99.863 99.608
128 100.000 100.000 99.982 99.891 99.662
129 100.000 100.000 99.982 99.894 99.675
130 100.000 100.000 99.982 99.897 99.687
131 100.000 100.000 99.982 99.899 99.695
132 100.000 100.000 99.982 99.899 99.696
133 100.000 100.000 99.982 99.903 99.712
134 100.000 100.000 99.982 99.905 99.720
135 100.000 100.000 99.982 99.907 99.728
136 100.000 100.000 99.982 99.908 99.733
137 100.000 100.000 99.982 99.911 99.745
138 100.000 100.000 99.982 99.914 99.753
139 100.000 100.000 99.982 99.916 99.761
140 100.000 100.000 99.982 99.919 99.772
141 100.000 100.000 99.982 99.919 99.773
142 100.000 100.000 99.982 99.920 99.777
143 100.000 100.000 99.982 99.922 99.785
144 100.000 100.000 99.982 99.924 99.794
145 100.000 100.000 99.982 99.924 99.796
146 100.000 100.000 99.982 99.926 99.804
147 100.000 100.000 99.991 99.953 99.854
148 100.000 100.000 99.991 99.953 99.855
149 100.000 100.000 99.991 99.955 99.863
150 100.000 100.000 99.991 99.956 99.867
151 100.000 100.000 99.991 99.956 99.867
152 100.000 100.000 99.991 99.958 99.875
153 100.000 100.000 99.991 99.959 99.880
154 100.000 100.000 99.991 99.959 99.881
155 100.000 100.000 99.991 99.961 99.889
156 100.000 100.000 99.991 99.964 99.896
157 100.000 100.000 99.991 99.964 99.898
158 100.000 100.000 99.991 99.965 99.902
159 100.000 100.000 99.991 99.965 99.903
160 100.000 100.000 99.991 99.965 99.904
161 100.000 100.000 99.991 99.965 99.905
162 100.000 100.000 99.991 99.967 99.913
163 100.000 100.000 99.991 99.968 99.918
164 100.000 100.000 99.991 99.969 99.922
165 100.000 100.000 99.991 99.969 99.923
166 100.000 100.000 99.991 99.969 99.924
167 100.000 100.000 100.000 99.996 99.975
168 100.000 100.000 100.000 99.997 99.979
169 100.000 100.000 100.000 99.998 99.984
170 100.000 100.000 100.000 99.998 99.984
171 100.000 100.000 100.000 99.998 99.985
172 100.000 100.000 100.000 99.998 99.986
173 100.000 100.000 100.000 99.998 99.987
174 100.000 100.000 100.000 99.998 99.988
175 100.000 100.000 100.000 99.999 99.992
176 100.000 100.000 100.000 99.999 99.993
177 100.000 100.000 100.000 99.999 99.994
178 100.000 100.000 100.000 100.000 99.998
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.619 71.291 60.381 50.518
5 95.349 86.559 76.230 65.817 56.069
6 97.674 90.382 80.947 70.967 61.313
7 97.674 91.425 83.153 74.099 65.039
8 100.000 95.249 87.778 79.027 69.942
9 100.000 96.176 89.661 81.632 72.997
10 100.000 96.871 91.138 83.755 75.571
11 100.000 97.219 92.015 85.177 77.453
12 100.000 97.567 92.837 86.465 79.132
13 100.000 97.914 93.658 87.745 80.785
14 100.000 98.146 94.258 88.733 82.106
15 100.000 98.378 94.849 89.712 83.437
16 100.000 98.378 95.034 90.202 84.266
17 100.000 98.610 95.588 91.081 85.426
18 100.000 98.725 95.938 91.726 86.367
19 100.000 98.841 96.280 92.330 87.214
20 100.000 98.957 96.612 92.923 88.063
21 100.000 99.189 97.111 93.656 88.981
22 100.000 99.305 97.406 94.158 89.689
23 100.000 99.305 97.498 94.418 90.154
24 100.000 99.305 97.581 94.650 90.562
25 100.000 99.421 97.840 95.051 91.095
26 100.000 99.537 98.098 95.446 91.611
27 100.000 99.537 98.172 95.659 91.996
28 100.000 99.652 98.421 96.026 92.456
29 100.000 99.768 98.671 96.391 92.911
30 100.000 99.884 98.920 96.768 93.404
31 100.000 99.884 98.985 96.955 93.743
32 100.000 100.000 99.234 97.322 94.206
33 100.000 100.000 99.289 97.480 94.488
34 100.000 100.000 99.335 97.616 94.742
35 100.000 100.000 99.382 97.758 95.013
36 100.000 100.000 99.428 97.900 95.288
37 100.000 100.000 99.465 98.020 95.527
38 100.000 100.000 99.502 98.136 95.753
39 100.000 100.000 99.538 98.240 95.940
40 100.000 100.000 99.575 98.355 96.167
41 100.000 100.000 99.612 98.460 96.354
42 100.000 100.000 99.649 98.570 96.563
43 100.000 100.000 99.686 98.684 96.782
44 100.000 100.000 99.723 98.791 96.980
45 100.000 100.000 99.751 98.878 97.151
46 100.000 100.000 99.778 98.961 97.307
47 100.000 100.000 99.806 99.051 97.489
48 100.000 100.000 99.834 99.139 97.663
49 100.000 100.000 99.852 99.196 97.775
50 100.000 100.000 99.871 99.254 97.892
51 100.000 100.000 99.880 99.281 97.943
52 100.000 100.000 99.889 99.316 98.019
53 100.000 100.000 99.898 99.351 98.100
54 100.000 100.000 99.908 99.382 98.167
55 100.000 100.000 99.917 99.409 98.218
56 100.000 100.000 99.926 99.438 98.273
57 100.000 100.000 99.935 99.475 98.360
58 100.000 100.000 99.945 99.505 98.421
59 100.000 100.000 99.954 99.532 98.473
60 100.000 100.000 99.963 99.560 98.527
61 100.000 100.000 99.972 99.588 98.578
62 100.000 100.000 99.982 99.616 98.632
63 100.000 100.000 99.991 99.643 98.683
64 100.000 100.000 100.000 99.671 98.737
65 100.000 100.000 100.000 99.675 98.752
66 100.000 100.000 100.000 99.685 98.789
67 100.000 100.000 100.000 99.694 98.819
68 100.000 100.000 100.000 99.699 98.839
69 100.000 100.000 100.000 99.708 98.873
70 100.000 100.000 100.000 99.717 98.904
71 100.000 100.000 100.000 99.726 98.938
72 100.000 100.000 100.000 99.736 98.975
73 100.000 100.000 100.000 99.743 98.997
74 100.000 100.000 100.000 99.748 99.017
75 100.000 100.000 100.000 99.753 99.036
76 100.000 100.000 100.000 99.759 99.059
77 100.000 100.000 100.000 99.761 99.067
78 100.000 100.000 100.000 99.767 99.087
79 100.000 100.000 100.000 99.772 99.106
80 100.000 100.000 100.000 99.779 99.132
81 100.000 100.000 100.000 99.788 99.162
82 100.000 100.000 100.000 99.793 99.182
83 100.000 100.000 100.000 99.799 99.205
84 100.000 100.000 100.000 99.803 99.222
85 100.000 100.000 100.000 99.806 99.233
86 100.000 100.000 100.000 99.816 99.267
87 100.000 100.000 100.000 99.823 99.293
88 100.000 100.000 100.000 99.828 99.313
89 100.000 100.000 100.000 99.832 99.329
90 100.000 100.000 100.000 99.838 99.348
91 100.000 100.000 100.000 99.841 99.360
92 100.000 100.000 100.000 99.846 99.380
93 100.000 100.000 100.000 99.851 99.399
94 100.000 100.000 100.000 99.855 99.415
95 100.000 100.000 100.000 99.856 99.419
96 100.000 100.000 100.000 99.860 99.435
97 100.000 100.000 100.000 99.865 99.450
98 100.000 100.000 100.000 99.869 99.466
99 100.000 100.000 100.000 99.871 99.474
100 100.000 100.000 100.000 99.877 99.497
101 100.000 100.000 100.000 99.880 99.508
102 100.000 100.000 100.000 99.885 99.528
103 100.000 100.000 100.000 99.890 99.544
104 100.000 100.000 100.000 99.894 99.559
105 100.000 100.000 100.000 99.897 99.571
106 100.000 100.000 100.000 99.900 99.583
107 100.000 100.000 100.000 99.901 99.587
108 100.000 100.000 100.000 99.902 99.591
109 100.000 100.000 100.000 99.905 99.604
110 100.000 100.000 100.000 99.908 99.616
111 100.000 100.000 100.000 99.913 99.632
112 100.000 100.000 100.000 99.915 99.641
113 100.000 100.000 100.000 99.916 99.646
114 100.000 100.000 100.000 99.916 99.648
115 100.000 100.000 100.000 99.919 99.660
116 100.000 100.000 100.000 99.922 99.671
117 100.000 100.000 100.000 99.924 99.679
118 100.000 100.000 100.000 99.927 99.691
119 100.000 100.000 100.000 99.928 99.696
120 100.000 100.000 100.000 99.934 99.719
121 100.000 100.000 100.000 99.938 99.731
122 100.000 100.000 100.000 99.940 99.739
123 100.000 100.000 100.000 99.941 99.744
124 100.000 100.000 100.000 99.944 99.756
125 100.000 100.000 100.000 99.944 99.757
126 100.000 100.000 100.000 99.946 99.765
127 100.000 100.000 100.000 99.947 99.769
128 100.000 100.000 100.000 99.947 99.770
129 100.000 100.000 100.000 99.948 99.775
130 100.000 100.000 100.000 99.948 99.775
131 100.000 100.000 100.000 99.950 99.783
132 100.000 100.000 100.000 99.951 99.788
133 100.000 100.000 100.000 99.955 99.803
134 100.000 100.000 100.000 99.959 99.819
135 100.000 100.000 100.000 99.960 99.824
136 100.000 100.000 100.000 99.963 99.832
137 100.000 100.000 100.000 99.963 99.833
138 100.000 100.000 100.000 99.964 99.837
139 100.000 100.000 100.000 99.966 99.846
140 100.000 100.000 100.000 99.967 99.851
141 100.000 100.000 100.000 99.969 99.859
142 100.000 100.000 100.000 99.970 99.864
143 100.000 100.000 100.000 99.970 99.864
144 100.000 100.000 100.000 99.971 99.869
145 100.000 100.000 100.000 99.973 99.877
146 100.000 100.000 100.000 99.974 99.881
147 100.000 100.000 100.000 99.977 99.893
148 100.000 100.000 100.000 99.980 99.904
149 100.000 100.000 100.000 99.982 99.912
150 100.000 100.000 100.000 99.982 99.913
151 100.000 100.000 100.000 99.983 99.918
152 100.000 100.000 100.000 99.985 99.926
153 100.000 100.000 100.000 99.988 99.934
154 100.000 100.000 100.000 99.991 99.945
155 100.000 100.000 100.000 99.991 99.946
156 100.000 100.000 100.000 99.992 99.952
157 100.000 100.000 100.000 99.994 99.959
158 100.000 100.000 100.000 99.996 99.967
159 100.000 100.000 100.000 99.997 99.972
160 100.000 100.000 100.000 99.997 99.972
161 100.000 100.000 100.000 99.997 99.973
162 100.000 100.000 100.000 99.997 99.974
163 100.000 100.000 100.000 99.997 99.975
164 100.000 100.000 100.000 99.997 99.977
165 100.000 100.000 100.000 99.997 99.978
166 100.000 100.000 100.000 99.997 99.979
167 100.000 100.000 100.000 99.997 99.980
168 100.000 100.000 100.000 99.998 99.985
169 100.000 100.000 100.000 99.998 99.986
170 100.000 100.000 100.000 99.999 99.990
171 100.000 100.000 100.000 99.999 99.991
172 100.000 100.000 100.000 99.999 99.992
173 100.000 100.000 100.000 99.999 99.992
174 100.000 100.000 100.000 99.999 99.993
175 100.000 100.000 100.000 99.999 99.993
176 100.000 100.000 100.000 99.999 99.994
177 100.000 100.000 100.000 100.000 99.998
178 100.000 100.000 100.000 100.000 99.999
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.722 68.596 58.165 48.823
5 95.349 86.327 75.870 65.445 55.754
6 97.674 90.267 80.809 70.881 61.305
7 97.674 90.846 82.110 72.833 63.749
8 100.000 94.322 86.135 77.062 67.971
9 100.000 95.017 87.630 79.227 70.602
10 100.000 96.060 89.689 82.005 73.792
11 100.000 96.640 90.907 83.753 75.921
12 100.000 97.103 91.951 85.325 77.902
13 100.000 97.567 92.966 86.827 79.771
14 100.000 97.914 93.788 88.109 81.431
15 100.000 98.030 94.175 88.844 82.513
16 100.000 98.146 94.517 89.467 83.422
17 100.000 98.262 94.849 90.067 84.294
18 100.000 98.494 95.384 90.898 85.376
19 100.000 98.725 95.920 91.727 86.451
20 100.000 98.725 96.012 91.993 86.934
21 100.000 98.957 96.529 92.778 87.944
22 100.000 99.073 96.852 93.343 88.740
23 100.000 99.189 97.157 93.864 89.474
24 100.000 99.189 97.240 94.099 89.892
25 100.000 99.305 97.526 94.572 90.544
26 100.000 99.305 97.609 94.805 90.960
27 100.000 99.305 97.692 95.036 91.368
28 100.000 99.305 97.757 95.226 91.718
29 100.000 99.305 97.821 95.413 92.062
30 100.000 99.305 97.886 95.599 92.399
31 100.000 99.305 97.932 95.737 92.661
32 100.000 99.305 97.988 95.894 92.945
33 100.000 99.305 98.034 96.031 93.199
34 100.000 99.421 98.292 96.427 93.715
35 100.000 99.421 98.320 96.519 93.904
36 100.000 99.421 98.348 96.610 94.087
37 100.000 99.537 98.597 96.982 94.564
38 100.000 99.537 98.615 97.050 94.713
39 100.000 99.537 98.652 97.161 94.923
40 100.000 99.652 98.902 97.531 95.394
41 100.000 99.652 98.938 97.639 95.596
42 100.000 99.768 99.188 98.008 96.062
43 100.000 99.884 99.437 98.376 96.529
44 100.000 99.884 99.474 98.482 96.724
45 100.000 99.884 99.502 98.568 96.891
46 100.000 100.000 99.751 98.933 97.347
47 100.000 100.000 99.769 98.997 97.480
48 100.000 100.000 99.806 99.101 97.667
49 100.000 100.000 99.825 99.163 97.794
50 100.000 100.000 99.843 99.224 97.920
51 100.000 100.000 99.852 99.262 98.007
52 100.000 100.000 99.871 99.319 98.120
53 100.000 100.000 99.880 99.354 98.200
54 100.000 100.000 99.898 99.408 98.302
55 100.000 100.000 99.898 99.421 98.348
56 100.000 100.000 99.898 99.432 98.388
57 100.000 100.000 99.908 99.465 98.457
58 100.000 100.000 99.908 99.476 98.497
59 100.000 100.000 99.908 99.487 98.533
60 100.000 100.000 99.908 99.496 98.567
61 100.000 100.000 99.908 99.504 98.597
62 100.000 100.000 99.908 99.513 98.627
63 100.000 100.000 99.908 99.520 98.654
64 100.000 100.000 99.908 99.527 98.680
65 100.000 100.000 99.908 99.534 98.707
66 100.000 100.000 99.908 99.541 98.730
67 100.000 100.000 99.917 99.568 98.780
68 100.000 100.000 99.917 99.574 98.803
69 100.000 100.000 99.926 99.601 98.854
70 100.000 100.000 99.926 99.607 98.877
71 100.000 100.000 99.935 99.634 98.928
72 100.000 100.000 99.945 99.662 98.979
73 100.000 100.000 99.945 99.668 99.002
74 100.000 100.000 99.954 99.695 99.053
75 100.000 100.000 99.954 99.701 99.076
76 100.000 100.000 99.963 99.728 99.127
77 100.000 100.000 99.972 99.755 99.177
78 100.000 100.000 99.982 99.782 99.229
79 100.000 100.000 99.991 99.809 99.280
80 100.000 100.000 99.991 99.816 99.303
81 100.000 100.000 100.000 99.843 99.354
82 100.000 100.000 100.000 99.848 99.374
83 100.000 100.000 100.000 99.853 99.393
84 100.000 100.000 100.000 99.858 99.413
85 100.000 100.000 100.000 99.864 99.432
86 100.000 100.000 100.000 99.869 99.451
87 100.000 100.000 100.000 99.873 99.467
88 100.000 100.000 100.000 99.877 99.482
89 100.000 100.000 100.000 99.881 99.498
90 100.000 100.000 100.000 99.885 99.514
91 100.000 100.000 100.000 99.890 99.530
92 100.000 100.000 100.000 99.894 99.547
93 100.000 100.000 100.000 99.898 99.562
94 100.000 100.000 100.000 99.902 99.578
95 100.000 100.000 100.000 99.906 99.593
96 100.000 100.000 100.000 99.910 99.609
97 100.000 100.000 100.000 99.915 99.624
98 100.000 100.000 100.000 99.918 99.636
99 100.000 100.000 100.000 99.921 99.648
100 100.000 100.000 100.000 99.924 99.660
101 100.000 100.000 100.000 99.927 99.671
102 100.000 100.000 100.000 99.930 99.684
103 100.000 100.000 100.000 99.933 99.696
104 100.000 100.000 100.000 99.936 99.708
105 100.000 100.000 100.000 99.940 99.721
106 100.000 100.000 100.000 99.943 99.733
107 100.000 100.000 100.000 99.945 99.742
108 100.000 100.000 100.000 99.947 99.750
109 100.000 100.000 100.000 99.949 99.758
110 100.000 100.000 100.000 99.951 99.766
111 100.000 100.000 100.000 99.953 99.774
112 100.000 100.000 100.000 99.955 99.782
113 100.000 100.000 100.000 99.957 99.790
114 100.000 100.000 100.000 99.959 99.799
115 100.000 100.000 100.000 99.961 99.807
116 100.000 100.000 100.000 99.964 99.815
117 100.000 100.000 100.000 99.966 99.823
118 100.000 100.000 100.000 99.968 99.831
119 100.000 100.000 100.000 99.970 99.840
120 100.000 100.000 100.000 99.972 99.848
121 100.000 100.000 100.000 99.974 99.857
122 100.000 100.000 100.000 99.976 99.865
123 100.000 100.000 100.000 99.978 99.873
124 100.000 100.000 100.000 99.980 99.881
125 100.000 100.000 100.000 99.982 99.889
126 100.000 100.000 100.000 99.984 99.897
127 100.000 100.000 100.000 99.986 99.905
128 100.000 100.000 100.000 99.989 99.913
129 100.000 100.000 100.000 99.991 99.921
130 100.000 100.000 100.000 99.993 99.929
131 100.000 100.000 100.000 99.994 99.934
132 100.000 100.000 100.000 99.995 99.939
133 100.000 100.000 100.000 99.996 99.944
134 100.000 100.000 100.000 99.997 99.949
135 100.000 100.000 100.000 99.998 99.954
136 100.000 100.000 100.000 99.999 99.959
137 100.000 100.000 100.000 100.000 99.964
138 100.000 100.000 100.000 100.000 99.964
139 100.000 100.000 100.000 100.000 99.965
140 100.000 100.000 100.000 100.000 99.966
141 100.000 100.000 100.000 100.000 99.967
142 100.000 100.000 100.000 100.000 99.967
143 100.000 100.000 100.000 100.000 99.968
144 100.000 100.000 100.000 100.000 99.969
145 100.000 100.000 100.000 100.000 99.970
146 100.000 100.000 100.000 100.000 99.971
147 100.000 100.000 100.000 100.000 99.972
148 100.000 100.000 100.000 100.000 99.973
149 100.000 100.000 100.000 100.000 99.974
150 100.000 100.000 100.000 100.000 99.974
151 100.000 100.000 100.000 100.000 99.975
152 100.000 100.000 100.000 100.000 99.977
153 100.000 100.000 100.000 100.000 99.977
154 100.000 100.000 100.000 100.000 99.978
155 100.000 100.000 100.000 100.000 99.979
156 100.000 100.000 100.000 100.000 99.979
157 100.000 100.000 100.000 100.000 99.981
158 100.000 100.000 100.000 100.000 99.981
159 100.000 100.000 100.000 100.000 99.982
160 100.000 100.000 100.000 100.000 99.983
161 100.000 100.000 100.000 100.000 99.984
162 100.000 100.000 100.000 100.000 99.984
163 100.000 100.000 100.000 100.000 99.986
164 100.000 100.000 100.000 100.000 99.986
165 100.000 100.000 100.000 100.000 99.988
166 100.000 100.000 100.000 100.000 99.988
167 100.000 100.000 100.000 100.000 99.990
168 100.000 100.000 100.000 100.000 99.990
169 100.000 100.000 100.000 100.000 99.991
170 100.000 100.000 100.000 100.000 99.992
171 100.000 100.000 100.000 100.000 99.993
172 100.000 100.000 100.000 100.000 99.994
173 100.000 100.000 100.000 100.000 99.994
174 100.000 100.000 100.000 100.000 99.995
175 100.000 100.000 100.000 100.000 99.995
176 100.000 100.000 100.000 100.000 99.996
177 100.000 100.000 100.000 100.000 99.997
178 100.000 100.000 100.000 100.000 99.998
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.503 71.125 60.228 50.409
5 95.349 86.327 75.852 65.403 55.691
6 95.349 87.601 78.445 68.973 59.835
7 95.349 88.528 80.384 71.714 63.102
8 95.349 89.108 81.723 73.754 65.676
9 97.674 92.932 86.357 78.710 70.634
10 97.674 93.163 87.012 79.849 72.214
11 97.674 93.859 88.461 81.905 74.684
12 97.674 94.670 90.067 84.089 77.221
13 100.000 97.914 93.640 87.720 80.778
14 100.000 98.030 94.018 88.436 81.834
15 100.000 98.262 94.628 89.451 83.208
16 100.000 98.378 95.006 90.161 84.244
17 100.000 98.494 95.357 90.801 85.172
18 100.000 98.494 95.486 91.157 85.795
19 100.000 98.494 95.597 91.468 86.346
20 100.000 98.610 95.901 91.991 87.084
21 100.000 98.610 95.994 92.247 87.536
22 100.000 98.841 96.492 92.989 88.483
23 100.000 98.841 96.594 93.272 88.980
24 100.000 99.073 97.101 94.033 89.950
25 100.000 99.073 97.185 94.264 90.360
26 100.000 99.073 97.240 94.434 90.684
27 100.000 99.073 97.286 94.576 90.956
28 100.000 99.073 97.323 94.691 91.183
29 100.000 99.073 97.406 94.925 91.597
30 100.000 99.073 97.434 95.017 91.787
31 100.000 99.073 97.461 95.109 91.974
32 100.000 99.189 97.729 95.533 92.543
33 100.000 99.189 97.775 95.672 92.807
34 100.000 99.189 97.794 95.736 92.942
35 100.000 99.305 98.043 96.111 93.430
36 100.000 99.305 98.052 96.151 93.525
37 100.000 99.305 98.098 96.284 93.769
38 100.000 99.305 98.145 96.416 94.010
39 100.000 99.305 98.154 96.454 94.098
40 100.000 99.305 98.181 96.540 94.267
41 100.000 99.305 98.218 96.646 94.462
42 100.000 99.421 98.477 97.038 94.965
43 100.000 99.421 98.514 97.144 95.161
44 100.000 99.421 98.532 97.205 95.284
45 100.000 99.421 98.551 97.264 95.403
46 100.000 99.421 98.560 97.299 95.484
47 100.000 99.537 98.809 97.666 95.944
48 100.000 99.537 98.818 97.703 96.031
49 100.000 99.652 99.068 98.071 96.494
50 100.000 99.652 99.077 98.105 96.571
51 100.000 99.652 99.086 98.141 96.650
52 100.000 99.768 99.335 98.507 97.110
53 100.000 99.884 99.585 98.874 97.569
54 100.000 99.884 99.603 98.931 97.681
55 100.000 99.884 99.612 98.963 97.748
56 100.000 99.884 99.612 98.972 97.783
57 100.000 99.884 99.622 99.006 97.858
58 100.000 100.000 99.871 99.372 98.314
59 100.000 100.000 99.889 99.429 98.426
60 100.000 100.000 99.889 99.439 98.460
61 100.000 100.000 99.889 99.450 98.500
62 100.000 100.000 99.889 99.457 98.527
63 100.000 100.000 99.889 99.466 98.557
64 100.000 100.000 99.889 99.473 98.584
65 100.000 100.000 99.889 99.480 98.611
66 100.000 100.000 99.908 99.533 98.709
67 100.000 100.000 99.908 99.542 98.739
68 100.000 100.000 99.908 99.548 98.762
69 100.000 100.000 99.908 99.556 98.792
70 100.000 100.000 99.908 99.563 98.815
71 100.000 100.000 99.908 99.569 98.838
72 100.000 100.000 99.908 99.573 98.855
73 100.000 100.000 99.908 99.578 98.875
74 100.000 100.000 99.908 99.584 98.897
75 100.000 100.000 99.908 99.590 98.917
76 100.000 100.000 99.908 99.595 98.936
77 100.000 100.000 99.908 99.599 98.953
78 100.000 100.000 99.908 99.603 98.969
79 100.000 100.000 99.908 99.608 98.988
80 100.000 100.000 99.908 99.614 99.008
81 100.000 100.000 99.908 99.617 99.021
82 100.000 100.000 99.908 99.621 99.037
83 100.000 100.000 99.908 99.625 99.052
84 100.000 100.000 99.908 99.629 99.068
85 100.000 100.000 99.908 99.633 99.084
86 100.000 100.000 99.908 99.637 99.097
87 100.000 100.000 99.908 99.640 99.109
88 100.000 100.000 99.908 99.644 99.124
89 100.000 100.000 99.908 99.647 99.137
90 100.000 100.000 99.908 99.651 99.152
91 100.000 100.000 99.908 99.653 99.161
92 100.000 100.000 99.917 99.680 99.212
93 100.000 100.000 99.917 99.683 99.224
94 100.000 100.000 99.917 99.687 99.236
95 100.000 100.000 99.917 99.689 99.245
96 100.000 100.000 99.917 99.691 99.254
97 100.000 100.000 99.917 99.695 99.269
98 100.000 100.000 99.917 99.697 99.278
99 100.000 100.000 99.917 99.700 99.290
100 100.000 100.000 99.926 99.727 99.341
101 100.000 100.000 99.926 99.729 99.349
102 100.000 100.000 99.935 99.756 99.400
103 100.000 100.000 99.945 99.783 99.450
104 100.000 100.000 99.945 99.785 99.459
105 100.000 100.000 99.954 99.813 99.510
106 100.000 100.000 99.963 99.840 99.560
107 100.000 100.000 99.972 99.867 99.611
108 100.000 100.000 99.982 99.894 99.661
109 100.000 100.000 99.991 99.921 99.712
110 100.000 100.000 100.000 99.948 99.763
111 100.000 100.000 100.000 99.950 99.771
112 100.000 100.000 100.000 99.952 99.779
113 100.000 100.000 100.000 99.954 99.788
114 100.000 100.000 100.000 99.956 99.796
115 100.000 100.000 100.000 99.958 99.804
116 100.000 100.000 100.000 99.959 99.809
117 100.000 100.000 100.000 99.963 99.821
118 100.000 100.000 100.000 99.965 99.829
119 100.000 100.000 100.000 99.967 99.837
120 100.000 100.000 100.000 99.968 99.842
121 100.000 100.000 100.000 99.970 99.850
122 100.000 100.000 100.000 99.972 99.858
123 100.000 100.000 100.000 99.974 99.866
124 100.000 100.000 100.000 99.976 99.874
125 100.000 100.000 100.000 99.978 99.882
126 100.000 100.000 100.000 99.980 99.890
127 100.000 100.000 100.000 99.982 99.898
128 100.000 100.000 100.000 99.984 99.906
129 100.000 100.000 100.000 99.986 99.914
130 100.000 100.000 100.000 99.989 99.922
131 100.000 100.000 100.000 99.990 99.926
132 100.000 100.000 100.000 99.992 99.934
133 100.000 100.000 100.000 99.994 99.942
134 100.000 100.000 100.000 99.996 99.950
135 100.000 100.000 100.000 99.997 99.954
136 100.000 100.000 100.000 99.999 99.962
137 100.000 100.000 100.000 99.999 99.963
138 100.000 100.000 100.000 99.999 99.965
139 100.000 100.000 100.000 99.999 99.966
140 100.000 100.000 100.000 99.999 99.967
141 100.000 100.000 100.000 99.999 99.968
142 100.000 100.000 100.000 99.999 99.969
143 100.000 100.000 100.000 99.999 99.970
144 100.000 100.000 100.000 99.999 99.972
145 100.000 100.000 100.000 99.999 99.973
146 100.000 100.000 100.000 99.999 99.974
147 100.000 100.000 100.000 99.999 99.974
148 100.000 100.000 100.000 99.999 99.975
149 100.000 100.000 100.000 99.999 99.976
150 100.000 100.000 100.000 99.999 99.977
151 100.000 100.000 100.000 99.999 99.978
152 100.000 100.000 100.000 99.999 99.979
153 100.000 100.000 100.000 99.999 99.980
154 100.000 100.000 100.000 99.999 99.981
155 100.000 100.000 100.000 99.999 99.982
156 100.000 100.000 100.000 99.999 99.983
157 100.000 100.000 100.000 100.000 99.987
158 100.000 100.000 100.000 100.000 99.988
159 100.000 100.000 100.000 100.000 99.988
160 100.000 100.000 100.000 100.000 99.989
161 100.000 100.000 100.000 100.000 99.989
162 100.000 100.000 100.000 100.000 99.990
163 100.000 100.000 100.000 100.000 99.991
164 100.000 100.000 100.000 100.000 99.991
165 100.000 100.000 100.000 100.000 99.992
166 100.000 100.000 100.000 100.000 99.993
167 100.000 100.000 100.000 100.000 99.993
168 100.000 100.000 100.000 100.000 99.994
169 100.000 100.000 100.000 100.000 99.994
170 100.000 100.000 100.000 100.000 99.995
171 100.000 100.000 100.000 100.000 99.996
172 100.000 100.000 100.000 100.000 99.996
173 100.000 100.000 100.000 100.000 99.997
174 100.000 100.000 100.000 100.000 99.998
175 100.000 100.000 100.000 100.000 99.998
176 100.000 100.000 100.000 100.000 99.999
177 100.000 100.000 100.000 100.000 99.999
178 100.000 100.000 100.000 100.000 100.000
179 100.000 100.000 100.000 100.000 100.000
180 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 64.286 34.687 19.287 11.372 7.125
4 71.429 41.794 24.402 14.787 9.398
5 80.952 48.900 29.141 17.945 11.545
6 85.714 53.807 32.895 20.632 13.457
7 90.476 58.545 36.696 23.463 15.504
8 95.238 63.283 40.497 26.293 17.551
9 97.619 67.174 44.064 29.071 19.589
10 100.000 70.558 47.208 31.602 21.506
11 100.000 72.927 50.023 34.046 23.407
12 100.000 75.296 52.816 36.481 25.314
13 100.000 77.327 55.256 38.663 27.073
14 100.000 79.188 57.602 40.802 28.803
15 100.000 80.880 59.784 42.840 30.490
16 100.000 82.572 61.872 44.759 32.080
17 100.000 83.926 63.749 46.603 33.661
18 100.000 85.110 65.415 48.287 35.148
19 100.000 86.125 66.753 49.601 36.311
20 100.000 87.140 68.207 51.102 37.667
21 100.000 87.986 69.521 52.517 38.978
22 100.000 88.832 70.695 53.718 40.076
23 100.000 89.679 72.032 55.171 41.426
24 100.000 90.525 73.393 56.662 42.811
25 100.000 91.371 74.801 58.227 44.272
26 100.000 92.047 75.692 59.107 45.058
27 100.000 92.724 76.771 60.308 46.210
28 100.000 93.401 78.038 61.782 47.617
29 100.000 94.078 79.024 62.833 48.612
30 100.000 94.755 80.009 63.884 49.607
31 100.000 95.262 80.784 64.753 50.471
32 100.000 95.770 81.628 65.723 51.429
33 100.000 96.277 82.543 66.822 52.536
34 100.000 96.785 83.388 67.803 53.514
35 100.000 97.293 84.233 68.785 54.496
36 100.000 97.800 85.031 69.680 55.375
37 100.000 98.308 85.734 70.409 56.057
38 100.000 98.646 86.415 71.272 56.963
39 100.000 98.985 87.025 72.034 57.773
40 100.000 99.323 87.541 72.613 58.352
41 100.000 99.662 88.104 73.294 59.073
42 100.000 99.831 88.433 73.717 59.535
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 64.286 34.518 19.193 11.340 7.118
4 71.429 41.455 24.214 14.723 9.383
5 76.190 46.870 28.578 17.881 11.587
6 83.333 52.961 33.060 21.039 13.782
7 85.714 57.191 36.931 24.009 15.922
8 88.095 61.421 40.779 26.958 18.050
9 90.476 65.313 44.392 29.795 20.134
10 92.857 68.697 47.489 32.277 22.022
11 95.238 71.912 50.493 34.738 23.920
12 97.619 74.958 53.332 37.086 25.752
13 97.619 76.819 55.655 39.215 27.492
14 97.619 78.511 57.837 41.263 29.195
15 97.619 80.034 59.808 43.129 30.767
16 97.619 81.049 61.427 44.850 32.313
17 97.619 82.064 62.928 46.432 33.757
18 97.619 83.249 64.524 48.008 35.142
19 97.619 84.264 66.025 49.579 36.565
20 97.619 85.279 67.480 51.091 37.940
21 97.619 86.294 68.888 52.523 39.233
22 97.619 87.310 70.296 53.954 40.526
23 97.619 88.325 71.680 55.338 41.755
24 97.619 88.832 72.712 56.603 43.013
25 97.619 89.509 73.792 57.815 44.185
26 97.619 90.186 74.871 59.016 45.337
27 100.000 92.217 76.420 60.195 46.253
28 100.000 92.893 77.452 61.326 47.338
29 100.000 93.570 78.461 62.410 48.361
30 100.000 94.247 79.446 63.460 49.356
31 100.000 94.924 80.432 64.511 50.350
32 100.000 95.431 81.300 65.530 51.364
33 100.000 95.770 82.074 66.527 52.390
34 100.000 96.108 82.802 67.460 53.363
35 100.000 96.785 83.693 68.339 54.149
36 100.000 97.293 84.491 69.251 55.064
37 100.000 97.800 85.265 70.120 55.928
38 100.000 98.308 86.016 70.945 56.739
39 100.000 98.646 86.649 71.733 57.566
40 100.000 99.154 87.353 72.463 58.248
41 100.000 99.492 87.916 73.138 58.957
42 100.000 99.831 88.433 73.717 59.535
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 61.905 33.503 18.841 11.233 7.090
4 71.429 41.455 24.261 14.771 9.413
5 80.952 49.239 29.517 18.208 11.687
6 85.714 54.653 33.881 21.366 13.891
7 88.095 59.391 38.104 24.503 16.094
8 90.476 63.621 41.952 27.452 18.222
9 90.476 65.990 44.721 29.864 20.118
10 92.857 68.866 47.583 32.309 22.030
11 95.238 71.912 50.446 34.684 23.881
12 100.000 76.142 53.778 37.221 25.778
13 100.000 77.834 55.913 39.210 27.435
14 100.000 79.357 57.907 41.113 29.044
15 100.000 80.372 59.573 42.866 30.592
16 100.000 81.387 61.098 44.480 32.062
17 100.000 82.741 62.905 46.239 33.578
18 100.000 83.926 64.547 47.896 35.046
19 100.000 84.941 66.002 49.408 36.422
20 100.000 85.787 67.316 50.839 37.761
21 100.000 86.802 68.771 52.340 39.116
22 100.000 87.986 70.366 53.917 40.502
23 100.000 88.832 71.633 55.289 41.794
24 100.000 89.848 73.041 56.715 43.079
25 100.000 90.863 74.449 58.142 44.365
26 100.000 91.709 75.669 59.423 45.554
27 100.000 92.724 77.030 60.774 46.756
28 100.000 93.401 78.109 61.975 47.908
29 100.000 94.078 79.141 63.107 48.993
30 100.000 94.416 79.939 64.131 50.038
31 100.000 95.093 80.924 65.181 51.033
32 100.000 95.601 81.746 66.120 51.963
33 100.000 95.939 82.426 66.983 52.869
34 100.000 96.447 83.247 67.921 53.799
35 100.000 96.954 84.045 68.822 54.691
36 100.000 97.462 84.819 69.691 55.554
37 100.000 98.139 85.711 70.570 56.340
38 100.000 98.477 86.274 71.251 57.061
39 100.000 98.816 86.861 71.948 57.773
40 100.000 99.323 87.565 72.677 58.456
41 100.000 99.662 88.081 73.256 59.034
42 100.000 99.831 88.409 73.680 59.497
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 59.524 32.318 18.348 11.045 7.025
4 66.667 39.594 23.627 14.562 9.346
5 73.810 46.701 28.789 18.026 11.648
6 78.571 51.777 32.895 21.050 13.797
7 80.952 54.992 36.039 23.666 15.800
8 83.333 58.376 39.207 26.235 17.752
9 88.095 62.944 42.891 29.012 19.781
10 88.095 65.144 45.519 31.333 21.623
11 88.095 67.343 48.170 33.676 23.476
12 90.476 70.051 50.634 35.757 25.159
13 92.857 73.096 53.426 38.046 26.945
14 92.857 74.450 55.256 39.842 28.500
15 97.619 78.680 58.517 42.282 30.312
16 97.619 80.034 60.347 44.078 31.864
17 100.000 82.910 62.905 46.164 33.510
18 100.000 84.095 64.547 47.810 34.961
19 100.000 84.941 65.931 49.322 36.357
20 100.000 86.125 67.597 50.995 37.828
21 100.000 86.971 68.935 52.448 39.177
22 100.000 87.817 70.202 53.809 40.453
23 100.000 88.832 71.656 55.311 41.809
24 100.000 90.017 73.229 56.855 43.166
25 100.000 90.863 74.519 58.238 44.450
26 100.000 91.878 75.880 59.611 45.696
27 100.000 92.555 76.983 60.844 46.876
28 100.000 93.401 78.179 62.088 48.026
29 100.000 94.078 79.165 63.155 49.056
30 100.000 94.416 79.962 64.179 50.101
31 100.000 94.755 80.666 65.069 51.022
32 100.000 95.262 81.488 66.007 51.952
33 100.000 95.770 82.309 66.945 52.882
34 100.000 96.447 83.271 67.959 53.838
35 100.000 97.124 84.233 68.972 54.794
36 100.000 97.631 85.007 69.841 55.658
37 100.000 97.970 85.570 70.522 56.379
38 100.000 98.308 86.157 71.219 57.091
39 100.000 98.816 86.861 71.948 57.773
40 100.000 99.323 87.565 72.677 58.456
41 100.000 99.662 88.081 73.256 59.034
42 100.000 99.831 88.409 73.680 59.497
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 66.667 35.533 19.521 11.426 7.134
4 76.190 43.486 24.941 14.964 9.457
5 85.714 51.100 30.103 18.369 11.724
6 90.476 57.191 34.937 21.747 14.010
7 92.857 61.929 39.160 24.883 16.212
8 95.238 66.328 43.149 27.913 18.377
9 97.619 70.051 46.574 30.615 20.386
10 100.000 73.435 49.789 33.242 22.385
11 100.000 75.296 52.112 35.376 24.134
12 100.000 77.157 54.575 37.671 25.996
13 100.000 79.019 56.875 39.778 27.727
14 100.000 80.541 59.033 41.880 29.488
15 100.000 81.895 61.004 43.842 31.165
16 100.000 83.249 62.764 45.536 32.626
17 100.000 84.433 64.406 47.182 34.077
18 100.000 85.448 65.814 48.619 35.379
19 100.000 86.464 67.222 50.051 36.672
20 100.000 87.479 68.583 51.402 37.874
21 100.000 88.494 70.014 52.866 39.194
22 100.000 89.340 71.258 54.179 40.411
23 100.000 90.186 72.642 55.713 41.844
24 100.000 91.032 74.050 57.257 43.268
25 100.000 91.709 75.364 58.774 44.692
26 100.000 92.386 76.396 59.895 45.753
27 100.000 93.063 77.288 60.774 46.539
28 100.000 93.570 78.179 61.847 47.628
29 100.000 94.078 78.954 62.715 48.492
30 100.000 94.585 79.798 63.707 49.494
31 100.000 95.093 80.526 64.474 50.215
32 100.000 95.601 81.253 65.240 50.937
33 100.000 96.108 82.074 66.179 51.867
34 100.000 96.447 82.708 66.967 52.693
35 100.000 96.785 83.388 67.825 53.587
36 100.000 97.124 83.881 68.361 54.114
37 100.000 97.462 84.420 69.004 54.796
38 100.000 97.800 85.031 69.771 55.615
39 100.000 97.970 85.617 70.629 56.555
40 100.000 98.139 86.133 71.374 57.383
41 100.000 98.308 86.509 71.894 57.977
42 100.000 98.477 86.908 72.452 58.609
43 100.000 98.646 87.353 73.101 59.352
44 100.000 98.816 87.682 73.540 59.848
45 100.000 98.985 88.034 74.007 60.362
46 100.000 99.154 88.409 74.532 60.967
47 100.000 99.323 88.761 75.031 61.553
48 100.000 99.492 89.254 75.739 62.343
49 100.000 99.662 89.653 76.296 62.975
50 100.000 99.831 89.911 76.591 63.280
51 100.000 100.000 90.216 76.977 63.704
52 100.000 100.000 90.357 77.261 64.090
53 100.000 100.000 90.802 78.055 65.031
54 100.000 100.000 90.990 78.441 65.553
55 100.000 100.000 91.061 78.586 65.754
56 100.000 100.000 91.107 78.687 65.904
57 100.000 100.000 91.248 78.961 66.263
58 100.000 100.000 91.436 79.336 66.764
59 100.000 100.000 91.741 79.888 67.437
60 100.000 100.000 91.929 80.264 67.938
61 100.000 100.000 92.046 80.505 68.269
62 100.000 100.000 92.281 80.961 68.861
63 100.000 100.000 92.374 81.165 69.153
64 100.000 100.000 92.562 81.545 69.667
65 100.000 100.000 92.679 81.781 69.987
66 100.000 100.000 92.726 81.883 70.137
67 100.000 100.000 92.844 82.130 70.485
68 100.000 100.000 92.867 82.189 70.583
69 100.000 100.000 92.914 82.290 70.732
70 100.000 100.000 92.985 82.441 70.949
71 100.000 100.000 93.102 82.682 71.284
72 100.000 100.000 93.172 82.843 71.524
73 100.000 100.000 93.290 83.084 71.855
74 100.000 100.000 93.407 83.320 72.175
75 100.000 100.000 93.454 83.427 72.339
76 100.000 100.000 93.595 83.722 72.750
77 100.000 100.000 93.688 83.915 73.018
78 100.000 100.000 93.759 84.065 73.234
79 100.000 100.000 93.782 84.119 73.318
80 100.000 100.000 94.017 84.521 73.789
81 100.000 100.000 94.111 84.714 74.057
82 100.000 100.000 94.392 85.245 74.727
83 100.000 100.000 94.674 85.781 75.398
84 100.000 100.000 94.721 85.904 75.601
85 100.000 100.000 94.791 86.076 75.873
86 100.000 100.000 94.979 86.462 76.395
87 100.000 100.000 95.190 86.880 76.940
88 100.000 100.000 95.237 87.014 77.171
89 100.000 100.000 95.331 87.229 77.491
90 100.000 100.000 95.448 87.464 77.813
91 100.000 100.000 95.565 87.711 78.161
92 100.000 100.000 95.636 87.856 78.362
93 100.000 100.000 95.753 88.113 78.734
94 100.000 100.000 95.847 88.306 79.002
95 100.000 100.000 95.870 88.381 79.141
96 100.000 100.000 95.964 88.590 79.448
97 100.000 100.000 95.988 88.671 79.599
98 100.000 100.000 96.058 88.832 79.844
99 100.000 100.000 96.293 89.277 80.411
100 100.000 100.000 96.387 89.491 80.724
101 100.000 100.000 96.457 89.647 80.953
102 100.000 100.000 96.621 89.990 81.423
103 100.000 100.000 96.809 90.370 81.937
104 100.000 100.000 96.832 90.424 82.020
105 100.000 100.000 96.973 90.698 82.379
106 100.000 100.000 97.255 91.212 83.013
107 100.000 100.000 97.349 91.405 83.281
108 100.000 100.000 97.396 91.507 83.431
109 100.000 100.000 97.489 91.722 83.751
110 100.000 100.000 97.630 92.022 84.174
111 100.000 100.000 97.724 92.220 84.455
112 100.000 100.000 97.748 92.285 84.566
113 100.000 100.000 97.794 92.386 84.716
114 100.000 100.000 97.888 92.590 85.012
115 100.000 100.000 97.912 92.649 85.112
116 100.000 100.000 97.959 92.762 85.289
117 100.000 100.000 98.053 92.955 85.558
118 100.000 100.000 98.076 93.025 85.682
119 100.000 100.000 98.170 93.228 85.977
120 100.000 100.000 98.264 93.421 86.246
121 100.000 100.000 98.358 93.625 86.534
122 100.000 100.000 98.381 93.738 86.761
123 100.000 100.000 98.405 93.797 86.859
124 100.000 100.000 98.451 93.898 87.009
125 100.000 100.000 98.522 94.054 87.239
126 100.000 100.000 98.710 94.429 87.740
127 100.000 100.000 98.756 94.531 87.889
128 100.000 100.000 98.850 94.724 88.158
129 100.000 100.000 98.921 94.874 88.374
130 100.000 100.000 98.968 94.976 88.524
131 100.000 100.000 99.085 95.233 88.895
132 100.000 100.000 99.108 95.303 89.019
133 100.000 100.000 99.179 95.459 89.249
134 100.000 100.000 99.202 95.512 89.332
135 100.000 100.000 99.249 95.646 89.563
136 100.000 100.000 99.273 95.711 89.674
137 100.000 100.000 99.320 95.823 89.852
138 100.000 100.000 99.413 96.016 90.120
139 100.000 100.000 99.460 96.129 90.297
140 100.000 100.000 99.484 96.193 90.410
141 100.000 100.000 99.578 96.397 90.706
142 100.000 100.000 99.648 96.552 90.935
143 100.000 100.000 99.695 96.708 91.213
144 100.000 100.000 99.742 96.810 91.362
145 100.000 100.000 99.812 96.981 91.632
146 100.000 100.000 99.883 97.132 91.849
147 100.000 100.000 99.906 97.201 91.974
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 61.905 33.841 19.052 11.318 7.116
4 73.810 42.640 24.707 14.911 9.448
5 80.952 49.915 29.986 18.428 11.768
6 83.333 54.146 33.904 21.447 13.938
7 85.714 57.868 37.424 24.251 16.014
8 92.857 63.621 41.600 27.216 18.108
9 95.238 67.343 45.002 29.891 20.097
10 95.238 69.543 47.701 32.293 21.996
11 100.000 73.942 51.103 34.813 23.846
12 100.000 76.142 53.754 37.156 25.699
13 100.000 78.173 56.288 39.456 27.550
14 100.000 80.203 58.799 41.735 29.390
15 100.000 81.557 60.770 43.697 31.067
16 100.000 82.910 62.694 45.611 32.714
17 100.000 84.095 64.313 47.236 34.155
18 100.000 85.448 66.002 48.812 35.486
19 100.000 86.464 67.480 50.362 36.899
20 100.000 87.479 68.935 51.863 38.255
21 100.000 88.325 70.272 53.316 39.605
22 100.000 89.171 71.563 54.699 40.888
23 100.000 90.017 72.853 56.104 42.208
24 100.000 90.863 74.097 57.439 43.462
25 100.000 91.709 75.340 58.758 44.688
26 100.000 92.386 76.466 60.008 45.862
27 100.000 93.063 77.546 61.209 47.014
28 100.000 93.739 78.531 62.286 48.065
29 100.000 94.247 79.423 63.326 49.089
30 100.000 94.247 80.033 64.350 50.234
31 100.000 94.755 80.924 65.412 51.303
32 100.000 95.262 81.793 66.420 52.299
33 100.000 95.431 82.450 67.374 53.331
34 100.000 95.939 83.294 68.361 54.324
35 100.000 96.447 84.092 69.283 55.260
36 100.000 96.954 84.866 70.146 56.113
37 100.000 97.293 85.547 71.004 57.006
38 100.000 97.462 86.110 71.830 57.921
39 100.000 97.800 86.720 72.581 58.709
40 100.000 97.800 87.142 73.315 59.567
41 100.000 98.139 87.752 74.066 60.355
42 100.000 98.139 88.104 74.704 61.115
43 100.000 98.139 88.456 75.331 61.867
44 100.000 98.308 88.878 75.932 62.550
45 100.000 98.646 89.371 76.500 63.153
46 100.000 98.816 89.770 77.063 63.796
47 100.000 98.985 90.145 77.583 64.390
48 100.000 99.154 90.521 78.103 64.983
49 100.000 99.323 90.896 78.623 65.577
50 100.000 99.323 91.154 79.116 66.207
51 100.000 99.492 91.506 79.615 66.794
52 100.000 99.492 91.741 80.065 67.372
53 100.000 99.492 91.976 80.500 67.916
54 100.000 99.662 92.304 80.950 68.439
55 100.000 99.831 92.633 81.374 68.902
56 100.000 99.831 92.867 81.835 69.505
57 100.000 99.831 93.078 82.253 70.050
58 100.000 99.831 93.266 82.628 70.551
59 100.000 99.831 93.454 82.998 71.041
60 100.000 99.831 93.641 83.395 71.591
61 100.000 99.831 93.829 83.770 72.092
62 100.000 99.831 94.017 84.151 72.606
63 100.000 100.000 94.275 84.446 72.911
64 100.000 100.000 94.439 84.784 73.373
65 100.000 100.000 94.603 85.100 73.784
66 100.000 100.000 94.744 85.384 74.170
67 100.000 100.000 94.885 85.700 74.625
68 100.000 100.000 95.026 85.974 74.984
69 100.000 100.000 95.167 86.253 75.356
70 100.000 100.000 95.307 86.553 75.777
71 100.000 100.000 95.448 86.837 76.156
72 100.000 100.000 95.589 87.121 76.535
73 100.000 100.000 95.730 87.400 76.907
74 100.000 100.000 95.870 87.695 77.318
75 100.000 100.000 96.011 88.001 77.752
76 100.000 100.000 96.129 88.253 78.111
77 100.000 100.000 96.246 88.483 78.418
78 100.000 100.000 96.363 88.719 78.738
79 100.000 100.000 96.481 88.976 79.111
80 100.000 100.000 96.574 89.180 79.405
81 100.000 100.000 96.668 89.373 79.673
82 100.000 100.000 96.762 89.641 80.110
83 100.000 100.000 96.856 89.872 80.467
84 100.000 100.000 96.950 90.070 80.750
85 100.000 100.000 97.044 90.269 81.031
86 100.000 100.000 97.137 90.499 81.388
87 100.000 100.000 97.231 90.714 81.708
88 100.000 100.000 97.302 90.864 81.924
89 100.000 100.000 97.372 91.009 82.125
90 100.000 100.000 97.443 91.175 82.379
91 100.000 100.000 97.513 91.357 82.671
92 100.000 100.000 97.583 91.534 82.952
93 100.000 100.000 97.654 91.700 83.205
94 100.000 100.000 97.724 91.856 83.435
95 100.000 100.000 97.794 92.017 83.679
96 100.000 100.000 97.865 92.188 83.949
97 100.000 100.000 97.935 92.360 84.219
98 100.000 100.000 98.006 92.510 84.435
99 100.000 100.000 98.076 92.660 84.652
100 100.000 100.000 98.146 92.815 84.881
101 100.000 100.000 98.217 92.966 85.097
102 100.000 100.000 98.287 93.148 85.391
103 100.000 100.000 98.358 93.346 85.722
104 100.000 100.000 98.428 93.502 85.952
105 100.000 100.000 98.475 93.604 86.101
106 100.000 100.000 98.522 93.722 86.294
107 100.000 100.000 98.569 93.829 86.458
108 100.000 100.000 98.616 93.931 86.608
109 100.000 100.000 98.663 94.049 86.798
110 100.000 100.000 98.710 94.156 86.963
111 100.000 100.000 98.756 94.268 87.141
112 100.000 100.000 98.803 94.381 87.322
113 100.000 100.000 98.850 94.504 87.527
114 100.000 100.000 98.897 94.606 87.677
115 100.000 100.000 98.944 94.708 87.827
116 100.000 100.000 98.991 94.810 87.976
117 100.000 100.000 99.038 94.923 88.154
118 100.000 100.000 99.085 95.030 88.317
119 100.000 100.000 99.132 95.137 88.481
120 100.000 100.000 99.179 95.239 88.631
121 100.000 100.000 99.226 95.394 88.908
122 100.000 100.000 99.273 95.518 89.110
123 100.000 100.000 99.320 95.636 89.302
124 100.000 100.000 99.366 95.737 89.452
125 100.000 100.000 99.413 95.861 89.654
126 100.000 100.000 99.460 95.968 89.818
127 100.000 100.000 99.507 96.086 90.007
128 100.000 100.000 99.531 96.156 90.131
129 100.000 100.000 99.554 96.225 90.256
130 100.000 100.000 99.578 96.279 90.340
131 100.000 100.000 99.601 96.343 90.454
132 100.000 100.000 99.625 96.429 90.621
133 100.000 100.000 99.648 96.488 90.719
134 100.000 100.000 99.672 96.542 90.802
135 100.000 100.000 99.695 96.601 90.900
136 100.000 100.000 99.718 96.660 90.998
137 100.000 100.000 99.742 96.719 91.096
138 100.000 100.000 99.765 96.772 91.179
139 100.000 100.000 99.789 96.837 91.290
140 100.000 100.000 99.812 96.890 91.373
141 100.000 100.000 99.836 96.955 91.486
142 100.000 100.000 99.859 97.014 91.586
143 100.000 100.000 99.883 97.110 91.778
144 100.000 100.000 99.906 97.164 91.862
145 100.000 100.000 99.930 97.223 91.960
146 100.000 100.000 99.953 97.282 92.058
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 57.143 31.641 18.254 11.072 7.053
4 66.667 39.425 23.557 14.557 9.357
5 76.190 47.208 28.860 18.042 11.661
6 80.952 52.623 33.224 21.200 13.865
7 85.714 57.530 37.166 24.122 15.966
8 88.095 61.083 40.544 26.851 18.013
9 90.476 64.129 43.548 29.382 19.970
10 90.476 66.328 46.293 31.832 21.898
11 95.238 71.066 50.047 34.615 23.916
12 97.619 74.281 53.050 37.076 25.813
13 97.619 76.142 55.397 39.237 27.581
14 97.619 77.834 57.626 41.344 29.330
15 100.000 80.711 60.300 43.590 31.115
16 100.000 82.403 62.412 45.542 32.733
17 100.000 83.249 63.820 47.113 34.201
18 100.000 84.772 65.767 48.952 35.755
19 100.000 85.956 67.433 50.630 37.234
20 100.000 87.310 69.193 52.319 38.684
21 100.000 88.494 70.765 53.868 40.050
22 100.000 89.171 71.938 55.209 41.336
23 100.000 90.017 73.205 56.565 42.600
24 100.000 90.863 74.449 57.884 43.828
25 100.000 91.540 75.575 59.134 45.002
26 100.000 92.047 76.513 60.265 46.138
27 100.000 92.893 77.710 61.498 47.264
28 100.000 93.232 78.461 62.474 48.287
29 100.000 93.570 79.235 63.471 49.309
30 100.000 94.078 80.103 64.490 50.323
31 100.000 94.416 80.807 65.412 51.310
32 100.000 94.755 81.511 66.334 52.283
33 100.000 95.093 82.168 67.192 53.213
34 100.000 95.262 82.708 67.986 54.102
35 100.000 95.431 83.200 68.709 54.927
36 100.000 95.939 83.951 69.551 55.772
37 100.000 96.447 84.702 70.388 56.608
38 100.000 96.447 85.031 70.994 57.353
39 100.000 96.616 85.500 71.664 58.104
40 100.000 96.785 85.969 72.334 58.853
41 100.000 97.124 86.579 73.079 59.628
42 100.000 97.293 87.025 73.728 60.371
43 100.000 97.462 87.471 74.366 61.094
44 100.000 97.800 88.034 75.058 61.840
45 100.000 97.800 88.315 75.594 62.511
46 100.000 97.970 88.738 76.200 63.206
47 100.000 98.139 89.137 76.779 63.879
48 100.000 98.646 89.864 77.545 64.600
49 100.000 98.816 90.263 78.114 65.257
50 100.000 98.985 90.662 78.671 65.889
51 100.000 99.154 91.061 79.229 66.522
52 100.000 99.154 91.295 79.685 67.114
53 100.000 99.154 91.530 80.140 67.705
54 100.000 99.154 91.741 80.559 68.251
55 100.000 99.154 91.952 80.972 68.791
56 100.000 99.154 92.140 81.363 69.327
57 100.000 99.154 92.281 81.701 69.830
58 100.000 99.154 92.468 82.076 70.331
59 100.000 99.154 92.656 82.451 70.832
60 100.000 99.154 92.820 82.800 71.319
61 100.000 99.154 93.031 83.191 71.807
62 100.000 99.323 93.360 83.631 72.302
63 100.000 99.492 93.688 84.071 72.805
64 100.000 99.492 93.853 84.414 73.279
65 100.000 99.492 94.017 84.751 73.741
66 100.000 99.492 94.181 85.089 74.203
67 100.000 99.662 94.486 85.497 74.671
68 100.000 99.662 94.627 85.808 75.119
69 100.000 99.662 94.791 86.135 75.557
70 100.000 99.662 94.908 86.408 75.964
71 100.000 99.662 95.073 86.725 76.374
72 100.000 99.662 95.214 87.019 76.785
73 100.000 99.662 95.354 87.314 77.197
74 100.000 99.662 95.495 87.604 77.595
75 100.000 99.831 95.800 87.990 78.018
76 100.000 100.000 96.105 88.376 78.442
77 100.000 100.000 96.246 88.660 78.828
78 100.000 100.000 96.340 88.896 79.193
79 100.000 100.000 96.457 89.148 79.553
80 100.000 100.000 96.551 89.368 79.888
81 100.000 100.000 96.668 89.609 80.221
82 100.000 100.000 96.786 89.850 80.556
83 100.000 100.000 96.856 90.038 80.861
84 100.000 100.000 96.950 90.247 81.169
85 100.000 100.000 97.044 90.451 81.458
86 100.000 100.000 97.137 90.655 81.750
87 100.000 100.000 97.231 90.858 82.048
88 100.000 100.000 97.325 91.057 82.329
89 100.000 100.000 97.419 91.250 82.597
90 100.000 100.000 97.489 91.421 82.865
91 100.000 100.000 97.560 91.593 83.133
92 100.000 100.000 97.630 91.759 83.390
93 100.000 100.000 97.701 91.925 83.647
94 100.000 100.000 97.771 92.086 83.890
95 100.000 100.000 97.841 92.247 84.130
96 100.000 100.000 97.912 92.403 84.359
97 100.000 100.000 97.982 92.558 84.592
98 100.000 100.000 98.053 92.714 84.822
99 100.000 100.000 98.099 92.842 85.040
100 100.000 100.000 98.170 92.992 85.258
101 100.000 100.000 98.240 93.142 85.474
102 100.000 100.000 98.311 93.293 85.691
103 100.000 100.000 98.381 93.443 85.907
104 100.000 100.000 98.451 93.587 86.109
105 100.000 100.000 98.498 93.711 86.310
106 100.000 100.000 98.545 93.834 86.516
107 100.000 100.000 98.592 93.957 86.717
108 100.000 100.000 98.639 94.075 86.906
109 100.000 100.000 98.663 94.172 87.098
110 100.000 100.000 98.710 94.284 87.276
111 100.000 100.000 98.756 94.397 87.455
112 100.000 100.000 98.803 94.510 87.632
113 100.000 100.000 98.850 94.622 87.810
114 100.000 100.000 98.897 94.735 87.989
115 100.000 100.000 98.944 94.847 88.167
116 100.000 100.000 98.991 94.960 88.344
117 100.000 100.000 99.015 95.046 88.513
118 100.000 100.000 99.061 95.153 88.679
119 100.000 100.000 99.108 95.260 88.844
120 100.000 100.000 99.155 95.362 88.993
121 100.000 100.000 99.202 95.464 89.143
122 100.000 100.000 99.249 95.566 89.293
123 100.000 100.000 99.296 95.668 89.443
124 100.000 100.000 99.343 95.770 89.593
125 100.000 100.000 99.390 95.872 89.742
126 100.000 100.000 99.437 95.973 89.892
127 100.000 100.000 99.484 96.075 90.042
128 100.000 100.000 99.531 96.177 90.194
129 100.000 100.000 99.554 96.252 90.332
130 100.000 100.000 99.578 96.327 90.471
131 100.000 100.000 99.601 96.397 90.597
132 100.000 100.000 99.625 96.467 90.722
133 100.000 100.000 99.648 96.536 90.846
134 100.000 100.000 99.672 96.601 90.957
135 100.000 100.000 99.695 96.665 91.068
136 100.000 100.000 99.718 96.729 91.179
137 100.000 100.000 99.742 96.794 91.292
138 100.000 100.000 99.765 96.858 91.405
139 100.000 100.000 99.789 96.922 91.516
140 100.000 100.000 99.812 96.981 91.614
141 100.000 100.000 99.836 97.035 91.697
142 100.000 100.000 99.859 97.089 91.780
143 100.000 100.000 99.883 97.142 91.863
144 100.000 100.000 99.906 97.196 91.947
145 100.000 100.000 99.930 97.249 92.030
146 100.000 100.000 99.953 97.303 92.113
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 42.857 21.658 12.107 7.292 4.656
3 52.381 29.272 17.269 10.696 6.923
4 59.524 36.041 22.220 14.074 9.200
5 71.429 44.501 27.616 17.533 11.476
6 78.571 51.100 32.426 20.830 13.716
7 80.952 56.176 36.861 24.052 15.944
8 85.714 61.421 41.107 27.157 18.130
9 90.476 66.159 44.932 30.025 20.212
10 95.238 70.728 48.545 32.727 22.192
11 95.238 73.096 51.337 35.162 24.099
12 97.619 76.311 54.411 37.698 26.044
13 100.000 79.188 57.109 39.966 27.840
14 100.000 80.372 58.893 41.810 29.467
15 100.000 81.726 60.863 43.772 31.144
16 100.000 83.080 62.740 45.633 32.755
17 100.000 84.095 64.336 47.327 34.283
18 100.000 85.618 66.283 49.166 35.836
19 100.000 86.633 67.785 50.732 37.251
20 100.000 87.648 69.310 52.324 38.684
21 100.000 88.325 70.507 53.697 39.997
22 100.000 89.509 72.055 55.214 41.336
23 100.000 90.186 73.158 56.453 42.526
24 100.000 90.694 74.097 57.579 43.653
25 100.000 91.371 75.153 58.748 44.777
26 100.000 92.047 76.208 59.916 45.901
27 100.000 92.555 77.100 60.978 46.970
28 100.000 92.724 77.687 61.852 47.946
29 100.000 93.232 78.578 62.903 48.995
30 100.000 93.739 79.446 63.922 50.016
31 100.000 94.078 80.150 64.828 50.972
32 100.000 94.585 80.948 65.750 51.907
33 100.000 94.755 81.534 66.608 52.847
34 100.000 94.755 81.863 67.230 53.627
35 100.000 95.431 82.872 68.313 54.650
36 100.000 95.601 83.365 69.037 55.471
37 100.000 95.939 84.045 69.894 56.364
38 100.000 95.939 84.374 70.500 57.109
39 100.000 96.277 85.007 71.294 57.947
40 100.000 96.785 85.805 72.195 58.838
41 100.000 97.124 86.415 72.951 59.637
42 100.000 97.631 87.213 73.851 60.525
43 100.000 97.800 87.635 74.468 61.240
44 100.000 97.970 88.081 75.111 61.975
45 100.000 97.970 88.339 75.610 62.616
46 100.000 97.970 88.620 76.135 63.275
47 100.000 97.970 88.902 76.671 63.946
48 100.000 98.139 89.301 77.245 64.613
49 100.000 98.308 89.700 77.814 65.270
50 100.000 98.477 90.122 78.409 65.941
51 100.000 98.646 90.521 78.972 66.585
52 100.000 98.646 90.756 79.422 67.163
53 100.000 98.646 90.967 79.840 67.716
54 100.000 98.816 91.366 80.398 68.349
55 100.000 98.816 91.577 80.811 68.891
56 100.000 98.816 91.788 81.224 69.432
57 100.000 98.816 91.976 81.610 69.954
58 100.000 98.816 92.163 81.985 70.455
59 100.000 98.816 92.328 82.328 70.928
60 100.000 98.985 92.679 82.811 71.483
61 100.000 98.985 92.867 83.181 71.973
62 100.000 98.985 93.055 83.545 72.454
63 100.000 99.154 93.360 83.958 72.942
64 100.000 99.154 93.501 84.258 73.366
65 100.000 99.323 93.829 84.698 73.869
66 100.000 99.323 93.993 85.025 74.307
67 100.000 99.492 94.298 85.432 74.782
68 100.000 99.492 94.439 85.727 75.193
69 100.000 99.492 94.580 86.022 75.603
70 100.000 99.492 94.721 86.317 76.014
71 100.000 99.492 94.862 86.607 76.413
72 100.000 99.662 95.190 87.046 76.909
73 100.000 99.662 95.331 87.330 77.297
74 100.000 99.662 95.495 87.652 77.719
75 100.000 99.662 95.612 87.904 78.079
76 100.000 99.662 95.730 88.151 78.427
77 100.000 99.662 95.824 88.365 78.747
78 100.000 99.662 95.941 88.612 79.093
79 100.000 99.662 96.035 88.826 79.413
80 100.000 99.662 96.176 89.111 79.792
81 100.000 99.662 96.269 89.320 80.102
82 100.000 99.662 96.387 89.561 80.437
83 100.000 99.662 96.481 89.770 80.746
84 100.000 99.662 96.574 89.979 81.053
85 100.000 99.662 96.668 90.183 81.349
86 100.000 99.662 96.762 90.387 81.645
87 100.000 99.662 96.856 90.596 81.948
88 100.000 99.662 96.950 90.799 82.244
89 100.000 99.662 97.044 91.003 82.540
90 100.000 99.662 97.091 91.143 82.780
91 100.000 99.831 97.372 91.486 83.152
92 100.000 99.831 97.466 91.689 83.440
93 100.000 99.831 97.489 91.786 83.634
94 100.000 99.831 97.536 91.915 83.853
95 100.000 99.831 97.560 92.011 84.045
96 100.000 99.831 97.630 92.172 84.289
97 100.000 99.831 97.724 92.365 84.557
98 100.000 99.831 97.794 92.521 84.790
99 100.000 99.831 97.841 92.644 84.995
100 100.000 99.831 97.912 92.799 85.225
101 100.000 99.831 97.982 92.955 85.454
102 100.000 99.831 98.053 93.110 85.683
103 100.000 99.831 98.099 93.234 85.885
104 100.000 99.831 98.146 93.357 86.087
105 100.000 99.831 98.193 93.475 86.279
106 100.000 99.831 98.217 93.561 86.445
107 100.000 99.831 98.287 93.711 86.662
108 100.000 99.831 98.358 93.861 86.878
109 100.000 99.831 98.428 94.011 87.094
110 100.000 100.000 98.686 94.306 87.399
111 100.000 100.000 98.756 94.456 87.616
112 100.000 100.000 98.803 94.574 87.804
113 100.000 100.000 98.827 94.654 87.958
114 100.000 100.000 98.874 94.767 88.135
115 100.000 100.000 98.921 94.880 88.313
116 100.000 100.000 98.968 94.992 88.490
117 100.000 100.000 99.038 95.137 88.692
118 100.000 100.000 99.085 95.244 88.858
119 100.000 100.000 99.132 95.351 89.023
120 100.000 100.000 99.179 95.453 89.175
121 100.000 100.000 99.202 95.523 89.300
122 100.000 100.000 99.249 95.625 89.450
123 100.000 100.000 99.273 95.695 89.574
124 100.000 100.000 99.320 95.796 89.724
125 100.000 100.000 99.366 95.898 89.874
126 100.000 100.000 99.413 96.000 90.023
127 100.000 100.000 99.460 96.102 90.173
128 100.000 100.000 99.507 96.204 90.323
129 100.000 100.000 99.531 96.274 90.447
130 100.000 100.000 99.578 96.376 90.597
131 100.000 100.000 99.601 96.440 90.710
132 100.000 100.000 99.625 96.504 90.820
133 100.000 100.000 99.648 96.569 90.931
134 100.000 100.000 99.672 96.633 91.042
135 100.000 100.000 99.695 96.697 91.153
136 100.000 100.000 99.718 96.756 91.251
137 100.000 100.000 99.742 96.815 91.349
138 100.000 100.000 99.765 96.874 91.447
139 100.000 100.000 99.789 96.928 91.531
140 100.000 100.000 99.812 96.981 91.614
141 100.000 100.000 99.836 97.035 91.697
142 100.000 100.000 99.859 97.089 91.780
143 100.000 100.000 99.883 97.142 91.863
144 100.000 100.000 99.906 97.196 91.947
145 100.000 100.000 99.930 97.249 92.030
146 100.000 100.000 99.953 97.303 92.113
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 66.667 35.533 19.521 11.426 7.134
4 76.190 43.486 24.941 14.964 9.457
5 80.952 49.577 29.775 18.342 11.743
6 85.714 55.668 34.608 21.720 14.028
7 92.857 62.098 39.301 24.964 16.249
8 95.238 66.836 43.524 28.100 18.451
9 97.619 70.389 46.715 30.594 20.310
10 100.000 73.435 49.648 33.049 22.218
11 100.000 75.635 52.229 35.301 23.994
12 100.000 77.496 54.364 37.151 25.464
13 100.000 79.357 56.617 39.183 27.119
14 100.000 81.049 58.611 40.952 28.552
15 100.000 82.572 60.676 42.947 30.237
16 100.000 83.926 62.694 44.957 31.943
17 100.000 85.279 64.735 46.989 33.661
18 100.000 86.464 66.448 48.737 35.207
19 100.000 87.648 68.067 50.319 36.566
20 100.000 88.663 69.451 51.702 37.796
21 100.000 89.509 70.695 53.016 39.013
22 100.000 90.355 72.102 54.560 40.437
23 100.000 91.201 73.322 55.836 41.615
24 100.000 92.047 74.542 57.107 42.780
25 100.000 92.893 75.809 58.426 43.974
26 100.000 93.570 76.795 59.466 44.945
27 100.000 94.247 77.945 60.742 46.138
28 100.000 94.755 78.672 61.509 46.859
29 100.000 95.262 79.587 62.619 47.983
30 100.000 95.770 80.432 63.616 48.986
31 100.000 96.108 80.995 64.286 49.683
32 100.000 96.447 81.699 65.181 50.613
33 100.000 96.785 82.426 66.131 51.617
34 100.000 97.124 82.942 66.715 52.207
35 100.000 97.462 83.458 67.294 52.786
36 100.000 97.800 84.186 68.222 53.746
37 100.000 98.139 85.007 69.267 54.798
38 100.000 98.477 85.687 70.141 55.726
39 100.000 98.816 86.204 70.720 56.305
40 100.000 99.154 86.861 71.546 57.163
41 100.000 99.323 87.166 71.932 57.586
42 100.000 99.492 87.494 72.361 58.062
43 100.000 99.662 88.175 73.379 59.179
44 100.000 99.831 88.503 73.803 59.641
45 100.000 100.000 88.832 74.243 60.136
46 100.000 100.000 89.043 74.650 60.665
47 100.000 100.000 89.230 75.015 61.142
48 100.000 100.000 89.559 75.631 61.904
49 100.000 100.000 89.747 76.007 62.405
50 100.000 100.000 89.840 76.205 62.687
51 100.000 100.000 89.958 76.436 62.994
52 100.000 100.000 90.075 76.671 63.313
53 100.000 100.000 90.521 77.449 64.223
54 100.000 100.000 90.756 77.948 64.900
55 100.000 100.000 90.943 78.312 65.377
56 100.000 100.000 91.389 79.090 66.287
57 100.000 100.000 91.436 79.197 66.452
58 100.000 100.000 91.600 79.524 66.890
59 100.000 100.000 91.835 79.969 67.458
60 100.000 100.000 92.046 80.355 67.935
61 100.000 100.000 92.140 80.569 68.251
62 100.000 100.000 92.210 80.714 68.452
63 100.000 100.000 92.351 81.020 68.887
64 100.000 100.000 92.398 81.127 69.052
65 100.000 100.000 92.445 81.250 69.253
66 100.000 100.000 92.445 81.283 69.336
67 100.000 100.000 92.586 81.577 69.747
68 100.000 100.000 92.633 81.690 69.924
69 100.000 100.000 92.679 81.803 70.102
70 100.000 100.000 92.726 81.910 70.265
71 100.000 100.000 92.773 82.012 70.414
72 100.000 100.000 92.820 82.114 70.564
73 100.000 100.000 92.938 82.371 70.934
74 100.000 100.000 92.985 82.489 71.125
75 100.000 100.000 93.055 82.660 71.393
76 100.000 100.000 93.196 82.955 71.788
77 100.000 100.000 93.407 83.374 72.334
78 100.000 100.000 93.430 83.438 72.445
79 100.000 100.000 93.618 83.813 72.942
80 100.000 100.000 93.712 84.006 73.210
81 100.000 100.000 93.993 84.553 73.917
82 100.000 100.000 94.040 84.687 74.142
83 100.000 100.000 94.064 84.757 74.268
84 100.000 100.000 94.087 84.821 74.379
85 100.000 100.000 94.158 84.966 74.581
86 100.000 100.000 94.181 85.030 74.692
87 100.000 100.000 94.369 85.373 75.117
88 100.000 100.000 94.369 85.406 75.200
89 100.000 100.000 94.510 85.690 75.587
90 100.000 100.000 94.627 85.920 75.894
91 100.000 100.000 94.674 86.070 76.160
92 100.000 100.000 94.721 86.172 76.310
93 100.000 100.000 94.862 86.456 76.696
94 100.000 100.000 94.932 86.633 76.970
95 100.000 100.000 95.026 86.826 77.238
96 100.000 100.000 95.096 86.977 77.454
97 100.000 100.000 95.214 87.245 77.850
98 100.000 100.000 95.354 87.550 78.285
99 100.000 100.000 95.354 87.582 78.372
100 100.000 100.000 95.378 87.641 78.470
101 100.000 100.000 95.495 87.883 78.801
102 100.000 100.000 95.589 88.086 79.096
103 100.000 100.000 95.659 88.231 79.298
104 100.000 100.000 95.706 88.333 79.448
105 100.000 100.000 95.753 88.446 79.625
106 100.000 100.000 95.894 88.719 79.984
107 100.000 100.000 96.176 89.255 80.655
108 100.000 100.000 96.269 89.448 80.927
109 100.000 100.000 96.340 89.604 81.156
110 100.000 100.000 96.340 89.636 81.247
111 100.000 100.000 96.340 89.652 81.293
112 100.000 100.000 96.387 89.754 81.443
113 100.000 100.000 96.481 89.958 81.735
114 100.000 100.000 96.481 89.968 81.769
115 100.000 100.000 96.527 90.076 81.933
116 100.000 100.000 96.692 90.397 82.355
117 100.000 100.000 96.762 90.569 82.621
118 100.000 100.000 96.786 90.622 82.704
119 100.000 100.000 96.809 90.676 82.788
120 100.000 100.000 96.809 90.698 82.847
121 100.000 100.000 96.856 90.799 82.996
122 100.000 100.000 96.903 90.901 83.146
123 100.000 100.000 96.950 91.014 83.324
124 100.000 100.000 96.997 91.121 83.488
125 100.000 100.000 97.067 91.293 83.756
126 100.000 100.000 97.114 91.400 83.921
127 100.000 100.000 97.114 91.421 83.984
128 100.000 100.000 97.137 91.507 84.145
129 100.000 100.000 97.161 91.598 84.319
130 100.000 100.000 97.161 91.609 84.350
131 100.000 100.000 97.161 91.641 84.433
132 100.000 100.000 97.255 91.850 84.731
133 100.000 100.000 97.255 91.893 84.838
134 100.000 100.000 97.302 91.995 84.988
135 100.000 100.000 97.325 92.054 85.086
136 100.000 100.000 97.396 92.204 85.302
137 100.000 100.000 97.396 92.210 85.319
138 100.000 100.000 97.396 92.285 85.502
139 100.000 100.000 97.396 92.295 85.534
140 100.000 100.000 97.419 92.376 85.687
141 100.000 100.000 97.419 92.392 85.731
142 100.000 100.000 97.513 92.596 86.024
143 100.000 100.000 97.536 92.660 86.136
144 100.000 100.000 97.583 92.762 86.286
145 100.000 100.000 97.607 92.815 86.369
146 100.000 100.000 97.630 92.869 86.453
147 100.000 100.000 97.654 92.923 86.536
148 100.000 100.000 97.654 92.923 86.540
149 100.000 100.000 97.654 92.928 86.556
150 100.000 100.000 97.701 93.030 86.706
151 100.000 100.000 97.748 93.142 86.884
152 100.000 100.000 97.771 93.207 86.998
153 100.000 100.000 97.818 93.330 87.200
154 100.000 100.000 97.818 93.346 87.246
155 100.000 100.000 97.888 93.496 87.462
156 100.000 100.000 97.888 93.496 87.466
157 100.000 100.000 97.912 93.577 87.616
158 100.000 100.000 97.912 93.582 87.632
159 100.000 100.000 97.912 93.614 87.721
160 100.000 100.000 97.912 93.641 87.797
161 100.000 100.000 97.912 93.646 87.814
162 100.000 100.000 97.959 93.748 87.963
163 100.000 100.000 97.959 93.759 87.995
164 100.000 100.000 98.006 93.861 88.145
165 100.000 100.000 98.029 93.920 88.243
166 100.000 100.000 98.053 93.995 88.383
167 100.000 100.000 98.099 94.102 88.552
168 100.000 100.000 98.123 94.172 88.681
169 100.000 100.000 98.123 94.193 88.740
170 100.000 100.000 98.170 94.295 88.890
171 100.000 100.000 98.170 94.376 89.090
172 100.000 100.000 98.193 94.429 89.173
173 100.000 100.000 98.287 94.633 89.465
174 100.000 100.000 98.358 94.778 89.667
175 100.000 100.000 98.498 95.073 90.077
176 100.000 100.000 98.592 95.260 90.332
177 100.000 100.000 98.592 95.266 90.349
178 100.000 100.000 98.639 95.368 90.499
179 100.000 100.000 98.639 95.373 90.515
180 100.000 100.000 98.686 95.528 90.789
181 100.000 100.000 98.733 95.630 90.939
182 100.000 100.000 98.756 95.684 91.022
183 100.000 100.000 98.780 95.737 91.105
184 100.000 100.000 98.780 95.754 91.155
185 100.000 100.000 98.803 95.813 91.253
186 100.000 100.000 98.827 95.866 91.336
187 100.000 100.000 98.850 95.952 91.499
188 100.000 100.000 98.850 95.963 91.531
189 100.000 100.000 98.874 96.032 91.658
190 100.000 100.000 98.874 96.048 91.704
191 100.000 100.000 98.921 96.150 91.856
192 100.000 100.000 98.921 96.161 91.887
193 100.000 100.000 98.921 96.172 91.919
194 100.000 100.000 98.921 96.193 91.980
195 100.000 100.000 98.921 96.220 92.056
196 100.000 100.000 98.944 96.274 92.139
197 100.000 100.000 98.944 96.290 92.185
198 100.000 100.000 98.968 96.349 92.283
199 100.000 100.000 99.015 96.451 92.433
200 100.000 100.000 99.015 96.472 92.492
201 100.000 100.000 99.061 96.574 92.642
202 100.000 100.000 99.085 96.628 92.725
203 100.000 100.000 99.108 96.681 92.808
204 100.000 100.000 99.108 96.729 92.932
205 100.000 100.000 99.108 96.740 92.964
206 100.000 100.000 99.108 96.756 93.008
207 100.000 100.000 99.108 96.767 93.040
208 100.000 100.000 99.108 96.772 93.056
209 100.000 100.000 99.108 96.778 93.073
210 100.000 100.000 99.108 96.794 93.119
211 100.000 100.000 99.132 96.847 93.204
212 100.000 100.000 99.155 96.901 93.287
213 100.000 100.000 99.155 96.906 93.306
214 100.000 100.000 99.155 96.928 93.365
215 100.000 100.000 99.155 96.933 93.384
216 100.000 100.000 99.179 96.992 93.482
217 100.000 100.000 99.179 96.997 93.498
218 100.000 100.000 99.226 97.121 93.700
219 100.000 100.000 99.226 97.121 93.707
220 100.000 100.000 99.226 97.126 93.724
221 100.000 100.000 99.320 97.341 94.044
222 100.000 100.000 99.320 97.357 94.088
223 100.000 100.000 99.343 97.410 94.171
224 100.000 100.000 99.343 97.416 94.188
225 100.000 100.000 99.343 97.432 94.238
226 100.000 100.000 99.343 97.432 94.242
227 100.000 100.000 99.390 97.560 94.460
228 100.000 100.000 99.413 97.614 94.543
229 100.000 100.000 99.460 97.721 94.706
230 100.000 100.000 99.460 97.732 94.737
231 100.000 100.000 99.460 97.737 94.756
232 100.000 100.000 99.460 97.743 94.772
233 100.000 100.000 99.554 97.946 95.068
234 100.000 100.000 99.554 97.963 95.114
235 100.000 100.000 99.578 98.016 95.198
236 100.000 100.000 99.578 98.038 95.257
237 100.000 100.000 99.578 98.043 95.273
238 100.000 100.000 99.578 98.043 95.275
239 100.000 100.000 99.578 98.064 95.334
240 100.000 100.000 99.578 98.086 95.394
241 100.000 100.000 99.578 98.091 95.410
242 100.000 100.000 99.578 98.097 95.427
243 100.000 100.000 99.578 98.107 95.458
244 100.000 100.000 99.601 98.161 95.542
245 100.000 100.000 99.625 98.215 95.625
246 100.000 100.000 99.625 98.215 95.627
247 100.000 100.000 99.672 98.327 95.804
248 100.000 100.000 99.695 98.397 95.926
249 100.000 100.000 99.695 98.408 95.961
250 100.000 100.000 99.695 98.418 95.993
251 100.000 100.000 99.695 98.429 96.024
252 100.000 100.000 99.695 98.450 96.087
253 100.000 100.000 99.742 98.552 96.237
254 100.000 100.000 99.742 98.558 96.253
255 100.000 100.000 99.742 98.568 96.289
256 100.000 100.000 99.742 98.601 96.376
257 100.000 100.000 99.742 98.606 96.392
258 100.000 100.000 99.765 98.660 96.475
259 100.000 100.000 99.765 98.665 96.492
260 100.000 100.000 99.812 98.778 96.670
261 100.000 100.000 99.812 98.788 96.701
262 100.000 100.000 99.836 98.853 96.812
263 100.000 100.000 99.836 98.858 96.829
264 100.000 100.000 99.836 98.858 96.836
265 100.000 100.000 99.859 98.912 96.919
266 100.000 100.000 99.859 98.917 96.936
267 100.000 100.000 99.859 98.944 97.010
268 100.000 100.000 99.859 98.944 97.012
269 100.000 100.000 99.859 98.954 97.043
270 100.000 100.000 99.859 98.965 97.075
271 100.000 100.000 99.906 99.067 97.224
272 100.000 100.000 99.930 99.121 97.308
273 100.000 100.000 99.930 99.185 97.470
274 100.000 100.000 99.930 99.185 97.474
275 100.000 100.000 99.930 99.190 97.491
276 100.000 100.000 99.930 99.196 97.507
277 100.000 100.000 99.930 99.217 97.568
278 100.000 100.000 99.930 99.223 97.585
279 100.000 100.000 99.930 99.233 97.616
280 100.000 100.000 99.930 99.239 97.633
281 100.000 100.000 99.930 99.239 97.635
282 100.000 100.000 99.930 99.244 97.653
283 100.000 100.000 99.930 99.260 97.698
284 100.000 100.000 99.930 99.287 97.770
285 100.000 100.000 99.930 99.308 97.829
286 100.000 100.000 99.930 99.319 97.860
287 100.000 100.000 99.930 99.324 97.877
288 100.000 100.000 99.930 99.335 97.909
289 100.000 100.000 99.930 99.335 97.910
290 100.000 100.000 99.930 99.341 97.927
291 100.000 100.000 99.930 99.351 97.962
292 100.000 100.000 99.930 99.351 97.964
293 100.000 100.000 99.930 99.362 97.995
294 100.000 100.000 99.930 99.367 98.012
295 100.000 100.000 99.930 99.367 98.014
296 100.000 100.000 99.930 99.373 98.031
297 100.000 100.000 99.930 99.378 98.047
298 100.000 100.000 99.930 99.383 98.064
299 100.000 100.000 99.930 99.389 98.081
300 100.000 100.000 99.930 99.394 98.097
301 100.000 100.000 99.930 99.399 98.119
302 100.000 100.000 99.930 99.410 98.154
303 100.000 100.000 99.930 99.421 98.186
304 100.000 100.000 99.930 99.442 98.245
305 100.000 100.000 99.930 99.464 98.308
306 100.000 100.000 99.930 99.469 98.325
307 100.000 100.000 99.930 99.485 98.371
308 100.000 100.000 99.930 99.496 98.402
309 100.000 100.000 99.953 99.550 98.485
310 100.000 100.000 99.953 99.555 98.502
311 100.000 100.000 99.953 99.560 98.519
312 100.000 100.000 99.953 99.566 98.535
313 100.000 100.000 99.953 99.566 98.537
314 100.000 100.000 99.953 99.571 98.554
315 100.000 100.000 99.953 99.582 98.589
316 100.000 100.000 100.000 99.694 98.767
317 100.000 100.000 100.000 99.716 98.826
318 100.000 100.000 100.000 99.727 98.857
319 100.000 100.000 100.000 99.732 98.874
320 100.000 100.000 100.000 99.743 98.905
321 100.000 100.000 100.000 99.748 98.922
322 100.000 100.000 100.000 99.748 98.931
323 100.000 100.000 100.000 99.748 98.935
324 100.000 100.000 100.000 99.753 98.951
325 100.000 100.000 100.000 99.759 98.968
326 100.000 100.000 100.000 99.759 98.970
327 100.000 100.000 100.000 99.759 98.977
328 100.000 100.000 100.000 99.759 98.981
329 100.000 100.000 100.000 99.769 99.016
330 100.000 100.000 100.000 99.780 99.048
331 100.000 100.000 100.000 99.786 99.064
332 100.000 100.000 100.000 99.791 99.081
333 100.000 100.000 100.000 99.812 99.140
334 100.000 100.000 100.000 99.823 99.172
335 100.000 100.000 100.000 99.828 99.188
336 100.000 100.000 100.000 99.834 99.205
337 100.000 100.000 100.000 99.845 99.236
338 100.000 100.000 100.000 99.866 99.295
339 100.000 100.000 100.000 99.866 99.299
340 100.000 100.000 100.000 99.866 99.301
341 100.000 100.000 100.000 99.877 99.336
342 100.000 100.000 100.000 99.877 99.338
343 100.000 100.000 100.000 99.887 99.369
344 100.000 100.000 100.000 99.893 99.386
345 100.000 100.000 100.000 99.903 99.417
346 100.000 100.000 100.000 99.914 99.449
347 100.000 100.000 100.000 99.920 99.466
348 100.000 100.000 100.000 99.920 99.469
349 100.000 100.000 100.000 99.930 99.504
350 100.000 100.000 100.000 99.936 99.521
351 100.000 100.000 100.000 99.941 99.538
352 100.000 100.000 100.000 99.946 99.554
353 100.000 100.000 100.000 99.957 99.586
354 100.000 100.000 100.000 99.973 99.630
355 100.000 100.000 100.000 99.979 99.647
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 61.905 33.841 19.052 11.318 7.116
4 71.429 41.794 24.472 14.857 9.438
5 78.571 48.900 29.634 18.321 11.741
6 85.714 55.330 34.374 21.613 13.991
7 88.095 59.391 38.128 24.530 16.112
8 92.857 63.959 41.835 27.334 18.159
9 97.619 68.866 45.730 30.197 20.214
10 100.000 72.420 49.061 32.867 22.215
11 100.000 74.958 51.971 35.355 24.140
12 100.000 77.157 54.622 37.708 26.009
13 100.000 79.188 57.156 39.998 27.844
14 100.000 81.049 59.456 42.110 29.582
15 100.000 82.572 61.520 44.116 31.283
16 100.000 84.095 63.491 45.987 32.864
17 100.000 85.279 65.275 47.820 34.475
18 100.000 86.464 66.917 49.467 35.927
19 100.000 87.479 68.466 51.102 37.408
20 100.000 88.663 70.108 52.748 38.859
21 100.000 89.679 71.563 54.254 40.228
22 100.000 90.694 72.994 55.718 41.548
23 100.000 91.371 74.214 57.128 42.896
24 100.000 92.047 75.340 58.394 44.104
25 100.000 92.724 76.420 59.605 45.276
26 100.000 93.401 77.475 60.790 46.430
27 100.000 93.739 78.297 61.879 47.567
28 100.000 94.416 79.282 62.930 48.562
29 100.000 94.585 79.962 63.932 49.653
30 100.000 95.093 80.831 64.951 50.667
31 100.000 95.601 81.699 65.964 51.674
32 100.000 96.108 82.543 66.962 52.688
33 100.000 96.616 83.365 67.894 53.607
34 100.000 96.954 84.069 68.801 54.563
35 100.000 97.124 84.679 69.696 55.537
36 100.000 97.631 85.453 70.565 56.401
37 100.000 97.970 86.133 71.428 57.309
38 100.000 98.308 86.767 72.205 58.112
39 100.000 98.646 87.400 72.988 58.927
40 100.000 98.816 87.870 73.701 59.767
41 100.000 98.985 88.339 74.377 60.529
42 100.000 99.323 88.855 74.956 61.107
43 100.000 99.323 89.183 75.551 61.832
44 100.000 99.492 89.606 76.157 62.528
45 100.000 99.492 89.934 76.752 63.252
46 100.000 99.492 90.239 77.320 63.959
47 100.000 99.492 90.544 77.883 64.656
48 100.000 99.492 90.826 78.414 65.325
49 100.000 99.492 91.084 78.907 65.956
50 100.000 99.492 91.342 79.406 66.599
51 100.000 99.492 91.577 79.862 67.191
52 100.000 99.492 91.811 80.307 67.759
53 100.000 99.662 92.140 80.746 68.255
54 100.000 99.831 92.468 81.207 68.802
55 100.000 99.831 92.679 81.626 69.347
56 100.000 99.831 92.891 82.028 69.863
57 100.000 99.831 93.102 82.441 70.403
58 100.000 99.831 93.313 82.859 70.956
59 100.000 99.831 93.501 83.213 71.406
60 100.000 99.831 93.688 83.583 71.894
61 100.000 100.000 93.970 83.926 72.265
62 100.000 100.000 94.158 84.296 72.756
63 100.000 100.000 94.345 84.671 73.257
64 100.000 100.000 94.533 85.025 73.706
65 100.000 100.000 94.697 85.352 74.144
66 100.000 100.000 94.862 85.700 74.631
67 100.000 100.000 95.002 85.985 75.010
68 100.000 100.000 95.143 86.290 75.444
69 100.000 100.000 95.284 86.585 75.857
70 100.000 100.000 95.425 86.875 76.258
71 100.000 100.000 95.565 87.148 76.617
72 100.000 100.000 95.683 87.389 76.953
73 100.000 100.000 95.800 87.636 77.301
74 100.000 100.000 95.917 87.883 77.649
75 100.000 100.000 96.035 88.129 77.996
76 100.000 100.000 96.152 88.387 78.368
77 100.000 100.000 96.269 88.623 78.688
78 100.000 100.000 96.387 88.864 79.023
79 100.000 100.000 96.481 89.057 79.291
80 100.000 100.000 96.574 89.271 79.614
81 100.000 100.000 96.668 89.480 79.919
82 100.000 100.000 96.762 89.673 80.188
83 100.000 100.000 96.856 89.888 80.507
84 100.000 100.000 96.950 90.092 80.796
85 100.000 100.000 97.044 90.285 81.064
86 100.000 100.000 97.137 90.478 81.332
87 100.000 100.000 97.231 90.703 81.678
88 100.000 100.000 97.325 90.917 81.998
89 100.000 100.000 97.419 91.110 82.266
90 100.000 100.000 97.489 91.261 82.482
91 100.000 100.000 97.560 91.432 82.754
92 100.000 100.000 97.630 91.588 82.984
93 100.000 100.000 97.701 91.732 83.185
94 100.000 100.000 97.771 91.893 83.429
95 100.000 100.000 97.841 92.054 83.671
96 100.000 100.000 97.912 92.204 83.888
97 100.000 100.000 97.982 92.392 84.193
98 100.000 100.000 98.053 92.547 84.422
99 100.000 100.000 98.123 92.697 84.639
100 100.000 100.000 98.193 92.848 84.855
101 100.000 100.000 98.264 93.030 85.147
102 100.000 100.000 98.334 93.207 85.428
103 100.000 100.000 98.381 93.319 85.606
104 100.000 100.000 98.428 93.432 85.783
105 100.000 100.000 98.475 93.561 85.998
106 100.000 100.000 98.522 93.684 86.201
107 100.000 100.000 98.569 93.813 86.419
108 100.000 100.000 98.616 93.915 86.569
109 100.000 100.000 98.663 94.038 86.771
110 100.000 100.000 98.710 94.167 86.989
111 100.000 100.000 98.756 94.279 87.168
112 100.000 100.000 98.803 94.402 87.370
113 100.000 100.000 98.850 94.504 87.520
114 100.000 100.000 98.897 94.606 87.669
115 100.000 100.000 98.944 94.708 87.819
116 100.000 100.000 98.991 94.831 88.021
117 100.000 100.000 99.038 94.933 88.171
118 100.000 100.000 99.085 95.046 88.350
119 100.000 100.000 99.132 95.158 88.527
120 100.000 100.000 99.179 95.271 88.705
121 100.000 100.000 99.226 95.373 88.855
122 100.000 100.000 99.249 95.427 88.938
123 100.000 100.000 99.273 95.485 89.036
124 100.000 100.000 99.296 95.550 89.149
125 100.000 100.000 99.320 95.609 89.247
126 100.000 100.000 99.343 95.679 89.373
127 100.000 100.000 99.366 95.743 89.485
128 100.000 100.000 99.390 95.813 89.611
129 100.000 100.000 99.413 95.866 89.694
130 100.000 100.000 99.437 95.941 89.833
131 100.000 100.000 99.460 96.038 90.025
132 100.000 100.000 99.484 96.113 90.164
133 100.000 100.000 99.507 96.172 90.262
134 100.000 100.000 99.531 96.225 90.345
135 100.000 100.000 99.554 96.290 90.456
136 100.000 100.000 99.578 96.365 90.595
137 100.000 100.000 99.601 96.445 90.747
138 100.000 100.000 99.625 96.510 90.857
139 100.000 100.000 99.648 96.585 90.996
140 100.000 100.000 99.672 96.638 91.079
141 100.000 100.000 99.695 96.708 91.205
142 100.000 100.000 99.718 96.767 91.303
143 100.000 100.000 99.742 96.821 91.386
144 100.000 100.000 99.765 96.906 91.549
145 100.000 100.000 99.789 96.976 91.673
146 100.000 100.000 99.812 97.030 91.756
147 100.000 100.000 99.836 97.089 91.858
148 100.000 100.000 99.859 97.142 91.941
149 100.000 100.000 99.883 97.201 92.039
150 100.000 100.000 99.906 97.266 92.150
151 100.000 100.000 99.930 97.325 92.248
152 100.000 100.000 99.953 97.378 92.331
153 100.000 100.000 99.977 97.432 92.415
154 100.000 100.000 100.000 97.501 92.540
155 100.000 100.000 100.000 97.512 92.572
156 100.000 100.000 100.000 97.518 92.588
157 100.000 100.000 100.000 97.534 92.633
158 100.000 100.000 100.000 97.544 92.666
159 100.000 100.000 100.000 97.560 92.712
160 100.000 100.000 100.000 97.566 92.731
161 100.000 100.000 100.000 97.593 92.805
162 100.000 100.000 100.000 97.614 92.864
163 100.000 100.000 100.000 97.630 92.910
164 100.000 100.000 100.000 97.668 93.006
165 100.000 100.000 100.000 97.700 93.093
166 100.000 100.000 100.000 97.716 93.138
167 100.000 100.000 100.000 97.727 93.169
168 100.000 100.000 100.000 97.748 93.228
169 100.000 100.000 100.000 97.759 93.260
170 100.000 100.000 100.000 97.775 93.306
171 100.000 100.000 100.000 97.780 93.326
172 100.000 100.000 100.000 97.786 93.345
173 100.000 100.000 100.000 97.807 93.404
174 100.000 100.000 100.000 97.812 93.420
175 100.000 100.000 100.000 97.823 93.452
176 100.000 100.000 100.000 97.845 93.511
177 100.000 100.000 100.000 97.855 93.543
178 100.000 100.000 100.000 97.877 93.605
179 100.000 100.000 100.000 97.946 93.777
180 100.000 100.000 100.000 97.963 93.822
181 100.000 100.000 100.000 97.979 93.866
182 100.000 100.000 100.000 97.984 93.885
183 100.000 100.000 100.000 98.005 93.944
184 100.000 100.000 100.000 98.016 93.975
185 100.000 100.000 100.000 98.027 94.010
186 100.000 100.000 100.000 98.059 94.094
187 100.000 100.000 100.000 98.075 94.140
188 100.000 100.000 100.000 98.102 94.212
189 100.000 100.000 100.000 98.113 94.243
190 100.000 100.000 100.000 98.123 94.279
191 100.000 100.000 100.000 98.198 94.458
192 100.000 100.000 100.000 98.204 94.475
193 100.000 100.000 100.000 98.247 94.585
194 100.000 100.000 100.000 98.257 94.621
195 100.000 100.000 100.000 98.322 94.780
196 100.000 100.000 100.000 98.333 94.811
197 100.000 100.000 100.000 98.365 94.898
198 100.000 100.000 100.000 98.370 94.915
199 100.000 100.000 100.000 98.386 94.961
200 100.000 100.000 100.000 98.402 95.007
201 100.000 100.000 100.000 98.429 95.079
202 100.000 100.000 100.000 98.445 95.125
203 100.000 100.000 100.000 98.467 95.185
204 100.000 100.000 100.000 98.483 95.233
205 100.000 100.000 100.000 98.499 95.277
206 100.000 100.000 100.000 98.515 95.325
207 100.000 100.000 100.000 98.531 95.370
208 100.000 100.000 100.000 98.542 95.401
209 100.000 100.000 100.000 98.552 95.447
210 100.000 100.000 100.000 98.568 95.493
211 100.000 100.000 100.000 98.601 95.577
212 100.000 100.000 100.000 98.606 95.593
213 100.000 100.000 100.000 98.649 95.704
214 100.000 100.000 100.000 98.660 95.736
215 100.000 100.000 100.000 98.665 95.754
216 100.000 100.000 100.000 98.670 95.771
217 100.000 100.000 100.000 98.676 95.787
218 100.000 100.000 100.000 98.697 95.847
219 100.000 100.000 100.000 98.713 95.891
220 100.000 100.000 100.000 98.719 95.908
221 100.000 100.000 100.000 98.724 95.924
222 100.000 100.000 100.000 98.745 95.984
223 100.000 100.000 100.000 98.751 96.000
224 100.000 100.000 100.000 98.756 96.017
225 100.000 100.000 100.000 98.761 96.033
226 100.000 100.000 100.000 98.778 96.080
227 100.000 100.000 100.000 98.778 96.089
228 100.000 100.000 100.000 98.788 96.120
229 100.000 100.000 100.000 98.831 96.231
230 100.000 100.000 100.000 98.837 96.248
231 100.000 100.000 100.000 98.842 96.266
232 100.000 100.000 100.000 98.853 96.298
233 100.000 100.000 100.000 98.879 96.370
234 100.000 100.000 100.000 98.912 96.457
235 100.000 100.000 100.000 98.917 96.474
236 100.000 100.000 100.000 98.922 96.490
237 100.000 100.000 100.000 98.944 96.549
238 100.000 100.000 100.000 98.949 96.568
239 100.000 100.000 100.000 98.971 96.627
240 100.000 100.000 100.000 99.003 96.714
241 100.000 100.000 100.000 99.013 96.745
242 100.000 100.000 100.000 99.019 96.762
243 100.000 100.000 100.000 99.035 96.808
244 100.000 100.000 100.000 99.040 96.825
245 100.000 100.000 100.000 99.056 96.869
246 100.000 100.000 100.000 99.056 96.873
247 100.000 100.000 100.000 99.078 96.932
248 100.000 100.000 100.000 99.089 96.967
249 100.000 100.000 100.000 99.094 96.986
250 100.000 100.000 100.000 99.105 97.017
251 100.000 100.000 100.000 99.121 97.065
252 100.000 100.000 100.000 99.131 97.097
253 100.000 100.000 100.000 99.137 97.115
254 100.000 100.000 100.000 99.147 97.150
255 100.000 100.000 100.000 99.164 97.197
256 100.000 100.000 100.000 99.169 97.213
257 100.000 100.000 100.000 99.180 97.245
258 100.000 100.000 100.000 99.190 97.276
259 100.000 100.000 100.000 99.212 97.339
260 100.000 100.000 100.000 99.217 97.356
261 100.000 100.000 100.000 99.282 97.518
262 100.000 100.000 100.000 99.292 97.550
263 100.000 100.000 100.000 99.303 97.581
264 100.000 100.000 100.000 99.303 97.585
265 100.000 100.000 100.000 99.330 97.661
266 100.000 100.000 100.000 99.330 97.664
267 100.000 100.000 100.000 99.335 97.681
268 100.000 100.000 100.000 99.341 97.701
269 100.000 100.000 100.000 99.351 97.733
270 100.000 100.000 100.000 99.357 97.750
271 100.000 100.000 100.000 99.378 97.809
272 100.000 100.000 100.000 99.389 97.840
273 100.000 100.000 100.000 99.394 97.857
274 100.000 100.000 100.000 99.405 97.888
275 100.000 100.000 100.000 99.410 97.905
276 100.000 100.000 100.000 99.416 97.923
277 100.000 100.000 100.000 99.421 97.942
278 100.000 100.000 100.000 99.426 97.958
279 100.000 100.000 100.000 99.432 97.975
280 100.000 100.000 100.000 99.442 98.007
281 100.000 100.000 100.000 99.448 98.023
282 100.000 100.000 100.000 99.469 98.082
283 100.000 100.000 100.000 99.501 98.169
284 100.000 100.000 100.000 99.501 98.173
285 100.000 100.000 100.000 99.501 98.175
286 100.000 100.000 100.000 99.512 98.206
287 100.000 100.000 100.000 99.534 98.265
288 100.000 100.000 100.000 99.544 98.297
289 100.000 100.000 100.000 99.555 98.328
290 100.000 100.000 100.000 99.555 98.330
291 100.000 100.000 100.000 99.566 98.362
292 100.000 100.000 100.000 99.571 98.378
293 100.000 100.000 100.000 99.576 98.395
294 100.000 100.000 100.000 99.587 98.426
295 100.000 100.000 100.000 99.593 98.443
296 100.000 100.000 100.000 99.598 98.460
297 100.000 100.000 100.000 99.598 98.463
298 100.000 100.000 100.000 99.609 98.495
299 100.000 100.000 100.000 99.609 98.498
300 100.000 100.000 100.000 99.619 98.530
301 100.000 100.000 100.000 99.625 98.547
302 100.000 100.000 100.000 99.635 98.582
303 100.000 100.000 100.000 99.646 98.613
304 100.000 100.000 100.000 99.651 98.630
305 100.000 100.000 100.000 99.662 98.661
306 100.000 100.000 100.000 99.668 98.678
307 100.000 100.000 100.000 99.668 98.685
308 100.000 100.000 100.000 99.700 98.772
309 100.000 100.000 100.000 99.721 98.831
310 100.000 100.000 100.000 99.727 98.848
311 100.000 100.000 100.000 99.732 98.866
312 100.000 100.000 100.000 99.753 98.926
313 100.000 100.000 100.000 99.764 98.957
314 100.000 100.000 100.000 99.769 98.974
315 100.000 100.000 100.000 99.780 99.005
316 100.000 100.000 100.000 99.786 99.022
317 100.000 100.000 100.000 99.818 99.109
318 100.000 100.000 100.000 99.823 99.125
319 100.000 100.000 100.000 99.823 99.129
320 100.000 100.000 100.000 99.823 99.133
321 100.000 100.000 100.000 99.823 99.135
322 100.000 100.000 100.000 99.828 99.151
323 100.000 100.000 100.000 99.828 99.153
324 100.000 100.000 100.000 99.828 99.160
325 100.000 100.000 100.000 99.828 99.162
326 100.000 100.000 100.000 99.828 99.164
327 100.000 100.000 100.000 99.839 99.196
328 100.000 100.000 100.000 99.861 99.255
329 100.000 100.000 100.000 99.861 99.257
330 100.000 100.000 100.000 99.866 99.273
331 100.000 100.000 100.000 99.871 99.295
332 100.000 100.000 100.000 99.877 99.312
333 100.000 100.000 100.000 99.887 99.344
334 100.000 100.000 100.000 99.887 99.345
335 100.000 100.000 100.000 99.898 99.377
336 100.000 100.000 100.000 99.909 99.408
337 100.000 100.000 100.000 99.920 99.440
338 100.000 100.000 100.000 99.925 99.456
339 100.000 100.000 100.000 99.925 99.458
340 100.000 100.000 100.000 99.930 99.475
341 100.000 100.000 100.000 99.941 99.510
342 100.000 100.000 100.000 99.946 99.527
343 100.000 100.000 100.000 99.946 99.530
344 100.000 100.000 100.000 99.946 99.532
345 100.000 100.000 100.000 99.952 99.549
346 100.000 100.000 100.000 99.962 99.580
347 100.000 100.000 100.000 99.968 99.597
348 100.000 100.000 100.000 99.973 99.614
349 100.000 100.000 100.000 99.979 99.630
350 100.000 100.000 100.000 99.984 99.647
351 100.000 100.000 100.000 99.984 99.650
352 100.000 100.000 100.000 99.984 99.654
353 100.000 100.000 100.000 99.989 99.671
354 100.000 100.000 100.000 99.989 99.673
355 100.000 100.000 100.000 99.995 99.689
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 59.524 32.826 18.700 11.211 7.088
4 69.048 40.778 24.097 14.728 9.400
5 76.190 47.547 29.071 18.128 11.687
6 80.952 53.299 33.693 21.420 13.947
7 83.333 57.868 37.799 24.503 16.131
8 88.095 62.944 41.952 27.575 18.309
9 90.476 66.328 45.096 30.122 20.251
10 95.238 70.897 48.733 32.851 22.250
11 97.619 74.112 51.807 35.387 24.195
12 97.619 75.973 54.223 37.633 26.028
13 100.000 79.019 57.039 39.955 27.842
14 100.000 80.541 59.057 41.896 29.489
15 100.000 81.387 60.535 43.553 31.022
16 100.000 82.741 62.459 45.456 32.653
17 100.000 84.095 64.313 47.284 34.236
18 100.000 84.772 65.580 48.753 35.634
19 100.000 85.787 67.081 50.319 37.049
20 100.000 86.633 68.419 51.772 38.399
21 100.000 87.310 69.568 53.107 39.703
22 100.000 88.494 71.164 54.683 41.088
23 100.000 89.340 72.478 56.104 42.410
24 100.000 90.017 73.627 57.407 43.656
25 100.000 90.186 74.402 58.522 44.829
26 100.000 90.863 75.504 59.756 46.008
27 100.000 91.709 76.725 61.037 47.198
28 100.000 92.386 77.757 62.168 48.287
29 100.000 92.555 78.390 63.096 49.287
30 100.000 93.232 79.399 64.200 50.356
31 100.000 93.570 80.103 65.101 51.297
32 100.000 93.909 80.784 65.975 52.226
33 100.000 94.585 81.746 66.994 53.193
34 100.000 95.093 82.543 67.916 54.128
35 100.000 95.431 83.200 68.752 55.020
36 100.000 95.601 83.717 69.508 55.872
37 100.000 95.939 84.374 70.334 56.738
38 100.000 96.616 85.312 71.304 57.642
39 100.000 96.954 85.946 72.103 58.489
40 100.000 97.293 86.579 72.902 59.339
41 100.000 97.293 86.884 73.497 60.101
42 100.000 97.293 87.189 74.066 60.811
43 100.000 97.800 87.893 74.838 61.590
44 100.000 97.800 88.198 75.401 62.287
45 100.000 97.800 88.480 75.937 62.958
46 100.000 97.970 88.878 76.521 63.646
47 100.000 97.970 89.160 77.047 64.305
48 100.000 98.477 89.887 77.814 65.026
49 100.000 98.816 90.427 78.446 65.684
50 100.000 98.816 90.662 78.902 66.276
51 100.000 98.985 91.037 79.433 66.894
52 100.000 98.985 91.272 79.888 67.485
53 100.000 99.154 91.647 80.409 68.079
54 100.000 99.154 91.811 80.784 68.621
55 100.000 99.154 91.999 81.181 69.166
56 100.000 99.154 92.210 81.599 69.712
57 100.000 99.323 92.539 82.065 70.272
58 100.000 99.492 92.891 82.553 70.838
59 100.000 99.492 93.078 82.939 71.363
60 100.000 99.492 93.290 83.341 71.879
61 100.000 99.492 93.477 83.717 72.380
62 100.000 99.492 93.665 84.092 72.881
63 100.000 99.492 93.853 84.462 73.371
64 100.000 99.492 94.017 84.805 73.847
65 100.000 99.662 94.345 85.245 74.342
66 100.000 99.831 94.650 85.647 74.804
67 100.000 99.831 94.791 85.947 75.228
68 100.000 99.831 94.932 86.247 75.651
69 100.000 99.831 95.073 86.548 76.075
70 100.000 99.831 95.214 86.843 76.485
71 100.000 99.831 95.307 87.089 76.874
72 100.000 99.831 95.425 87.352 77.260
73 100.000 99.831 95.542 87.609 77.632
74 100.000 99.831 95.636 87.845 77.996
75 100.000 99.831 95.753 88.103 78.368
76 100.000 99.831 95.894 88.381 78.741
77 100.000 99.831 96.011 88.633 79.102
78 100.000 99.831 96.105 88.859 79.446
79 100.000 99.831 96.222 89.105 79.794
80 100.000 99.831 96.340 89.352 80.141
81 100.000 99.831 96.457 89.598 80.489
82 100.000 99.831 96.527 89.797 80.818
83 100.000 99.831 96.621 90.017 81.151
84 100.000 99.831 96.715 90.226 81.460
85 100.000 99.831 96.809 90.429 81.748
86 100.000 99.831 96.879 90.612 82.044
87 100.000 99.831 96.973 90.810 82.329
88 100.000 99.831 97.067 91.009 82.608
89 100.000 99.831 97.114 91.164 82.886
90 100.000 100.000 97.372 91.459 83.191
91 100.000 100.000 97.466 91.657 83.477
92 100.000 100.000 97.560 91.850 83.745
93 100.000 100.000 97.607 92.000 84.010
94 100.000 100.000 97.677 92.172 84.280
95 100.000 100.000 97.748 92.338 84.533
96 100.000 100.000 97.818 92.504 84.790
97 100.000 100.000 97.888 92.671 85.047
98 100.000 100.000 97.959 92.831 85.288
99 100.000 100.000 98.029 92.992 85.534
100 100.000 100.000 98.099 93.148 85.763
101 100.000 100.000 98.170 93.298 85.979
102 100.000 100.000 98.217 93.427 86.197
103 100.000 100.000 98.287 93.577 86.414
104 100.000 100.000 98.358 93.727 86.630
105 100.000 100.000 98.428 93.877 86.847
106 100.000 100.000 98.498 94.027 87.063
107 100.000 100.000 98.569 94.177 87.279
108 100.000 100.000 98.639 94.327 87.496
109 100.000 100.000 98.686 94.451 87.701
110 100.000 100.000 98.733 94.574 87.906
111 100.000 100.000 98.780 94.697 88.111
112 100.000 100.000 98.827 94.821 88.319
113 100.000 100.000 98.874 94.944 88.520
114 100.000 100.000 98.921 95.057 88.705
115 100.000 100.000 98.968 95.169 88.883
116 100.000 100.000 99.015 95.282 89.060
117 100.000 100.000 99.061 95.389 89.225
118 100.000 100.000 99.061 95.453 89.387
119 100.000 100.000 99.108 95.561 89.552
120 100.000 100.000 99.155 95.668 89.717
121 100.000 100.000 99.179 95.754 89.887
122 100.000 100.000 99.226 95.861 90.055
123 100.000 100.000 99.249 95.947 90.221
124 100.000 100.000 99.249 96.011 90.377
125 100.000 100.000 99.273 96.091 90.532
126 100.000 100.000 99.320 96.193 90.682
127 100.000 100.000 99.343 96.274 90.837
128 100.000 100.000 99.366 96.354 90.992
129 100.000 100.000 99.413 96.456 91.142
130 100.000 100.000 99.437 96.536 91.296
131 100.000 100.000 99.484 96.638 91.446
132 100.000 100.000 99.484 96.692 91.581
133 100.000 100.000 99.507 96.767 91.719
134 100.000 100.000 99.531 96.842 91.860
135 100.000 100.000 99.554 96.917 92.000
136 100.000 100.000 99.578 96.987 92.126
137 100.000 100.000 99.601 97.056 92.252
138 100.000 100.000 99.625 97.126 92.378
139 100.000 100.000 99.648 97.190 92.488
140 100.000 100.000 99.672 97.255 92.599
141 100.000 100.000 99.695 97.319 92.712
142 100.000 100.000 99.718 97.384 92.825
143 100.000 100.000 99.742 97.448 92.938
144 100.000 100.000 99.765 97.512 93.052
145 100.000 100.000 99.789 97.577 93.163
146 100.000 100.000 99.789 97.614 93.263
147 100.000 100.000 99.812 97.673 93.361
148 100.000 100.000 99.836 97.732 93.459
149 100.000 100.000 99.859 97.791 93.557
150 100.000 100.000 99.883 97.850 93.655
151 100.000 100.000 99.883 97.888 93.752
152 100.000 100.000 99.883 97.925 93.851
153 100.000 100.000 99.883 97.963 93.955
154 100.000 100.000 99.906 98.022 94.057
155 100.000 100.000 99.906 98.054 94.144
156 100.000 100.000 99.906 98.086 94.230
157 100.000 100.000 99.930 98.140 94.314
158 100.000 100.000 99.953 98.193 94.397
159 100.000 100.000 99.953 98.225 94.484
160 100.000 100.000 99.977 98.279 94.567
161 100.000 100.000 99.977 98.311 94.654
162 100.000 100.000 99.977 98.343 94.745
163 100.000 100.000 99.977 98.375 94.835
164 100.000 100.000 100.000 98.429 94.918
165 100.000 100.000 100.000 98.456 94.990
166 100.000 100.000 100.000 98.483 95.063
167 100.000 100.000 100.000 98.509 95.135
168 100.000 100.000 100.000 98.536 95.211
169 100.000 100.000 100.000 98.563 95.283
170 100.000 100.000 100.000 98.590 95.355
171 100.000 100.000 100.000 98.617 95.427
172 100.000 100.000 100.000 98.638 95.490
173 100.000 100.000 100.000 98.660 95.549
174 100.000 100.000 100.000 98.681 95.608
175 100.000 100.000 100.000 98.702 95.667
176 100.000 100.000 100.000 98.724 95.728
177 100.000 100.000 100.000 98.745 95.791
178 100.000 100.000 100.000 98.767 95.850
179 100.000 100.000 100.000 98.788 95.911
180 100.000 100.000 100.000 98.810 95.971
181 100.000 100.000 100.000 98.831 96.033
182 100.000 100.000 100.000 98.853 96.093
183 100.000 100.000 100.000 98.874 96.152
184 100.000 100.000 100.000 98.896 96.211
185 100.000 100.000 100.000 98.917 96.274
186 100.000 100.000 100.000 98.938 96.335
187 100.000 100.000 100.000 98.960 96.394
188 100.000 100.000 100.000 98.981 96.453
189 100.000 100.000 100.000 99.003 96.512
190 100.000 100.000 100.000 99.024 96.572
191 100.000 100.000 100.000 99.046 96.631
192 100.000 100.000 100.000 99.067 96.690
193 100.000 100.000 100.000 99.089 96.751
194 100.000 100.000 100.000 99.105 96.797
195 100.000 100.000 100.000 99.121 96.843
196 100.000 100.000 100.000 99.137 96.893
197 100.000 100.000 100.000 99.153 96.945
198 100.000 100.000 100.000 99.169 96.991
199 100.000 100.000 100.000 99.185 97.038
200 100.000 100.000 100.000 99.201 97.086
201 100.000 100.000 100.000 99.217 97.132
202 100.000 100.000 100.000 99.233 97.178
203 100.000 100.000 100.000 99.249 97.224
204 100.000 100.000 100.000 99.265 97.271
205 100.000 100.000 100.000 99.282 97.317
206 100.000 100.000 100.000 99.298 97.365
207 100.000 100.000 100.000 99.314 97.409
208 100.000 100.000 100.000 99.324 97.441
209 100.000 100.000 100.000 99.335 97.472
210 100.000 100.000 100.000 99.346 97.504
211 100.000 100.000 100.000 99.357 97.535
212 100.000 100.000 100.000 99.367 97.566
213 100.000 100.000 100.000 99.378 97.598
214 100.000 100.000 100.000 99.389 97.629
215 100.000 100.000 100.000 99.399 97.661
216 100.000 100.000 100.000 99.410 97.694
217 100.000 100.000 100.000 99.421 97.725
218 100.000 100.000 100.000 99.432 97.761
219 100.000 100.000 100.000 99.442 97.792
220 100.000 100.000 100.000 99.453 97.823
221 100.000 100.000 100.000 99.464 97.855
222 100.000 100.000 100.000 99.475 97.886
223 100.000 100.000 100.000 99.485 97.918
224 100.000 100.000 100.000 99.496 97.949
225 100.000 100.000 100.000 99.507 97.981
226 100.000 100.000 100.000 99.517 98.012
227 100.000 100.000 100.000 99.528 98.044
228 100.000 100.000 100.000 99.539 98.075
229 100.000 100.000 100.000 99.550 98.106
230 100.000 100.000 100.000 99.560 98.138
231 100.000 100.000 100.000 99.571 98.171
232 100.000 100.000 100.000 99.582 98.204
233 100.000 100.000 100.000 99.593 98.236
234 100.000 100.000 100.000 99.603 98.267
235 100.000 100.000 100.000 99.614 98.299
236 100.000 100.000 100.000 99.625 98.334
237 100.000 100.000 100.000 99.635 98.365
238 100.000 100.000 100.000 99.646 98.397
239 100.000 100.000 100.000 99.657 98.428
240 100.000 100.000 100.000 99.668 98.463
241 100.000 100.000 100.000 99.678 98.495
242 100.000 100.000 100.000 99.684 98.513
243 100.000 100.000 100.000 99.689 98.530
244 100.000 100.000 100.000 99.694 98.558
245 100.000 100.000 100.000 99.700 98.574
246 100.000 100.000 100.000 99.705 98.591
247 100.000 100.000 100.000 99.710 98.608
248 100.000 100.000 100.000 99.716 98.624
249 100.000 100.000 100.000 99.721 98.641
250 100.000 100.000 100.000 99.727 98.657
251 100.000 100.000 100.000 99.732 98.676
252 100.000 100.000 100.000 99.737 98.694
253 100.000 100.000 100.000 99.743 98.711
254 100.000 100.000 100.000 99.748 98.730
255 100.000 100.000 100.000 99.753 98.746
256 100.000 100.000 100.000 99.759 98.763
257 100.000 100.000 100.000 99.764 98.780
258 100.000 100.000 100.000 99.769 98.796
259 100.000 100.000 100.000 99.775 98.813
260 100.000 100.000 100.000 99.780 98.831
261 100.000 100.000 100.000 99.786 98.848
262 100.000 100.000 100.000 99.791 98.865
263 100.000 100.000 100.000 99.796 98.881
264 100.000 100.000 100.000 99.802 98.898
265 100.000 100.000 100.000 99.807 98.915
266 100.000 100.000 100.000 99.812 98.931
267 100.000 100.000 100.000 99.818 98.948
268 100.000 100.000 100.000 99.823 98.964
269 100.000 100.000 100.000 99.828 98.981
270 100.000 100.000 100.000 99.834 98.998
271 100.000 100.000 100.000 99.839 99.014
272 100.000 100.000 100.000 99.845 99.031
273 100.000 100.000 100.000 99.850 99.050
274 100.000 100.000 100.000 99.855 99.066
275 100.000 100.000 100.000 99.861 99.083
276 100.000 100.000 100.000 99.866 99.099
277 100.000 100.000 100.000 99.871 99.116
278 100.000 100.000 100.000 99.877 99.133
279 100.000 100.000 100.000 99.882 99.151
280 100.000 100.000 100.000 99.887 99.168
281 100.000 100.000 100.000 99.893 99.184
282 100.000 100.000 100.000 99.898 99.201
283 100.000 100.000 100.000 99.903 99.221
284 100.000 100.000 100.000 99.909 99.240
285 100.000 100.000 100.000 99.914 99.257
286 100.000 100.000 100.000 99.920 99.275
287 100.000 100.000 100.000 99.925 99.292
288 100.000 100.000 100.000 99.930 99.308
289 100.000 100.000 100.000 99.936 99.325
290 100.000 100.000 100.000 99.941 99.342
291 100.000 100.000 100.000 99.946 99.358
292 100.000 100.000 100.000 99.952 99.377
293 100.000 100.000 100.000 99.957 99.393
294 100.000 100.000 100.000 99.962 99.410
295 100.000 100.000 100.000 99.968 99.429
296 100.000 100.000 100.000 99.973 99.447
297 100.000 100.000 100.000 99.979 99.464
298 100.000 100.000 100.000 99.984 99.480
299 100.000 100.000 100.000 99.989 99.497
300 100.000 100.000 100.000 99.995 99.514
301 100.000 100.000 100.000 100.000 99.530
302 100.000 100.000 100.000 100.000 99.532
303 100.000 100.000 100.000 100.000 99.540
304 100.000 100.000 100.000 100.000 99.543
305 100.000 100.000 100.000 100.000 99.545
306 100.000 100.000 100.000 100.000 99.549
307 100.000 100.000 100.000 100.000 99.552
308 100.000 100.000 100.000 100.000 99.554
309 100.000 100.000 100.000 100.000 99.556
310 100.000 100.000 100.000 100.000 99.560
311 100.000 100.000 100.000 100.000 99.564
312 100.000 100.000 100.000 100.000 99.565
313 100.000 100.000 100.000 100.000 99.569
314 100.000 100.000 100.000 100.000 99.573
315 100.000 100.000 100.000 100.000 99.577
316 100.000 100.000 100.000 100.000 99.580
317 100.000 100.000 100.000 100.000 99.584
318 100.000 100.000 100.000 100.000 99.588
319 100.000 100.000 100.000 100.000 99.591
320 100.000 100.000 100.000 100.000 99.599
321 100.000 100.000 100.000 100.000 99.601
322 100.000 100.000 100.000 100.000 99.604
323 100.000 100.000 100.000 100.000 99.606
324 100.000 100.000 100.000 100.000 99.610
325 100.000 100.000 100.000 100.000 99.614
326 100.000 100.000 100.000 100.000 99.615
327 100.000 100.000 100.000 100.000 99.617
328 100.000 100.000 100.000 100.000 99.619
329 100.000 100.000 100.000 100.000 99.621
330 100.000 100.000 100.000 100.000 99.623
331 100.000 100.000 100.000 100.000 99.625
332 100.000 100.000 100.000 100.000 99.626
333 100.000 100.000 100.000 100.000 99.636
334 100.000 100.000 100.000 100.000 99.638
335 100.000 100.000 100.000 100.000 99.639
336 100.000 100.000 100.000 100.000 99.641
337 100.000 100.000 100.000 100.000 99.645
338 100.000 100.000 100.000 100.000 99.647
339 100.000 100.000 100.000 100.000 99.650
340 100.000 100.000 100.000 100.000 99.654
341 100.000 100.000 100.000 100.000 99.658
342 100.000 100.000 100.000 100.000 99.665
343 100.000 100.000 100.000 100.000 99.673
344 100.000 100.000 100.000 100.000 99.675
345 100.000 100.000 100.000 100.000 99.678
346 100.000 100.000 100.000 100.000 99.680
347 100.000 100.000 100.000 100.000 99.684
348 100.000 100.000 100.000 100.000 99.686
349 100.000 100.000 100.000 100.000 99.687
350 100.000 100.000 100.000 100.000 99.689
351 100.000 100.000 100.000 100.000 99.691
352 100.000 100.000 100.000 100.000 99.695
353 100.000 100.000 100.000 100.000 99.697
354 100.000 100.000 100.000 100.000 99.700
355 100.000 100.000 100.000 100.000 99.704
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 59.524 32.487 18.442 11.077 7.033
4 64.286 37.563 22.548 14.101 9.181
5 71.429 43.993 27.241 17.345 11.402
6 76.190 49.577 31.699 20.535 13.614
7 85.714 56.853 36.603 23.806 15.826
8 88.095 60.406 39.934 26.487 17.843
9 90.476 64.129 43.383 29.221 19.879
10 92.857 66.836 46.011 31.494 21.702
11 92.857 69.205 48.780 33.902 23.590
12 92.857 71.743 51.783 36.486 25.575
13 92.857 74.112 54.528 38.867 27.444
14 95.238 76.819 57.062 41.033 29.192
15 95.238 78.680 59.362 43.134 30.913
16 95.238 79.526 60.770 44.689 32.358
17 95.238 80.372 62.224 46.292 33.828
18 97.619 83.418 64.969 48.512 35.547
19 97.619 84.772 66.729 50.212 37.018
20 100.000 86.971 68.724 51.965 38.497
21 100.000 87.986 70.202 53.498 39.884
22 100.000 89.002 71.633 54.962 41.204
23 100.000 89.679 72.736 56.206 42.404
24 100.000 90.525 74.003 57.557 43.660
25 100.000 91.032 74.988 58.748 44.842
26 100.000 91.878 76.255 60.099 46.097
27 100.000 92.217 77.030 61.117 47.168
28 100.000 92.724 77.921 62.174 48.228
29 100.000 93.232 78.789 63.198 49.259
30 100.000 93.570 79.493 64.115 50.236
31 100.000 94.078 80.338 65.112 51.247
32 100.000 94.247 80.878 65.911 52.148
33 100.000 94.247 81.300 66.656 53.030
34 100.000 94.247 81.628 67.289 53.831
35 100.000 94.755 82.567 68.393 54.909
36 100.000 94.924 83.153 69.251 55.848
37 100.000 95.431 83.951 70.168 56.775
38 100.000 95.601 84.444 70.897 57.605
39 100.000 95.939 85.054 71.664 58.428
40 100.000 96.277 85.687 72.457 59.265
41 100.000 96.447 86.086 73.052 59.977
42 100.000 96.616 86.532 73.701 60.725
43 100.000 96.616 86.861 74.296 61.449
44 100.000 97.293 87.823 75.304 62.393
45 100.000 97.293 88.104 75.830 63.051
46 100.000 97.293 88.339 76.296 63.663
47 100.000 97.462 88.738 76.870 64.330
48 100.000 97.800 89.301 77.551 65.052
49 100.000 97.970 89.700 78.119 65.708
50 100.000 98.646 90.591 79.020 66.546
51 100.000 98.816 90.967 79.556 67.175
52 100.000 98.816 91.178 79.980 67.739
53 100.000 98.816 91.389 80.403 68.303
54 100.000 98.816 91.553 80.757 68.800
55 100.000 98.816 91.741 81.138 69.314
56 100.000 98.816 91.882 81.454 69.775
57 100.000 98.816 92.093 81.872 70.320
58 100.000 98.985 92.398 82.301 70.840
59 100.000 99.154 92.726 82.757 71.378
60 100.000 99.154 92.891 83.100 71.851
61 100.000 99.154 93.055 83.443 72.325
62 100.000 99.323 93.407 83.926 72.879
63 100.000 99.323 93.595 84.296 73.369
64 100.000 99.323 93.759 84.634 73.832
65 100.000 99.492 94.111 85.111 74.374
66 100.000 99.492 94.181 85.320 74.725
67 100.000 99.492 94.275 85.556 75.093
68 100.000 99.492 94.416 85.851 75.503
69 100.000 99.492 94.533 86.113 75.888
70 100.000 99.492 94.627 86.349 76.252
71 100.000 99.492 94.744 86.612 76.635
72 100.000 99.662 95.073 87.052 77.131
73 100.000 99.662 95.167 87.277 77.477
74 100.000 99.662 95.260 87.502 77.821
75 100.000 99.662 95.401 87.786 78.207
76 100.000 99.662 95.542 88.065 78.582
77 100.000 99.662 95.683 88.349 78.961
78 100.000 99.662 95.777 88.564 79.281
79 100.000 99.662 95.870 88.773 79.592
80 100.000 99.662 95.988 89.014 79.929
81 100.000 99.662 96.082 89.223 80.237
82 100.000 99.662 96.199 89.464 80.572
83 100.000 99.831 96.504 89.850 80.996
84 100.000 99.831 96.598 90.060 81.304
85 100.000 99.831 96.668 90.236 81.586
86 100.000 99.831 96.739 90.419 81.874
87 100.000 99.831 96.832 90.622 82.172
88 100.000 99.831 96.926 90.826 82.468
89 100.000 99.831 96.997 90.992 82.726
90 100.000 99.831 97.067 91.159 82.984
91 100.000 99.831 97.161 91.357 83.266
92 100.000 99.831 97.231 91.529 83.529
93 100.000 99.831 97.278 91.657 83.749
94 100.000 99.831 97.349 91.823 84.002
95 100.000 99.831 97.396 91.952 84.221
96 100.000 99.831 97.443 92.081 84.439
97 100.000 99.831 97.536 92.274 84.707
98 100.000 99.831 97.630 92.467 84.975
99 100.000 99.831 97.677 92.596 85.190
100 100.000 99.831 97.677 92.660 85.352
101 100.000 99.831 97.724 92.783 85.558
102 100.000 99.831 97.771 92.907 85.763
103 100.000 99.831 97.841 93.062 85.994
104 100.000 99.831 97.888 93.185 86.199
105 100.000 99.831 97.959 93.341 86.429
106 100.000 99.831 98.029 93.496 86.658
107 100.000 99.831 98.099 93.652 86.887
108 100.000 99.831 98.146 93.775 87.089
109 100.000 99.831 98.146 93.839 87.244
110 100.000 99.831 98.170 93.925 87.411
111 100.000 100.000 98.428 94.220 87.716
112 100.000 100.000 98.475 94.338 87.906
113 100.000 100.000 98.522 94.456 88.097
114 100.000 100.000 98.592 94.606 88.313
115 100.000 100.000 98.663 94.756 88.529
116 100.000 100.000 98.710 94.874 88.720
117 100.000 100.000 98.780 95.024 88.936
118 100.000 100.000 98.780 95.078 89.075
119 100.000 100.000 98.850 95.228 89.291
120 100.000 100.000 98.850 95.282 89.430
121 100.000 100.000 98.874 95.362 89.583
122 100.000 100.000 98.921 95.475 89.763
123 100.000 100.000 98.968 95.587 89.940
124 100.000 100.000 99.015 95.700 90.118
125 100.000 100.000 99.061 95.813 90.295
126 100.000 100.000 99.085 95.888 90.438
127 100.000 100.000 99.085 95.931 90.552
128 100.000 100.000 99.132 96.038 90.719
129 100.000 100.000 99.179 96.145 90.883
130 100.000 100.000 99.226 96.252 91.048
131 100.000 100.000 99.273 96.359 91.213
132 100.000 100.000 99.273 96.402 91.323
133 100.000 100.000 99.296 96.472 91.453
134 100.000 100.000 99.320 96.542 91.581
135 100.000 100.000 99.343 96.611 91.708
136 100.000 100.000 99.366 96.681 91.834
137 100.000 100.000 99.413 96.783 91.986
138 100.000 100.000 99.437 96.853 92.111
139 100.000 100.000 99.460 96.922 92.237
140 100.000 100.000 99.484 96.992 92.363
141 100.000 100.000 99.531 97.094 92.513
142 100.000 100.000 99.554 97.158 92.629
143 100.000 100.000 99.554 97.190 92.720
144 100.000 100.000 99.601 97.292 92.869
145 100.000 100.000 99.648 97.394 93.019
146 100.000 100.000 99.648 97.421 93.102
147 100.000 100.000 99.648 97.453 93.189
148 100.000 100.000 99.672 97.518 93.302
149 100.000 100.000 99.672 97.550 93.389
150 100.000 100.000 99.672 97.577 93.465
151 100.000 100.000 99.672 97.603 93.541
152 100.000 100.000 99.672 97.630 93.615
153 100.000 100.000 99.695 97.689 93.715
154 100.000 100.000 99.718 97.748 93.814
155 100.000 100.000 99.718 97.775 93.888
156 100.000 100.000 99.742 97.834 93.986
157 100.000 100.000 99.742 97.861 94.058
158 100.000 100.000 99.765 97.920 94.156
159 100.000 100.000 99.789 97.979 94.254
160 100.000 100.000 99.812 98.038 94.352
161 100.000 100.000 99.836 98.097 94.451
162 100.000 100.000 99.859 98.156 94.549
163 100.000 100.000 99.859 98.182 94.621
164 100.000 100.000 99.859 98.209 94.693
165 100.000 100.000 99.883 98.268 94.791
166 100.000 100.000 99.883 98.295 94.863
167 100.000 100.000 99.883 98.322 94.935
168 100.000 100.000 99.883 98.343 94.998
169 100.000 100.000 99.883 98.365 95.061
170 100.000 100.000 99.883 98.386 95.124
171 100.000 100.000 99.883 98.408 95.186
172 100.000 100.000 99.883 98.429 95.249
173 100.000 100.000 99.906 98.483 95.336
174 100.000 100.000 99.906 98.504 95.397
175 100.000 100.000 99.906 98.526 95.458
176 100.000 100.000 99.906 98.536 95.505
177 100.000 100.000 99.906 98.558 95.564
178 100.000 100.000 99.906 98.579 95.623
179 100.000 100.000 99.906 98.601 95.682
180 100.000 100.000 99.906 98.622 95.741
181 100.000 100.000 99.906 98.644 95.800
182 100.000 100.000 99.906 98.665 95.860
183 100.000 100.000 99.930 98.719 95.945
184 100.000 100.000 99.930 98.740 96.004
185 100.000 100.000 99.953 98.794 96.089
186 100.000 100.000 99.953 98.810 96.141
187 100.000 100.000 99.953 98.831 96.200
188 100.000 100.000 99.953 98.853 96.259
189 100.000 100.000 99.953 98.874 96.318
190 100.000 100.000 99.953 98.896 96.377
191 100.000 100.000 99.953 98.917 96.437
192 100.000 100.000 99.977 98.971 96.520
193 100.000 100.000 100.000 99.024 96.603
194 100.000 100.000 100.000 99.040 96.653
195 100.000 100.000 100.000 99.056 96.701
196 100.000 100.000 100.000 99.072 96.749
197 100.000 100.000 100.000 99.089 96.797
198 100.000 100.000 100.000 99.105 96.845
199 100.000 100.000 100.000 99.121 96.893
200 100.000 100.000 100.000 99.137 96.940
201 100.000 100.000 100.000 99.153 96.986
202 100.000 100.000 100.000 99.169 97.032
203 100.000 100.000 100.000 99.185 97.078
204 100.000 100.000 100.000 99.201 97.124
205 100.000 100.000 100.000 99.217 97.171
206 100.000 100.000 100.000 99.233 97.217
207 100.000 100.000 100.000 99.249 97.263
208 100.000 100.000 100.000 99.265 97.309
209 100.000 100.000 100.000 99.282 97.354
210 100.000 100.000 100.000 99.292 97.389
211 100.000 100.000 100.000 99.303 97.424
212 100.000 100.000 100.000 99.314 97.459
213 100.000 100.000 100.000 99.324 97.492
214 100.000 100.000 100.000 99.335 97.526
215 100.000 100.000 100.000 99.346 97.559
216 100.000 100.000 100.000 99.357 97.592
217 100.000 100.000 100.000 99.367 97.626
218 100.000 100.000 100.000 99.378 97.659
219 100.000 100.000 100.000 99.389 97.690
220 100.000 100.000 100.000 99.399 97.722
221 100.000 100.000 100.000 99.410 97.753
222 100.000 100.000 100.000 99.421 97.785
223 100.000 100.000 100.000 99.432 97.816
224 100.000 100.000 100.000 99.442 97.848
225 100.000 100.000 100.000 99.453 97.879
226 100.000 100.000 100.000 99.464 97.910
227 100.000 100.000 100.000 99.475 97.942
228 100.000 100.000 100.000 99.485 97.973
229 100.000 100.000 100.000 99.496 98.005
230 100.000 100.000 100.000 99.507 98.036
231 100.000 100.000 100.000 99.517 98.068
232 100.000 100.000 100.000 99.528 98.099
233 100.000 100.000 100.000 99.539 98.130
234 100.000 100.000 100.000 99.550 98.162
235 100.000 100.000 100.000 99.560 98.193
236 100.000 100.000 100.000 99.571 98.225
237 100.000 100.000 100.000 99.582 98.256
238 100.000 100.000 100.000 99.593 98.288
239 100.000 100.000 100.000 99.603 98.319
240 100.000 100.000 100.000 99.614 98.351
241 100.000 100.000 100.000 99.625 98.382
242 100.000 100.000 100.000 99.635 98.413
243 100.000 100.000 100.000 99.646 98.445
244 100.000 100.000 100.000 99.657 98.476
245 100.000 100.000 100.000 99.662 98.497
246 100.000 100.000 100.000 99.668 98.517
247 100.000 100.000 100.000 99.673 98.535
248 100.000 100.000 100.000 99.678 98.554
249 100.000 100.000 100.000 99.684 98.572
250 100.000 100.000 100.000 99.689 98.591
251 100.000 100.000 100.000 99.694 98.609
252 100.000 100.000 100.000 99.700 98.628
253 100.000 100.000 100.000 99.705 98.645
254 100.000 100.000 100.000 99.710 98.661
255 100.000 100.000 100.000 99.710 98.670
256 100.000 100.000 100.000 99.716 98.687
257 100.000 100.000 100.000 99.721 98.704
258 100.000 100.000 100.000 99.727 98.720
259 100.000 100.000 100.000 99.732 98.737
260 100.000 100.000 100.000 99.737 98.754
261 100.000 100.000 100.000 99.743 98.770
262 100.000 100.000 100.000 99.748 98.787
263 100.000 100.000 100.000 99.753 98.804
264 100.000 100.000 100.000 99.759 98.820
265 100.000 100.000 100.000 99.764 98.837
266 100.000 100.000 100.000 99.769 98.853
267 100.000 100.000 100.000 99.775 98.870
268 100.000 100.000 100.000 99.780 98.887
269 100.000 100.000 100.000 99.786 98.903
270 100.000 100.000 100.000 99.791 98.920
271 100.000 100.000 100.000 99.796 98.937
272 100.000 100.000 100.000 99.802 98.953
273 100.000 100.000 100.000 99.807 98.970
274 100.000 100.000 100.000 99.812 98.987
275 100.000 100.000 100.000 99.818 99.003
276 100.000 100.000 100.000 99.823 99.020
277 100.000 100.000 100.000 99.828 99.037
278 100.000 100.000 100.000 99.834 99.053
279 100.000 100.000 100.000 99.839 99.070
280 100.000 100.000 100.000 99.845 99.086
281 100.000 100.000 100.000 99.850 99.103
282 100.000 100.000 100.000 99.855 99.120
283 100.000 100.000 100.000 99.861 99.136
284 100.000 100.000 100.000 99.866 99.153
285 100.000 100.000 100.000 99.871 99.170
286 100.000 100.000 100.000 99.877 99.186
287 100.000 100.000 100.000 99.882 99.203
288 100.000 100.000 100.000 99.887 99.220
289 100.000 100.000 100.000 99.893 99.236
290 100.000 100.000 100.000 99.898 99.253
291 100.000 100.000 100.000 99.903 99.270
292 100.000 100.000 100.000 99.909 99.286
293 100.000 100.000 100.000 99.914 99.303
294 100.000 100.000 100.000 99.920 99.319
295 100.000 100.000 100.000 99.925 99.336
296 100.000 100.000 100.000 99.930 99.353
297 100.000 100.000 100.000 99.936 99.369
298 100.000 100.000 100.000 99.941 99.386
299 100.000 100.000 100.000 99.946 99.403
300 100.000 100.000 100.000 99.952 99.419
301 100.000 100.000 100.000 99.957 99.436
302 100.000 100.000 100.000 99.962 99.453
303 100.000 100.000 100.000 99.968 99.469
304 100.000 100.000 100.000 99.973 99.486
305 100.000 100.000 100.000 99.979 99.503
306 100.000 100.000 100.000 99.984 99.519
307 100.000 100.000 100.000 99.989 99.536
308 100.000 100.000 100.000 99.995 99.552
309 100.000 100.000 100.000 100.000 99.569
310 100.000 100.000 100.000 100.000 99.577
311 100.000 100.000 100.000 100.000 99.584
312 100.000 100.000 100.000 100.000 99.591
313 100.000 100.000 100.000 100.000 99.597
314 100.000 100.000 100.000 100.000 99.601
315 100.000 100.000 100.000 100.000 99.604
316 100.000 100.000 100.000 100.000 99.608
317 100.000 100.000 100.000 100.000 99.612
318 100.000 100.000 100.000 100.000 99.615
319 100.000 100.000 100.000 100.000 99.619
320 100.000 100.000 100.000 100.000 99.623
321 100.000 100.000 100.000 100.000 99.626
322 100.000 100.000 100.000 100.000 99.630
323 100.000 100.000 100.000 100.000 99.634
324 100.000 100.000 100.000 100.000 99.638
325 100.000 100.000 100.000 100.000 99.641
326 100.000 100.000 100.000 100.000 99.645
327 100.000 100.000 100.000 100.000 99.649
328 100.000 100.000 100.000 100.000 99.652
329 100.000 100.000 100.000 100.000 99.656
330 100.000 100.000 100.000 100.000 99.658
331 100.000 100.000 100.000 100.000 99.660
332 100.000 100.000 100.000 100.000 99.662
333 100.000 100.000 100.000 100.000 99.663
334 100.000 100.000 100.000 100.000 99.665
335 100.000 100.000 100.000 100.000 99.667
336 100.000 100.000 100.000 100.000 99.669
337 100.000 100.000 100.000 100.000 99.671
338 100.000 100.000 100.000 100.000 99.673
339 100.000 100.000 100.000 100.000 99.675
340 100.000 100.000 100.000 100.000 99.676
341 100.000 100.000 100.000 100.000 99.678
342 100.000 100.000 100.000 100.000 99.680
343 100.000 100.000 100.000 100.000 99.682
344 100.000 100.000 100.000 100.000 99.684
345 100.000 100.000 100.000 100.000 99.686
346 100.000 100.000 100.000 100.000 99.687
347 100.000 100.000 100.000 100.000 99.689
348 100.000 100.000 100.000 100.000 99.691
349 100.000 100.000 100.000 100.000 99.693
350 100.000 100.000 100.000 100.000 99.695
351 100.000 100.000 100.000 100.000 99.697
352 100.000 100.000 100.000 100.000 99.699
353 100.000 100.000 100.000 100.000 99.700
354 100.000 100.000 100.000 100.000 99.702
355 100.000 100.000 100.000 100.000 99.704
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 61.905 33.841 19.052 11.318 7.116
4 73.810 42.809 24.801 14.943 9.455
5 83.333 50.761 30.221 18.482 11.778
6 85.714 55.499 34.444 21.618 13.980
7 92.857 61.421 38.691 24.588 16.062
8 95.238 65.990 42.820 27.693 18.257
9 97.619 70.051 46.574 30.610 20.378
10 100.000 73.773 49.883 33.156 22.255
11 100.000 75.804 52.346 35.365 24.032
12 100.000 77.834 54.974 37.773 25.959
13 100.000 79.865 57.461 40.025 27.781
14 100.000 81.557 59.596 42.008 29.427
15 100.000 82.910 61.427 43.805 30.982
16 100.000 84.264 63.257 45.590 32.517
17 100.000 85.618 65.275 47.601 34.223
18 100.000 86.802 66.964 49.327 35.758
19 100.000 87.817 68.419 50.818 37.097
20 100.000 88.832 69.826 52.228 38.345
21 100.000 89.679 71.187 53.708 39.710
22 100.000 90.525 72.478 55.096 41.004
23 100.000 91.371 73.698 56.367 42.169
24 100.000 92.217 74.918 57.648 43.359
25 100.000 92.893 75.903 58.699 44.353
26 100.000 93.570 76.983 59.906 45.515
27 100.000 94.078 77.992 61.123 46.715
28 100.000 94.585 78.930 62.243 47.830
29 100.000 95.093 79.775 63.214 48.788
30 100.000 95.601 80.690 64.308 49.888
31 100.000 96.108 81.417 65.074 50.609
32 100.000 96.447 82.309 66.243 51.800
33 100.000 96.785 82.802 66.779 52.327
34 100.000 97.124 83.623 67.836 53.400
35 100.000 97.462 84.280 68.656 54.254
36 100.000 97.800 84.984 69.557 55.199
37 100.000 98.139 85.476 70.103 55.750
38 100.000 98.477 85.969 70.640 56.277
39 100.000 98.646 86.368 71.192 56.897
40 100.000 98.816 86.649 71.540 57.281
41 100.000 98.985 87.236 72.398 58.221
42 100.000 99.154 87.565 72.838 58.716
43 100.000 99.323 88.104 73.626 59.593
44 100.000 99.492 88.456 74.114 60.159
45 100.000 99.662 88.761 74.500 60.582
46 100.000 99.831 89.019 74.795 60.887
47 100.000 100.000 89.277 75.090 61.192
48 100.000 100.000 89.465 75.433 61.618
49 100.000 100.000 89.676 75.830 62.123
50 100.000 100.000 89.887 76.248 62.672
51 100.000 100.000 90.099 76.661 63.212
52 100.000 100.000 90.216 76.897 63.532
53 100.000 100.000 90.333 77.133 63.852
54 100.000 100.000 90.544 77.556 64.416
55 100.000 100.000 90.591 77.679 64.617
56 100.000 100.000 90.779 78.044 65.094
57 100.000 100.000 90.967 78.409 65.571
58 100.000 100.000 91.154 78.763 66.021
59 100.000 100.000 91.248 78.966 66.313
60 100.000 100.000 91.412 79.342 66.851
61 100.000 100.000 91.483 79.502 67.091
62 100.000 100.000 91.835 80.157 67.899
63 100.000 100.000 92.163 80.752 68.624
64 100.000 100.000 92.210 80.870 68.815
65 100.000 100.000 92.374 81.207 69.277
66 100.000 100.000 92.445 81.363 69.506
67 100.000 100.000 92.656 81.781 70.058
68 100.000 100.000 92.750 81.974 70.327
69 100.000 100.000 92.961 82.382 70.856
70 100.000 100.000 93.008 82.484 71.006
71 100.000 100.000 93.219 82.929 71.611
72 100.000 100.000 93.360 83.202 71.970
73 100.000 100.000 93.430 83.368 72.227
74 100.000 100.000 93.430 83.433 72.382
75 100.000 100.000 93.454 83.497 72.493
76 100.000 100.000 93.524 83.642 72.694
77 100.000 100.000 93.618 83.845 72.990
78 100.000 100.000 93.665 83.979 73.216
79 100.000 100.000 93.853 84.333 73.665
80 100.000 100.000 94.134 84.848 74.300
81 100.000 100.000 94.228 85.062 74.620
82 100.000 100.000 94.298 85.234 74.886
83 100.000 100.000 94.345 85.347 75.063
84 100.000 100.000 94.369 85.406 75.161
85 100.000 100.000 94.416 85.518 75.339
86 100.000 100.000 94.510 85.717 75.618
87 100.000 100.000 94.603 85.899 75.858
88 100.000 100.000 94.603 85.942 75.969
89 100.000 100.000 94.603 85.958 76.016
90 100.000 100.000 94.627 86.033 76.154
91 100.000 100.000 94.791 86.371 76.617
92 100.000 100.000 94.791 86.398 76.689
93 100.000 100.000 94.838 86.521 76.894
94 100.000 100.000 94.908 86.676 77.123
95 100.000 100.000 94.955 86.789 77.301
96 100.000 100.000 95.049 86.982 77.569
97 100.000 100.000 95.096 87.116 77.795
98 100.000 100.000 95.143 87.223 77.957
99 100.000 100.000 95.167 87.282 78.055
100 100.000 100.000 95.260 87.475 78.324
101 100.000 100.000 95.448 87.845 78.814
102 100.000 100.000 95.495 87.974 79.026
103 100.000 100.000 95.612 88.226 79.385
104 100.000 100.000 95.659 88.349 79.587
105 100.000 100.000 95.753 88.564 79.899
106 100.000 100.000 95.753 88.596 79.990
107 100.000 100.000 95.777 88.665 80.115
108 100.000 100.000 95.824 88.778 80.293
109 100.000 100.000 95.824 88.853 80.472
110 100.000 100.000 95.847 88.917 80.583
111 100.000 100.000 95.894 89.030 80.761
112 100.000 100.000 95.964 89.202 81.025
113 100.000 100.000 96.082 89.443 81.360
114 100.000 100.000 96.082 89.486 81.471
115 100.000 100.000 96.129 89.598 81.648
116 100.000 100.000 96.129 89.614 81.693
117 100.000 100.000 96.152 89.673 81.791
118 100.000 100.000 96.152 89.684 81.822
119 100.000 100.000 96.222 89.829 82.024
120 100.000 100.000 96.246 89.888 82.125
121 100.000 100.000 96.340 90.092 82.418
122 100.000 100.000 96.410 90.242 82.634
123 100.000 100.000 96.410 90.269 82.708
124 100.000 100.000 96.457 90.370 82.858
125 100.000 100.000 96.527 90.521 83.076
126 100.000 100.000 96.574 90.622 83.229
127 100.000 100.000 96.598 90.676 83.313
128 100.000 100.000 96.598 90.730 83.448
129 100.000 100.000 96.668 90.891 83.692
130 100.000 100.000 96.715 90.998 83.855
131 100.000 100.000 96.786 91.143 84.058
132 100.000 100.000 96.809 91.202 84.158
133 100.000 100.000 96.832 91.261 84.256
134 100.000 100.000 96.832 91.271 84.287
135 100.000 100.000 96.856 91.325 84.370
136 100.000 100.000 96.856 91.346 84.430
137 100.000 100.000 96.856 91.357 84.461
138 100.000 100.000 96.879 91.421 84.572
139 100.000 100.000 96.903 91.502 84.722
140 100.000 100.000 96.950 91.614 84.905
141 100.000 100.000 96.950 91.625 84.936
142 100.000 100.000 96.997 91.727 85.086
143 100.000 100.000 97.020 91.791 85.197
144 100.000 100.000 97.044 91.845 85.280
145 100.000 100.000 97.067 91.899 85.363
146 100.000 100.000 97.114 92.011 85.541
147 100.000 100.000 97.231 92.242 85.848
148 100.000 100.000 97.278 92.365 86.050
149 100.000 100.000 97.278 92.381 86.094
150 100.000 100.000 97.325 92.499 86.286
151 100.000 100.000 97.372 92.606 86.449
152 100.000 100.000 97.443 92.767 86.689
153 100.000 100.000 97.443 92.815 86.813
154 100.000 100.000 97.466 92.869 86.898
155 100.000 100.000 97.466 92.885 86.945
156 100.000 100.000 97.466 92.896 86.976
157 100.000 100.000 97.489 92.949 87.059
158 100.000 100.000 97.489 92.960 87.091
159 100.000 100.000 97.513 93.041 87.242
160 100.000 100.000 97.536 93.100 87.340
161 100.000 100.000 97.536 93.121 87.399
162 100.000 100.000 97.536 93.142 87.459
163 100.000 100.000 97.607 93.293 87.675
164 100.000 100.000 97.607 93.298 87.692
165 100.000 100.000 97.630 93.352 87.775
166 100.000 100.000 97.724 93.550 88.056
167 100.000 100.000 97.724 93.604 88.191
168 100.000 100.000 97.724 93.614 88.222
169 100.000 100.000 97.748 93.689 88.361
170 100.000 100.000 97.771 93.743 88.444
171 100.000 100.000 97.818 93.845 88.594
172 100.000 100.000 97.818 93.877 88.681
173 100.000 100.000 97.818 93.898 88.740
174 100.000 100.000 97.841 93.952 88.823
175 100.000 100.000 97.888 94.059 88.988
176 100.000 100.000 97.888 94.070 89.019
177 100.000 100.000 97.888 94.070 89.023
178 100.000 100.000 97.888 94.081 89.054
179 100.000 100.000 97.888 94.102 89.117
180 100.000 100.000 97.888 94.129 89.189
181 100.000 100.000 97.912 94.188 89.289
182 100.000 100.000 97.912 94.188 89.291
183 100.000 100.000 97.912 94.204 89.339
184 100.000 100.000 97.912 94.226 89.400
185 100.000 100.000 97.912 94.236 89.432
186 100.000 100.000 97.912 94.236 89.434
187 100.000 100.000 97.912 94.242 89.450
188 100.000 100.000 97.912 94.258 89.496
189 100.000 100.000 97.912 94.258 89.504
190 100.000 100.000 97.935 94.311 89.591
191 100.000 100.000 97.935 94.322 89.622
192 100.000 100.000 97.935 94.333 89.654
193 100.000 100.000 98.006 94.483 89.870
194 100.000 100.000 98.099 94.676 90.138
195 100.000 100.000 98.099 94.687 90.170
196 100.000 100.000 98.099 94.719 90.260
197 100.000 100.000 98.123 94.778 90.358
198 100.000 100.000 98.170 94.880 90.508
199 100.000 100.000 98.193 94.933 90.591
200 100.000 100.000 98.264 95.078 90.793
201 100.000 100.000 98.358 95.271 91.061
202 100.000 100.000 98.358 95.287 91.109
203 100.000 100.000 98.381 95.351 91.220
204 100.000 100.000 98.428 95.480 91.434
205 100.000 100.000 98.451 95.534 91.518
206 100.000 100.000 98.451 95.555 91.577
207 100.000 100.000 98.451 95.566 91.612
208 100.000 100.000 98.451 95.587 91.671
209 100.000 100.000 98.451 95.587 91.679
210 100.000 100.000 98.451 95.587 91.680
211 100.000 100.000 98.451 95.603 91.727
212 100.000 100.000 98.522 95.754 91.943
213 100.000 100.000 98.522 95.759 91.961
214 100.000 100.000 98.592 95.904 92.163
215 100.000 100.000 98.592 95.925 92.222
216 100.000 100.000 98.592 95.936 92.254
217 100.000 100.000 98.592 95.941 92.272
218 100.000 100.000 98.592 95.947 92.289
219 100.000 100.000 98.592 95.952 92.307
220 100.000 100.000 98.592 95.952 92.315
221 100.000 100.000 98.592 95.963 92.350
222 100.000 100.000 98.616 96.016 92.433
223 100.000 100.000 98.616 96.022 92.450
224 100.000 100.000 98.663 96.124 92.599
225 100.000 100.000 98.663 96.140 92.646
226 100.000 100.000 98.663 96.156 92.692
227 100.000 100.000 98.710 96.258 92.842
228 100.000 100.000 98.710 96.258 92.849
229 100.000 100.000 98.710 96.263 92.866
230 100.000 100.000 98.710 96.274 92.897
231 100.000 100.000 98.710 96.295 92.956
232 100.000 100.000 98.733 96.359 93.069
233 100.000 100.000 98.733 96.365 93.086
234 100.000 100.000 98.733 96.376 93.117
235 100.000 100.000 98.733 96.381 93.134
236 100.000 100.000 98.756 96.435 93.219
237 100.000 100.000 98.756 96.445 93.250
238 100.000 100.000 98.756 96.461 93.295
239 100.000 100.000 98.780 96.520 93.393
240 100.000 100.000 98.780 96.526 93.411
241 100.000 100.000 98.803 96.585 93.509
242 100.000 100.000 98.803 96.611 93.581
243 100.000 100.000 98.827 96.670 93.679
244 100.000 100.000 98.827 96.670 93.683
245 100.000 100.000 98.827 96.692 93.742
246 100.000 100.000 98.827 96.703 93.774
247 100.000 100.000 98.827 96.713 93.805
248 100.000 100.000 98.827 96.724 93.837
249 100.000 100.000 98.827 96.729 93.853
250 100.000 100.000 98.827 96.745 93.898
251 100.000 100.000 98.850 96.799 93.981
252 100.000 100.000 98.850 96.831 94.071
253 100.000 100.000 98.850 96.837 94.092
254 100.000 100.000 98.874 96.890 94.175
255 100.000 100.000 98.874 96.912 94.236
256 100.000 100.000 98.874 96.922 94.267
257 100.000 100.000 98.874 96.928 94.284
258 100.000 100.000 98.874 96.939 94.327
259 100.000 100.000 98.874 96.949 94.358
260 100.000 100.000 98.874 96.960 94.389
261 100.000 100.000 98.874 96.965 94.408
262 100.000 100.000 98.874 96.971 94.425
263 100.000 100.000 98.874 96.976 94.441
264 100.000 100.000 98.874 96.987 94.473
265 100.000 100.000 98.874 96.997 94.504
266 100.000 100.000 98.897 97.062 94.615
267 100.000 100.000 98.897 97.083 94.674
268 100.000 100.000 98.897 97.083 94.678
269 100.000 100.000 98.897 97.094 94.709
270 100.000 100.000 98.897 97.094 94.711
271 100.000 100.000 98.897 97.099 94.728
272 100.000 100.000 98.897 97.105 94.745
273 100.000 100.000 98.944 97.207 94.894
274 100.000 100.000 98.944 97.212 94.911
275 100.000 100.000 98.944 97.217 94.929
276 100.000 100.000 98.944 97.217 94.933
277 100.000 100.000 98.968 97.271 95.016
278 100.000 100.000 98.968 97.271 95.018
279 100.000 100.000 98.968 97.271 95.026
280 100.000 100.000 98.991 97.325 95.109
281 100.000 100.000 98.991 97.335 95.140
282 100.000 100.000 98.991 97.351 95.185
283 100.000 100.000 98.991 97.351 95.192
284 100.000 100.000 98.991 97.351 95.198
285 100.000 100.000 98.991 97.357 95.214
286 100.000 100.000 98.991 97.357 95.222
287 100.000 100.000 98.991 97.362 95.238
288 100.000 100.000 98.991 97.362 95.240
289 100.000 100.000 99.015 97.416 95.323
290 100.000 100.000 99.015 97.426 95.355
291 100.000 100.000 99.015 97.432 95.371
292 100.000 100.000 99.015 97.442 95.405
293 100.000 100.000 99.015 97.448 95.421
294 100.000 100.000 99.038 97.507 95.519
295 100.000 100.000 99.273 97.957 96.091
296 100.000 100.000 99.273 97.968 96.122
297 100.000 100.000 99.296 98.022 96.205
298 100.000 100.000 99.296 98.032 96.239
299 100.000 100.000 99.296 98.038 96.259
300 100.000 100.000 99.296 98.048 96.290
301 100.000 100.000 99.296 98.054 96.307
302 100.000 100.000 99.296 98.054 96.311
303 100.000 100.000 99.296 98.054 96.313
304 100.000 100.000 99.296 98.064 96.344
305 100.000 100.000 99.296 98.081 96.388
306 100.000 100.000 99.296 98.091 96.420
307 100.000 100.000 99.296 98.091 96.422
308 100.000 100.000 99.296 98.097 96.438
309 100.000 100.000 99.320 98.150 96.522
310 100.000 100.000 99.320 98.156 96.538
311 100.000 100.000 99.320 98.161 96.555
312 100.000 100.000 99.320 98.161 96.557
313 100.000 100.000 99.320 98.166 96.573
314 100.000 100.000 99.320 98.166 96.577
315 100.000 100.000 99.320 98.177 96.609
316 100.000 100.000 99.320 98.188 96.644
317 100.000 100.000 99.320 98.188 96.646
318 100.000 100.000 99.320 98.188 96.653
319 100.000 100.000 99.320 98.193 96.671
320 100.000 100.000 99.320 98.193 96.673
321 100.000 100.000 99.320 98.209 96.718
322 100.000 100.000 99.531 98.627 97.263
323 100.000 100.000 99.531 98.627 97.271
324 100.000 100.000 99.554 98.681 97.354
325 100.000 100.000 99.554 98.692 97.385
326 100.000 100.000 99.578 98.745 97.470
327 100.000 100.000 99.578 98.745 97.476
328 100.000 100.000 99.578 98.756 97.507
329 100.000 100.000 99.578 98.756 97.511
330 100.000 100.000 99.578 98.767 97.542
331 100.000 100.000 99.601 98.820 97.626
332 100.000 100.000 99.601 98.820 97.627
333 100.000 100.000 99.601 98.826 97.644
334 100.000 100.000 99.601 98.826 97.646
335 100.000 100.000 99.601 98.842 97.692
336 100.000 100.000 99.601 98.847 97.709
337 100.000 100.000 99.601 98.847 97.714
338 100.000 100.000 99.601 98.858 97.750
339 100.000 100.000 99.601 98.863 97.766
340 100.000 100.000 99.601 98.863 97.774
341 100.000 100.000 99.601 98.869 97.790
342 100.000 100.000 99.601 98.869 97.794
343 100.000 100.000 99.601 98.874 97.811
344 100.000 100.000 99.601 98.879 97.827
345 100.000 100.000 99.601 98.879 97.831
346 100.000 100.000 99.601 98.890 97.862
347 100.000 100.000 99.601 98.901 97.894
348 100.000 100.000 99.601 98.906 97.910
349 100.000 100.000 99.601 98.912 97.927
350 100.000 100.000 99.601 98.917 97.944
351 100.000 100.000 99.601 98.917 97.947
352 100.000 100.000 99.601 98.928 97.979
353 100.000 100.000 99.601 98.933 97.995
354 100.000 100.000 99.601 98.944 98.027
355 100.000 100.000 99.601 98.949 98.044
356 100.000 100.000 99.601 98.954 98.060
357 100.000 100.000 99.648 99.056 98.210
358 100.000 100.000 99.789 99.341 98.589
359 100.000 100.000 99.789 99.346 98.606
360 100.000 100.000 99.789 99.346 98.609
361 100.000 100.000 99.789 99.351 98.626
362 100.000 100.000 99.789 99.351 98.630
363 100.000 100.000 99.789 99.351 98.633
364 100.000 100.000 99.789 99.357 98.654
365 100.000 100.000 99.789 99.362 98.672
366 100.000 100.000 99.789 99.362 98.674
367 100.000 100.000 99.789 99.362 98.676
368 100.000 100.000 99.789 99.362 98.680
369 100.000 100.000 99.789 99.373 98.715
370 100.000 100.000 99.789 99.378 98.731
371 100.000 100.000 99.789 99.383 98.748
372 100.000 100.000 99.789 99.389 98.765
373 100.000 100.000 99.789 99.389 98.768
374 100.000 100.000 99.789 99.394 98.785
375 100.000 100.000 99.812 99.448 98.868
376 100.000 100.000 99.812 99.448 98.872
377 100.000 100.000 99.812 99.448 98.874
378 100.000 100.000 99.812 99.485 98.970
379 100.000 100.000 99.812 99.485 98.974
380 100.000 100.000 99.812 99.485 98.976
381 100.000 100.000 99.812 99.496 99.007
382 100.000 100.000 99.812 99.496 99.011
383 100.000 100.000 99.812 99.496 99.013
384 100.000 100.000 99.812 99.501 99.029
385 100.000 100.000 99.812 99.501 99.033
386 100.000 100.000 99.812 99.501 99.037
387 100.000 100.000 99.812 99.501 99.040
388 100.000 100.000 99.812 99.501 99.044
389 100.000 100.000 99.812 99.507 99.061
390 100.000 100.000 99.812 99.512 99.077
391 100.000 100.000 99.883 99.662 99.294
392 100.000 100.000 99.883 99.668 99.310
393 100.000 100.000 99.883 99.668 99.314
394 100.000 100.000 99.930 99.769 99.464
395 100.000 100.000 99.930 99.769 99.466
396 100.000 100.000 99.930 99.769 99.467
397 100.000 100.000 99.930 99.780 99.499
398 100.000 100.000 99.930 99.780 99.501
399 100.000 100.000 99.930 99.780 99.504
400 100.000 100.000 99.930 99.780 99.508
401 100.000 100.000 99.930 99.780 99.510
402 100.000 100.000 99.930 99.786 99.527
403 100.000 100.000 99.930 99.791 99.543
404 100.000 100.000 99.930 99.791 99.545
405 100.000 100.000 99.930 99.796 99.562
406 100.000 100.000 99.930 99.796 99.565
407 100.000 100.000 99.930 99.796 99.567
408 100.000 100.000 99.930 99.796 99.571
409 100.000 100.000 99.930 99.802 99.588
410 100.000 100.000 99.930 99.802 99.589
411 100.000 100.000 99.930 99.802 99.593
412 100.000 100.000 99.930 99.802 99.597
413 100.000 100.000 99.930 99.802 99.601
414 100.000 100.000 99.930 99.812 99.632
415 100.000 100.000 99.930 99.818 99.649
416 100.000 100.000 99.930 99.818 99.650
417 100.000 100.000 99.930 99.818 99.652
418 100.000 100.000 99.930 99.823 99.669
419 100.000 100.000 99.930 99.834 99.700
420 100.000 100.000 99.930 99.839 99.717
421 100.000 100.000 99.930 99.845 99.734
422 100.000 100.000 99.930 99.845 99.736
423 100.000 100.000 99.930 99.845 99.739
424 100.000 100.000 99.930 99.845 99.741
425 100.000 100.000 99.930 99.845 99.745
426 100.000 100.000 99.930 99.845 99.747
427 100.000 100.000 99.930 99.845 99.750
428 100.000 100.000 99.930 99.845 99.752
429 100.000 100.000 99.930 99.845 99.756
430 100.000 100.000 99.930 99.845 99.758
431 100.000 100.000 99.930 99.845 99.760
432 100.000 100.000 99.930 99.845 99.761
433 100.000 100.000 99.930 99.845 99.765
434 100.000 100.000 99.930 99.845 99.767
435 100.000 100.000 100.000 99.995 99.983
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 61.905 33.841 19.052 11.318 7.116
4 71.429 41.794 24.472 14.857 9.438
5 78.571 48.393 29.329 18.203 11.707
6 80.952 52.623 33.247 21.221 13.877
7 85.714 58.037 37.588 24.358 16.070
8 90.476 63.283 41.741 27.355 18.180
9 95.238 68.020 45.519 30.165 20.216
10 95.238 70.728 48.592 32.760 22.196
11 97.619 74.112 51.760 35.328 24.149
12 100.000 76.481 54.223 37.564 25.976
13 100.000 78.511 56.710 39.805 27.781
14 100.000 79.865 58.775 41.874 29.534
15 100.000 81.557 60.934 43.906 31.235
16 100.000 83.249 63.069 45.906 32.909
17 100.000 84.772 65.063 47.820 34.538
18 100.000 86.125 66.823 49.509 35.988
19 100.000 87.140 68.325 51.096 37.439
20 100.000 88.156 69.803 52.608 38.785
21 100.000 88.832 71.023 54.013 40.126
22 100.000 89.679 72.313 55.418 41.450
23 100.000 90.694 73.698 56.833 42.741
24 100.000 91.540 74.965 58.185 43.997
25 100.000 92.047 75.997 59.455 45.265
26 100.000 92.724 77.053 60.629 46.399
27 100.000 93.570 78.203 61.793 47.458
28 100.000 94.078 79.141 62.924 48.594
29 100.000 94.585 79.986 63.895 49.552
30 100.000 94.755 80.643 64.854 50.593
31 100.000 94.924 81.276 65.777 51.586
32 100.000 95.093 81.910 66.726 52.636
33 100.000 95.601 82.708 67.626 53.527
34 100.000 96.108 83.505 68.533 54.430
35 100.000 96.616 84.280 69.406 55.303
36 100.000 96.954 84.960 70.270 56.207
37 100.000 97.462 85.687 71.085 57.026
38 100.000 97.970 86.415 71.851 57.747
39 100.000 98.308 87.048 72.640 58.574
40 100.000 98.308 87.471 73.374 59.432
41 100.000 98.646 88.081 74.157 60.286
42 100.000 98.816 88.550 74.865 61.113
43 100.000 98.985 89.019 75.540 61.875
44 100.000 99.154 89.489 76.221 62.648
45 100.000 99.154 89.840 76.848 63.400
46 100.000 99.154 90.169 77.444 64.133
47 100.000 99.323 90.568 78.001 64.765
48 100.000 99.323 90.873 78.564 65.462
49 100.000 99.323 91.178 79.127 66.159
50 100.000 99.662 91.671 79.663 66.686
51 100.000 99.662 91.952 80.199 67.358
52 100.000 99.662 92.187 80.666 67.972
53 100.000 99.831 92.515 81.106 68.467
54 100.000 99.831 92.726 81.524 69.020
55 100.000 99.831 92.938 81.947 69.584
56 100.000 99.831 93.149 82.366 70.130
57 100.000 100.000 93.454 82.752 70.553
58 100.000 100.000 93.641 83.138 71.078
59 100.000 100.000 93.829 83.513 71.579
60 100.000 100.000 93.993 83.840 72.018
61 100.000 100.000 94.158 84.178 72.480
62 100.000 100.000 94.322 84.521 72.955
63 100.000 100.000 94.486 84.843 73.381
64 100.000 100.000 94.650 85.170 73.819
65 100.000 100.000 94.815 85.502 74.268
66 100.000 100.000 94.955 85.786 74.647
67 100.000 100.000 95.096 86.092 75.082
68 100.000 100.000 95.237 86.381 75.481
69 100.000 100.000 95.378 86.666 75.864
70 100.000 100.000 95.519 86.971 76.299
71 100.000 100.000 95.636 87.218 76.643
72 100.000 100.000 95.753 87.481 77.027
73 100.000 100.000 95.870 87.738 77.399
74 100.000 100.000 95.988 87.990 77.759
75 100.000 100.000 96.105 88.247 78.131
76 100.000 100.000 96.222 88.489 78.462
77 100.000 100.000 96.340 88.730 78.793
78 100.000 100.000 96.457 88.982 79.154
79 100.000 100.000 96.551 89.191 79.459
80 100.000 100.000 96.645 89.373 79.699
81 100.000 100.000 96.739 89.620 80.091
82 100.000 100.000 96.832 89.824 80.389
83 100.000 100.000 96.926 90.017 80.657
84 100.000 100.000 97.020 90.215 80.936
85 100.000 100.000 97.114 90.440 81.280
86 100.000 100.000 97.208 90.655 81.598
87 100.000 100.000 97.302 90.864 81.905
88 100.000 100.000 97.396 91.062 82.187
89 100.000 100.000 97.466 91.228 82.440
90 100.000 100.000 97.536 91.384 82.669
91 100.000 100.000 97.607 91.539 82.900
92 100.000 100.000 97.677 91.716 83.181
93 100.000 100.000 97.748 91.893 83.464
94 100.000 100.000 97.818 92.059 83.721
95 100.000 100.000 97.888 92.226 83.975
96 100.000 100.000 97.959 92.376 84.191
97 100.000 100.000 98.029 92.531 84.420
98 100.000 100.000 98.099 92.681 84.637
99 100.000 100.000 98.146 92.794 84.814
100 100.000 100.000 98.193 92.912 85.005
101 100.000 100.000 98.240 93.067 85.284
102 100.000 100.000 98.287 93.169 85.434
103 100.000 100.000 98.334 93.293 85.635
104 100.000 100.000 98.381 93.394 85.787
105 100.000 100.000 98.428 93.523 86.001
106 100.000 100.000 98.475 93.636 86.179
107 100.000 100.000 98.522 93.738 86.329
108 100.000 100.000 98.569 93.866 86.547
109 100.000 100.000 98.616 93.995 86.761
110 100.000 100.000 98.663 94.108 86.939
111 100.000 100.000 98.710 94.209 87.089
112 100.000 100.000 98.756 94.333 87.292
113 100.000 100.000 98.803 94.435 87.442
114 100.000 100.000 98.850 94.536 87.592
115 100.000 100.000 98.897 94.649 87.769
116 100.000 100.000 98.944 94.778 87.987
117 100.000 100.000 98.991 94.896 88.178
118 100.000 100.000 99.038 94.998 88.328
119 100.000 100.000 99.085 95.105 88.492
120 100.000 100.000 99.132 95.228 88.694
121 100.000 100.000 99.179 95.330 88.844
122 100.000 100.000 99.226 95.448 89.034
123 100.000 100.000 99.273 95.630 89.369
124 100.000 100.000 99.320 95.732 89.519
125 100.000 100.000 99.343 95.786 89.602
126 100.000 100.000 99.366 95.839 89.685
127 100.000 100.000 99.390 95.914 89.824
128 100.000 100.000 99.413 96.006 90.001
129 100.000 100.000 99.437 96.081 90.140
130 100.000 100.000 99.460 96.145 90.253
131 100.000 100.000 99.484 96.204 90.351
132 100.000 100.000 99.507 96.268 90.462
133 100.000 100.000 99.531 96.354 90.624
134 100.000 100.000 99.554 96.413 90.722
135 100.000 100.000 99.578 96.483 90.848
136 100.000 100.000 99.601 96.552 90.974
137 100.000 100.000 99.625 96.606 91.057
138 100.000 100.000 99.648 96.665 91.155
139 100.000 100.000 99.672 96.735 91.290
140 100.000 100.000 99.695 96.794 91.388
141 100.000 100.000 99.718 96.869 91.531
142 100.000 100.000 99.742 96.922 91.614
143 100.000 100.000 99.765 96.981 91.712
144 100.000 100.000 99.789 97.046 91.823
145 100.000 100.000 99.812 97.121 91.965
146 100.000 100.000 99.836 97.180 92.063
147 100.000 100.000 99.859 97.239 92.161
148 100.000 100.000 99.883 97.298 92.259
149 100.000 100.000 99.906 97.351 92.342
150 100.000 100.000 99.930 97.405 92.426
151 100.000 100.000 99.953 97.459 92.509
152 100.000 100.000 99.977 97.523 92.622
153 100.000 100.000 100.000 97.577 92.705
154 100.000 100.000 100.000 97.582 92.723
155 100.000 100.000 100.000 97.587 92.744
156 100.000 100.000 100.000 97.614 92.818
157 100.000 100.000 100.000 97.641 92.892
158 100.000 100.000 100.000 97.684 93.003
159 100.000 100.000 100.000 97.705 93.062
160 100.000 100.000 100.000 97.716 93.093
161 100.000 100.000 100.000 97.721 93.112
162 100.000 100.000 100.000 97.743 93.173
163 100.000 100.000 100.000 97.764 93.232
164 100.000 100.000 100.000 97.807 93.345
165 100.000 100.000 100.000 97.818 93.376
166 100.000 100.000 100.000 97.829 93.408
167 100.000 100.000 100.000 97.850 93.467
168 100.000 100.000 100.000 97.866 93.511
169 100.000 100.000 100.000 97.877 93.543
170 100.000 100.000 100.000 97.888 93.574
171 100.000 100.000 100.000 97.914 93.648
172 100.000 100.000 100.000 97.920 93.666
173 100.000 100.000 100.000 97.925 93.683
174 100.000 100.000 100.000 97.957 93.770
175 100.000 100.000 100.000 97.968 93.801
176 100.000 100.000 100.000 97.989 93.861
177 100.000 100.000 100.000 98.022 93.948
178 100.000 100.000 100.000 98.064 94.058
179 100.000 100.000 100.000 98.113 94.186
180 100.000 100.000 100.000 98.118 94.205
181 100.000 100.000 100.000 98.150 94.291
182 100.000 100.000 100.000 98.161 94.323
183 100.000 100.000 100.000 98.193 94.406
184 100.000 100.000 100.000 98.209 94.452
185 100.000 100.000 100.000 98.231 94.515
186 100.000 100.000 100.000 98.241 94.550
187 100.000 100.000 100.000 98.247 94.567
188 100.000 100.000 100.000 98.257 94.598
189 100.000 100.000 100.000 98.279 94.658
190 100.000 100.000 100.000 98.295 94.704
191 100.000 100.000 100.000 98.300 94.722
192 100.000 100.000 100.000 98.311 94.754
193 100.000 100.000 100.000 98.327 94.800
194 100.000 100.000 100.000 98.343 94.848
195 100.000 100.000 100.000 98.343 94.855
196 100.000 100.000 100.000 98.365 94.915
197 100.000 100.000 100.000 98.413 95.042
198 100.000 100.000 100.000 98.434 95.101
199 100.000 100.000 100.000 98.440 95.118
200 100.000 100.000 100.000 98.445 95.137
201 100.000 100.000 100.000 98.450 95.153
202 100.000 100.000 100.000 98.456 95.172
203 100.000 100.000 100.000 98.467 95.207
204 100.000 100.000 100.000 98.488 95.266
205 100.000 100.000 100.000 98.520 95.353
206 100.000 100.000 100.000 98.526 95.371
207 100.000 100.000 100.000 98.531 95.388
208 100.000 100.000 100.000 98.536 95.407
209 100.000 100.000 100.000 98.552 95.453
210 100.000 100.000 100.000 98.563 95.488
211 100.000 100.000 100.000 98.574 95.519
212 100.000 100.000 100.000 98.585 95.554
213 100.000 100.000 100.000 98.590 95.571
214 100.000 100.000 100.000 98.606 95.616
215 100.000 100.000 100.000 98.611 95.632
216 100.000 100.000 100.000 98.638 95.704
217 100.000 100.000 100.000 98.660 95.767
218 100.000 100.000 100.000 98.670 95.799
219 100.000 100.000 100.000 98.676 95.815
220 100.000 100.000 100.000 98.676 95.823
221 100.000 100.000 100.000 98.697 95.885
222 100.000 100.000 100.000 98.708 95.919
223 100.000 100.000 100.000 98.724 95.963
224 100.000 100.000 100.000 98.729 95.980
225 100.000 100.000 100.000 98.740 96.011
226 100.000 100.000 100.000 98.740 96.013
227 100.000 100.000 100.000 98.745 96.032
228 100.000 100.000 100.000 98.745 96.033
229 100.000 100.000 100.000 98.756 96.065
230 100.000 100.000 100.000 98.761 96.082
231 100.000 100.000 100.000 98.767 96.102
232 100.000 100.000 100.000 98.778 96.137
233 100.000 100.000 100.000 98.788 96.172
234 100.000 100.000 100.000 98.788 96.176
235 100.000 100.000 100.000 98.799 96.207
236 100.000 100.000 100.000 98.810 96.239
237 100.000 100.000 100.000 98.815 96.255
238 100.000 100.000 100.000 98.820 96.272
239 100.000 100.000 100.000 98.826 96.289
240 100.000 100.000 100.000 98.831 96.305
241 100.000 100.000 100.000 98.869 96.401
242 100.000 100.000 100.000 98.879 96.433
243 100.000 100.000 100.000 98.944 96.596
244 100.000 100.000 100.000 98.954 96.629
245 100.000 100.000 100.000 98.965 96.660
246 100.000 100.000 100.000 98.981 96.707
247 100.000 100.000 100.000 98.987 96.723
248 100.000 100.000 100.000 98.997 96.758
249 100.000 100.000 100.000 99.003 96.775
250 100.000 100.000 100.000 99.013 96.808
251 100.000 100.000 100.000 99.024 96.843
252 100.000 100.000 100.000 99.024 96.845
253 100.000 100.000 100.000 99.035 96.877
254 100.000 100.000 100.000 99.040 96.893
255 100.000 100.000 100.000 99.046 96.910
256 100.000 100.000 100.000 99.051 96.927
257 100.000 100.000 100.000 99.072 96.986
258 100.000 100.000 100.000 99.083 97.017
259 100.000 100.000 100.000 99.083 97.019
260 100.000 100.000 100.000 99.099 97.069
261 100.000 100.000 100.000 99.105 97.086
262 100.000 100.000 100.000 99.126 97.149
263 100.000 100.000 100.000 99.137 97.180
264 100.000 100.000 100.000 99.147 97.211
265 100.000 100.000 100.000 99.153 97.228
266 100.000 100.000 100.000 99.153 97.232
267 100.000 100.000 100.000 99.158 97.248
268 100.000 100.000 100.000 99.164 97.265
269 100.000 100.000 100.000 99.174 97.296
270 100.000 100.000 100.000 99.180 97.313
271 100.000 100.000 100.000 99.180 97.315
272 100.000 100.000 100.000 99.190 97.346
273 100.000 100.000 100.000 99.196 97.365
274 100.000 100.000 100.000 99.196 97.367
275 100.000 100.000 100.000 99.217 97.426
276 100.000 100.000 100.000 99.228 97.457
277 100.000 100.000 100.000 99.233 97.476
278 100.000 100.000 100.000 99.255 97.535
279 100.000 100.000 100.000 99.260 97.552
280 100.000 100.000 100.000 99.271 97.583
281 100.000 100.000 100.000 99.276 97.600
282 100.000 100.000 100.000 99.287 97.631
283 100.000 100.000 100.000 99.292 97.653
284 100.000 100.000 100.000 99.357 97.816
285 100.000 100.000 100.000 99.378 97.875
286 100.000 100.000 100.000 99.378 97.879
287 100.000 100.000 100.000 99.389 97.910
288 100.000 100.000 100.000 99.399 97.942
289 100.000 100.000 100.000 99.416 97.986
290 100.000 100.000 100.000 99.421 98.003
291 100.000 100.000 100.000 99.421 98.005
292 100.000 100.000 100.000 99.421 98.007
293 100.000 100.000 100.000 99.426 98.023
294 100.000 100.000 100.000 99.437 98.062
295 100.000 100.000 100.000 99.448 98.093
296 100.000 100.000 100.000 99.448 98.101
297 100.000 100.000 100.000 99.453 98.117
298 100.000 100.000 100.000 99.453 98.119
299 100.000 100.000 100.000 99.464 98.151
300 100.000 100.000 100.000 99.464 98.154
301 100.000 100.000 100.000 99.469 98.171
302 100.000 100.000 100.000 99.480 98.203
303 100.000 100.000 100.000 99.501 98.265
304 100.000 100.000 100.000 99.507 98.282
305 100.000 100.000 100.000 99.507 98.286
306 100.000 100.000 100.000 99.512 98.302
307 100.000 100.000 100.000 99.517 98.319
308 100.000 100.000 100.000 99.517 98.321
309 100.000 100.000 100.000 99.528 98.352
310 100.000 100.000 100.000 99.539 98.384
311 100.000 100.000 100.000 99.539 98.387
312 100.000 100.000 100.000 99.544 98.408
313 100.000 100.000 100.000 99.566 98.467
314 100.000 100.000 100.000 99.576 98.498
315 100.000 100.000 100.000 99.587 98.530
316 100.000 100.000 100.000 99.593 98.547
317 100.000 100.000 100.000 99.603 98.578
318 100.000 100.000 100.000 99.603 98.582
319 100.000 100.000 100.000 99.609 98.598
320 100.000 100.000 100.000 99.614 98.615
321 100.000 100.000 100.000 99.614 98.619
322 100.000 100.000 100.000 99.619 98.635
323 100.000 100.000 100.000 99.625 98.652
324 100.000 100.000 100.000 99.625 98.656
325 100.000 100.000 100.000 99.635 98.687
326 100.000 100.000 100.000 99.641 98.704
327 100.000 100.000 100.000 99.646 98.720
328 100.000 100.000 100.000 99.646 98.724
329 100.000 100.000 100.000 99.646 98.726
330 100.000 100.000 100.000 99.657 98.757
331 100.000 100.000 100.000 99.662 98.778
332 100.000 100.000 100.000 99.668 98.794
333 100.000 100.000 100.000 99.673 98.811
334 100.000 100.000 100.000 99.673 98.813
335 100.000 100.000 100.000 99.678 98.829
336 100.000 100.000 100.000 99.689 98.861
337 100.000 100.000 100.000 99.694 98.878
338 100.000 100.000 100.000 99.700 98.894
339 100.000 100.000 100.000 99.705 98.911
340 100.000 100.000 100.000 99.710 98.927
341 100.000 100.000 100.000 99.710 98.929
342 100.000 100.000 100.000 99.710 98.933
343 100.000 100.000 100.000 99.721 98.968
344 100.000 100.000 100.000 99.732 99.000
345 100.000 100.000 100.000 99.732 99.003
346 100.000 100.000 100.000 99.732 99.005
347 100.000 100.000 100.000 99.748 99.051
348 100.000 100.000 100.000 99.759 99.083
349 100.000 100.000 100.000 99.764 99.099
350 100.000 100.000 100.000 99.764 99.101
351 100.000 100.000 100.000 99.775 99.133
352 100.000 100.000 100.000 99.775 99.140
353 100.000 100.000 100.000 99.775 99.144
354 100.000 100.000 100.000 99.775 99.148
355 100.000 100.000 100.000 99.780 99.164
356 100.000 100.000 100.000 99.786 99.181
357 100.000 100.000 100.000 99.786 99.184
358 100.000 100.000 100.000 99.786 99.188
359 100.000 100.000 100.000 99.786 99.190
360 100.000 100.000 100.000 99.786 99.194
361 100.000 100.000 100.000 99.791 99.210
362 100.000 100.000 100.000 99.791 99.212
363 100.000 100.000 100.000 99.802 99.244
364 100.000 100.000 100.000 99.807 99.260
365 100.000 100.000 100.000 99.812 99.277
366 100.000 100.000 100.000 99.818 99.295
367 100.000 100.000 100.000 99.828 99.327
368 100.000 100.000 100.000 99.828 99.329
369 100.000 100.000 100.000 99.834 99.345
370 100.000 100.000 100.000 99.834 99.347
371 100.000 100.000 100.000 99.834 99.351
372 100.000 100.000 100.000 99.834 99.353
373 100.000 100.000 100.000 99.850 99.399
374 100.000 100.000 100.000 99.850 99.403
375 100.000 100.000 100.000 99.850 99.406
376 100.000 100.000 100.000 99.850 99.412
377 100.000 100.000 100.000 99.855 99.429
378 100.000 100.000 100.000 99.861 99.445
379 100.000 100.000 100.000 99.866 99.462
380 100.000 100.000 100.000 99.871 99.479
381 100.000 100.000 100.000 99.871 99.482
382 100.000 100.000 100.000 99.882 99.514
383 100.000 100.000 100.000 99.882 99.517
384 100.000 100.000 100.000 99.882 99.525
385 100.000 100.000 100.000 99.893 99.556
386 100.000 100.000 100.000 99.893 99.558
387 100.000 100.000 100.000 99.903 99.593
388 100.000 100.000 100.000 99.903 99.597
389 100.000 100.000 100.000 99.903 99.601
390 100.000 100.000 100.000 99.903 99.604
391 100.000 100.000 100.000 99.909 99.621
392 100.000 100.000 100.000 99.909 99.623
393 100.000 100.000 100.000 99.909 99.625
394 100.000 100.000 100.000 99.914 99.641
395 100.000 100.000 100.000 99.925 99.673
396 100.000 100.000 100.000 99.925 99.676
397 100.000 100.000 100.000 99.930 99.693
398 100.000 100.000 100.000 99.941 99.724
399 100.000 100.000 100.000 99.946 99.741
400 100.000 100.000 100.000 99.946 99.745
401 100.000 100.000 100.000 99.946 99.749
402 100.000 100.000 100.000 99.952 99.765
403 100.000 100.000 100.000 99.952 99.769
404 100.000 100.000 100.000 99.952 99.771
405 100.000 100.000 100.000 99.952 99.782
406 100.000 100.000 100.000 99.952 99.784
407 100.000 100.000 100.000 99.952 99.787
408 100.000 100.000 100.000 99.952 99.791
409 100.000 100.000 100.000 99.952 99.795
410 100.000 100.000 100.000 99.952 99.798
411 100.000 100.000 100.000 99.957 99.815
412 100.000 100.000 100.000 99.962 99.832
413 100.000 100.000 100.000 99.962 99.839
414 100.000 100.000 100.000 99.962 99.843
415 100.000 100.000 100.000 99.968 99.859
416 100.000 100.000 100.000 99.973 99.876
417 100.000 100.000 100.000 99.973 99.880
418 100.000 100.000 100.000 99.979 99.896
419 100.000 100.000 100.000 99.979 99.900
420 100.000 100.000 100.000 99.979 99.904
421 100.000 100.000 100.000 99.979 99.906
422 100.000 100.000 100.000 99.979 99.908
423 100.000 100.000 100.000 99.979 99.909
424 100.000 100.000 100.000 99.979 99.911
425 100.000 100.000 100.000 99.984 99.928
426 100.000 100.000 100.000 99.984 99.932
427 100.000 100.000 100.000 99.995 99.963
428 100.000 100.000 100.000 99.995 99.967
429 100.000 100.000 100.000 99.995 99.970
430 100.000 100.000 100.000 99.995 99.974
431 100.000 100.000 100.000 99.995 99.976
432 100.000 100.000 100.000 100.000 99.993
433 100.000 100.000 100.000 100.000 99.994
434 100.000 100.000 100.000 100.000 99.996
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 57.143 31.641 18.254 11.072 7.053
4 64.286 38.579 23.322 14.503 9.348
5 73.810 46.024 28.414 17.903 11.626
6 83.333 53.807 33.740 21.409 13.941
7 88.095 59.222 38.081 24.546 16.134
8 90.476 63.113 41.764 27.457 18.267
9 95.238 68.697 46.222 30.647 20.478
10 100.000 73.435 50.000 33.457 22.514
11 100.000 75.465 52.557 35.784 24.384
12 100.000 76.988 54.716 37.896 26.161
13 100.000 79.019 57.227 40.164 27.984
14 100.000 80.372 59.338 42.282 29.767
15 100.000 82.234 61.614 44.368 31.487
16 100.000 83.249 63.210 46.078 33.038
17 100.000 84.772 65.204 47.992 34.667
18 100.000 86.125 67.058 49.810 36.230
19 100.000 87.310 68.700 51.466 37.702
20 100.000 88.325 70.155 52.994 39.107
21 100.000 89.171 71.422 54.356 40.383
22 100.000 89.679 72.407 55.568 41.602
23 100.000 90.355 73.534 56.849 42.841
24 100.000 91.201 74.847 58.238 44.098
25 100.000 91.878 75.950 59.477 45.289
26 100.000 92.555 77.053 60.715 46.480
27 100.000 92.893 77.851 61.771 47.590
28 100.000 93.401 78.719 62.822 48.664
29 100.000 93.909 79.634 63.911 49.751
30 100.000 94.416 80.526 64.967 50.811
31 100.000 94.755 81.276 65.937 51.822
32 100.000 95.262 82.145 66.956 52.836
33 100.000 95.431 82.708 67.798 53.784
34 100.000 95.601 83.294 68.656 54.724
35 100.000 95.939 83.975 69.535 55.665
36 100.000 96.108 84.514 70.334 56.566
37 100.000 96.616 85.312 71.256 57.501
38 100.000 96.954 85.946 72.050 58.339
39 100.000 97.293 86.579 72.838 59.166
40 100.000 97.462 87.001 73.471 59.911
41 100.000 97.631 87.447 74.125 60.669
42 100.000 97.800 87.893 74.768 61.403
43 100.000 97.970 88.339 75.412 62.137
44 100.000 97.970 88.620 75.958 62.840
45 100.000 98.139 89.066 76.602 63.574
46 100.000 98.308 89.465 77.192 64.277
47 100.000 98.308 89.747 77.722 64.946
48 100.000 98.477 90.145 78.291 65.599
49 100.000 98.646 90.544 78.854 66.244
50 100.000 98.816 90.896 79.358 66.845
51 100.000 98.816 91.107 79.781 67.409
52 100.000 99.154 91.600 80.350 68.012
53 100.000 99.323 91.976 80.864 68.593
54 100.000 99.323 92.187 81.283 69.138
55 100.000 99.323 92.374 81.669 69.664
56 100.000 99.323 92.562 82.055 70.189
57 100.000 99.492 92.914 82.532 70.731
58 100.000 99.492 93.102 82.907 71.232
59 100.000 99.492 93.290 83.272 71.709
60 100.000 99.492 93.454 83.615 72.182
61 100.000 99.662 93.782 84.054 72.678
62 100.000 99.662 93.947 84.392 73.140
63 100.000 99.831 94.252 84.800 73.615
64 100.000 99.831 94.416 85.132 74.067
65 100.000 99.831 94.580 85.465 74.518
66 100.000 99.831 94.721 85.770 74.954
67 100.000 99.831 94.838 86.054 75.385
68 100.000 99.831 94.979 86.355 75.809
69 100.000 99.831 95.120 86.650 76.221
70 100.000 99.831 95.260 86.944 76.631
71 100.000 99.831 95.401 87.239 77.042
72 100.000 99.831 95.542 87.534 77.456
73 100.000 99.831 95.683 87.824 77.856
74 100.000 99.831 95.800 88.092 78.251
75 100.000 99.831 95.941 88.376 78.638
76 100.000 99.831 96.082 88.660 79.017
77 100.000 99.831 96.222 88.939 79.389
78 100.000 99.831 96.363 89.212 79.747
79 100.000 99.831 96.481 89.459 80.095
80 100.000 99.831 96.574 89.684 80.439
81 100.000 99.831 96.692 89.931 80.787
82 100.000 99.831 96.786 90.151 81.118
83 100.000 99.831 96.879 90.360 81.426
84 100.000 99.831 96.973 90.569 81.735
85 100.000 99.831 97.020 90.735 82.037
86 100.000 99.831 97.067 90.901 82.334
87 100.000 99.831 97.137 91.084 82.630
88 100.000 99.831 97.184 91.244 82.921
89 100.000 100.000 97.443 91.539 83.226
90 100.000 100.000 97.489 91.695 83.503
91 100.000 100.000 97.536 91.850 83.781
92 100.000 100.000 97.630 92.049 84.060
93 100.000 100.000 97.724 92.242 84.328
94 100.000 100.000 97.818 92.435 84.596
95 100.000 100.000 97.888 92.606 84.862
96 100.000 100.000 97.959 92.773 85.119
97 100.000 100.000 98.029 92.939 85.376
98 100.000 100.000 98.099 93.100 85.621
99 100.000 100.000 98.170 93.260 85.866
100 100.000 100.000 98.217 93.400 86.109
101 100.000 100.000 98.287 93.561 86.355
102 100.000 100.000 98.334 93.695 86.584
103 100.000 100.000 98.405 93.845 86.800
104 100.000 100.000 98.475 93.995 87.017
105 100.000 100.000 98.522 94.124 87.231
106 100.000 100.000 98.569 94.247 87.436
107 100.000 100.000 98.616 94.370 87.642
108 100.000 100.000 98.663 94.494 87.849
109 100.000 100.000 98.710 94.612 88.039
110 100.000 100.000 98.756 94.730 88.230
111 100.000 100.000 98.803 94.842 88.411
112 100.000 100.000 98.850 94.955 88.588
113 100.000 100.000 98.897 95.067 88.770
114 100.000 100.000 98.944 95.180 88.949
115 100.000 100.000 98.944 95.244 89.104
116 100.000 100.000 98.991 95.351 89.269
117 100.000 100.000 99.038 95.459 89.434
118 100.000 100.000 99.061 95.544 89.600
119 100.000 100.000 99.061 95.609 89.763
120 100.000 100.000 99.108 95.716 89.931
121 100.000 100.000 99.155 95.818 90.081
122 100.000 100.000 99.179 95.898 90.234
123 100.000 100.000 99.226 96.000 90.384
124 100.000 100.000 99.249 96.081 90.538
125 100.000 100.000 99.296 96.183 90.687
126 100.000 100.000 99.343 96.284 90.837
127 100.000 100.000 99.390 96.386 90.987
128 100.000 100.000 99.413 96.461 91.126
129 100.000 100.000 99.437 96.536 91.266
130 100.000 100.000 99.437 96.590 91.405
131 100.000 100.000 99.460 96.665 91.545
132 100.000 100.000 99.484 96.735 91.671
133 100.000 100.000 99.507 96.804 91.797
134 100.000 100.000 99.531 96.869 91.908
135 100.000 100.000 99.531 96.912 92.022
136 100.000 100.000 99.554 96.976 92.135
137 100.000 100.000 99.554 97.019 92.246
138 100.000 100.000 99.578 97.083 92.357
139 100.000 100.000 99.601 97.148 92.468
140 100.000 100.000 99.625 97.212 92.581
141 100.000 100.000 99.648 97.276 92.696
142 100.000 100.000 99.672 97.341 92.807
143 100.000 100.000 99.695 97.405 92.918
144 100.000 100.000 99.718 97.469 93.028
145 100.000 100.000 99.742 97.528 93.126
146 100.000 100.000 99.765 97.587 93.224
147 100.000 100.000 99.789 97.646 93.322
148 100.000 100.000 99.812 97.705 93.422
149 100.000 100.000 99.812 97.743 93.522
150 100.000 100.000 99.836 97.802 93.620
151 100.000 100.000 99.836 97.839 93.720
152 100.000 100.000 99.859 97.898 93.818
153 100.000 100.000 99.883 97.957 93.918
154 100.000 100.000 99.883 97.995 94.014
155 100.000 100.000 99.906 98.054 94.112
156 100.000 100.000 99.906 98.086 94.199
157 100.000 100.000 99.906 98.118 94.286
158 100.000 100.000 99.930 98.172 94.369
159 100.000 100.000 99.953 98.225 94.452
160 100.000 100.000 99.953 98.257 94.539
161 100.000 100.000 99.953 98.290 94.626
162 100.000 100.000 99.977 98.343 94.709
163 100.000 100.000 99.977 98.375 94.796
164 100.000 100.000 100.000 98.429 94.880
165 100.000 100.000 100.000 98.461 94.966
166 100.000 100.000 100.000 98.493 95.053
167 100.000 100.000 100.000 98.526 95.140
168 100.000 100.000 100.000 98.552 95.216
169 100.000 100.000 100.000 98.579 95.288
170 100.000 100.000 100.000 98.606 95.368
171 100.000 100.000 100.000 98.633 95.440
172 100.000 100.000 100.000 98.660 95.512
173 100.000 100.000 100.000 98.686 95.588
174 100.000 100.000 100.000 98.713 95.664
175 100.000 100.000 100.000 98.735 95.723
176 100.000 100.000 100.000 98.756 95.793
177 100.000 100.000 100.000 98.778 95.860
178 100.000 100.000 100.000 98.799 95.922
179 100.000 100.000 100.000 98.820 95.984
180 100.000 100.000 100.000 98.842 96.045
181 100.000 100.000 100.000 98.863 96.104
182 100.000 100.000 100.000 98.885 96.167
183 100.000 100.000 100.000 98.906 96.226
184 100.000 100.000 100.000 98.928 96.289
185 100.000 100.000 100.000 98.949 96.348
186 100.000 100.000 100.000 98.971 96.411
187 100.000 100.000 100.000 98.992 96.474
188 100.000 100.000 100.000 99.013 96.533
189 100.000 100.000 100.000 99.035 96.592
190 100.000 100.000 100.000 99.056 96.655
191 100.000 100.000 100.000 99.078 96.714
192 100.000 100.000 100.000 99.099 96.777
193 100.000 100.000 100.000 99.121 96.836
194 100.000 100.000 100.000 99.142 96.895
195 100.000 100.000 100.000 99.164 96.958
196 100.000 100.000 100.000 99.185 97.021
197 100.000 100.000 100.000 99.201 97.067
198 100.000 100.000 100.000 99.217 97.113
199 100.000 100.000 100.000 99.233 97.163
200 100.000 100.000 100.000 99.249 97.213
201 100.000 100.000 100.000 99.265 97.259
202 100.000 100.000 100.000 99.282 97.306
203 100.000 100.000 100.000 99.298 97.354
204 100.000 100.000 100.000 99.314 97.400
205 100.000 100.000 100.000 99.330 97.444
206 100.000 100.000 100.000 99.346 97.491
207 100.000 100.000 100.000 99.362 97.537
208 100.000 100.000 100.000 99.373 97.570
209 100.000 100.000 100.000 99.383 97.603
210 100.000 100.000 100.000 99.394 97.635
211 100.000 100.000 100.000 99.405 97.666
212 100.000 100.000 100.000 99.416 97.698
213 100.000 100.000 100.000 99.426 97.729
214 100.000 100.000 100.000 99.437 97.762
215 100.000 100.000 100.000 99.448 97.794
216 100.000 100.000 100.000 99.458 97.825
217 100.000 100.000 100.000 99.469 97.859
218 100.000 100.000 100.000 99.480 97.890
219 100.000 100.000 100.000 99.491 97.925
220 100.000 100.000 100.000 99.501 97.958
221 100.000 100.000 100.000 99.512 97.994
222 100.000 100.000 100.000 99.523 98.025
223 100.000 100.000 100.000 99.534 98.056
224 100.000 100.000 100.000 99.544 98.090
225 100.000 100.000 100.000 99.555 98.125
226 100.000 100.000 100.000 99.566 98.156
227 100.000 100.000 100.000 99.576 98.188
228 100.000 100.000 100.000 99.587 98.223
229 100.000 100.000 100.000 99.598 98.254
230 100.000 100.000 100.000 99.609 98.288
231 100.000 100.000 100.000 99.619 98.325
232 100.000 100.000 100.000 99.630 98.362
233 100.000 100.000 100.000 99.641 98.393
234 100.000 100.000 100.000 99.651 98.424
235 100.000 100.000 100.000 99.662 98.458
236 100.000 100.000 100.000 99.673 98.489
237 100.000 100.000 100.000 99.684 98.521
238 100.000 100.000 100.000 99.694 98.552
239 100.000 100.000 100.000 99.705 98.584
240 100.000 100.000 100.000 99.716 98.615
241 100.000 100.000 100.000 99.721 98.632
242 100.000 100.000 100.000 99.727 98.648
243 100.000 100.000 100.000 99.732 98.665
244 100.000 100.000 100.000 99.737 98.683
245 100.000 100.000 100.000 99.743 98.700
246 100.000 100.000 100.000 99.748 98.717
247 100.000 100.000 100.000 99.753 98.733
248 100.000 100.000 100.000 99.759 98.750
249 100.000 100.000 100.000 99.764 98.767
250 100.000 100.000 100.000 99.769 98.783
251 100.000 100.000 100.000 99.775 98.802
252 100.000 100.000 100.000 99.780 98.818
253 100.000 100.000 100.000 99.786 98.835
254 100.000 100.000 100.000 99.791 98.852
255 100.000 100.000 100.000 99.796 98.868
256 100.000 100.000 100.000 99.802 98.885
257 100.000 100.000 100.000 99.807 98.902
258 100.000 100.000 100.000 99.812 98.918
259 100.000 100.000 100.000 99.818 98.937
260 100.000 100.000 100.000 99.823 98.953
261 100.000 100.000 100.000 99.828 98.970
262 100.000 100.000 100.000 99.834 98.987
263 100.000 100.000 100.000 99.839 99.003
264 100.000 100.000 100.000 99.845 99.020
265 100.000 100.000 100.000 99.850 99.037
266 100.000 100.000 100.000 99.855 99.053
267 100.000 100.000 100.000 99.861 99.070
268 100.000 100.000 100.000 99.866 99.088
269 100.000 100.000 100.000 99.871 99.107
270 100.000 100.000 100.000 99.877 99.123
271 100.000 100.000 100.000 99.882 99.144
272 100.000 100.000 100.000 99.887 99.160
273 100.000 100.000 100.000 99.893 99.177
274 100.000 100.000 100.000 99.898 99.196
275 100.000 100.000 100.000 99.903 99.216
276 100.000 100.000 100.000 99.909 99.233
277 100.000 100.000 100.000 99.914 99.249
278 100.000 100.000 100.000 99.920 99.266
279 100.000 100.000 100.000 99.925 99.283
280 100.000 100.000 100.000 99.930 99.299
281 100.000 100.000 100.000 99.936 99.316
282 100.000 100.000 100.000 99.941 99.332
283 100.000 100.000 100.000 99.946 99.351
284 100.000 100.000 100.000 99.952 99.368
285 100.000 100.000 100.000 99.957 99.384
286 100.000 100.000 100.000 99.962 99.405
287 100.000 100.000 100.000 99.968 99.421
288 100.000 100.000 100.000 99.973 99.438
289 100.000 100.000 100.000 99.979 99.454
290 100.000 100.000 100.000 99.984 99.471
291 100.000 100.000 100.000 99.989 99.488
292 100.000 100.000 100.000 99.995 99.504
293 100.000 100.000 100.000 100.000 99.521
294 100.000 100.000 100.000 100.000 99.525
295 100.000 100.000 100.000 100.000 99.532
296 100.000 100.000 100.000 100.000 99.536
297 100.000 100.000 100.000 100.000 99.538
298 100.000 100.000 100.000 100.000 99.540
299 100.000 100.000 100.000 100.000 99.541
300 100.000 100.000 100.000 100.000 99.543
301 100.000 100.000 100.000 100.000 99.545
302 100.000 100.000 100.000 100.000 99.547
303 100.000 100.000 100.000 100.000 99.549
304 100.000 100.000 100.000 100.000 99.551
305 100.000 100.000 100.000 100.000 99.554
306 100.000 100.000 100.000 100.000 99.562
307 100.000 100.000 100.000 100.000 99.564
308 100.000 100.000 100.000 100.000 99.567
309 100.000 100.000 100.000 100.000 99.571
310 100.000 100.000 100.000 100.000 99.575
311 100.000 100.000 100.000 100.000 99.578
312 100.000 100.000 100.000 100.000 99.582
313 100.000 100.000 100.000 100.000 99.586
314 100.000 100.000 100.000 100.000 99.589
315 100.000 100.000 100.000 100.000 99.593
316 100.000 100.000 100.000 100.000 99.597
317 100.000 100.000 100.000 100.000 99.601
318 100.000 100.000 100.000 100.000 99.602
319 100.000 100.000 100.000 100.000 99.606
320 100.000 100.000 100.000 100.000 99.610
321 100.000 100.000 100.000 100.000 99.612
322 100.000 100.000 100.000 100.000 99.615
323 100.000 100.000 100.000 100.000 99.623
324 100.000 100.000 100.000 100.000 99.625
325 100.000 100.000 100.000 100.000 99.626
326 100.000 100.000 100.000 100.000 99.630
327 100.000 100.000 100.000 100.000 99.632
328 100.000 100.000 100.000 100.000 99.634
329 100.000 100.000 100.000 100.000 99.641
330 100.000 100.000 100.000 100.000 99.643
331 100.000 100.000 100.000 100.000 99.647
332 100.000 100.000 100.000 100.000 99.649
333 100.000 100.000 100.000 100.000 99.652
334 100.000 100.000 100.000 100.000 99.654
335 100.000 100.000 100.000 100.000 99.658
336 100.000 100.000 100.000 100.000 99.660
337 100.000 100.000 100.000 100.000 99.662
338 100.000 100.000 100.000 100.000 99.665
339 100.000 100.000 100.000 100.000 99.671
340 100.000 100.000 100.000 100.000 99.678
341 100.000 100.000 100.000 100.000 99.680
342 100.000 100.000 100.000 100.000 99.682
343 100.000 100.000 100.000 100.000 99.686
344 100.000 100.000 100.000 100.000 99.687
345 100.000 100.000 100.000 100.000 99.691
346 100.000 100.000 100.000 100.000 99.695
347 100.000 100.000 100.000 100.000 99.699
348 100.000 100.000 100.000 100.000 99.700
349 100.000 100.000 100.000 100.000 99.704
350 100.000 100.000 100.000 100.000 99.708
351 100.000 100.000 100.000 100.000 99.712
352 100.000 100.000 100.000 100.000 99.713
353 100.000 100.000 100.000 100.000 99.715
354 100.000 100.000 100.000 100.000 99.723
355 100.000 100.000 100.000 100.000 99.724
356 100.000 100.000 100.000 100.000 99.726
357 100.000 100.000 100.000 100.000 99.728
358 100.000 100.000 100.000 100.000 99.730
359 100.000 100.000 100.000 100.000 99.734
360 100.000 100.000 100.000 100.000 99.737
361 100.000 100.000 100.000 100.000 99.745
362 100.000 100.000 100.000 100.000 99.749
363 100.000 100.000 100.000 100.000 99.752
364 100.000 100.000 100.000 100.000 99.754
365 100.000 100.000 100.000 100.000 99.756
366 100.000 100.000 100.000 100.000 99.760
367 100.000 100.000 100.000 100.000 99.761
368 100.000 100.000 100.000 100.000 99.776
369 100.000 100.000 100.000 100.000 99.778
370 100.000 100.000 100.000 100.000 99.782
371 100.000 100.000 100.000 100.000 99.789
372 100.000 100.000 100.000 100.000 99.793
373 100.000 100.000 100.000 100.000 99.797
374 100.000 100.000 100.000 100.000 99.800
375 100.000 100.000 100.000 100.000 99.802
376 100.000 100.000 100.000 100.000 99.806
377 100.000 100.000 100.000 100.000 99.810
378 100.000 100.000 100.000 100.000 99.813
379 100.000 100.000 100.000 100.000 99.817
380 100.000 100.000 100.000 100.000 99.821
381 100.000 100.000 100.000 100.000 99.822
382 100.000 100.000 100.000 100.000 99.826
383 100.000 100.000 100.000 100.000 99.830
384 100.000 100.000 100.000 100.000 99.835
385 100.000 100.000 100.000 100.000 99.837
386 100.000 100.000 100.000 100.000 99.841
387 100.000 100.000 100.000 100.000 99.843
388 100.000 100.000 100.000 100.000 99.847
389 100.000 100.000 100.000 100.000 99.850
390 100.000 100.000 100.000 100.000 99.852
391 100.000 100.000 100.000 100.000 99.854
392 100.000 100.000 100.000 100.000 99.858
393 100.000 100.000 100.000 100.000 99.861
394 100.000 100.000 100.000 100.000 99.863
395 100.000 100.000 100.000 100.000 99.865
396 100.000 100.000 100.000 100.000 99.869
397 100.000 100.000 100.000 100.000 99.872
398 100.000 100.000 100.000 100.000 99.874
399 100.000 100.000 100.000 100.000 99.876
400 100.000 100.000 100.000 100.000 99.880
401 100.000 100.000 100.000 100.000 99.883
402 100.000 100.000 100.000 100.000 99.887
403 100.000 100.000 100.000 100.000 99.889
404 100.000 100.000 100.000 100.000 99.893
405 100.000 100.000 100.000 100.000 99.895
406 100.000 100.000 100.000 100.000 99.896
407 100.000 100.000 100.000 100.000 99.900
408 100.000 100.000 100.000 100.000 99.908
409 100.000 100.000 100.000 100.000 99.911
410 100.000 100.000 100.000 100.000 99.913
411 100.000 100.000 100.000 100.000 99.917
412 100.000 100.000 100.000 100.000 99.920
413 100.000 100.000 100.000 100.000 99.922
414 100.000 100.000 100.000 100.000 99.924
415 100.000 100.000 100.000 100.000 99.928
416 100.000 100.000 100.000 100.000 99.930
417 100.000 100.000 100.000 100.000 99.932
418 100.000 100.000 100.000 100.000 99.935
419 100.000 100.000 100.000 100.000 99.937
420 100.000 100.000 100.000 100.000 99.941
421 100.000 100.000 100.000 100.000 99.945
422 100.000 100.000 100.000 100.000 99.946
423 100.000 100.000 100.000 100.000 99.948
424 100.000 100.000 100.000 100.000 99.952
425 100.000 100.000 100.000 100.000 99.956
426 100.000 100.000 100.000 100.000 99.967
427 100.000 100.000 100.000 100.000 99.970
428 100.000 100.000 100.000 100.000 99.972
429 100.000 100.000 100.000 100.000 99.976
430 100.000 100.000 100.000 100.000 99.982
431 100.000 100.000 100.000 100.000 99.983
432 100.000 100.000 100.000 100.000 99.987
433 100.000 100.000 100.000 100.000 99.991
434 100.000 100.000 100.000 100.000 99.994
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 42.857 21.658 12.107 7.292 4.656
3 54.762 30.288 17.644 10.831 6.970
4 61.905 36.717 22.337 14.074 9.191
5 69.048 42.978 26.889 17.238 11.375
6 73.810 48.562 31.347 20.428 13.586
7 80.952 54.992 36.039 23.672 15.807
8 80.952 57.360 38.878 26.165 17.760
9 80.952 59.729 41.671 28.610 19.683
10 83.333 62.944 44.721 31.124 21.617
11 85.714 66.497 48.076 33.816 23.629
12 90.476 71.066 51.689 36.518 25.610
13 92.857 73.773 54.247 38.711 27.376
14 92.857 75.635 56.546 40.813 29.097
15 95.238 78.173 58.916 42.866 30.780
16 95.238 79.357 60.605 44.587 32.308
17 95.238 80.880 62.529 46.405 33.854
18 100.000 85.110 65.791 48.845 35.666
19 100.000 85.787 67.034 50.276 37.025
20 100.000 86.802 68.536 51.842 38.440
21 100.000 87.648 69.873 53.295 39.790
22 100.000 88.663 71.305 54.775 41.139
23 100.000 89.679 72.877 56.421 42.606
24 100.000 90.355 73.956 57.648 43.808
25 100.000 91.371 75.411 59.144 45.156
26 100.000 91.878 76.396 60.335 46.338
27 100.000 92.386 77.311 61.434 47.449
28 100.000 93.232 78.555 62.753 48.677
29 100.000 93.739 79.423 63.787 49.729
30 100.000 93.909 80.033 64.683 50.715
31 100.000 94.247 80.760 65.621 51.699
32 100.000 94.247 81.206 66.399 52.605
33 100.000 94.416 81.793 67.251 53.538
34 100.000 94.585 82.309 68.012 54.400
35 100.000 95.431 83.482 69.213 55.499
36 100.000 95.939 84.280 70.136 56.434
37 100.000 96.277 84.913 70.945 57.305
38 100.000 96.785 85.758 71.921 58.274
39 100.000 97.124 86.368 72.693 59.108
40 100.000 97.462 86.978 73.455 59.920
41 100.000 97.462 87.236 73.964 60.586
42 100.000 97.631 87.705 74.639 61.348
43 100.000 97.631 87.940 75.106 61.967
44 100.000 97.800 88.386 75.749 62.701
45 100.000 97.800 88.644 76.248 63.345
46 100.000 98.308 89.395 77.079 64.170
47 100.000 98.308 89.747 77.712 64.922
48 100.000 98.477 90.169 78.312 65.606
49 100.000 98.477 90.286 78.623 66.091
50 100.000 98.646 90.685 79.186 66.738
51 100.000 98.646 90.873 79.583 67.284
52 100.000 98.646 91.084 80.006 67.848
53 100.000 98.646 91.295 80.430 68.412
54 100.000 98.646 91.506 80.848 68.965
55 100.000 98.985 92.046 81.486 69.634
56 100.000 98.985 92.234 81.862 70.139
57 100.000 98.985 92.445 82.280 70.684
58 100.000 98.985 92.586 82.596 71.143
59 100.000 99.154 92.938 83.084 71.709
60 100.000 99.323 93.243 83.513 72.228
61 100.000 99.323 93.430 83.888 72.730
62 100.000 99.323 93.571 84.199 73.175
63 100.000 99.323 93.759 84.569 73.665
64 100.000 99.323 93.876 84.848 74.085
65 100.000 99.323 94.040 85.186 74.547
66 100.000 99.323 94.134 85.432 74.939
67 100.000 99.323 94.252 85.700 75.335
68 100.000 99.323 94.345 85.931 75.694
69 100.000 99.323 94.416 86.129 76.023
70 100.000 99.323 94.557 86.419 76.423
71 100.000 99.323 94.697 86.708 76.822
72 100.000 99.323 94.838 86.998 77.221
73 100.000 99.323 94.955 87.255 77.593
74 100.000 99.323 95.026 87.448 77.909
75 100.000 99.323 95.073 87.609 78.200
76 100.000 99.323 95.143 87.802 78.516
77 100.000 99.492 95.448 88.194 78.958
78 100.000 99.662 95.753 88.585 79.398
79 100.000 99.662 95.847 88.805 79.731
80 100.000 99.831 96.176 89.228 80.193
81 100.000 99.831 96.269 89.443 80.515
82 100.000 99.831 96.340 89.625 80.809
83 100.000 99.831 96.481 89.909 81.188
84 100.000 99.831 96.574 90.124 81.508
85 100.000 99.831 96.715 90.408 81.887
86 100.000 99.831 96.809 90.612 82.187
87 100.000 99.831 96.809 90.698 82.394
88 100.000 99.831 96.903 90.901 82.689
89 100.000 99.831 96.973 91.073 82.958
90 100.000 99.831 97.067 91.271 83.242
91 100.000 99.831 97.137 91.437 83.499
92 100.000 99.831 97.208 91.604 83.756
93 100.000 99.831 97.255 91.738 83.986
94 100.000 99.831 97.302 91.872 84.215
95 100.000 99.831 97.396 92.070 84.496
96 100.000 99.831 97.466 92.231 84.742
97 100.000 99.831 97.536 92.392 84.986
98 100.000 99.831 97.583 92.521 85.204
99 100.000 99.831 97.677 92.714 85.473
100 100.000 99.831 97.771 92.907 85.741
101 100.000 99.831 97.818 93.030 85.948
102 100.000 99.831 97.888 93.185 86.181
103 100.000 99.831 97.935 93.309 86.386
104 100.000 99.831 98.006 93.464 86.617
105 100.000 99.831 98.076 93.620 86.848
106 100.000 99.831 98.146 93.775 87.078
107 100.000 99.831 98.217 93.931 87.307
108 100.000 99.831 98.264 94.049 87.501
109 100.000 99.831 98.264 94.113 87.656
110 100.000 99.831 98.334 94.263 87.873
111 100.000 100.000 98.592 94.558 88.178
112 100.000 100.000 98.639 94.676 88.368
113 100.000 100.000 98.710 94.826 88.585
114 100.000 100.000 98.756 94.939 88.768
115 100.000 100.000 98.780 95.019 88.923
116 100.000 100.000 98.827 95.132 89.103
117 100.000 100.000 98.897 95.276 89.308
118 100.000 100.000 98.921 95.357 89.461
119 100.000 100.000 98.944 95.437 89.615
120 100.000 100.000 99.015 95.582 89.820
121 100.000 100.000 99.061 95.695 89.998
122 100.000 100.000 99.108 95.807 90.175
123 100.000 100.000 99.155 95.920 90.353
124 100.000 100.000 99.202 96.032 90.530
125 100.000 100.000 99.226 96.107 90.673
126 100.000 100.000 99.273 96.215 90.841
127 100.000 100.000 99.273 96.258 90.955
128 100.000 100.000 99.296 96.333 91.094
129 100.000 100.000 99.320 96.408 91.233
130 100.000 100.000 99.366 96.515 91.397
131 100.000 100.000 99.413 96.622 91.562
132 100.000 100.000 99.460 96.729 91.727
133 100.000 100.000 99.460 96.762 91.821
134 100.000 100.000 99.484 96.831 91.947
135 100.000 100.000 99.484 96.869 92.047
136 100.000 100.000 99.507 96.939 92.172
137 100.000 100.000 99.507 96.976 92.272
138 100.000 100.000 99.531 97.046 92.398
139 100.000 100.000 99.554 97.115 92.524
140 100.000 100.000 99.601 97.217 92.673
141 100.000 100.000 99.648 97.319 92.823
142 100.000 100.000 99.672 97.384 92.938
143 100.000 100.000 99.672 97.421 93.034
144 100.000 100.000 99.672 97.453 93.121
145 100.000 100.000 99.672 97.485 93.208
146 100.000 100.000 99.672 97.512 93.287
147 100.000 100.000 99.672 97.544 93.374
148 100.000 100.000 99.672 97.577 93.461
149 100.000 100.000 99.672 97.609 93.548
150 100.000 100.000 99.672 97.641 93.635
151 100.000 100.000 99.695 97.705 93.746
152 100.000 100.000 99.695 97.732 93.822
153 100.000 100.000 99.695 97.759 93.898
154 100.000 100.000 99.695 97.786 93.973
155 100.000 100.000 99.695 97.812 94.049
156 100.000 100.000 99.718 97.871 94.149
157 100.000 100.000 99.718 97.898 94.223
158 100.000 100.000 99.742 97.957 94.323
159 100.000 100.000 99.765 98.016 94.421
160 100.000 100.000 99.765 98.043 94.493
161 100.000 100.000 99.789 98.102 94.591
162 100.000 100.000 99.812 98.161 94.689
163 100.000 100.000 99.836 98.220 94.787
164 100.000 100.000 99.836 98.247 94.859
165 100.000 100.000 99.836 98.274 94.931
166 100.000 100.000 99.836 98.300 95.003
167 100.000 100.000 99.859 98.359 95.101
168 100.000 100.000 99.883 98.418 95.199
169 100.000 100.000 99.906 98.477 95.297
170 100.000 100.000 99.906 98.504 95.370
171 100.000 100.000 99.906 98.526 95.432
172 100.000 100.000 99.906 98.547 95.495
173 100.000 100.000 99.906 98.568 95.558
174 100.000 100.000 99.906 98.590 95.621
175 100.000 100.000 99.906 98.611 95.684
176 100.000 100.000 99.906 98.633 95.745
177 100.000 100.000 99.906 98.654 95.806
178 100.000 100.000 99.906 98.676 95.865
179 100.000 100.000 99.906 98.697 95.924
180 100.000 100.000 99.906 98.719 95.984
181 100.000 100.000 99.906 98.740 96.043
182 100.000 100.000 99.906 98.761 96.102
183 100.000 100.000 99.906 98.783 96.161
184 100.000 100.000 99.906 98.804 96.220
185 100.000 100.000 99.906 98.826 96.279
186 100.000 100.000 99.930 98.879 96.364
187 100.000 100.000 99.930 98.901 96.424
188 100.000 100.000 99.930 98.922 96.483
189 100.000 100.000 99.930 98.944 96.542
190 100.000 100.000 99.930 98.965 96.601
191 100.000 100.000 99.930 98.987 96.660
192 100.000 100.000 99.930 99.008 96.719
193 100.000 100.000 99.953 99.062 96.805
194 100.000 100.000 99.977 99.115 96.888
195 100.000 100.000 100.000 99.169 96.971
196 100.000 100.000 100.000 99.185 97.019
197 100.000 100.000 100.000 99.201 97.067
198 100.000 100.000 100.000 99.217 97.115
199 100.000 100.000 100.000 99.233 97.163
200 100.000 100.000 100.000 99.249 97.211
201 100.000 100.000 100.000 99.265 97.259
202 100.000 100.000 100.000 99.282 97.306
203 100.000 100.000 100.000 99.298 97.352
204 100.000 100.000 100.000 99.314 97.398
205 100.000 100.000 100.000 99.330 97.444
206 100.000 100.000 100.000 99.346 97.491
207 100.000 100.000 100.000 99.362 97.537
208 100.000 100.000 100.000 99.373 97.572
209 100.000 100.000 100.000 99.383 97.607
210 100.000 100.000 100.000 99.394 97.642
211 100.000 100.000 100.000 99.405 97.676
212 100.000 100.000 100.000 99.416 97.709
213 100.000 100.000 100.000 99.426 97.742
214 100.000 100.000 100.000 99.437 97.775
215 100.000 100.000 100.000 99.448 97.809
216 100.000 100.000 100.000 99.458 97.842
217 100.000 100.000 100.000 99.469 97.875
218 100.000 100.000 100.000 99.480 97.909
219 100.000 100.000 100.000 99.491 97.942
220 100.000 100.000 100.000 99.501 97.975
221 100.000 100.000 100.000 99.512 98.007
222 100.000 100.000 100.000 99.523 98.038
223 100.000 100.000 100.000 99.534 98.069
224 100.000 100.000 100.000 99.544 98.101
225 100.000 100.000 100.000 99.555 98.132
226 100.000 100.000 100.000 99.566 98.164
227 100.000 100.000 100.000 99.576 98.195
228 100.000 100.000 100.000 99.587 98.227
229 100.000 100.000 100.000 99.598 98.258
230 100.000 100.000 100.000 99.609 98.289
231 100.000 100.000 100.000 99.619 98.321
232 100.000 100.000 100.000 99.630 98.352
233 100.000 100.000 100.000 99.641 98.384
234 100.000 100.000 100.000 99.651 98.415
235 100.000 100.000 100.000 99.662 98.447
236 100.000 100.000 100.000 99.673 98.478
237 100.000 100.000 100.000 99.684 98.510
238 100.000 100.000 100.000 99.684 98.524
239 100.000 100.000 100.000 99.689 98.547
240 100.000 100.000 100.000 99.694 98.567
241 100.000 100.000 100.000 99.700 98.587
242 100.000 100.000 100.000 99.705 98.608
243 100.000 100.000 100.000 99.710 98.628
244 100.000 100.000 100.000 99.716 98.648
245 100.000 100.000 100.000 99.716 98.659
246 100.000 100.000 100.000 99.721 98.678
247 100.000 100.000 100.000 99.727 98.696
248 100.000 100.000 100.000 99.732 98.715
249 100.000 100.000 100.000 99.737 98.733
250 100.000 100.000 100.000 99.743 98.750
251 100.000 100.000 100.000 99.748 98.767
252 100.000 100.000 100.000 99.753 98.783
253 100.000 100.000 100.000 99.759 98.800
254 100.000 100.000 100.000 99.764 98.817
255 100.000 100.000 100.000 99.769 98.833
256 100.000 100.000 100.000 99.775 98.850
257 100.000 100.000 100.000 99.780 98.866
258 100.000 100.000 100.000 99.786 98.883
259 100.000 100.000 100.000 99.791 98.900
260 100.000 100.000 100.000 99.796 98.916
261 100.000 100.000 100.000 99.802 98.933
262 100.000 100.000 100.000 99.807 98.950
263 100.000 100.000 100.000 99.812 98.966
264 100.000 100.000 100.000 99.818 98.983
265 100.000 100.000 100.000 99.823 99.000
266 100.000 100.000 100.000 99.828 99.016
267 100.000 100.000 100.000 99.834 99.033
268 100.000 100.000 100.000 99.839 99.050
269 100.000 100.000 100.000 99.845 99.066
270 100.000 100.000 100.000 99.850 99.083
271 100.000 100.000 100.000 99.855 99.099
272 100.000 100.000 100.000 99.861 99.116
273 100.000 100.000 100.000 99.866 99.133
274 100.000 100.000 100.000 99.871 99.149
275 100.000 100.000 100.000 99.877 99.166
276 100.000 100.000 100.000 99.882 99.183
277 100.000 100.000 100.000 99.887 99.199
278 100.000 100.000 100.000 99.893 99.216
279 100.000 100.000 100.000 99.898 99.233
280 100.000 100.000 100.000 99.903 99.249
281 100.000 100.000 100.000 99.909 99.266
282 100.000 100.000 100.000 99.914 99.283
283 100.000 100.000 100.000 99.920 99.299
284 100.000 100.000 100.000 99.925 99.316
285 100.000 100.000 100.000 99.930 99.332
286 100.000 100.000 100.000 99.936 99.349
287 100.000 100.000 100.000 99.941 99.366
288 100.000 100.000 100.000 99.946 99.382
289 100.000 100.000 100.000 99.952 99.399
290 100.000 100.000 100.000 99.957 99.416
291 100.000 100.000 100.000 99.962 99.432
292 100.000 100.000 100.000 99.968 99.449
293 100.000 100.000 100.000 99.973 99.466
294 100.000 100.000 100.000 99.979 99.482
295 100.000 100.000 100.000 99.984 99.499
296 100.000 100.000 100.000 99.989 99.516
297 100.000 100.000 100.000 99.995 99.532
298 100.000 100.000 100.000 100.000 99.549
299 100.000 100.000 100.000 100.000 99.556
300 100.000 100.000 100.000 100.000 99.564
301 100.000 100.000 100.000 100.000 99.571
302 100.000 100.000 100.000 100.000 99.578
303 100.000 100.000 100.000 100.000 99.586
304 100.000 100.000 100.000 100.000 99.593
305 100.000 100.000 100.000 100.000 99.601
306 100.000 100.000 100.000 100.000 99.608
307 100.000 100.000 100.000 100.000 99.615
308 100.000 100.000 100.000 100.000 99.623
309 100.000 100.000 100.000 100.000 99.630
310 100.000 100.000 100.000 100.000 99.638
311 100.000 100.000 100.000 100.000 99.643
312 100.000 100.000 100.000 100.000 99.647
313 100.000 100.000 100.000 100.000 99.650
314 100.000 100.000 100.000 100.000 99.654
315 100.000 100.000 100.000 100.000 99.658
316 100.000 100.000 100.000 100.000 99.662
317 100.000 100.000 100.000 100.000 99.665
318 100.000 100.000 100.000 100.000 99.669
319 100.000 100.000 100.000 100.000 99.673
320 100.000 100.000 100.000 100.000 99.676
321 100.000 100.000 100.000 100.000 99.680
322 100.000 100.000 100.000 100.000 99.684
323 100.000 100.000 100.000 100.000 99.687
324 100.000 100.000 100.000 100.000 99.691
325 100.000 100.000 100.000 100.000 99.695
326 100.000 100.000 100.000 100.000 99.699
327 100.000 100.000 100.000 100.000 99.702
328 100.000 100.000 100.000 100.000 99.706
329 100.000 100.000 100.000 100.000 99.710
330 100.000 100.000 100.000 100.000 99.713
331 100.000 100.000 100.000 100.000 99.717
332 100.000 100.000 100.000 100.000 99.721
333 100.000 100.000 100.000 100.000 99.724
334 100.000 100.000 100.000 100.000 99.728
335 100.000 100.000 100.000 100.000 99.732
336 100.000 100.000 100.000 100.000 99.736
337 100.000 100.000 100.000 100.000 99.739
338 100.000 100.000 100.000 100.000 99.743
339 100.000 100.000 100.000 100.000 99.747
340 100.000 100.000 100.000 100.000 99.750
341 100.000 100.000 100.000 100.000 99.754
342 100.000 100.000 100.000 100.000 99.758
343 100.000 100.000 100.000 100.000 99.761
344 100.000 100.000 100.000 100.000 99.765
345 100.000 100.000 100.000 100.000 99.769
346 100.000 100.000 100.000 100.000 99.773
347 100.000 100.000 100.000 100.000 99.776
348 100.000 100.000 100.000 100.000 99.780
349 100.000 100.000 100.000 100.000 99.784
350 100.000 100.000 100.000 100.000 99.787
351 100.000 100.000 100.000 100.000 99.791
352 100.000 100.000 100.000 100.000 99.795
353 100.000 100.000 100.000 100.000 99.798
354 100.000 100.000 100.000 100.000 99.802
355 100.000 100.000 100.000 100.000 99.806
356 100.000 100.000 100.000 100.000 99.810
357 100.000 100.000 100.000 100.000 99.813
358 100.000 100.000 100.000 100.000 99.817
359 100.000 100.000 100.000 100.000 99.821
360 100.000 100.000 100.000 100.000 99.824
361 100.000 100.000 100.000 100.000 99.828
362 100.000 100.000 100.000 100.000 99.832
363 100.000 100.000 100.000 100.000 99.835
364 100.000 100.000 100.000 100.000 99.839
365 100.000 100.000 100.000 100.000 99.843
366 100.000 100.000 100.000 100.000 99.847
367 100.000 100.000 100.000 100.000 99.850
368 100.000 100.000 100.000 100.000 99.854
369 100.000 100.000 100.000 100.000 99.858
370 100.000 100.000 100.000 100.000 99.861
371 100.000 100.000 100.000 100.000 99.865
372 100.000 100.000 100.000 100.000 99.869
373 100.000 100.000 100.000 100.000 99.872
374 100.000 100.000 100.000 100.000 99.876
375 100.000 100.000 100.000 100.000 99.880
376 100.000 100.000 100.000 100.000 99.883
377 100.000 100.000 100.000 100.000 99.887
378 100.000 100.000 100.000 100.000 99.891
379 100.000 100.000 100.000 100.000 99.895
380 100.000 100.000 100.000 100.000 99.896
381 100.000 100.000 100.000 100.000 99.898
382 100.000 100.000 100.000 100.000 99.900
383 100.000 100.000 100.000 100.000 99.902
384 100.000 100.000 100.000 100.000 99.904
385 100.000 100.000 100.000 100.000 99.906
386 100.000 100.000 100.000 100.000 99.908
387 100.000 100.000 100.000 100.000 99.909
388 100.000 100.000 100.000 100.000 99.911
389 100.000 100.000 100.000 100.000 99.913
390 100.000 100.000 100.000 100.000 99.915
391 100.000 100.000 100.000 100.000 99.917
392 100.000 100.000 100.000 100.000 99.919
393 100.000 100.000 100.000 100.000 99.920
394 100.000 100.000 100.000 100.000 99.922
395 100.000 100.000 100.000 100.000 99.924
396 100.000 100.000 100.000 100.000 99.926
397 100.000 100.000 100.000 100.000 99.928
398 100.000 100.000 100.000 100.000 99.930
399 100.000 100.000 100.000 100.000 99.932
400 100.000 100.000 100.000 100.000 99.933
401 100.000 100.000 100.000 100.000 99.935
402 100.000 100.000 100.000 100.000 99.937
403 100.000 100.000 100.000 100.000 99.939
404 100.000 100.000 100.000 100.000 99.941
405 100.000 100.000 100.000 100.000 99.943
406 100.000 100.000 100.000 100.000 99.945
407 100.000 100.000 100.000 100.000 99.946
408 100.000 100.000 100.000 100.000 99.948
409 100.000 100.000 100.000 100.000 99.950
410 100.000 100.000 100.000 100.000 99.952
411 100.000 100.000 100.000 100.000 99.954
412 100.000 100.000 100.000 100.000 99.956
413 100.000 100.000 100.000 100.000 99.957
414 100.000 100.000 100.000 100.000 99.959
415 100.000 100.000 100.000 100.000 99.961
416 100.000 100.000 100.000 100.000 99.963
417 100.000 100.000 100.000 100.000 99.965
418 100.000 100.000 100.000 100.000 99.967
419 100.000 100.000 100.000 100.000 99.969
420 100.000 100.000 100.000 100.000 99.970
421 100.000 100.000 100.000 100.000 99.972
422 100.000 100.000 100.000 100.000 99.974
423 100.000 100.000 100.000 100.000 99.976
424 100.000 100.000 100.000 100.000 99.978
425 100.000 100.000 100.000 100.000 99.980
426 100.000 100.000 100.000 100.000 99.982
427 100.000 100.000 100.000 100.000 99.983
428 100.000 100.000 100.000 100.000 99.985
429 100.000 100.000 100.000 100.000 99.987
430 100.000 100.000 100.000 100.000 99.989
431 100.000 100.000 100.000 100.000 99.991
432 100.000 100.000 100.000 100.000 99.993
433 100.000 100.000 100.000 100.000 99.994
434 100.000 100.000 100.000 100.000 99.996
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.557 67.361 58.835
8 97.368 89.555 80.347 71.159 62.542
9 100.000 93.088 84.092 74.853 66.091
10 100.000 93.856 85.664 77.056 68.721
11 100.000 94.624 87.147 79.050 71.033
12 100.000 95.392 88.616 81.004 73.276
13 100.000 96.160 90.100 82.990 75.565
14 100.000 96.928 91.569 84.927 77.757
15 100.000 97.696 93.038 86.869 79.963
16 100.000 98.464 94.506 88.806 82.154
17 100.000 99.232 96.034 90.906 84.627
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.347 62.970 52.942 44.301
5 89.474 78.648 68.243 58.664 50.095
6 92.105 82.488 72.738 63.484 54.998
7 94.737 86.022 76.601 67.465 58.989
8 97.368 89.555 80.376 71.230 62.647
9 100.000 93.088 84.122 74.918 66.180
10 100.000 93.856 85.679 77.095 68.783
11 100.000 94.624 87.206 79.194 71.258
12 100.000 95.392 88.704 81.219 73.611
13 100.000 96.160 90.203 83.249 75.983
14 100.000 96.928 91.686 85.236 78.272
15 100.000 97.696 93.155 87.177 80.477
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 78.947 61.290 47.474 36.905 28.873
3 84.211 69.585 56.962 46.428 37.793
4 86.842 75.115 64.233 54.358 45.643
5 89.474 78.802 68.478 58.904 50.294
6 92.105 82.488 72.723 63.448 54.940
7 94.737 86.022 76.572 67.396 58.886
8 97.368 89.555 80.405 71.303 62.763
9 100.000 93.088 84.151 74.989 66.290
10 100.000 93.856 85.679 77.093 68.778
11 100.000 94.624 87.206 79.194 71.260
12 100.000 95.392 88.719 81.261 73.688
13 100.000 96.160 90.217 83.294 76.066
14 100.000 96.928 91.686 85.240 78.285
15 100.000 97.696 93.155 87.181 80.490
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 89.474 79.724 70.535 61.935 54.002
7 92.105 83.257 74.354 65.811 57.839
8 94.737 86.790 78.173 69.682 61.660
9 97.368 90.323 81.933 73.411 65.262
10 100.000 93.856 85.693 77.135 68.854
11 100.000 94.624 87.221 79.237 71.335
12 100.000 95.392 88.719 81.272 73.719
13 100.000 96.160 90.217 83.294 76.066
14 100.000 96.928 91.686 85.240 78.285
15 100.000 97.696 93.155 87.181 80.490
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.347 62.970 52.942 44.301
5 89.474 78.648 68.199 58.561 49.940
6 92.105 82.335 72.429 63.075 54.547
7 94.737 86.022 76.601 67.463 58.983
8 97.368 89.555 80.479 71.477 63.025
9 100.000 93.088 84.269 75.277 66.739
10 100.000 93.702 85.517 77.042 68.885
11 100.000 94.316 86.722 78.683 70.814
12 100.000 94.931 87.985 80.480 73.005
13 100.000 95.545 89.189 82.120 74.929
14 100.000 96.006 90.115 83.420 76.510
15 100.000 96.467 91.040 84.700 78.027
16 100.000 96.928 91.965 85.980 79.541
17 100.000 97.389 92.891 87.260 81.056
18 100.000 97.849 93.816 88.544 82.587
19 100.000 98.157 94.477 89.530 83.849
20 100.000 98.464 95.138 90.500 85.057
21 100.000 98.771 95.799 91.470 86.268
22 100.000 99.078 96.460 92.443 87.489
23 100.000 99.386 97.121 93.419 88.717
24 100.000 99.539 97.488 94.005 89.501
25 100.000 99.693 97.870 94.628 90.342
26 100.000 99.846 98.237 95.212 91.119
27 100.000 100.000 98.605 95.797 91.896
28 100.000 100.000 98.707 96.073 92.373
29 100.000 100.000 98.796 96.310 92.782
30 100.000 100.000 98.884 96.555 93.220
31 100.000 100.000 98.972 96.793 93.630
32 100.000 100.000 99.060 97.034 94.054
33 100.000 100.000 99.163 97.318 94.556
34 100.000 100.000 99.222 97.489 94.871
35 100.000 100.000 99.324 97.764 95.346
36 100.000 100.000 99.412 98.002 95.757
37 100.000 100.000 99.471 98.164 96.046
38 100.000 100.000 99.559 98.401 96.456
39 100.000 100.000 99.618 98.564 96.743
40 100.000 100.000 99.706 98.801 97.153
41 100.000 100.000 99.765 98.963 97.441
42 100.000 100.000 99.824 99.126 97.728
43 100.000 100.000 99.882 99.288 98.015
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.552 71.650 63.284
9 100.000 93.088 84.342 75.451 67.002
10 100.000 93.702 85.590 77.218 69.154
11 100.000 94.316 86.810 78.916 71.203
12 100.000 94.931 88.014 80.568 73.164
13 100.000 95.545 89.219 82.207 75.088
14 100.000 96.160 90.423 83.852 77.030
15 100.000 96.774 91.627 85.483 78.929
16 100.000 97.389 92.832 87.122 80.852
17 100.000 98.003 94.036 88.749 82.737
18 100.000 98.464 94.962 90.037 84.281
19 100.000 98.771 95.623 91.007 85.489
20 100.000 98.925 96.005 91.630 86.331
21 100.000 99.078 96.387 92.267 87.219
22 100.000 99.232 96.769 92.900 88.095
23 100.000 99.386 97.150 93.523 88.938
24 100.000 99.539 97.532 94.147 89.786
25 100.000 99.693 97.914 94.770 90.627
26 100.000 99.846 98.296 95.393 91.468
27 100.000 100.000 98.678 96.016 92.310
28 100.000 100.000 98.766 96.257 92.736
29 100.000 100.000 98.854 96.495 93.146
30 100.000 100.000 98.942 96.732 93.556
31 100.000 100.000 99.031 96.969 93.965
32 100.000 100.000 99.119 97.207 94.377
33 100.000 100.000 99.207 97.444 94.787
34 100.000 100.000 99.295 97.681 95.197
35 100.000 100.000 99.383 97.919 95.607
36 100.000 100.000 99.471 98.156 96.018
37 100.000 100.000 99.530 98.318 96.305
38 100.000 100.000 99.589 98.481 96.593
39 100.000 100.000 99.647 98.643 96.880
40 100.000 100.000 99.706 98.805 97.168
41 100.000 100.000 99.765 98.967 97.455
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.675 67.637 59.246
8 97.368 89.555 80.567 71.684 63.334
9 100.000 93.088 84.327 75.417 66.951
10 100.000 93.702 85.576 77.180 69.091
11 100.000 94.316 86.795 78.878 71.139
12 100.000 94.931 87.999 80.531 73.107
13 100.000 95.545 89.204 82.172 75.036
14 100.000 96.160 90.408 83.813 76.966
15 100.000 96.774 91.613 85.451 78.884
16 100.000 97.389 92.817 87.082 80.782
17 100.000 98.003 94.022 88.709 82.665
18 100.000 98.618 95.226 90.336 84.548
19 100.000 98.771 95.623 91.007 85.488
20 100.000 98.925 96.005 91.634 86.344
21 100.000 99.078 96.387 92.259 87.192
22 100.000 99.232 96.769 92.884 88.041
23 100.000 99.386 97.150 93.506 88.883
24 100.000 99.539 97.532 94.129 89.724
25 100.000 99.693 97.914 94.752 90.565
26 100.000 99.846 98.296 95.375 91.408
27 100.000 100.000 98.678 95.997 92.248
28 100.000 100.000 98.766 96.245 92.692
29 100.000 100.000 98.854 96.486 93.116
30 100.000 100.000 98.942 96.728 93.541
31 100.000 100.000 99.031 96.965 93.951
32 100.000 100.000 99.119 97.202 94.361
33 100.000 100.000 99.207 97.440 94.770
34 100.000 100.000 99.295 97.677 95.180
35 100.000 100.000 99.383 97.915 95.590
36 100.000 100.000 99.471 98.152 96.002
37 100.000 100.000 99.530 98.318 96.304
38 100.000 100.000 99.589 98.481 96.592
39 100.000 100.000 99.647 98.643 96.879
40 100.000 100.000 99.706 98.805 97.167
41 100.000 100.000 99.765 98.967 97.454
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.868 76.293 67.053 58.532
8 97.368 89.401 80.185 71.096 62.610
9 100.000 93.088 84.224 75.173 66.585
10 100.000 93.702 85.532 77.076 68.937
11 100.000 94.316 86.780 78.837 71.071
12 100.000 94.931 87.985 80.493 73.042
13 100.000 95.545 89.189 82.136 74.978
14 100.000 96.160 90.394 83.777 76.909
15 100.000 96.774 91.598 85.410 78.812
16 100.000 97.389 92.803 87.037 80.696
17 100.000 98.003 94.007 88.664 82.580
18 100.000 98.464 94.932 89.950 84.117
19 100.000 98.771 95.593 90.920 85.327
20 100.000 98.925 95.990 91.591 86.267
21 100.000 99.078 96.387 92.263 87.206
22 100.000 99.232 96.769 92.890 88.062
23 100.000 99.386 97.150 93.514 88.911
24 100.000 99.539 97.532 94.137 89.753
25 100.000 99.693 97.914 94.760 90.594
26 100.000 99.846 98.296 95.383 91.434
27 100.000 100.000 98.678 96.006 92.274
28 100.000 100.000 98.766 96.253 92.719
29 100.000 100.000 98.854 96.495 93.144
30 100.000 100.000 98.942 96.732 93.556
31 100.000 100.000 99.031 96.969 93.968
32 100.000 100.000 99.119 97.207 94.378
33 100.000 100.000 99.207 97.444 94.787
34 100.000 100.000 99.295 97.681 95.197
35 100.000 100.000 99.383 97.919 95.607
36 100.000 100.000 99.471 98.156 96.017
37 100.000 100.000 99.530 98.318 96.305
38 100.000 100.000 99.589 98.481 96.593
39 100.000 100.000 99.647 98.643 96.880
40 100.000 100.000 99.706 98.805 97.168
41 100.000 100.000 99.765 98.967 97.455
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.150 62.732 54.193
7 94.737 86.022 76.542 67.329 58.789
8 97.368 89.555 80.288 71.013 62.310
9 100.000 93.088 84.122 74.908 66.153
10 100.000 93.856 85.693 77.119 68.805
11 100.000 94.624 87.192 79.152 71.182
12 100.000 95.392 88.690 81.182 73.554
13 100.000 96.160 90.159 83.120 75.744
14 100.000 96.928 91.657 85.142 78.092
15 100.000 97.696 93.126 87.088 80.310
16 100.000 98.464 94.624 89.122 82.693
17 100.000 99.232 96.093 91.068 84.910
18 100.000 100.000 97.562 93.005 87.101
19 100.000 100.000 97.679 93.330 87.673
20 100.000 100.000 97.767 93.581 88.132
21 100.000 100.000 97.855 93.819 88.545
22 100.000 100.000 97.944 94.060 88.969
23 100.000 100.000 98.061 94.383 89.536
24 100.000 100.000 98.135 94.588 89.903
25 100.000 100.000 98.223 94.825 90.316
26 100.000 100.000 98.281 94.987 90.606
27 100.000 100.000 98.370 95.225 91.015
28 100.000 100.000 98.443 95.434 91.397
29 100.000 100.000 98.502 95.596 91.684
30 100.000 100.000 98.575 95.797 92.038
31 100.000 100.000 98.663 96.034 92.449
32 100.000 100.000 98.751 96.271 92.858
33 100.000 100.000 98.840 96.509 93.269
34 100.000 100.000 98.913 96.710 93.621
35 100.000 100.000 98.957 96.835 93.851
36 100.000 100.000 99.031 97.036 94.204
37 100.000 100.000 99.119 97.274 94.616
38 100.000 100.000 99.177 97.436 94.904
39 100.000 100.000 99.236 97.598 95.191
40 100.000 100.000 99.266 97.687 95.363
41 100.000 100.000 99.295 97.777 95.535
42 100.000 100.000 99.310 97.827 95.642
43 100.000 100.000 99.324 97.878 95.750
44 100.000 100.000 99.368 98.004 95.980
45 100.000 100.000 99.383 98.055 96.087
46 100.000 100.000 99.383 98.069 96.137
47 100.000 100.000 99.398 98.119 96.244
48 100.000 100.000 99.486 98.357 96.654
49 100.000 100.000 99.486 98.371 96.703
50 100.000 100.000 99.545 98.533 96.991
51 100.000 100.000 99.574 98.623 97.163
52 100.000 100.000 99.633 98.785 97.450
53 100.000 100.000 99.662 98.874 97.622
54 100.000 100.000 99.706 99.000 97.852
55 100.000 100.000 99.780 99.201 98.204
56 100.000 100.000 99.794 99.251 98.311
57 100.000 100.000 99.838 99.377 98.540
58 100.000 100.000 99.853 99.428 98.648
59 100.000 100.000 99.853 99.442 98.697
60 100.000 100.000 99.882 99.531 98.869
61 100.000 100.000 99.882 99.546 98.919
62 100.000 100.000 99.927 99.671 99.148
63 100.000 100.000 99.956 99.761 99.320
64 100.000 100.000 99.956 99.775 99.370
65 100.000 100.000 99.971 99.826 99.477
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.572 67.396 58.886
8 97.368 89.555 80.449 71.408 62.923
9 100.000 93.088 84.254 75.242 66.687
10 100.000 93.856 85.781 77.352 69.194
11 100.000 94.624 87.294 79.423 71.636
12 100.000 95.392 88.793 81.464 74.038
13 100.000 96.160 90.276 83.460 76.359
14 100.000 96.928 91.745 85.422 78.628
15 100.000 97.696 93.214 87.360 80.822
16 100.000 98.464 94.683 89.297 83.015
17 100.000 99.232 96.152 91.234 85.209
18 100.000 100.000 97.620 93.172 87.402
19 100.000 100.000 97.709 93.409 87.817
20 100.000 100.000 97.797 93.665 88.286
21 100.000 100.000 97.885 93.902 88.699
22 100.000 100.000 97.973 94.139 89.112
23 100.000 100.000 98.061 94.381 89.536
24 100.000 100.000 98.149 94.618 89.946
25 100.000 100.000 98.237 94.855 90.357
26 100.000 100.000 98.325 95.097 90.781
27 100.000 100.000 98.414 95.334 91.191
28 100.000 100.000 98.502 95.571 91.601
29 100.000 100.000 98.590 95.809 92.012
30 100.000 100.000 98.678 96.046 92.422
31 100.000 100.000 98.766 96.288 92.846
32 100.000 100.000 98.854 96.525 93.255
33 100.000 100.000 98.942 96.768 93.688
34 100.000 100.000 99.031 97.006 94.098
35 100.000 100.000 99.119 97.243 94.508
36 100.000 100.000 99.207 97.480 94.917
37 100.000 100.000 99.251 97.606 95.147
38 100.000 100.000 99.295 97.732 95.377
39 100.000 100.000 99.339 97.858 95.606
40 100.000 100.000 99.383 97.984 95.836
41 100.000 100.000 99.427 98.109 96.066
42 100.000 100.000 99.471 98.235 96.295
43 100.000 100.000 99.515 98.361 96.525
44 100.000 100.000 99.559 98.487 96.756
45 100.000 100.000 99.603 98.612 96.985
46 100.000 100.000 99.647 98.738 97.216
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.676
49 100.000 100.000 99.780 99.116 97.906
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.365
52 100.000 100.000 99.912 99.493 98.595
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.714 76.058 66.807 58.315
8 97.368 89.247 79.950 70.852 62.398
9 100.000 92.934 83.975 74.896 66.323
10 100.000 93.856 85.752 77.283 69.092
11 100.000 94.624 87.221 79.239 71.341
12 100.000 95.392 88.690 81.194 73.590
13 100.000 96.160 90.159 83.146 75.828
14 100.000 96.928 91.627 85.098 78.066
15 100.000 97.696 93.096 87.039 80.271
16 100.000 98.464 94.565 88.980 82.477
17 100.000 99.232 96.034 90.922 84.681
18 100.000 100.000 97.503 92.863 86.886
19 100.000 100.000 97.650 93.251 87.543
20 100.000 100.000 97.797 93.638 88.200
21 100.000 100.000 97.885 93.888 88.651
22 100.000 100.000 97.973 94.137 89.102
23 100.000 100.000 98.061 94.385 89.547
24 100.000 100.000 98.149 94.632 89.993
25 100.000 100.000 98.237 94.870 90.404
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.228
28 100.000 100.000 98.502 95.582 91.638
29 100.000 100.000 98.590 95.819 92.049
30 100.000 100.000 98.678 96.056 92.460
31 100.000 100.000 98.766 96.294 92.870
32 100.000 100.000 98.854 96.531 93.281
33 100.000 100.000 98.942 96.768 93.692
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.251 97.606 95.151
38 100.000 100.000 99.295 97.732 95.380
39 100.000 100.000 99.339 97.858 95.610
40 100.000 100.000 99.383 97.984 95.840
41 100.000 100.000 99.427 98.109 96.069
42 100.000 100.000 99.471 98.235 96.299
43 100.000 100.000 99.515 98.361 96.529
44 100.000 100.000 99.559 98.487 96.758
45 100.000 100.000 99.603 98.612 96.988
46 100.000 100.000 99.647 98.738 97.218
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.677
49 100.000 100.000 99.780 99.116 97.907
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.366
52 100.000 100.000 99.912 99.493 98.596
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.714 76.058 66.807 58.315
8 97.368 89.247 79.950 70.852 62.398
9 100.000 92.934 83.975 74.896 66.323
10 100.000 93.856 85.752 77.283 69.092
11 100.000 94.624 87.221 79.239 71.341
12 100.000 95.392 88.690 81.194 73.590
13 100.000 96.160 90.159 83.146 75.828
14 100.000 96.928 91.627 85.098 78.066
15 100.000 97.696 93.096 87.039 80.271
16 100.000 98.464 94.565 88.980 82.477
17 100.000 99.232 96.034 90.922 84.681
18 100.000 100.000 97.503 92.863 86.886
19 100.000 100.000 97.650 93.251 87.543
20 100.000 100.000 97.797 93.638 88.200
21 100.000 100.000 97.885 93.888 88.651
22 100.000 100.000 97.973 94.137 89.102
23 100.000 100.000 98.061 94.385 89.547
24 100.000 100.000 98.149 94.632 89.993
25 100.000 100.000 98.237 94.870 90.404
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.228
28 100.000 100.000 98.502 95.582 91.639
29 100.000 100.000 98.590 95.819 92.050
30 100.000 100.000 98.678 96.056 92.461
31 100.000 100.000 98.766 96.294 92.871
32 100.000 100.000 98.854 96.531 93.282
33 100.000 100.000 98.942 96.768 93.692
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.251 97.606 95.151
38 100.000 100.000 99.295 97.732 95.380
39 100.000 100.000 99.339 97.858 95.610
40 100.000 100.000 99.383 97.984 95.840
41 100.000 100.000 99.427 98.109 96.069
42 100.000 100.000 99.471 98.235 96.299
43 100.000 100.000 99.515 98.361 96.529
44 100.000 100.000 99.559 98.487 96.758
45 100.000 100.000 99.603 98.612 96.988
46 100.000 100.000 99.647 98.738 97.218
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.677
49 100.000 100.000 99.780 99.116 97.907
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.366
52 100.000 100.000 99.912 99.493 98.596
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.449 71.408 62.925
9 100.000 93.088 84.195 75.088 66.434
10 100.000 93.856 85.664 77.030 68.638
11 100.000 94.624 87.147 79.036 70.987
12 100.000 95.392 88.646 81.057 73.327
13 100.000 96.160 90.129 83.051 75.641
14 100.000 96.928 91.627 85.071 77.980
15 100.000 97.696 93.111 87.061 80.282
16 100.000 98.464 94.683 89.279 82.953
17 100.000 99.232 96.152 91.222 85.166
18 100.000 100.000 97.620 93.159 87.358
19 100.000 100.000 97.679 93.318 87.632
20 100.000 100.000 97.767 93.555 88.045
21 100.000 100.000 97.855 93.792 88.455
22 100.000 100.000 97.914 93.963 88.771
23 100.000 100.000 97.973 94.121 89.044
24 100.000 100.000 98.002 94.212 89.223
25 100.000 100.000 98.090 94.458 89.662
26 100.000 100.000 98.179 94.695 90.072
27 100.000 100.000 98.193 94.738 90.152
28 100.000 100.000 98.281 94.975 90.565
29 100.000 100.000 98.340 95.133 90.838
30 100.000 100.000 98.428 95.375 91.264
31 100.000 100.000 98.472 95.505 91.507
32 100.000 100.000 98.531 95.663 91.781
33 100.000 100.000 98.561 95.746 91.933
34 100.000 100.000 98.649 95.983 92.344
35 100.000 100.000 98.678 96.066 92.496
36 100.000 100.000 98.766 96.304 92.906
37 100.000 100.000 98.781 96.355 93.012
38 100.000 100.000 98.810 96.438 93.163
39 100.000 100.000 98.854 96.559 93.381
40 100.000 100.000 98.913 96.722 93.669
41 100.000 100.000 99.001 96.959 94.079
42 100.000 100.000 99.031 97.042 94.230
43 100.000 100.000 99.045 97.085 94.309
44 100.000 100.000 99.104 97.247 94.597
45 100.000 100.000 99.148 97.369 94.812
46 100.000 100.000 99.163 97.411 94.892
47 100.000 100.000 99.207 97.533 95.108
48 100.000 100.000 99.236 97.616 95.261
49 100.000 100.000 99.266 97.700 95.412
50 100.000 100.000 99.354 97.937 95.822
51 100.000 100.000 99.354 97.945 95.850
52 100.000 100.000 99.354 97.949 95.865
53 100.000 100.000 99.354 97.957 95.894
54 100.000 100.000 99.354 97.959 95.901
55 100.000 100.000 99.412 98.117 96.176
56 100.000 100.000 99.412 98.124 96.197
57 100.000 100.000 99.471 98.286 96.485
58 100.000 100.000 99.530 98.448 96.772
59 100.000 100.000 99.589 98.610 97.059
60 100.000 100.000 99.618 98.698 97.224
61 100.000 100.000 99.633 98.740 97.304
62 100.000 100.000 99.633 98.746 97.325
63 100.000 100.000 99.692 98.911 97.620
64 100.000 100.000 99.692 98.915 97.634
65 100.000 100.000 99.692 98.919 97.649
66 100.000 100.000 99.721 99.002 97.800
67 100.000 100.000 99.750 99.085 97.951
68 100.000 100.000 99.780 99.168 98.102
69 100.000 100.000 99.794 99.211 98.182
70 100.000 100.000 99.794 99.215 98.196
71 100.000 100.000 99.794 99.221 98.218
72 100.000 100.000 99.838 99.343 98.434
73 100.000 100.000 99.838 99.347 98.448
74 100.000 100.000 99.868 99.430 98.599
75 100.000 100.000 99.868 99.434 98.614
76 100.000 100.000 99.868 99.434 98.615
77 100.000 100.000 99.882 99.477 98.694
78 100.000 100.000 99.882 99.487 98.730
79 100.000 100.000 99.882 99.493 98.751
80 100.000 100.000 99.882 99.499 98.773
81 100.000 100.000 99.882 99.507 98.801
82 100.000 100.000 99.912 99.590 98.953
83 100.000 100.000 99.912 99.594 98.967
84 100.000 100.000 99.912 99.594 98.968
85 100.000 100.000 99.912 99.598 98.983
86 100.000 100.000 99.912 99.606 99.011
87 100.000 100.000 99.912 99.613 99.033
88 100.000 100.000 99.912 99.613 99.034
89 100.000 100.000 99.912 99.623 99.069
90 100.000 100.000 99.941 99.706 99.220
91 100.000 100.000 99.956 99.753 99.314
92 100.000 100.000 99.956 99.753 99.314
93 100.000 100.000 99.956 99.757 99.329
94 100.000 100.000 99.956 99.761 99.344
95 100.000 100.000 99.956 99.769 99.373
96 100.000 100.000 99.956 99.775 99.394
97 100.000 100.000 99.985 99.858 99.545
98 100.000 100.000 99.985 99.858 99.546
99 100.000 100.000 99.985 99.858 99.547
100 100.000 100.000 99.985 99.864 99.568
101 100.000 100.000 99.985 99.870 99.590
102 100.000 100.000 100.000 99.913 99.669
103 100.000 100.000 100.000 99.919 99.691
104 100.000 100.000 100.000 99.919 99.691
105 100.000 100.000 100.000 99.919 99.692
106 100.000 100.000 100.000 99.921 99.700
107 100.000 100.000 100.000 99.925 99.714
108 100.000 100.000 100.000 99.927 99.722
109 100.000 100.000 100.000 99.931 99.737
110 100.000 100.000 100.000 99.935 99.751
111 100.000 100.000 100.000 99.943 99.780
112 100.000 100.000 100.000 99.945 99.788
113 100.000 100.000 100.000 99.949 99.802
114 100.000 100.000 100.000 99.951 99.810
115 100.000 100.000 100.000 99.955 99.825
116 100.000 100.000 100.000 99.955 99.825
117 100.000 100.000 100.000 99.959 99.840
118 100.000 100.000 100.000 99.959 99.841
119 100.000 100.000 100.000 99.963 99.856
120 100.000 100.000 100.000 99.968 99.870
121 100.000 100.000 100.000 99.976 99.899
122 100.000 100.000 100.000 99.980 99.914
123 100.000 100.000 100.000 99.984 99.929
124 100.000 100.000 100.000 99.984 99.929
125 100.000 100.000 100.000 99.988 99.944
126 100.000 100.000 100.000 99.988 99.945
127 100.000 100.000 100.000 99.988 99.946
128 100.000 100.000 100.000 99.992 99.960
129 100.000 100.000 100.000 99.992 99.961
130 100.000 100.000 100.000 99.992 99.962
131 100.000 100.000 100.000 99.992 99.963
132 100.000 100.000 100.000 99.994 99.970
133 100.000 100.000 100.000 99.994 99.971
134 100.000 100.000 100.000 99.994 99.972
135 100.000 100.000 100.000 99.998 99.986
136 100.000 100.000 100.000 99.998 99.987
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.559 69.501
11 100.000 94.624 87.338 79.511 71.739
12 100.000 95.392 88.807 81.478 74.025
13 100.000 96.160 90.276 83.428 76.257
14 100.000 96.928 91.745 85.390 78.525
15 100.000 97.696 93.214 87.335 80.745
16 100.000 98.464 94.683 89.283 82.970
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.421 87.851
20 100.000 100.000 97.797 93.663 88.276
21 100.000 100.000 97.885 93.900 88.688
22 100.000 100.000 97.973 94.137 89.098
23 100.000 100.000 98.061 94.379 89.523
24 100.000 100.000 98.149 94.616 89.934
25 100.000 100.000 98.237 94.853 90.346
26 100.000 100.000 98.325 95.091 90.757
27 100.000 100.000 98.414 95.328 91.167
28 100.000 100.000 98.502 95.569 91.592
29 100.000 100.000 98.590 95.807 92.002
30 100.000 100.000 98.678 96.044 92.413
31 100.000 100.000 98.766 96.286 92.838
32 100.000 100.000 98.854 96.523 93.248
33 100.000 100.000 98.942 96.764 93.672
34 100.000 100.000 99.031 97.002 94.082
35 100.000 100.000 99.119 97.239 94.492
36 100.000 100.000 99.207 97.476 94.902
37 100.000 100.000 99.266 97.639 95.190
38 100.000 100.000 99.324 97.797 95.464
39 100.000 100.000 99.383 97.959 95.752
40 100.000 100.000 99.442 98.121 96.039
41 100.000 100.000 99.486 98.247 96.269
42 100.000 100.000 99.530 98.373 96.499
43 100.000 100.000 99.574 98.499 96.728
44 100.000 100.000 99.618 98.625 96.958
45 100.000 100.000 99.662 98.750 97.187
46 100.000 100.000 99.706 98.876 97.417
47 100.000 100.000 99.750 99.002 97.647
48 100.000 100.000 99.794 99.128 97.876
49 100.000 100.000 99.824 99.211 98.028
50 100.000 100.000 99.853 99.298 98.193
51 100.000 100.000 99.882 99.385 98.359
52 100.000 100.000 99.912 99.468 98.510
53 100.000 100.000 99.941 99.556 98.675
54 100.000 100.000 99.971 99.639 98.826
55 100.000 100.000 99.985 99.686 98.920
56 100.000 100.000 100.000 99.732 99.013
57 100.000 100.000 100.000 99.736 99.028
58 100.000 100.000 100.000 99.738 99.036
59 100.000 100.000 100.000 99.740 99.044
60 100.000 100.000 100.000 99.742 99.052
61 100.000 100.000 100.000 99.757 99.101
62 100.000 100.000 100.000 99.767 99.137
63 100.000 100.000 100.000 99.767 99.138
64 100.000 100.000 100.000 99.771 99.152
65 100.000 100.000 100.000 99.773 99.160
66 100.000 100.000 100.000 99.777 99.175
67 100.000 100.000 100.000 99.783 99.197
68 100.000 100.000 100.000 99.797 99.246
69 100.000 100.000 100.000 99.801 99.261
70 100.000 100.000 100.000 99.805 99.276
71 100.000 100.000 100.000 99.805 99.277
72 100.000 100.000 100.000 99.813 99.305
73 100.000 100.000 100.000 99.817 99.320
74 100.000 100.000 100.000 99.821 99.335
75 100.000 100.000 100.000 99.821 99.336
76 100.000 100.000 100.000 99.826 99.350
77 100.000 100.000 100.000 99.832 99.372
78 100.000 100.000 100.000 99.834 99.379
79 100.000 100.000 100.000 99.838 99.394
80 100.000 100.000 100.000 99.840 99.402
81 100.000 100.000 100.000 99.840 99.402
82 100.000 100.000 100.000 99.844 99.417
83 100.000 100.000 100.000 99.844 99.418
84 100.000 100.000 100.000 99.848 99.433
85 100.000 100.000 100.000 99.856 99.461
86 100.000 100.000 100.000 99.860 99.476
87 100.000 100.000 100.000 99.862 99.484
88 100.000 100.000 100.000 99.870 99.512
89 100.000 100.000 100.000 99.878 99.541
90 100.000 100.000 100.000 99.878 99.542
91 100.000 100.000 100.000 99.886 99.571
92 100.000 100.000 100.000 99.890 99.585
93 100.000 100.000 100.000 99.899 99.614
94 100.000 100.000 100.000 99.903 99.629
95 100.000 100.000 100.000 99.907 99.643
96 100.000 100.000 100.000 99.907 99.644
97 100.000 100.000 100.000 99.907 99.645
98 100.000 100.000 100.000 99.913 99.666
99 100.000 100.000 100.000 99.913 99.667
100 100.000 100.000 100.000 99.923 99.703
101 100.000 100.000 100.000 99.929 99.724
102 100.000 100.000 100.000 99.935 99.746
103 100.000 100.000 100.000 99.941 99.767
104 100.000 100.000 100.000 99.947 99.789
105 100.000 100.000 100.000 99.955 99.818
106 100.000 100.000 100.000 99.955 99.818
107 100.000 100.000 100.000 99.963 99.847
108 100.000 100.000 100.000 99.963 99.848
109 100.000 100.000 100.000 99.963 99.849
110 100.000 100.000 100.000 99.966 99.856
111 100.000 100.000 100.000 99.968 99.864
112 100.000 100.000 100.000 99.970 99.871
113 100.000 100.000 100.000 99.970 99.872
114 100.000 100.000 100.000 99.972 99.880
115 100.000 100.000 100.000 99.972 99.880
116 100.000 100.000 100.000 99.976 99.895
117 100.000 100.000 100.000 99.980 99.910
118 100.000 100.000 100.000 99.980 99.910
119 100.000 100.000 100.000 99.980 99.911
120 100.000 100.000 100.000 99.980 99.912
121 100.000 100.000 100.000 99.980 99.913
122 100.000 100.000 100.000 99.988 99.941
123 100.000 100.000 100.000 99.988 99.942
124 100.000 100.000 100.000 99.988 99.943
125 100.000 100.000 100.000 99.992 99.958
126 100.000 100.000 100.000 99.992 99.958
127 100.000 100.000 100.000 99.992 99.959
128 100.000 100.000 100.000 99.992 99.960
129 100.000 100.000 100.000 99.992 99.961
130 100.000 100.000 100.000 99.996 99.975
131 100.000 100.000 100.000 99.996 99.976
132 100.000 100.000 100.000 99.996 99.977
133 100.000 100.000 100.000 99.996 99.978
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.563 69.515
11 100.000 94.624 87.338 79.525 71.783
12 100.000 95.392 88.807 81.487 74.051
13 100.000 96.160 90.276 83.438 76.289
14 100.000 96.928 91.745 85.390 78.526
15 100.000 97.696 93.214 87.341 80.762
16 100.000 98.464 94.683 89.289 82.988
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.425 87.865
20 100.000 100.000 97.797 93.667 88.291
21 100.000 100.000 97.885 93.908 88.717
22 100.000 100.000 97.973 94.149 89.143
23 100.000 100.000 98.061 94.391 89.569
24 100.000 100.000 98.149 94.632 89.995
25 100.000 100.000 98.237 94.870 90.406
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.226
28 100.000 100.000 98.502 95.582 91.636
29 100.000 100.000 98.590 95.819 92.046
30 100.000 100.000 98.678 96.056 92.456
31 100.000 100.000 98.766 96.294 92.866
32 100.000 100.000 98.854 96.531 93.277
33 100.000 100.000 98.942 96.768 93.687
34 100.000 100.000 99.031 97.006 94.097
35 100.000 100.000 99.119 97.243 94.507
36 100.000 100.000 99.207 97.480 94.917
37 100.000 100.000 99.266 97.643 95.205
38 100.000 100.000 99.324 97.805 95.492
39 100.000 100.000 99.383 97.967 95.780
40 100.000 100.000 99.442 98.130 96.067
41 100.000 100.000 99.501 98.292 96.355
42 100.000 100.000 99.545 98.418 96.584
43 100.000 100.000 99.589 98.543 96.814
44 100.000 100.000 99.633 98.669 97.044
45 100.000 100.000 99.677 98.795 97.273
46 100.000 100.000 99.706 98.882 97.438
47 100.000 100.000 99.736 98.969 97.603
48 100.000 100.000 99.765 99.057 97.768
49 100.000 100.000 99.794 99.144 97.934
50 100.000 100.000 99.824 99.231 98.099
51 100.000 100.000 99.853 99.314 98.250
52 100.000 100.000 99.882 99.397 98.401
53 100.000 100.000 99.912 99.481 98.553
54 100.000 100.000 99.941 99.564 98.704
55 100.000 100.000 99.971 99.647 98.855
56 100.000 100.000 99.985 99.694 98.948
57 100.000 100.000 100.000 99.740 99.041
58 100.000 100.000 100.000 99.755 99.091
59 100.000 100.000 100.000 99.769 99.140
60 100.000 100.000 100.000 99.783 99.190
61 100.000 100.000 100.000 99.797 99.239
62 100.000 100.000 100.000 99.807 99.275
63 100.000 100.000 100.000 99.817 99.310
64 100.000 100.000 100.000 99.828 99.346
65 100.000 100.000 100.000 99.838 99.381
66 100.000 100.000 100.000 99.848 99.417
67 100.000 100.000 100.000 99.858 99.452
68 100.000 100.000 100.000 99.868 99.488
69 100.000 100.000 100.000 99.878 99.523
70 100.000 100.000 100.000 99.888 99.559
71 100.000 100.000 100.000 99.899 99.594
72 100.000 100.000 100.000 99.909 99.630
73 100.000 100.000 100.000 99.919 99.665
74 100.000 100.000 100.000 99.929 99.701
75 100.000 100.000 100.000 99.939 99.736
76 100.000 100.000 100.000 99.949 99.773
77 100.000 100.000 100.000 99.959 99.808
78 100.000 100.000 100.000 99.970 99.844
79 100.000 100.000 100.000 99.980 99.879
80 100.000 100.000 100.000 99.988 99.908
81 100.000 100.000 100.000 99.994 99.929
82 100.000 100.000 100.000 100.000 99.951
83 100.000 100.000 100.000 100.000 99.952
84 100.000 100.000 100.000 100.000 99.952
85 100.000 100.000 100.000 100.000 99.954
86 100.000 100.000 100.000 100.000 99.955
87 100.000 100.000 100.000 100.000 99.955
88 100.000 100.000 100.000 100.000 99.956
89 100.000 100.000 100.000 100.000 99.957
90 100.000 100.000 100.000 100.000 99.958
91 100.000 100.000 100.000 100.000 99.959
92 100.000 100.000 100.000 100.000 99.960
93 100.000 100.000 100.000 100.000 99.961
94 100.000 100.000 100.000 100.000 99.961
95 100.000 100.000 100.000 100.000 99.962
96 100.000 100.000 100.000 100.000 99.963
97 100.000 100.000 100.000 100.000 99.964
98 100.000 100.000 100.000 100.000 99.964
99 100.000 100.000 100.000 100.000 99.965
100 100.000 100.000 100.000 100.000 99.967
101 100.000 100.000 100.000 100.000 99.968
102 100.000 100.000 100.000 100.000 99.968
103 100.000 100.000 100.000 100.000 99.969
104 100.000 100.000 100.000 100.000 99.970
105 100.000 100.000 100.000 100.000 99.971
106 100.000 100.000 100.000 100.000 99.971
107 100.000 100.000 100.000 100.000 99.972
108 100.000 100.000 100.000 100.000 99.973
109 100.000 100.000 100.000 100.000 99.974
110 100.000 100.000 100.000 100.000 99.974
111 100.000 100.000 100.000 100.000 99.975
112 100.000 100.000 100.000 100.000 99.976
113 100.000 100.000 100.000 100.000 99.977
114 100.000 100.000 100.000 100.000 99.977
115 100.000 100.000 100.000 100.000 99.978
116 100.000 100.000 100.000 100.000 99.979
117 100.000 100.000 100.000 100.000 99.980
118 100.000 100.000 100.000 100.000 99.980
119 100.000 100.000 100.000 100.000 99.981
120 100.000 100.000 100.000 100.000 99.982
121 100.000 100.000 100.000 100.000 99.983
122 100.000 100.000 100.000 100.000 99.983
123 100.000 100.000 100.000 100.000 99.984
124 100.000 100.000 100.000 100.000 99.985
125 100.000 100.000 100.000 100.000 99.986
126 100.000 100.000 100.000 100.000 99.986
127 100.000 100.000 100.000 100.000 99.987
128 100.000 100.000 100.000 100.000 99.988
129 100.000 100.000 100.000 100.000 99.989
130 100.000 100.000 100.000 100.000 99.989
131 100.000 100.000 100.000 100.000 99.990
132 100.000 100.000 100.000 100.000 99.991
133 100.000 100.000 100.000 100.000 99.992
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.563 69.515
11 100.000 94.624 87.338 79.525 71.783
12 100.000 95.392 88.807 81.487 74.051
13 100.000 96.160 90.276 83.438 76.289
14 100.000 96.928 91.745 85.390 78.526
15 100.000 97.696 93.214 87.339 80.758
16 100.000 98.464 94.683 89.289 82.989
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.425 87.865
20 100.000 100.000 97.797 93.667 88.291
21 100.000 100.000 97.885 93.908 88.717
22 100.000 100.000 97.973 94.149 89.143
23 100.000 100.000 98.061 94.391 89.570
24 100.000 100.000 98.149 94.632 89.996
25 100.000 100.000 98.237 94.870 90.406
26 100.000 100.000 98.325 95.107 90.817
27 100.000 100.000 98.414 95.344 91.227
28 100.000 100.000 98.502 95.582 91.638
29 100.000 100.000 98.590 95.819 92.049
30 100.000 100.000 98.678 96.056 92.459
31 100.000 100.000 98.766 96.294 92.870
32 100.000 100.000 98.854 96.531 93.280
33 100.000 100.000 98.942 96.768 93.691
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.266 97.643 95.209
38 100.000 100.000 99.324 97.805 95.496
39 100.000 100.000 99.383 97.967 95.783
40 100.000 100.000 99.442 98.130 96.071
41 100.000 100.000 99.486 98.255 96.301
42 100.000 100.000 99.530 98.381 96.530
43 100.000 100.000 99.559 98.468 96.695
44 100.000 100.000 99.589 98.556 96.860
45 100.000 100.000 99.618 98.643 97.025
46 100.000 100.000 99.647 98.730 97.190
47 100.000 100.000 99.677 98.813 97.342
48 100.000 100.000 99.706 98.896 97.493
49 100.000 100.000 99.736 98.980 97.644
50 100.000 100.000 99.765 99.063 97.795
51 100.000 100.000 99.794 99.146 97.946
52 100.000 100.000 99.824 99.229 98.097
53 100.000 100.000 99.853 99.312 98.248
54 100.000 100.000 99.882 99.395 98.399
55 100.000 100.000 99.912 99.479 98.550
56 100.000 100.000 99.927 99.525 98.644
57 100.000 100.000 99.941 99.572 98.737
58 100.000 100.000 99.956 99.619 98.830
59 100.000 100.000 99.971 99.665 98.923
60 100.000 100.000 99.985 99.712 99.017
61 100.000 100.000 100.000 99.759 99.110
62 100.000 100.000 100.000 99.773 99.160
63 100.000 100.000 100.000 99.787 99.209
64 100.000 100.000 100.000 99.797 99.245
65 100.000 100.000 100.000 99.807 99.280
66 100.000 100.000 100.000 99.817 99.316
67 100.000 100.000 100.000 99.828 99.351
68 100.000 100.000 100.000 99.838 99.387
69 100.000 100.000 100.000 99.848 99.422
70 100.000 100.000 100.000 99.858 99.458
71 100.000 100.000 100.000 99.868 99.493
72 100.000 100.000 100.000 99.878 99.529
73 100.000 100.000 100.000 99.888 99.564
74 100.000 100.000 100.000 99.899 99.600
75 100.000 100.000 100.000 99.909 99.635
76 100.000 100.000 100.000 99.919 99.671
77 100.000 100.000 100.000 99.929 99.706
78 100.000 100.000 100.000 99.939 99.742
79 100.000 100.000 100.000 99.947 99.770
80 100.000 100.000 100.000 99.953 99.792
81 100.000 100.000 100.000 99.959 99.813
82 100.000 100.000 100.000 99.966 99.835
83 100.000 100.000 100.000 99.972 99.856
84 100.000 100.000 100.000 99.978 99.878
85 100.000 100.000 100.000 99.984 99.900
86 100.000 100.000 100.000 99.988 99.914
87 100.000 100.000 100.000 99.992 99.929
88 100.000 100.000 100.000 99.996 99.944
89 100.000 100.000 100.000 100.000 99.958
90 100.000 100.000 100.000 100.000 99.959
91 100.000 100.000 100.000 100.000 99.960
92 100.000 100.000 100.000 100.000 99.961
93 100.000 100.000 100.000 100.000 99.961
94 100.000 100.000 100.000 100.000 99.962
95 100.000 100.000 100.000 100.000 99.963
96 100.000 100.000 100.000 100.000 99.964
97 100.000 100.000 100.000 100.000 99.964
98 100.000 100.000 100.000 100.000 99.965
99 100.000 100.000 100.000 100.000 99.966
100 100.000 100.000 100.000 100.000 99.967
101 100.000 100.000 100.000 100.000 99.968
102 100.000 100.000 100.000 100.000 99.968
103 100.000 100.000 100.000 100.000 99.969
104 100.000 100.000 100.000 100.000 99.970
105 100.000 100.000 100.000 100.000 99.971
106 100.000 100.000 100.000 100.000 99.971
107 100.000 100.000 100.000 100.000 99.972
108 100.000 100.000 100.000 100.000 99.973
109 100.000 100.000 100.000 100.000 99.974
110 100.000 100.000 100.000 100.000 99.974
111 100.000 100.000 100.000 100.000 99.975
112 100.000 100.000 100.000 100.000 99.976
113 100.000 100.000 100.000 100.000 99.977
114 100.000 100.000 100.000 100.000 99.977
115 100.000 100.000 100.000 100.000 99.978
116 100.000 100.000 100.000 100.000 99.979
117 100.000 100.000 100.000 100.000 99.980
118 100.000 100.000 100.000 100.000 99.980
119 100.000 100.000 100.000 100.000 99.981
120 100.000 100.000 100.000 100.000 99.982
121 100.000 100.000 100.000 100.000 99.983
122 100.000 100.000 100.000 100.000 99.983
123 100.000 100.000 100.000 100.000 99.984
124 100.000 100.000 100.000 100.000 99.985
125 100.000 100.000 100.000 100.000 99.986
126 100.000 100.000 100.000 100.000 99.986
127 100.000 100.000 100.000 100.000 99.987
128 100.000 100.000 100.000 100.000 99.988
129 100.000 100.000 100.000 100.000 99.989
130 100.000 100.000 100.000 100.000 99.989
131 100.000 100.000 100.000 100.000 99.990
132 100.000 100.000 100.000 100.000 99.991
133 100.000 100.000 100.000 100.000 99.992
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 57.500 31.395 18.105 11.144 7.277
3 70.000 42.359 25.843 16.375 10.836
4 75.000 49.668 32.050 20.986 14.150
5 82.500 56.478 37.430 25.080 17.220
6 87.500 61.960 41.982 28.594 19.891
7 90.000 66.279 46.182 32.097 22.655
8 92.500 69.601 49.348 34.818 24.901
9 95.000 72.591 52.142 37.237 26.933
10 97.500 75.415 54.852 39.656 29.000
11 100.000 78.239 57.438 41.889 30.886
12 100.000 79.734 59.756 44.345 33.123
13 100.000 81.063 61.577 46.189 34.801
14 100.000 82.392 63.439 48.112 36.577
15 100.000 83.555 65.053 49.770 38.109
16 100.000 84.551 66.377 51.090 39.308
17 100.000 85.548 67.763 52.520 40.640
18 100.000 86.545 69.150 53.951 41.971
19 100.000 87.375 70.536 55.531 43.514
20 100.000 88.206 71.674 56.689 44.578
21 100.000 89.037 72.812 57.846 45.642
22 100.000 89.701 73.846 59.013 46.803
23 100.000 90.365 74.798 60.022 47.765
24 100.000 91.030 75.874 61.266 49.024
25 100.000 91.528 76.619 62.089 49.840
26 100.000 92.027 77.488 63.152 50.964
27 100.000 92.525 78.233 63.975 51.779
28 100.000 93.023 79.019 64.868 52.673
29 100.000 93.522 79.806 65.761 53.567
30 100.000 94.020 80.633 66.741 54.582
31 100.000 94.352 81.233 67.498 55.408
32 100.000 94.684 81.833 68.242 56.199
33 100.000 95.017 82.351 68.838 56.814
34 100.000 95.349 82.909 69.512 57.528
35 100.000 95.681 83.468 70.194 58.265
36 100.000 96.013 84.026 70.868 58.979
37 100.000 96.346 84.606 71.592 59.770
38 100.000 96.678 85.164 72.266 60.484
39 100.000 96.844 85.599 72.894 61.207
40 100.000 97.010 85.992 73.452 61.853
41 100.000 97.176 86.261 73.792 62.245
42 100.000 97.342 86.530 74.114 62.592
43 100.000 97.508 86.944 74.722 63.316
44 100.000 97.674 87.378 75.350 64.039
45 100.000 97.841 87.772 75.921 64.718
46 100.000 98.007 88.041 76.244 65.065
47 100.000 98.173 88.434 76.802 65.711
48 100.000 98.339 88.744 77.224 66.213
49 100.000 98.505 89.075 77.670 66.726
50 100.000 98.671 89.427 78.158 67.294
51 100.000 98.837 89.758 78.605 67.807
52 100.000 99.003 90.068 79.014 68.276
53 100.000 99.169 90.379 79.424 68.746
54 100.000 99.336 90.648 79.763 69.137
55 100.000 99.502 91.082 80.408 69.905
56 100.000 99.668 91.393 80.817 70.374
57 100.000 99.834 91.724 81.264 70.887
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 80.000 55.316 37.016 24.964 17.203
6 85.000 60.465 41.548 28.677 20.122
7 87.500 64.452 45.396 31.915 22.716
8 92.500 69.269 49.328 35.016 25.149
9 95.000 72.924 52.866 38.035 27.599
10 97.500 75.914 55.659 40.454 29.631
11 100.000 78.738 58.225 42.646 31.462
12 100.000 80.066 60.087 44.568 33.238
13 100.000 81.395 61.949 46.491 35.015
14 100.000 82.392 63.542 48.274 36.743
15 100.000 83.389 65.032 49.899 38.315
16 100.000 84.385 66.418 51.329 39.646
17 100.000 85.382 67.805 52.760 40.978
18 100.000 86.213 69.088 54.199 42.404
19 100.000 87.209 70.432 55.560 43.657
20 100.000 88.040 71.612 56.804 44.843
21 100.000 88.870 72.750 57.962 45.907
22 100.000 89.701 73.888 59.120 46.971
23 100.000 90.365 74.922 60.286 48.132
24 100.000 91.030 75.936 61.423 49.270
25 100.000 91.694 76.929 62.519 50.353
26 100.000 92.359 77.881 63.528 51.315
27 100.000 92.857 78.709 64.508 52.330
28 100.000 93.355 79.495 65.401 53.224
29 100.000 93.854 80.240 66.224 54.040
30 100.000 94.020 80.799 67.080 55.043
31 100.000 94.352 81.419 67.874 55.912
32 100.000 94.684 82.019 68.618 56.704
33 100.000 95.017 82.599 69.338 57.484
34 100.000 95.349 83.178 70.045 58.232
35 100.000 95.681 83.737 70.719 58.946
36 100.000 95.847 84.171 71.348 59.669
37 100.000 96.013 84.606 71.976 60.393
38 100.000 96.346 85.123 72.572 61.008
39 100.000 96.678 85.640 73.167 61.623
40 100.000 97.010 86.158 73.763 62.238
41 100.000 97.176 86.571 74.366 62.951
42 100.000 97.342 86.965 74.925 63.597
43 100.000 97.508 87.337 75.462 64.242
44 100.000 97.674 87.709 76.004 64.898
45 100.000 97.841 88.061 76.492 65.465
46 100.000 98.007 88.413 76.971 66.013
47 100.000 98.173 88.765 77.459 66.580
48 100.000 98.339 89.096 77.906 67.093
49 100.000 98.505 89.427 78.353 67.607
50 100.000 98.671 89.737 78.762 68.076
51 100.000 98.837 90.048 79.171 68.545
52 100.000 99.003 90.358 79.585 69.025
53 100.000 99.169 90.668 79.994 69.495
54 100.000 99.336 90.958 80.371 69.933
55 100.000 99.502 91.227 80.693 70.280
56 100.000 99.668 91.496 81.016 70.628
57 100.000 99.834 91.765 81.338 70.976
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 77.500 54.319 36.706 24.881 17.185
6 82.500 59.468 41.237 28.594 20.103
7 85.000 63.289 44.920 31.700 22.596
8 90.000 67.774 48.624 34.723 25.040
9 95.000 72.591 52.555 37.824 27.473
10 97.500 75.415 55.204 40.165 29.480
11 100.000 78.073 57.645 42.319 31.342
12 100.000 79.070 59.197 44.039 33.007
13 100.000 80.233 60.852 45.776 34.638
14 100.000 81.561 62.632 47.550 36.238
15 100.000 82.724 64.246 49.208 37.770
16 100.000 83.721 65.673 50.717 39.200
17 100.000 84.718 67.080 52.190 40.586
18 100.000 85.714 68.487 53.662 41.972
19 100.000 86.545 69.708 54.985 43.256
20 100.000 87.209 70.764 56.192 44.472
21 100.000 88.206 72.088 57.511 45.671
22 100.000 89.037 73.246 58.715 46.802
23 100.000 89.867 74.384 59.873 47.866
24 100.000 90.698 75.522 61.030 48.929
25 100.000 91.196 76.350 62.019 49.965
26 100.000 91.528 77.033 62.924 50.967
27 100.000 92.027 77.840 63.871 51.948
28 100.000 92.691 78.792 64.880 52.910
29 100.000 93.189 79.578 65.790 53.848
30 100.000 93.522 80.219 66.621 54.761
31 100.000 94.020 81.006 67.514 55.654
32 100.000 94.352 81.626 68.304 56.513
33 100.000 94.850 82.371 69.127 57.329
34 100.000 95.183 82.930 69.818 58.086
35 100.000 95.349 83.364 70.446 58.810
36 100.000 95.515 83.799 71.075 59.533
37 100.000 95.681 84.192 71.645 60.212
38 100.000 95.847 84.585 72.204 60.857
39 100.000 96.013 84.978 72.762 61.503
40 100.000 96.179 85.371 73.320 62.148
41 100.000 96.512 85.889 73.916 62.764
42 100.000 96.844 86.406 74.511 63.379
43 100.000 97.176 86.923 75.106 63.994
44 100.000 97.508 87.441 75.702 64.609
45 100.000 97.841 87.958 76.297 65.225
46 100.000 98.007 88.310 76.789 65.805
47 100.000 98.173 88.661 77.277 66.372
48 100.000 98.339 89.013 77.765 66.940
49 100.000 98.505 89.365 78.245 67.487
50 100.000 98.671 89.696 78.692 68.000
51 100.000 98.837 90.027 79.138 68.514
52 100.000 99.003 90.337 79.552 68.994
53 100.000 99.169 90.648 79.961 69.463
54 100.000 99.336 90.958 80.371 69.933
55 100.000 99.502 91.227 80.710 70.324
56 100.000 99.668 91.496 81.032 70.672
57 100.000 99.834 91.765 81.355 71.020
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 70.000 41.694 25.347 16.123 10.733
4 75.000 48.505 31.181 20.543 13.968
5 75.000 52.658 35.837 24.472 17.008
6 77.500 56.312 39.623 27.838 19.773
7 85.000 62.791 44.527 31.506 22.539
8 90.000 67.608 48.500 34.677 25.050
9 92.500 70.764 51.500 37.282 27.227
10 95.000 73.588 54.149 39.623 29.235
11 95.000 75.083 56.218 41.732 31.157
12 97.500 77.741 58.680 43.923 33.062
13 97.500 79.070 60.501 45.776 34.761
14 100.000 81.728 62.880 47.823 36.501
15 100.000 82.724 64.349 49.411 38.029
16 100.000 83.721 65.798 50.961 39.514
17 100.000 84.551 67.101 52.417 40.935
18 100.000 85.548 68.529 53.926 42.365
19 100.000 86.545 69.915 55.357 43.696
20 100.000 87.209 70.929 56.494 44.834
21 100.000 88.040 72.108 57.739 46.020
22 100.000 88.704 73.102 58.835 47.103
23 100.000 89.037 73.743 59.666 48.016
24 100.000 89.535 74.529 60.576 48.954
25 100.000 90.365 75.667 61.733 50.018
26 100.000 91.196 76.805 62.891 51.082
27 100.000 91.694 77.571 63.760 51.964
28 100.000 92.193 78.336 64.628 52.847
29 100.000 92.857 79.288 65.637 53.809
30 100.000 93.355 80.074 66.530 54.702
31 100.000 93.854 80.861 67.423 55.596
32 100.000 94.186 81.440 68.143 56.377
33 100.000 94.518 82.019 68.862 57.157
34 100.000 94.850 82.619 69.607 57.949
35 100.000 95.017 83.054 70.235 58.672
36 100.000 95.183 83.426 70.773 59.318
37 100.000 95.515 84.006 71.480 60.066
38 100.000 95.847 84.564 72.154 60.779
39 100.000 96.013 84.958 72.712 61.425
40 100.000 96.179 85.351 73.270 62.071
41 100.000 96.346 85.744 73.829 62.716
42 100.000 96.512 86.096 74.317 63.284
43 100.000 96.844 86.613 74.912 63.899
44 100.000 97.176 87.130 75.508 64.515
45 100.000 97.508 87.647 76.103 65.130
46 100.000 97.841 88.165 76.699 65.745
47 100.000 98.173 88.682 77.294 66.360
48 100.000 98.339 89.013 77.749 66.894
49 100.000 98.505 89.365 78.229 67.441
50 100.000 98.671 89.696 78.675 67.954
51 100.000 98.837 90.006 79.089 68.435
52 100.000 99.003 90.317 79.498 68.904
53 100.000 99.169 90.627 79.907 69.373
54 100.000 99.336 90.937 80.317 69.843
55 100.000 99.502 91.227 80.693 70.280
56 100.000 99.668 91.496 81.032 70.672
57 100.000 99.834 91.765 81.355 71.020
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 85.000 56.146 36.478 24.286 16.699
6 85.000 60.299 41.134 28.214 19.739
7 90.000 65.449 45.665 31.927 22.657
8 92.500 69.435 49.514 35.165 25.251
9 95.000 72.757 52.700 37.919 27.531
10 97.500 75.748 55.493 40.338 29.563
11 100.000 78.405 57.914 42.460 31.390
12 100.000 79.734 59.735 44.304 33.068
13 100.000 81.063 61.556 46.148 34.746
14 100.000 82.392 63.418 48.071 36.523
15 100.000 83.555 65.156 49.915 38.236
16 100.000 84.718 66.770 51.573 39.768
17 100.000 85.714 68.156 53.004 41.099
18 100.000 86.711 69.584 54.514 42.529
19 100.000 87.542 70.722 55.671 43.593
20 100.000 88.372 72.026 57.143 45.053
21 100.000 89.203 73.164 58.301 46.117
22 100.000 90.033 74.302 59.459 47.181
23 100.000 90.698 75.336 60.634 48.363
24 100.000 91.362 76.371 61.800 49.524
25 100.000 92.027 77.364 62.895 50.607
26 100.000 92.525 78.274 64.024 51.799
27 100.000 93.023 79.040 64.893 52.682
28 100.000 93.522 79.826 65.802 53.619
29 100.000 94.020 80.612 66.696 54.513
30 100.000 94.352 81.212 67.448 55.327
31 100.000 94.684 81.730 68.044 55.943
32 100.000 95.017 82.288 68.718 56.656
33 100.000 95.349 82.888 69.462 57.448
34 100.000 95.681 83.489 70.215 58.263
35 100.000 96.013 84.047 70.897 58.999
36 100.000 96.346 84.564 71.492 59.615
37 100.000 96.512 84.833 71.823 59.985
38 100.000 96.678 85.206 72.357 60.620
39 100.000 96.844 85.496 72.725 61.035
40 100.000 97.010 85.930 73.370 61.802
41 100.000 97.176 86.323 73.945 62.491
42 100.000 97.342 86.716 74.503 63.137
43 100.000 97.508 87.109 75.061 63.783
44 100.000 97.674 87.399 75.429 64.197
45 100.000 97.841 87.813 76.041 64.930
46 100.000 98.007 88.185 76.566 65.542
47 100.000 98.173 88.496 76.984 66.032
48 100.000 98.339 88.827 77.430 66.545
49 100.000 98.505 89.179 77.918 67.113
50 100.000 98.671 89.448 78.257 67.504
51 100.000 98.837 89.758 78.667 67.974
52 100.000 99.003 90.068 79.076 68.443
53 100.000 99.169 90.420 79.564 69.011
54 100.000 99.336 90.772 80.052 69.579
55 100.000 99.502 91.165 80.623 70.257
56 100.000 99.668 91.517 81.111 70.825
57 100.000 99.834 91.786 81.433 71.173
58 100.000 100.000 92.055 81.764 71.544
59 100.000 100.000 92.158 81.987 71.867
60 100.000 100.000 92.220 82.124 72.069
61 100.000 100.000 92.262 82.219 72.216
62 100.000 100.000 92.882 83.273 73.432
63 100.000 100.000 93.503 84.328 74.648
64 100.000 100.000 93.710 84.745 75.214
65 100.000 100.000 93.834 85.006 75.581
66 100.000 100.000 93.896 85.142 75.783
67 100.000 100.000 94.103 85.535 76.288
68 100.000 100.000 94.269 85.883 76.777
69 100.000 100.000 94.413 86.185 77.199
70 100.000 100.000 94.455 86.280 77.346
71 100.000 100.000 94.682 86.722 77.923
72 100.000 100.000 94.765 86.904 78.192
73 100.000 100.000 94.807 86.999 78.339
74 100.000 100.000 94.972 87.346 78.828
75 100.000 100.000 95.034 87.483 79.030
76 100.000 100.000 95.076 87.578 79.177
77 100.000 100.000 95.241 87.909 79.622
78 100.000 100.000 95.303 88.045 79.824
79 100.000 100.000 95.345 88.140 79.971
80 100.000 100.000 95.448 88.368 80.304
81 100.000 100.000 95.593 88.670 80.726
82 100.000 100.000 95.634 88.773 80.896
83 100.000 100.000 95.717 88.947 81.142
84 100.000 100.000 95.820 89.187 81.509
85 100.000 100.000 95.903 89.369 81.778
86 100.000 100.000 95.924 89.422 81.870
87 100.000 100.000 95.986 89.567 82.095
88 100.000 100.000 96.027 89.662 82.242
89 100.000 100.000 96.069 89.757 82.389
90 100.000 100.000 96.131 89.894 82.591
91 100.000 100.000 96.172 89.993 82.750
92 100.000 100.000 96.234 90.134 82.964
93 100.000 100.000 96.338 90.353 83.277
94 100.000 100.000 96.379 90.448 83.424
95 100.000 100.000 96.420 90.543 83.571
96 100.000 100.000 96.462 90.638 83.718
97 100.000 100.000 96.524 90.787 83.955
98 100.000 100.000 96.586 90.923 84.157
99 100.000 100.000 96.648 91.060 84.359
100 100.000 100.000 96.689 91.155 84.506
101 100.000 100.000 96.710 91.205 84.586
102 100.000 100.000 96.814 91.428 84.910
103 100.000 100.000 96.855 91.523 85.057
104 100.000 100.000 96.917 91.668 85.282
105 100.000 100.000 96.938 91.721 85.374
106 100.000 100.000 96.979 91.817 85.521
107 100.000 100.000 97.103 92.077 85.889
108 100.000 100.000 97.145 92.172 86.036
109 100.000 100.000 97.186 92.267 86.183
110 100.000 100.000 97.289 92.491 86.506
111 100.000 100.000 97.352 92.631 86.720
112 100.000 100.000 97.414 92.784 86.966
113 100.000 100.000 97.434 92.838 87.058
114 100.000 100.000 97.496 92.999 87.327
115 100.000 100.000 97.538 93.103 87.497
116 100.000 100.000 97.579 93.198 87.644
117 100.000 100.000 97.600 93.251 87.737
118 100.000 100.000 97.662 93.388 87.938
119 100.000 100.000 97.724 93.524 88.140
120 100.000 100.000 97.765 93.636 88.331
121 100.000 100.000 97.827 93.772 88.533
122 100.000 100.000 97.848 93.822 88.613
123 100.000 100.000 97.910 93.959 88.815
124 100.000 100.000 97.931 94.021 88.931
125 100.000 100.000 98.034 94.240 89.243
126 100.000 100.000 98.076 94.335 89.390
127 100.000 100.000 98.283 94.728 89.895
128 100.000 100.000 98.324 94.823 90.042
129 100.000 100.000 98.345 94.881 90.145
130 100.000 100.000 98.365 94.951 90.281
131 100.000 100.000 98.407 95.046 90.429
132 100.000 100.000 98.448 95.141 90.576
133 100.000 100.000 98.510 95.294 90.821
134 100.000 100.000 98.572 95.431 91.023
135 100.000 100.000 98.614 95.526 91.170
136 100.000 100.000 98.655 95.621 91.317
137 100.000 100.000 98.696 95.716 91.465
138 100.000 100.000 98.738 95.828 91.655
139 100.000 100.000 98.800 95.964 91.857
140 100.000 100.000 98.821 96.018 91.950
141 100.000 100.000 98.862 96.130 92.141
142 100.000 100.000 98.883 96.183 92.233
143 100.000 100.000 98.903 96.237 92.325
144 100.000 100.000 98.965 96.373 92.527
145 100.000 100.000 99.007 96.477 92.697
146 100.000 100.000 99.028 96.526 92.778
147 100.000 100.000 99.069 96.622 92.925
148 100.000 100.000 99.110 96.717 93.072
149 100.000 100.000 99.152 96.828 93.263
150 100.000 100.000 99.234 97.027 93.575
151 100.000 100.000 99.255 97.081 93.668
152 100.000 100.000 99.297 97.176 93.815
153 100.000 100.000 99.317 97.238 93.930
154 100.000 100.000 99.359 97.333 94.077
155 100.000 100.000 99.379 97.387 94.170
156 100.000 100.000 99.421 97.490 94.340
157 100.000 100.000 99.462 97.585 94.487
158 100.000 100.000 99.545 97.767 94.756
159 100.000 100.000 99.565 97.821 94.848
160 100.000 100.000 99.586 97.875 94.941
161 100.000 100.000 99.607 97.924 95.021
162 100.000 100.000 99.628 97.978 95.113
163 100.000 100.000 99.669 98.073 95.261
164 100.000 100.000 99.690 98.135 95.376
165 100.000 100.000 99.710 98.193 95.479
166 100.000 100.000 99.752 98.288 95.626
167 100.000 100.000 99.855 98.511 95.950
168 100.000 100.000 99.897 98.615 96.120
169 100.000 100.000 99.959 98.755 96.334
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 75.000 48.837 31.450 20.692 14.035
5 80.000 55.648 37.285 25.113 17.270
6 87.500 62.458 42.541 29.045 20.207
7 90.000 65.781 45.976 32.147 22.801
8 92.500 69.435 49.555 35.235 25.329
9 95.000 72.425 52.473 37.841 27.542
10 97.500 75.249 55.121 40.181 29.550
11 97.500 76.910 57.397 42.472 31.608
12 100.000 79.568 59.797 44.556 33.392
13 100.000 81.063 61.825 46.595 35.236
14 100.000 82.558 63.811 48.555 36.982
15 100.000 83.555 65.529 50.490 38.825
16 100.000 84.884 67.308 52.264 40.426
17 100.000 86.047 68.881 53.852 41.880
18 100.000 87.043 70.308 55.361 43.310
19 100.000 88.040 71.695 56.800 44.662
20 100.000 89.037 73.081 58.231 45.993
21 100.000 89.867 74.281 59.513 47.222
22 100.000 90.698 75.419 60.671 48.286
23 100.000 91.196 76.329 61.808 49.498
24 100.000 91.860 77.323 62.904 50.582
25 100.000 91.860 77.943 63.958 51.798
26 100.000 92.359 78.771 64.938 52.813
27 100.000 92.857 79.599 65.918 53.828
28 100.000 93.355 80.426 66.898 54.843
29 100.000 93.854 81.233 67.853 55.845
30 100.000 94.186 81.937 68.796 56.891
31 100.000 94.518 82.599 69.661 57.838
32 100.000 94.850 83.220 70.454 58.708
33 100.000 95.183 83.840 71.253 59.587
34 100.000 95.515 84.440 72.009 60.412
35 100.000 95.847 84.999 72.683 61.126
36 100.000 96.013 85.433 73.328 61.893
37 100.000 96.179 85.868 73.957 62.617
38 100.000 96.346 86.303 74.585 63.340
39 100.000 96.512 86.737 75.231 64.107
40 100.000 96.678 87.172 75.871 64.864
41 100.000 96.844 87.565 76.430 65.509
42 100.000 97.010 87.958 76.988 66.155
43 100.000 97.176 88.351 77.546 66.800
44 100.000 97.342 88.703 78.034 67.368
45 100.000 97.508 89.054 78.522 67.936
46 100.000 97.674 89.406 79.010 68.504
47 100.000 97.841 89.758 79.502 69.084
48 100.000 98.007 90.110 79.990 69.652
49 100.000 98.173 90.461 80.478 70.219
50 100.000 98.339 90.813 80.966 70.787
51 100.000 98.505 91.165 81.454 71.355
52 100.000 98.671 91.496 81.901 71.868
53 100.000 98.837 91.827 82.347 72.381
54 100.000 99.003 92.158 82.810 72.938
55 100.000 99.169 92.489 83.265 73.474
56 100.000 99.336 92.820 83.716 73.998
57 100.000 99.502 93.131 84.125 74.468
58 100.000 99.668 93.441 84.543 74.958
59 100.000 99.834 93.751 84.952 75.427
60 100.000 100.000 94.062 85.370 75.917
61 100.000 100.000 94.269 85.763 76.422
62 100.000 100.000 94.475 86.156 76.926
63 100.000 100.000 94.682 86.548 77.431
64 100.000 100.000 94.827 86.859 77.876
65 100.000 100.000 94.972 87.169 78.321
66 100.000 100.000 95.096 87.429 78.688
67 100.000 100.000 95.220 87.690 79.055
68 100.000 100.000 95.345 87.950 79.422
69 100.000 100.000 95.448 88.178 79.758
70 100.000 100.000 95.551 88.401 80.081
71 100.000 100.000 95.655 88.624 80.405
72 100.000 100.000 95.758 88.848 80.728
73 100.000 100.000 95.841 89.042 81.030
74 100.000 100.000 95.924 89.224 81.299
75 100.000 100.000 96.007 89.406 81.567
76 100.000 100.000 96.089 89.604 81.880
77 100.000 100.000 96.172 89.786 82.148
78 100.000 100.000 96.255 89.968 82.417
79 100.000 100.000 96.317 90.105 82.619
80 100.000 100.000 96.379 90.245 82.833
81 100.000 100.000 96.441 90.382 83.035
82 100.000 100.000 96.503 90.522 83.249
83 100.000 100.000 96.565 90.667 83.474
84 100.000 100.000 96.627 90.820 83.719
85 100.000 100.000 96.689 90.965 83.944
86 100.000 100.000 96.752 91.114 84.179
87 100.000 100.000 96.814 91.267 84.425
88 100.000 100.000 96.876 91.403 84.626
89 100.000 100.000 96.938 91.540 84.828
90 100.000 100.000 97.000 91.676 85.030
91 100.000 100.000 97.062 91.812 85.232
92 100.000 100.000 97.124 91.949 85.434
93 100.000 100.000 97.186 92.085 85.636
94 100.000 100.000 97.248 92.222 85.837
95 100.000 100.000 97.310 92.358 86.039
96 100.000 100.000 97.372 92.495 86.241
97 100.000 100.000 97.434 92.631 86.443
98 100.000 100.000 97.496 92.768 86.645
99 100.000 100.000 97.558 92.904 86.847
100 100.000 100.000 97.621 93.041 87.048
101 100.000 100.000 97.683 93.177 87.250
102 100.000 100.000 97.745 93.313 87.452
103 100.000 100.000 97.807 93.466 87.698
104 100.000 100.000 97.869 93.603 87.900
105 100.000 100.000 97.931 93.739 88.101
106 100.000 100.000 97.993 93.876 88.303
107 100.000 100.000 98.055 94.012 88.505
108 100.000 100.000 98.117 94.149 88.707
109 100.000 100.000 98.158 94.244 88.854
110 100.000 100.000 98.200 94.347 89.024
111 100.000 100.000 98.241 94.447 89.184
112 100.000 100.000 98.283 94.542 89.331
113 100.000 100.000 98.324 94.653 89.522
114 100.000 100.000 98.365 94.765 89.712
115 100.000 100.000 98.407 94.868 89.883
116 100.000 100.000 98.448 94.963 90.030
117 100.000 100.000 98.490 95.075 90.221
118 100.000 100.000 98.531 95.170 90.368
119 100.000 100.000 98.572 95.265 90.515
120 100.000 100.000 98.614 95.360 90.662
121 100.000 100.000 98.655 95.455 90.809
122 100.000 100.000 98.696 95.559 90.979
123 100.000 100.000 98.738 95.654 91.127
124 100.000 100.000 98.779 95.749 91.274
125 100.000 100.000 98.821 95.844 91.421
126 100.000 100.000 98.862 95.943 91.580
127 100.000 100.000 98.903 96.047 91.750
128 100.000 100.000 98.945 96.142 91.897
129 100.000 100.000 98.986 96.241 92.057
130 100.000 100.000 99.028 96.336 92.204
131 100.000 100.000 99.069 96.431 92.351
132 100.000 100.000 99.110 96.526 92.498
133 100.000 100.000 99.152 96.622 92.645
134 100.000 100.000 99.193 96.717 92.792
135 100.000 100.000 99.234 96.812 92.939
136 100.000 100.000 99.276 96.907 93.087
137 100.000 100.000 99.317 97.019 93.277
138 100.000 100.000 99.338 97.089 93.414
139 100.000 100.000 99.359 97.151 93.529
140 100.000 100.000 99.379 97.213 93.645
141 100.000 100.000 99.400 97.275 93.760
142 100.000 100.000 99.421 97.337 93.876
143 100.000 100.000 99.441 97.391 93.968
144 100.000 100.000 99.462 97.440 94.048
145 100.000 100.000 99.483 97.498 94.153
146 100.000 100.000 99.503 97.556 94.256
147 100.000 100.000 99.524 97.614 94.360
148 100.000 100.000 99.545 97.664 94.440
149 100.000 100.000 99.565 97.726 94.555
150 100.000 100.000 99.586 97.775 94.636
151 100.000 100.000 99.607 97.829 94.728
152 100.000 100.000 99.628 97.879 94.808
153 100.000 100.000 99.648 97.932 94.901
154 100.000 100.000 99.669 97.982 94.981
155 100.000 100.000 99.690 98.036 95.073
156 100.000 100.000 99.710 98.090 95.166
157 100.000 100.000 99.731 98.139 95.246
158 100.000 100.000 99.752 98.193 95.338
159 100.000 100.000 99.772 98.247 95.431
160 100.000 100.000 99.793 98.300 95.523
161 100.000 100.000 99.814 98.362 95.639
162 100.000 100.000 99.834 98.416 95.731
163 100.000 100.000 99.855 98.470 95.823
164 100.000 100.000 99.876 98.524 95.916
165 100.000 100.000 99.897 98.573 95.996
166 100.000 100.000 99.917 98.627 96.089
167 100.000 100.000 99.938 98.681 96.181
168 100.000 100.000 99.959 98.743 96.296
169 100.000 100.000 99.979 98.797 96.389
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 43.023 25.926 16.354 10.818
4 80.000 50.166 31.533 20.552 13.924
5 80.000 54.319 36.189 24.480 16.964
6 85.000 60.133 41.196 28.417 19.960
7 87.500 64.452 45.438 31.973 22.769
8 92.500 69.435 49.576 35.260 25.347
9 95.000 72.591 52.576 37.865 27.525
10 97.500 75.581 55.411 40.355 29.635
11 100.000 78.405 58.059 42.695 31.642
12 100.000 79.568 59.921 44.717 33.528
13 100.000 80.731 61.701 46.653 35.365
14 100.000 82.226 63.687 48.613 37.111
15 100.000 83.056 65.115 50.271 38.753
16 100.000 84.385 66.915 52.082 40.396
17 100.000 85.548 68.487 53.670 41.851
18 100.000 86.711 70.039 55.241 43.305
19 100.000 87.874 71.612 56.829 44.759
20 100.000 87.874 72.233 57.884 45.975
21 100.000 88.704 73.453 59.207 47.259
22 100.000 89.535 74.674 60.530 48.543
23 100.000 90.199 75.729 61.742 49.768
24 100.000 91.030 76.888 62.929 50.858
25 100.000 91.528 77.736 63.946 51.917
26 100.000 92.027 78.585 64.963 52.976
27 100.000 92.359 79.247 65.844 53.962
28 100.000 92.857 80.033 66.754 54.899
29 100.000 93.355 80.819 67.663 55.837
30 100.000 93.688 81.440 68.465 56.727
31 100.000 94.020 82.061 69.259 57.596
32 100.000 94.352 82.682 70.053 58.466
33 100.000 94.684 83.282 70.806 59.280
34 100.000 94.850 83.716 71.451 60.047
35 100.000 95.017 84.151 72.096 60.815
36 100.000 95.183 84.585 72.741 61.582
37 100.000 95.515 85.144 73.415 62.296
38 100.000 95.847 85.702 74.089 63.009
39 100.000 96.013 86.116 74.685 63.699
40 100.000 96.179 86.509 75.251 64.367
41 100.000 96.512 87.047 75.888 65.037
42 100.000 96.844 87.585 76.525 65.707
43 100.000 97.176 88.123 77.162 66.377
44 100.000 97.342 88.516 77.720 67.023
45 100.000 97.508 88.910 78.278 67.669
46 100.000 97.674 89.303 78.836 68.314
47 100.000 97.841 89.696 79.395 68.960
48 100.000 98.007 90.048 79.891 69.551
49 100.000 98.173 90.399 80.379 70.119
50 100.000 98.339 90.751 80.867 70.686
51 100.000 98.505 91.082 81.322 71.220
52 100.000 98.671 91.413 81.776 71.754
53 100.000 98.837 91.744 82.231 72.290
54 100.000 99.003 92.075 82.686 72.826
55 100.000 99.169 92.406 83.141 73.363
56 100.000 99.336 92.737 83.596 73.896
57 100.000 99.336 92.944 83.989 74.401
58 100.000 99.336 93.151 84.382 74.905
59 100.000 99.336 93.358 84.774 75.410
60 100.000 99.502 93.689 85.221 75.923
61 100.000 99.668 94.020 85.668 76.436
62 100.000 99.834 94.331 86.089 76.938
63 100.000 100.000 94.641 86.499 77.408
64 100.000 100.000 94.765 86.780 77.830
65 100.000 100.000 94.889 87.057 78.241
66 100.000 100.000 95.013 87.326 78.631
67 100.000 100.000 95.138 87.586 78.998
68 100.000 100.000 95.241 87.814 79.334
69 100.000 100.000 95.345 88.037 79.657
70 100.000 100.000 95.448 88.260 79.981
71 100.000 100.000 95.551 88.484 80.304
72 100.000 100.000 95.655 88.707 80.627
73 100.000 100.000 95.758 88.926 80.940
74 100.000 100.000 95.841 89.112 81.221
75 100.000 100.000 95.924 89.298 81.502
76 100.000 100.000 96.007 89.480 81.770
77 100.000 100.000 96.089 89.662 82.039
78 100.000 100.000 96.172 89.844 82.308
79 100.000 100.000 96.255 90.026 82.576
80 100.000 100.000 96.338 90.208 82.845
81 100.000 100.000 96.420 90.390 83.114
82 100.000 100.000 96.483 90.543 83.359
83 100.000 100.000 96.545 90.696 83.605
84 100.000 100.000 96.607 90.849 83.851
85 100.000 100.000 96.669 90.998 84.085
86 100.000 100.000 96.731 91.143 84.310
87 100.000 100.000 96.793 91.287 84.535
88 100.000 100.000 96.855 91.432 84.760
89 100.000 100.000 96.917 91.577 84.985
90 100.000 100.000 96.979 91.717 85.199
91 100.000 100.000 97.041 91.858 85.413
92 100.000 100.000 97.103 91.994 85.615
93 100.000 100.000 97.165 92.131 85.817
94 100.000 100.000 97.227 92.267 86.019
95 100.000 100.000 97.289 92.404 86.220
96 100.000 100.000 97.352 92.540 86.422
97 100.000 100.000 97.414 92.677 86.624
98 100.000 100.000 97.476 92.813 86.826
99 100.000 100.000 97.538 92.950 87.028
100 100.000 100.000 97.600 93.086 87.230
101 100.000 100.000 97.662 93.223 87.431
102 100.000 100.000 97.724 93.359 87.633
103 100.000 100.000 97.786 93.495 87.835
104 100.000 100.000 97.848 93.632 88.037
105 100.000 100.000 97.910 93.768 88.239
106 100.000 100.000 97.972 93.905 88.441
107 100.000 100.000 98.034 94.041 88.642
108 100.000 100.000 98.076 94.153 88.833
109 100.000 100.000 98.117 94.265 89.024
110 100.000 100.000 98.158 94.368 89.194
111 100.000 100.000 98.200 94.471 89.365
112 100.000 100.000 98.241 94.575 89.535
113 100.000 100.000 98.283 94.678 89.705
114 100.000 100.000 98.324 94.781 89.875
115 100.000 100.000 98.365 94.877 90.022
116 100.000 100.000 98.407 94.972 90.170
117 100.000 100.000 98.448 95.067 90.317
118 100.000 100.000 98.490 95.162 90.464
119 100.000 100.000 98.531 95.257 90.611
120 100.000 100.000 98.572 95.352 90.758
121 100.000 100.000 98.614 95.447 90.905
122 100.000 100.000 98.655 95.542 91.052
123 100.000 100.000 98.696 95.637 91.199
124 100.000 100.000 98.738 95.733 91.347
125 100.000 100.000 98.779 95.828 91.494
126 100.000 100.000 98.821 95.923 91.641
127 100.000 100.000 98.862 96.018 91.788
128 100.000 100.000 98.903 96.113 91.935
129 100.000 100.000 98.945 96.208 92.082
130 100.000 100.000 98.986 96.303 92.229
131 100.000 100.000 99.028 96.398 92.376
132 100.000 100.000 99.069 96.493 92.524
133 100.000 100.000 99.110 96.589 92.671
134 100.000 100.000 99.152 96.684 92.818
135 100.000 100.000 99.193 96.779 92.965
136 100.000 100.000 99.234 96.874 93.112
137 100.000 100.000 99.276 96.969 93.259
138 100.000 100.000 99.317 97.064 93.406
139 100.000 100.000 99.359 97.159 93.553
140 100.000 100.000 99.379 97.221 93.669
141 100.000 100.000 99.400 97.283 93.784
142 100.000 100.000 99.421 97.345 93.900
143 100.000 100.000 99.441 97.407 94.015
144 100.000 100.000 99.462 97.469 94.131
145 100.000 100.000 99.483 97.527 94.234
146 100.000 100.000 99.503 97.585 94.338
147 100.000 100.000 99.524 97.643 94.441
148 100.000 100.000 99.545 97.697 94.533
149 100.000 100.000 99.565 97.750 94.626
150 100.000 100.000 99.586 97.804 94.718
151 100.000 100.000 99.607 97.858 94.811
152 100.000 100.000 99.628 97.912 94.903
153 100.000 100.000 99.648 97.966 94.995
154 100.000 100.000 99.669 98.019 95.088
155 100.000 100.000 99.690 98.073 95.180
156 100.000 100.000 99.710 98.127 95.273
157 100.000 100.000 99.731 98.181 95.365
158 100.000 100.000 99.752 98.234 95.457
159 100.000 100.000 99.772 98.288 95.550
160 100.000 100.000 99.793 98.342 95.642
161 100.000 100.000 99.814 98.396 95.735
162 100.000 100.000 99.834 98.449 95.827
163 100.000 100.000 99.855 98.503 95.920
164 100.000 100.000 99.876 98.553 96.000
165 100.000 100.000 99.897 98.602 96.080
166 100.000 100.000 99.917 98.652 96.160
167 100.000 100.000 99.938 98.702 96.241
168 100.000 100.000 99.959 98.751 96.321
169 100.000 100.000 99.979 98.801 96.401
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 55.000 30.233 17.670 10.999 7.234
3 70.000 41.694 25.347 16.123 10.733
4 77.500 49.169 31.264 20.523 13.950
5 80.000 54.817 36.665 24.798 17.143
6 82.500 59.302 41.092 28.503 20.053
7 85.000 63.455 45.169 31.944 22.794
8 92.500 69.934 50.052 35.579 25.526
9 95.000 72.924 52.886 38.068 27.636
10 97.500 75.748 55.535 40.409 29.643
11 97.500 77.243 57.604 42.517 31.565
12 100.000 79.734 59.963 44.684 33.488
13 100.000 80.565 61.556 46.557 35.307
14 100.000 81.561 63.170 48.365 37.050
15 100.000 82.724 64.825 50.114 38.706
16 100.000 84.053 66.646 51.958 40.384
17 100.000 85.216 68.260 53.624 41.937
18 100.000 86.545 70.039 55.398 43.537
19 100.000 87.708 71.633 57.024 45.035
20 100.000 88.538 72.874 58.388 46.374
21 100.000 88.538 73.495 59.443 47.590
22 100.000 89.203 74.550 60.650 48.805
23 100.000 89.701 75.378 61.655 49.875
24 100.000 90.698 76.723 63.015 51.129
25 100.000 91.196 77.550 64.012 52.186
26 100.000 91.694 78.399 65.029 53.245
27 100.000 92.193 79.205 65.976 54.226
28 100.000 92.359 79.681 66.691 55.071
29 100.000 92.525 80.157 67.407 55.916
30 100.000 93.355 81.295 68.565 56.980
31 100.000 93.688 81.895 69.330 57.824
32 100.000 93.854 82.330 69.975 58.591
33 100.000 94.020 82.764 70.620 59.358
34 100.000 94.352 83.364 71.377 60.184
35 100.000 94.850 84.109 72.208 61.023
36 100.000 95.183 84.668 72.898 61.780
37 100.000 95.349 85.082 73.502 62.493
38 100.000 95.681 85.682 74.246 63.284
39 100.000 96.013 86.282 74.991 64.076
40 100.000 96.346 86.840 75.673 64.812
41 100.000 96.346 87.068 76.128 65.423
42 100.000 96.512 87.441 76.661 66.058
43 100.000 96.678 87.834 77.220 66.703
44 100.000 96.844 88.227 77.778 67.349
45 100.000 97.010 88.620 78.336 67.994
46 100.000 97.176 89.013 78.894 68.640
47 100.000 97.342 89.385 79.419 69.254
48 100.000 97.342 89.551 79.775 69.764
49 100.000 97.674 90.089 80.412 70.433
50 100.000 97.841 90.461 80.937 71.045
51 100.000 98.007 90.813 81.429 71.625
52 100.000 98.339 91.330 82.033 72.261
53 100.000 98.671 91.848 82.628 72.876
54 100.000 98.837 92.179 83.083 73.410
55 100.000 99.003 92.510 83.538 73.944
56 100.000 99.003 92.717 83.931 74.448
57 100.000 99.003 92.924 84.324 74.953
58 100.000 99.003 93.131 84.717 75.457
59 100.000 99.169 93.462 85.163 75.971
60 100.000 99.336 93.772 85.581 76.461
61 100.000 99.502 94.082 85.994 76.942
62 100.000 99.502 94.227 86.296 77.364
63 100.000 99.668 94.538 86.706 77.833
64 100.000 99.834 94.848 87.115 78.303
65 100.000 99.834 94.972 87.380 78.682
66 100.000 99.834 95.117 87.673 79.083
67 100.000 99.834 95.241 87.934 79.450
68 100.000 99.834 95.345 88.157 79.774
69 100.000 100.000 95.613 88.488 80.145
70 100.000 100.000 95.717 88.707 80.457
71 100.000 100.000 95.820 88.926 80.770
72 100.000 100.000 95.903 89.108 81.038
73 100.000 100.000 95.965 89.261 81.285
74 100.000 100.000 96.048 89.443 81.554
75 100.000 100.000 96.131 89.625 81.823
76 100.000 100.000 96.193 89.778 82.068
77 100.000 100.000 96.255 89.931 82.314
78 100.000 100.000 96.317 90.084 82.559
79 100.000 100.000 96.379 90.237 82.805
80 100.000 100.000 96.441 90.382 83.030
81 100.000 100.000 96.503 90.526 83.255
82 100.000 100.000 96.565 90.671 83.480
83 100.000 100.000 96.627 90.816 83.705
84 100.000 100.000 96.689 90.961 83.930
85 100.000 100.000 96.752 91.105 84.155
86 100.000 100.000 96.814 91.250 84.380
87 100.000 100.000 96.876 91.391 84.594
88 100.000 100.000 96.917 91.502 84.784
89 100.000 100.000 96.958 91.614 84.975
90 100.000 100.000 97.000 91.726 85.166
91 100.000 100.000 97.041 91.837 85.357
92 100.000 100.000 97.083 91.949 85.548
93 100.000 100.000 97.124 92.061 85.739
94 100.000 100.000 97.165 92.172 85.930
95 100.000 100.000 97.227 92.309 86.132
96 100.000 100.000 97.289 92.445 86.334
97 100.000 100.000 97.352 92.582 86.535
98 100.000 100.000 97.414 92.718 86.737
99 100.000 100.000 97.476 92.854 86.939
100 100.000 100.000 97.538 92.991 87.141
101 100.000 100.000 97.600 93.127 87.343
102 100.000 100.000 97.662 93.264 87.545
103 100.000 100.000 97.724 93.400 87.746
104 100.000 100.000 97.786 93.537 87.948
105 100.000 100.000 97.848 93.673 88.150
106 100.000 100.000 97.910 93.810 88.352
107 100.000 100.000 97.972 93.946 88.554
108 100.000 100.000 98.034 94.083 88.756
109 100.000 100.000 98.096 94.219 88.957
110 100.000 100.000 98.138 94.322 89.128
111 100.000 100.000 98.179 94.426 89.298
112 100.000 100.000 98.221 94.529 89.468
113 100.000 100.000 98.262 94.633 89.638
114 100.000 100.000 98.303 94.736 89.808
115 100.000 100.000 98.345 94.839 89.979
116 100.000 100.000 98.386 94.934 90.126
117 100.000 100.000 98.427 95.030 90.273
118 100.000 100.000 98.469 95.125 90.420
119 100.000 100.000 98.510 95.220 90.567
120 100.000 100.000 98.552 95.315 90.714
121 100.000 100.000 98.593 95.410 90.861
122 100.000 100.000 98.634 95.505 91.009
123 100.000 100.000 98.676 95.600 91.156
124 100.000 100.000 98.717 95.695 91.303
125 100.000 100.000 98.759 95.790 91.450
126 100.000 100.000 98.800 95.886 91.597
127 100.000 100.000 98.841 95.981 91.744
128 100.000 100.000 98.883 96.076 91.891
129 100.000 100.000 98.924 96.171 92.038
130 100.000 100.000 98.965 96.266 92.186
131 100.000 100.000 99.007 96.361 92.333
132 100.000 100.000 99.048 96.456 92.480
133 100.000 100.000 99.090 96.551 92.627
134 100.000 100.000 99.131 96.646 92.774
135 100.000 100.000 99.172 96.742 92.921
136 100.000 100.000 99.214 96.837 93.068
137 100.000 100.000 99.255 96.932 93.215
138 100.000 100.000 99.297 97.027 93.363
139 100.000 100.000 99.338 97.122 93.510
140 100.000 100.000 99.379 97.217 93.657
141 100.000 100.000 99.400 97.279 93.772
142 100.000 100.000 99.421 97.341 93.888
143 100.000 100.000 99.441 97.403 94.003
144 100.000 100.000 99.462 97.465 94.119
145 100.000 100.000 99.483 97.527 94.234
146 100.000 100.000 99.503 97.589 94.350
147 100.000 100.000 99.524 97.647 94.453
148 100.000 100.000 99.545 97.701 94.546
149 100.000 100.000 99.565 97.755 94.638
150 100.000 100.000 99.586 97.808 94.730
151 100.000 100.000 99.607 97.862 94.823
152 100.000 100.000 99.628 97.916 94.915
153 100.000 100.000 99.648 97.970 95.008
154 100.000 100.000 99.669 98.023 95.100
155 100.000 100.000 99.690 98.077 95.192
156 100.000 100.000 99.710 98.131 95.285
157 100.000 100.000 99.731 98.185 95.377
158 100.000 100.000 99.752 98.238 95.470
159 100.000 100.000 99.772 98.292 95.562
160 100.000 100.000 99.793 98.346 95.654
161 100.000 100.000 99.814 98.400 95.747
162 100.000 100.000 99.834 98.453 95.839
163 100.000 100.000 99.855 98.503 95.920
164 100.000 100.000 99.876 98.553 96.000
165 100.000 100.000 99.897 98.602 96.080
166 100.000 100.000 99.917 98.652 96.160
167 100.000 100.000 99.938 98.702 96.241
168 100.000 100.000 99.959 98.751 96.321
169 100.000 100.000 99.979 98.801 96.401
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 77.500 54.153 36.582 24.819 17.161
6 85.000 60.631 41.486 28.487 19.927
7 90.000 65.781 45.810 31.923 22.608
8 92.500 69.767 49.638 35.128 25.169
9 95.000 72.924 52.638 37.733 27.346
10 97.500 75.748 55.535 40.421 29.669
11 100.000 78.571 58.101 42.613 31.500
12 100.000 80.066 60.170 44.721 33.422
13 100.000 81.561 62.239 46.822 35.326
14 100.000 83.056 64.329 48.948 37.248
15 100.000 84.219 65.901 50.536 38.703
16 100.000 85.382 67.474 52.123 40.157
17 100.000 86.379 68.777 53.397 41.289
18 100.000 87.375 70.122 54.757 42.542
19 100.000 88.372 71.591 56.329 44.032
20 100.000 89.203 72.771 57.574 45.217
21 100.000 90.033 73.909 58.731 46.281
22 100.000 90.864 75.460 60.551 48.056
23 100.000 91.362 76.309 61.572 49.126
24 100.000 91.860 77.136 62.552 50.142
25 100.000 92.359 77.985 63.569 51.201
26 100.000 92.857 78.854 64.628 52.314
27 100.000 93.355 79.764 65.769 53.536
28 100.000 93.688 80.364 66.522 54.351
29 100.000 94.020 81.006 67.357 55.275
30 100.000 94.352 81.523 67.953 55.890
31 100.000 94.684 82.164 68.792 56.824
32 100.000 95.017 82.744 69.499 57.572
33 100.000 95.349 83.344 70.252 58.387
34 100.000 95.681 83.944 71.004 59.201
35 100.000 95.847 84.378 71.650 59.968
36 100.000 96.013 84.813 72.295 60.736
37 100.000 96.179 85.082 72.617 61.083
38 100.000 96.346 85.433 73.122 61.695
39 100.000 96.512 85.806 73.655 62.330
40 100.000 96.678 86.220 74.259 63.042
41 100.000 96.844 86.551 74.705 63.555
42 100.000 97.010 86.903 75.193 64.123
43 100.000 97.176 87.172 75.516 64.471
44 100.000 97.342 87.544 76.041 65.082
45 100.000 97.508 87.896 76.521 65.630
46 100.000 97.674 88.206 76.930 66.099
47 100.000 97.841 88.496 77.298 66.513
48 100.000 98.007 88.785 77.666 66.928
49 100.000 98.173 89.137 78.154 67.496
50 100.000 98.339 89.406 78.477 67.844
51 100.000 98.505 89.799 79.051 68.533
52 100.000 98.671 90.130 79.498 69.046
53 100.000 98.837 90.399 79.821 69.394
54 100.000 99.003 90.710 80.230 69.863
55 100.000 99.169 91.020 80.639 70.333
56 100.000 99.336 91.372 81.127 70.900
57 100.000 99.502 91.641 81.450 71.248
58 100.000 99.668 91.930 81.818 71.663
59 100.000 99.834 92.220 82.186 72.077
60 100.000 100.000 92.489 82.508 72.425
61 100.000 100.000 92.551 82.649 72.639
62 100.000 100.000 92.572 82.707 72.742
63 100.000 100.000 92.593 82.756 72.823
64 100.000 100.000 92.717 83.025 73.213
65 100.000 100.000 92.800 83.199 73.459
66 100.000 100.000 92.903 83.418 73.771
67 100.000 100.000 93.048 83.720 74.193
68 100.000 100.000 93.131 83.910 74.485
69 100.000 100.000 93.172 84.005 74.632
70 100.000 100.000 93.213 84.100 74.779
71 100.000 100.000 93.358 84.402 75.201
72 100.000 100.000 93.462 84.626 75.524
73 100.000 100.000 93.524 84.779 75.770
74 100.000 100.000 93.524 84.787 75.795
75 100.000 100.000 93.544 84.845 75.899
76 100.000 100.000 93.606 84.985 76.113
77 100.000 100.000 93.627 85.035 76.193
78 100.000 100.000 93.669 85.138 76.361
79 100.000 100.000 93.731 85.279 76.575
80 100.000 100.000 93.855 85.539 76.942
81 100.000 100.000 93.938 85.721 77.211
82 100.000 100.000 94.041 85.945 77.534
83 100.000 100.000 94.082 86.040 77.681
84 100.000 100.000 94.165 86.222 77.950
85 100.000 100.000 94.248 86.404 78.219
86 100.000 100.000 94.289 86.507 78.389
87 100.000 100.000 94.331 86.602 78.536
88 100.000 100.000 94.351 86.664 78.652
89 100.000 100.000 94.455 86.887 78.975
90 100.000 100.000 94.475 86.945 79.078
91 100.000 100.000 94.558 87.127 79.347
92 100.000 100.000 94.558 87.131 79.360
93 100.000 100.000 94.641 87.313 79.629
94 100.000 100.000 94.641 87.318 79.643
95 100.000 100.000 94.682 87.429 79.833
96 100.000 100.000 94.703 87.479 79.914
97 100.000 100.000 94.786 87.669 80.203
98 100.000 100.000 94.848 87.805 80.405
99 100.000 100.000 94.889 87.901 80.552
100 100.000 100.000 94.931 87.996 80.699
101 100.000 100.000 95.055 88.256 81.066
102 100.000 100.000 95.096 88.360 81.237
103 100.000 100.000 95.096 88.364 81.250
104 100.000 100.000 95.158 88.500 81.452
105 100.000 100.000 95.158 88.525 81.521
106 100.000 100.000 95.220 88.661 81.723
107 100.000 100.000 95.282 88.798 81.925
108 100.000 100.000 95.427 89.100 82.347
109 100.000 100.000 95.427 89.116 82.395
110 100.000 100.000 95.551 89.377 82.762
111 100.000 100.000 95.613 89.513 82.964
112 100.000 100.000 95.634 89.563 83.045
113 100.000 100.000 95.676 89.658 83.192
114 100.000 100.000 95.820 89.960 83.614
115 100.000 100.000 95.841 90.010 83.694
116 100.000 100.000 95.882 90.121 83.885
117 100.000 100.000 95.924 90.216 84.032
118 100.000 100.000 95.945 90.266 84.112
119 100.000 100.000 96.007 90.402 84.314
120 100.000 100.000 96.027 90.469 84.438
121 100.000 100.000 96.048 90.518 84.518
122 100.000 100.000 96.089 90.613 84.665
123 100.000 100.000 96.172 90.812 84.978
124 100.000 100.000 96.214 90.907 85.125
125 100.000 100.000 96.214 90.915 85.150
126 100.000 100.000 96.234 90.973 85.254
127 100.000 100.000 96.255 91.035 85.369
128 100.000 100.000 96.255 91.043 85.395
129 100.000 100.000 96.296 91.138 85.542
130 100.000 100.000 96.317 91.188 85.622
131 100.000 100.000 96.379 91.329 85.836
132 100.000 100.000 96.379 91.341 85.873
133 100.000 100.000 96.379 91.349 85.898
134 100.000 100.000 96.400 91.399 85.978
135 100.000 100.000 96.441 91.494 86.126
136 100.000 100.000 96.483 91.593 86.285
137 100.000 100.000 96.483 91.597 86.298
138 100.000 100.000 96.503 91.647 86.379
139 100.000 100.000 96.545 91.742 86.526
140 100.000 100.000 96.586 91.837 86.673
141 100.000 100.000 96.607 91.887 86.753
142 100.000 100.000 96.648 91.982 86.900
143 100.000 100.000 96.648 91.986 86.913
144 100.000 100.000 96.648 91.994 86.939
145 100.000 100.000 96.689 92.089 87.086
146 100.000 100.000 96.689 92.098 87.112
147 100.000 100.000 96.689 92.106 87.137
148 100.000 100.000 96.689 92.114 87.163
149 100.000 100.000 96.731 92.209 87.310
150 100.000 100.000 96.772 92.305 87.457
151 100.000 100.000 96.814 92.416 87.648
152 100.000 100.000 96.855 92.511 87.795
153 100.000 100.000 96.855 92.520 87.821
154 100.000 100.000 96.876 92.586 87.945
155 100.000 100.000 96.896 92.635 88.025
156 100.000 100.000 96.896 92.644 88.050
157 100.000 100.000 96.896 92.648 88.064
158 100.000 100.000 96.938 92.751 88.234
159 100.000 100.000 97.000 92.888 88.436
160 100.000 100.000 97.621 93.942 89.652
161 100.000 100.000 97.683 94.078 89.853
162 100.000 100.000 97.683 94.083 89.867
163 100.000 100.000 97.683 94.087 89.880
164 100.000 100.000 97.724 94.182 90.027
165 100.000 100.000 97.724 94.190 90.053
166 100.000 100.000 97.724 94.203 90.089
167 100.000 100.000 97.724 94.211 90.115
168 100.000 100.000 97.724 94.219 90.140
169 100.000 100.000 97.765 94.314 90.288
170 100.000 100.000 97.786 94.364 90.368
171 100.000 100.000 97.786 94.372 90.393
172 100.000 100.000 97.827 94.467 90.540
173 100.000 100.000 97.848 94.521 90.633
174 100.000 100.000 97.848 94.525 90.646
175 100.000 100.000 97.931 94.707 90.915
176 100.000 100.000 97.952 94.757 90.995
177 100.000 100.000 97.972 94.806 91.075
178 100.000 100.000 97.993 94.856 91.156
179 100.000 100.000 97.993 94.872 91.204
180 100.000 100.000 98.034 94.968 91.351
181 100.000 100.000 98.034 94.980 91.388
182 100.000 100.000 98.055 95.030 91.468
183 100.000 100.000 98.076 95.083 91.561
184 100.000 100.000 98.117 95.178 91.708
185 100.000 100.000 98.158 95.274 91.855
186 100.000 100.000 98.200 95.369 92.002
187 100.000 100.000 98.200 95.373 92.015
188 100.000 100.000 98.221 95.439 92.139
189 100.000 100.000 98.241 95.489 92.220
190 100.000 100.000 98.241 95.505 92.268
191 100.000 100.000 98.262 95.555 92.348
192 100.000 100.000 98.262 95.559 92.362
193 100.000 100.000 98.262 95.563 92.375
194 100.000 100.000 98.303 95.658 92.522
195 100.000 100.000 98.303 95.671 92.559
196 100.000 100.000 98.303 95.675 92.572
197 100.000 100.000 98.365 95.811 92.774
198 100.000 100.000 98.386 95.861 92.854
199 100.000 100.000 98.448 95.997 93.056
200 100.000 100.000 98.448 96.001 93.069
201 100.000 100.000 98.448 96.010 93.095
202 100.000 100.000 98.448 96.014 93.108
203 100.000 100.000 98.448 96.039 93.178
204 100.000 100.000 98.490 96.134 93.325
205 100.000 100.000 98.531 96.229 93.472
206 100.000 100.000 98.531 96.237 93.497
207 100.000 100.000 98.531 96.245 93.523
208 100.000 100.000 98.552 96.295 93.603
209 100.000 100.000 98.572 96.345 93.684
210 100.000 100.000 98.572 96.357 93.720
211 100.000 100.000 98.572 96.365 93.746
212 100.000 100.000 98.572 96.373 93.771
213 100.000 100.000 98.634 96.510 93.973
214 100.000 100.000 98.634 96.522 94.009
215 100.000 100.000 98.634 96.531 94.035
216 100.000 100.000 98.655 96.580 94.115
217 100.000 100.000 98.676 96.630 94.195
218 100.000 100.000 98.696 96.679 94.276
219 100.000 100.000 98.696 96.688 94.301
220 100.000 100.000 98.696 96.692 94.315
221 100.000 100.000 98.696 96.696 94.328
222 100.000 100.000 98.717 96.750 94.420
223 100.000 100.000 98.738 96.804 94.513
224 100.000 100.000 98.738 96.808 94.526
225 100.000 100.000 98.759 96.857 94.606
226 100.000 100.000 98.759 96.866 94.632
227 100.000 100.000 98.759 96.870 94.645
228 100.000 100.000 98.800 96.965 94.792
229 100.000 100.000 98.821 97.014 94.873
230 100.000 100.000 98.821 97.019 94.886
231 100.000 100.000 98.821 97.027 94.912
232 100.000 100.000 98.821 97.039 94.948
233 100.000 100.000 98.821 97.043 94.961
234 100.000 100.000 98.862 97.138 95.109
235 100.000 100.000 98.862 97.143 95.122
236 100.000 100.000 98.903 97.238 95.269
237 100.000 100.000 98.945 97.333 95.416
238 100.000 100.000 98.986 97.428 95.563
239 100.000 100.000 98.986 97.436 95.589
240 100.000 100.000 99.028 97.531 95.736
241 100.000 100.000 99.069 97.626 95.883
242 100.000 100.000 99.069 97.635 95.909
243 100.000 100.000 99.069 97.643 95.934
244 100.000 100.000 99.110 97.738 96.081
245 100.000 100.000 99.110 97.746 96.107
246 100.000 100.000 99.110 97.755 96.132
247 100.000 100.000 99.110 97.763 96.158
248 100.000 100.000 99.110 97.767 96.171
249 100.000 100.000 99.110 97.771 96.185
250 100.000 100.000 99.110 97.775 96.198
251 100.000 100.000 99.152 97.870 96.345
252 100.000 100.000 99.152 97.875 96.358
253 100.000 100.000 99.152 97.879 96.372
254 100.000 100.000 99.152 97.887 96.397
255 100.000 100.000 99.152 97.891 96.411
256 100.000 100.000 99.152 97.899 96.436
257 100.000 100.000 99.152 97.908 96.462
258 100.000 100.000 99.172 97.957 96.542
259 100.000 100.000 99.379 98.350 97.047
260 100.000 100.000 99.379 98.358 97.072
261 100.000 100.000 99.379 98.367 97.098
262 100.000 100.000 99.379 98.371 97.111
263 100.000 100.000 99.379 98.375 97.124
264 100.000 100.000 99.379 98.391 97.173
265 100.000