Combinatorial Test Generation for Multiple Input Models with Shared Parameters.

Combinatorial testing typically considers a single input model and creates a single test set that achieves -way coverage. This paper addresses the problem of combinatorial test generation for multiple input models with shared parameters. We formally define the problem and propose an efficient approach to generating multiple test sets, one for each input model, that together satisfy -way coverage for all of these input models while minimizing the amount of redundancy between these test sets.

Monte Carlo studies of bootstrap variability in ROC analysis with data dependency.

ROC analysis involving two large datasets is an important method for analyzing statistics of interest for decision making of a classifier in many disciplines. And data dependency due to multiple use of the same subjects exists ubiquitously in order to generate more samples because of limited resources. Hence, a two-layer data structure is constructed and the nonparametric two-sample two-layer bootstrap is employed to estimate standard errors of statistics of interest derived from two sets of data, such as a weighted sum of two probabilities.

The Impact of Data Dependence on Speaker Recognition Evaluation.

The data dependency due to multiple use of the same subjects has impact on the standard error (SE) of the detection cost function (DCF) in speaker recognition evaluation. The DCF is defined as a weighted sum of the probabilities of type I and type II errors at a given threshold. A two-layer data structure is constructed: target scores are grouped into target sets based on the dependency, and likewise for non-target scores.

A novel measure and significance testing in data analysis of cell image segmentation.

Background: Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods.

Measuring and Specifying Combinatorial Coverage of Test Input Configurations.

A key issue in testing is how many tests are needed for a required level of coverage or fault detection. Estimates are often based on error rates in initial testing, or on code coverage. For example, tests may be run until a desired level of statement or branch coverage is achieved.

Estimating -way Fault Profile Evolution During Testing.

Empirical studies have shown that most software interaction faults involve one or two variables interacting, with progressively fewer triggered by three or more, and no failure has been reported involving more than six variables interacting. This paper introduces a hypothesis for the origin of this distribution, with implications for removal of interaction faults and reliability growth.

Validation of Nonparametric Two-Sample Bootstrap in ROC Analysis on Large Datasets.

The nonparametric two-sample bootstrap is applied to computing uncertainties of measures in ROC analysis on large datasets in areas such as biometrics, speaker recognition, etc., when the analytical method cannot be used. Its validation was studied by computing the SE of the area under ROC curve using the well-established analytical Mann-Whitney-statistic method and also using the bootstrap.

Removing divergence of JCGM documents from the GUM ( 1993) and repairing other defects.

The mission of the Joint Committee for Guides in Metrology (JCGM) is to maintain and promote the use of the Guide to the Expression of Uncertainty in Measurement (GUM) and the International Vocabulary of Metrology (VIM, second edition). The JCGM has produced the third edition of the VIM (referred to as VIM3) and a number of documents; some of which are referred to as supplements to the GUM. We are concerned with the Supplement 1 (GUM-S1) and the document JCGM 104.

Anharmonic Vibrational Frequency Calculations Are Not Worthwhile for Small Basis Sets.

Anharmonic calculations using vibrational perturbation theory are known to provide near-spectroscopic accuracy when combined with high-level ab initio potential energy functions. However, performance with economical, popular electronic structure methods is less well characterized. We compare the accuracy of harmonic and anharmonic predictions from Hartree-Fock, second-order perturbation, and density functional theories combined with 6-31G(d) and 6-31+G(d,p) basis sets.

Combining Results From Multiple Evaluations of the Same Measurand.

According to the Guide to the Expression of Uncertainty in Measurement (GUM), a result of measurement consists of a measured value together with its associated standard uncertainty. The measured value and the standard uncertainty are interpreted as the expected value and the standard deviation of a state-of-knowledge probability distribution attributed to the measurand. We discuss the term metrological compatibility introduced by the International Vocabulary of Metrology, third edition (VIM3) for lack of significant differences between two or more results of measurement for the same measurand.

Measures, Uncertainties, and Significance Test in Operational ROC Analysis.

In receiver operating characteristic (ROC) analysis, the sampling variability can result in uncertainties of performance measures. Thus, while evaluating and comparing the performances of algorithms, the measurement uncertainties must be taken into account. The key issue is how to calculate the uncertainties of performance measures in ROC analysis.

Assessing Differences Between Results Determined According to the Guide to the Expression of Uncertainty in Measurement.

In some metrology applications multiple results of measurement for a common measurand are obtained and it is necessary to determine whether the results agree with each other. A result of measurement based on the Guide to the Expression of Uncertainty in Measurement (GUM) consists of a measured value together with its associated standard uncertainty. In the GUM, the measured value is regarded as the expected value and the standard uncertainty is regarded as the standard deviation, both known values, of a state-of-knowledge probability distribution.

Scaling Factors and Uncertainties for ab Initio Anharmonic Vibrational Frequencies.

To predict the vibrational spectra of molecules, ab initio calculations are often used to compute harmonic frequencies, which are usually scaled by empirical factors as an approximate correction for errors in the force constants and for anharmonic effects. Anharmonic computations of fundamental frequencies are becoming increasingly popular. We report scaling factors, along with their associated uncertainties, for anharmonic (second-order perturbation theory) predictions from HF, MP2, and B3LYP calculations using the 6-31G(d) and 6-31+G(d,p) basis sets.

Uncertainties in scaling factors for ab initio vibrational zero-point energies.

Vibrational zero-point energies (ZPEs) determined from ab initio calculations are often scaled by empirical factors. An empirical scaling factor partially compensates for the effects arising from vibrational anharmonicity and incomplete treatment of electron correlation. These effects are not random but are systematic.

Refining the In-Parameter-Order Strategy for Constructing Covering Arrays.

Covering arrays are structures for well-representing extremely large input spaces and are used to efficiently implement blackbox testing for software and hardware. This paper proposes refinements over the In-Parameter-Order strategy (for arbitrary t). When constructing homogeneous-alphabet covering arrays, these refinements reduce runtime in nearly all cases by a factor of more than 5 and in some cases by factors as large as 280.

Uncertainties in scaling factors for ab initio vibrational frequencies.

Vibrational frequencies determined from ab initio calculations are often scaled by empirical factors. An empirical scaling factor partially compensates for the errors arising from vibrational anharmonicity and incomplete treatment of electron correlation. These errors are not random but are systematic biases.

Taguchi's Orthogonal Arrays Are Classical Designs of Experiments.

Taguchi's catalog of orthogonal arrays is based on the mathematical theory of factorial designs and difference sets developed by R. C. Bose and his associates.