This section describes experiments that were performed on five datasets, taken from the Irvine collection (Murphy & Aha, 1994). The five datasets are described in detail in Appendix A. All five datasets involve medical problems. The test costs are based on information from the Ontario Ministry of Health (1992). The main purpose of the experiments is to gain insight into the behavior of ICET. The other cost-sensitive algorithms, EG2, CS-ID3, and IDX, are included mainly as benchmarks for evaluating ICET. C4.5 is also included as a benchmark, to illustrate the behavior of an algorithm that makes no use of cost information. The main conclusion of these experiments is that ICET performs significantly better than its competitors, under a wide range of conditions. With access to the Irvine collection and the information in Appendix A, it should be possible for other researchers to duplicate the results reported here.
Medical datasets frequently have missing values. We conjecture that many missing values in medical datasets are missing because the doctor involved in generating the dataset decided that a particular test was not economically justified for a particular patient. Thus there may be information content in the fact that a certain value is missing. There may be many reasons for missing values other than the cost of the tests. For example, perhaps the doctor forgot to order the test or perhaps the patient failed to show up for the test. However, it seems likely that there is often information content in the fact that a value is missing. For our experiments, this information content should be hidden from the learning algorithms, since using it (at least in the testing sets) would be a form of cheating. Two of the five datasets we selected had some missing data. To avoid accusations of cheating, we decided to preprocess the datasets so that the data presented to the algorithms had no missing values. This preprocessing is described in Appendices A.2 and A.3.
Note that ICET is capable of handling missing values without preprocessing -- it inherits this ability from its C4.5 component. We preprocessed the data only to avoid accusations of cheating, not because ICET requires preprocessed data.
For the experiments, each dataset was randomly split into 10 pairs of training and testing sets. Each training set consisted of two thirds of the dataset and each testing set consisted of the remaining one third. The same 10 pairs were used in all experiments, in order to facilitate comparison of results across experiments.
There are three groups of experiments. The first group of experiments examines the baseline performance of the algorithms. The second group considers how robust ICET is under a variety of conditions. The final group looks at how ICET searches bias space.