An Evaluation of the Minimum Expected Cost Metric for Intrusion Detection Systems

The base-rate fallacy of intrusion detection systems (IDSs) states that that in many cases, the false positive rate is the limiting factor for performance improvement of IDSs. In this paper, we present a quantitative evaluation on the Minimum Expected Cost (MEC) used for the IDS evaluation. MEC metric uses a decision tree and the Bayesian theorem for the deduction of the expected cost associated with false positives, which can be minimized by the determination of an operating point from the corresponding ROC curve. Comparing to similar approaches, MEC includes more environmental factors, such as the hostility of an operating environment and the prior possibility of intrusions. In addition, it quantitatively supports decisions that may be contrary to the decisions solely based on the reports from an IDS. By showing a detailed quantitative reasoning process, we demonstrate how MEC validates the base-rate fallacy problem from a new perspective. Based on the study we also proposed potential ways to improve the MEC metric.