Dynamic Similarity and Distance Measures Based on Quantiles

Abstract

Data Mining emerges in response to technological advances and considers the treatment of large amounts of data. The aim of Data Mining is the extraction of new, valid, comprehensible and useful knowledge by the construction of a simple model that describes the data and can also be used in prediction tasks. The challenge of extracting knowledge from data is an interdisciplinary discipline and draws upon research in statistics, pattern recognition and machine learning among others.

A common technique for identifying natural groups hidden in data is clustering. Clustering is a process that automatically discovers structure in data and does not require any supervision. The model’s performance relies heavily on the choice of an appropriate measure. It is important to use the appropriate similarity metric to measure the proximity between two objects, but the separability of clusters must also be taken into account.

This paper addresses the problem of comparing two or more sets of overlapping data as a basis for comparing different partitions of quantitative data. An approach that uses statistical concepts to measure the distance between partitions is presented. The data’s descriptive knowledge is expressed by means of a boxplot that allows for the construction of clusters taking into account conditional probabilities.