The clustering center needs to be set manually is the biggest problem of K-means algorithm, and it is usually impossible to determine the classification of data in reality. In order to solve the problem, we propose a new OCC K-means algorithm. Different from the traditional algorithm, which generates the clustering center in the way of random selection, this algorithm carries out necessary preprocessing, and uses UPGMA and maximum and minimum distance algorithm to screen data points for the ones that can reflect data distribution characteristics as the initial clustering center, so as to improve the accuracy of clustering. From the two experimental results, it can be found that in different data sets, the improved algorithm is better in the measurement of clustering accuracy, recall, F-measurement than the traditional K-means algorithm. This is because the center point selected by OCC algorithm comes from different and data-intensive areas, and noise data and edge data interference to the experiment are excluded in the process of screening. At the same time, in order to conform to the trend of big data development, the parallelization implementation is carried out on Spark platform with Scala language, which improves the ability of the algorithm to deal with massive data, and the better parallelization of the algorithm is verified by experimental indexes.%聚类中心需要手动设置是K-means算法最大的问题,而通常情况是并不能确定现实中数据的分类情况.为了解决这一问题,提出了一种新的OCC K-means算法.不同于传统算法以随机选择的方式产生聚类中心,该算法进行必要的预处理,利用UPGMA和最大最小距离算法对数据点进行筛选,得到可以反映数据分布特征的点,并作为初始的聚类中心,以提高聚类的精度.从两次的实验结果可以对比出,在不同的数据集上,改进算法在衡量聚类效果的准确率、召回率、F-测量值上的表现要优于传统K-means算法.这是因为OCC算法选择的中心点来自于不同的且数据密集的区域,并在筛选的过程中排除了噪声数据、边缘数据对实验的干扰;同时为了契合大数据发展潮流,使用Scala语言在Spark平台进行了并行化实现,提高了算法处理海量数据的能力,并通过实验指标验证了算法具有良好的并行化能力.
展开▼
