基于分布式聚类的作物生长环境异常检测方法

doi:10.3969/j.issn.0255-8297.2018.06.013

应用科学学报 ›› 2018, Vol. 36 ›› Issue (6): 1010-1021.doi: 10.3969/j.issn.0255-8297.2018.06.013

基于分布式聚类的作物生长环境异常检测方法

余玥^1,2, 邓丽^1,2, 庞洪霖^1,2, 费敏锐^1,2

1. 上海大学机电工程与自动化学院, 上海 200072;
2. 上海市电站自动化技术重点实验室, 上海 200072

收稿日期:2016-12-20 修回日期:2017-05-31 出版日期:2018-12-31 发布日期:2018-12-31
通信作者: 邓丽,副教授,研究方向:机器学习、智能优化算法等,E-mail:dengli@shu.edu.cn E-mail:dengli@shu.edu.cn
基金资助:
上海市科委重点项目基金（No.14DZ1206302）资助

Environmental Anomaly Detection Method during Crop Growth Based on Distributed Clustering

YU Yue^1,2, DENG Li^1,2, PANG Hong-lin^1,2, FEI Min-rui^1,2

1. School of Mechatronics Engineering and Automation, Shanghai University, Shanghai 200072, China;
2. Shanghai Key Laboratory of Power Station Automation Technology, Shanghai 200072, China

Received:2016-12-20 Revised:2017-05-31 Online:2018-12-31 Published:2018-12-31

摘要/Abstract

摘要： 为了处理大量分布式存储的农业环境数据，提高农业生产效率，对高斯混合模型聚类算法进行了改进，提出了一种基于分布式聚类的农业环境数据异常检测方法.在Spark分布式计算框架下，首先对数据进行粗聚类，得到初始化模型；然后利用Spark迭代更新模型直至稳定，其中Map阶段将样本点分配到模型，Reduce阶段更新模型个数及参数；最后利用聚类结果，实现环境异常值的检测.实验结果表明该方法可行有效.

关键词: 高斯混合模型聚类, 农业环境数据, 异常检测, Spark

Abstract: In order to process the massive agricultural environmental data stored in distributed system and improve the production efciency, the clustering algorithm based on Gaussian Mixture Model (GMM) is modifed in this paper. Based on this, an environmental anomaly detection method during crop growth is proposed. Under the Spark distributed computing framework, frstly, a pre-clustering algorithm is adopted to initialize the models. Secondly, Spark is utilized to update the models iterationally until it gets stable. In each iteration, Map phase distributes sample points to the models, Reduce phase renews the numbers of models and parameters. Finally, the detection of environmental anomaly is completed by taking advantages of the clustering result. The experimental results show that this approach is practically feasible and effective.

Key words: agriculture environmental data, anomaly detection, Spark, Gaussian mixture model (GMM) clustering

中图分类号:

P277

余玥, 邓丽, 庞洪霖, 费敏锐. 基于分布式聚类的作物生长环境异常检测方法[J]. 应用科学学报, 2018, 36(6): 1010-1021.

YU Yue, DENG Li, PANG Hong-lin, FEI Min-rui. Environmental Anomaly Detection Method during Crop Growth Based on Distributed Clustering[J]. Journal of Applied Sciences, 2018, 36(6): 1010-1021.

参考文献

[1] Ruß G, Kruse R, Scheider M. A clustering approach for management zone delineation in precision agriculture[J]. Journal of Applied Physics, 2010, 26(9):1165-1172.
[2] Russ G, Kruse R. Machine learning methods for spatial clustering on precision agriculture data[C]//Scandinavian Conference on Artifcial Intelligence, 2011, 227:40-49.
[3] Cao L, Zhang X, San X, Zhao Y, Chen G. Application of fuzzy clustering algorithm in precision agriculture[C]//2012 IEEE World Automation Congress (WAC), 2012:1-4.
[4] Ananthara M G, Arunkumar T, Hemavathy R. CRY-An improved crop yield prediction model using bee hive clustering approach for agricultural data sets[C]//IEEE International Conference on Pattern Recognition, Informatics and Mobile Engineering, 2013:473-478.
[5] Wu M, Wang Y, Liao Z. A new clustering algorithm for sensor data streams in an agricultural IoT[C]//IEEE International Conference on High Performance Computing and Communications, Zhangjiajie. 2013:2373-2378.
[6] 时雷. 基于物联网的小麦生长环境数据采集与数据挖掘技术研究[D]. 郑州:河南农业大学,2013.
[7] 曹振丽. 面向养殖环境监测的数据流处理方法研究[D]. 北京:中国农业大学,2015.
[8] Oh S H, Lee W S. An anomaly intrusion detection method by clustering normal user behavior[J]. Computers & Security, 2003, 22(7):596-612.
[9] Li H, Achim A, Bull D. Unsupervised video anomaly detection using feature clustering[J]. IET Signal Processing, 2012, 6(5):521-533.
[10] Ahmed M, Mahmood A N, Maher M J. Heart disease diagnosis using co-clustering[M]. Germany:Springer International Publishing, 2015:61-70.
[11] Bilmes J A. A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models[R]. International Computer Science Institute Technical Report, UC Berkeley. 1998.
[12] Figueiredo M A T, Jain A K. Unsupervised learning of fnite mixture models[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2002, 24(3):381-396.
[13] Yang M S, Lai C Y, Lin C Y. A robust EM clustering algorithm for Gaussian mixture models[J]. Pattern Recognition, 2012, 45(11):3950-3961.
[14] Zaharia M, Chowdhury M, Franklin M J, Shenker S, Stoica I. Spark:cluster computing with working sets[C]//Usenix Conference on Hot Topics in Cloud Computing, USENIX Association, 2010:1765-1773.
[15] 侯加林. 温室番茄生长发育模拟模型的研究[D]. 北京:中国农业大学,2005.
[16] Mccallum A. Efcient clustering of high-dimensional data sets with application to reference matching[J]. Knowledge Discovery & Data Mining, 2010:169-178.

基于分布式聚类的作物生长环境异常检测方法

Environmental Anomaly Detection Method during Crop Growth Based on Distributed Clustering

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	成宝芝，赵春晖，王玉磊. 结合光谱解混的高光谱图像异常目标检测SVDD算法[J]. 应用科学学报, 2012, 30(1): 82-88.
[2]	田新广;段洣毅;孙春来;李文法. 采用shell命令和隐Markov模型进行网络用户行为异常检测 [J]. 应用科学学报, 2008, 26(2): 175-175 .