中文地名数据清洗中的重复字段匹配

doi:10.3969/j.issn.0255-8297.2013.02.017

应用科学学报 ›› 2013, Vol. 31 ›› Issue (2): 212-220.doi: 10.3969/j.issn.0255-8297.2013.02.017

• 计算机科学与应用 • 上一篇

中文地名数据清洗中的重复字段匹配

叶鸥1，张璟1,2，李军怀1

1. 西安理工大学计算机科学与工程学院，西安710048
2. 西安交通大学机械制造系统工程国家重点实验室，西安710048

收稿日期:2012-03-15 修回日期:2012-07-18 出版日期:2013-03-25 发布日期:2012-07-18
通信作者: 叶鸥，博士生，研究方向：分布式计算与Web技术，E-mail：785669070@qq.com
作者简介:叶鸥，博士生，研究方向：分布式计算与Web技术，E-mail：785669070@qq.com；张璟，教授，博导，研究方向：分布式计算、虚拟化技术、云计算，E-mail：zhangjing@xaut.edu.cn
基金资助:
国家自然科学基金(No.61172018)；国家“863”高技术研究发展计划基金(No.2007AA010305)；陇西水利科技计划项目基金
资助

Duplicate Field Matching for Data Cleaning of Chinese Placenames

YE Ou1, ZHANG Jing1,2, LI Jun-huai1

1. School of Computer Science and Engineering, Xi’an University of Technology, Xi’an 710048, China
2. State Key Laboratory for Manufacturing Systems Engineer, Xi’an Jiaotong University, Xi’an 710048, China

Received:2012-03-15 Revised:2012-07-18 Online:2013-03-25 Published:2012-07-18

摘要/Abstract

摘要： 为了提高中文地名字段匹配的准确度，提出中文地名的相似重复检测与清洗策略和矩阵相似重复匹配方法. 该清洗策略采用矩阵相似重复匹配方法，通过矩阵运算计算中文地名数据之间相同字符或者词语出现的频率，利用频率计算两者之间的语义相似度和结构相似度，然后将语义和结构相似性有机结合，作为相似性判断和重复数据清洗的依据. 实验验证了该方法的可行性和有效性. 结果表明，该方法对于中文地名字段匹配在查准率和召回率方面优于已有的其他算法.

关键词: 数据清洗, 字段匹配, 矩阵相似重复匹配, 中文地名, 语义相似度, 结构相似度

Abstract: To improve accuracy of field matching of Chinese placenames, an approximate duplicate detection and cleaning strategy and a matrix approximate duplicate matching method are proposed. In the strategy, a matrix approximate duplicate matching method is used. Frequencies of the same Chinese characters or word between two Chinese placenames can first be calculated with a matrix operation. Semantic similarity and structure similarity can be calculated using the frequencies. By combining semantic and structure similarities, they are considered as the basis of duplicate detection and data cleaning. Simulation experiments are conducted to prove feasibility and validity of the method, showing that the matrix approximate duplicate matching method is better than other existing methods in terms of precision and recall ratio.

Key words: data cleaning, field matching, matrix approximately duplicate matching, Chinese placename, semantic similarity, structure similarity

中图分类号:

TP311

叶鸥1，张璟1,2，李军怀1. 中文地名数据清洗中的重复字段匹配[J]. 应用科学学报, 2013, 31(2): 212-220.

YE Ou1, ZHANG Jing1,2, LI Jun-huai1. Duplicate Field Matching for Data Cleaning of Chinese Placenames[J]. Journal of Applied Sciences, 2013, 31(2): 212-220.

参考文献

[1]郭志懋，周傲英．数据质量和数据清洗研究综述 [J].软件学报，2002, 13(11): 2076-2082．
[2]FAN Wenfei． Extending dependencies with conditions for data cleaning [J]．2008 8^th IEEE International Conference on: Computer and Information Technology, 2008: 185-190．
[3]NEELY M P. Data quality tools for data warehousing: a small sample survey [C]// Proceedings of MIT Conference on Information Quality, Center for Technology in Government, University at Albany/ SUNY: Germany (1998).
[4]QIU Yuefeng, TIAN Zongping, JI Wenyun. An efficient approach for detecting approximately duplicate database records [J]．Chinese Journal of Computers, 2001, 24(1): 69-77．
[5]EKTEFA M, SIDI F, IBRAHIM H, JABAR M A, MEMAR S, RAMLI A．A threshold-based similarity measure for duplicate detection [C]//2011 IEEE Conference on Open Systems(ICOS), 2011: 37-41．
[6]TREVOR C K．Automated detection of duplicate free-form English bug reports [J]．MS Computer Science Thesis, Department of Computer Science, Morgantown, West Virginia, USA, 2009, West Virginia University．
[7]RUNESON P, ALEXANDERSSON M, NYHOLM O．Detection of duplicate defect reports using natural language processing [C]//ICSE 2007: Proceedings of the 29^th international conference on Software Engineering, Washington, DC, USA, 2007: 499-510.
[8]NARAYANA V A, PREMCHAND P, GOVARDHAN A．A novel and efficient approach for near duplicate page detection in Web crawling [C]//2009 IEEE International Advance Computing Conference (IACC), 2009: 1492-1496．
[9]WANG Bin, LI Zhiwei, LI Mingjing, MA Weiying．Large-scale duplicate detection for Web image search [C]//2006 IEEE International Conference on Multimedia and Expo, 2006: 353-356．
[10]MASEK W, PATERSON M A．Fast algorithm computing string edit distance [J]．Journal of Computer System Science, 1980, 20(1): 18-31．
[11]LIAO Hsienyu, MENG_Laiyin, YI Cheng．A parallel implementation of the Smith-Waterman algorithm for massive sequences searching [C]//IEMBS’04．26^th Annual International Coference of the IEEE: Engineering in Medicine and Biology Society, 2004, 2: 2817-2820．
[12]WINKLER W E. The state record linkage and current research problems [J]．Technical report, Statistics of Income Division, Internal Revenue Service Publication (1999)．
[13]LI Guohui, DU Xiaokun, HU Fangxiao, YANG Bing, TANG Xiaohong．Structure matching method based on functional dependencies [J]．Journal of Software, 2009, 20(10): 2667-2678．

中文地名数据清洗中的重复字段匹配

Duplicate Field Matching for Data Cleaning of Chinese Placenames

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 1

编辑推荐

Metrics

本文评价