Abstract
Frequent itemsets mining is a common and useful task in data mining. But most of the current mining algorithms can’t be used in very large text database. In this paper, we propose a novel and efficient parallel algorithm parTFI which is used to find top-k frequent itemsets with specified minimum length in very large text database. Base on a simple data structure H-struct, parTFI uses a novel logical vertical data partition technique to mine top-k frequent itemsets at each mining server parallel. Our performance study shows that when processing very large sparse text database, parTFI outperforms Apriori and FP-growth, two efficient frequent iemsets mining algorithms, even when both are running with the better tuned min_support. Furthermore, by creating H-struct dynamically, parTFI can suit even huge dataset that most other algorithms can’t process.
This project is sponsored by national 863 high technology development foundation (No.2004AA112020, No.2003AA115210 and No.2003AA111020).
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Antonie, M.-L., Zaïane, O.R.: Text Document Categorization by Term Association. In: Proc. of the IEEE 2002 International Conference on Data Mining (ICDM 2002), Maebashi City, Japan, December 9 - 12, pp. 19–26 (2002)
Beil, F., Ester, M., Xu, X.: Frequent Term-Based Text Clustering. In: ACM SIGKDD (2002)
Agrawal, R., Srikant, R.: Fast algorithms for mining association rules. In: VLDB 1994 (1994)
Han, J., Pei, J., Yin, Y.: Mining frequent patterns without candidate generation. In: SIGMOD 2000 (2000)
Zheng, Z., Kohavi, R., Mason, L.: Real World Performance of Association Rule Algorithms. In: Proc. of KDD 2001 (2001)
Agrawal, R., Shafer, J.: Parallel and Distributed Association Mining: A Survey. "Parallel Mining of Association Rules". IEEE Trans. Knowledge and Data Eng. 8(6), 962–969 (1996)
Zaki, M.J., Hsiao, C.J.: CHARM: An efficient algorithm for closed itemset mining. In: Grossman, R., et al. (eds.) Proc. of the 2nd SIAM Int’l. Conf. on Data Mining, pp. 12–28. SIAM, Arlington (2002)
Han, J., Pei, J., Yin, Y.: Mining frequent patterns without candidate generation. In: SIGMOD 2000, pp. 1–12 (2000)
Pei, J., Han, J., Lu, H., Nishio, S., Tang, S., Yang, D.: H-Mine: Hyper-Structure Mining of Frequent Patterns in Large Databases. In: Proc. of the 2001 IEEE ICDM Conf., San Jose, CA, USA (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, Y., Jia, Y., Yang, S. (2005). Parallel Mining of Top-K Frequent Itemsets in Very Large Text Database. In: Fan, W., Wu, Z., Yang, J. (eds) Advances in Web-Age Information Management. WAIM 2005. Lecture Notes in Computer Science, vol 3739. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11563952_68
Download citation
DOI: https://doi.org/10.1007/11563952_68
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29227-2
Online ISBN: 978-3-540-32087-6
eBook Packages: Computer ScienceComputer Science (R0)Springer Nature Proceedings Computer Science
