CN100495405C - Hierarchy clustering method of successive dichotomy for document in large scale - Google Patents

Abstract

A method for clustering large-capacity text includes presenting vector space of text, calculating similarity between each two texts, embedding pattern into dimensional space and using K-means algorithm to cluster texts to be two types, carrying out successive bisect till requirement is satisfied and pattern is not divided any more.

Description

The hierarchy clustering method of successive dichotomy for document in large scale

Technical field

The invention belongs to the text message technical field, be specifically related to a kind of clustering method of extensive text.

Background technology

Along with popularizing of internet, increasing people likes the medium that are used as stating one's views with network.A lot of forums, blog, the chatroom all provides abundant public opinion information, how to become an important problems with these information of computer automatic analysis.Text cluster is a kind of technology that can utilize computing machine automatically text message to be sorted out, and through after the cluster, those articles that belong to same topic will be classified as same class, thereby make things convenient for the user to search reading.Mainly contain following text cluster methods at present:

1, K-means is a kind of fast based on the clustering algorithm of optimizing criterion.This algorithm is looked for k initial class center at the beginning at random.Then each text is assigned to the center in its that nearest class, obtain the classification of every piece of text after, recomputate the center of each class.So iterate, till the variation no longer obviously at class center.The advantage of this method is that speed is fast, is not very desirable but may obtain cluster result, and the number of class needs manually given in advance.

2, hierarchical clustering algorithm begins to regard each text as a class, then, merges two classes the most similar, till the number of class is 1 at every turn.Class is represented to the similarity that similarity between the class is used in two the most similar in these two classes texts.The advantage of this method is, can not know the number of class at the beginning, by continuous merging, constituted one tree, and the user can obtain the respective classified system according to the needs of oneself.The advantage of this method is, the number of class does not need artificial appointment at the beginning, but shortcoming is the cluster poor effect.

3, spectral clustering algorithm calculates text similarity between any two, and n piece of writing text has constituted the non-directed graph that contains n node like this, and the weight on limit is exactly the similarity between these two texts between the node.The spectral clustering algorithm is attempted this figure is embedded into the space of a low-dimensional and is got on, and makes that the bigger limit of weights can keep as much as possible among the figure, and the less limit of weights then can be ignored.After the lower dimensional space that obtains each text is represented, can carry out cluster with any one of above-mentioned two kinds of algorithms.The advantage of this algorithm is that the cluster effect is better, and shortcoming is that speed is very slow.

Summary of the invention

It is effective that purpose of the present invention proposes a kind of cluster, the clustering algorithm of the extensive text that computing velocity is fast.

The clustering method of the extensive text that the present invention proposes is an advantage of having drawn above-mentioned second the third algorithm.And improve the algorithm that forms.Its core technology is made up of two parts: figure embeds and cluster.First's technology is similar with the spectral clustering algorithm, figure need be embedded on the space of a low-dimensional, and different is that the dimension of lower dimensional space is defined as one dimension here.In fact exactly all texts have been done an ordering this moment.Then text is gathered into two classes with K-means or hierarchical clustering algorithm.Promptly figure was carried out two minutes one by one.So-called " two minutes one by one " promptly are that the subgraph that obtains is proceeded " two minutes ", enough closely then stop to cut apart up to each subgraph that obtains.

Concrete steps comprise: the vector space of text is represented; The similarity of text is calculated in twos; Figure is embedded into the one-dimensional space, and uses K-means or hierarchical clustering algorithm, and figure is carried out cluster, and figure is divided into two classes; One by one two minutes again,, figure is no longer done cutting up to meeting the demands.

Advantage of the present invention is as follows:

Because the hierarchical clustering algorithm of " two minutes one by one " is a kind of improvement algorithm, thus its advantage can from the comparison of existing algorithm draw.

1, the number of class does not need people's prior appointment.With respect to the number that needs artificial specified class in the K-means algorithm, the hierarchical clustering algorithm of " two minutes one by one " only needs to decide a threshold values in advance as stopping to cut apart condition, and is identical with hierarchical clustering algorithm.This is very easily in actual applications.

2, effective.Because the hierarchical clustering algorithm of " two minutes one by one " has adopted figure to embed, so its cluster result is very close with the spectral clustering algorithm significantly better than K-means and hierarchical clustering algorithm.

3, speed is fast.In the spectral clustering algorithm, dimensionality reduction has consumed a large amount of time, and particularly the number as fruit is many especially, the also corresponding increase of the dimension that is fallen so, and the time overhead of spectral clustering algorithm will be very big this moment.And the cluster time complexity of K-means is directly proportional with the number of class, also can consume a large amount of time when the number of class is very big.And in " two minutes one by one " method, text is divided into two classes at every turn, and the required dimension that falls is minimum one dimension, and obtains k class and only need make log and operate for k time, and this compares with the K-means algorithm with the spectral clustering algorithm, and advantage is fairly obvious.

In sum, that the hierarchical clustering algorithm of " two minutes one by one " has is automatic, effective, characteristic fast, is a kind of more outstanding text cluster algorithm.

Embodiment

Basic procedure is after text table is shown as space vector, to calculate the similarity between the text in twos, obtains figure, and carries out cluster with the hierarchical clustering algorithm of " two minutes one by one ".

1, the vector space of text is represented.

Suppose to have now n piece of writing article, occurred m speech altogether.Then every piece of vector representation that article is tieed up with a m, n piece of writing article has constituted the matrix of m * n, is designated as M.M _IjRepresent the tfidf value of i word in j piece of writing article: $M_{\mathrm{ij}}={\mathrm{tf}}_{\mathrm{ij}}\×\log \frac{n}{d{f}_i},$ Tf wherein _IjRepresent the frequency that i speech occurs in j piece of writing article, df _iExpression comprises the article number of i speech.In order to eliminate the difference of text length, text table is shown as after the vector, do normalized again, each vector is long divided by its mould:

$X_{\mathrm{ij}}=\frac{M_{\mathrm{ij}}}{\sqrt{\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{M}_{\mathrm{ij}}^2}}$

Like this, just text table has been shown as in the space mould length and is 1 vector.

2, the calculating of figure.

Calculate the similarity between the text in twos.Similarity between two texts.I.e. included angle cosine between the vector of two texts.N piece of writing text has constituted the non-directed graph that contains n node, and the weight on limit is exactly the similarity between these two texts between the node.Its similarity matrix is represented with S.

3, cut apart:

A) figure is embedded into 1 dimension space: compute matrix L=D*S*D, wherein D is a diagonal matrix, Then calculate time big eigenwert characteristic of correspondence vector y of L.The component y of y then _iJust represented the position of i piece of writing document on the one-dimensional space.

B) cut apart: ask the average y ' of vectorial y, if y _i0, then i piece of writing document is assigned to the first kind, otherwise assign to second class.

4, differentiate:

If the minimal weight on the limit in the subgraph that obtains surpasses a value given in advance, think that then all documents have belonged to same topic among this figure, do not do cutting; Otherwise got back to for the 3rd step.

Claims (1)

1, a kind of hierarchy clustering method of successive dichotomy for document in large scale, it is characterized in that concrete steps are as follows: the vector space of text is represented; The similarity of text is calculated in twos; Figure is embedded into the one-dimensional space, and uses K-means or hierarchical clustering algorithm, and figure is carried out cluster, and figure is divided into two classes; One by one two minutes again, up to meeting the demands, figure is no longer done cutting, wherein:

(1) vector space of text is represented:

Suppose to have n piece of writing article, occurred m speech altogether, then every piece of vector representation that article is tieed up with a m, n piece of writing article has constituted the matrix of m * n, is designated as M, M _IjRepresent the tfidf value of i word in j piece of writing article: $M_{\mathrm{ij}}={\mathrm{tf}}_{\mathrm{ij}}\×\log \frac{n}{{\mathrm{df}}_i},$ Tf wherein _IjRepresent the frequency that i speech occurs in j piece of writing article, df _iExpression comprises the article number of i speech; Text table is shown as after the vector, does normalized again:

$X_{\mathrm{ij}}=\frac{M_{\mathrm{ij}}}{\sqrt{\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{M}_{\mathrm{ij}}^2}}$

(2) calculating of figure:

Calculate the similarity between the text in twos, its similarity matrix is represented with S; Similarity in its Chinese is the included angle cosine between the vector of two texts, and the n of a non-directed graph node is made of n piece of writing text, and the weight on limit is two similarities between the text between the node;

(3) cut apart:

A) figure is embedded into the one-dimensional space: compute matrix L=D * S * D, wherein D is a diagonal matrix, $D_{\mathrm{ii}}=\frac{1}{\sqrt{\underset{j=1}{\overset{n}{\mathrm{\Σ}}}{S}_{\mathrm{ij}}}}$ , then calculate time big eigenwert characteristic of correspondence vector y, then the component y of y of L _iJust represented the position of i piece of writing document on the one-dimensional space;

B) cut apart: ask the average y ' of vectorial y, if y _i0, then i piece of writing document is assigned to the first kind, otherwise assign to second class;

(4) differentiate:

If the minimal weight on the limit in the subgraph that obtains surpasses a value given in advance, think that then all documents have belonged to same topic in the subgraph after cutting apart, do not do cutting; Otherwise got back to for (3) step.