AN OVERBROAD PATENT ON Methods of detecting similar documents - This application from Nhn seeks to patent the idea of...Extracting an entity from two or more web documents, calculating weights based on the two or more documents, and determining a similarity between the two or more documents! 10 minutes of your time can help narrow US patent applications before they become patents. Follow @askpatents on twitter to help.

QUESTION - Have you seen anything that was published before 5/4/2011 that discusses:

• Methods for detecting similar documents by matching and comparing extracted entities from the documents

If so, please submit evidence of prior art as an answer to this question. We welcome multiple answers from the same individual.

EXTRA CREDIT - A characteristic index for each of the web documents is calculated based on weight of the corresponding entity. A similarity between the web documents is determined based on the calculated characteristic indices

TITLE: Method and device to detect similar documents

Summary: [Translated from Legalese into English] Extracting an entity from two web documents, calculating weights for the entity based on importance contribution elements corresponding to both of the two web documents, and determining a similarity between the two web documents based on the calculated weights.

• Publication Number: US 20120284270 A1
• Application Number: US 13/462,592
• Assignee: Nhn
• Prior Art Date: Seeking prior Art predating 5/4/2011
• Open for Challenge at USPTO: Open through 5/7/2013
• Link to Google Prior Art Search - "Find Prior Art"

Claim 1 requires each and every step below:

A method that uses a processor for detecting similar documents, comprising:

1. Extracting an entity from each of a first web document and a second web document;

2. Determining an importance contribution element corresponding to each of the web documents;

3. Calculating, using the processor, weights for each entity based on the determined importance contribution elements; and

4. Determining whether the web documents are similar documents based on the calculated weights. Determining whether the web documents are similar documents based on the calculated weights.

In English this means:

A method for detecting similar documents, the method comprises the following steps:

1. Extracting an entity from two web documents;

2. Determining an importance contribution element corresponding to both the web documents;

3. Calculating weights for each entity based on the determined importance contribution elements; and

4. Determining whether the two web documents are similar documents based on the calculated weights.

Good prior art would be evidence of a system that did each and every one of these steps prior to 5/4/2011

"You're probably aware of ten pieces of art that meet this criteria already... separately, the applicant is claiming Each importance contribution element comprises at least one of a hash value of the corresponding entity, and a frequency of which the corresponding entity is duplicated within the corresponding web document. Further, the calculating the weights comprises using an inverted document frequency concept, the inverted document frequency concept increasing a weight of each entity in inverse proportion to decreasing frequency of duplication of the respective entity. "

"Method and device to detect similar documents" from the Applicant

What is good prior art? Please see our FAQ.

Want to help? Please vote or comment on submissions below. We welcome you to post your own request for prior art on other questionable US Patent Applications.