US20110191327A1

US20110191327A1 - Method for Human Ranking of Search Results

Info

Publication number: US20110191327A1
Application number: US12/944,842
Authority: US
Inventors: Bryant Christopher Lee
Original assignee: Advanced Research LLC
Current assignee: Advanced Research LLC
Priority date: 2010-01-31
Filing date: 2010-11-12
Publication date: 2011-08-04
Also published as: US8099406B2; US20110191317A1

Abstract

A search engine may perform a search for a user search query. A rankings of result items may be returned as a result of the search. The user may enter new suggested rankings for some of the items. The new suggested rankings may change the rankings of result items when other users search for the same search query at a later time.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. provisional patent application Ser. No. 61/299,994 filed Jan. 31, 2010 and U.S. provisional patent application Ser. No. 61/302,117 filed Feb. 6, 2010, which are both incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to search engines, and more particularly, to methods of ranking search results and to user interfaces that are used to display search results.

BACKGROUND

Search engines assist users in finding relevant documents or other informational content from among a larger corpus. A common application of search engines is to assist users in finding information, documents, and files that are located on a network. A common application of search engines is to assist users in finding information, documents, and files on the World Wide Web (“Web”), which comprises a massive network of Web documents. The user submits a word or phrase he or she is interested in, called a search query, and the search engine seeks to provide a list of high quality Web documents that are relevant to that query. The list of Web documents is usually ranked with higher-quality and more relevant documents appearing first.
The ranking is conventionally done either by examination of the contents of the Web documents or by examining the link structure of the Web documents. For example, a search engine may examine the words on a web document and assign an importance ranking according to how similar or related the words on the web document are to the search query. For example, a search engine may examine the link structure of a web document and assign an importance ranking according to which other documents link to the document and which other documents the document links to. For example, a search engine may assign an importance ranking to a document based on the user's search history. For example, a search engine may assign an importance ranking to a document based on whether the user has previously navigated to and made a purchase from that document.
One problem with search engines is that the methods of ranking documents often do not produce rankings that users believe to be an accurate reflection of relevance or quality. Often, a Web document that users believe to be the highest quality and most relevant is not listed first or second or anywhere near the top of the results list. A second problem is that users who notice this problem do not have a means to fix it so that future users who search for the same query can receive better search results.

SUMMARY OF THE INVENTION

One aspect of the invention is directed to a method that includes receiving at a computer system a search query from a user. The method includes performing a search over a set of data in response to the search query. The method includes generating a rankings of a set of result items from the search over the set of data. The method includes adjusting the rankings of the set of result items by a method of using human suggested rankings comprising: retrieving from a database previously suggested rankings of result items that were provided by other users; and adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. The method includes displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database to be available for potential later use.
One aspect of the invention is directed to a method that includes receiving at a computer system a search query from a user. The method includes performing a search over a set of data in response to the search query. The method includes generating a rankings of a set of result items from the search over the set of data. The method includes adjusting the rankings of the set of result items by a method of using human suggested rankings comprising: retrieving from a database previously suggested rankings of result items that were provided by other users; the previously suggested rankings of result items containing suggested rankings for some of the result items in the set of result items; the other users having conducted a search using the same search query earlier in time; the other users having entered the suggested rankings of result items after being displayed with the results of their search using the same search query earlier in time; and adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. The method includes displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database to be available for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.
One aspect of the invention is directed to adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items by a method comprising: re-ranking one or more result items in the set of result items to the last user-suggested ranking the one or more result items received from the other users, the other users having conducted a search using the same search query earlier in time.
One aspect of the invention is directed to adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items by a method comprising: re-ranking one or more result items in the set of result items to an average user-suggested ranking the one or more result items have received from the other users, the other users having conducted a search using the same search query earlier in time.
One aspect of the invention is directed to adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items by a method comprising: assigning two weights, the first weight being determined based on user-suggested rankings one or more result items in the set of result items have received from the other users, the other users having conducted a search using the same search query earlier in time, and the second weight being determined by the content, link structure, relevancy, quality, or other substantive qualities of one or more result items in the set of result items, and the two weights being used in combination to adjust the rankings of the one or more result items in the set of result items.
The invention improves on the prior art in at least the following ways: it may improve the quality of rankings of the set of result items returned as a result of a user search so that more relevant result items are ranked higher and less relevant result items are ranked lower; and it allows users to improve the quality of rankings of the set of result items for later users.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an exemplary environment in which one embodiment of the present invention may operate;

FIG. 2 is a flow diagram illustrating operations that may be performed for human ranking of search results in one embodiment of the present invention;

FIG. 3 is a diagram illustrating an exemplary results document that could be produced in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an exemplary network environment for implementation of one embodiment of the present invention. The network environment 100 may include multiple clients 110 connected to one or more servers 120 via a network 140. Network 140 may include a local area network (LAN), a wide area network (WAN), a telephone network, such as the Public Switched Telephone Network (PSTN), an intranet, the Internet, or a combination of networks. Two clients 110 and one server 120 have been illustrated for simplicity, though in practice there may be more or fewer clients and servers.
Clients 110 may be personal computers, digital assistants, personal digital assistants, cellular phones, mobile phones, smart phones, pagers, digital tablets, laptop computers, Internet appliances, and other processor-based devices. Clients may be client computing devices. Client computing devices may be personal computers, digital assistants, personal digital assistants, cellular phones, mobile phones, smart phones, pagers, digital tablets, laptop computers, Internet appliances, and other processor-based devices. Servers 120 may include entities that process, search, and/or maintain documents or files in a manner consistent with the principles of the invention. Server 120 may be implemented as a number of networked server devices, though it is illustrated as a single entity.
Server 120 may include a search engine 125 that can receive search queries from clients 110. In one embodiment, the search engine 125 may function as a query-based web page search engine. The search engine may cause the output, such as the display, of search results on the client 110. The search results may be in the form a results document. The results document may contain a set of result items.
In one embodiment, the search engine 125 processes search queries for documents located on the network. It may process queries by searching the content database 127 in which documents and/or index information about documents on the network are stored. It may return a set of documents and/or index information (called the set of result items) and may rank this set of result items according to relevancy and quality.
In one embodiment, the search engine then accesses the user rankings database 128 to access previous user-suggested rankings of result items for the search query. A user-suggested ranking of a result item is a ranking of the result item that was suggested by a user. The search engine may adjust the results rankings based on these suggested rankings.
Those skilled in the art will appreciate that the documents and/or index information about documents in the content database 127 and the user-suggested ranks in the user rankings database 128 may be stored in a single database or in more or fewer databases. Alternatively, they may be stored in a networked system of databases. It is for simplicity of illustration that these elements are described as residing in two different databases. The term database as used in this patent may refer to a computer database or any data storage or network of data storage.
FIG. 2 is a flow chart illustrating an example method 200 that may be performed by search engine 125 consistent with aspects of the invention. Search engine 125 may receive a search query from a user (act 201). The search query may be received at the server 120. The user may provide the search query either at the server 120 or at a client 110 that is connected to the server 120 over the network 140. In response, the search engine may perform a search over a set of data, where the set of data is at the computer system or accessible to the computer system (act 202). The set of data that is searched over may be the content database 127. The search engine may generate a rankings of a set of result items returned from the search over the set of data (act 203). The step of generating a rankings of a set of result items may be performed by any conventional or nonconventional method of generating a rankings of a set of result items such as but not limited to: analysis of the content of the document and other documents or analysis of the link structure of the document and other documents. The search engine may adjust the rankings of the set of result items by a method of using human suggested rankings (act 204). Act 204 may include retrieving from the user rankings database 128 a previously suggested rankings of result items that were provided by a other users. The previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items. The previously suggested rankings of result items in the user rankings database 128 may have been provided by the other users, the other users having conducted a search using the same search query earlier in time. The previously suggested rankings of result items in the database may have been provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time. Act 204 may include adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items. The search engine may display the adjusted rankings of the set of result items in a results document (act 205). The user may enter suggested rankings for zero, one, or more result items in the results document (act 206). Such suggested rankings are stored in the user rankings database 128 to be available for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.
A suggested ranking for a result item in the user rankings database 128 may be a database entry with information regarding: the user who made the user-suggested ranking, the suggested ranking of the result item, the name or identifying information of the result item, the search query used by the user who made the user-suggested ranking to obtain the result item, the date and time that the suggested ranking was entered, and other information. A suggested ranking of a result item may be a suggestion that the result item be ranked at a certain ranking. For example, a suggested ranking for website A may be a suggestion that the website A be ranked third in a set of result items returned when a user conducts a search using search query B. A suggested ranking is not merely an indication that a user recommended or did not recommend a website; is not merely an indication that a user voted up or voted down a website; is not merely a rating of a website on a scale for example from 1 to 5. Rather a suggested ranking indicates that a user made a suggestion that the result item be ranked at a certain ranking in the set of result items.
In this patent, the term “same search query” is defined as a search query that is similar or substantially the same. Two search queries need not be exactly identical to be the “same search query.” For example, a search query of “Michael Jordan” may be the “same search query” as “the Michael Jordan,” “person Michael Jordan,” “Michael Jordan basketball,” “Michael J basketball player,” and so on in some embodiments of the invention.
In one embodiment, the step in act 204 of adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items may be accomplished by a method comprising: re-ranking one or more result items in the set of result items to the last user-suggested ranking the one or more result items received from the other users, the other users having conducted a search using the same search query earlier in time. Additional procedures may be used in this adjustment method as follows: When a result item is re-ranked, it may displace a result item that was ranked at that location in the original ranking, the original ranking being the ranking of the result items before the adjustment; said result item is moved down one rank if the re-ranked result item was ranked below it in the original ranking, and said result item is moved up one rank if the re-ranked result item was above it in the original ranking; this movement of said first result item may displace a second result item; the second result item is moved down one rank if the first result item was moved down, and the second result item is moved up if the first result item was moved up; the process is repeated for a third, fourth, fifth, and so on result items as necessary until no result items need to be re-ranked. Other procedures may be used to deal with the situation in which a result item that is re-ranked displaces a result item that was ranked at that location in the original ranking. For example, the result item that was ranked at that location in the original ranking could be deleted from the set of result items.
In one embodiment, the step in act 204 of adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items may be accomplished by a method comprising: re-ranking one or more result items in the set of result items to an average user-suggested ranking the one or more result items have received from the other users, the other users having conducted a search using the same search query earlier in time. The average may be a mean, median, mode, weighted average, or other kind of average. Additional procedures may be used in this adjustment method as follows: When a result item is re-ranked, it may displace a result item that was ranked at that location in the original ranking, the original ranking being the ranking of the result items before the adjustment; said result item is moved down one rank if the re-ranked result item was ranked below it in the original ranking, and said result item is moved up one rank if the re-ranked result item was above it in the original ranking; this movement of said first result item may displace a second result item; the second result item is moved down one rank if the first result item was moved down, and the second result item is moved up if the first result item was moved up; the process is repeated for a third, fourth, fifth, and so on result items as necessary until no result items need to be re-ranked. Other procedures may be used to deal with the situation in which a result item that is re-ranked displaces a result item that was ranked at that location in the original ranking. For example, the result item that was ranked at that location in the original ranking could be deleted from the set of result items.
In one embodiment, the step in act 204 of adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items may be accomplished by a method comprising: assigning two weights, the first weight being determined based on user-suggested rankings one or more result items in the set of result items have received from the other users, the other users having conducted a search using the same search query earlier in time, and the second weight being determined by the content, link structure, relevancy, quality, or other substantive qualities of one or more result items in the set of result items, and the two weights being used in combination to adjust the rankings of the one or more result items in the set of result items. Additional procedures may be used in this adjustment method as follows: When a result item is re-ranked, it may displace a result item that was ranked at that location in the original ranking, the original ranking being the ranking of the result items before the adjustment; said result item is moved down one rank if the re-ranked result item was ranked below it in the original ranking, and said result item is moved up one rank if the re-ranked result item was above it in the original ranking; this movement of said first result item may displace a second result item; the second result item is moved down one rank if the first result item was moved down, and the second result item is moved up if the first result item was moved up; the process is repeated for a third, fourth, fifth, and so on result items as necessary until no result items need to be re-ranked. Other procedures may be used to deal with the situation in which a result item that is re-ranked displaces a result item that was ranked at that location in the original ranking. For example, the result item that was ranked at that location in the original ranking could be deleted from the set of result items.
In one embodiment, when a user suggests a new ranking for an item, the item is re-ranked to the suggested ranking, the other items are displaced accordingly, and the rankings of all or less than all of the items in the search results list are stored in the user rankings database. In this embodiment, adjusting the rankings of the search items according to user-suggested rankings may be accomplished by retrieving the rankings for all or less than all of the items from the user rankings database.
Those skilled in the art will recognize that adjusting the rankings of the search items according to user suggested rankings (act 204) may be performed according to many different methods and processes. The foregoing description of preferred embodiments of act 204 provides illustration and description, but is not intended to be exhaustive or to limit the implementation of act 204 to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. For example, although many of the operations described above were described in a particular order, many of the operations are amenable to being performed simultaneously or in different orders to still achieve the same or equivalent results.
In one embodiment, the user may not suggest rankings of items unless the user has a user account and is logged in to the user account. In one embodiment, the user may not suggest rankings of items unless the user has editing permissions. Editing permissions may be given to all or less than all of the users of the invention. Editing permissions may be given or taken away from a user at different times. Some users may receive editing permissions when they create a user account. Other users may not receive editing permissions when they create a user account. There may be different levels of editing permissions.
In one embodiment, the user must pass a test to show he or she is a human and not a computer program before the user can enter suggested rankings for the result items in the results document.
In one embodiment, the user may only enter suggested rankings for the result items in the results document a limited number of times over a limited time period.
In one embodiment, the user and the other users are divided into one or more classes. The user and the other users may be in multiple classes at once. The user may be in a class or more than one class. In this embodiment, suggested rankings entered by a user are only used to adjust rankings made in response to another user's search query if the two users are of the same class, for example both users being of the class “interested in sports.” It may be appreciated that users may be assigned to classes in many ways and not just according to interest area. Other kinds of classes may be: users who are female, users who are over the age of 25, users who have a college education, users who work in the computer software industry, and so on. It may be appreciated that classes may be automatically assigned, assigned by running a computer program, assigned by manual intervention, or specified by the user. In this embodiment where the user and the other users are divided into one or more classes, act 204 may be performed by a method comprising: retrieving from the database a previously suggested rankings of result items that were provided by a other users; wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time, the other users being in the same class as the user; wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items.
FIG. 3 is a diagram illustrating an exemplary results document 300 displaying the ranked results returned in response to a search query. The result items on results document 300 are exemplary result items returned by search engine 125 for the exemplary search “Michael Jordan.” In one embodiment, each result item may be listed with a title 301, a summary of relevant content 302, and URL 303. The title 301 may be linked to the document. In other embodiments, the result items may be listed with more, less, or different information.
In one embodiment, the user may enter suggested rankings for one or more result items listed on the results document by selecting and moving the one or more result items to a higher or lower ranking. In one embodiment, this may be accomplished by clicking on the title 301, summary of relevant content 302, or URL 303 to select the result item and then moving the result item. In one embodiment, some or all result items may be accompanied by a move button 304 such that the user may click the move button and then drag the item to a new location in the list that corresponds to a new ranking. Selecting and moving the one or more result items to a higher or lower ranking can be accomplished in many ways, using many different kinds of user interface, and the above examples of ways to select and move a result item are exemplary and not limiting. The new user-suggested ranking may be stored as a new user-suggested ranking in the user rankings database 128.
In one embodiment, the results document includes user interface elements next to one or more result items. The user may enter suggested rankings for the one or more result items via the user interface elements. In one embodiment, some or all result items may be accompanied by a text input box where the user may enter a new suggested ranking. The new user-suggested ranking may be stored as a new user-suggested ranking in the user rankings database 128.
Those skilled in the art will recognize that many different kinds of user interface actions may be used to allow the user to enter a new user-suggested ranking and cause that ranking to be stored in the user rankings database 128. The foregoing description of preferred embodiments provides illustration and description, but is not intended to be exhaustive or to limit implementation to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention.
In one embodiment, a merge component 310 is included on the results document. In one embodiment, the search query received from the user is one search query from a set of merged search queries, and adjusting the rankings of the set of result items by a method of using human suggested rankings is performed by a method comprising: retrieving from the database a previously suggested rankings of result items that were provided by a other users; wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using any search query from the set of merged search queries earlier in time; wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using any search query from the set of merged search queries earlier in time; adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items.
The results document may include a merge component 310 that displays information about one or more sets of merged search queries or a link to a merge component 310 that displays information about one or more sets of merged search queries. The results document may include user interface elements that may be used to add one or more search queries to a set of merged search queries or to remove one or more search queries from a set of merged search queries. In one embodiment of the merge component 310, a text input field 311 allows entering a search query and a merge button 312 causes that search query to be added to the set of merged search queries 313. The set of merged search queries may include the search query submitted by the user to generate the current results document. Each search query in the set of merged search queries may return the same results document, and new user suggested rankings of items on the results document may be used to adjust the rankings of items when searching for any of the merged search queries. An unmerge button 314 next to each currently merged search query may, when clicked, cause the search query to be removed from the set of merged search queries.
In one embodiment, the results document includes a version control component 320 that displays information about one or more previous versions of the results document or a link to a version control component 320 that displays information about one or more previous versions of the results document. Each of the one or more previous versions of the results document may display a past adjusted rankings of the set of result items, the past adjusted rankings of the set of result items being generated at the current time or having been generated at a previous time by a method of using human suggested rankings comprising: retrieving from the database a previously suggested rankings of result items that were provided by a other users; wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items; wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time; wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time; wherein the other users entered the suggested rankings of result items prior to a specific time, the specific time being associated with the one or more previous versions of the results document; adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items.
In one embodiment of the version control component 320, the component may include information about the current version of the results document in a current version display 321. The component may include date information 322, time information 323, and user information 324 for previous versions of the results document, a previous version of the results document being a document displaying the ranking of the set of result items adjusted according to the user-suggested rankings of items provided by users up to a specific time that is earlier than the current time, the specific time being associated with the previous version of the results document. The date information 322 and time information 323 may reflect the specific earlier time that is associated with the previous version of the results document. The user information 324 may reflect the last user to enter a user suggested ranking before or at that specific time. Each of the current and previous version listings may include a first checkbox 325 and a second checkbox 326 allowing the selection of two previous versions of the results document simultaneously. The user may choose to view a differences between two or more previous versions of the results document. A display of the differences between the two selected previous versions may appear when the user clicks the compare button 327. The show button 328 may cause a previous version to be displayed, the user choosing which previous version to display using the first checkbox 325 or another interface element.
In one embodiment, the results document includes a discussion component 330 for user discussion or a link to a discussion component 330 for user discussion, the discussion component for user discussion comprising a wiki, a forum, a bulletin board, a chat room, or other interface component configured for user discussion.

CONCLUSION

By adjusting search result rankings with user-suggested rankings, users may receive rankings of result items that are more accurate with respect to the rankings that users believe the result items should have.
The foregoing description of preferred embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. For example, although many of the operations described above were described in a particular order, many of the operations are amenable to being performed simultaneously or in different orders to still achieve the same or equivalent results.

Claims

1. A method for human ranking of search engine results, the method performed by a computer system having access to a database, the method comprising:

receiving at the computer system a search query from a user;

wherein the search query is provided by the user either at the computer system or at a client computing device connected to the computer system over a network;

performing a search over a set of data in response to the search query, the set of data being at the computer system or accessible to the computer system;

generating a rankings of a set of result items returned from the search over the set of data;

adjusting the rankings of the set of result items by a method of using human suggested rankings comprising:

retrieving from the database a previously suggested rankings of result items that were provided by a other users;

wherein the previously suggested rankings of result items contains suggested rankings for zero, one, or more of the result items in the set of result items;

wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time;

wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using the same search query earlier in time;

adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items;

displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database to be available for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.

2. The method of claim 1, wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: re-ranking one or more result items in the set of result items to the last user-suggested ranking the one or more result items received from the other users, the other users having conducted a search using the same search query earlier in time.

3. The method of claim 1, wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: re-ranking one or more result items in the set of result items to an average user-suggested ranking the one or more result items have received from the other users, the other users having conducted a search using the same search query earlier in time.

4. The method of claim 1, wherein adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items from the other users to obtain an adjusted rankings of the set of result items is performed by a method comprising: assigning two weights, the first weight being determined based on user-suggested rankings one or more result items in the set of result items have received from the other users, the other users having conducted a search using the same search query earlier in time, and the second weight being determined by the content, link structure, relevancy, quality, or other substantive qualities of one or more result items in the set of result items, and the two weights being used in combination to adjust the rankings of the one or more result items in the set of result items.

5. The method of claim 1, wherein the results document includes a discussion component for user discussion or a link to a discussion component for user discussion, the discussion component for user discussion comprising a wiki, a forum, a bulletin board, a chat room, or other interface component configured for user discussion.

6. The method of claim 1, wherein the results document includes a version control component that displays information about one or more previous versions of the results document or a link to a version control component that displays information about one or more previous versions of the results document;

wherein each of the one or more previous versions of the results document display a past adjusted rankings of the set of result items, the past adjusted rankings of the set of result items being generated at the current time or having been generated at a previous time by a method of using human suggested rankings comprising:

wherein the other users entered the suggested rankings of result items prior to a specific time, the specific time being associated with the one or more previous versions of the results document;

adjusting the rankings of the set of result items based on the retrieved previously suggested rankings of result items that were provided by the other users to obtain an adjusted rankings of the set of result items.

7. The method of claim 6, wherein the user may choose to view a differences between two or more previous versions of the results document.

8. The method of claim 1, wherein the results document includes user interface elements next to one or more result items;

wherein the user may enter suggested rankings for the one or more result items via the user interface elements.

9. The method of claim 1, wherein the user may enter suggested rankings for one or more result items listed on the results document by selecting and moving the one or more result items to a higher or lower ranking.

10. The method of claim 1, wherein the user may only enter suggested rankings for the result items in the results document if the user has editing permissions.

11. The method of claim 1, wherein the user must pass a test to show he or she is a human and not a computer program before the user can enter suggested rankings for the result items in the results document.

12. The method of claim 1, wherein the user may only enter suggested rankings for the result items in the results document a limited number of times over a limited time period.

13. The method of claim 1, wherein the search query received from the user is one search query from a set of merged search queries;

wherein adjusting the rankings of the set of result items by a method of using human suggested rankings is instead performed by a method comprising:

wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using any search query from the set of merged search queries earlier in time;

wherein the previously suggested rankings of result items in the database were provided by the other users by entering suggested rankings of result items, the result items having been displayed in a prior results document that was returned after the other users conducted a search using any search query from the set of merged search queries earlier in time;

14. The method of claim 13, wherein the results document includes a merge component that displays information about one or more sets of merged search queries or a link to a merge component that displays information about one or more sets of merged search queries.

15. The method of claim 13, wherein the results document includes user interface elements that may be used to add one or more search queries to a set of merged search queries or to remove one or more search queries from a set of merged search queries.

16. The method of claim 1, wherein the user and the other users are divided into one or more classes;

wherein the user is in a class;

wherein the previously suggested rankings of result items in the database were provided by the other users, the other users having conducted a search using the same search query earlier in time, the other users being in the same class as the user;

17. A system including at least one computer system having access to a database, the system configured to:

receive at the computer system a search query from a user;

perform a search over a set of data in response to the search query, the set of data being at the computer system or accessible to the computer system;

generate a rankings of a set of result items returned from the search over the set of data;

adjust the rankings of the set of result items by a method of using human suggested rankings comprising:

display the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.

18. A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method of human ranking of search engine results, said method using a computer system having access to a database, said method comprising:

receiving at the computer system a search query from a user;

displaying the adjusted rankings of the set of result items in a results document, wherein the user may enter suggested rankings for zero, one, or more result items in the results document, such suggested rankings to be stored in the database for potential later use in adjusting the rankings of a later set of result items returned in response to a later search queries of a later users.