HELP SAVE Information Retrieval for future generations.
This application from FACEBOOK seeks to patent distributed database query engine for low query latency database analyses.
10 minutes of your time can help narrow this EP patent application before it becomes a patent.
QUESTION - Have you seen anything that was published before 07.01.2013 that discusses:
- the benefits of low-latency queries on a very large amount of data stored in a large scale storage cluster (e.g. a Hadoop cluster), which stores metadata in a system catalog (e.g. Hive Metastore);
- a distributed query engine based on a serving tree computation framework including a gateway server and a plurality of work nodes. The gateway server divides a query task into partial tasks. Each worker node of the engine handles a partial task to generate an intermediate query result in memory. The intermediate query result can be an approximate intermediate result generated by scanning a portion of the related data.
- The gateway server receives the intermediate query results and constructs a query result for the query task based on the intermediate query results.
If you've ever seen anything like this before, please submit evidence of prior art as an answer to this question -- one piece of prior art per answer. We welcome multiple answers from a single individual.
EXTRA CREDIT - Include a reference to anything that meets all the criteria to the question above AND solves the problem of high query latency for the Hive system. In particular, for interactive analyses.
The patent application in question is:
TITLE: SYSTEM AND METHOD FOR DISTRIBUTED DATABASE QUERY ENGINES
- Application Number: [16176465.9] (https://www.google.com/patents/US20140195558)
- Assignee: Facebook
- Prior Art Date: Seeking prior Art predating [07.01.2013]
Claim 12 requires each and every step below:
A method comprising:
receiving from a client device a database query for a database containing data stored in a distributed storage cluster that has a plurality of data nodes; dividing the database query into a plurality of partial queries;
sending each of the partial queries to a respective worker node of a plurality of worker nodes, wherein each worker node is a service running on a data node of the distributed storage cluster;
retrieving a plurality of intermediate results for the partial queries from the worker nodes, wherein each intermediate result is processed by a respective worker node of the worker nodes by scanning related data stored in a data node on which the perspective worker node runs; and
generating a query result based on the plurality of intermediate results.
A system comprising:
a gateway server configured to generate a plurality of partial queries from a database query for a database containing data stored in a distributed storage cluster that has a plurality of data nodes, and to construct a query result based on a plurality of intermediate results; and
a plurality of worker nodes, wherein each worker node of the plurality of worker nodes is configured to process a respective partial query of the plurality of partial queries by scanning data related to the respective partial query and stored on at least one data node of the distributed storage cluster,
and wherein each worker node of the plurality of worker nodes is further configured to generate an intermediate result of the plurality of intermediate results that is stored in a memory of that worker node.