Downloaded 118 times
More Related Content
Similar to FedX - Optimization Techniques for Federated Query Processing on Linked Data
FedX - Optimization Techniques for Federated Query Processing on Linked Data
- 1. FedX: Optimization Techniques forFederated Query Processing on Linked Data ISWC 2011 – October 26th Andreas Schwarte1, Peter Haase1, Katja Hose2, Ralf Schenkel2, and Michael Schmidt1 1fluidOperations AG, Walldorf 2Max-Planck Institute for Informatics, Saarbrücken
- 2. Outline • Motivation • Federated Query Processing • FedX Query Processing Model • Optimization Techniques • Evaluation • Conclusion & Outlook 2
- 3. Motivation Query processing involving multiple distributed data sources, e.g. Linked Open Data cloud New York Times DBpedia Query both data collections in an integrated way 3
- 4. Federated Query Processing Federation mediator at the server Virtual integration of (remote) data sources Communication via SPARQL protocol SPARQL Data Source SPARQL Federation Query Data Source Mediator SPARQL Data Source 4
- 5. Federated Query Processing Example Query Find US presidents and associated news articles SELECT ?President ?Party ?TopicPage WHERE { ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . ?nytPresident owl:sameAs ?President . ?President dbpedia:party ?Party . ?nytPresident nytimes:topicPage ?TopicPage . } 5
- 6. Federated Query Processing Query: SELECT?President ?Party ?TopicPage WHERE { ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . ?nytPresident owl:sameAs ?President . SPARQL ... } DBpedia Federation “Barack Obama” Mediator “George W. Bush” ... ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . SPARQL “Barack Obama” “George W. Bush” NYTimes ... 6
- 7. Federated Query Processing Query: SELECT ?President ?Party ?TopicPage WHERE { ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . ?nytPresident owl:sameAs ?President . SPARQL ... } DBpedia Federation yago:Obama Mediator ?nytPresident owl:sameAs “Barack Obama” . SPARQL “Barack Obama” Input: “George W. Bush” NYTimes ... “Barack Obama”, yago:Obama nyt:Obama Output: “Barack Obama”, nyt:Obama 7
- 8. Federated Query Processing Query: SELECT ?President ?Party ?TopicPage WHERE { ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . SPARQL ... } DBpedia Federation Mediator ?nytPresident owl:sameAs “George W. Bush” . SPARQL “Barack Obama” Input: “George W. Bush” NYTimes ... “Barack Obama”, yago:Obama nyt:Bush Output: “Barack Obama”, nyt:Obama “George W. Bush”, nyt:Bush ... and so on for the other intermediate mappings and triple patterns ... 8
- 9. FedX Query ProcessingModel Scenario: Efficient SPARQL query processing on multiple distributed sources Data sources are known and accessible as SPARQL endpoints • FedX is designed to be fully compatible with SPARQL 1.0 No a-priori knowledge about data sources • No local preprocessing of the data sources required • No need for pre-computed statistics On-demand federation setup 9
- 10. Challenges to FederatedQuery Processing 1) Involve only relevant sources in the evaluation Avoid: Subqueries are sent to all sources, although potentially irrelevant 2) Compute joins close to the data Avoid: All joins are executed locally in a nested loop fashion 3) Reduce remote communication Avoid: Nested loop join that causes many remote requests 10
- 11. Optimization Techniques 1. SourceSelection: Idea: Triple patterns are annotated with relevant sources • Sources that can contribute information for a particular triple pattern • SPARQL ASK requests in conjunction with a local cache • After a warm-up period the cache learns the capabilities of the data sources During Source Selection remote requests can be avoided 2. Exclusive Groups: Idea: Group triple patterns with the same single relevant source • Evaluation in a single (remote) subquery • Push join to the endpoint 11
- 12. Optimization Techniques (2) Example:Source Selection + Exclusive Groups SELECT ?President ?Party ?TopicPage WHERE { Source Selection ?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates . @ DBpedia ?President dbpedia:party ?Party . @ DBpedia Exclusive Group ?nytPresident owl:sameAs ?President . @ DBpedia, NYTimes ?nytPresident nytimes:topicPage ?TopicPage . @ NYTimes } Advantages: Avoid sending subqueries to sources that are not relevant Delegate joins to the endpoint by forming exclusive groups (i.e. executing the respective patterns in a single subquery) 12
- 13. Optimization Techniques (3) 3.Join Order: Idea: Iteratively determine the join order based on count-heuristic: • Count free variables of triple patterns and groups • Consider "resolved" variable mappings from earlier iteration 4. Bound Joins: Idea: Compute joins in a block nested loop fashion: • Reduce the number of requests by "vectored" evaluation of a set of input bindings • Renaming and Post-Processing technique for SPARQL 1.0 13
- 14. Optimization Techniques (4) Example:Bound Joins SELECT ?President ?Party ?TopicPage WHERE { ?President rdf:type dbpedia:PresidentsOfTheUnitedStates . ?President dbpedia:party ?Party . ?nytPresident owl:sameAs ?President . ?nytPresident nytimes:topicPage ?TopicPage . } Assume that the following intermediate results have been computed as input for the last triple pattern Block Input Before (NLJ) “Barack Obama” SELECT ?TopicPage WHERE { “Barack Obama” nytimes:topicPage ?TopicPage } “George W. Bush” SELECT ?TopicPage WHERE { “George W. Bush” nytimes:topicPage ?TopicPage } … … Now: Evaluation in a single remote request using a SPARQL UNION construct + local post processing (SPARQL 1.0) 14
- 15. Optimization Example 1 SPARQL Query 2 Unoptimized Internal Representation Compute Micronutrients using Drugbank and KEGG 4x Local Join SELECT ?drug ?title WHERE { = ?drug drugbank:drugCategory drugbank-cat:micronutrient . ?drug drugbank:casRegistryNumber ?id . 4x NLJ ?keggDrug rdf:type kegg:Drug . ?keggDrug bio2rdf:xRef ?id . ?keggDrug dc:title ?title . } 3 Optimized Internal Representation Exlusive Group Remote Join 15
- 16. Evaluation Based on FedBenchbenchmark suite 14 queries from the Cross Domain (CD) and Life Science (LS) collections Real-World Data from the Linked Open Data cloud Federation with 5 (CD) and 4 (LS) data sources Queries vary in complexity, size, structure, and sources involved Benchmark environment HP Proliant 2GHz 4Core, 32GB RAM 20GB RAM for server (federation mediator) Local copies of the SPARQL endpoint to ensure reproducibility and reliability of the service • Provided by the FedBench Framework 16
- 17. Evaluation (2) a) Evaluation times of Cross Domain (CD) and Life Science (LS) queries 17
- 18. Evaluation (3) b) Number of requests sent to the endpoints Runtimes AliBaba: >600s DARQ: >600s FedX: 0.109s Runtimes AliBaba: >600s DARQ: 133s FedX: 1.4s 18
- 19. FedX – TheBigger Picture Information Workbench: Integration of Virtualized Data Sources as a Service Application Layer Semantic Wiki Collaboration Reporting & Analytics Visual Exploration Virtualization Layer Transparent & On-Demand Integration of Data Sources Data Layer SPARQL SPARQL SPARQL Endpoint Endpoint Endpoint Data Registries Metadata CKAN, data.gov, etc. Registry Data Source Data Source + Enterprise Data Data Source 19
- 20. Conclusion and Outlook FedX:Efficient SPARQL query processing in federated setting Available as open source software: http://www.fluidops.com/fedx • Implementation as a Sesame SAIL Novel join processing strategies, grouping techniques, source selection • Significant improvement of query performance compared to state-of-the-art systems On-demand federation setup without any preprocessing of data sources Compatible with existing SPARQL 1.0 endpoints Outlook SPARQL 1.1 Federation Extension in Sesame 2.6 • Ongoing integration of FedX’ optimization techniques into Sesame Federation Extensions in FedX • SERVICE keyword for optional user input to improve source selection • BINDINGS clauses for bound joins (for SPARQL 1.1 endpoints) (Remote) statistics to improve join order (e.g. VoID) 20
- 21. Visit our exhibitorbooth or arrange a private live demo! Further information on FedX http://www.fluidops.com/fedx THANK YOU CONTACT: fluid Operations Altrottstr. 31 Walldorf, Germany Email: [email protected] website: www.fluidOps.com Tel.: +49 6227 3846-527
- 22. Query Cross Domain3 (CD3) Find US presidents, their party and associated news SELECT ?president ?party ?page WHERE { ?president <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/President> . ?president <http://dbpedia.org/ontology/nationality> <http://dbpedia.org/resource/United_States> . ?president <http://dbpedia.org/ontology/party> ?party . ?x <http://data.nytimes.com/elements/topicPage> ?page . ?x <http://www.w3.org/2002/07/owl#sameAs> ?president . }
- 23. Query Life Science3 (LS3) Find drugs and interaction drugs (+ the effect) SELECT ?Drug ?IntDrug ?IntEffect WHERE { ?Drug <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Drug> . ?y <http://www.w3.org/2002/07/owl#sameAs> ?Drug . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/interactionDrug1> ?y . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/interactionDrug2> ?IntDrug . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/text> ?IntEffect . }
Editor's Notes
- #4 Example scenarioDBpedia and New York Times collectionsDBpedia as structured knowledge baseNew York Times as a news provider
- #11 Minimize the number of remote requests due to intelligent algebraic query rewritings
- #18 Query Performance improved significantly for most queriesImprovements by more than an order of magnitudeNo timeouts, no evaluation errorsQuery CD3SELECT ?president ?party ?page WHERE { ?president <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/President> . ?president <http://dbpedia.org/ontology/nationality> <http://dbpedia.org/resource/United_States> . ?president <http://dbpedia.org/ontology/party> ?party . ?x <http://data.nytimes.com/elements/topicPage> ?page . ?x <http://www.w3.org/2002/07/owl#sameAs> ?president .}Query LS3SELECT ?Drug ?IntDrug ?IntEffect WHERE { ?Drug <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://dbpedia.org/ontology/Drug> . ?y <http://www.w3.org/2002/07/owl#sameAs> ?Drug . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/interactionDrug1> ?y . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/interactionDrug2> ?IntDrug . ?Int <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/text> ?IntEffect . }