Skip to main content
SpringerLink
Log in

Sparklify: A Scalable Software Component for Efficient Evaluation of SPARQL Queries over Distributed RDF Datasets

  • Conference paper
  • First Online:
The Semantic Web – ISWC 2019 (ISWC 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11779))

Included in the following conference series:

Abstract

One of the key traits of Big Data is its complexity in terms of representation, structure, or formats. One existing way to deal with it is offered by Semantic Web standards. Among them, RDF – which proposes to model data with triples representing edges in a graph – has received a large success and the semantically annotated data has grown steadily towards a massive scale. Therefore, there is a need for scalable and efficient query engines capable of retrieving such information. In this paper, we propose Sparklify: a scalable software component for efficient evaluation of SPARQL queries over distributed RDF datasets. It uses Sparqlify as a SPARQL-to-SQL rewriter for translating SPARQL queries into Spark executable code. Our preliminary results demonstrate that our approach is more extensible, efficient, and scalable as compared to state-of-the-art approaches. Sparklify is integrated into a larger SANSA framework and it serves as a default query engine and has been used by at least three external use scenarios.

Resource type Software Framework

Website http://sansa-stack.net/sparklify/

Permanent URL https://doi.org/10.6084/m9.figshare.7963193

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.domo.com/learn/data-never-sleeps-5.

  2. 2.

    https://www.w3.org/TR/rdf11-primer/.

  3. 3.

    https://www.w3.org/TR/sparql11-overview/.

  4. 4.

    http://lodstats.aksw.org/.

  5. 5.

    https://goo.gl/mJTkPp.

  6. 6.

    https://github.com/SANSA-Stack/SANSA-Query/tree/develop/sansa-query-spark/src/main/scala/net/sansa_stack/query/spark/sparqlify.

  7. 7.

    http://linkedgeodata.org.

  8. 8.

    http://sansa-stack.net/.

  9. 9.

    https://github.com/SmartDataAnalytics/Sparqlify.

  10. 10.

    https://www.w3.org/TR/r2rml/.

  11. 11.

    http://sml.aksw.org/.

  12. 12.

    https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html.

  13. 13.

    https://jena.apache.org/documentation/query/.

  14. 14.

    https://aleth.io/.

  15. 15.

    https://github.com/ConsenSys/EthOn.

  16. 16.

    https://bit.ly/2YX7CXG.

  17. 17.

    https://www.specialprivacy.eu.

  18. 18.

    http://slipo.eu/.

  19. 19.

    https://pig.apache.org/.

  20. 20.

    https://impala.apache.org/.

  21. 21.

    https://accumulo.apache.org.

References

  1. Aluç, G., Hartig, O., Özsu, M.T., Daudjee, K.: Diversified stress testing of RDF data management systems. In: Mika, R., et al. (eds.) ISWC 2014. LNCS, vol. 8796, pp. 197–212. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11964-9_13

    Chapter  Google Scholar 

  2. Armbrust, M., et al.: Spark SQL: relational data processing in Spark. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1383–1394. ACM, New York (2015)

    Google Scholar 

  3. Calvanese, D., et al.: Ontop: answering SPARQL queries over relational databases. Semant. Web 8, 471–487 (2017)

    Article  Google Scholar 

  4. Erling, O., Mikhailov, I.: Virtuoso: RDF support in a native RDBMS. In: de Virgilio, R., Giunchiglia, F., Tanca, L. (eds.) Semantic Web Information Management, pp. 501–519. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-04329-1_21

    Chapter  Google Scholar 

  5. Ermilov, I., et al.: The Tale of Sansa Spark. In 16th International Semantic Web Conference, Poster & Demos (2017)

    Google Scholar 

  6. Faye, D.C., Curé, O., Blin, G.: A survey of RDF storage approaches. Revue Africaine de la Recherche en Informatique et Mathématiques Appliquées 15, 11–35 (2012)

    Google Scholar 

  7. Graux, D., Jachiet, L., Genevès, P., Layaïda, N.: SPARQLGX: efficient distributed evaluation of SPARQL with Apache Spark. In: Groth, P., et al. (eds.) The Semantic Web - ISWC 2016. LNCS, pp. 80–87. Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-46547-0_9

    Chapter  Google Scholar 

  8. Graux, D., Jachiet, L., Geneves, P., Layaïda, N.: A multi-criteria experimental ranking of distributed SPARQL evaluators. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 693–702. IEEE (2018)

    Google Scholar 

  9. Guo, Y., Pan, Z., Heflin, J.: LUBM: a benchmark for owl knowledge base systems. J. Web Semant. 3, 158–182 (2005)

    Article  Google Scholar 

  10. Kaoudi, Z., Manolescu, I.: RDF in the clouds: a survey. VLDB J.-Int. J. Very Large Data Bases 24(1), 67–91 (2015)

    Article  Google Scholar 

  11. Lehmann, J., et al.: Distributed semantic analytics using the SANSA stack. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10588, pp. 147–155. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68204-4_15

    Chapter  Google Scholar 

  12. Neumann, T., Weikum, G.: RDF-3X: a RISC-style engine for RDF. Proc. VLDB Endow. 1(1), 647–659 (2008)

    Article  Google Scholar 

  13. Punnoose, R., Crainiceanu, A., Rapp, D.: Rya: a scalable RDF triple store for the clouds. In: Proceedings of the 1st International Workshop on Cloud Intelligence, Cloud-I 2012, pp. 4:1–4:8. ACM, New York (2012)

    Google Scholar 

  14. Rohloff, K., Schantz, R.E.: High-performance, massively scalable distributed systems using the MapReduce software framework: the SHARD triple-store. In: Programming Support Innovations for Emerging Distributed Applications, PSI EtA 2010, pp. 4:1–4:5. ACM, New York (2010)

    Google Scholar 

  15. Schätzle, A., Przyjaciel-Zablocki, M., Lausen, G.: PigSPARQL: mapping SPARQL to Pig Latin. In: Proceedings of the International Workshop on Semantic Web Information Management, SWIM 2011, pp. 4:1–4:8. ACM, New York (2011)

    Google Scholar 

  16. Schätzle, A., Przyjaciel-Zablocki, M., Neu, A., Lausen, G.: Sempala: interactive SPARQL query processing on Hadoop. In: Mika, P., et al. (eds.) ISWC 2014. LNCS, vol. 8796, pp. 164–179. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11964-9_11

    Chapter  Google Scholar 

  17. Schätzle, A., Przyjaciel-Zablocki, M., Skilevic, S., Lausen, G.: S2RDF: RDF querying with SPARQL on Spark. Proc. VLDB Endow. 9(10), 804–815 (2016)

    Article  Google Scholar 

  18. Stadler, C., Unbehauen, J., Westphal, P., Sherif, M.A., Lehmann, J.: Simplified RDB2RDF mapping. In: Proceedings of the 8th Workshop on Linked Data on the Web, LDOW 2015, Florence, Italy (2015)

    Google Scholar 

  19. Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX (2012)

    Google Scholar 

Download references

Acknowledgment

This work was partly supported by the EU Horizon2020 projects BigDataOcean (GA no. 732310), Boost4.0 (GA no. 780732), SLIPO (GA no. 731581) and QROWD (GA no. 723088); and by the ADAPT Centre for Digital Content Technology funded under the SFI Research Centres Programme (Grant 13/RC/2106) and co-funded under the European Regional Development Fund.

Author information

Authors and Affiliations

  1. Institute for Applied Informatics (InfAI), University of Leipzig, Leipzig, Germany

    Claus Stadler

  2. Smart Data Analytics, University of Bonn, Bonn, Germany

    Gezim Sejdiu & Jens Lehmann

  3. Enterprise Information Systems, Fraunhofer IAIS, Sankt Augustin, Germany

    Damien Graux & Jens Lehmann

  4. ADAPT Centre, Trinity College of Dublin, Dublin, Ireland

    Damien Graux

Authors
  1. Claus Stadler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gezim Sejdiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Damien Graux
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jens Lehmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claus Stadler .

Editor information

Editors and Affiliations

  1. Fondazione Bruno Kessler, Trento, Italy

    Chiara Ghidini

  2. Linköping University, Linköping, Sweden

    Olaf Hartig

  3. University of Bonn, Bonn, Germany

    Maria Maleshkova

  4. University of Economics Prague, Prague, Czech Republic

    Vojtěch Svátek

  5. University of Illinois at Chicago, Chicago, IL, USA

    Isabel Cruz

  6. University of Chile, Santiago, Chile

    Aidan Hogan

  7. Memect Technology, Beijing, China

    Jie Song

  8. Mines Saint-Etienne, Saint-Etienne, France

    Maxime Lefrançois

  9. Inria Sophia Antipolis - Méditerranée, Sophia Antipolis, France

    Fabien Gandon

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Stadler, C., Sejdiu, G., Graux, D., Lehmann, J. (2019). Sparklify: A Scalable Software Component for Efficient Evaluation of SPARQL Queries over Distributed RDF Datasets. In: Ghidini, C., et al. The Semantic Web – ISWC 2019. ISWC 2019. Lecture Notes in Computer Science(), vol 11779. Springer, Cham. https://doi.org/10.1007/978-3-030-30796-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30796-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30795-0

  • Online ISBN: 978-3-030-30796-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation