Skip to main content
SpringerLink
Log in

Big Data programming with Apache Spark

  • Chapter
  • First Online:
Data Science in Practice

Part of the book series: Studies in Big Data ((SBD,volume 46))

Abstract

In this chapter we give an introduction to Apache Spark, a Big Data programming framework. We describe the framework’s core aspects as well as some of the challenges that parallel and distributed computing entail.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    See “Scala: From a Functional Programming Perspective” [3] for a quick introduction to the programming language; and/or “Programming in Scala” [2] for a more thorough description of the language.

  2. 2.

    Read, Eval, Print and Loop. See http://docs.scala-lang.org/overviews/repl/overview.html.

  3. 3.

    See more options by running spark-shell --help.

  4. 4.

    A more technical description of RDDs can be found in [4].

  5. 5.

    The number of partitions in which a file is divided is, if not stated otherwise, decided by Spark based on file block size. File block size is 32MB on a local file system, and 128MB on YARN. The minimum number of partitions is 2, which would be the case of small files such as README.md (3.8 K).

  6. 6.

    Immutability is a key concept in functional programming, and an important aspect for reliable parallel programming.

  7. 7.

    List of transformations: http://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations.

  8. 8.

    Lineage is the official name given in the Spark documentation.

  9. 9.

    List of actions: http://spark.apache.org/docs/latest/rdd-programming-guide.html#actions.

  10. 10.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD.

  11. 11.

    http://spark.apache.org/docs/latest/rdd-programming-guide.html#shared-variables.

  12. 12.

    http://spark.apache.org/docs/latest/rdd-programming-guide.html#shuffle-operations.

  13. 13.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions.

  14. 14.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions.

  15. 15.

    Serialization occurs when data is sent over the network e.g. when a shuffle operation takes place.

  16. 16.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameReader.

  17. 17.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Column.

  18. 18.

    http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$.

  19. 19.

    Weather from 2012: http://academictorrents.com/details/16be344abd95d58afd4860445f4a927b7eb1a89d.

  20. 20.

    A DStream is collection of RDDs, which are collections of distributed elements. This might sound confusing but bear with us.

  21. 21.

    How to build self-contained applications: http://spark.apache.org/docs/latest/quick-start.html#self-contained-applications.

  22. 22.

    Library dependencies can be found in the Maven repository: https://mvnrepository.com/.

  23. 23.

    http://spark.apache.org/docs/latest/streaming-programming-guide.html#basic-sources.

  24. 24.

    http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#input-sources.

  25. 25.

    A new feature called Continuous Processing, which will allow handling data elements as soon as they arrive, is being released.

References

  1. Jacobs, A. (2009). The pathologies of big data. Communications of the ACM, 52(8), 36–44.

    Article  Google Scholar 

  2. Odersky, M., Spoon, L., & Venners, B. (2011). Programming in Scala, 2 edn. Artima Press.

    Google Scholar 

  3. Torra, V. (2016). Scala: From a functional programming perspective: An introduction to the programming language. Cham, Switzerland: Springer.

    Book  Google Scholar 

  4. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., & Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing, in Proceedings of the 9th USENIX conference on networked systems design and implementation, USENIX Association, Berkeley, CA, USA, NSDI’12, p. 2

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Skövde, Skövde, Sweden

    Elio Ventocilla

Authors
  1. Elio Ventocilla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elio Ventocilla .

Editor information

Editors and Affiliations

  1. University of Skövde, Skövde, Sweden

    Alan Said

  2. University of Skövde, Skövde, Sweden

    Vicenç Torra

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Ventocilla, E. (2019). Big Data programming with Apache Spark. In: Said, A., Torra, V. (eds) Data Science in Practice. Studies in Big Data, vol 46. Springer, Cham. https://doi.org/10.1007/978-3-319-97556-6_10

Download citation

Publish with us

Policies and ethics

Search

Navigation