In my previous post I talked about of RDDs as an abstraction of parallel data processing. Today, I’d like to briefly discuss and set an example for accumulators and broadcast variables. Accumulators counters or sums that can be reliably used in parallel processing native support for numeric types, extensions possible via API workers can modify…
When starting with Apache Spark, a “lightning-fast cluster computing” engine, it is important to understand how Spark fits into the Hadoop ecosystem. This article provides a brief overview of Spark’s distinctive features and its ties Hadoop. Hadoop has been around for about 12 years and it dominated the space of Big Data by providing reliable distributed processing of…
Surprising as it sounds, Eclipse doesn’t support sbt out of the box, not even in the Scala IDE. At least I wasn’t able to find a way of how to generate an sbt project from within Eclipse. Hence, I wrote my own bash script which generates a ready-to-use Eclipse-compliant minimalistic sbt project. The script sbt-eclipse.sh (see…
As a Scala newbie I was struggling to understand the benefits of currying in a real-life scenario. Most of the examples I could find were a bit too academic for my taste. Eventually, the coin dropped and I realized that currying enables to derive specialized methods out of a general one in an elegant and…