Monitoring Spark with Graphite and Grafana

Folks at Hammer Lab have shared the results of their efforts on Monitoring Spark with Graphite and Grafana. The post explains how to use Spark's MetricSystem to direct relevant stats from Executors, Drivers and the JVM to various Sinks - in this case, allowing them to create Graphite/Grafana dashboards.…

Read more

Spark Streaming and Kafka

If you are using Spark Streaming with Kafka, Micheal G. Noll wrote a very comprehensive guide on Integrating Kafka and Spark Streaming where he provides a lot of detail on parallelism, perfomance tuning and scalability. You can also find a complete code example on github.…

Read more
Proudly published with Ghost