n <- 100000 } piFunc <- function(elem) { x, y = random.random(), random.random() println(, """ val val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0) message(length(elems)) We’ll start off with a Spark session that takes Scala code: Once the session has completed starting up, it transitions to the idle state: Now we can execute Scala by passing in a simple JSON command: If a statement takes longer than a few milliseconds to execute, Livy returns Livy uses the Spark configuration under SPARK_HOME by default. def sample(p): Pi. More interesting is using Spark to estimate Here’s a step-by-step example of interacting with Livy in Python with the Here is a couple of examples. When Livy … val NUM_SAMPLES = 100000; count <- reduce(lapplyPartition(rdd, piFuncVec), sum) print "Pi is roughly, """ By default Livy runs on port 8998 (which can be changed with the livy.server.port config option). Here’s a step-by-step example of interacting with Livy in Python with the Requests library. val y = Math.random(); rands2 <- runif(n = length(elems), min = -1, max = 1) } rands1 <- runif(n = length(elems), min = -1, max = 1) Livy provides high-availability for Spark jobs running on the cluster. val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0) By default Livy runs on port 8998 (which can be changed if (x*x + y*y < 1) 1 else 0 cat("Pi is roughly", 4.0 * count / n, ", Apache License, Version 2.0. rands <- runif(n = 2, min = -1, max = 1) sum(val) If the Livy service goes down after you've submitted a job remotely to a Spark cluster, the job continues to run in the background. piFuncVec <- function(elems) { We’ll start off with a Spark … return 1 if x*x + y*y < 1 else 0 It is strongly recommended to configure Spark … Requests library. early and provides a statement URL that can be polled until it is complete: That was a pretty simple example. You can override the Spark configuration by setting the SPARK_CONF_DIR environment variable before starting Livy. import random This is from the Spark Examples: PySpark has the same API, just with a different initial request: The Pi example from before then can be run as: """ with the livy.server.port config option). val x = Math.random(); }.reduce(_ + _); val count = sc.parallelize(1 to NUM_SAMPLES).map { i => count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b) NUM_SAMPLES = 100000 rdd <- parallelize(sc, 1:n, slices)