Skip to content

Commit 18c22c0

Browse files
committed
Added executor configuration parameters to the Makefile and README.md
Signed-off-by: Manuel Carmona <manu.carmona90@gmail.com>
1 parent e4edafa commit 18c22c0

2 files changed

Lines changed: 12 additions & 1 deletion

File tree

Makefile

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ SBT = ./sbt ++$(SCALA_VERSION)
1313
APP_NAME = "QuerySetApp"
1414
APP_MAIN_CLASS = "tech.sourced.queryset.Main"
1515
SPARK_MASTER ?= "local[*]"
16+
NUM_EXECUTORS ?= 1
17+
EXECUTOR_MEM ?= 1G
18+
EXECUTORS_CORES ?= 4
1619
UBER_JAR = "target/scala-2.11/queryset-0.1.0.jar"
1720
REPOS_PATH ?= "src/main/resources/siva-files"
1821
REPOS_FORMAT ?= "siva"
@@ -27,6 +30,9 @@ run:
2730
--name $(APP_NAME) \
2831
--class $(APP_MAIN_CLASS) \
2932
--master $(SPARK_MASTER) \
33+
--num-executors $(NUM_EXECUTORS) \
34+
--executor-memory $(EXECUTOR_MEM) \
35+
--total-executor-cores $(EXECUTORS_CORES) \
3036
$(UBER_JAR) $(APP_ARGS)
3137

3238
build: clean

README.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,21 @@ $SPARK_HOME/bin/spark-submit \
99
--name "QuerySetApp" \
1010
--class "tech.sourced.queryset.Main" \
1111
--master $SPARK_MASTER \
12+
--num-executors $NUM_EXECUTORS \
13+
--executor-memory $EXECUTOR_MEM \
14+
--total-executor-cores $EXECUTOR_CORES \
1215
path/to/queryset-0.1.0.jar $REPOS_PATH $REPOS_FORMAT
1316
```
1417

1518
- `SPARK_HOME` environment variable must point to the directory where `Spark` was downloaded (e.g. `/usr/local/spark`)
1619

1720
- `SPARK_MASTER` environment variable must point to the [master URL](https://spark.apache.org/docs/latest/submitting-applications.html#master-urls) for the cluster (e.g. `spark://p-spark-master:7077`)
1821

22+
- `NUM_EXECUTORS`, `EXECUTOR_MEM`, `EXECUTOR_CORES`: [Executor configuration parameters](https://spark.apache.org/docs/latest/submitting-applications.html) (eg: `3`, `4G`, `64`).
23+
1924
- `REPOS_PATH` must point to the directory which contains the repositories.
2025

21-
- `REPOS_FORMAT` must specify the repositories' format (e.g. `siva`)
26+
- `REPOS_FORMAT` must specify the repositories' format (e.g. `siva`, `standard`)
2227

2328
#### Development
2429
- First you should set the `Spark` directory:

0 commit comments

Comments
 (0)