[Spark] K8s로 spark cluster 구축하기

관련 깃허브

https://github.com/bitnami/charts/tree/main/bitnami/spark

charts/bitnami/spark at main · bitnami/charts

1.  Helm 다운로드 

helm install my-spark-release oci://registry-1.docker.io/bitnamicharts/spark

 

2.  상태 확인

helm status my-spark-release

 

3. 웹 ui 확인

# localhost:80로 ui 포트포워딩 
kubectl port-forward svc/my-spark-release-master-svc 8080:80

 

4.  Spark job Test

#컨테이너 접속
kubectl exec -it my-spark-release-master-0 -- /bin/bash  
#example 실행
spark-submit --master spark://my-spark-release-master-svc:7077 /opt/bitnami/spark/examples/src/main/python/pi.py

 

 

# 커스텀 Spark 이미지 적용하는 방법

# values.yaml 
image:
  registry: docker.io
  repository: your repo
  tag: your tag
  pullPolicy: IfNotPresent
  
#이미지 적용 
helm upgrade my-spark-release -f values.yaml