반응형
spark 하드웨어 옵션 계산
-
spark-submit 하드웨어 옵션 체크하기Data Engineering/Apache Spark 2022. 2. 7. 14:44
페이지 참고사항 (계산식) https://aws.amazon.com/ko/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/#:~:text=For example%2C the default for,application based on the workloads. 계산된 옵션 목록 executor-cores executor-memory spark.driver.cores driver-memory num_executors spark.default.parallelism Linux Cpu, Memory 확인 방법 # Cpu core 확인 lscpu # Memory 확인 free..