spark-optimization

SUMMARY

What it does

This Agent Skill provides production patterns for optimizing Apache Spark jobs, covering partitioning strategies, memory management, shuffle optimization, and performance tuning. The source document includes guidance on enabling Adaptive Query Execution (AQE), using columnar formats like Parquet/Delta, broadcasting small tables, monitoring the Spark UI, and right-sizing partitions. It also lists common pitfalls to avoid, such as collecting large data to the driver, unnecessary UDFs, over-caching, ignoring data skew, and using count() for existence checks. The skill is intended for use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.