Introducing Lightning Engine — the next generation of Apache Spark performance

Apache Spark lets organizations analyze vast amounts of data for use cases like ETL, data science, machine learning and more. However, achieving high performance and cost efficiency at scale can be challenging. Users often encounter bottlenecks related to query execution speed, data input/output (I/O), and resource utilization, leading to increased processing times and higher infrastructure costs. 

At Google Cloud, we understand these challenges deeply. To deliver best-in-class performance for Spark, we’re introducing Lightning Engine (in preview), our latest and most powerful Spark engine yet, engineered to unlock the true potential of your lakehouse.

What is Lightning Engine?

Lightning Engine is a multi-layer optimization engine focusing on both the traditional optimization techniques like query and execution optimizations, as well as curated optimizations in the file-system layer and data-access connectors. 

For example, at a 10TB dataset size, Lightning Engine accelerates Spark query performance by 3.6x on TPC-H-like workloads* when compared to open source Spark running on similar infrastructure.

Some of Lightning Engine’s key enhancements, illustrated in Figure 1, include:

• Query optimizer: Lightning Engine incorporates a significantly enhanced Spark optimizer, leveraging Google's expertise from engines like F1 and Procella. This advanced optimizer introduces features such as an optimized Bloom filters implementation based on listing call statistics, subquery fusion to consolidate scans, partial aggregation pushdown to minimize shuffling, enhanced adaptive query execution for join removal and exchange reuse, advanced inferred filters for semi-join pushdowns, dynamic in-filter generation for efficient row-group pruning in Iceberg and Delta tables, and more such optimizations. Taken together, these lead to substantial scan and shuffle reductions.

• Execution engine: Lightning Engine’s execution engine enhances performance through a native implementation based on Apache Gluten and Velox that have been specifically designed to leverage Google’s hardware. This includes unified memory management, for dynamic switching between off-heap and on-heap memory without requiring changes to existing Spark configurations. Lightning Engine also includes expanded support for operators, functions and Spark data types, as well as intelligence to automatically identify opportunities to utilize the native engine for optimal pushdown operations. 

• Shuffle: Lightning Engine incorporates columnar shuffle with an optimized serializer-deserializer to minimize shuffle data. 

• File parsers: Lightning Engine includes a specialized parquet parser to do prefetching, intelligent caching and advanced in-filtering to reduce data scans and metadata operations.

• Connectors: Lightning Engine enhances connectivity to Google Cloud Storage and BigQuery to optimize the performance of its native engine. The improved Cloud Storage connector minimizes metadata operations to reduce costs, while an optimized file output committer unlocks performance and reliability for Spark workloads. Additionally, the new native BigQuery connector streamlines data transfer by directly transmitting data in Apache Arrow format to the engine, eliminating the overhead of row-to-columnar conversions.

Lightning Engine is compatible with Apache Spark DataFrame and SQL APIs, enabling seamless workload execution without requiring modifications to existing code.

Why Lightning Engine?

Compared to other cloud Spark solutions, Lightning Engine offers superior performance and cost savings. Support for open formats like Apache Iceberg and Delta Lake, combined with BigQuery and Google Cloud’s advanced AI/ML, can help you improve business efficiency.

Lightning Engine also provides improved performance over DIY Spark implementations, which can translate to substantial cost savings, allowing you to focus on  core business challenges rather than platform maintenance

Key benefits of Lightning Engine:

• Boosted performance: Leverages a new Spark processing engine with vectorized execution, built-in intelligent caching, and optimized storage I/O to deliver significantly faster query performance.

• Industry-leading price-performance: Delivers superior performance and cost efficiency, allowing users to process more data for less.

• Intuitive lakehouse integration: Works with Apache Iceberg, Delta Lake and Google Cloud services like BigQuery and Vertex AI, providing a unified data analytics and AI platform.

• Enhanced data access: Optimized connectors for Cloud Storage and BigQuery help to improve data access latency, reduced metadata operations, and increased throughput.

• Flexible deployments: Available in both serverless and cluster-based configurations.

While Lightning Engine offers substantial performance gains, the specific impact varies with the workload. It is best suited for compute-intensive tasks leveraging Spark Dataframe APIs and Spark SQL queries, rather than I/O-bound operations. 

The future of Spark on Google Cloud

With the new Lightning Engine high-performance query engine for data, we’re excited to bring the best of Google’s scale, performance, and engineering excellence to Apache Spark workloads, driving innovation and empowering developers globally. This is only the beginning, and we already have plans to make it even faster over the coming months!

Lightning Engine is available in preview in both Google Cloud Serverless for Apache Spark and Dataproc on Google Compute Engine premium tiers. Both services already feature GPU support for accelerated machine learning workloads, and job monitoring tools for operational efficiency. Request early access to the private preview here.

^{Special thanks to Newton Alex, Sr. Engineering Manager, for his contributions to this blog post.}