ARTICLE AD BOX
I want to share an open-source benchmark project I recently concluded. It's a session-level clickstream data preprocessing pipeline comparing Polars and Pandas for ML feature engineering (specifically, targeting LightGBM).
I think the community might find the results highly relevant, as it perfectly demonstrates the performance cliff between Python UDFs and the native expression engine.
Key Findings:
Phase 1 (The UDF Trap): Calculating sequence Shannon entropy via map_elements became the absolute bottleneck, consuming ~99% of the pipeline time and restricting the end-to-end Polars advantage to ~3x over Pandas. Phase 3 (Native Expressions): By replacing the Python callback with a native explode + group_by + .entropy(normalize=True, base=2) expression, the bottleneck was entirely eliminated. Final Result: The fully native Polars pipeline achieved a 146x speedup on the MSCI dataset and a 186x speedup on the Retailrocket dataset compared to Pandas .apply().Thank you for building such an incredible tool. The fact that operations like entropy are available natively made this optimization incredibly clean.
Repository with all notebooks, EDA, and raw timing CSVs: [https://git.disroot.org/Machine_Learning-XanIAGPL/clickstream-polars-benchmark]
Disclaimer: I am not a Polars expert. Although I have knowledge of Machine Learning, Pandas, and Python, all the code I generate is made by AI, following my instructions, suggestions/questions, corrections, and step-by-step joint reviews of the generated code.
