Broadcasting DataFrames across NumPy array dimensions

I'm working with a large Pandas DataFrame and a multi-dimensional NumPy array. My goal is to efficiently "broadcast" a specific column of the DataFrame across one or more dimensions of the NumPy array, performing an element-wise operation.

Let's say I have a DataFrame df like this:

import pandas as pd import numpy as np data = {'id': range(100), 'value': np.random.rand(100)} df = pd.DataFrame(data)

And a NumPy array arr with shape (10, 5, 100, 20):

arr = np.random.rand(10, 5, 100, 20)

I want to multiply df['value'] by arr such that df['value'][i] is multiplied by arr[:, :, i, :] for all i. In essence, df['value'] should align with the 3rd dimension of arr.

A solution might involve iterating or using np.apply_along_axis which is often slow for large arrays:

result_slow = np.zeros_like(arr) for i in range(df.shape[0]): result_slow[:, :, i, :] = arr[:, :, i, :] * df['value'].iloc[i]

This works, but for much larger arr (e.g., millions in the third dimension) and df, it becomes computationally expensive.

How can I solve this multiplication efficiently, leveraging NumPy's broadcasting without explicit loops or apply_along_axis, to multiply the df['value'] column along a specific axis (the 3rd axis in this case) of the NumPy array?

I'm looking for a solution that not only performs well for large datasets, but is also memory-efficient.