Hello team. Seem that the image version sagemaker-spark-processing:3.1-cpu is not using an updated version of PyArrow, required for pandas_udf functions. Error returned during the execution of the Processing Job:
2023-02-21T20:06:56.503+01:00ImportError: PyArrow >= 1.0.0 must be installed; however, it was not found.
Can you please update the Spark images? Thank you