rules_engine.execute_rules does a deepcopy of the dataset

[gerrycampion]

Refer to the line here:

cdisc-rules-engine/cdisc_rules_engine/rules_engine.py

Line 313 in cdb8cdb

dataset = deepcopy(dataset)

If the dataset is large, this is an expensive operation for cpu, disk and ram. We should find a better way to do this.

Update: Check out Copy-on-Write documentation and refer to Dataframe copy

[SFJohnson24]

https://github.com/static-frame/static-frame may be a way to preserve the original dataframe and prevent the need for a deepcopy

[alexfurmenkov]

@gerrycampion , @SFJohnson24 What do you think about updating the pandas version to 3.0.*?
pandas CoW says that

Copy-on-Write is now the default with pandas 3.0.

Alternatively, we can enable it by adding
pd.options.mode.copy_on_write = True at the start of validation.

[SFJohnson24]

that is a large version bump. I do think this behavior will prevent mutation which was what the deep copy was set out to do. Preprocessor does a copy() at the beginning. If all operations are just adding computed columns and check operators are read-only, I dont see why this wouldnt work.