Memory not released when deleting or clearing Fireducks DataFrame

[Sree-dutt]

When a fireducks.DataFrame object is deleted or cleared, the memory it occupies is not released back to the system,references are still internally available. This causes memory to accumulate over time.
Deleted it using del df or clear its contents
Ran the garbage collection manually gc.collect()
Observed that the memory is not released
Fireducks version -- 1.2.0
@qsourav

[qsourav]

Dear @Sree-dutt,

Thank you for reporting the issue.

It might be caused by any of the two possible reasons (although it's difficult to confirm without looking into the actual use case):

1. We avoid data copying between operations as much as possible using a shared pointer for each column. Therefore, you may notice no significant memory gain even when we perform some operations like concat, projections, reset_index, etc. As a result, the buffer holding the column values will not be deallocated until all its associated objects are released.

In short, if OpA -> OpB -> OpC do not involve any new allocation (just share the immutable buffer), you will not notice any deallocation when objects associated with OpB, OpC are released.

2. In FireDucks, the backend (C++) side memory is managed by the arrow memory pool. Therefore, even though the garbage collection releases the python objects, the associated memory on the backend side might not be returned to the OS (right after gc is called). Hence, it might be difficult to confirm the released memory, but it could be reused for further allocation.

In other words, gc.collect() must have released the memory that does not have any reference left at the Python side, but it might be difficult to confirm at the OS level since it is managed by a different memory pool at the backend side.

Thanks and Regards,
Sourav Saha

[qsourav]

Dear @Sree-dutt,

We have fixed one potential issue related to delayed memory deallocation with FireDucks 1.4.0.
Can you please check if your previously reported situation gets improved with this version?

Looking forward to hearing from you.

Thanks and Regards,
Sourav Saha

[qsourav]

Hi @Sree-dutt,

Closing the issue as completed.

Please feel free to reopen in case it isn't solved for your case.

Thanks and Regards,
Sourav Saha