Performance issue with 0.16.1

[jenky]

Upgrade from 0.15.1

I have a CLI command that finishes in less than 1 second. After the upgrade, the command freezes without any changes to the source code. I'm not sure what's happening under the hood, as there are no error logs or anything similar.

Basically, it reads the parquet files from S3, joins them with another data frame (loaded from the database), performs some aggregations (groupBy(), sum(), rank()), renames the columns, and then calls collect() and write(to_output())

One thing I've tried to debug is that the command uses collect() to display the output and it might lead to memory issues. Once it is removed, the command completes but takes about 12 seconds and double the memory usage compare the 0.15.1 with collect()

[stloyd]

Could you paste the methods used in the overall workflow? Could the refactoring of the renaming functionality affect something? (can be simplified, ofc, just overall size & method called would be sufficient for the start of debugging).

[norberttech]

hey @jenky let me ask few questions first.

After the upgrade, the command freezes without any changes to the source code.

is it possible that the size of the datasets that you are processing changed in the meantime?

without any changes to the source code

Did you actually tried to downgrade back to 0.15.1 and run the same code? It's still being processed under 1 minute?

renames the columns,

Last release mostly changed things around renames the columns.
Could you share the part of the code related to that rename?

Basically, it reads the parquet files from S3, joins them with another data frame (loaded from the database), performs some aggregations (groupBy(), sum(), rank()), renames the columns, and then calls collect() and write(to_output())

In general those two methods:

• collect()
• to_output()

Are not very efficient, to_output specifically is desinged for debugging/development purposes.
collect() on the other hand needs to collect all rows into the memory and merge them together, in most cases it should be avoided unlless your dataset size is constant and not bigger than available memory.

Also like @stloyd already menitoned, if you could share with me the code of your Flow Pipeline and maybe some sample data, it would make it a lot easier for me to try to reproduce it locally, otherwise we are pretty much guessing.

If you don't want to share the code here you can send it to me over email contact@norbert.tech or reach out to me at our Discord Server

[jenky]

is it possible that the size of the datasets that you are processing changed in the meantime?

No, the dataset is the same.

Did you actually tried to downgrade back to 0.15.1 and run the same code? It's still being processed under 1 minute?

Yes.

Last release mostly changed things around renames the columns.
Could you share the part of the code related to that rename?

I was aware of this change and noticed that the internal flow pipelines are different, with more pipelines being run.

In general those two methods:

collect()
to_output()
Are not very efficient, to_output specifically is desinged for debugging/development purposes.

Yep, my CLI command is used for debugging. Basically, I want to confirm the results before importing the data into the database.

If you don't want to share the code here you can send it to me over email contact@norbert.tech or reach out to me at our Discord Server

I've sent you the email. The database dataset has been replaced with a CSV. Technically, it won't change over time.

[norberttech]

hey @jenky
first of all, thank you for providing code samples that helped me reproduce this issue, it was super helpful! 🤝

I was able to localize the regression introduced here #1640

What was happening is that each pipeline that was wrapping another one, instead of returning just those pipelines was also iterating recursively through them.

It wasn't really visible until certain combination of pipelines put together 🤦‍♂

Long story short here was pretty much an infinite loop inside of pipelines optimization mechanism.

Patch is coming once merged I will release it as 0.16.2.

Thank you!!

[norberttech]

0.16.2 It is now available, sorry for the inconvenience!