clean up open file handles properly#620
Conversation
|
I've disabled the The equivalent tests for Python 2.4 and 2.7 are still active and seem to be robust against lots of open file handles (probably because they lack the bug in Python 2.6 mentioned above, and because the |
|
It fixes the resource leak in the non-error case, but leaks are still possible in case read() throws. There should be an exception-safe idiom to read a file in python, given that there were 4 places to fix, we might want a function that just returns the file contents. |
easybuild/tools/systemtools.py
Outdated
There was a problem hiding this comment.
make a function openread() that does this.
There was a problem hiding this comment.
Created read_file and write_file functions, and used them everywhere. Please rereview (unit tests pass).
…ting files, and use them
There was a problem hiding this comment.
Add a note here about not using finally since this is not supported in python 2.4.
|
reviewed, ok |
clean up open file handles properly
4 participants
As became apparent through the easybuild-easyconfigs unit tests as soon as the total number of available easyconfigs passed the magic threshold of 1024, lots of file handles are left dangling in the current framework.
This becomes especially clear when using
eb --dep-graphon the full set of available easyconfigs, for example in combination with Python 2.6 since that triggers a bug (http://bugs.python.org/issue3392), see https://jenkins1.ugent.be/view/EasyBuild/job/easybuild-easyconfigs_unit-test_hpcugent_develop/24/testReport/junit/test.easyconfigs.easyconfigs/EasyConfigTest/test_dep_graph/.This patch fixes the problems, by properly closing open files for
/proc/cpuinfoand easyblock log files. Part of the patch is forvsc/utils/fancylogger.py, so this should be fixed in https://github.com/hpcugent/vsc-base instead.A nice way of pinpointing the source of the growing set of dangling open file handles is posted at http://stackoverflow.com/questions/2023608/check-what-files-are-open-in-python.