Add large-file tests for modules using sendfile(2)

[giampaolo]

BPO	37096
Nosy	@vstinner, @giampaolo, @encukou, @stratakis
PRs	bpo-37096: Add large-file tests for modules using sendfile(2) #13676
Files	disk-space.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-02-17.11:23:56.052>
created_at = <Date 2019-05-30.07:18:19.110>
labels = ['type-feature', 'tests', '3.9']
title = 'Add large-file tests for modules using sendfile(2)'
updated_at = <Date 2020-02-17.11:23:56.051>
user = 'https://github.com/giampaolo'

bugs.python.org fields:

activity = <Date 2020-02-17.11:23:56.051>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-17.11:23:56.052>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-05-30.07:18:19.110>
creator = 'giampaolo.rodola'
dependencies = []
files = ['48636']
hgrepos = []
issue_num = 37096
keywords = ['patch']
message_count = 14.0
messages = ['343951', '353533', '353552', '353699', '353701', '353703', '353704', '353709', '356368', '357859', '357863', '357944', '360325', '362136']
nosy_count = 5.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'petr.viktorin', 'rosslagerwall', 'cstratak']
pr_nums = ['13676']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37096'
versions = ['Python 3.9']

[giampaolo]

The need for this emerged in:
https://bugs.python.org/issue36610#msg343948.
We currently use sendfile(2) syscall in high-level shutil.copyfile() on Linux and socket.sendfile() on UNIX. In addition this PR also tests shutil.copyfile() implementation on OSX which takes advantage of fcopyfile(2) syscall. The goal is to make sure "fast-copy" syscalls can handle files > 2G. On my Ubuntu 18.04 with SSD disk this introduces a 5 secs slowdown to the test run.

According to:
golang/go#13892
...problems may arise on SunOS, so if a BB will turn red this is expected.

[giampaolo]

New changeset 5bcc6d8 by Giampaolo Rodola in branch 'master':
bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)
5bcc6d8

[vstinner]

TestCopyfile fails on 32-bit system: bpo-38319.

[stratakis]

After this change I get some disk space issues on the Fedora rawhide buildbot for the clang installed build only (and strangely enough not for the other jobs). There are currently around 9GB of free space there:

https://buildbot.python.org/all/#/builders/188/builds/1110

======================================================================
ERROR: test_tofromfile (test.test_array.DoubleTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64.clang-installed/build/target/lib/python3.9/test/test_array.py",
line 374, in test_tofromfile
    f.close()
OSError: [Errno 28] No space left on device

[giampaolo]

Sorry about that. I'm attaching a patch which skips those tests if there's not enough free disk space. If you confirm it works I'll make a PR.

[vstinner]

Should we decorate the new files with @requires('largefile', '...')? Documentation:

largefile - It is okay to run some test that may create huge
            files.  These tests can take a long time and may
            consume >2 GiB of disk space temporarily.

It may be interesting to add an option to regrtest to specify how much space can be used by tests, an option similar to -M/--memlimit which is for memory.

I don't recall if largefile is enabled by default on buildbots or not :-p