timeit called from within Python should allow autoranging

[scottdaniels]

BPO	6422
Nosy	@rhettinger, @amauryfa, @ncoghlan, @pitrou, @vstinner, @rbtcollins, @merwok, @stevendaprano, @asvetlov, @csabella
Files	timeit.patch: Patch including changes to timeit, test, and doc timeit-autorange.patch: minimal proof-of-concept patch to add autorange method to Timer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/stevendaprano'
closed_at = <Date 2019-03-28.09:50:03.233>
created_at = <Date 2009-07-05.15:15:10.511>
labels = ['type-feature', 'library']
title = 'timeit called from within Python should allow autoranging'
updated_at = <Date 2019-03-28.09:50:03.232>
user = 'https://bugs.python.org/scottdaniels'

bugs.python.org fields:

activity = <Date 2019-03-28.09:50:03.232>
actor = 'cheryl.sabella'
assignee = 'steven.daprano'
closed = True
closed_date = <Date 2019-03-28.09:50:03.233>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2009-07-05.15:15:10.511>
creator = 'scott_daniels'
dependencies = []
files = ['14472', '42816']
hgrepos = []
issue_num = 6422
keywords = ['patch']
message_count = 30.0
messages = ['90157', '90245', '90264', '90275', '122955', '123711', '164027', '164070', '164071', '164216', '164217', '168796', '238352', '238354', '238879', '265319', '265322', '265324', '265326', '265360', '265418', '265442', '272140', '272142', '272676', '272704', '338958', '339011', '339024', '339026']
nosy_count = 13.0
nosy_names = ['rhettinger', 'scott_daniels', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'rbcollins', 'eric.araujo', 'steven.daprano', 'asvetlov', 'tshepang', 'python-dev', 'cheryl.sabella']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6422'
versions = ['Python 3.6']

[scottdaniels]

timeit.main has a _very_ handy autoranging facility to pick an
appropriate number of repetitions when not specified. The autoranging
code should be lifted to a method on Timer instances (so non-main code
can use it). If number is specified as 0 or None, I would like to use
the results of that autoranging code in Timer.repeat and Timer.timeit.

Patch to follow.

[scottdaniels]

I've got the code "working" on trunk2 for my tests.
Should I port to py3K before checking in, and give diffs from there, or
what?

[amauryfa]

You can still upload available patches to this tracker.

[rhettinger]

I would like to look at this in context of all the other proposed build-
outs to timeit.

[rhettinger]

This does not conflict with the other proposed changes to timeit and it is in-line with Guido's desire that to expose useful parts currently buried in the command-line logic.

Amaury, you've shown an interest. Would you like to apply it?

[pitrou]

Not sure why you chose 0.11 here. It should probably be 0.2 as in the command-line code.
As for applying the patch, this can't be done before 3.2 is released.

[rhettinger]

Looking at this again after more time has passes, I still think exposing autoranging is a good idea but I don't like the patch as it stands. It "infects" the API in a number of places and makes the overall module harder to use and learn.

Ideally, there should be a cleaner interface, or more limited API change, or a separate high level function that can autorange existing functions without changing their API.

Anyone care to propose a cleaner API?

[ncoghlan]

In bpo-5442, I proposed leaving the architecture of the module alone, and simply exposing the main module functionality as a high level helper function:

def measure(stmt="pass", setup="pass", timer=default_timer,
            repeat=default_repeat, number=default_number,
            verbosity=0, precision=3)

The return value would simply be a (number, results) 2-tuple with the number of iterations per test (which may have been calculated automatically), and then a list of the results. To get "timeit" style behavior, simply set "repeat=1".

[ncoghlan]

Oops, that link reference should have been to bpo-5441.

[vstinner]

Hi, I wrote recently a similar function because timeit is not reliable by default. Results look random and require to run the same benchmark 3 times or more on the command line.

https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py

By default, the benchmark takes at least 5 measures, one measure should be greater than 100 ms, and the benchmark should not be longer than 1 second. I chose these parameters to get reliable results on microbenchmarks like "abc".encode("utf-8").

The calibration function uses also the precision of the timer. The user may define a minimum time (of one measure) smaller than the timer precision, so the calibration function tries to solve such issue. The calibration computes the number of loops and the number of repetitions.

Look at BenchmarkRunner.calibrate_timer() and BenchmarkRunner.run_benchmark().
https://bitbucket.org/haypo/misc/src/bfacfb9a1224/python/benchmark.py#cl-362