Add benchmark script to compare two different builds of s5cmd#471

boraberke · 2022-07-22T11:26:24Z

This python script allow us to compare two different build (from either version tag, PR number or commit tag) performance under various scenarios. These scenarios include:

Upload, Download, Remove many small sized file
Upload, Download, Remove large file
Upload, Download, Remove very large file

To change the scenarios, you should edit it inside the bench.py for now. In the future, this could be read from a file. From each scenario, user should not forget to change the file size and file count keeping in mind the restrictions of their system.

To run use the following syntax:

usage: bench.py [-h] [-s OLD NEW] [-w WARMUP] [-r RUNS] [-o OUTPUT_FILE_NAME] -b BUCKET [-l LOCAL_PATH] [-p PREFIX] [-hf HYPERFINE_EXTRA_FLAGS] [-sf S5CMD_EXTRA_FLAGS]

Compare performance of two different builds of s5cmd.

optional arguments:
  -h, --help            show this help message and exit
  -s OLD NEW, --s5cmd OLD NEW
                        Reference to old and new s5cmd.It can be a decimal indicating PR number,any of the version tags like v2.0.0 or any commit tag. Additionally it can be 'latest_release' or
                        'master'. (default: ('latest_release', 'master'))
  -w WARMUP, --warmup WARMUP
                        Number of program executions before the actual benchmark: (default: 2)
  -r RUNS, --runs RUNS  Number of runs to perform for each command (default: 10)
  -o OUTPUT_FILE_NAME, --output_file_name OUTPUT_FILE_NAME
                        Name of the output file (default: summary.md)
  -b BUCKET, --bucket BUCKET
                        Name of the bucket in remote (default: None)
  -l LOCAL_PATH, --local-path LOCAL_PATH
                        specify a local path for temporary files to be loaded. (default: None)
  -p PREFIX, --prefix PREFIX
                        Key prefix to be used while uploading to a specified bucket (default: s5cmd-benchmarks-)
  -hf HYPERFINE_EXTRA_FLAGS, --hyperfine-extra-flags HYPERFINE_EXTRA_FLAGS
                        hyperfine global extra flags. Write in between quotation marks and start with a space to avoid bugs. (default: None)
  -sf S5CMD_EXTRA_FLAGS, --s5cmd-extra-flags S5CMD_EXTRA_FLAGS
                        s5cmd global extra flags. Write in between quotation marks and start with a space to avoid bugs. (default: None)

Examples

./bench.py --bucket tempbucket --s5cmd v2.0.0 456 --warmup 2 --runs 10

Above command will compare v2.0.0 to PR:456 with 2 warmup runs and 10 benchmark runs.

./bench.py --bucket tempbucket --s5cmd v2.0.0 456 --warmup 2 --runs 10 -sf " --log error" -hf " --show-output"

When using -hf and -sf flags, use quotes like above and start with an empty space. If not started with an empty space, it might give an error. This is a known issue with argparse and this discussion can be useful to understand the problem deeper.

Assumes that user has two copies of s5cmd one is default commandline app, second one is in the current directory. A bucket name MUST be provided with -b flag. User can optionally: - add a key prefix with -k flag. - specify hyperfine warmup counts with -w flag - specify hyperfine runs counts with -r flag. Example calls `./benchmark.sh -b mcanktmpbuck` `./benchmark.sh -w 1 -r 4 -b mcanktmpbuck -k example_key_prefix `

… the tests. user can specify either version(tag), commit hash, or the PR number of the s5cmd that are to be used in the tests. Two versions of the s5cmd will be used namely old and new. They can be specified by -o and -n flags, respectively. Though version specified by o does not have to be older than that of specified by -n flag. Default values of versions are v1.4.0 for old (-o), and v2.0.0 for new (-n). Example execution that compares performance of #456 with the v1.4.0: ` ./benchmark.sh -b mys5cmbuck -n 456 -o v1.4.0 -k 1829` Also note that user must provide a proper bucket which she has write/read/delete access. Co-Authored-By: boraberke <67373739+boraberke@users.noreply.github.com>

- Measure the download and delete speeds - quote shell variables, see also "shellcheck(SC2086)"

…into benchmark-script

… call

boraberke · 2022-07-22T11:27:08Z

Results will be both printed out to console and saved to bench_results.md file, which looks like below:

Benchmark summary:

Scenario	Summary
upload small files	'PR:473' ran 2.65 times faster than 'version:v2.0.0'
download small files	'PR:473' ran 1.03 times faster than 'version:v2.0.0'
upload large files	'version:v2.0.0' ran 1.09 times faster than 'PR:473'
download large files	'PR:473' ran 1.24 times faster than 'version:v2.0.0'
remove small files	'PR:473' ran 1.07 times faster than 'version:v2.0.0'
remove large files	'PR:473' ran 1.11 times faster than 'version:v2.0.0'

Detailed summary:

Scenario	Command	Mean [ms]	Min [ms]	Max [ms]	Relative
upload small files	`PR:473`	241.2	241.2	241.2	1.00
upload small files	`version:v2.0.0`	639.9	639.9	639.9	2.65
download small files	`PR:473`	19.0	19.0	19.0	1.00
download small files	`version:v2.0.0`	19.5	19.5	19.5	1.03
upload large files	`PR:473`	153.2	153.2	153.2	1.09
upload large files	`version:v2.0.0`	140.0	140.0	140.0	1.00
download large files	`PR:473`	20.3	20.3	20.3	1.00
download large files	`version:v2.0.0`	25.2	25.2	25.2	1.24
remove small files	`PR:473`	28.1	28.1	28.1	1.00
remove small files	`version:v2.0.0`	30.0	30.0	30.0	1.07
remove large files	`PR:473`	27.5	27.5	27.5	1.00
remove large files	`version:v2.0.0`	30.6	30.6	30.6	1.11

sonmezonur

Please add --help command to your benchmark script. Otherwise users need to read this documentation before using this script

boraberke · 2022-07-27T08:39:04Z

Please add --help command to your benchmark script. Otherwise users need to read this documentation before using this script

Just updated from shell script to python script. Users now can use --help command to see the usage of this script 👍

also delete unnecessary file.

benchmark/bench.py

Before this commit, for each individual command, a scenario was required. These scenarios were dependent to each other. However, with this commit, any scenario will run `upload`, `download`, `remove` commands for a specified file size and file count.

Add a wait of 10 seconds after preparation to overcome errors due to eventual consistency of s3

benchmark/bench.py

kucukaslan and others added 11 commits July 19, 2022 14:17

choose appropriate command to create temporary file for each OS

ea9fad2

- Measure the download and delete speeds - quote shell variables, see also "shellcheck(SC2086)"

Merge branch 'master' into benchmark-script

d15f2fc

Beautify and refactor code

4857cc8

Merge branch 'benchmark-script' of https://github.com/Kucukaslan/s5cmd …

ce69e27

…into benchmark-script

add optional flags to add global flags to hyperfine (h) and s5cmd (f)…

eaf4e31

… call

Add color and fix bugs

1280eba

Add Markdown Bench Results

240e725

Add brief summary to output file

8cc4b53

Add file size and file count as parameters

7b54c7a

boraberke requested a review from a team as a code owner July 22, 2022 11:26

boraberke requested review from igungor and sonmezonur and removed request for a team July 22, 2022 11:26

boraberke added 2 commits July 22, 2022 14:28

Move under benchmark folder

10fd45d

Fix typo

f63e3b9

sonmezonur reviewed Jul 25, 2022

View reviewed changes

boraberke added 2 commits July 27, 2022 11:17

Switch from shell to python

8975da0

Update help

d31d2b0

boraberke changed the title ~~Add benchmark script to compare Upload/Download/Remove speed~~ Add benchmark script to compare two different builds of s5cmd Jul 27, 2022

boraberke and others added 7 commits July 27, 2022 13:18

Refactor code

596f07a

remove unused imports

9938ec4

also delete unnecessary file.

add default value for s5cmd-extra-flags as empty string

fa4b70c

add print error.

df4f5aa

add teardown for finished scenarios

8dc009e

add output flag

1d01b05

add more output to summary.md file

674a117

ilkinulas reviewed Aug 3, 2022

View reviewed changes

benchmark/bench.py Outdated Show resolved Hide resolved

benchmark/bench.py Outdated Show resolved Hide resolved

boraberke dismissed igungor’s stale review via 3c45159 August 3, 2022 14:32

Improve prepare of remove

9048201

Add a wait of 10 seconds after preparation to overcome errors due to eventual consistency of s3

kucukaslan mentioned this pull request Aug 8, 2022

Concurrency flag performance #418

Open

boraberke added 6 commits August 10, 2022 17:57

fix pull request checkout

63f9da3

fix time unit to seconds

4a2a691

Update CHANGELOG.md

b2da91d

change milliseconds to seconds

475861f

fix time unit to seconds

167529c

remove stdout print to a file

d1f4c4d

sonmezonur previously approved these changes Aug 13, 2022

View reviewed changes

ilkinulas reviewed Aug 14, 2022

View reviewed changes

benchmark/bench.py Show resolved Hide resolved

benchmark/bench.py Outdated Show resolved Hide resolved

benchmark/bench.py Outdated Show resolved Hide resolved

remove raise SystemExit

fe5b05d

boraberke dismissed sonmezonur’s stale review via fe5b05d August 15, 2022 13:22

add external dependency check

a86b709

ilkinulas reviewed Aug 15, 2022

View reviewed changes

benchmark/bench.py Outdated Show resolved Hide resolved

boraberke added 4 commits August 16, 2022 11:38

change default compared s5cmd builds

8498689

change summary file creation order to after run

6540bbc

add readme file for benchmark script

f4e9004

rename method

8369112

ilkinulas reviewed Aug 16, 2022

View reviewed changes

benchmark/bench.py Outdated Show resolved Hide resolved

boraberke added 2 commits August 16, 2022 16:08

escape parantheses

213f7d5

add unit tests

3d6ea84

ilkinulas previously approved these changes Aug 16, 2022

View reviewed changes

remove unnecessary comments

2df9f01

boraberke dismissed ilkinulas’s stale review via 2df9f01 August 16, 2022 15:57

ilkinulas approved these changes Aug 17, 2022

View reviewed changes

sonmezonur approved these changes Aug 17, 2022

View reviewed changes

igungor merged commit 914e701 into peak:master Aug 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmark script to compare two different builds of s5cmd#471