Update Brotli to v1.0.9 #44107

saucecontrol · 2020-10-31T17:26:50Z

This updates Brotli to the v1.0.9 release tag, which includes the x64 performance improvements detailed in dotnet/corefx#35315

Benchmark results from the dotnet/performance repo:

BenchmarkDotNet=v0.12.1.1448-nightly, OS=Windows 10.0.19042
Intel Core i7-6700K CPU 4.00GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=5.0.100-rc.2.20479.15
  [Host]     : .NET 5.0.0 (5.0.20.47505), X64 RyuJIT
  Job-XCWZJW : .NET 5.0 (42.42.42.42424), X64 RyuJIT
  Job-EJFMQA : .NET 5.0 (42.42.42.42424), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  IterationTime=250.0000 ms  
MaxIterationCount=20  MinIterationCount=15  WarmupCount=1

|                  Method | Branch |   level |             file |         Mean |       Error |      StdDev | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------------------ -------- |-------- |----------------- |-------------:|------------:|------------:|------:|--------:|------:|------:|------:|----------:|
|      Compress_WithState | master | Optimal | TestDocument.pdf | 461,408.5 μs | 1,300.48 μs | 1,216.47 μs |  1.00 |    0.00 |     - |     - |     - |   3,432 B |
|      Compress_WithState | pr     | Optimal | TestDocument.pdf | 465,973.2 μs | 1,174.48 μs | 1,098.61 μs |  1.01 |    0.00 |     - |     - |     - |   2,152 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Optimal | TestDocument.pdf |     848.7 μs |     4.92 μs |     4.11 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|    Decompress_WithState | pr     | Optimal | TestDocument.pdf |     852.5 μs |     7.96 μs |     7.06 μs |  1.01 |    0.01 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Optimal | TestDocument.pdf | 461,102.0 μs | 1,137.91 μs | 1,064.40 μs |  1.00 |    0.00 |     - |     - |     - |   3,112 B |
|   Compress_WithoutState | pr     | Optimal | TestDocument.pdf | 464,783.9 μs |   647.42 μs |   540.63 μs |  1.01 |    0.00 |     - |     - |     - |   1,464 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Optimal | TestDocument.pdf |     852.8 μs |     6.13 μs |     5.44 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Optimal | TestDocument.pdf |     847.7 μs |     4.19 μs |     3.92 μs |  0.99 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|      Compress_WithState | master | Optimal |      alice29.txt | 209,089.6 μs |   420.32 μs |   328.16 μs |  1.00 |    0.00 |     - |     - |     - |      64 B |
|      Compress_WithState | pr     | Optimal |      alice29.txt | 193,220.6 μs |   719.41 μs |   637.74 μs |  0.92 |    0.00 |     - |     - |     - |      64 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Optimal |      alice29.txt |     440.1 μs |     1.97 μs |     1.85 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|    Decompress_WithState | pr     | Optimal |      alice29.txt |     444.7 μs |     3.19 μs |     2.98 μs |  1.01 |    0.01 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Optimal |      alice29.txt | 209,511.9 μs | 1,022.20 μs |   956.17 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|   Compress_WithoutState | pr     | Optimal |      alice29.txt | 193,809.0 μs |   864.14 μs |   808.32 μs |  0.93 |    0.00 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Optimal |      alice29.txt |     442.4 μs |     2.91 μs |     2.58 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Optimal |      alice29.txt |     446.4 μs |     3.17 μs |     2.81 μs |  1.01 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|      Compress_WithState | master | Optimal |              sum |  49,844.3 μs |   392.49 μs |   367.13 μs |  1.00 |    0.00 |     - |     - |     - |      40 B |
|      Compress_WithState | pr     | Optimal |              sum |  46,203.0 μs |   252.14 μs |   223.51 μs |  0.93 |    0.01 |     - |     - |     - |      40 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Optimal |              sum |     156.8 μs |     0.63 μs |     0.56 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|    Decompress_WithState | pr     | Optimal |              sum |     154.3 μs |     0.69 μs |     0.65 μs |  0.98 |    0.00 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Optimal |              sum |  49,673.0 μs |   186.22 μs |   165.08 μs |  1.00 |    0.00 |     - |     - |     - |       8 B |
|   Compress_WithoutState | pr     | Optimal |              sum |  46,215.3 μs |   257.85 μs |   241.19 μs |  0.93 |    0.01 |     - |     - |     - |       8 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Optimal |              sum |     156.0 μs |     0.24 μs |     0.20 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Optimal |              sum |     153.7 μs |     0.34 μs |     0.29 μs |  0.99 |    0.00 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|      Compress_WithState | master | Fastest | TestDocument.pdf |     367.5 μs |     2.58 μs |     2.16 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|      Compress_WithState | pr     | Fastest | TestDocument.pdf |     352.9 μs |     5.20 μs |     4.35 μs |  0.96 |    0.01 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Fastest | TestDocument.pdf |     262.6 μs |     1.02 μs |     0.96 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|    Decompress_WithState | pr     | Fastest | TestDocument.pdf |     267.5 μs |     1.42 μs |     1.25 μs |  1.02 |    0.01 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Fastest | TestDocument.pdf |     367.8 μs |     2.90 μs |     2.72 μs |  1.00 |    0.00 |     - |     - |     - |         - |
|   Compress_WithoutState | pr     | Fastest | TestDocument.pdf |     362.6 μs |     3.48 μs |     2.90 μs |  0.99 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Fastest | TestDocument.pdf |     262.3 μs |     1.06 μs |     0.99 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Fastest | TestDocument.pdf |     267.6 μs |     1.32 μs |     1.17 μs |  1.02 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|      Compress_WithState | master | Fastest |      alice29.txt |   1,052.1 μs |    46.79 μs |    48.05 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|      Compress_WithState | pr     | Fastest |      alice29.txt |     791.2 μs |    18.67 μs |    18.33 μs |  0.75 |    0.02 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Fastest |      alice29.txt |     455.6 μs |     2.47 μs |     1.93 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|    Decompress_WithState | pr     | Fastest |      alice29.txt |     455.4 μs |     5.81 μs |     5.43 μs |  1.00 |    0.01 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Fastest |      alice29.txt |   1,032.1 μs |    29.12 μs |    29.91 μs |  1.00 |    0.00 |     - |     - |     - |         - |
|   Compress_WithoutState | pr     | Fastest |      alice29.txt |     812.5 μs |    57.99 μs |    59.55 μs |  0.79 |    0.04 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Fastest |      alice29.txt |     453.0 μs |     1.10 μs |     0.86 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Fastest |      alice29.txt |     455.3 μs |     5.39 μs |     5.04 μs |  1.00 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|      Compress_WithState | master | Fastest |              sum |     181.6 μs |     3.54 μs |     3.48 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|      Compress_WithState | pr     | Fastest |              sum |     153.7 μs |     2.89 μs |     2.70 μs |  0.85 |    0.03 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|    Decompress_WithState | master | Fastest |              sum |     118.8 μs |     0.33 μs |     0.28 μs |  1.00 |    0.00 |     - |     - |     - |      33 B |
|    Decompress_WithState | pr     | Fastest |              sum |     118.8 μs |     0.29 μs |     0.24 μs |  1.00 |    0.00 |     - |     - |     - |      32 B |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
|   Compress_WithoutState | master | Fastest |              sum |     181.3 μs |     0.94 μs |     0.79 μs |  1.00 |    0.00 |     - |     - |     - |         - |
|   Compress_WithoutState | pr     | Fastest |              sum |     151.6 μs |     0.88 μs |     0.74 μs |  0.84 |    0.01 |     - |     - |     - |         - |
|                         |        |         |                  |              |             |             |       |         |       |       |       |           |
| Decompress_WithoutState | master | Fastest |              sum |     118.3 μs |     0.19 μs |     0.16 μs |  1.00 |    0.00 |     - |     - |     - |         - |
| Decompress_WithoutState | pr     | Fastest |              sum |     118.5 μs |     0.54 μs |     0.50 μs |  1.00 |    0.00 |     - |     - |     - |         - |

Closes #44073

ghost · 2020-10-31T17:26:56Z

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

stephentoub · 2020-11-02T16:00:57Z

Thanks, @saucecontrol. Perf results look good: a few tiny regressions that appear to be within or close to noise, and a few measurable improvements.

stephentoub · 2020-11-02T16:09:29Z

Thanks, @saucecontrol! This LGTM.

I downloaded the 1.0.9 source and compared it to both what we currently have and what's in your PR, just to confirm the diff. I noticed the new brotli source doesn't have the common/dictionary.bin and common/dictionary.bin.br that we currently have checked in; are those still needed, or should they be deleted? I'm also not sure where our fuzz directory comes from: is that something that came with a previous version of brotli and is no longer there, or is that something we added separately from the brotli source?

stephentoub · 2020-11-02T16:20:54Z

cc: @carlossanlop @ericstj

saucecontrol · 2020-11-02T18:15:39Z

@stephentoub, I'm not sure what you mean about the dictionary and fuzz files. They're still in the Brotli repo: https://github.com/google/brotli/tree/v1.0.9/c/common and https://github.com/google/brotli/tree/v1.0.9/c/fuzz

We don't use the fuzz folder, but I think I remember seeing a comment from an earlier PR that tried to delete them, and the decision was we should just have an untouched copy of the c folder from source repo here.

stephentoub · 2020-11-02T19:59:48Z

I'm not sure what you mean about the dictionary and fuzz files. They're still in the Brotli repo

Interesting. They're in the repo, but they're not in the tagged 1.0.9 zip or tar.gz downloads (which is what I was looking at):
https://github.com/google/brotli/releases/tag/v1.0.9

saucecontrol · 2020-11-02T20:11:44Z

They're definitely not needed for the build. I'm happy to delete them if you prefer to keep this copy minimal.

stephentoub · 2020-11-02T20:15:18Z

It's not relevant to your change, so don't worry about it. We might want to subsequently dump the unnecessary files, though. I'm happy to defer to @ericstj / @carlossanlop on that.

stephentoub · 2020-11-02T20:16:03Z

Thanks for fixing this.

saucecontrol added 2 commits October 31, 2020 10:20

sync brotli source

f8d76f5

update runtime project

fb9a728

Dotnet-GitSync-Bot added the area-Infrastructure-libraries label Oct 31, 2020

GrabYourPitchforks added area-System.IO.Compression and removed area-Infrastructure-libraries labels Oct 31, 2020

saucecontrol marked this pull request as ready for review November 2, 2020 03:44

saucecontrol changed the title ~~Update brotli to v1.0.9~~ Update Brotli to v1.0.9 Nov 2, 2020

stephentoub approved these changes Nov 2, 2020

View reviewed changes

stephentoub merged commit 84e9562 into dotnet:master Nov 2, 2020

saucecontrol deleted the brotli-update branch November 2, 2020 20:16

ghost locked as resolved and limited conversation to collaborators Dec 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update Brotli to v1.0.9 #44107

Update Brotli to v1.0.9 #44107

Uh oh!

saucecontrol commented Oct 31, 2020 •

edited

Loading

Uh oh!

ghost commented Oct 31, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

saucecontrol commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020 •

edited

Loading

Uh oh!

saucecontrol commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Update Brotli to v1.0.9 #44107

Update Brotli to v1.0.9 #44107

Uh oh!

Conversation

saucecontrol commented Oct 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Oct 31, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

saucecontrol commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saucecontrol commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

stephentoub commented Nov 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saucecontrol commented Oct 31, 2020 •

edited

Loading

stephentoub commented Nov 2, 2020 •

edited

Loading