Skip to content

bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve#19639

Merged
miss-islington merged 5 commits intopython:masterfrom
ZackerySpytz:bpo-34204-shelve-pickle-DEFAULT_PROTOCOL
Oct 29, 2020
Merged

bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve#19639
miss-islington merged 5 commits intopython:masterfrom
ZackerySpytz:bpo-34204-shelve-pickle-DEFAULT_PROTOCOL

Conversation

@ZackerySpytz
Copy link
Contributor

@ZackerySpytz ZackerySpytz commented Apr 21, 2020 •
edited by miss-islington
Loading

Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.

https://bugs.python.org/issue34204

Automerge-Triggered-By: GH:csabella

@ZackerySpytz
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve
1c9214e 
Use pickle.DEFAULT_PROTOCOL (currently 4) in shelve instead of a
hardcoded 3.
corona10
corona10 reviewed Apr 22, 2020
View reviewed changes
Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you like to add NEWS on what's new?
It looks like noticiable changing for user.

@corona10 corona10 closed this Apr 23, 2020
@corona10 corona10 reopened this Apr 23, 2020
@corona10 corona10 requested a review from vstinner April 23, 2020 13:12
@corona10
Copy link
Member

corona10 commented Apr 23, 2020

@vstinner Do you have any ideas?

@vstinner
Copy link
Member

vstinner commented Apr 23, 2020

@vstinner Do you have any ideas?

I would prefer to get the approval of @pitrou or @applio on such change. pickle can be used to share data between different Python versions. I'm not sure about the backward compatibility here.

@pitrou
Copy link
Member

pitrou commented Apr 23, 2020

I don't think this is ok to change. cc @rhettinger

@vstinner
Copy link
Member

vstinner commented Apr 23, 2020

Oh sorry, I misunderstood the PR. I read that it changes the default pickle version in multiprocessing (oops), but it's about the shelve module :-)

@csabella
Copy link
Contributor

csabella commented May 27, 2020

@corona10 @vstinner @pitrou @applio There are two approvals on this PR, but I'm not sure if there's consensus. Does it need more discussion on the bug tracker or should it be merged?

@corona10
Copy link
Member

corona10 commented May 27, 2020

@csabella

Does it need more discussion on the bug tracker

IMHO, we need more discussion.

vstinner
vstinner approved these changes May 27, 2020
View reviewed changes
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if the PR is updated for Python 3.10 (right now, it says "new in 3.9").

Sorry, first I read that this PR was for multiprocessing... For shelve, IMO it's reaonsable to use the latest pickle version. The main drawback is that shelve files created by Python 3.10 can no longer be read by Python 3.7 and older: protocol 5 was introduced in Python 3.8.
https://docs.python.org/dev/library/pickle.html#data-stream-format

IMHO it's a reasonable limitation. Also, it's well documented that the pickle protocol can easily controlled by the protocol parameter of shelve.open():
https://docs.python.org/dev/library/shelve.html#shelve.open

@pitrou @corona10: What do you think?

@vstinner
Copy link
Member

vstinner commented May 27, 2020

Oh, @serhiy-storchaka created https://bugs.python.org/issue34204 so it would be nice to also hear @serhiy-storchaka on this PR :-)

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented May 27, 2020

We can also use just None instead of pickle.DEFAULT_PROTOCOL.

@vstinner
Copy link
Member

vstinner commented May 27, 2020

We can also use just None instead of pickle.DEFAULT_PROTOCOL.

I prefer to have s._protocol = pickle.DEFAULT_PROTOCOL than s._protocol = None, it's more explicit.

@pitrou
Copy link
Member

pitrou commented May 27, 2020

Also, it's well documented that the pickle protocol can easily controlled by the protocol parameter of shelve.open()

Yes, so if it's well documented, what's the point of changing the default and breaking compatibility?

@vstinner
Copy link
Member

vstinner commented May 27, 2020

Hum, I measured the file disk depending on the pickle protocol. The efficiency depends on values: https://bugs.python.org/issue34204

@marco-c
Copy link

marco-c commented Oct 19, 2020

@ZackerySpytz are you going to update the PR?
I have opened a new issue and PR as I didn't notice this, I will close my PR if you are planning on updating yours.

@marco-c
Copy link

marco-c commented Oct 28, 2020

My PR to fix the same issue is #22751.

@ZackerySpytz
Copy link
Contributor Author

ZackerySpytz commented Oct 28, 2020

I have just pushed an update.

@marco-c
Copy link

marco-c commented Oct 28, 2020

I have just pushed an update.

Thanks, I'll close mine then!

@miss-islington miss-islington merged commit df59273 into python:master Oct 29, 2020
shihai1991 added a commit to shihai1991/cpython that referenced this pull request Oct 29, 2020
@shihai1991
Merge remote-tracking branch 'origin/master' into bpo_41073_add_types…
b3f992a 
…lots1

* origin/master: (365 commits)
  bpo-42029: Remove IRIX code (pythonGH-23023)
  bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (pythonGH-22953)
  bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (pythonGH-19639)
  bpo-41805: Documentation for PEP 585 (pythonGH-22615)
  bpo-42161: Micro-optimize _collections._count_elements() (pythonGH-23008)
  bpo-42161: Remove private _PyLong_Zero and _PyLong_One (pythonGH-23003)
  bpo-42099: Fix reference to ob_type in unionobject.c and ceval (pythonGH-22829)
  bpo-41659: Disallow curly brace directly after primary (pythonGH-22996)
  bpo-6761: Enhance __call__ documentation (pythonGH-7987)
  bpo-42161: Modules/ uses _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22998)
  bpo-41474, Makefile: Add dependency on cpython/frameobject.h (pythonGH-22999)
  bpo-42157: Rename unicodedata.ucnhash_CAPI (pythonGH-22994)
  bpo-42161: Use _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22995)
  bpo-30681: Support invalid date format or value in email Date header (pythonGH-22090)
  bpo-42161: Add _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22993)
  bpo-42123: Run the parser two times and only enable invalid rules on the second run (pythonGH-22111)
  bpo-42157: Convert unicodedata.UCD to heap type (pythonGH-22991)
  bpo-42157: unicodedata avoids references to UCD_Type (pythonGH-22990)
  bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (pythonGH-22654)
  bpo-1635741: _PyUnicode_Name_CAPI moves to internal C API (pythonGH-22713)
  ...
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021
@ZackerySpytz @adorilson
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (pythonGH-19639)
e3cfa33 
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
@serhiy-storchaka serhiy-storchaka mentioned this pull request Apr 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 corona10 approved these changes

@vstinner vstinner vstinner approved these changes

@rhettinger rhettinger rhettinger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.