gh-104773: PEP 594: Remove cgi and cgitb modules#104775

vstinner · 2023-05-22T23:50:29Z

test_pyclbr no longer tests the cgi module.

Issue: PEP 594: Remove stdlib modules scheduled for deletion in Python 3.13 #104773

📚 Documentation preview 📚: https://cpython-previews--104775.org.readthedocs.build/

vstinner · 2023-05-23T00:03:27Z

PR updated to fix the doc: Warning: (...)/Doc/library/security_warnings.rst:12: WARNING: undefined label: cgi-security.

vstinner · 2023-05-23T00:37:20Z

Update: fix test_pyclbr.

vstinner · 2023-05-23T07:28:13Z

It seems like cgi.escape() was used in Python 2. There is still compatibility code in the wild using it. Example with distlib/compat.py:

try:
    from html import escape
except ImportError:
    from cgi import escape

src: https://github.com/pypa/distlib/blob/05375908c1b2d6b0e74bdeb574569d3609db9f56/distlib/compat.py#L470-L474

vstinner · 2023-05-23T07:32:40Z

Using import cgi|from cgi regex, I found a bunch of PyPI top 5,000 projects (PYPI-2023-04-13) using cgi and cgitb modules.

Affected projects (91):

Cheetah3-3.2.6.post1
CherryPy (18.8.0)
Cython (0.29.34)
FormEncode (2.0.1)
Mako (1.2.4)
PTable (0.9.2)
Paste (3.5.2)
PasteDeploy (3.0.1)
PasteScript (3.3.0)
PyVISA (1.13.0)
Pyro4 (4.82)
ShopifyAPI (12.3.0)
Tempita (0.5.2)
Twisted (22.10.0)
WMI (1.5.1)
WebHelpers (1.3)
WebOb (1.8.7)
WebTest (3.0.0)
Whoosh (2.7.4)
alexapy (1.26.5)
aliyun-python-sdk-core (2.13.36)
ansible (7.4.0)
asana (3.2.0)
astropy (5.2.2)
asyncio (3.4.3)
atlassian-jwt-auth (17.0.0)
boltons (23.0.0)
boto (2.49.0)
braintree (4.19.0)
chispa (0.9.2)
cinemagoer (2022.12.27)
conda (4.3.16)
coreapi (2.3.3)
cssutils (2.6.0)
devpi-client (6.0.4)
distlib (0.3.6)
docxtpl (0.16.6)
eli5 (0.13.0)
emails (0.6)
eventlet (0.33.3)
falcon (3.1.1)
feedparser (6.0.10)
fissix (21.11.13)
gevent (22.10.2)
google-apitools (0.5.32)
gsutil (5.23)
gviz_api (1.10.0)
htmlmin (0.1.12)
ibm-cos-sdk-core (2.13.0)
isort (5.12.0)
jenkinsapi (0.3.13)
json-ref-dict (0.7.1)
json2html (1.3.0)
jusText (3.0.0)
lxml (4.9.2)
mechanize (0.4.8)
mercurial (6.4.1)
mozlog (7.1.1)
nibabel (5.1.0)
nltk (3.8.1)
numpy (1.24.2)
oauth (1.0.1)
oauth2client (4.1.3)
odfpy (1.4.1)
oletools (0.60.1)
os_sys (2.1.4)
pdfminer (20191125)
peppercorn (0.6)
pex (2.1.131)
prov (2.0.0)
pyLDAvis (3.4.0)
pyrocko (2023.3.27)
pysaml2 (7.4.1)
pystache (0.6.0)
python-mimeparse (1.6.0)
python-openid (2.2.5)
python3-openid (3.2.0)
pyvmomi (8.0.0.1.2)
scapy (2.5.0)
sentry-sdk (1.19.1)
setuptools (67.6.1)
simplekml (1.3.6)
speedtest-cli (2.1.3)
supervisor (4.2.5)
tableauserverclient (0.25)
tqdm (4.65.0)
twine (4.0.2)
uri (2.0.1)
web.py (0.62)
webapp2 (2.5.2)
zeep (4.2.1)

Details

Code:

Cheetah3-3.2.6.post1/Cheetah/Filters.py: # These substitutions are copied from cgi.escape().
Cheetah3-3.2.6.post1/Cheetah/Template.py: import cgi  # Used by .webInput() if the template is a CGI script.
Cheetah3-3.2.6.post1/Cheetah/Tests/Regressions.py: from cgi import escape as html_escape
CherryPy-18.8.0/cherrypy/lib/covercp.py: import cgi
CherryPy-18.8.0/cherrypy/lib/httputil.py: from cgi import parse_header
Cython-0.29.34/Cython/Tempita/_tempita.py: import cgi
FormEncode-2.0.1/formencode/fieldstorage.py: import cgi
FormEncode-2.0.1/formencode/htmlgen.py: from cgi import escape
FormEncode-2.0.1/formencode/rewritingparser.py: from cgi import escape
FormEncode-2.0.1/formencode/validators.py: import cgi
Mako-1.2.4/examples/wsgi/run_wsgi.py: import cgi
PTable-0.9.2/prettytable/_compact.py: from cgi import escape
Paste-3.5.2/paste/auth/open_id.py: import cgi
Paste-3.5.2/paste/cgitb_catcher.py: import cgitb
Paste-3.5.2/paste/debug/doctest_webapp.py: import cgi
Paste-3.5.2/paste/debug/prints.py: import cgi
Paste-3.5.2/paste/debug/profile.py: import cgi
Paste-3.5.2/paste/debug/watchthreads.py: import cgi
Paste-3.5.2/paste/debug/wdg_validate.py: import cgi
Paste-3.5.2/paste/exceptions/errormiddleware.py: import cgi
Paste-3.5.2/paste/request.py: import cgi
Paste-3.5.2/paste/util/multidict.py: import cgi
Paste-3.5.2/tests/cgiapp_data/form.cgi: import cgi
Paste-3.5.2/tests/test_fixture.py: import cgi
Paste-3.5.2/tests/test_multidict.py: import cgi
Paste-3.5.2/tests/test_wsgiwrappers.py: import cgi
PasteDeploy-3.0.1/src/paste/deploy/paster_templates/paste_deploy/+package+/sampleapp.py_tmpl: import cgi
PasteScript-3.3.0/paste/script/copydir.py: import cgi as html
PasteScript-3.3.0/paste/script/exe.py: # Maybe import cgitb or something?
PasteScript-3.3.0/paste/script/testapp.py: import cgi as html
PyVISA-1.13.0/pyvisa/thirdparty/prettytable.py: from cgi import escape
Pyro4-4.82/src/Pyro4/utils/httpgateway.py: import cgi
ShopifyAPI-12.3.0/shopify/collection.py: import cgi
Tempita-0.5.2/tempita/__init__.py: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger15.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger16.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger17.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger18.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger19.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger19a.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger19b.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger19c.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger20.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger21.tac: import cgi
Twisted-22.10.0/docs/core/howto/tutorial/listings/finger/finger22.py: import cgi
Twisted-22.10.0/src/twisted/web/http.py: import cgi
Twisted-22.10.0/src/twisted/web/test/test_cgi.py: import cgi
WMI-1.5.1/wmiweb.py: from cgi import escape
WebHelpers-1.3/webhelpers/util.py: from cgi import parse_qs
WebHelpers-1.3/webhelpers/util.py: import cgi
WebOb-1.8.7/src/webob/compat.py: from cgi import FieldStorage as _cgi_FieldStorage
WebOb-1.8.7/src/webob/compat.py: from cgi import FieldStorage as cgi_FieldStorage
WebOb-1.8.7/src/webob/compat.py: from cgi import escape
WebOb-1.8.7/src/webob/compat.py: from cgi import parse_header
WebOb-1.8.7/src/webob/compat.py: import cgi
WebOb-1.8.7/tests/test_compat.py: from webob.compat import cgi_FieldStorage
WebOb-1.8.7/tests/test_compat.py: from webob.compat import cgi_FieldStorage
WebOb-1.8.7/tests/test_compat.py: from webob.compat import cgi_FieldStorage
WebOb-1.8.7/tests/test_compat.py: from webob.compat import cgi_FieldStorage
WebOb-1.8.7/tests/test_in_wsgiref.py: import cgi
WebOb-1.8.7/tests/test_multidict.py: from cgi import FieldStorage
WebOb-1.8.7/tests/test_multidict.py: from cgi import FieldStorage
WebOb-1.8.7/tests/test_multidict.py: from cgi import FieldStorage
WebOb-1.8.7/tests/test_request.py: from cgi import FieldStorage
WebOb-1.8.7/tests/test_request.py: import cgi
WebOb-1.8.7/tests/test_request.py: import cgi
WebOb-1.8.7/tests/test_request.py: import cgi
WebTest-3.0.0/tests/test_forms.py: import cgi
Whoosh-2.7.4/src/whoosh/compat.py: import cgi
alexapy-1.26.5/alexapy/aiohttp/helpers.py: import cgi
aliyun-python-sdk-core-2.13.36/aliyunsdkcore/vendored/requests/utils.py: import cgi
ansible-7.4.0/ansible_collections/community/general/plugins/modules/campfire.py: import cgi
asana-3.2.0/tests/test_client_attachments.py: import cgi
astropy-5.2.2/cextern/cfitsio/docs/changes.txt: that even fits files returned from cgi queries which have the wrong
asyncio-3.4.3/examples/crawl.py: import cgi
atlassian-jwt-auth-17.0.0/atlassian_jwt_auth/key.py: import cgi
boltons-23.0.0/boltons/tableutils.py: from cgi import escape as html_escape
boltons-23.0.0/boltons/tableutils.py: from cgi import escape as html_escape
boto-2.49.0/boto/ecs/item.py: import cgi
braintree-4.19.0/braintree/apple_pay_gateway.py: from cgi import escape
chispa-0.9.2/chispa/prettytable.py: from cgi import escape
cinemagoer-2022.12.27/imdb/helpers.py: from cgi import escape
cinemagoer-2022.12.27/imdb/parser/http/piculet.py: from cgi import escape as html_escape
conda-4.3.16/conda/gateways/adapters/ftp.py: import cgi
coreapi-2.3.3/coreapi/codecs/download.py: import cgi
cssutils-2.6.0/cssutils/_fetchgae.py: import cgi
cssutils-2.6.0/cssutils/tests/test_util.py: import cgi
cssutils-2.6.0/encutils/__init__.py: import cgi
cssutils-2.6.0/tools/try.py: import cgi
devpi-client-6.0.4/testing/simpypi.py: from cgi import escape  # type: ignore
distlib-0.3.6/tests/test_database.py: import cgi
docxtpl-0.16.6/docxtpl/listing.py: from cgi import escape
docxtpl-0.16.6/docxtpl/richtext.py: from cgi import escape
docxtpl-0.16.6/docxtpl/template.py: from cgi import escape  # noqa: F401
eli5-0.13.0/eli5/formatters/html.py: from cgi import escape  # type: ignore
emails-0.6/emails/loader/helpers.py: import cgi
eventlet-0.33.3/tests/wsgi_test.py: import cgi
falcon-3.1.1/falcon/asgi/multipart.py: import cgi
falcon-3.1.1/falcon/media/multipart.py: import cgi
falcon-3.1.1/falcon/testing/helpers.py: import cgi
falcon-3.1.1/falcon/vendor/mimeparse/mimeparse.py: import cgi
feedparser-6.0.10/feedparser/encodings.py: import cgi
fissix-21.11.13/fissix/tests/test_util.py: self.assertEqual(str(node), "import cgi\nbar()\n\n")
gevent-22.10.2/examples/webproxy.py: from cgi import escape
gevent-22.10.2/src/greentest/2.7/test_httpservers.py: import cgi
gevent-22.10.2/src/greentest/2.7pypy/test_httpservers.py: import cgi
gevent-22.10.2/src/greentest/3.7pypy/test_httpservers.py: import cgi
google-apitools-0.5.32/apitools/base/protorpclite/test_util.py: import cgi
gsutil-5.23/gslib/vendored/boto/boto/ecs/item.py: import cgi
gsutil-5.23/gslib/vendored/oauth2client/oauth2client/contrib/appengine.py: import cgi
gviz_api-1.10.0/gviz_api.py: import cgi as html  # Only used for .escape()
htmlmin-0.1.12/htmlmin/escape.py: from cgi import escape
htmlmin-0.1.12/htmlmin/main.py: import cgi
ibm-cos-sdk-core-2.13.0/ibm_botocore/utils.py: import cgi
isort-5.12.0/tests/unit/profiles/test_django.py: """import cgi
jenkinsapi-0.3.13/examples/low_level/post_watcher.py: import cgi
jenkinsapi-0.3.13/jenkinsapi/utils/simple_post_logger.py: import cgi
json-ref-dict-0.7.1/json_ref_dict/loader.py: import cgi
json2html-1.3.0/json2html/jsonconv.py: from cgi import escape as html_escape
jusText-3.0.0/justext/_compat.py: from cgi import escape
lxml-4.9.2/doc/s5/ep2008/atom.py: import cgi
lxml-4.9.2/src/lxml/doctestcompare.py: from cgi import escape as html_escape
lxml-4.9.2/src/lxml/html/diff.py: from cgi import escape as html_escape
mechanize-0.4.8/examples/forms/echo.cgi: import cgi
mercurial-6.4.1/contrib/hgweb.fcgi: # import cgitb; cgitb.enable()
mercurial-6.4.1/contrib/hgweb.wsgi: #import cgitb; cgitb.enable()
mercurial-6.4.1/hgweb.cgi: # import cgitb; cgitb.enable()
mercurial-6.4.1/tests/test-clone-cgi.t: > import cgitb
mercurial-6.4.1/tests/test-mq.t: > import cgitb
mercurial-6.4.1/tests/test-newcgi.t: > import cgitb
mercurial-6.4.1/tests/test-newcgi.t: > import cgitb
mercurial-6.4.1/tests/test-newercgi.t: > import cgitb
mercurial-6.4.1/tests/test-newercgi.t: > import cgitb
mercurial-6.4.1/tests/test-oldcgi.t: > import cgitb, os, sys
mercurial-6.4.1/tests/test-oldcgi.t: > import cgitb, sys
mercurial-6.4.1/tests/test-push-cgi.t: > import cgitb
mozlog-7.1.1/mozlog/formatters/html/html.py: from cgi import escape
nibabel-5.1.0/tools/dicomfs.wsgi: import cgi
nltk-3.8.1/nltk/tree/prettyprinter.py: from cgi import escape
numpy-1.24.2/tools/npy_tempita/__init__.py: from cgi import escape as html_escape
oauth-1.0.1/oauth/oauth.py: import cgi
oauth2client-4.1.3/oauth2client/contrib/appengine.py: import cgi
odfpy-1.4.1/contrib/html2odt/html2odt.py: from cgi import escape,parse_header
odfpy-1.4.1/contrib/html2odt/shtml2odt.py: from cgi import escape,parse_header
oletools-0.60.1/oletools/thirdparty/prettytable/prettytable.py: from cgi import escape
os_sys-2.1.4/bars/bars/notebook.py: from cgi import escape
os_sys-2.1.4/bars/notebook.py: from cgi import escape
os_sys-2.1.4/os_sys/download.py: import cgi
os_sys-2.1.4/server/core/handlers/wsgi.py: import cgi
os_sys-2.1.4/server/core/management/templates.py: import cgi
os_sys-2.1.4/server/http/multipartparser.py: import cgi
pdfminer-20191125/tools/pdf2html.cgi: #import cgitb; cgitb.enable()
pdfminer-20191125/tools/pdf2html.cgi: import cgi, logging, traceback, random
peppercorn-0.6/peppercorn/tests.py: from cgi import FieldStorage
pex-2.1.131/pex/vendor/_vendored/pip/pip/_internal/index/collector.py: import cgi
pex-2.1.131/pex/vendor/_vendored/pip/pip/_internal/network/download.py: import cgi
prov-2.0.0/src/prov/dot.py: from cgi import escape
pyLDAvis-3.4.0/pyLDAvis/lib/python3.11/site-packages/setuptools/_distutils/config.py: import cgi
pyrocko-2023.3.27/src/gf/server.py: from cgi import escape
pyrocko-2023.3.27/src/gf/server.py: import cgi
pysaml2-7.4.1/src/saml2/httputil.py: import cgi
pysaml2-7.4.1/src/saml2/pack.py: import cgi as html  # type: ignore[no-redef]
pystache-0.6.0/pystache/defaults.py: from cgi import escape
python-mimeparse-1.6.0/mimeparse.py: import cgi
python-openid-2.2.5/contrib/openid-parse: import cgi, re, subprocess, sys
python-openid-2.2.5/examples/consumer.py: import cgi
python-openid-2.2.5/examples/consumer.py: import cgitb
python-openid-2.2.5/examples/djopenid/server/views.py: import cgi
python-openid-2.2.5/examples/server.py: import cgi
python-openid-2.2.5/examples/server.py: import cgitb
python-openid-2.2.5/openid/consumer/consumer.py: import cgi
python-openid-2.2.5/openid/test/test_auth_request.py: import cgi
python-openid-2.2.5/openid/test/test_consumer.py: import cgi
python-openid-2.2.5/openid/test/test_message.py: import cgi
python-openid-2.2.5/openid/test/test_server.py: import cgi
python3-openid-3.2.0/contrib/openid-parse: import cgi, re, subprocess, sys
python3-openid-3.2.0/examples/consumer.py: import cgitb
python3-openid-3.2.0/examples/djopenid/server/views.py: import cgi
python3-openid-3.2.0/examples/server.py: import cgitb
pyvmomi-8.0.0.1.2/pyVim/sso.py: from cgi import escape
scapy-2.5.0/scapy/compat.py: import cgi
sentry-sdk-1.19.1/sentry_sdk/integrations/pyramid.py: from webob.compat import cgi_FieldStorage  # type: ignore
sentry-sdk-1.19.1/sentry_sdk/utils.py: from cgi import parse_qs  # type: ignore
setuptools-67.6.1/setuptools/_distutils/config.py: import cgi
simplekml-1.3.6/simplekml/base.py: import cgi
speedtest-cli-2.1.3/speedtest.py: from cgi import parse_qs
supervisor-4.2.5/supervisor/compat.py: from cgi import escape
tableauserverclient-0.25/tableauserverclient/server/endpoint/datasources_endpoint.py: import cgi
tableauserverclient-0.25/tableauserverclient/server/endpoint/flows_endpoint.py: import cgi
tableauserverclient-0.25/tableauserverclient/server/endpoint/workbooks_endpoint.py: import cgi
tqdm-4.65.0/tqdm/notebook.py: from cgi import escape
twine-4.0.2/twine/commands/check.py: import cgi
uri-2.0.1/uri/compat.py: from cgi import escape
web.py-0.62/web/webapi.py: import cgi
webapp2-2.5.2/tests/request_test.py: import cgi
webapp2-2.5.2/tests/request_test.py: import cgi
webapp2-2.5.2/webapp2.py: import cgi
zeep-4.2.1/src/zeep/utils.py: import cgi

hugovk

🧹

vstinner · 2023-05-24T08:06:06Z

test_importlib modified the environment on the Ubuntu job. I wrote a fix for that: #104840

* Replace "cgi" with "!cgi" in the Sphinx documentation to avoid warnings on broken references. * test_pyclbr no longer tests the cgi module.

vstinner · 2023-05-24T08:30:45Z

I re-ran the Ubuntu job and test_importlib failed again.

test_importlib modified the environment on the Ubuntu job. I wrote a fix for that: #104840

I merged the test_importlib fix. I rebased my PR on top of it.

vstinner · 2023-05-24T09:06:03Z

Bye bye cgi! I added instructions on how to port existing code.

vstinner · 2023-05-31T12:39:15Z

@JulienPalard: Oh, I'm surprising that sphinx-lint missing this typo:

* :pep:`594`: Remove the :mod:`!cgi`` and :mod:`!cgitb` modules,
  deprecated in Python 3.11.

vstinner requested a review from ethanfurman as a code owner May 22, 2023 23:50

bedevere-bot added the awaiting core review label May 22, 2023

bedevere-bot mentioned this pull request May 22, 2023

PEP 594: Remove stdlib modules scheduled for deletion in Python 3.13 #104773

Closed

19 tasks

vstinner force-pushed the remove_cgi branch from 82772f2 to b32e052 Compare May 23, 2023 00:03

vstinner changed the title ~~gh-104773: PEP 594: Remove cgi and cgitb modules~~ gh-104773: PEP 594: Remove cgi and cgitb modules May 23, 2023

vstinner force-pushed the remove_cgi branch 2 times, most recently from 7a4a015 to 7cbc500 Compare May 23, 2023 00:36

gpshead approved these changes May 23, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 23, 2023

vstinner force-pushed the remove_cgi branch from 7cbc500 to b0d94a4 Compare May 23, 2023 07:22

hugovk self-requested a review May 23, 2023 09:44

hugovk approved these changes May 23, 2023

View reviewed changes

vstinner force-pushed the remove_cgi branch 2 times, most recently from c0001e7 to 734ae51 Compare May 24, 2023 07:38

vstinner added 2 commits May 24, 2023 10:30

pythongh-104773: PEP 594: Remove cgi and cgitb modules

10b3f34

* Replace "cgi" with "!cgi" in the Sphinx documentation to avoid warnings on broken references. * test_pyclbr no longer tests the cgi module.

Explain how to port existing code

5ad9cf6

vstinner force-pushed the remove_cgi branch from 734ae51 to 5ad9cf6 Compare May 24, 2023 08:30

vstinner enabled auto-merge (squash) May 24, 2023 08:57

vstinner merged commit 08d5923 into python:main May 24, 2023

bedevere-bot removed the awaiting merge label May 24, 2023

vstinner deleted the remove_cgi branch May 24, 2023 09:05

JelleZijlstra mentioned this pull request May 25, 2023

[BUG] Cython uses deprecated cgi module cython/cython#5449

Closed

vstinner mentioned this pull request May 31, 2023

Detect typo :mod:!cgi` sphinx-contrib/sphinx-lint#64

Closed

Uzume mentioned this pull request May 18, 2024

[WORKAROUND] Unable to build PyYAML < 6.0.1 from source or sdist yaml/pyyaml#736

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

gh-104773: PEP 594: Remove cgi and cgitb modules#104775