Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org [duplicate]

Question

I'm practicing the code from 'Web Scraping with Python', and I keep having this certificate problem:

from urllib.request import urlopen 
from bs4 import BeautifulSoup 
import re

pages = set()
def getLinks(pageUrl):
    global pages
    html = urlopen("http://en.wikipedia.org"+pageUrl)
    bsObj = BeautifulSoup(html)
    for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href'] 
                print(newPage) 
                pages.add(newPage) 
                getLinks(newPage)
getLinks("")

The error is:

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>

Btw,I was also practicing scrapy, but kept getting the problem: command not found: scrapy (I tried all sorts of solutions online but none works... really frustrating)

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)> — Catherine4j
– Catherine4j, Commented May 8, 2018 at 14:32
and... please tell me the reason behind this error, really want to know~~thanks!! — Catherine4j
– Catherine4j, Commented May 8, 2018 at 14:36
There are 529 existing questions on SSL: CERTIFICATE_VERIFY_FAILED, please figure out which is your solution then close this as duplicate. — smci
– smci, Commented May 9, 2018 at 0:02
export SSL_CERT_DIR=/etc/ssl/certs worked for me on Mac OS Big Sur. — TrigonaMinima
– TrigonaMinima, Commented Oct 8, 2021 at 20:55

Jey Miranda · Accepted Answer · 2018-11-14 23:57:27Z

1067

Once upon a time I stumbled with this issue. If you're using macOS go to Macintosh HD > Applications > Python3.6 folder (or whatever version of python you're using) > double click on "Install Certificates.command" file. :D

answered Nov 14, 2018 at 23:57

Jey Miranda

10.8k1 gold badge13 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

19 Comments

Catherine4j Over a year ago

oh I don't have this Install Certificates.command file on my mac... don't know why TAT

Sampo Over a year ago

If you install Python using Homebrew that file does not exist. The solution is here: stackoverflow.com/a/44649450/412896

Aakash Basu Over a year ago

How can I do it for Windows 10? No such file exists.

ihightower Over a year ago

See answer from @Rambod which is fast and quick and solved for me. import ssl ssl._create_default_https_context = ssl._create_unverified_context that's it and it worked like a charm. Yes there may be security isue, but for one off scripts and known source.. who gives a *

WHOATEMYNOODLES Over a year ago

What's the window 10 solution?

|

Rambod · Accepted Answer · 2020-03-13 13:22:38Z

235

to use unverified ssl you can add this to your code:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

answered Mar 13, 2020 at 13:22

Rambod

2,7431 gold badge17 silver badges15 bronze badges

5 Comments

Nagri Over a year ago

this is good because why do I have to make system changes if all I need to add is two lines of code.

Michael Altfield Over a year ago

this answer should probably mention that this code could introduce huge security concerns depending on what the user does with the data after it's scraped..

Rambod Over a year ago

for sure when you use unverified certificate you are at risk. and this line of code only good if you know what data source is and its not good for production mode to set it up as default. question is about how to scarp some unverified page this code wont make big security concern for this situation at least.

thewaywewere Over a year ago

this works in my case as a temporary measure

Cyborg Over a year ago

1 line: import ssl; ssl._create_default_https_context = ssl._create_unverified_context thanks @Rambod 👍

Hillsie · Accepted Answer · 2019-09-17 07:01:18Z

48

This terminal command:

open /Applications/Python\ 3.7/Install\ Certificates.command

Found here: https://stackoverflow.com/a/57614113/6207266

Resolved it for me. With my config

pip install --upgrade certifi

had no impact.

answered Sep 17, 2019 at 7:01

Hillsie

7251 gold badge7 silver badges16 bronze badges

3 Comments

Hillsie Over a year ago

@MiguelSilva open is specific to MacOS. You have Google colab running with MacOS? Wouldn't know about colab.

Miguel Tomás Over a year ago

In my case, I'm running on Opera browser in a win10 intel I7 16Gb MacBook Pro

Hillsie Over a year ago

@MiguelSilva The above command is at the operating system level and independent of the browser. Its related to Python. Additionally, there is one constant in this industry and it's Change. So what worked in the past, might not work in the present.

quant · Accepted Answer · 2018-11-15 11:34:27Z

36

To solve this:

All you need to do is to install Python certificates! A common issue on macOS.

Open these files:

Install Certificates.command
Update Shell Profile.command

Simply Run these two scripts and you wont have this issue any more.

Hope this helps!

edited Nov 15, 2018 at 11:34

quant

2,2142 gold badges22 silver badges33 bronze badges

answered Nov 15, 2018 at 10:39

Azim

4994 silver badges8 bronze badges

6 Comments

Kai Over a year ago

I don't see a Python directory in my Application Dir.. I'm using Anaconda and created a new env for python 3.6 since I previously had python 2.7 installed (which apparently did not create a folder in my Applications Dir either!)... Can't find the Certificates.command on my mac

baxx Over a year ago

where are these supposed to be run from?

Betty Over a year ago

@Azim: how about in Ubuntu?

oeter Over a year ago

Why did this answer even get upvoted at all? It doesn't specify where these commands are supposed to be run, and it lacks information about those "two scripts"

sovemp Over a year ago

I'm running into his problem with a Python lambda in AWS, so this doesn't help because I don't have access to this.

|

Hemant · Accepted Answer · 2019-09-15 05:28:43Z

28

For novice users, you can go in the Applications folder and expand the Python 3.7 folder. Now first run (or double click) the Install Certificates.command and then Update Shell Profile.command

answered Sep 15, 2019 at 5:28

Hemant

4,6868 gold badges43 silver badges43 bronze badges

Comments

Tony · Accepted Answer · 2023-06-08 19:44:08Z

27

This is the only solution that worked for me in a windows pc

pip install pip_system_certs
pip install python-certifi-win32

answered Jun 8, 2023 at 19:44

Tony

4476 silver badges5 bronze badges

7 Comments

bdutta74 Over a year ago

Worked for me on getting PrivateGPT to work locally in Ubuntu 22.04 WSL

janicebaratheon Over a year ago

Worked on Windows 11 under company's VPN.

user3408245 Over a year ago

Worked for me after applying a pip version update for simple-salesforce library and login.

FMFF Over a year ago

I didn't have to execute the second pip install. Just the first one helped me.

tb. Jan 2 at 14:15

Try the first line only before the second. python-certifi-win32 jacked things up for me worse than when I started, but pip_system_certs ultimately seemed to have resolved the issue for me.

|

VIC3KING · Accepted Answer · 2023-06-12 21:05:01Z

24

If you're running on a Mac you could just search for Install Certificates.command on the spotlight and hit enter.

edited Jun 12, 2023 at 21:05

answered Nov 28, 2019 at 6:44

VIC3KING

9321 gold badge7 silver badges12 bronze badges

2 Comments

Jayakrishnan Over a year ago

For mac users, this is all you need.

Ermiya Eskandary Over a year ago

This is correct however make sure you are doing it for the correct Python version. If you have multiple Python installations, do /Applications/Python\ 3.12/Install\ Certificates.command for example explicitly with the specific version,.

liakoyras · Accepted Answer · 2019-09-12 09:39:05Z

15

For anyone who is using anaconda, you would install the certifi package, see more at:

https://anaconda.org/anaconda/certifi

To install, type this line in your terminal:

conda install -c anaconda certifi

edited Sep 12, 2019 at 9:39

liakoyras

1,20514 silver badges28 bronze badges

answered Sep 12, 2019 at 8:38

Amy Mou

1511 silver badge3 bronze badges

3 Comments

Kai Over a year ago

I have this problem too. Can't create an environment. I just installed miniconda3 (macOS) and I see certifi installed ('conda list')..

vielfarbig Over a year ago

Works, but need a reselection of the kernel in a jupyter notebook.

log0 Over a year ago

that worked for me on ubuntu 22 inside conda

Muzammil-cyber · Accepted Answer · 2021-07-29 16:14:29Z

14

open /Applications/Python\ 3.7/Install\ Certificates.command

Try this command in terminal

answered Jul 29, 2021 at 16:14

Muzammil-cyber

1811 silver badge7 bronze badges

2 Comments

eddyoc Over a year ago

For me this simply yields

ERROR: Could not install packages due to an OSError: Cannot move the non-empty directory '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi-2021.10.8.dist-info/': Lacking write permission to '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi-2021.10.8.dist-info/'.

Muzammil-cyber Over a year ago

@eddyoc Find the Certificates.command manually and run it. Hopefully, that would work

Alexis Berson · Accepted Answer · 2019-07-27 02:21:12Z

9

Two steps worked for me : - going Macintosh HD > Applications > Python3.7 folder - click on "Install Certificates.command"

answered Jul 27, 2019 at 2:21

Alexis Berson

911 silver badge1 bronze badge

1 Comment

Krishnadas PC Over a year ago

It still worked perfectly on python 3.10 as well.

Alexandre Crivellaro · Accepted Answer · 2020-02-07 00:04:25Z

8

I could find this solution and is working fine:

cd /Applications/Python\ 3.7/
./Install\ Certificates.command

answered Feb 7, 2020 at 0:04

Alexandre Crivellaro

9139 silver badges10 bronze badges

Comments

Milovan Tomašević · Accepted Answer · 2021-02-07 23:21:01Z

I had the same error and solved the problem by running the program code below:

# install_certifi.py
#
# sample script to install or update a set of default Root Certificates
# for the ssl module.  Uses the certificates provided by the certifi package:
#       https://pypi.python.org/pypi/certifi

import os
import os.path
import ssl
import stat
import subprocess
import sys

STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
             | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP
             | stat.S_IROTH |                stat.S_IXOTH )


def main():
    openssl_dir, openssl_cafile = os.path.split(
        ssl.get_default_verify_paths().openssl_cafile)

    print(" -- pip install --upgrade certifi")
    subprocess.check_call([sys.executable,
        "-E", "-s", "-m", "pip", "install", "--upgrade", "certifi"])

    import certifi

    # change working directory to the default SSL directory
    os.chdir(openssl_dir)
    relpath_to_certifi_cafile = os.path.relpath(certifi.where())
    print(" -- removing any existing file or link")
    try:
        os.remove(openssl_cafile)
    except FileNotFoundError:
        pass
    print(" -- creating symlink to certifi certificate bundle")
    os.symlink(relpath_to_certifi_cafile, openssl_cafile)
    print(" -- setting permissions")
    os.chmod(openssl_cafile, STAT_0o775)
    print(" -- update complete")

if __name__ == '__main__':
    main()

Patrick Suzuki · Accepted Answer · 2018-08-10 03:44:26Z

6

Take a look at this post, it seems like for later versions of Python, certificates are not pre installed which seems to cause this error. You should be able to run the following command to install the certifi package: /Applications/Python\ 3.6/Install\ Certificates.command

Post 1: urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error

Post 2: Airbrake error: urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

answered Aug 10, 2018 at 3:44

Patrick Suzuki

571 silver badge5 bronze badges

Comments

Catherine4j · Accepted Answer · 2018-05-09 13:33:52Z

3

i didn't solve the problem, sadly. but managed to make to codes work (almost all of my codes have this probelm btw) the local issuer certificate problem happens under python3.7 so i changed back to python2.7 QAQ and all that needed to change including "from urllib2 import urlopen" instead of "from urllib.request import urlopen" so sad...

answered May 9, 2018 at 13:33

Catherine4j

3,2182 gold badges10 silver badges10 bronze badges

Comments

Tobias Domhan · Accepted Answer · 2023-03-02 10:45:46Z

2

I had the problem that python somehow was trying to use a cert.pem file that didn't exist. This can be seen by running:

import ssl
paths = ssl.get_default_verify_paths()

The openssl_cafile pointed to /etc/ssl/cert.pem which did not exist under that path.

Setting SSL_CERT_FILE to a path that does exist solved the problem:

export SSL_CERT_FILE=/etc/pki/tls/cert.pem

answered Mar 2, 2023 at 10:45

Tobias Domhan

3,3161 gold badge21 silver badges13 bronze badges

1 Comment

arivero Over a year ago

I noticed this problem in a slurm + modules cluster, typically a module for python sets different path.

Nitin · Accepted Answer · 2019-09-12 09:46:58Z

0

Use requests library. Try this solution, or just add https:// before the URL:

import requests
from bs4 import BeautifulSoup
import re

pages = set()
def getLinks(pageUrl):
    global pages
    html = requests.get("http://en.wikipedia.org"+pageUrl, verify=False).text
    bsObj = BeautifulSoup(html)
    for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href']
                print(newPage)
                pages.add(newPage)
                getLinks(newPage)
getLinks("")

Check if this works for you

answered Sep 12, 2019 at 9:46

Nitin

2441 silver badge7 bronze badges

Comments

user3303164 · Accepted Answer · 2020-03-22 18:25:03Z

I'm a relative novice compared to all the experts on Stack Overflow.

I have 2 versions of jupyter notebook running (one through a fresh Anaconda Navigator installation and one through ????). I think this is because Anaconda was installed as a local installation on my Mac (per Anaconda instructions).

I already had python 3.7 installed. After that, I used my terminal to open jupyter notebook and I think that it put another version globally onto my Mac.

However, I'm not sure because I'm just learning through trial and error!

I did the terminal command:

conda install -c anaconda certifi

(as directed above, but it didn't work.)

My python 3.7 is installed on OS Catalina10.15.3 in:

/Library/Python/3.7/site-packages AND
~/Library/Python/3.7/lib/python/site-packages

The certificate is at:

~/Library/Python/3.7/lib/python/site-packages/certifi-2019.11.28.dist-info

I tried to find the Install Certificate.command ... but couldn't find it through looking through the file structures...not in Applications...not in links above.

I finally installed it by finding it through Spotlight (as someone suggested above). And it double clicked automatically and installed ANOTHER certificate in the same folder as:

~/Library/Python/3.7/lib/python/site-packages/

NONE of the above solved anything for me...I still got the same error.

So, I solved the problem by:

closing my jupyter notebook.
opening Anaconda Navigator.
opening jupyter notebook through the Navigator GUI (instead of through Terminal).
opening my notebook and running the code.

I can't tell you why this worked. But it solved the problem for me.

I just want to save someone the hassle next time. If someone can tell my why it worked, that would be terrific.

I didn't try the other terminal commands because of the 2 versions of jupyter notebook that I knew were a problem. I just don't know how to fix that.

Saurabh · Accepted Answer · 2020-07-14 05:28:38Z

-1

This will work. Set the environment variable PYTHONHTTPSVERIFY to 0.

By typing linux command:

export PYTHONHTTPSVERIFY = 0

OR

Using in python code:

import os
os.environ["PYTHONHTTPSVERIFY"] = "0"

answered Jul 14, 2020 at 5:28

Saurabh

711 silver badge7 bronze badges

1 Comment

钟智强 Over a year ago

this doesn't help at all!

Weston A. Greene · Accepted Answer · 2020-10-01 08:56:49Z

-1

For me the problem was that I was setting REQUESTS_CA_BUNDLE in my .bash_profile

/Users/westonagreene/.bash_profile:
...
export REQUESTS_CA_BUNDLE=/usr/local/etc/openssl/cert.pem
...

Once I set REQUESTS_CA_BUNDLE to blank (i.e. removed from .bash_profile), requests worked again.

export REQUESTS_CA_BUNDLE=""

The problem only exhibited when executing python requests via a CLI (Command Line Interface). If I ran requests.get(URL, CERT) it resolved just fine.

Mac OS Catalina (10.15.6). Pyenv of 3.6.11. Error message I was getting: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

My answer elsewhere: https://stackoverflow.com/a/64151964/4420657

answered Oct 1, 2020 at 8:56

Weston A. Greene

1611 gold badge2 silver badges11 bronze badges

Comments

tedy58 · Accepted Answer · 2020-11-27 13:07:25Z

I am using Debian 10 buster and try download a file with youtube-dl and get this error: sudo youtube-dl -k https://youtu.be/uscis0CnDjk

[youtube] uscis0CnDjk: Downloading webpage ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)> (caused by URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

Certificates with python2 and python3.8 are installed correctly, but i persistent receive the same error. finally (which is not the best solution, but works for me was to eliminate the certificate check as it is given as an option in youtube-dl) whith this command sudo youtube-dl -k --no-check-certificate https://youtu.be/uscis0CnDjk

FriendFX · Accepted Answer · 2021-05-13 02:45:44Z

I am seeing this issue on a Ubuntu 20.04 system and none of the "real fixes" (like this one) helped.

While Firefox was willing to open the site just fine neither GNOME Web (i.e. Epiphany) nor Python3 or wget were accepting the certificate. After some searching, I came across this answer on ServerFault which lists two common reasons:

The certificate is really signed by an unknown CA (for instance an internal CA).

The certificate is signed with an intermediate CA certificate from one of the well known CA's and the remote server is misconfigured in the regard that it doesn't include that intermediate CA certificate as a CA chain it's response.

You can use the Qualys SSL Labs website to check the site's certificates and if there are issues, contact the site's administrator to have it fixed.

If you really need to work around the issue right now, I'd recommend a temporary solution like Rambod's confined to the site(s) you're trying to access.

SimfikDuke · Accepted Answer · 2022-01-13 11:27:02Z

-1

BTW guys if you are getting the same error using aiohttp just put verify_ssl=False argument into your TCPConnector:

import aiohttp
...

async with aiohttp.ClientSession(
    connector=aiohttp.TCPConnector(verify_ssl=False)
) as session:
    async with session.get(url) as response:
        body = await response.text()

answered Jan 13, 2022 at 11:27

SimfikDuke

1,17710 silver badges22 bronze badges

Comments

Ishmam Choudhury · Accepted Answer · 2022-08-03 18:53:30Z

-1

Make sure your websockets is >=10.0

Additional to: Install Certificates.command Update Shell Profile.command

pip3 install websockets==10.0

answered Aug 3, 2022 at 18:53

Ishmam Choudhury

11 bronze badge

1 Comment

Aaron Meese Over a year ago

Could you add some context to your answer, so readers know why the websockets version is important? Thanks! :)

thirukarthika nadar · Accepted Answer · 2023-07-15 13:26:03Z

-1

''''

import pymongo

from pymongo.mongo_client import MongoClient

CONNECTION_STRING = "mongodb+srv://username:[email protected]/?retryWrites=true&w=majority**&ssl_cert_reqs=CERT_NONE**"

client = pymongo.MongoClient(CONNECTION_STRING )

''''

In windows I tried to connect mongodb with jupyter notebook, finally by adding &ssl_cert_reqs=CERT_NONE in my CONNECTION_STRING helped me.

This basically disables SSL certificate verification (not recommended for production)

answered Jul 15, 2023 at 13:26

thirukarthika nadar

1

Comments

arun n a · Accepted Answer · 2023-10-09 16:55:37Z

-1

I got this error "_ssl.c:1091" in python 3.7 environment on ubuntu 18.04. No solution mentioned worked for me. Finally, through trial and error when I created a new environment with 3.11.5 and related libraries along with conda 11.7, resolved the issue for me.

edited Oct 9, 2023 at 16:55

arun n a

91612 silver badges33 bronze badges

answered Oct 5, 2023 at 4:17

Arvind Chandel

1

1 Comment

arun n a Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Chandan · Accepted Answer · 2024-01-03 07:56:49Z

Setting these in .zprofile and .zshenv in Mac-sonoma os worked.

This worked after looking at the results of the below snippet :

import ssl
paths = ssl.get_default_verify_paths()
print(paths)

DefaultVerifyPaths(cafile='/Library/Application Support/somefolder/somefolder/download/nscacert.pem', capath='/Library/Application Support/somefolder/somefolder/download/', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Users/chvr/anaconda3/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Users/chvr/anaconda3/ssl/certs')

export SSL_CERT_DIR="/Library/Application Support/folder/folder/download/"
export SSL_CERT_FILE="/Library/Application Support/folder/folder/download/nscacert.pem"

Execute these commands in terminal and ensure .zhrc / .zprofile file has these 2 path entries. If not add it manually and restart your python terminal.

Wayne Wignes · Accepted Answer · 2022-01-08 18:22:44Z

-2

I am using anaconda on windows. Was getting the same error until I tried the following;

import urllib.request
link = 'http://docs.python.org'
with urllib.request.urlopen(link) as response:
    htmlSource = response.read()

which I got from the stackoverflow thread on using urlopen:

Python urllib urlopen not working

answered Jan 8, 2022 at 18:22

Wayne Wignes

1

Collectives™ on Stack Overflow

Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org [duplicate]

27 Answers 27

19 Comments

5 Comments

3 Comments

6 Comments

Comments

7 Comments

2 Comments

3 Comments

2 Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

27 Answers 27

19 Comments

5 Comments

3 Comments

6 Comments

Comments

7 Comments

2 Comments

3 Comments

2 Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Linked

Related