405 Method not allowed when connecting to JupyterHub

[dobos]

Bug: Notebook Editor, Interactive Window, Editor cells

This is a follow up to https://github.com/microsoft/vscode-python/issues/6309 which has been locked but the issue persists.

Steps to cause the bug to occur

1. Run a password-protected jupyterhub server at some URL accessible from the machine running vscode, e.g. https://localhost:4443/jupyter/
2. Generate API token in JupyterHub (go to control panel, token link on top)
3. Set up "python.dataScience.jupyterServerURI" to point to the jupyterhub root URL with the token generated above, e.g. https://localhost:4443/jupyter/?token={guid_comes_here}
4. Try to run a jupyter notebook cell.

Actual behavior

You get the following error message:

Error: Failed to connect to remote Jupyter notebook.
Check that the Jupyter Server URI setting has a valid running server specified.
https://localhost:4443/jupyter/
TypeError: request to https://localhost:4443/jupyter/api/contents/?1579541567509 failed, reason: connect ECONNREFUSED 127.0.0.1:4443

Expected behavior

The error message above is correct since vscode is trying to access the jupyterhub API which is password protected. Instead, it should access the notebook API with the specified token. For instance, if I copy the following URL into an in-private window of a browser, I can access the single user jupyter server without entering the jupyterhub password:

• https://localhost:4443/jupyter/user/{username}/tree/home/{username}?token={guid}

Your Jupyter and/or Python environment

Please provide as much info as you readily know

• Jupyter server running: Remote
• Extension version: 2020.1.58038
• VS Code version: 1.41.1
• Setting python.jediEnabled: false
• Python and/or Anaconda version: 3.7.3
• OS: Linux (distro): CentOS (SL7.1)
• Virtual environment: conda

Developer Tools Console Output

jupyter.log

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer

[IanMatthewHuff]

Hey @dobos . Thanks for reporting this. In step microsoft/vscode-python#3 you look like you are pointing at the root URL. This doesn't match up with the instructions that I've seen for connecting to a JupyterHub instance provided by the Jupyter team:
https://blog.jupyter.org/connect-to-a-jupyterhub-from-visual-studio-code-ed7ed3a31bcb

Could you try with the user path seen in step 3 on that page?

[dobos]

@IanMatthewHuff Yep, apparently I forgot to report that part of the problem. The behavior is similar when I specify the user root. If I use the url in the form https://localhost:4443/jupyter/user/{username}/?token={guid} vscode sends a similar request and I get this error:

Error: Failed to connect to remote Jupyter notebook.
Check that the Jupyter Server URI setting has a valid running server specified.
https://localhost:4443/jupyter/user/{username}/
TypeError: request to https://localhost:4443/jupyter/user/{username}/api/contents/?1579701006572 failed, reason: connect ECONNREFUSED 127.0.0.1:4443

Whereas if I copy the url (the original one with the guid, not with the number, I don't even know what that is) into the browser it redirects me to the correct user page without asking the password. I use an in-private window to avoid existing auth cookies. Once authenticated the URL in the form https://localhost:4443/jupyter/user/{username}/api/contents/?1579701006572 also works so I'm guessing the problem is that the jupyter extension first should request the original root URL with the token. This is not happening, at least it's not showing in the javascript console. I'll try to go through the process again from scratch and create a fiddler log too.

[IanMatthewHuff]

Thanks, the fiddler log would help here. In general JupyterHub isn't something that we explicitly support yet but the API token usually works as a basic workaround. Also if you run your connection scenario you could go to VSCode->Help->Toggle Developer tools. Then in that window as you run your repro you can save off the log from the Console section and paste it here. That would have our diagnostics spew that might help me to see what's going on.

[dobos]

Any tips on how to direct traffic through fiddler from the python extensions? This is my settings.json

    "http.proxy": "http://localhost:8888",
    "http.proxyStrictSSL": false,
    "http.proxySupport": "override",

I've tried on instead of override too.

I can see the dev tools console output but it doesn't say much about network traffic and no requests show up in the network tab either. I already uploaded a console log with the original comment above. I've tried it by running vscode in local mode on my windows pc and also in remote mode on a linux server, results are the same. I see no entry in the log that a request to the jupyter server is made with the token. Nor I see that the request made to the jupyter API contains the auth cookie that the server should set when a request with the token is made.

[IanMatthewHuff]

One thing to try. Could you swap out the localhost with the actual ip address of the server that you are connecting to? Would just like to rule that out as a possible issue. I'm looking at a successful remote connection log myself right now, and the token used in the connection isn't actually shown in the log at all. The number after api/contents or api/kernels is just the kernel instance that is getting connected to.

Also, would there be any configuration on your machine for something like a firewall / proxy that would block / restrict websocket usage? Some of the cases where we've seen a URL that can be resolved in the browser, but cannot be resolved in our VSCode connection hit issues there since we use jupyter services websockets to connect.

[dobos]

I'm connecting to a remote server via an SSH tunnel, so I can't use the IP address. I can connect to the single user server with the token just fine from a browser running on the very same machine where VS Code is running (using in-private mode, so no cookies are set at start). I'm going to check fiddler again. I could already see the communication over the tunnel, by setting the URL to https://localhost.:4443/... (with a dot after localhost) but it's not decrypting the https traffic for some reason, although all settings seem to be fine.

[rchiodo]

I just submitted a change that should allow somebody to connect to Jupyter Hub. We're looking for feedback as we're not sure what sort of authentication people normally use. The change supports username/password at the moment.

Essentially you just give us the base URI of the jupyterhub server, we'll ping it to see if it is a jupyter hub server and then we query the login page with a username/password you give us. If that works we then use the REST api on the jupyterhub server to get a token and start a notebook server for that user.

It looks something like so:

[rchiodo]

For somebody validating this, please follow the extra steps I added to the test plan (so that's validated too).
https://github.com/microsoft/vscode-python/blob/master/.github/test_plan.md

[IanMatthewHuff]

Might come poke at this some more, or come back to it on my windows machine. But the docker instructions were not working for me on the Mac. Can't validate just yet.

[IanMatthewHuff]

Validated. Nice work @rchiodo.