Python 3.13: an unexpected `Authorization` header because of NETRC file

[yermulnik]

When sending POST request using Python 3.13 with requests v2.32.3 there appear an unexpected Authorization header.
This is not the case with Python 3.10 with the same version of requests lib.

Expected Result

POST request does not get updated with an unexpected Authorization header.

Actual Result

Python 3.13

> ~/tmp/zzz.py 2>&1 | egrep -m 1 "^send: "
send: b'POST /oauth2/v1/device/authorize HTTP/1.1\r\nHost: okta.oktapreview.com\r\nUser-Agent: gimme-aws-creds 2.8.2;linux;3.13.2\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nContent-Length: 52\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic Og==\r\n\r\n'

Python 3.10

> ~/tmp/zzz.py 2>&1 | egrep -m 1 "^send: "
send: b'POST /oauth2/v1/device/authorize HTTP/1.1\r\nHost: okta.oktapreview.com\r\nUser-Agent: gimme-aws-creds 2.8.2;linux;3.13.2\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: application/json\r\nConnection: keep-alive\r\nContent-Length: 52\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'

Reproduction Steps

import requests
import logging

try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

data = {
    "client_id": "<client_id>",
    "scope": "openid okta.apps.sso"
}
url = "https://okta.oktapreview.com/oauth2/v1/device/authorize"
headers = {'User-Agent': 'gimme-aws-creds 2.8.2;linux;3.13.2', 'Accept': 'application/json'}

response = requests.post(url, data=data, headers=headers, verify=True)

print(response.text)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.4.1"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.13.2"
  },
  "platform": {
    "release": "6.8.0-52-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30400010"
  },
  "urllib3": {
    "version": "2.3.0"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Ref: Nike-Inc/gimme-aws-creds#485

[sigmavirus24]

This is strange as Og== is base64 encoded from :.

My best guess is that adding trust_env=False would fix this on Python 3.13. Why it fixes it though, is likely do to something in the netrc standard library.

If you check ~/.netrc do you have any configuration for the particular host you care about?

[sigmavirus24]

Are the versions of requests the same between the two environments as well?

[yermulnik]

My best guess is that adding trust_env=False would fix this on Python 3.13. Why it fixes it though, is likely do to something in the netrc standard library.

If you check ~/.netrc do you have any configuration for the particular host you care about?

Can confirm the ~/.netrc is the issue and the only matching host entry for the host is the default entry:

default
        macdef init
                set prompt "[%n@%M][%/]> "

Seems to have nothing to do with auth though, but removing this section does fix the issue (similarly setting trust_env=False for requests session fixes the issue too).

Since my issue is with another tool that utilizes requests, do you by any chance have a hint of whether trust_env value can be modified (set False) via an external approach like an Env var?

Are the versions of requests the same between the two environments as well?

Yes.

> for i in 3.10 3.13; do echo -n "$i: "; python$i -m requests.help | jq -r '.requests.version'; done
3.10: 2.32.3
3.13: 2.32.3

[sigmavirus24]

It would be ironic to have an environment variable telling the library to not trust the environment.

[yermulnik]

Haha, agree =)) I think I'm just too focused on .netrc part of this.

Looks like my best (and sort of the only) option at the moment is to try and PR to the tool, that I rely upon, a flag to switch trust_env for requests (and accept the fact that this will also break use cases where someone relies upon proxy env vars) 🤔

Am I right assuming that the Authorization header in my conditions (where there are no auth parameters set for the host via .netrc) is added by Python 3.13 rather than by requests lib or is it something that requests can potentially rectify or remediate?

Thanks.

[sigmavirus24]

Can you reproduce it just with the standard library? If not, can you write a failing test for https://github.com/psf/requests/blob/main/src%2Frequests%2Futils.py#L207

[yermulnik]

Can you reproduce it just with the standard library?

Sorry, I can't seem to figure out what you mean 😕 Isn't the «Reproduction Steps» from the issue description what you're asking about? 🤔

#!/usr/bin/env python3.13
import requests

data = {
    "client_id": "0[…]7",
    "scope": "openid okta.apps.sso"
}
url = "https://[…].oktapreview.com/oauth2/v1/device/authorize"
headers = {'Accept': 'application/json'}

response = requests.post(url, data=data, headers=headers, verify=True)

print(response.text)

> ~/tmp/zzz.py
{"errorCode":"invalid_client","errorSummary":"Invalid value for 'client_id' parameter.","errorLink":"invalid_client","errorId":"oaeohowcwaASwevPhC5oeoeHA","errorCauses":[]}

> NETRC="" ~/tmp/zzz.py
{"device_code":"c[…]f","user_code":"L[…]B","verification_uri":"https://[…].oktapreview.com/activate","verification_uri_complete":"https://[…].oktapreview.com/activate?user_code=L[…]B","expires_in":600,"interval":5}

If not, can you write a failing test for main/src%2Frequests%2Futils.py#L207

I could try... though I'm not a Python dev and it can take me a challenge. If you could provide some guidance probably (or point out to an existing test example): I probably need to put a .netrc file somewhere under /tests/ dir (create new dir like files/ and add file to it?); the known to me endpoint that returns failure in this case is my employer's Okta tenant (incl Okta App Client ID), which I don't think is a good endpoint to run test against from within pfs/requests repo; so on 🤔

[yermulnik]

@sigmavirus24 Any hints?
I do understand that my case is pretty much rare and uncommon, though it's probably better to rectify this glitch sooner or later and I'll be happy to assist testing the fix. Thank you.

[fabrice-lebas]

How can we raise interest in this issue? This blocks access to any service accepting bearer token, or using SLT instead of the content of netrc.

[nateprewitt]

It turns out this is actually a change in Python 3.11. The netrc update is in the docs but missed the release notes. For 3.11+, it treats a netrc entry without username, account or password as valid. That means a file can have a valid default profile that we shouldn't be using.

What's happening in Requests is we used to get a NetrcParseError that defaulted the value to None skipping this. Now, that produces ('', '', '') for our _netrc. That tuple is in turn "truthy" and used to create an Authorization header with two empty values (:).

I've made a PR that skips using the netrc entries entirely if we can't find any info. I'd argue what we're doing now is wrong. If someone really wants to include : as their Authorization header value, they can do it through auth=('', ''). Grabbing random entries that aren't intended for use should skipped.

Once that's merged, this should be fixed in the next release. Thanks for the report @yermulnik!