Start Fresh will now attempt to remove Steam Shortcuts

Added functions to clean environment variables and run the NSLGameScanner script. Updated deletion logic for various directories and files.

.claude/settings.json

@@ -0,0 +1,42 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(backlog:*)",
+      "Bash(ruff:*)",
+      "Bash(pytest:*)",
+      "Bash(python:*)",
+      "Bash(pre-commit:*)",
+      "Bash(uv:*)",
+      "mcp__context7__get-library-docs",
+      "mcp__context7__resolve-library-id",
+      "mcp__serena__activate_project",
+      "mcp__serena__check_onboarding_performed",
+      "mcp__serena__find_symbol",
+      "mcp__serena__get_symbols_overview",
+      "mcp__serena__insert_after_symbol",
+      "mcp__serena__insert_before_symbol",
+      "mcp__serena__list_dir",
+      "mcp__serena__onboarding",
+      "mcp__serena__replace_symbol_body",
+      "mcp__serena__search_for_pattern",
+      "mcp__serena__think_about_collected_information",
+      "mcp__serena__think_about_whether_you_are_done",
+      "mcp__serena__write_memory"
+    ],
+    "deny": [ ]
+  }
+  // TODO: instate at a later time
+  // "hooks": {
+  //   "PostToolUse": [
+  //     {
+  //       "matcher": "Edit|MultiEdit|Write",
+  //       "hooks": [
+  //         {
+  //           "type": "command",
+  //           "command": "git diff --name-only HEAD | xargs -r pre-commit run --files"
+  //         }
+  //       ]
+  //     }
+  //   ]
+  // }
+}

.devcontainer/devcontainer.json

@@ -0,0 +1,32 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/python
+{
+  "name": "Dev Environment",
+  "build": {
+    "dockerfile": "../Dockerfile"
+  },
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "aaron-bond.better-comments",
+        "codezombiech.gitignore",
+        "eamodio.gitlens",
+        "EditorConfig.EditorConfig",
+        "GitHub.copilot-chat",
+        "GitHub.copilot",
+        "mads-hartmann.bash-ide-vscode",
+        "ms-azuretools.vscode-docker",
+        "ms-python.python",
+        "ms-vscode.atom-keybindings",
+        "ms-vsliveshare.vsliveshare",
+        "redhat.vscode-yaml",
+        "timonwong.shellcheck",
+        "yzhang.markdown-all-in-one"
+      ]
+    }
+  },
+//   "forwardPorts": [
+//     8080,
+//     8081
+//   ]
+}

.dockerignore

@@ -0,0 +1,13 @@
+!poetry.lock
+!pyproject.toml
+.cache
+.devcontainer
+.git
+.gitignore
+.pytest_cache
+.tool-versions
+.venv
+.vscode
+**/__pycache__
+Dockerfile*
+README.md

.editorconfig

@@ -0,0 +1,34 @@
+# EditorConfig is awesome: https://EditorConfig.org
+
+# top-most EditorConfig file
+root = true
+
+# Unix-style newlines and whitespace cleanup
+[*]
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+# Protect whitespace in markdown files
+[*.md]
+trim_trailing_whitespace = false
+
+# general formatting
+[*.{bash,go,sh,zsh,justfile,Makefile}]
+indent_style = tab
+indent_size = 4
+
+# Set default charset
+[*.{html,xml,js,css,py}]
+charset = utf-8
+
+# python
+[*.py]
+indent_style = space
+indent_size = 4
+
+# webdev et al
+[*.{html,xml,js,css,json,gql,lua,tf,tfvars,yml,yaml}]
+indent_style = space
+indent_size = 2

.github/FUNDING.yml

@@ -0,0 +1,14 @@
+# These are supported funding model platforms
+
+github: moraroy
+patreon: moraroy
+open_collective: # Replace with a single Open Collective username
+ko_fi: moraroy
+buy_me_a_coffee: moraroy
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: moraroy
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+

.github/ISSUE_TEMPLATE/🐞-bug-report--linux---steamos-.md

@@ -0,0 +1,85 @@
+---
+name: "\U0001F41E Bug Report (Linux / SteamOS)"
+about: Report a bug or problem that you have experienced with NonSteamLaunchers
+title: bug
+labels: bug
+assignees: ''
+
+---
+
+## 🐞 Describe the Bug
+A clear and concise description of the issue you’re experiencing with **NonSteamLaunchers (NSL)**.  
+Include any error messages, crashes, or unexpected behavior.  
+
+**Which version are you using?**  
+- [ ] NonSteamLaunchers.desktop (Desktop Version)
+- [ ] Decky Plugin (`NonSteamLaunchersDecky`)
+
+---
+
+## 🔁 To Reproduce
+Steps to reproduce the behavior:
+1. (Optional) Switch to Desktop Mode.
+2. Launch NSL using your selected method above.
+3. Select and install one or more launchers (Epic, EA, GOG, etc.)
+4. Wait for the installation to complete or for any errors to appear.
+5. (If applicable) Return to Gaming Mode and attempt to launch the installed launcher(s).
+6. Observe the issue.
+
+*Example:*  
+> Epic Games Launcher installs successfully but fails to open in Gaming Mode (Proton GE 9-5).
+
+---
+
+## ✅ Expected Behavior
+A clear and concise description of what you expected to happen.  
+Example:  
+> The Epic Games Launcher should open in Gaming Mode and allow login.
+
+---
+
+## 🖼️ Screenshots / Logs
+If applicable, add screenshots or logs to help explain your problem.
+
+**Log locations:**
+- Script/Desktop version:  
+  `/home/deck/Downloads/NonSteamLaunchers-install.log`
+- Decky Plugin version:  
+  `/home/deck/homebrew/logs/NonSteamLaunchers/`
+
+Please attach relevant sections or errors from the logs if possible.
+
+---
+
+## 🧰 System Information
+Please complete the following details:
+
+**Device Type**
+- [ ] Steam Deck  
+- [ ] Desktop PC  
+- [ ] Laptop  
+- [ ] Other: (please specify)
+
+**Operating System**
+- [ ] SteamOS (Arch-based)  
+- [ ] Other Linux Distro (please specify)
+
+**Details**
+- **SteamOS Version:** e.g. 3.6.5  
+- **Kernel Version:** (run `uname -r` in Konsole)  
+- **Proton Version Used:** e.g. Proton GE 9-5 / Proton Experimental  
+- **Launchers Affected:** Epic / GOG / EA / Battle.net / etc.  
+- **NSL Version:** (from GitHub release tag, Decky Plugin version, or script header)  
+- **Install Method:** Script (.sh) / Desktop Launcher (.desktop) / Decky Plugin  
+
+---
+
+## 💬 Additional Context
+Add any other context about the problem here, such as:
+- [ ] Occurs only in Gaming Mode  
+- [ ] Occurs only in Desktop Mode  
+- [ ] Reinstalling NSL did not fix the issue  
+- [ ] Clearing compatibility data did not help  
+- [ ] I am using a custom Proton build  
+
+Provide any temporary fixes or additional details below:

.github/ISSUE_TEMPLATE/💡-feature-request--linux---steamos-.md

@@ -0,0 +1,55 @@
+---
+name: "\U0001F4A1 Feature Request (Linux / SteamOS)"
+about: Whats the big idea?
+title: enhancement
+labels: enhancement
+assignees: ''
+
+---
+
+## 💭 Is your feature request related to a problem?
+A clear and concise description of what the problem is.  
+Example:  
+> I'm always frustrated when NonSteamLaunchers creates duplicates in the scanner!
+
+---
+
+## 🚀 Describe the Solution You’d Like
+A clear and concise description of what you’d like to see added or improved.  
+Example:  
+> Add a booster seat feature so NSL can launch into space.
+
+---
+
+## 🔄 Describe Alternatives You’ve Considered
+List any alternative approaches, workarounds, or existing tools you’ve tried.  
+Example:  
+> I currently use the Decky Plugin version to manage launchers, but it could be better!
+
+---
+
+## 🧩 Scope of Feature
+Please indicate what area of NSL your request affects:
+
+- [ ] Desktop (`NonSteamLaunchers.desktop`)
+- [ ] Decky Plugin (`NonSteamLaunchersDecky`)
+- [ ] Proton / Compatibility Integration
+- [ ] UI / User Experience
+- [ ] Other (please specify below)
+
+---
+
+## 🧰 System Context
+(Optional, but helpful for context)
+- **Device:** Steam Deck / Desktop PC / Laptop / Other  
+- **OS / Distro:** SteamOS (Arch-based) / Other Linux Distro (specify)  
+- **Current NSL Version:** (from GitHub release tag, Decky Plugin version, or script header)  
+
+---
+
+## 💬 Additional Context
+Add any other context, mockups, or screenshots related to your suggestion here.  
+You can also include:
+- Why this feature would improve the Linux/Steam Deck experience  
+- Whether it should be optional or enabled by default  
+- If it relates to an existing issue or pull request

.github/dependabot.yml

@@ -0,0 +1,18 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem
+# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#directory
+
+version: 2
+updates:
+  - package-ecosystem: "pip"      # package manager
+    directory: "/"                # Files stored in repository root
+    schedule:
+      interval: "weekly"
+      day: "saturday"
+      time: "10:00"
+      timezone: "America/Chicago"
+    open-pull-requests-limit: 5
+    versioning-strategy: increase-if-necessary

.github/workflows/release-please.yml

@@ -0,0 +1,29 @@
+name: Release Please
+
+on:
+  push:
+    branches:
+      - 'main'
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  release-please:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Release with release-please
+        uses: googleapis/release-please-action@v4
+        with:
+          token: ${{ secrets.RELEASE_PLEASE_TOKEN }}
+          config-file: release-please-config.json
+          manifest-file: .release-please-manifest.json
+          repo-url: moraroy/NonSteamLaunchers-On-Steam-Deck
+          fork: false
+          include-component-in-tag: false
+          skip-github-release: false
+          skip-github-pull-request: false

.gitignore

@@ -0,0 +1,301 @@
+# ETC
+*.out
+.serena/
+CLAUDE.md
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+.pnpm-debug.log*
+
+# Diagnostic reports (https://nodejs.org/api/report.html)
+report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+*.lcov
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (https://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Snowpack dependency directory (https://snowpack.dev/)
+web_modules/
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional stylelint cache
+.stylelintcache
+
+# Microbundle cache
+.rpt2_cache/
+.rts2_cache_cjs/
+.rts2_cache_es/
+.rts2_cache_umd/
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variable files
+.env
+.env.development.local
+.env.test.local
+.env.production.local
+.env.local
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+.parcel-cache
+
+# Next.js build output
+.next
+out
+
+# Nuxt.js build / generate output
+.nuxt
+dist
+
+# Gatsby files
+.cache/
+# Comment in the public line in if your project uses Gatsby and not Next.js
+# https://nextjs.org/blog/next-9-1#public-directory-support
+# public
+
+# vuepress build output
+.vuepress/dist
+
+# vuepress v2.x temp and cache directory
+.temp
+.cache
+
+# Docusaurus cache and generated files
+.docusaurus
+
+# Serverless directories
+.serverless/
+
+# FuseBox cache
+.fusebox/
+
+# DynamoDB Local files
+.dynamodb/
+
+# TernJS port file
+.tern-port
+
+# Stores VSCode versions used for testing VSCode extensions
+.vscode-test
+
+# yarn v2
+.yarn/cache
+.yarn/unplugged
+.yarn/build-state.yml
+.yarn/install-state.gz
+.pnp.*
+
+# INCLUDE
+!**/.gitkeep
+!**/*.example
+!**/AGENTS.md

.pre-commit-config.yaml

@@ -0,0 +1,43 @@
+fail_fast: true
+
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.2.2
+    hooks:
+      - id: ruff
+        args: [--fix, --exit-non-zero-on-fix]
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+    - id: check-yaml
+    - id: check-added-large-files
+      args: ['--maxkb=1024']
+    - id: check-docstring-first
+      exclude: |
+          (?x)^(
+              hello.py
+          )$
+    - id: check-executables-have-shebangs
+    - id: check-merge-conflict
+    - id: check-shebang-scripts-are-executable
+    - id: check-symlinks
+    - id: check-toml
+    - id: check-yaml
+      args: [--unsafe]
+    - id: debug-statements
+    - id: destroyed-symlinks
+    - id: detect-private-key
+    - id: end-of-file-fixer
+    - id: fix-byte-order-marker
+    - id: mixed-line-ending
+    - id: pretty-format-json
+      # https://pre-commit.com/#regular-expressions
+      exclude: |
+          (?x)^(
+              .devcontainer/devcontainer.json|
+              .vscode/launch.json|
+              .vscode/settings.json|
+              .vscode/extensions.json
+          )$
+      args: ['--autofix', '--indent=2', '--no-sort-keys']
+    - id: requirements-txt-fixer

.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "4.2.3"
+}

.shellcheckrc

@@ -0,0 +1,15 @@
+disable=SC1073
+disable=SC1078
+disable=SC1079
+disable=SC1091
+disable=SC2016
+disable=SC2034
+disable=SC2046
+disable=SC2086
+disable=SC2153
+disable=SC2155
+disable=SC2164
+disable=SC2236
+disable=SC2317
+disable=SC3037
+disable=SC3045

.tool-versions

@@ -0,0 +1,2 @@
+python 3.11.13
+uv     0.8.8

.vscode/extensions.json

@@ -0,0 +1,18 @@
+{
+  "recommendations": [
+    "aaron-bond.better-comments",
+    "codezombiech.gitignore",
+    "eamodio.gitlens",
+    "EditorConfig.EditorConfig",
+    "GitHub.copilot-chat",
+    "GitHub.copilot",
+    "mads-hartmann.bash-ide-vscode",
+    "ms-azuretools.vscode-docker",
+    "ms-python.python",
+    "ms-vscode.atom-keybindings",
+    "ms-vsliveshare.vsliveshare",
+    "redhat.vscode-yaml",
+    "timonwong.shellcheck",
+    "yzhang.markdown-all-in-one"
+  ]
+}

.vscode/launch.json

@@ -0,0 +1,92 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    // pytest: https://stackoverflow.com/questions/70259564/how-to-debug-the-current-python-test-file-with-pytest-in-vs-code
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python: Current File",
+            "type": "python",
+            "request": "launch",
+            "program": "${file}",
+            "console": "integratedTerminal",
+            "cwd": "${fileDirname}",
+            "pythonArgs": [
+                "-Xfrozen_modules=off"
+            ],
+            "env": {
+                "PYDEVD_DISABLE_FILE_VALIDATION": "1"
+            }
+            // "args": ["-i", "response.xml", "-o", "response.csv"],
+            // "args": ["-d", "/Volumes/Data"],
+            // "args": ["-f", "menubar.dmg"],
+            // "args": ["-h"],
+        },
+        {
+            "name": "Python: Django",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/manage.py",
+            "console": "integratedTerminal",
+            "args": [
+                "runserver"
+            ],
+            "django": true,
+            "justMyCode": true
+        },
+        {
+            "name": "Python: Flask",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/.venv/bin/flask",
+            "console": "integratedTerminal",
+            "args": [
+                "run",
+                "--host",
+                "0.0.0.0",
+                "--port",
+                "8000"
+            ],
+            "justMyCode": true
+        },
+        {
+            "name": "Python: Flet",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/.venv/bin/flet",
+            "console": "integratedTerminal",
+            "args": [
+                "run",
+                "main.py",
+                "-d"
+            ],
+            "env": {
+                "PYDEVD_DISABLE_FILE_VALIDATION": "1",
+            },
+            "justMyCode": true
+        },
+        {
+            "name": "Python: Debug Tests",
+            "type": "python",
+            "request": "launch",
+            "module": "pytest",
+            "args": [
+                "${file}"
+            ],
+            "console": "integratedTerminal"
+        },
+        {
+            "name": "Node: Current File",
+            "type": "node",
+            "request": "launch",
+            "program": "${file}",
+            "console": "integratedTerminal",
+            "skipFiles": [
+                "<node_internals>/**"
+            ],
+            // "runtimeExecutable": "${env:HOME}/.n/bin/node"
+        }
+    ],
+    "ansible.python.interpreterPath": "${workspaceFolder}/.venv/bin/python"
+}

.vscode/settings.json

@@ -0,0 +1,14 @@
+{
+  "[python]": {
+    "editor.insertSpaces": true,
+    "editor.tabSize": 4,
+  },
+    "python.testing.pytestArgs": [
+        "."
+    ],
+    // ! only disable analysis (pylance) if ruff is installed
+    // https://github.com/astral-sh/ruff-vscode
+    // "python.analysis.ignore": [
+    //   "*"
+    // ],
+}

AGENTS.md

@@ -0,0 +1,121 @@
+# AGENTS.md
+
+This file provides guidance to LLMs when working with code in this repository.
+
+## Project Overview
+
+NonSteamLaunchers is a tool that installs various game launchers (Epic Games, EA App, GOG Galaxy, etc.) under a single Proton prefix on Steam Deck and Linux systems. It automatically adds these launchers to the Steam Library and can scan for installed games in real-time.
+
+## Development Commands
+
+### Python Environment Setup
+
+```bash
+# Create and activate virtual environment using UV
+uv venv --python ">=3.11,<3.13"
+source .venv/bin/activate
+
+# Install dependencies
+uv pip install -r pyproject.toml --all-extras
+```
+
+### Code Quality
+
+```bash
+# Format Python code
+ruff format .
+
+# Check Python linting
+ruff check .
+
+# Fix linting issues automatically
+ruff check . --fix
+
+# Run pre-commit hooks manually
+pre-commit run --all-files
+```
+
+### Running the Project
+
+```bash
+# Main installer script
+./NonSteamLaunchers.sh
+
+# Game scanner service
+python NSLGameScanner.py
+
+# Install specific launcher via command line
+/bin/bash -c 'curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash -s -- "Epic Games"'
+```
+
+## Architecture
+
+### Core Components
+
+- **NonSteamLaunchers.sh**: Main Bash script that handles launcher installation, Steam integration, and UI
+- **NSLGameScanner.py**: Python service that monitors installed launchers and automatically adds games to Steam library
+- **config.py**: Configuration management for paths and settings
+- **Modules/**: Vendored Python dependencies (steamgrid, requests, vdf, etc.)
+
+### Key Paths
+
+- Compatdata directory: `~/.local/share/Steam/steamapps/compatdata/`
+- NonSteamLaunchers prefix: `~/.local/share/Steam/steamapps/compatdata/NonSteamLaunchers/`
+- Logs: `/home/deck/Downloads/NonSteamLaunchers-install.log`
+- Game saves backup: `/home/deck/NSLGameSaves`
+
+### Integration Points
+
+- **Steam**: Adds shortcuts via Steam's shortcuts.vdf file
+- **Proton**: Uses UMU Launcher and GE-Proton for Windows compatibility
+- **Decky Loader**: Plugin available for Steam Deck Game Mode
+- **Ludusavi**: Pre-configured for game save backups
+
+## Code Style
+
+### Python
+
+- Formatter: Ruff (Black-compatible, 130 char line length)
+- Python version: 3.11-3.12
+- Indentation: 4 spaces
+- Follow PEP 8 with exceptions defined in ruff.toml
+
+### Shell Scripts
+
+- Indentation: Tabs
+- Follow shellcheck rules (see .shellcheckrc)
+- Use bash shebang: `#!/bin/bash`
+
+### Pre-commit Hooks
+
+Pre-commit is configured with:
+
+- Ruff for Python formatting and linting
+- File checks (YAML, JSON, large files, private keys)
+- End-of-file fixing and line ending normalization
+
+## Testing
+
+Currently, there is no formal test suite. When implementing new features:
+
+- Test installation of launchers manually
+- Verify Steam library integration
+- Check game scanner functionality
+- Test on both Desktop and Game Mode
+
+## Command Line Arguments
+
+The main script supports:
+
+- Launcher names: `"Epic Games"`, `"EA App"`, `"GOG Galaxy"`, etc.
+- Uninstall: `"Uninstall Epic Games"`
+- Utilities: `"Start Fresh"`, `"Update Proton-GE"`, `"Stop NSLGameScanner"`
+- SD Card: `"Move to SD Card" "LauncherName"`
+
+## Important Notes
+
+- The script modifies Steam configuration files - handle with care
+- Game scanner runs as a systemd service
+- Community notes feature uses `#nsl` hashtag
+- MicroSD card support requires proper mount points
+- Cloud saves via Ludusavi are backed up to `/home/deck/NSLGameSaves`

CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## [3.8.2](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/compare/v3.8.1...v3.8.2) (2024-08-06)
+
+
+### Bug Fixes
+
+* use correct home folder for user ([76466fd](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/commit/76466fdcc6f5473dc004aec365a4c58a9057eeee))
+
+## [3.8.1](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/compare/v3.8.0...v3.8.1) (2024-02-29)
+
+
+### Documentation
+
+* conventional commits ([6b41710](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/commit/6b4171090dca8695856c1e98330973f729547081))
+* formatting and pre-commit hooks ([b8f72f2](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/commit/b8f72f2dd1a542d08225e9ffbdebd1187262e468))
+* update README.md ([61e19be](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/commit/61e19bee16871aeb46a9f8ee9734ca89f5b0a82c))
+* update README.md ([3f6a7c9](https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/commit/3f6a7c952665d1ec4146bd454a8ef38c1c2fbe46))

DONTREADME.md

@@ -0,0 +1 @@
+Oh wow. It seems like you forced yourself to read this by your own free will. Thats known as a choice. A decision. You must be greatly offended with yourself by now. By the end of this sentence, your consent will be taken away from you. Maybe go read the README file instead?

Dockerfile

@@ -0,0 +1,136 @@
+# syntax=docker/dockerfile:1.6
+
+# full semver just for python base image
+ARG PYTHON_VERSION=3.11.4
+
+FROM python:${PYTHON_VERSION}-slim-bullseye AS builder
+
+# avoid stuck build due to user prompt
+ARG DEBIAN_FRONTEND=noninteractive
+
+# update apt repos and install dependencies
+RUN apt -qq update && apt -qq install \
+    --no-install-recommends -y \
+    curl \
+    gcc \
+    libpq-dev \
+    python3-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# pip env vars
+ENV PIP_NO_CACHE_DIR=off
+ENV PIP_DISABLE_PIP_VERSION_CHECK=on
+ENV PIP_DEFAULT_TIMEOUT=100
+
+# poetry env vars
+ENV POETRY_HOME="/opt/poetry"
+ENV POETRY_VERSION=1.5.1
+ENV POETRY_VIRTUALENVS_IN_PROJECT=true
+ENV POETRY_NO_INTERACTION=1
+
+# path
+ENV VENV="/opt/venv"
+ENV PATH="$POETRY_HOME/bin:$VENV/bin:$PATH"
+
+COPY requirements.txt requirements.txt
+
+RUN python -m venv $VENV \
+    && . "${VENV}/bin/activate"\
+    && python -m pip install "poetry==${POETRY_VERSION}" \
+    && python -m pip install -r requirements.txt
+
+FROM python:${PYTHON_VERSION}-slim-bullseye AS runner
+
+# setup standard non-root user for use downstream
+ENV USER_NAME=appuser
+ENV USER_GROUP=appuser
+ENV HOME="/home/${USER_NAME}"
+ENV HOSTNAME="${HOST:-localhost}"
+ENV VENV="/opt/venv"
+
+ENV PATH="${VENV}/bin:${VENV}/lib/python${PYTHON_VERSION}/site-packages:/usr/local/bin:${HOME}/.local/bin:/bin:/usr/bin:/usr/share/doc:$PATH"
+
+# standardise on locale, don't generate .pyc, enable tracebacks on seg faults
+ENV LANG C.UTF-8
+ENV LC_ALL C.UTF-8
+ENV PYTHONDONTWRITEBYTECODE 1
+ENV PYTHONFAULTHANDLER 1
+
+# workers per core
+# https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker/blob/master/README.md#web_concurrency
+ENV WEB_CONCURRENCY=2
+
+# avoid stuck build due to user prompt
+ARG DEBIAN_FRONTEND=noninteractive
+
+# install dependencies
+RUN apt -qq update && apt -qq install \
+    --no-install-recommends -y \
+    bat \
+    curl \
+    dpkg \
+    git \
+    iputils-ping \
+    lsof \
+    p7zip \
+    perl \
+    shellcheck \
+    tldr \
+    tree \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN groupadd $USER_NAME \
+    && useradd -m $USER_NAME -g $USER_GROUP
+
+# create read/write dirs
+RUN <<EOF
+#!/usr/bin/env bash
+mkdir -p /app/{certs,staticfiles}
+chown -R "${USER_NAME}:${USER_GROUP}" /app/
+EOF
+
+USER $USER_NAME
+WORKDIR $HOME
+
+COPY --from=builder --chown=${USER_NAME}:${USER_GROUP} $VENV $VENV
+
+# qol: tooling
+RUN <<EOF
+#!/usr/bin/env bash
+# gh
+curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
+chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
+echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null
+apt update && apt install gh -y
+apt remove dpkg -y
+rm -rf /var/lib/apt/lists/*
+
+# fzf
+git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
+yes | ~/.fzf/install
+EOF
+
+# qol: .bashrc
+RUN tee -a $HOME/.bashrc <<EOF
+# shared history
+HISTFILE=/var/tmp/.bash_history
+HISTFILESIZE=100
+HISTSIZE=100
+
+stty -ixon
+
+[ -f ~/.fzf.bash ] && . ~/.fzf.bash
+
+# aliases
+alias ..='cd ../'
+alias ...='cd ../../'
+alias ll='ls -la --color=auto'
+EOF
+
+# $PATH
+ENV PATH=$VENV_PATH/bin:$HOME/.local/bin:$PATH
+
+# port needed by app
+EXPOSE 8000
+
+CMD ["sleep", "infinity"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 moraroy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Modules/charset_normalizer/__init__.py

@@ -0,0 +1,46 @@
+# -*- coding: utf-8 -*-
+"""
+Charset-Normalizer
+~~~~~~~~~~~~~~
+The Real First Universal Charset Detector.
+A library that helps you read text from an unknown charset encoding.
+Motivated by chardet, This package is trying to resolve the issue by taking a new approach.
+All IANA character set names for which the Python core library provides codecs are supported.
+
+Basic usage:
+   >>> from charset_normalizer import from_bytes
+   >>> results = from_bytes('Bсеки човек има право на образование. Oбразованието!'.encode('utf_8'))
+   >>> best_guess = results.best()
+   >>> str(best_guess)
+   'Bсеки човек има право на образование. Oбразованието!'
+
+Others methods and usages are available - see the full documentation
+at <https://github.com/Ousret/charset_normalizer>.
+:copyright: (c) 2021 by Ahmed TAHRI
+:license: MIT, see LICENSE for more details.
+"""
+import logging
+
+from .api import from_bytes, from_fp, from_path, is_binary
+from .legacy import detect
+from .models import CharsetMatch, CharsetMatches
+from .utils import set_logging_handler
+from .version import VERSION, __version__
+
+__all__ = (
+    "from_fp",
+    "from_path",
+    "from_bytes",
+    "is_binary",
+    "detect",
+    "CharsetMatch",
+    "CharsetMatches",
+    "__version__",
+    "VERSION",
+    "set_logging_handler",
+)
+
+# Attach a NullHandler to the top level logger by default
+# https://docs.python.org/3.3/howto/logging.html#configuring-logging-for-a-library
+
+logging.getLogger("charset_normalizer").addHandler(logging.NullHandler())

Modules/charset_normalizer/__main__.py

@@ -0,0 +1,4 @@
+from .cli import cli_detect
+
+if __name__ == "__main__":
+    cli_detect()

Modules/charset_normalizer/api.py

@@ -0,0 +1,626 @@
+import logging
+from os import PathLike
+from typing import BinaryIO, List, Optional, Set, Union
+
+from .cd import (
+    coherence_ratio,
+    encoding_languages,
+    mb_encoding_languages,
+    merge_coherence_ratios,
+)
+from .constant import IANA_SUPPORTED, TOO_BIG_SEQUENCE, TOO_SMALL_SEQUENCE, TRACE
+from .md import mess_ratio
+from .models import CharsetMatch, CharsetMatches
+from .utils import (
+    any_specified_encoding,
+    cut_sequence_chunks,
+    iana_name,
+    identify_sig_or_bom,
+    is_cp_similar,
+    is_multi_byte_encoding,
+    should_strip_sig_or_bom,
+)
+
+# Will most likely be controversial
+# logging.addLevelName(TRACE, "TRACE")
+logger = logging.getLogger("charset_normalizer")
+explain_handler = logging.StreamHandler()
+explain_handler.setFormatter(
+    logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
+)
+
+
+def from_bytes(
+    sequences: Union[bytes, bytearray],
+    steps: int = 5,
+    chunk_size: int = 512,
+    threshold: float = 0.2,
+    cp_isolation: Optional[List[str]] = None,
+    cp_exclusion: Optional[List[str]] = None,
+    preemptive_behaviour: bool = True,
+    explain: bool = False,
+    language_threshold: float = 0.1,
+    enable_fallback: bool = True,
+) -> CharsetMatches:
+    """
+    Given a raw bytes sequence, return the best possibles charset usable to render str objects.
+    If there is no results, it is a strong indicator that the source is binary/not text.
+    By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
+    And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
+
+    The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
+    but never take it for granted. Can improve the performance.
+
+    You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
+    purpose.
+
+    This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
+    By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
+    toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
+    Custom logging format and handler can be set manually.
+    """
+
+    if not isinstance(sequences, (bytearray, bytes)):
+        raise TypeError(
+            "Expected object of type bytes or bytearray, got: {0}".format(
+                type(sequences)
+            )
+        )
+
+    if explain:
+        previous_logger_level: int = logger.level
+        logger.addHandler(explain_handler)
+        logger.setLevel(TRACE)
+
+    length: int = len(sequences)
+
+    if length == 0:
+        logger.debug("Encoding detection on empty bytes, assuming utf_8 intention.")
+        if explain:
+            logger.removeHandler(explain_handler)
+            logger.setLevel(previous_logger_level or logging.WARNING)
+        return CharsetMatches([CharsetMatch(sequences, "utf_8", 0.0, False, [], "")])
+
+    if cp_isolation is not None:
+        logger.log(
+            TRACE,
+            "cp_isolation is set. use this flag for debugging purpose. "
+            "limited list of encoding allowed : %s.",
+            ", ".join(cp_isolation),
+        )
+        cp_isolation = [iana_name(cp, False) for cp in cp_isolation]
+    else:
+        cp_isolation = []
+
+    if cp_exclusion is not None:
+        logger.log(
+            TRACE,
+            "cp_exclusion is set. use this flag for debugging purpose. "
+            "limited list of encoding excluded : %s.",
+            ", ".join(cp_exclusion),
+        )
+        cp_exclusion = [iana_name(cp, False) for cp in cp_exclusion]
+    else:
+        cp_exclusion = []
+
+    if length <= (chunk_size * steps):
+        logger.log(
+            TRACE,
+            "override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.",
+            steps,
+            chunk_size,
+            length,
+        )
+        steps = 1
+        chunk_size = length
+
+    if steps > 1 and length / steps < chunk_size:
+        chunk_size = int(length / steps)
+
+    is_too_small_sequence: bool = len(sequences) < TOO_SMALL_SEQUENCE
+    is_too_large_sequence: bool = len(sequences) >= TOO_BIG_SEQUENCE
+
+    if is_too_small_sequence:
+        logger.log(
+            TRACE,
+            "Trying to detect encoding from a tiny portion of ({}) byte(s).".format(
+                length
+            ),
+        )
+    elif is_too_large_sequence:
+        logger.log(
+            TRACE,
+            "Using lazy str decoding because the payload is quite large, ({}) byte(s).".format(
+                length
+            ),
+        )
+
+    prioritized_encodings: List[str] = []
+
+    specified_encoding: Optional[str] = (
+        any_specified_encoding(sequences) if preemptive_behaviour else None
+    )
+
+    if specified_encoding is not None:
+        prioritized_encodings.append(specified_encoding)
+        logger.log(
+            TRACE,
+            "Detected declarative mark in sequence. Priority +1 given for %s.",
+            specified_encoding,
+        )
+
+    tested: Set[str] = set()
+    tested_but_hard_failure: List[str] = []
+    tested_but_soft_failure: List[str] = []
+
+    fallback_ascii: Optional[CharsetMatch] = None
+    fallback_u8: Optional[CharsetMatch] = None
+    fallback_specified: Optional[CharsetMatch] = None
+
+    results: CharsetMatches = CharsetMatches()
+
+    sig_encoding, sig_payload = identify_sig_or_bom(sequences)
+
+    if sig_encoding is not None:
+        prioritized_encodings.append(sig_encoding)
+        logger.log(
+            TRACE,
+            "Detected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.",
+            len(sig_payload),
+            sig_encoding,
+        )
+
+    prioritized_encodings.append("ascii")
+
+    if "utf_8" not in prioritized_encodings:
+        prioritized_encodings.append("utf_8")
+
+    for encoding_iana in prioritized_encodings + IANA_SUPPORTED:
+        if cp_isolation and encoding_iana not in cp_isolation:
+            continue
+
+        if cp_exclusion and encoding_iana in cp_exclusion:
+            continue
+
+        if encoding_iana in tested:
+            continue
+
+        tested.add(encoding_iana)
+
+        decoded_payload: Optional[str] = None
+        bom_or_sig_available: bool = sig_encoding == encoding_iana
+        strip_sig_or_bom: bool = bom_or_sig_available and should_strip_sig_or_bom(
+            encoding_iana
+        )
+
+        if encoding_iana in {"utf_16", "utf_32"} and not bom_or_sig_available:
+            logger.log(
+                TRACE,
+                "Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.",
+                encoding_iana,
+            )
+            continue
+        if encoding_iana in {"utf_7"} and not bom_or_sig_available:
+            logger.log(
+                TRACE,
+                "Encoding %s won't be tested as-is because detection is unreliable without BOM/SIG.",
+                encoding_iana,
+            )
+            continue
+
+        try:
+            is_multi_byte_decoder: bool = is_multi_byte_encoding(encoding_iana)
+        except (ModuleNotFoundError, ImportError):
+            logger.log(
+                TRACE,
+                "Encoding %s does not provide an IncrementalDecoder",
+                encoding_iana,
+            )
+            continue
+
+        try:
+            if is_too_large_sequence and is_multi_byte_decoder is False:
+                str(
+                    sequences[: int(50e4)]
+                    if strip_sig_or_bom is False
+                    else sequences[len(sig_payload) : int(50e4)],
+                    encoding=encoding_iana,
+                )
+            else:
+                decoded_payload = str(
+                    sequences
+                    if strip_sig_or_bom is False
+                    else sequences[len(sig_payload) :],
+                    encoding=encoding_iana,
+                )
+        except (UnicodeDecodeError, LookupError) as e:
+            if not isinstance(e, LookupError):
+                logger.log(
+                    TRACE,
+                    "Code page %s does not fit given bytes sequence at ALL. %s",
+                    encoding_iana,
+                    str(e),
+                )
+            tested_but_hard_failure.append(encoding_iana)
+            continue
+
+        similar_soft_failure_test: bool = False
+
+        for encoding_soft_failed in tested_but_soft_failure:
+            if is_cp_similar(encoding_iana, encoding_soft_failed):
+                similar_soft_failure_test = True
+                break
+
+        if similar_soft_failure_test:
+            logger.log(
+                TRACE,
+                "%s is deemed too similar to code page %s and was consider unsuited already. Continuing!",
+                encoding_iana,
+                encoding_soft_failed,
+            )
+            continue
+
+        r_ = range(
+            0 if not bom_or_sig_available else len(sig_payload),
+            length,
+            int(length / steps),
+        )
+
+        multi_byte_bonus: bool = (
+            is_multi_byte_decoder
+            and decoded_payload is not None
+            and len(decoded_payload) < length
+        )
+
+        if multi_byte_bonus:
+            logger.log(
+                TRACE,
+                "Code page %s is a multi byte encoding table and it appear that at least one character "
+                "was encoded using n-bytes.",
+                encoding_iana,
+            )
+
+        max_chunk_gave_up: int = int(len(r_) / 4)
+
+        max_chunk_gave_up = max(max_chunk_gave_up, 2)
+        early_stop_count: int = 0
+        lazy_str_hard_failure = False
+
+        md_chunks: List[str] = []
+        md_ratios = []
+
+        try:
+            for chunk in cut_sequence_chunks(
+                sequences,
+                encoding_iana,
+                r_,
+                chunk_size,
+                bom_or_sig_available,
+                strip_sig_or_bom,
+                sig_payload,
+                is_multi_byte_decoder,
+                decoded_payload,
+            ):
+                md_chunks.append(chunk)
+
+                md_ratios.append(
+                    mess_ratio(
+                        chunk,
+                        threshold,
+                        explain is True and 1 <= len(cp_isolation) <= 2,
+                    )
+                )
+
+                if md_ratios[-1] >= threshold:
+                    early_stop_count += 1
+
+                if (early_stop_count >= max_chunk_gave_up) or (
+                    bom_or_sig_available and strip_sig_or_bom is False
+                ):
+                    break
+        except (
+            UnicodeDecodeError
+        ) as e:  # Lazy str loading may have missed something there
+            logger.log(
+                TRACE,
+                "LazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %s",
+                encoding_iana,
+                str(e),
+            )
+            early_stop_count = max_chunk_gave_up
+            lazy_str_hard_failure = True
+
+        # We might want to check the sequence again with the whole content
+        # Only if initial MD tests passes
+        if (
+            not lazy_str_hard_failure
+            and is_too_large_sequence
+            and not is_multi_byte_decoder
+        ):
+            try:
+                sequences[int(50e3) :].decode(encoding_iana, errors="strict")
+            except UnicodeDecodeError as e:
+                logger.log(
+                    TRACE,
+                    "LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %s",
+                    encoding_iana,
+                    str(e),
+                )
+                tested_but_hard_failure.append(encoding_iana)
+                continue
+
+        mean_mess_ratio: float = sum(md_ratios) / len(md_ratios) if md_ratios else 0.0
+        if mean_mess_ratio >= threshold or early_stop_count >= max_chunk_gave_up:
+            tested_but_soft_failure.append(encoding_iana)
+            logger.log(
+                TRACE,
+                "%s was excluded because of initial chaos probing. Gave up %i time(s). "
+                "Computed mean chaos is %f %%.",
+                encoding_iana,
+                early_stop_count,
+                round(mean_mess_ratio * 100, ndigits=3),
+            )
+            # Preparing those fallbacks in case we got nothing.
+            if (
+                enable_fallback
+                and encoding_iana in ["ascii", "utf_8", specified_encoding]
+                and not lazy_str_hard_failure
+            ):
+                fallback_entry = CharsetMatch(
+                    sequences, encoding_iana, threshold, False, [], decoded_payload
+                )
+                if encoding_iana == specified_encoding:
+                    fallback_specified = fallback_entry
+                elif encoding_iana == "ascii":
+                    fallback_ascii = fallback_entry
+                else:
+                    fallback_u8 = fallback_entry
+            continue
+
+        logger.log(
+            TRACE,
+            "%s passed initial chaos probing. Mean measured chaos is %f %%",
+            encoding_iana,
+            round(mean_mess_ratio * 100, ndigits=3),
+        )
+
+        if not is_multi_byte_decoder:
+            target_languages: List[str] = encoding_languages(encoding_iana)
+        else:
+            target_languages = mb_encoding_languages(encoding_iana)
+
+        if target_languages:
+            logger.log(
+                TRACE,
+                "{} should target any language(s) of {}".format(
+                    encoding_iana, str(target_languages)
+                ),
+            )
+
+        cd_ratios = []
+
+        # We shall skip the CD when its about ASCII
+        # Most of the time its not relevant to run "language-detection" on it.
+        if encoding_iana != "ascii":
+            for chunk in md_chunks:
+                chunk_languages = coherence_ratio(
+                    chunk,
+                    language_threshold,
+                    ",".join(target_languages) if target_languages else None,
+                )
+
+                cd_ratios.append(chunk_languages)
+
+        cd_ratios_merged = merge_coherence_ratios(cd_ratios)
+
+        if cd_ratios_merged:
+            logger.log(
+                TRACE,
+                "We detected language {} using {}".format(
+                    cd_ratios_merged, encoding_iana
+                ),
+            )
+
+        results.append(
+            CharsetMatch(
+                sequences,
+                encoding_iana,
+                mean_mess_ratio,
+                bom_or_sig_available,
+                cd_ratios_merged,
+                decoded_payload,
+            )
+        )
+
+        if (
+            encoding_iana in [specified_encoding, "ascii", "utf_8"]
+            and mean_mess_ratio < 0.1
+        ):
+            logger.debug(
+                "Encoding detection: %s is most likely the one.", encoding_iana
+            )
+            if explain:
+                logger.removeHandler(explain_handler)
+                logger.setLevel(previous_logger_level)
+            return CharsetMatches([results[encoding_iana]])
+
+        if encoding_iana == sig_encoding:
+            logger.debug(
+                "Encoding detection: %s is most likely the one as we detected a BOM or SIG within "
+                "the beginning of the sequence.",
+                encoding_iana,
+            )
+            if explain:
+                logger.removeHandler(explain_handler)
+                logger.setLevel(previous_logger_level)
+            return CharsetMatches([results[encoding_iana]])
+
+    if len(results) == 0:
+        if fallback_u8 or fallback_ascii or fallback_specified:
+            logger.log(
+                TRACE,
+                "Nothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.",
+            )
+
+        if fallback_specified:
+            logger.debug(
+                "Encoding detection: %s will be used as a fallback match",
+                fallback_specified.encoding,
+            )
+            results.append(fallback_specified)
+        elif (
+            (fallback_u8 and fallback_ascii is None)
+            or (
+                fallback_u8
+                and fallback_ascii
+                and fallback_u8.fingerprint != fallback_ascii.fingerprint
+            )
+            or (fallback_u8 is not None)
+        ):
+            logger.debug("Encoding detection: utf_8 will be used as a fallback match")
+            results.append(fallback_u8)
+        elif fallback_ascii:
+            logger.debug("Encoding detection: ascii will be used as a fallback match")
+            results.append(fallback_ascii)
+
+    if results:
+        logger.debug(
+            "Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.",
+            results.best().encoding,  # type: ignore
+            len(results) - 1,
+        )
+    else:
+        logger.debug("Encoding detection: Unable to determine any suitable charset.")
+
+    if explain:
+        logger.removeHandler(explain_handler)
+        logger.setLevel(previous_logger_level)
+
+    return results
+
+
+def from_fp(
+    fp: BinaryIO,
+    steps: int = 5,
+    chunk_size: int = 512,
+    threshold: float = 0.20,
+    cp_isolation: Optional[List[str]] = None,
+    cp_exclusion: Optional[List[str]] = None,
+    preemptive_behaviour: bool = True,
+    explain: bool = False,
+    language_threshold: float = 0.1,
+    enable_fallback: bool = True,
+) -> CharsetMatches:
+    """
+    Same thing than the function from_bytes but using a file pointer that is already ready.
+    Will not close the file pointer.
+    """
+    return from_bytes(
+        fp.read(),
+        steps,
+        chunk_size,
+        threshold,
+        cp_isolation,
+        cp_exclusion,
+        preemptive_behaviour,
+        explain,
+        language_threshold,
+        enable_fallback,
+    )
+
+
+def from_path(
+    path: Union[str, bytes, PathLike],  # type: ignore[type-arg]
+    steps: int = 5,
+    chunk_size: int = 512,
+    threshold: float = 0.20,
+    cp_isolation: Optional[List[str]] = None,
+    cp_exclusion: Optional[List[str]] = None,
+    preemptive_behaviour: bool = True,
+    explain: bool = False,
+    language_threshold: float = 0.1,
+    enable_fallback: bool = True,
+) -> CharsetMatches:
+    """
+    Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
+    Can raise IOError.
+    """
+    with open(path, "rb") as fp:
+        return from_fp(
+            fp,
+            steps,
+            chunk_size,
+            threshold,
+            cp_isolation,
+            cp_exclusion,
+            preemptive_behaviour,
+            explain,
+            language_threshold,
+            enable_fallback,
+        )
+
+
+def is_binary(
+    fp_or_path_or_payload: Union[PathLike, str, BinaryIO, bytes],  # type: ignore[type-arg]
+    steps: int = 5,
+    chunk_size: int = 512,
+    threshold: float = 0.20,
+    cp_isolation: Optional[List[str]] = None,
+    cp_exclusion: Optional[List[str]] = None,
+    preemptive_behaviour: bool = True,
+    explain: bool = False,
+    language_threshold: float = 0.1,
+    enable_fallback: bool = False,
+) -> bool:
+    """
+    Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
+    Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
+    are disabled to be stricter around ASCII-compatible but unlikely to be a string.
+    """
+    if isinstance(fp_or_path_or_payload, (str, PathLike)):
+        guesses = from_path(
+            fp_or_path_or_payload,
+            steps=steps,
+            chunk_size=chunk_size,
+            threshold=threshold,
+            cp_isolation=cp_isolation,
+            cp_exclusion=cp_exclusion,
+            preemptive_behaviour=preemptive_behaviour,
+            explain=explain,
+            language_threshold=language_threshold,
+            enable_fallback=enable_fallback,
+        )
+    elif isinstance(
+        fp_or_path_or_payload,
+        (
+            bytes,
+            bytearray,
+        ),
+    ):
+        guesses = from_bytes(
+            fp_or_path_or_payload,
+            steps=steps,
+            chunk_size=chunk_size,
+            threshold=threshold,
+            cp_isolation=cp_isolation,
+            cp_exclusion=cp_exclusion,
+            preemptive_behaviour=preemptive_behaviour,
+            explain=explain,
+            language_threshold=language_threshold,
+            enable_fallback=enable_fallback,
+        )
+    else:
+        guesses = from_fp(
+            fp_or_path_or_payload,
+            steps=steps,
+            chunk_size=chunk_size,
+            threshold=threshold,
+            cp_isolation=cp_isolation,
+            cp_exclusion=cp_exclusion,
+            preemptive_behaviour=preemptive_behaviour,
+            explain=explain,
+            language_threshold=language_threshold,
+            enable_fallback=enable_fallback,
+        )
+
+    return not guesses

Modules/charset_normalizer/cd.py

@@ -0,0 +1,395 @@
+import importlib
+from codecs import IncrementalDecoder
+from collections import Counter
+from functools import lru_cache
+from typing import Counter as TypeCounter, Dict, List, Optional, Tuple
+
+from .constant import (
+    FREQUENCIES,
+    KO_NAMES,
+    LANGUAGE_SUPPORTED_COUNT,
+    TOO_SMALL_SEQUENCE,
+    ZH_NAMES,
+)
+from .md import is_suspiciously_successive_range
+from .models import CoherenceMatches
+from .utils import (
+    is_accentuated,
+    is_latin,
+    is_multi_byte_encoding,
+    is_unicode_range_secondary,
+    unicode_range,
+)
+
+
+def encoding_unicode_range(iana_name: str) -> List[str]:
+    """
+    Return associated unicode ranges in a single byte code page.
+    """
+    if is_multi_byte_encoding(iana_name):
+        raise IOError("Function not supported on multi-byte code page")
+
+    decoder = importlib.import_module(
+        "encodings.{}".format(iana_name)
+    ).IncrementalDecoder
+
+    p: IncrementalDecoder = decoder(errors="ignore")
+    seen_ranges: Dict[str, int] = {}
+    character_count: int = 0
+
+    for i in range(0x40, 0xFF):
+        chunk: str = p.decode(bytes([i]))
+
+        if chunk:
+            character_range: Optional[str] = unicode_range(chunk)
+
+            if character_range is None:
+                continue
+
+            if is_unicode_range_secondary(character_range) is False:
+                if character_range not in seen_ranges:
+                    seen_ranges[character_range] = 0
+                seen_ranges[character_range] += 1
+                character_count += 1
+
+    return sorted(
+        [
+            character_range
+            for character_range in seen_ranges
+            if seen_ranges[character_range] / character_count >= 0.15
+        ]
+    )
+
+
+def unicode_range_languages(primary_range: str) -> List[str]:
+    """
+    Return inferred languages used with a unicode range.
+    """
+    languages: List[str] = []
+
+    for language, characters in FREQUENCIES.items():
+        for character in characters:
+            if unicode_range(character) == primary_range:
+                languages.append(language)
+                break
+
+    return languages
+
+
+@lru_cache()
+def encoding_languages(iana_name: str) -> List[str]:
+    """
+    Single-byte encoding language association. Some code page are heavily linked to particular language(s).
+    This function does the correspondence.
+    """
+    unicode_ranges: List[str] = encoding_unicode_range(iana_name)
+    primary_range: Optional[str] = None
+
+    for specified_range in unicode_ranges:
+        if "Latin" not in specified_range:
+            primary_range = specified_range
+            break
+
+    if primary_range is None:
+        return ["Latin Based"]
+
+    return unicode_range_languages(primary_range)
+
+
+@lru_cache()
+def mb_encoding_languages(iana_name: str) -> List[str]:
+    """
+    Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
+    This function does the correspondence.
+    """
+    if (
+        iana_name.startswith("shift_")
+        or iana_name.startswith("iso2022_jp")
+        or iana_name.startswith("euc_j")
+        or iana_name == "cp932"
+    ):
+        return ["Japanese"]
+    if iana_name.startswith("gb") or iana_name in ZH_NAMES:
+        return ["Chinese"]
+    if iana_name.startswith("iso2022_kr") or iana_name in KO_NAMES:
+        return ["Korean"]
+
+    return []
+
+
+@lru_cache(maxsize=LANGUAGE_SUPPORTED_COUNT)
+def get_target_features(language: str) -> Tuple[bool, bool]:
+    """
+    Determine main aspects from a supported language if it contains accents and if is pure Latin.
+    """
+    target_have_accents: bool = False
+    target_pure_latin: bool = True
+
+    for character in FREQUENCIES[language]:
+        if not target_have_accents and is_accentuated(character):
+            target_have_accents = True
+        if target_pure_latin and is_latin(character) is False:
+            target_pure_latin = False
+
+    return target_have_accents, target_pure_latin
+
+
+def alphabet_languages(
+    characters: List[str], ignore_non_latin: bool = False
+) -> List[str]:
+    """
+    Return associated languages associated to given characters.
+    """
+    languages: List[Tuple[str, float]] = []
+
+    source_have_accents = any(is_accentuated(character) for character in characters)
+
+    for language, language_characters in FREQUENCIES.items():
+        target_have_accents, target_pure_latin = get_target_features(language)
+
+        if ignore_non_latin and target_pure_latin is False:
+            continue
+
+        if target_have_accents is False and source_have_accents:
+            continue
+
+        character_count: int = len(language_characters)
+
+        character_match_count: int = len(
+            [c for c in language_characters if c in characters]
+        )
+
+        ratio: float = character_match_count / character_count
+
+        if ratio >= 0.2:
+            languages.append((language, ratio))
+
+    languages = sorted(languages, key=lambda x: x[1], reverse=True)
+
+    return [compatible_language[0] for compatible_language in languages]
+
+
+def characters_popularity_compare(
+    language: str, ordered_characters: List[str]
+) -> float:
+    """
+    Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
+    The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
+    Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
+    """
+    if language not in FREQUENCIES:
+        raise ValueError("{} not available".format(language))
+
+    character_approved_count: int = 0
+    FREQUENCIES_language_set = set(FREQUENCIES[language])
+
+    ordered_characters_count: int = len(ordered_characters)
+    target_language_characters_count: int = len(FREQUENCIES[language])
+
+    large_alphabet: bool = target_language_characters_count > 26
+
+    for character, character_rank in zip(
+        ordered_characters, range(0, ordered_characters_count)
+    ):
+        if character not in FREQUENCIES_language_set:
+            continue
+
+        character_rank_in_language: int = FREQUENCIES[language].index(character)
+        expected_projection_ratio: float = (
+            target_language_characters_count / ordered_characters_count
+        )
+        character_rank_projection: int = int(character_rank * expected_projection_ratio)
+
+        if (
+            large_alphabet is False
+            and abs(character_rank_projection - character_rank_in_language) > 4
+        ):
+            continue
+
+        if (
+            large_alphabet is True
+            and abs(character_rank_projection - character_rank_in_language)
+            < target_language_characters_count / 3
+        ):
+            character_approved_count += 1
+            continue
+
+        characters_before_source: List[str] = FREQUENCIES[language][
+            0:character_rank_in_language
+        ]
+        characters_after_source: List[str] = FREQUENCIES[language][
+            character_rank_in_language:
+        ]
+        characters_before: List[str] = ordered_characters[0:character_rank]
+        characters_after: List[str] = ordered_characters[character_rank:]
+
+        before_match_count: int = len(
+            set(characters_before) & set(characters_before_source)
+        )
+
+        after_match_count: int = len(
+            set(characters_after) & set(characters_after_source)
+        )
+
+        if len(characters_before_source) == 0 and before_match_count <= 4:
+            character_approved_count += 1
+            continue
+
+        if len(characters_after_source) == 0 and after_match_count <= 4:
+            character_approved_count += 1
+            continue
+
+        if (
+            before_match_count / len(characters_before_source) >= 0.4
+            or after_match_count / len(characters_after_source) >= 0.4
+        ):
+            character_approved_count += 1
+            continue
+
+    return character_approved_count / len(ordered_characters)
+
+
+def alpha_unicode_split(decoded_sequence: str) -> List[str]:
+    """
+    Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
+    Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
+    One containing the latin letters and the other hebrew.
+    """
+    layers: Dict[str, str] = {}
+
+    for character in decoded_sequence:
+        if character.isalpha() is False:
+            continue
+
+        character_range: Optional[str] = unicode_range(character)
+
+        if character_range is None:
+            continue
+
+        layer_target_range: Optional[str] = None
+
+        for discovered_range in layers:
+            if (
+                is_suspiciously_successive_range(discovered_range, character_range)
+                is False
+            ):
+                layer_target_range = discovered_range
+                break
+
+        if layer_target_range is None:
+            layer_target_range = character_range
+
+        if layer_target_range not in layers:
+            layers[layer_target_range] = character.lower()
+            continue
+
+        layers[layer_target_range] += character.lower()
+
+    return list(layers.values())
+
+
+def merge_coherence_ratios(results: List[CoherenceMatches]) -> CoherenceMatches:
+    """
+    This function merge results previously given by the function coherence_ratio.
+    The return type is the same as coherence_ratio.
+    """
+    per_language_ratios: Dict[str, List[float]] = {}
+    for result in results:
+        for sub_result in result:
+            language, ratio = sub_result
+            if language not in per_language_ratios:
+                per_language_ratios[language] = [ratio]
+                continue
+            per_language_ratios[language].append(ratio)
+
+    merge = [
+        (
+            language,
+            round(
+                sum(per_language_ratios[language]) / len(per_language_ratios[language]),
+                4,
+            ),
+        )
+        for language in per_language_ratios
+    ]
+
+    return sorted(merge, key=lambda x: x[1], reverse=True)
+
+
+def filter_alt_coherence_matches(results: CoherenceMatches) -> CoherenceMatches:
+    """
+    We shall NOT return "English—" in CoherenceMatches because it is an alternative
+    of "English". This function only keeps the best match and remove the em-dash in it.
+    """
+    index_results: Dict[str, List[float]] = dict()
+
+    for result in results:
+        language, ratio = result
+        no_em_name: str = language.replace("—", "")
+
+        if no_em_name not in index_results:
+            index_results[no_em_name] = []
+
+        index_results[no_em_name].append(ratio)
+
+    if any(len(index_results[e]) > 1 for e in index_results):
+        filtered_results: CoherenceMatches = []
+
+        for language in index_results:
+            filtered_results.append((language, max(index_results[language])))
+
+        return filtered_results
+
+    return results
+
+
+@lru_cache(maxsize=2048)
+def coherence_ratio(
+    decoded_sequence: str, threshold: float = 0.1, lg_inclusion: Optional[str] = None
+) -> CoherenceMatches:
+    """
+    Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
+    A layer = Character extraction by alphabets/ranges.
+    """
+
+    results: List[Tuple[str, float]] = []
+    ignore_non_latin: bool = False
+
+    sufficient_match_count: int = 0
+
+    lg_inclusion_list = lg_inclusion.split(",") if lg_inclusion is not None else []
+    if "Latin Based" in lg_inclusion_list:
+        ignore_non_latin = True
+        lg_inclusion_list.remove("Latin Based")
+
+    for layer in alpha_unicode_split(decoded_sequence):
+        sequence_frequencies: TypeCounter[str] = Counter(layer)
+        most_common = sequence_frequencies.most_common()
+
+        character_count: int = sum(o for c, o in most_common)
+
+        if character_count <= TOO_SMALL_SEQUENCE:
+            continue
+
+        popular_character_ordered: List[str] = [c for c, o in most_common]
+
+        for language in lg_inclusion_list or alphabet_languages(
+            popular_character_ordered, ignore_non_latin
+        ):
+            ratio: float = characters_popularity_compare(
+                language, popular_character_ordered
+            )
+
+            if ratio < threshold:
+                continue
+            elif ratio >= 0.8:
+                sufficient_match_count += 1
+
+            results.append((language, round(ratio, 4)))
+
+            if sufficient_match_count >= 3:
+                break
+
+    return sorted(
+        filter_alt_coherence_matches(results), key=lambda x: x[1], reverse=True
+    )

Modules/charset_normalizer/cli/__init__.py

@@ -0,0 +1,6 @@
+from .__main__ import cli_detect, query_yes_no
+
+__all__ = (
+    "cli_detect",
+    "query_yes_no",
+)

Modules/charset_normalizer/cli/__main__.py

@@ -0,0 +1,296 @@
+import argparse
+import sys
+from json import dumps
+from os.path import abspath, basename, dirname, join, realpath
+from platform import python_version
+from typing import List, Optional
+from unicodedata import unidata_version
+
+import charset_normalizer.md as md_module
+from charset_normalizer import from_fp
+from charset_normalizer.models import CliDetectionResult
+from charset_normalizer.version import __version__
+
+
+def query_yes_no(question: str, default: str = "yes") -> bool:
+    """Ask a yes/no question via input() and return their answer.
+
+    "question" is a string that is presented to the user.
+    "default" is the presumed answer if the user just hits <Enter>.
+        It must be "yes" (the default), "no" or None (meaning
+        an answer is required of the user).
+
+    The "answer" return value is True for "yes" or False for "no".
+
+    Credit goes to (c) https://stackoverflow.com/questions/3041986/apt-command-line-interface-like-yes-no-input
+    """
+    valid = {"yes": True, "y": True, "ye": True, "no": False, "n": False}
+    if default is None:
+        prompt = " [y/n] "
+    elif default == "yes":
+        prompt = " [Y/n] "
+    elif default == "no":
+        prompt = " [y/N] "
+    else:
+        raise ValueError("invalid default answer: '%s'" % default)
+
+    while True:
+        sys.stdout.write(question + prompt)
+        choice = input().lower()
+        if default is not None and choice == "":
+            return valid[default]
+        elif choice in valid:
+            return valid[choice]
+        else:
+            sys.stdout.write("Please respond with 'yes' or 'no' " "(or 'y' or 'n').\n")
+
+
+def cli_detect(argv: Optional[List[str]] = None) -> int:
+    """
+    CLI assistant using ARGV and ArgumentParser
+    :param argv:
+    :return: 0 if everything is fine, anything else equal trouble
+    """
+    parser = argparse.ArgumentParser(
+        description="The Real First Universal Charset Detector. "
+        "Discover originating encoding used on text file. "
+        "Normalize text to unicode."
+    )
+
+    parser.add_argument(
+        "files", type=argparse.FileType("rb"), nargs="+", help="File(s) to be analysed"
+    )
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        default=False,
+        dest="verbose",
+        help="Display complementary information about file if any. "
+        "Stdout will contain logs about the detection process.",
+    )
+    parser.add_argument(
+        "-a",
+        "--with-alternative",
+        action="store_true",
+        default=False,
+        dest="alternatives",
+        help="Output complementary possibilities if any. Top-level JSON WILL be a list.",
+    )
+    parser.add_argument(
+        "-n",
+        "--normalize",
+        action="store_true",
+        default=False,
+        dest="normalize",
+        help="Permit to normalize input file. If not set, program does not write anything.",
+    )
+    parser.add_argument(
+        "-m",
+        "--minimal",
+        action="store_true",
+        default=False,
+        dest="minimal",
+        help="Only output the charset detected to STDOUT. Disabling JSON output.",
+    )
+    parser.add_argument(
+        "-r",
+        "--replace",
+        action="store_true",
+        default=False,
+        dest="replace",
+        help="Replace file when trying to normalize it instead of creating a new one.",
+    )
+    parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        default=False,
+        dest="force",
+        help="Replace file without asking if you are sure, use this flag with caution.",
+    )
+    parser.add_argument(
+        "-t",
+        "--threshold",
+        action="store",
+        default=0.2,
+        type=float,
+        dest="threshold",
+        help="Define a custom maximum amount of chaos allowed in decoded content. 0. <= chaos <= 1.",
+    )
+    parser.add_argument(
+        "--version",
+        action="version",
+        version="Charset-Normalizer {} - Python {} - Unicode {} - SpeedUp {}".format(
+            __version__,
+            python_version(),
+            unidata_version,
+            "OFF" if md_module.__file__.lower().endswith(".py") else "ON",
+        ),
+        help="Show version information and exit.",
+    )
+
+    args = parser.parse_args(argv)
+
+    if args.replace is True and args.normalize is False:
+        print("Use --replace in addition of --normalize only.", file=sys.stderr)
+        return 1
+
+    if args.force is True and args.replace is False:
+        print("Use --force in addition of --replace only.", file=sys.stderr)
+        return 1
+
+    if args.threshold < 0.0 or args.threshold > 1.0:
+        print("--threshold VALUE should be between 0. AND 1.", file=sys.stderr)
+        return 1
+
+    x_ = []
+
+    for my_file in args.files:
+        matches = from_fp(my_file, threshold=args.threshold, explain=args.verbose)
+
+        best_guess = matches.best()
+
+        if best_guess is None:
+            print(
+                'Unable to identify originating encoding for "{}". {}'.format(
+                    my_file.name,
+                    "Maybe try increasing maximum amount of chaos."
+                    if args.threshold < 1.0
+                    else "",
+                ),
+                file=sys.stderr,
+            )
+            x_.append(
+                CliDetectionResult(
+                    abspath(my_file.name),
+                    None,
+                    [],
+                    [],
+                    "Unknown",
+                    [],
+                    False,
+                    1.0,
+                    0.0,
+                    None,
+                    True,
+                )
+            )
+        else:
+            x_.append(
+                CliDetectionResult(
+                    abspath(my_file.name),
+                    best_guess.encoding,
+                    best_guess.encoding_aliases,
+                    [
+                        cp
+                        for cp in best_guess.could_be_from_charset
+                        if cp != best_guess.encoding
+                    ],
+                    best_guess.language,
+                    best_guess.alphabets,
+                    best_guess.bom,
+                    best_guess.percent_chaos,
+                    best_guess.percent_coherence,
+                    None,
+                    True,
+                )
+            )
+
+            if len(matches) > 1 and args.alternatives:
+                for el in matches:
+                    if el != best_guess:
+                        x_.append(
+                            CliDetectionResult(
+                                abspath(my_file.name),
+                                el.encoding,
+                                el.encoding_aliases,
+                                [
+                                    cp
+                                    for cp in el.could_be_from_charset
+                                    if cp != el.encoding
+                                ],
+                                el.language,
+                                el.alphabets,
+                                el.bom,
+                                el.percent_chaos,
+                                el.percent_coherence,
+                                None,
+                                False,
+                            )
+                        )
+
+            if args.normalize is True:
+                if best_guess.encoding.startswith("utf") is True:
+                    print(
+                        '"{}" file does not need to be normalized, as it already came from unicode.'.format(
+                            my_file.name
+                        ),
+                        file=sys.stderr,
+                    )
+                    if my_file.closed is False:
+                        my_file.close()
+                    continue
+
+                dir_path = dirname(realpath(my_file.name))
+                file_name = basename(realpath(my_file.name))
+
+                o_: List[str] = file_name.split(".")
+
+                if args.replace is False:
+                    o_.insert(-1, best_guess.encoding)
+                    if my_file.closed is False:
+                        my_file.close()
+                elif (
+                    args.force is False
+                    and query_yes_no(
+                        'Are you sure to normalize "{}" by replacing it ?'.format(
+                            my_file.name
+                        ),
+                        "no",
+                    )
+                    is False
+                ):
+                    if my_file.closed is False:
+                        my_file.close()
+                    continue
+
+                try:
+                    x_[0].unicode_path = join(dir_path, ".".join(o_))
+
+                    with open(x_[0].unicode_path, "w", encoding="utf-8") as fp:
+                        fp.write(str(best_guess))
+                except IOError as e:
+                    print(str(e), file=sys.stderr)
+                    if my_file.closed is False:
+                        my_file.close()
+                    return 2
+
+        if my_file.closed is False:
+            my_file.close()
+
+    if args.minimal is False:
+        print(
+            dumps(
+                [el.__dict__ for el in x_] if len(x_) > 1 else x_[0].__dict__,
+                ensure_ascii=True,
+                indent=4,
+            )
+        )
+    else:
+        for my_file in args.files:
+            print(
+                ", ".join(
+                    [
+                        el.encoding or "undefined"
+                        for el in x_
+                        if el.path == abspath(my_file.name)
+                    ]
+                )
+            )
+
+    return 0
+
+
+if __name__ == "__main__":
+    cli_detect()

Modules/charset_normalizer/legacy.py

@@ -0,0 +1,54 @@
+from typing import Any, Dict, Optional, Union
+from warnings import warn
+
+from .api import from_bytes
+from .constant import CHARDET_CORRESPONDENCE
+
+
+def detect(
+    byte_str: bytes, should_rename_legacy: bool = False, **kwargs: Any
+) -> Dict[str, Optional[Union[str, float]]]:
+    """
+    chardet legacy method
+    Detect the encoding of the given byte string. It should be mostly backward-compatible.
+    Encoding name will match Chardet own writing whenever possible. (Not on encoding name unsupported by it)
+    This function is deprecated and should be used to migrate your project easily, consult the documentation for
+    further information. Not planned for removal.
+
+    :param byte_str:     The byte sequence to examine.
+    :param should_rename_legacy:  Should we rename legacy encodings
+                                  to their more modern equivalents?
+    """
+    if len(kwargs):
+        warn(
+            f"charset-normalizer disregard arguments '{','.join(list(kwargs.keys()))}' in legacy function detect()"
+        )
+
+    if not isinstance(byte_str, (bytearray, bytes)):
+        raise TypeError(  # pragma: nocover
+            "Expected object of type bytes or bytearray, got: "
+            "{0}".format(type(byte_str))
+        )
+
+    if isinstance(byte_str, bytearray):
+        byte_str = bytes(byte_str)
+
+    r = from_bytes(byte_str).best()
+
+    encoding = r.encoding if r is not None else None
+    language = r.language if r is not None and r.language != "Unknown" else ""
+    confidence = 1.0 - r.chaos if r is not None else None
+
+    # Note: CharsetNormalizer does not return 'UTF-8-SIG' as the sig get stripped in the detection/normalization process
+    # but chardet does return 'utf-8-sig' and it is a valid codec name.
+    if r is not None and encoding == "utf_8" and r.bom:
+        encoding += "_sig"
+
+    if should_rename_legacy is False and encoding in CHARDET_CORRESPONDENCE:
+        encoding = CHARDET_CORRESPONDENCE[encoding]
+
+    return {
+        "encoding": encoding,
+        "language": language,
+        "confidence": confidence,
+    }

Modules/charset_normalizer/md.py

@@ -0,0 +1,615 @@
+from functools import lru_cache
+from logging import getLogger
+from typing import List, Optional
+
+from .constant import (
+    COMMON_SAFE_ASCII_CHARACTERS,
+    TRACE,
+    UNICODE_SECONDARY_RANGE_KEYWORD,
+)
+from .utils import (
+    is_accentuated,
+    is_arabic,
+    is_arabic_isolated_form,
+    is_case_variable,
+    is_cjk,
+    is_emoticon,
+    is_hangul,
+    is_hiragana,
+    is_katakana,
+    is_latin,
+    is_punctuation,
+    is_separator,
+    is_symbol,
+    is_thai,
+    is_unprintable,
+    remove_accent,
+    unicode_range,
+)
+
+
+class MessDetectorPlugin:
+    """
+    Base abstract class used for mess detection plugins.
+    All detectors MUST extend and implement given methods.
+    """
+
+    def eligible(self, character: str) -> bool:
+        """
+        Determine if given character should be fed in.
+        """
+        raise NotImplementedError  # pragma: nocover
+
+    def feed(self, character: str) -> None:
+        """
+        The main routine to be executed upon character.
+        Insert the logic in witch the text would be considered chaotic.
+        """
+        raise NotImplementedError  # pragma: nocover
+
+    def reset(self) -> None:  # pragma: no cover
+        """
+        Permit to reset the plugin to the initial state.
+        """
+        raise NotImplementedError
+
+    @property
+    def ratio(self) -> float:
+        """
+        Compute the chaos ratio based on what your feed() has seen.
+        Must NOT be lower than 0.; No restriction gt 0.
+        """
+        raise NotImplementedError  # pragma: nocover
+
+
+class TooManySymbolOrPunctuationPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._punctuation_count: int = 0
+        self._symbol_count: int = 0
+        self._character_count: int = 0
+
+        self._last_printable_char: Optional[str] = None
+        self._frenzy_symbol_in_word: bool = False
+
+    def eligible(self, character: str) -> bool:
+        return character.isprintable()
+
+    def feed(self, character: str) -> None:
+        self._character_count += 1
+
+        if (
+            character != self._last_printable_char
+            and character not in COMMON_SAFE_ASCII_CHARACTERS
+        ):
+            if is_punctuation(character):
+                self._punctuation_count += 1
+            elif (
+                character.isdigit() is False
+                and is_symbol(character)
+                and is_emoticon(character) is False
+            ):
+                self._symbol_count += 2
+
+        self._last_printable_char = character
+
+    def reset(self) -> None:  # pragma: no cover
+        self._punctuation_count = 0
+        self._character_count = 0
+        self._symbol_count = 0
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count == 0:
+            return 0.0
+
+        ratio_of_punctuation: float = (
+            self._punctuation_count + self._symbol_count
+        ) / self._character_count
+
+        return ratio_of_punctuation if ratio_of_punctuation >= 0.3 else 0.0
+
+
+class TooManyAccentuatedPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._character_count: int = 0
+        self._accentuated_count: int = 0
+
+    def eligible(self, character: str) -> bool:
+        return character.isalpha()
+
+    def feed(self, character: str) -> None:
+        self._character_count += 1
+
+        if is_accentuated(character):
+            self._accentuated_count += 1
+
+    def reset(self) -> None:  # pragma: no cover
+        self._character_count = 0
+        self._accentuated_count = 0
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count < 8:
+            return 0.0
+
+        ratio_of_accentuation: float = self._accentuated_count / self._character_count
+        return ratio_of_accentuation if ratio_of_accentuation >= 0.35 else 0.0
+
+
+class UnprintablePlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._unprintable_count: int = 0
+        self._character_count: int = 0
+
+    def eligible(self, character: str) -> bool:
+        return True
+
+    def feed(self, character: str) -> None:
+        if is_unprintable(character):
+            self._unprintable_count += 1
+        self._character_count += 1
+
+    def reset(self) -> None:  # pragma: no cover
+        self._unprintable_count = 0
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count == 0:
+            return 0.0
+
+        return (self._unprintable_count * 8) / self._character_count
+
+
+class SuspiciousDuplicateAccentPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._successive_count: int = 0
+        self._character_count: int = 0
+
+        self._last_latin_character: Optional[str] = None
+
+    def eligible(self, character: str) -> bool:
+        return character.isalpha() and is_latin(character)
+
+    def feed(self, character: str) -> None:
+        self._character_count += 1
+        if (
+            self._last_latin_character is not None
+            and is_accentuated(character)
+            and is_accentuated(self._last_latin_character)
+        ):
+            if character.isupper() and self._last_latin_character.isupper():
+                self._successive_count += 1
+            # Worse if its the same char duplicated with different accent.
+            if remove_accent(character) == remove_accent(self._last_latin_character):
+                self._successive_count += 1
+        self._last_latin_character = character
+
+    def reset(self) -> None:  # pragma: no cover
+        self._successive_count = 0
+        self._character_count = 0
+        self._last_latin_character = None
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count == 0:
+            return 0.0
+
+        return (self._successive_count * 2) / self._character_count
+
+
+class SuspiciousRange(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._suspicious_successive_range_count: int = 0
+        self._character_count: int = 0
+        self._last_printable_seen: Optional[str] = None
+
+    def eligible(self, character: str) -> bool:
+        return character.isprintable()
+
+    def feed(self, character: str) -> None:
+        self._character_count += 1
+
+        if (
+            character.isspace()
+            or is_punctuation(character)
+            or character in COMMON_SAFE_ASCII_CHARACTERS
+        ):
+            self._last_printable_seen = None
+            return
+
+        if self._last_printable_seen is None:
+            self._last_printable_seen = character
+            return
+
+        unicode_range_a: Optional[str] = unicode_range(self._last_printable_seen)
+        unicode_range_b: Optional[str] = unicode_range(character)
+
+        if is_suspiciously_successive_range(unicode_range_a, unicode_range_b):
+            self._suspicious_successive_range_count += 1
+
+        self._last_printable_seen = character
+
+    def reset(self) -> None:  # pragma: no cover
+        self._character_count = 0
+        self._suspicious_successive_range_count = 0
+        self._last_printable_seen = None
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count <= 24:
+            return 0.0
+
+        ratio_of_suspicious_range_usage: float = (
+            self._suspicious_successive_range_count * 2
+        ) / self._character_count
+
+        return ratio_of_suspicious_range_usage
+
+
+class SuperWeirdWordPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._word_count: int = 0
+        self._bad_word_count: int = 0
+        self._foreign_long_count: int = 0
+
+        self._is_current_word_bad: bool = False
+        self._foreign_long_watch: bool = False
+
+        self._character_count: int = 0
+        self._bad_character_count: int = 0
+
+        self._buffer: str = ""
+        self._buffer_accent_count: int = 0
+
+    def eligible(self, character: str) -> bool:
+        return True
+
+    def feed(self, character: str) -> None:
+        if character.isalpha():
+            self._buffer += character
+            if is_accentuated(character):
+                self._buffer_accent_count += 1
+            if (
+                self._foreign_long_watch is False
+                and (is_latin(character) is False or is_accentuated(character))
+                and is_cjk(character) is False
+                and is_hangul(character) is False
+                and is_katakana(character) is False
+                and is_hiragana(character) is False
+                and is_thai(character) is False
+            ):
+                self._foreign_long_watch = True
+            return
+        if not self._buffer:
+            return
+        if (
+            character.isspace() or is_punctuation(character) or is_separator(character)
+        ) and self._buffer:
+            self._word_count += 1
+            buffer_length: int = len(self._buffer)
+
+            self._character_count += buffer_length
+
+            if buffer_length >= 4:
+                if self._buffer_accent_count / buffer_length > 0.34:
+                    self._is_current_word_bad = True
+                # Word/Buffer ending with an upper case accentuated letter are so rare,
+                # that we will consider them all as suspicious. Same weight as foreign_long suspicious.
+                if (
+                    is_accentuated(self._buffer[-1])
+                    and self._buffer[-1].isupper()
+                    and all(_.isupper() for _ in self._buffer) is False
+                ):
+                    self._foreign_long_count += 1
+                    self._is_current_word_bad = True
+            if buffer_length >= 24 and self._foreign_long_watch:
+                camel_case_dst = [
+                    i
+                    for c, i in zip(self._buffer, range(0, buffer_length))
+                    if c.isupper()
+                ]
+                probable_camel_cased: bool = False
+
+                if camel_case_dst and (len(camel_case_dst) / buffer_length <= 0.3):
+                    probable_camel_cased = True
+
+                if not probable_camel_cased:
+                    self._foreign_long_count += 1
+                    self._is_current_word_bad = True
+
+            if self._is_current_word_bad:
+                self._bad_word_count += 1
+                self._bad_character_count += len(self._buffer)
+                self._is_current_word_bad = False
+
+            self._foreign_long_watch = False
+            self._buffer = ""
+            self._buffer_accent_count = 0
+        elif (
+            character not in {"<", ">", "-", "=", "~", "|", "_"}
+            and character.isdigit() is False
+            and is_symbol(character)
+        ):
+            self._is_current_word_bad = True
+            self._buffer += character
+
+    def reset(self) -> None:  # pragma: no cover
+        self._buffer = ""
+        self._is_current_word_bad = False
+        self._foreign_long_watch = False
+        self._bad_word_count = 0
+        self._word_count = 0
+        self._character_count = 0
+        self._bad_character_count = 0
+        self._foreign_long_count = 0
+
+    @property
+    def ratio(self) -> float:
+        if self._word_count <= 10 and self._foreign_long_count == 0:
+            return 0.0
+
+        return self._bad_character_count / self._character_count
+
+
+class CjkInvalidStopPlugin(MessDetectorPlugin):
+    """
+    GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and
+    can be easily detected. Searching for the overuse of '丅' and '丄'.
+    """
+
+    def __init__(self) -> None:
+        self._wrong_stop_count: int = 0
+        self._cjk_character_count: int = 0
+
+    def eligible(self, character: str) -> bool:
+        return True
+
+    def feed(self, character: str) -> None:
+        if character in {"丅", "丄"}:
+            self._wrong_stop_count += 1
+            return
+        if is_cjk(character):
+            self._cjk_character_count += 1
+
+    def reset(self) -> None:  # pragma: no cover
+        self._wrong_stop_count = 0
+        self._cjk_character_count = 0
+
+    @property
+    def ratio(self) -> float:
+        if self._cjk_character_count < 16:
+            return 0.0
+        return self._wrong_stop_count / self._cjk_character_count
+
+
+class ArchaicUpperLowerPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._buf: bool = False
+
+        self._character_count_since_last_sep: int = 0
+
+        self._successive_upper_lower_count: int = 0
+        self._successive_upper_lower_count_final: int = 0
+
+        self._character_count: int = 0
+
+        self._last_alpha_seen: Optional[str] = None
+        self._current_ascii_only: bool = True
+
+    def eligible(self, character: str) -> bool:
+        return True
+
+    def feed(self, character: str) -> None:
+        is_concerned = character.isalpha() and is_case_variable(character)
+        chunk_sep = is_concerned is False
+
+        if chunk_sep and self._character_count_since_last_sep > 0:
+            if (
+                self._character_count_since_last_sep <= 64
+                and character.isdigit() is False
+                and self._current_ascii_only is False
+            ):
+                self._successive_upper_lower_count_final += (
+                    self._successive_upper_lower_count
+                )
+
+            self._successive_upper_lower_count = 0
+            self._character_count_since_last_sep = 0
+            self._last_alpha_seen = None
+            self._buf = False
+            self._character_count += 1
+            self._current_ascii_only = True
+
+            return
+
+        if self._current_ascii_only is True and character.isascii() is False:
+            self._current_ascii_only = False
+
+        if self._last_alpha_seen is not None:
+            if (character.isupper() and self._last_alpha_seen.islower()) or (
+                character.islower() and self._last_alpha_seen.isupper()
+            ):
+                if self._buf is True:
+                    self._successive_upper_lower_count += 2
+                    self._buf = False
+                else:
+                    self._buf = True
+            else:
+                self._buf = False
+
+        self._character_count += 1
+        self._character_count_since_last_sep += 1
+        self._last_alpha_seen = character
+
+    def reset(self) -> None:  # pragma: no cover
+        self._character_count = 0
+        self._character_count_since_last_sep = 0
+        self._successive_upper_lower_count = 0
+        self._successive_upper_lower_count_final = 0
+        self._last_alpha_seen = None
+        self._buf = False
+        self._current_ascii_only = True
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count == 0:
+            return 0.0
+
+        return self._successive_upper_lower_count_final / self._character_count
+
+
+class ArabicIsolatedFormPlugin(MessDetectorPlugin):
+    def __init__(self) -> None:
+        self._character_count: int = 0
+        self._isolated_form_count: int = 0
+
+    def reset(self) -> None:  # pragma: no cover
+        self._character_count = 0
+        self._isolated_form_count = 0
+
+    def eligible(self, character: str) -> bool:
+        return is_arabic(character)
+
+    def feed(self, character: str) -> None:
+        self._character_count += 1
+
+        if is_arabic_isolated_form(character):
+            self._isolated_form_count += 1
+
+    @property
+    def ratio(self) -> float:
+        if self._character_count < 8:
+            return 0.0
+
+        isolated_form_usage: float = self._isolated_form_count / self._character_count
+
+        return isolated_form_usage
+
+
+@lru_cache(maxsize=1024)
+def is_suspiciously_successive_range(
+    unicode_range_a: Optional[str], unicode_range_b: Optional[str]
+) -> bool:
+    """
+    Determine if two Unicode range seen next to each other can be considered as suspicious.
+    """
+    if unicode_range_a is None or unicode_range_b is None:
+        return True
+
+    if unicode_range_a == unicode_range_b:
+        return False
+
+    if "Latin" in unicode_range_a and "Latin" in unicode_range_b:
+        return False
+
+    if "Emoticons" in unicode_range_a or "Emoticons" in unicode_range_b:
+        return False
+
+    # Latin characters can be accompanied with a combining diacritical mark
+    # eg. Vietnamese.
+    if ("Latin" in unicode_range_a or "Latin" in unicode_range_b) and (
+        "Combining" in unicode_range_a or "Combining" in unicode_range_b
+    ):
+        return False
+
+    keywords_range_a, keywords_range_b = unicode_range_a.split(
+        " "
+    ), unicode_range_b.split(" ")
+
+    for el in keywords_range_a:
+        if el in UNICODE_SECONDARY_RANGE_KEYWORD:
+            continue
+        if el in keywords_range_b:
+            return False
+
+    # Japanese Exception
+    range_a_jp_chars, range_b_jp_chars = (
+        unicode_range_a
+        in (
+            "Hiragana",
+            "Katakana",
+        ),
+        unicode_range_b in ("Hiragana", "Katakana"),
+    )
+    if (range_a_jp_chars or range_b_jp_chars) and (
+        "CJK" in unicode_range_a or "CJK" in unicode_range_b
+    ):
+        return False
+    if range_a_jp_chars and range_b_jp_chars:
+        return False
+
+    if "Hangul" in unicode_range_a or "Hangul" in unicode_range_b:
+        if "CJK" in unicode_range_a or "CJK" in unicode_range_b:
+            return False
+        if unicode_range_a == "Basic Latin" or unicode_range_b == "Basic Latin":
+            return False
+
+    # Chinese/Japanese use dedicated range for punctuation and/or separators.
+    if ("CJK" in unicode_range_a or "CJK" in unicode_range_b) or (
+        unicode_range_a in ["Katakana", "Hiragana"]
+        and unicode_range_b in ["Katakana", "Hiragana"]
+    ):
+        if "Punctuation" in unicode_range_a or "Punctuation" in unicode_range_b:
+            return False
+        if "Forms" in unicode_range_a or "Forms" in unicode_range_b:
+            return False
+        if unicode_range_a == "Basic Latin" or unicode_range_b == "Basic Latin":
+            return False
+
+    return True
+
+
+@lru_cache(maxsize=2048)
+def mess_ratio(
+    decoded_sequence: str, maximum_threshold: float = 0.2, debug: bool = False
+) -> float:
+    """
+    Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
+    """
+
+    detectors: List[MessDetectorPlugin] = [
+        md_class() for md_class in MessDetectorPlugin.__subclasses__()
+    ]
+
+    length: int = len(decoded_sequence) + 1
+
+    mean_mess_ratio: float = 0.0
+
+    if length < 512:
+        intermediary_mean_mess_ratio_calc: int = 32
+    elif length <= 1024:
+        intermediary_mean_mess_ratio_calc = 64
+    else:
+        intermediary_mean_mess_ratio_calc = 128
+
+    for character, index in zip(decoded_sequence + "\n", range(length)):
+        for detector in detectors:
+            if detector.eligible(character):
+                detector.feed(character)
+
+        if (
+            index > 0 and index % intermediary_mean_mess_ratio_calc == 0
+        ) or index == length - 1:
+            mean_mess_ratio = sum(dt.ratio for dt in detectors)
+
+            if mean_mess_ratio >= maximum_threshold:
+                break
+
+    if debug:
+        logger = getLogger("charset_normalizer")
+
+        logger.log(
+            TRACE,
+            "Mess-detector extended-analysis start. "
+            f"intermediary_mean_mess_ratio_calc={intermediary_mean_mess_ratio_calc} mean_mess_ratio={mean_mess_ratio} "
+            f"maximum_threshold={maximum_threshold}",
+        )
+
+        if len(decoded_sequence) > 16:
+            logger.log(TRACE, f"Starting with: {decoded_sequence[:16]}")
+            logger.log(TRACE, f"Ending with: {decoded_sequence[-16::]}")
+
+        for dt in detectors:  # pragma: nocover
+            logger.log(TRACE, f"{dt.__class__}: {dt.ratio}")
+
+    return round(mean_mess_ratio, 3)

Modules/charset_normalizer/models.py

@@ -0,0 +1,340 @@
+from encodings.aliases import aliases
+from hashlib import sha256
+from json import dumps
+from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
+
+from .constant import TOO_BIG_SEQUENCE
+from .utils import iana_name, is_multi_byte_encoding, unicode_range
+
+
+class CharsetMatch:
+    def __init__(
+        self,
+        payload: bytes,
+        guessed_encoding: str,
+        mean_mess_ratio: float,
+        has_sig_or_bom: bool,
+        languages: "CoherenceMatches",
+        decoded_payload: Optional[str] = None,
+    ):
+        self._payload: bytes = payload
+
+        self._encoding: str = guessed_encoding
+        self._mean_mess_ratio: float = mean_mess_ratio
+        self._languages: CoherenceMatches = languages
+        self._has_sig_or_bom: bool = has_sig_or_bom
+        self._unicode_ranges: Optional[List[str]] = None
+
+        self._leaves: List[CharsetMatch] = []
+        self._mean_coherence_ratio: float = 0.0
+
+        self._output_payload: Optional[bytes] = None
+        self._output_encoding: Optional[str] = None
+
+        self._string: Optional[str] = decoded_payload
+
+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, CharsetMatch):
+            raise TypeError(
+                "__eq__ cannot be invoked on {} and {}.".format(
+                    str(other.__class__), str(self.__class__)
+                )
+            )
+        return self.encoding == other.encoding and self.fingerprint == other.fingerprint
+
+    def __lt__(self, other: object) -> bool:
+        """
+        Implemented to make sorted available upon CharsetMatches items.
+        """
+        if not isinstance(other, CharsetMatch):
+            raise ValueError
+
+        chaos_difference: float = abs(self.chaos - other.chaos)
+        coherence_difference: float = abs(self.coherence - other.coherence)
+
+        # Below 1% difference --> Use Coherence
+        if chaos_difference < 0.01 and coherence_difference > 0.02:
+            return self.coherence > other.coherence
+        elif chaos_difference < 0.01 and coherence_difference <= 0.02:
+            # When having a difficult decision, use the result that decoded as many multi-byte as possible.
+            # preserve RAM usage!
+            if len(self._payload) >= TOO_BIG_SEQUENCE:
+                return self.chaos < other.chaos
+            return self.multi_byte_usage > other.multi_byte_usage
+
+        return self.chaos < other.chaos
+
+    @property
+    def multi_byte_usage(self) -> float:
+        return 1.0 - (len(str(self)) / len(self.raw))
+
+    def __str__(self) -> str:
+        # Lazy Str Loading
+        if self._string is None:
+            self._string = str(self._payload, self._encoding, "strict")
+        return self._string
+
+    def __repr__(self) -> str:
+        return "<CharsetMatch '{}' bytes({})>".format(self.encoding, self.fingerprint)
+
+    def add_submatch(self, other: "CharsetMatch") -> None:
+        if not isinstance(other, CharsetMatch) or other == self:
+            raise ValueError(
+                "Unable to add instance <{}> as a submatch of a CharsetMatch".format(
+                    other.__class__
+                )
+            )
+
+        other._string = None  # Unload RAM usage; dirty trick.
+        self._leaves.append(other)
+
+    @property
+    def encoding(self) -> str:
+        return self._encoding
+
+    @property
+    def encoding_aliases(self) -> List[str]:
+        """
+        Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.
+        """
+        also_known_as: List[str] = []
+        for u, p in aliases.items():
+            if self.encoding == u:
+                also_known_as.append(p)
+            elif self.encoding == p:
+                also_known_as.append(u)
+        return also_known_as
+
+    @property
+    def bom(self) -> bool:
+        return self._has_sig_or_bom
+
+    @property
+    def byte_order_mark(self) -> bool:
+        return self._has_sig_or_bom
+
+    @property
+    def languages(self) -> List[str]:
+        """
+        Return the complete list of possible languages found in decoded sequence.
+        Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.
+        """
+        return [e[0] for e in self._languages]
+
+    @property
+    def language(self) -> str:
+        """
+        Most probable language found in decoded sequence. If none were detected or inferred, the property will return
+        "Unknown".
+        """
+        if not self._languages:
+            # Trying to infer the language based on the given encoding
+            # Its either English or we should not pronounce ourselves in certain cases.
+            if "ascii" in self.could_be_from_charset:
+                return "English"
+
+            # doing it there to avoid circular import
+            from charset_normalizer.cd import encoding_languages, mb_encoding_languages
+
+            languages = (
+                mb_encoding_languages(self.encoding)
+                if is_multi_byte_encoding(self.encoding)
+                else encoding_languages(self.encoding)
+            )
+
+            if len(languages) == 0 or "Latin Based" in languages:
+                return "Unknown"
+
+            return languages[0]
+
+        return self._languages[0][0]
+
+    @property
+    def chaos(self) -> float:
+        return self._mean_mess_ratio
+
+    @property
+    def coherence(self) -> float:
+        if not self._languages:
+            return 0.0
+        return self._languages[0][1]
+
+    @property
+    def percent_chaos(self) -> float:
+        return round(self.chaos * 100, ndigits=3)
+
+    @property
+    def percent_coherence(self) -> float:
+        return round(self.coherence * 100, ndigits=3)
+
+    @property
+    def raw(self) -> bytes:
+        """
+        Original untouched bytes.
+        """
+        return self._payload
+
+    @property
+    def submatch(self) -> List["CharsetMatch"]:
+        return self._leaves
+
+    @property
+    def has_submatch(self) -> bool:
+        return len(self._leaves) > 0
+
+    @property
+    def alphabets(self) -> List[str]:
+        if self._unicode_ranges is not None:
+            return self._unicode_ranges
+        # list detected ranges
+        detected_ranges: List[Optional[str]] = [
+            unicode_range(char) for char in str(self)
+        ]
+        # filter and sort
+        self._unicode_ranges = sorted(list({r for r in detected_ranges if r}))
+        return self._unicode_ranges
+
+    @property
+    def could_be_from_charset(self) -> List[str]:
+        """
+        The complete list of encoding that output the exact SAME str result and therefore could be the originating
+        encoding.
+        This list does include the encoding available in property 'encoding'.
+        """
+        return [self._encoding] + [m.encoding for m in self._leaves]
+
+    def output(self, encoding: str = "utf_8") -> bytes:
+        """
+        Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.
+        Any errors will be simply ignored by the encoder NOT replaced.
+        """
+        if self._output_encoding is None or self._output_encoding != encoding:
+            self._output_encoding = encoding
+            self._output_payload = str(self).encode(encoding, "replace")
+
+        return self._output_payload  # type: ignore
+
+    @property
+    def fingerprint(self) -> str:
+        """
+        Retrieve the unique SHA256 computed using the transformed (re-encoded) payload. Not the original one.
+        """
+        return sha256(self.output()).hexdigest()
+
+
+class CharsetMatches:
+    """
+    Container with every CharsetMatch items ordered by default from most probable to the less one.
+    Act like a list(iterable) but does not implements all related methods.
+    """
+
+    def __init__(self, results: Optional[List[CharsetMatch]] = None):
+        self._results: List[CharsetMatch] = sorted(results) if results else []
+
+    def __iter__(self) -> Iterator[CharsetMatch]:
+        yield from self._results
+
+    def __getitem__(self, item: Union[int, str]) -> CharsetMatch:
+        """
+        Retrieve a single item either by its position or encoding name (alias may be used here).
+        Raise KeyError upon invalid index or encoding not present in results.
+        """
+        if isinstance(item, int):
+            return self._results[item]
+        if isinstance(item, str):
+            item = iana_name(item, False)
+            for result in self._results:
+                if item in result.could_be_from_charset:
+                    return result
+        raise KeyError
+
+    def __len__(self) -> int:
+        return len(self._results)
+
+    def __bool__(self) -> bool:
+        return len(self._results) > 0
+
+    def append(self, item: CharsetMatch) -> None:
+        """
+        Insert a single match. Will be inserted accordingly to preserve sort.
+        Can be inserted as a submatch.
+        """
+        if not isinstance(item, CharsetMatch):
+            raise ValueError(
+                "Cannot append instance '{}' to CharsetMatches".format(
+                    str(item.__class__)
+                )
+            )
+        # We should disable the submatch factoring when the input file is too heavy (conserve RAM usage)
+        if len(item.raw) <= TOO_BIG_SEQUENCE:
+            for match in self._results:
+                if match.fingerprint == item.fingerprint and match.chaos == item.chaos:
+                    match.add_submatch(item)
+                    return
+        self._results.append(item)
+        self._results = sorted(self._results)
+
+    def best(self) -> Optional["CharsetMatch"]:
+        """
+        Simply return the first match. Strict equivalent to matches[0].
+        """
+        if not self._results:
+            return None
+        return self._results[0]
+
+    def first(self) -> Optional["CharsetMatch"]:
+        """
+        Redundant method, call the method best(). Kept for BC reasons.
+        """
+        return self.best()
+
+
+CoherenceMatch = Tuple[str, float]
+CoherenceMatches = List[CoherenceMatch]
+
+
+class CliDetectionResult:
+    def __init__(
+        self,
+        path: str,
+        encoding: Optional[str],
+        encoding_aliases: List[str],
+        alternative_encodings: List[str],
+        language: str,
+        alphabets: List[str],
+        has_sig_or_bom: bool,
+        chaos: float,
+        coherence: float,
+        unicode_path: Optional[str],
+        is_preferred: bool,
+    ):
+        self.path: str = path
+        self.unicode_path: Optional[str] = unicode_path
+        self.encoding: Optional[str] = encoding
+        self.encoding_aliases: List[str] = encoding_aliases
+        self.alternative_encodings: List[str] = alternative_encodings
+        self.language: str = language
+        self.alphabets: List[str] = alphabets
+        self.has_sig_or_bom: bool = has_sig_or_bom
+        self.chaos: float = chaos
+        self.coherence: float = coherence
+        self.is_preferred: bool = is_preferred
+
+    @property
+    def __dict__(self) -> Dict[str, Any]:  # type: ignore
+        return {
+            "path": self.path,
+            "encoding": self.encoding,
+            "encoding_aliases": self.encoding_aliases,
+            "alternative_encodings": self.alternative_encodings,
+            "language": self.language,
+            "alphabets": self.alphabets,
+            "has_sig_or_bom": self.has_sig_or_bom,
+            "chaos": self.chaos,
+            "coherence": self.coherence,
+            "unicode_path": self.unicode_path,
+            "is_preferred": self.is_preferred,
+        }
+
+    def to_json(self) -> str:
+        return dumps(self.__dict__, ensure_ascii=True, indent=4)

Modules/charset_normalizer/utils.py

@@ -0,0 +1,421 @@
+import importlib
+import logging
+import unicodedata
+from codecs import IncrementalDecoder
+from encodings.aliases import aliases
+from functools import lru_cache
+from re import findall
+from typing import Generator, List, Optional, Set, Tuple, Union
+
+from _multibytecodec import MultibyteIncrementalDecoder
+
+from .constant import (
+    ENCODING_MARKS,
+    IANA_SUPPORTED_SIMILAR,
+    RE_POSSIBLE_ENCODING_INDICATION,
+    UNICODE_RANGES_COMBINED,
+    UNICODE_SECONDARY_RANGE_KEYWORD,
+    UTF8_MAXIMAL_ALLOCATION,
+)
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_accentuated(character: str) -> bool:
+    try:
+        description: str = unicodedata.name(character)
+    except ValueError:
+        return False
+    return (
+        "WITH GRAVE" in description
+        or "WITH ACUTE" in description
+        or "WITH CEDILLA" in description
+        or "WITH DIAERESIS" in description
+        or "WITH CIRCUMFLEX" in description
+        or "WITH TILDE" in description
+        or "WITH MACRON" in description
+        or "WITH RING ABOVE" in description
+    )
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def remove_accent(character: str) -> str:
+    decomposed: str = unicodedata.decomposition(character)
+    if not decomposed:
+        return character
+
+    codes: List[str] = decomposed.split(" ")
+
+    return chr(int(codes[0], 16))
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def unicode_range(character: str) -> Optional[str]:
+    """
+    Retrieve the Unicode range official name from a single character.
+    """
+    character_ord: int = ord(character)
+
+    for range_name, ord_range in UNICODE_RANGES_COMBINED.items():
+        if character_ord in ord_range:
+            return range_name
+
+    return None
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_latin(character: str) -> bool:
+    try:
+        description: str = unicodedata.name(character)
+    except ValueError:
+        return False
+    return "LATIN" in description
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_punctuation(character: str) -> bool:
+    character_category: str = unicodedata.category(character)
+
+    if "P" in character_category:
+        return True
+
+    character_range: Optional[str] = unicode_range(character)
+
+    if character_range is None:
+        return False
+
+    return "Punctuation" in character_range
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_symbol(character: str) -> bool:
+    character_category: str = unicodedata.category(character)
+
+    if "S" in character_category or "N" in character_category:
+        return True
+
+    character_range: Optional[str] = unicode_range(character)
+
+    if character_range is None:
+        return False
+
+    return "Forms" in character_range and character_category != "Lo"
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_emoticon(character: str) -> bool:
+    character_range: Optional[str] = unicode_range(character)
+
+    if character_range is None:
+        return False
+
+    return "Emoticons" in character_range or "Pictographs" in character_range
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_separator(character: str) -> bool:
+    if character.isspace() or character in {"｜", "+", "<", ">"}:
+        return True
+
+    character_category: str = unicodedata.category(character)
+
+    return "Z" in character_category or character_category in {"Po", "Pd", "Pc"}
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_case_variable(character: str) -> bool:
+    return character.islower() != character.isupper()
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_cjk(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "CJK" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_hiragana(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "HIRAGANA" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_katakana(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "KATAKANA" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_hangul(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "HANGUL" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_thai(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "THAI" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_arabic(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "ARABIC" in character_name
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_arabic_isolated_form(character: str) -> bool:
+    try:
+        character_name = unicodedata.name(character)
+    except ValueError:
+        return False
+
+    return "ARABIC" in character_name and "ISOLATED FORM" in character_name
+
+
+@lru_cache(maxsize=len(UNICODE_RANGES_COMBINED))
+def is_unicode_range_secondary(range_name: str) -> bool:
+    return any(keyword in range_name for keyword in UNICODE_SECONDARY_RANGE_KEYWORD)
+
+
+@lru_cache(maxsize=UTF8_MAXIMAL_ALLOCATION)
+def is_unprintable(character: str) -> bool:
+    return (
+        character.isspace() is False  # includes \n \t \r \v
+        and character.isprintable() is False
+        and character != "\x1A"  # Why? Its the ASCII substitute character.
+        and character != "\ufeff"  # bug discovered in Python,
+        # Zero Width No-Break Space located in 	Arabic Presentation Forms-B, Unicode 1.1 not acknowledged as space.
+    )
+
+
+def any_specified_encoding(sequence: bytes, search_zone: int = 8192) -> Optional[str]:
+    """
+    Extract using ASCII-only decoder any specified encoding in the first n-bytes.
+    """
+    if not isinstance(sequence, bytes):
+        raise TypeError
+
+    seq_len: int = len(sequence)
+
+    results: List[str] = findall(
+        RE_POSSIBLE_ENCODING_INDICATION,
+        sequence[: min(seq_len, search_zone)].decode("ascii", errors="ignore"),
+    )
+
+    if len(results) == 0:
+        return None
+
+    for specified_encoding in results:
+        specified_encoding = specified_encoding.lower().replace("-", "_")
+
+        encoding_alias: str
+        encoding_iana: str
+
+        for encoding_alias, encoding_iana in aliases.items():
+            if encoding_alias == specified_encoding:
+                return encoding_iana
+            if encoding_iana == specified_encoding:
+                return encoding_iana
+
+    return None
+
+
+@lru_cache(maxsize=128)
+def is_multi_byte_encoding(name: str) -> bool:
+    """
+    Verify is a specific encoding is a multi byte one based on it IANA name
+    """
+    return name in {
+        "utf_8",
+        "utf_8_sig",
+        "utf_16",
+        "utf_16_be",
+        "utf_16_le",
+        "utf_32",
+        "utf_32_le",
+        "utf_32_be",
+        "utf_7",
+    } or issubclass(
+        importlib.import_module("encodings.{}".format(name)).IncrementalDecoder,
+        MultibyteIncrementalDecoder,
+    )
+
+
+def identify_sig_or_bom(sequence: bytes) -> Tuple[Optional[str], bytes]:
+    """
+    Identify and extract SIG/BOM in given sequence.
+    """
+
+    for iana_encoding in ENCODING_MARKS:
+        marks: Union[bytes, List[bytes]] = ENCODING_MARKS[iana_encoding]
+
+        if isinstance(marks, bytes):
+            marks = [marks]
+
+        for mark in marks:
+            if sequence.startswith(mark):
+                return iana_encoding, mark
+
+    return None, b""
+
+
+def should_strip_sig_or_bom(iana_encoding: str) -> bool:
+    return iana_encoding not in {"utf_16", "utf_32"}
+
+
+def iana_name(cp_name: str, strict: bool = True) -> str:
+    cp_name = cp_name.lower().replace("-", "_")
+
+    encoding_alias: str
+    encoding_iana: str
+
+    for encoding_alias, encoding_iana in aliases.items():
+        if cp_name in [encoding_alias, encoding_iana]:
+            return encoding_iana
+
+    if strict:
+        raise ValueError("Unable to retrieve IANA for '{}'".format(cp_name))
+
+    return cp_name
+
+
+def range_scan(decoded_sequence: str) -> List[str]:
+    ranges: Set[str] = set()
+
+    for character in decoded_sequence:
+        character_range: Optional[str] = unicode_range(character)
+
+        if character_range is None:
+            continue
+
+        ranges.add(character_range)
+
+    return list(ranges)
+
+
+def cp_similarity(iana_name_a: str, iana_name_b: str) -> float:
+    if is_multi_byte_encoding(iana_name_a) or is_multi_byte_encoding(iana_name_b):
+        return 0.0
+
+    decoder_a = importlib.import_module(
+        "encodings.{}".format(iana_name_a)
+    ).IncrementalDecoder
+    decoder_b = importlib.import_module(
+        "encodings.{}".format(iana_name_b)
+    ).IncrementalDecoder
+
+    id_a: IncrementalDecoder = decoder_a(errors="ignore")
+    id_b: IncrementalDecoder = decoder_b(errors="ignore")
+
+    character_match_count: int = 0
+
+    for i in range(255):
+        to_be_decoded: bytes = bytes([i])
+        if id_a.decode(to_be_decoded) == id_b.decode(to_be_decoded):
+            character_match_count += 1
+
+    return character_match_count / 254
+
+
+def is_cp_similar(iana_name_a: str, iana_name_b: str) -> bool:
+    """
+    Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
+    the function cp_similarity.
+    """
+    return (
+        iana_name_a in IANA_SUPPORTED_SIMILAR
+        and iana_name_b in IANA_SUPPORTED_SIMILAR[iana_name_a]
+    )
+
+
+def set_logging_handler(
+    name: str = "charset_normalizer",
+    level: int = logging.INFO,
+    format_string: str = "%(asctime)s | %(levelname)s | %(message)s",
+) -> None:
+    logger = logging.getLogger(name)
+    logger.setLevel(level)
+
+    handler = logging.StreamHandler()
+    handler.setFormatter(logging.Formatter(format_string))
+    logger.addHandler(handler)
+
+
+def cut_sequence_chunks(
+    sequences: bytes,
+    encoding_iana: str,
+    offsets: range,
+    chunk_size: int,
+    bom_or_sig_available: bool,
+    strip_sig_or_bom: bool,
+    sig_payload: bytes,
+    is_multi_byte_decoder: bool,
+    decoded_payload: Optional[str] = None,
+) -> Generator[str, None, None]:
+    if decoded_payload and is_multi_byte_decoder is False:
+        for i in offsets:
+            chunk = decoded_payload[i : i + chunk_size]
+            if not chunk:
+                break
+            yield chunk
+    else:
+        for i in offsets:
+            chunk_end = i + chunk_size
+            if chunk_end > len(sequences) + 8:
+                continue
+
+            cut_sequence = sequences[i : i + chunk_size]
+
+            if bom_or_sig_available and strip_sig_or_bom is False:
+                cut_sequence = sig_payload + cut_sequence
+
+            chunk = cut_sequence.decode(
+                encoding_iana,
+                errors="ignore" if is_multi_byte_decoder else "strict",
+            )
+
+            # multi-byte bad cutting detector and adjustment
+            # not the cleanest way to perform that fix but clever enough for now.
+            if is_multi_byte_decoder and i > 0:
+                chunk_partial_size_chk: int = min(chunk_size, 16)
+
+                if (
+                    decoded_payload
+                    and chunk[:chunk_partial_size_chk] not in decoded_payload
+                ):
+                    for j in range(i, i - 4, -1):
+                        cut_sequence = sequences[j:chunk_end]
+
+                        if bom_or_sig_available and strip_sig_or_bom is False:
+                            cut_sequence = sig_payload + cut_sequence
+
+                        chunk = cut_sequence.decode(encoding_iana, errors="ignore")
+
+                        if chunk[:chunk_partial_size_chk] in decoded_payload:
+                            break
+
+            yield chunk

Modules/charset_normalizer/version.py

@@ -0,0 +1,6 @@
+"""
+Expose version
+"""
+
+__version__ = "3.8.2"
+VERSION = __version__.split(".")

Modules/urllib3/__init__.py

@@ -0,0 +1,149 @@
+"""
+Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more
+"""
+
+from __future__ import annotations
+
+# Set default logging handler to avoid "No handler found" warnings.
+import logging
+import typing
+import warnings
+from logging import NullHandler
+
+from . import exceptions
+from ._base_connection import _TYPE_BODY
+from ._collections import HTTPHeaderDict
+from ._version import __version__
+from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
+from .filepost import _TYPE_FIELDS, encode_multipart_formdata
+from .poolmanager import PoolManager, ProxyManager, proxy_from_url
+from .response import BaseHTTPResponse, HTTPResponse
+from .util.request import make_headers
+from .util.retry import Retry
+from .util.timeout import Timeout
+
+# Ensure that Python is compiled with OpenSSL 1.1.1+
+# If the 'ssl' module isn't available at all that's
+# fine, we only care if the module is available.
+try:
+    import ssl
+except ImportError:
+    pass
+else:
+    if not ssl.OPENSSL_VERSION.startswith("OpenSSL "):  # Defensive:
+        warnings.warn(
+            "urllib3 v2 only supports OpenSSL 1.1.1+, currently "
+            f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
+            "See: https://github.com/urllib3/urllib3/issues/3020",
+            exceptions.NotOpenSSLWarning,
+        )
+    elif ssl.OPENSSL_VERSION_INFO < (1, 1, 1):  # Defensive:
+        raise ImportError(
+            "urllib3 v2 only supports OpenSSL 1.1.1+, currently "
+            f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
+            "See: https://github.com/urllib3/urllib3/issues/2168"
+        )
+
+__author__ = "Andrey Petrov (andrey.petrov@shazow.net)"
+__license__ = "MIT"
+__version__ = __version__
+
+__all__ = (
+    "HTTPConnectionPool",
+    "HTTPHeaderDict",
+    "HTTPSConnectionPool",
+    "PoolManager",
+    "ProxyManager",
+    "HTTPResponse",
+    "Retry",
+    "Timeout",
+    "add_stderr_logger",
+    "connection_from_url",
+    "disable_warnings",
+    "encode_multipart_formdata",
+    "make_headers",
+    "proxy_from_url",
+    "request",
+    "BaseHTTPResponse",
+)
+
+logging.getLogger(__name__).addHandler(NullHandler())
+
+
+def add_stderr_logger(
+    level: int = logging.DEBUG,
+) -> logging.StreamHandler[typing.TextIO]:
+    """
+    Helper for quickly adding a StreamHandler to the logger. Useful for
+    debugging.
+
+    Returns the handler after adding it.
+    """
+    # This method needs to be in this __init__.py to get the __name__ correct
+    # even if urllib3 is vendored within another package.
+    logger = logging.getLogger(__name__)
+    handler = logging.StreamHandler()
+    handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
+    logger.addHandler(handler)
+    logger.setLevel(level)
+    logger.debug("Added a stderr logging handler to logger: %s", __name__)
+    return handler
+
+
+# ... Clean up.
+del NullHandler
+
+
+# All warning filters *must* be appended unless you're really certain that they
+# shouldn't be: otherwise, it's very hard for users to use most Python
+# mechanisms to silence them.
+# SecurityWarning's always go off by default.
+warnings.simplefilter("always", exceptions.SecurityWarning, append=True)
+# InsecurePlatformWarning's don't vary between requests, so we keep it default.
+warnings.simplefilter("default", exceptions.InsecurePlatformWarning, append=True)
+
+
+def disable_warnings(category: type[Warning] = exceptions.HTTPWarning) -> None:
+    """
+    Helper for quickly disabling all urllib3 warnings.
+    """
+    warnings.simplefilter("ignore", category)
+
+
+_DEFAULT_POOL = PoolManager()
+
+
+def request(
+    method: str,
+    url: str,
+    *,
+    body: _TYPE_BODY | None = None,
+    fields: _TYPE_FIELDS | None = None,
+    headers: typing.Mapping[str, str] | None = None,
+    preload_content: bool | None = True,
+    decode_content: bool | None = True,
+    redirect: bool | None = True,
+    retries: Retry | bool | int | None = None,
+    timeout: Timeout | float | int | None = 3,
+    json: typing.Any | None = None,
+) -> BaseHTTPResponse:
+    """
+    A convenience, top-level request method. It uses a module-global ``PoolManager`` instance.
+    Therefore, its side effects could be shared across dependencies relying on it.
+    To avoid side effects create a new ``PoolManager`` instance and use it instead.
+    The method does not accept low-level ``**urlopen_kw`` keyword arguments.
+    """
+
+    return _DEFAULT_POOL.request(
+        method,
+        url,
+        body=body,
+        fields=fields,
+        headers=headers,
+        preload_content=preload_content,
+        decode_content=decode_content,
+        redirect=redirect,
+        retries=retries,
+        timeout=timeout,
+        json=json,
+    )

Modules/urllib3/_base_connection.py

@@ -0,0 +1,172 @@
+from __future__ import annotations
+
+import typing
+
+from .util.connection import _TYPE_SOCKET_OPTIONS
+from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
+from .util.url import Url
+
+_TYPE_BODY = typing.Union[bytes, typing.IO[typing.Any], typing.Iterable[bytes], str]
+
+
+class ProxyConfig(typing.NamedTuple):
+    ssl_context: ssl.SSLContext | None
+    use_forwarding_for_https: bool
+    assert_hostname: None | str | Literal[False]
+    assert_fingerprint: str | None
+
+
+class _ResponseOptions(typing.NamedTuple):
+    # TODO: Remove this in favor of a better
+    # HTTP request/response lifecycle tracking.
+    request_method: str
+    request_url: str
+    preload_content: bool
+    decode_content: bool
+    enforce_content_length: bool
+
+
+if typing.TYPE_CHECKING:
+    import ssl
+    from typing import Literal, Protocol
+
+    from .response import BaseHTTPResponse
+
+    class BaseHTTPConnection(Protocol):
+        default_port: typing.ClassVar[int]
+        default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
+
+        host: str
+        port: int
+        timeout: None | (
+            float
+        )  # Instance doesn't store _DEFAULT_TIMEOUT, must be resolved.
+        blocksize: int
+        source_address: tuple[str, int] | None
+        socket_options: _TYPE_SOCKET_OPTIONS | None
+
+        proxy: Url | None
+        proxy_config: ProxyConfig | None
+
+        is_verified: bool
+        proxy_is_verified: bool | None
+
+        def __init__(
+            self,
+            host: str,
+            port: int | None = None,
+            *,
+            timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+            source_address: tuple[str, int] | None = None,
+            blocksize: int = 8192,
+            socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
+            proxy: Url | None = None,
+            proxy_config: ProxyConfig | None = None,
+        ) -> None:
+            ...
+
+        def set_tunnel(
+            self,
+            host: str,
+            port: int | None = None,
+            headers: typing.Mapping[str, str] | None = None,
+            scheme: str = "http",
+        ) -> None:
+            ...
+
+        def connect(self) -> None:
+            ...
+
+        def request(
+            self,
+            method: str,
+            url: str,
+            body: _TYPE_BODY | None = None,
+            headers: typing.Mapping[str, str] | None = None,
+            # We know *at least* botocore is depending on the order of the
+            # first 3 parameters so to be safe we only mark the later ones
+            # as keyword-only to ensure we have space to extend.
+            *,
+            chunked: bool = False,
+            preload_content: bool = True,
+            decode_content: bool = True,
+            enforce_content_length: bool = True,
+        ) -> None:
+            ...
+
+        def getresponse(self) -> BaseHTTPResponse:
+            ...
+
+        def close(self) -> None:
+            ...
+
+        @property
+        def is_closed(self) -> bool:
+            """Whether the connection either is brand new or has been previously closed.
+            If this property is True then both ``is_connected`` and ``has_connected_to_proxy``
+            properties must be False.
+            """
+
+        @property
+        def is_connected(self) -> bool:
+            """Whether the connection is actively connected to any origin (proxy or target)"""
+
+        @property
+        def has_connected_to_proxy(self) -> bool:
+            """Whether the connection has successfully connected to its proxy.
+            This returns False if no proxy is in use. Used to determine whether
+            errors are coming from the proxy layer or from tunnelling to the target origin.
+            """
+
+    class BaseHTTPSConnection(BaseHTTPConnection, Protocol):
+        default_port: typing.ClassVar[int]
+        default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
+
+        # Certificate verification methods
+        cert_reqs: int | str | None
+        assert_hostname: None | str | Literal[False]
+        assert_fingerprint: str | None
+        ssl_context: ssl.SSLContext | None
+
+        # Trusted CAs
+        ca_certs: str | None
+        ca_cert_dir: str | None
+        ca_cert_data: None | str | bytes
+
+        # TLS version
+        ssl_minimum_version: int | None
+        ssl_maximum_version: int | None
+        ssl_version: int | str | None  # Deprecated
+
+        # Client certificates
+        cert_file: str | None
+        key_file: str | None
+        key_password: str | None
+
+        def __init__(
+            self,
+            host: str,
+            port: int | None = None,
+            *,
+            timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+            source_address: tuple[str, int] | None = None,
+            blocksize: int = 16384,
+            socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
+            proxy: Url | None = None,
+            proxy_config: ProxyConfig | None = None,
+            cert_reqs: int | str | None = None,
+            assert_hostname: None | str | Literal[False] = None,
+            assert_fingerprint: str | None = None,
+            server_hostname: str | None = None,
+            ssl_context: ssl.SSLContext | None = None,
+            ca_certs: str | None = None,
+            ca_cert_dir: str | None = None,
+            ca_cert_data: None | str | bytes = None,
+            ssl_minimum_version: int | None = None,
+            ssl_maximum_version: int | None = None,
+            ssl_version: int | str | None = None,  # Deprecated
+            cert_file: str | None = None,
+            key_file: str | None = None,
+            key_password: str | None = None,
+        ) -> None:
+            ...

Modules/urllib3/_collections.py

@@ -0,0 +1,483 @@
+from __future__ import annotations
+
+import typing
+from collections import OrderedDict
+from enum import Enum, auto
+from threading import RLock
+
+if typing.TYPE_CHECKING:
+    # We can only import Protocol if TYPE_CHECKING because it's a development
+    # dependency, and is not available at runtime.
+    from typing import Protocol
+
+    from typing_extensions import Self
+
+    class HasGettableStringKeys(Protocol):
+        def keys(self) -> typing.Iterator[str]:
+            ...
+
+        def __getitem__(self, key: str) -> str:
+            ...
+
+
+__all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"]
+
+
+# Key type
+_KT = typing.TypeVar("_KT")
+# Value type
+_VT = typing.TypeVar("_VT")
+# Default type
+_DT = typing.TypeVar("_DT")
+
+ValidHTTPHeaderSource = typing.Union[
+    "HTTPHeaderDict",
+    typing.Mapping[str, str],
+    typing.Iterable[typing.Tuple[str, str]],
+    "HasGettableStringKeys",
+]
+
+
+class _Sentinel(Enum):
+    not_passed = auto()
+
+
+def ensure_can_construct_http_header_dict(
+    potential: object,
+) -> ValidHTTPHeaderSource | None:
+    if isinstance(potential, HTTPHeaderDict):
+        return potential
+    elif isinstance(potential, typing.Mapping):
+        # Full runtime checking of the contents of a Mapping is expensive, so for the
+        # purposes of typechecking, we assume that any Mapping is the right shape.
+        return typing.cast(typing.Mapping[str, str], potential)
+    elif isinstance(potential, typing.Iterable):
+        # Similarly to Mapping, full runtime checking of the contents of an Iterable is
+        # expensive, so for the purposes of typechecking, we assume that any Iterable
+        # is the right shape.
+        return typing.cast(typing.Iterable[typing.Tuple[str, str]], potential)
+    elif hasattr(potential, "keys") and hasattr(potential, "__getitem__"):
+        return typing.cast("HasGettableStringKeys", potential)
+    else:
+        return None
+
+
+class RecentlyUsedContainer(typing.Generic[_KT, _VT], typing.MutableMapping[_KT, _VT]):
+    """
+    Provides a thread-safe dict-like container which maintains up to
+    ``maxsize`` keys while throwing away the least-recently-used keys beyond
+    ``maxsize``.
+
+    :param maxsize:
+        Maximum number of recent elements to retain.
+
+    :param dispose_func:
+        Every time an item is evicted from the container,
+        ``dispose_func(value)`` is called.  Callback which will get called
+    """
+
+    _container: typing.OrderedDict[_KT, _VT]
+    _maxsize: int
+    dispose_func: typing.Callable[[_VT], None] | None
+    lock: RLock
+
+    def __init__(
+        self,
+        maxsize: int = 10,
+        dispose_func: typing.Callable[[_VT], None] | None = None,
+    ) -> None:
+        super().__init__()
+        self._maxsize = maxsize
+        self.dispose_func = dispose_func
+        self._container = OrderedDict()
+        self.lock = RLock()
+
+    def __getitem__(self, key: _KT) -> _VT:
+        # Re-insert the item, moving it to the end of the eviction line.
+        with self.lock:
+            item = self._container.pop(key)
+            self._container[key] = item
+            return item
+
+    def __setitem__(self, key: _KT, value: _VT) -> None:
+        evicted_item = None
+        with self.lock:
+            # Possibly evict the existing value of 'key'
+            try:
+                # If the key exists, we'll overwrite it, which won't change the
+                # size of the pool. Because accessing a key should move it to
+                # the end of the eviction line, we pop it out first.
+                evicted_item = key, self._container.pop(key)
+                self._container[key] = value
+            except KeyError:
+                # When the key does not exist, we insert the value first so that
+                # evicting works in all cases, including when self._maxsize is 0
+                self._container[key] = value
+                if len(self._container) > self._maxsize:
+                    # If we didn't evict an existing value, and we've hit our maximum
+                    # size, then we have to evict the least recently used item from
+                    # the beginning of the container.
+                    evicted_item = self._container.popitem(last=False)
+
+        # After releasing the lock on the pool, dispose of any evicted value.
+        if evicted_item is not None and self.dispose_func:
+            _, evicted_value = evicted_item
+            self.dispose_func(evicted_value)
+
+    def __delitem__(self, key: _KT) -> None:
+        with self.lock:
+            value = self._container.pop(key)
+
+        if self.dispose_func:
+            self.dispose_func(value)
+
+    def __len__(self) -> int:
+        with self.lock:
+            return len(self._container)
+
+    def __iter__(self) -> typing.NoReturn:
+        raise NotImplementedError(
+            "Iteration over this class is unlikely to be threadsafe."
+        )
+
+    def clear(self) -> None:
+        with self.lock:
+            # Copy pointers to all values, then wipe the mapping
+            values = list(self._container.values())
+            self._container.clear()
+
+        if self.dispose_func:
+            for value in values:
+                self.dispose_func(value)
+
+    def keys(self) -> set[_KT]:  # type: ignore[override]
+        with self.lock:
+            return set(self._container.keys())
+
+
+class HTTPHeaderDictItemView(typing.Set[typing.Tuple[str, str]]):
+    """
+    HTTPHeaderDict is unusual for a Mapping[str, str] in that it has two modes of
+    address.
+
+    If we directly try to get an item with a particular name, we will get a string
+    back that is the concatenated version of all the values:
+
+    >>> d['X-Header-Name']
+    'Value1, Value2, Value3'
+
+    However, if we iterate over an HTTPHeaderDict's items, we will optionally combine
+    these values based on whether combine=True was called when building up the dictionary
+
+    >>> d = HTTPHeaderDict({"A": "1", "B": "foo"})
+    >>> d.add("A", "2", combine=True)
+    >>> d.add("B", "bar")
+    >>> list(d.items())
+    [
+        ('A', '1, 2'),
+        ('B', 'foo'),
+        ('B', 'bar'),
+    ]
+
+    This class conforms to the interface required by the MutableMapping ABC while
+    also giving us the nonstandard iteration behavior we want; items with duplicate
+    keys, ordered by time of first insertion.
+    """
+
+    _headers: HTTPHeaderDict
+
+    def __init__(self, headers: HTTPHeaderDict) -> None:
+        self._headers = headers
+
+    def __len__(self) -> int:
+        return len(list(self._headers.iteritems()))
+
+    def __iter__(self) -> typing.Iterator[tuple[str, str]]:
+        return self._headers.iteritems()
+
+    def __contains__(self, item: object) -> bool:
+        if isinstance(item, tuple) and len(item) == 2:
+            passed_key, passed_val = item
+            if isinstance(passed_key, str) and isinstance(passed_val, str):
+                return self._headers._has_value_for_header(passed_key, passed_val)
+        return False
+
+
+class HTTPHeaderDict(typing.MutableMapping[str, str]):
+    """
+    :param headers:
+        An iterable of field-value pairs. Must not contain multiple field names
+        when compared case-insensitively.
+
+    :param kwargs:
+        Additional field-value pairs to pass in to ``dict.update``.
+
+    A ``dict`` like container for storing HTTP Headers.
+
+    Field names are stored and compared case-insensitively in compliance with
+    RFC 7230. Iteration provides the first case-sensitive key seen for each
+    case-insensitive pair.
+
+    Using ``__setitem__`` syntax overwrites fields that compare equal
+    case-insensitively in order to maintain ``dict``'s api. For fields that
+    compare equal, instead create a new ``HTTPHeaderDict`` and use ``.add``
+    in a loop.
+
+    If multiple fields that are equal case-insensitively are passed to the
+    constructor or ``.update``, the behavior is undefined and some will be
+    lost.
+
+    >>> headers = HTTPHeaderDict()
+    >>> headers.add('Set-Cookie', 'foo=bar')
+    >>> headers.add('set-cookie', 'baz=quxx')
+    >>> headers['content-length'] = '7'
+    >>> headers['SET-cookie']
+    'foo=bar, baz=quxx'
+    >>> headers['Content-Length']
+    '7'
+    """
+
+    _container: typing.MutableMapping[str, list[str]]
+
+    def __init__(self, headers: ValidHTTPHeaderSource | None = None, **kwargs: str):
+        super().__init__()
+        self._container = {}  # 'dict' is insert-ordered
+        if headers is not None:
+            if isinstance(headers, HTTPHeaderDict):
+                self._copy_from(headers)
+            else:
+                self.extend(headers)
+        if kwargs:
+            self.extend(kwargs)
+
+    def __setitem__(self, key: str, val: str) -> None:
+        # avoid a bytes/str comparison by decoding before httplib
+        if isinstance(key, bytes):
+            key = key.decode("latin-1")
+        self._container[key.lower()] = [key, val]
+
+    def __getitem__(self, key: str) -> str:
+        val = self._container[key.lower()]
+        return ", ".join(val[1:])
+
+    def __delitem__(self, key: str) -> None:
+        del self._container[key.lower()]
+
+    def __contains__(self, key: object) -> bool:
+        if isinstance(key, str):
+            return key.lower() in self._container
+        return False
+
+    def setdefault(self, key: str, default: str = "") -> str:
+        return super().setdefault(key, default)
+
+    def __eq__(self, other: object) -> bool:
+        maybe_constructable = ensure_can_construct_http_header_dict(other)
+        if maybe_constructable is None:
+            return False
+        else:
+            other_as_http_header_dict = type(self)(maybe_constructable)
+
+        return {k.lower(): v for k, v in self.itermerged()} == {
+            k.lower(): v for k, v in other_as_http_header_dict.itermerged()
+        }
+
+    def __ne__(self, other: object) -> bool:
+        return not self.__eq__(other)
+
+    def __len__(self) -> int:
+        return len(self._container)
+
+    def __iter__(self) -> typing.Iterator[str]:
+        # Only provide the originally cased names
+        for vals in self._container.values():
+            yield vals[0]
+
+    def discard(self, key: str) -> None:
+        try:
+            del self[key]
+        except KeyError:
+            pass
+
+    def add(self, key: str, val: str, *, combine: bool = False) -> None:
+        """Adds a (name, value) pair, doesn't overwrite the value if it already
+        exists.
+
+        If this is called with combine=True, instead of adding a new header value
+        as a distinct item during iteration, this will instead append the value to
+        any existing header value with a comma. If no existing header value exists
+        for the key, then the value will simply be added, ignoring the combine parameter.
+
+        >>> headers = HTTPHeaderDict(foo='bar')
+        >>> headers.add('Foo', 'baz')
+        >>> headers['foo']
+        'bar, baz'
+        >>> list(headers.items())
+        [('foo', 'bar'), ('foo', 'baz')]
+        >>> headers.add('foo', 'quz', combine=True)
+        >>> list(headers.items())
+        [('foo', 'bar, baz, quz')]
+        """
+        # avoid a bytes/str comparison by decoding before httplib
+        if isinstance(key, bytes):
+            key = key.decode("latin-1")
+        key_lower = key.lower()
+        new_vals = [key, val]
+        # Keep the common case aka no item present as fast as possible
+        vals = self._container.setdefault(key_lower, new_vals)
+        if new_vals is not vals:
+            # if there are values here, then there is at least the initial
+            # key/value pair
+            assert len(vals) >= 2
+            if combine:
+                vals[-1] = vals[-1] + ", " + val
+            else:
+                vals.append(val)
+
+    def extend(self, *args: ValidHTTPHeaderSource, **kwargs: str) -> None:
+        """Generic import function for any type of header-like object.
+        Adapted version of MutableMapping.update in order to insert items
+        with self.add instead of self.__setitem__
+        """
+        if len(args) > 1:
+            raise TypeError(
+                f"extend() takes at most 1 positional arguments ({len(args)} given)"
+            )
+        other = args[0] if len(args) >= 1 else ()
+
+        if isinstance(other, HTTPHeaderDict):
+            for key, val in other.iteritems():
+                self.add(key, val)
+        elif isinstance(other, typing.Mapping):
+            for key, val in other.items():
+                self.add(key, val)
+        elif isinstance(other, typing.Iterable):
+            other = typing.cast(typing.Iterable[typing.Tuple[str, str]], other)
+            for key, value in other:
+                self.add(key, value)
+        elif hasattr(other, "keys") and hasattr(other, "__getitem__"):
+            # THIS IS NOT A TYPESAFE BRANCH
+            # In this branch, the object has a `keys` attr but is not a Mapping or any of
+            # the other types indicated in the method signature. We do some stuff with
+            # it as though it partially implements the Mapping interface, but we're not
+            # doing that stuff safely AT ALL.
+            for key in other.keys():
+                self.add(key, other[key])
+
+        for key, value in kwargs.items():
+            self.add(key, value)
+
+    @typing.overload
+    def getlist(self, key: str) -> list[str]:
+        ...
+
+    @typing.overload
+    def getlist(self, key: str, default: _DT) -> list[str] | _DT:
+        ...
+
+    def getlist(
+        self, key: str, default: _Sentinel | _DT = _Sentinel.not_passed
+    ) -> list[str] | _DT:
+        """Returns a list of all the values for the named field. Returns an
+        empty list if the key doesn't exist."""
+        try:
+            vals = self._container[key.lower()]
+        except KeyError:
+            if default is _Sentinel.not_passed:
+                # _DT is unbound; empty list is instance of List[str]
+                return []
+            # _DT is bound; default is instance of _DT
+            return default
+        else:
+            # _DT may or may not be bound; vals[1:] is instance of List[str], which
+            # meets our external interface requirement of `Union[List[str], _DT]`.
+            return vals[1:]
+
+    def _prepare_for_method_change(self) -> Self:
+        """
+        Remove content-specific header fields before changing the request
+        method to GET or HEAD according to RFC 9110, Section 15.4.
+        """
+        content_specific_headers = [
+            "Content-Encoding",
+            "Content-Language",
+            "Content-Location",
+            "Content-Type",
+            "Content-Length",
+            "Digest",
+            "Last-Modified",
+        ]
+        for header in content_specific_headers:
+            self.discard(header)
+        return self
+
+    # Backwards compatibility for httplib
+    getheaders = getlist
+    getallmatchingheaders = getlist
+    iget = getlist
+
+    # Backwards compatibility for http.cookiejar
+    get_all = getlist
+
+    def __repr__(self) -> str:
+        return f"{type(self).__name__}({dict(self.itermerged())})"
+
+    def _copy_from(self, other: HTTPHeaderDict) -> None:
+        for key in other:
+            val = other.getlist(key)
+            self._container[key.lower()] = [key, *val]
+
+    def copy(self) -> HTTPHeaderDict:
+        clone = type(self)()
+        clone._copy_from(self)
+        return clone
+
+    def iteritems(self) -> typing.Iterator[tuple[str, str]]:
+        """Iterate over all header lines, including duplicate ones."""
+        for key in self:
+            vals = self._container[key.lower()]
+            for val in vals[1:]:
+                yield vals[0], val
+
+    def itermerged(self) -> typing.Iterator[tuple[str, str]]:
+        """Iterate over all headers, merging duplicate ones together."""
+        for key in self:
+            val = self._container[key.lower()]
+            yield val[0], ", ".join(val[1:])
+
+    def items(self) -> HTTPHeaderDictItemView:  # type: ignore[override]
+        return HTTPHeaderDictItemView(self)
+
+    def _has_value_for_header(self, header_name: str, potential_value: str) -> bool:
+        if header_name in self:
+            return potential_value in self._container[header_name.lower()][1:]
+        return False
+
+    def __ior__(self, other: object) -> HTTPHeaderDict:
+        # Supports extending a header dict in-place using operator |=
+        # combining items with add instead of __setitem__
+        maybe_constructable = ensure_can_construct_http_header_dict(other)
+        if maybe_constructable is None:
+            return NotImplemented
+        self.extend(maybe_constructable)
+        return self
+
+    def __or__(self, other: object) -> HTTPHeaderDict:
+        # Supports merging header dicts using operator |
+        # combining items with add instead of __setitem__
+        maybe_constructable = ensure_can_construct_http_header_dict(other)
+        if maybe_constructable is None:
+            return NotImplemented
+        result = self.copy()
+        result.extend(maybe_constructable)
+        return result
+
+    def __ror__(self, other: object) -> HTTPHeaderDict:
+        # Supports merging header dicts using operator | when other is on left side
+        # combining items with add instead of __setitem__
+        maybe_constructable = ensure_can_construct_http_header_dict(other)
+        if maybe_constructable is None:
+            return NotImplemented
+        result = type(self)(maybe_constructable)
+        result.extend(self)
+        return result

Modules/urllib3/_request_methods.py

@@ -0,0 +1,217 @@
+from __future__ import annotations
+
+import json as _json
+import typing
+from urllib.parse import urlencode
+
+from ._base_connection import _TYPE_BODY
+from ._collections import HTTPHeaderDict
+from .filepost import _TYPE_FIELDS, encode_multipart_formdata
+from .response import BaseHTTPResponse
+
+__all__ = ["RequestMethods"]
+
+_TYPE_ENCODE_URL_FIELDS = typing.Union[
+    typing.Sequence[typing.Tuple[str, typing.Union[str, bytes]]],
+    typing.Mapping[str, typing.Union[str, bytes]],
+]
+
+
+class RequestMethods:
+    """
+    Convenience mixin for classes who implement a :meth:`urlopen` method, such
+    as :class:`urllib3.HTTPConnectionPool` and
+    :class:`urllib3.PoolManager`.
+
+    Provides behavior for making common types of HTTP request methods and
+    decides which type of request field encoding to use.
+
+    Specifically,
+
+    :meth:`.request_encode_url` is for sending requests whose fields are
+    encoded in the URL (such as GET, HEAD, DELETE).
+
+    :meth:`.request_encode_body` is for sending requests whose fields are
+    encoded in the *body* of the request using multipart or www-form-urlencoded
+    (such as for POST, PUT, PATCH).
+
+    :meth:`.request` is for making any kind of request, it will look up the
+    appropriate encoding format and use one of the above two methods to make
+    the request.
+
+    Initializer parameters:
+
+    :param headers:
+        Headers to include with all requests, unless other headers are given
+        explicitly.
+    """
+
+    _encode_url_methods = {"DELETE", "GET", "HEAD", "OPTIONS"}
+
+    def __init__(self, headers: typing.Mapping[str, str] | None = None) -> None:
+        self.headers = headers or {}
+
+    def urlopen(
+        self,
+        method: str,
+        url: str,
+        body: _TYPE_BODY | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        encode_multipart: bool = True,
+        multipart_boundary: str | None = None,
+        **kw: typing.Any,
+    ) -> BaseHTTPResponse:  # Abstract
+        raise NotImplementedError(
+            "Classes extending RequestMethods must implement "
+            "their own ``urlopen`` method."
+        )
+
+    def request(
+        self,
+        method: str,
+        url: str,
+        body: _TYPE_BODY | None = None,
+        fields: _TYPE_FIELDS | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        json: typing.Any | None = None,
+        **urlopen_kw: typing.Any,
+    ) -> BaseHTTPResponse:
+        """
+        Make a request using :meth:`urlopen` with the appropriate encoding of
+        ``fields`` based on the ``method`` used.
+
+        This is a convenience method that requires the least amount of manual
+        effort. It can be used in most situations, while still having the
+        option to drop down to more specific methods when necessary, such as
+        :meth:`request_encode_url`, :meth:`request_encode_body`,
+        or even the lowest level :meth:`urlopen`.
+        """
+        method = method.upper()
+
+        if json is not None and body is not None:
+            raise TypeError(
+                "request got values for both 'body' and 'json' parameters which are mutually exclusive"
+            )
+
+        if json is not None:
+            if headers is None:
+                headers = self.headers.copy()  # type: ignore
+            if not ("content-type" in map(str.lower, headers.keys())):
+                headers["Content-Type"] = "application/json"  # type: ignore
+
+            body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode(
+                "utf-8"
+            )
+
+        if body is not None:
+            urlopen_kw["body"] = body
+
+        if method in self._encode_url_methods:
+            return self.request_encode_url(
+                method,
+                url,
+                fields=fields,  # type: ignore[arg-type]
+                headers=headers,
+                **urlopen_kw,
+            )
+        else:
+            return self.request_encode_body(
+                method, url, fields=fields, headers=headers, **urlopen_kw
+            )
+
+    def request_encode_url(
+        self,
+        method: str,
+        url: str,
+        fields: _TYPE_ENCODE_URL_FIELDS | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        **urlopen_kw: str,
+    ) -> BaseHTTPResponse:
+        """
+        Make a request using :meth:`urlopen` with the ``fields`` encoded in
+        the url. This is useful for request methods like GET, HEAD, DELETE, etc.
+        """
+        if headers is None:
+            headers = self.headers
+
+        extra_kw: dict[str, typing.Any] = {"headers": headers}
+        extra_kw.update(urlopen_kw)
+
+        if fields:
+            url += "?" + urlencode(fields)
+
+        return self.urlopen(method, url, **extra_kw)
+
+    def request_encode_body(
+        self,
+        method: str,
+        url: str,
+        fields: _TYPE_FIELDS | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        encode_multipart: bool = True,
+        multipart_boundary: str | None = None,
+        **urlopen_kw: str,
+    ) -> BaseHTTPResponse:
+        """
+        Make a request using :meth:`urlopen` with the ``fields`` encoded in
+        the body. This is useful for request methods like POST, PUT, PATCH, etc.
+
+        When ``encode_multipart=True`` (default), then
+        :func:`urllib3.encode_multipart_formdata` is used to encode
+        the payload with the appropriate content type. Otherwise
+        :func:`urllib.parse.urlencode` is used with the
+        'application/x-www-form-urlencoded' content type.
+
+        Multipart encoding must be used when posting files, and it's reasonably
+        safe to use it in other times too. However, it may break request
+        signing, such as with OAuth.
+
+        Supports an optional ``fields`` parameter of key/value strings AND
+        key/filetuple. A filetuple is a (filename, data, MIME type) tuple where
+        the MIME type is optional. For example::
+
+            fields = {
+                'foo': 'bar',
+                'fakefile': ('foofile.txt', 'contents of foofile'),
+                'realfile': ('barfile.txt', open('realfile').read()),
+                'typedfile': ('bazfile.bin', open('bazfile').read(),
+                              'image/jpeg'),
+                'nonamefile': 'contents of nonamefile field',
+            }
+
+        When uploading a file, providing a filename (the first parameter of the
+        tuple) is optional but recommended to best mimic behavior of browsers.
+
+        Note that if ``headers`` are supplied, the 'Content-Type' header will
+        be overwritten because it depends on the dynamic random boundary string
+        which is used to compose the body of the request. The random boundary
+        string can be explicitly set with the ``multipart_boundary`` parameter.
+        """
+        if headers is None:
+            headers = self.headers
+
+        extra_kw: dict[str, typing.Any] = {"headers": HTTPHeaderDict(headers)}
+        body: bytes | str
+
+        if fields:
+            if "body" in urlopen_kw:
+                raise TypeError(
+                    "request got values for both 'fields' and 'body', can only specify one."
+                )
+
+            if encode_multipart:
+                body, content_type = encode_multipart_formdata(
+                    fields, boundary=multipart_boundary
+                )
+            else:
+                body, content_type = (
+                    urlencode(fields),  # type: ignore[arg-type]
+                    "application/x-www-form-urlencoded",
+                )
+
+            extra_kw["body"] = body
+            extra_kw["headers"].setdefault("Content-Type", content_type)
+
+        extra_kw.update(urlopen_kw)
+
+        return self.urlopen(method, url, **extra_kw)

Modules/urllib3/_version.py

@@ -0,0 +1,4 @@
+# This file is protected via CODEOWNERS
+from __future__ import annotations
+
+__version__ = "2.1.0"

Modules/urllib3/connection.py

@@ -0,0 +1,905 @@
+from __future__ import annotations
+
+import datetime
+import logging
+import os
+import re
+import socket
+import sys
+import typing
+import warnings
+from http.client import HTTPConnection as _HTTPConnection
+from http.client import HTTPException as HTTPException  # noqa: F401
+from http.client import ResponseNotReady
+from socket import timeout as SocketTimeout
+
+if typing.TYPE_CHECKING:
+    from typing import Literal
+
+    from .response import HTTPResponse
+    from .util.ssl_ import _TYPE_PEER_CERT_RET_DICT
+    from .util.ssltransport import SSLTransport
+
+from ._collections import HTTPHeaderDict
+from .util.response import assert_header_parsing
+from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT, Timeout
+from .util.util import to_str
+from .util.wait import wait_for_read
+
+try:  # Compiled with SSL?
+    import ssl
+
+    BaseSSLError = ssl.SSLError
+except (ImportError, AttributeError):
+    ssl = None  # type: ignore[assignment]
+
+    class BaseSSLError(BaseException):  # type: ignore[no-redef]
+        pass
+
+
+from ._base_connection import _TYPE_BODY
+from ._base_connection import ProxyConfig as ProxyConfig
+from ._base_connection import _ResponseOptions as _ResponseOptions
+from ._version import __version__
+from .exceptions import (
+    ConnectTimeoutError,
+    HeaderParsingError,
+    NameResolutionError,
+    NewConnectionError,
+    ProxyError,
+    SystemTimeWarning,
+)
+from .util import SKIP_HEADER, SKIPPABLE_HEADERS, connection, ssl_
+from .util.request import body_to_chunks
+from .util.ssl_ import assert_fingerprint as _assert_fingerprint
+from .util.ssl_ import (
+    create_urllib3_context,
+    is_ipaddress,
+    resolve_cert_reqs,
+    resolve_ssl_version,
+    ssl_wrap_socket,
+)
+from .util.ssl_match_hostname import CertificateError, match_hostname
+from .util.url import Url
+
+# Not a no-op, we're adding this to the namespace so it can be imported.
+ConnectionError = ConnectionError
+BrokenPipeError = BrokenPipeError
+
+
+log = logging.getLogger(__name__)
+
+port_by_scheme = {"http": 80, "https": 443}
+
+# When it comes time to update this value as a part of regular maintenance
+# (ie test_recent_date is failing) update it to ~6 months before the current date.
+RECENT_DATE = datetime.date(2022, 1, 1)
+
+_CONTAINS_CONTROL_CHAR_RE = re.compile(r"[^-!#$%&'*+.^_`|~0-9a-zA-Z]")
+
+_HAS_SYS_AUDIT = hasattr(sys, "audit")
+
+
+class HTTPConnection(_HTTPConnection):
+    """
+    Based on :class:`http.client.HTTPConnection` but provides an extra constructor
+    backwards-compatibility layer between older and newer Pythons.
+
+    Additional keyword parameters are used to configure attributes of the connection.
+    Accepted parameters include:
+
+    - ``source_address``: Set the source address for the current connection.
+    - ``socket_options``: Set specific options on the underlying socket. If not specified, then
+      defaults are loaded from ``HTTPConnection.default_socket_options`` which includes disabling
+      Nagle's algorithm (sets TCP_NODELAY to 1) unless the connection is behind a proxy.
+
+      For example, if you wish to enable TCP Keep Alive in addition to the defaults,
+      you might pass:
+
+      .. code-block:: python
+
+         HTTPConnection.default_socket_options + [
+             (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
+         ]
+
+      Or you may want to disable the defaults by passing an empty list (e.g., ``[]``).
+    """
+
+    default_port: typing.ClassVar[int] = port_by_scheme["http"]  # type: ignore[misc]
+
+    #: Disable Nagle's algorithm by default.
+    #: ``[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]``
+    default_socket_options: typing.ClassVar[connection._TYPE_SOCKET_OPTIONS] = [
+        (socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+    ]
+
+    #: Whether this connection verifies the host's certificate.
+    is_verified: bool = False
+
+    #: Whether this proxy connection verified the proxy host's certificate.
+    # If no proxy is currently connected to the value will be ``None``.
+    proxy_is_verified: bool | None = None
+
+    blocksize: int
+    source_address: tuple[str, int] | None
+    socket_options: connection._TYPE_SOCKET_OPTIONS | None
+
+    _has_connected_to_proxy: bool
+    _response_options: _ResponseOptions | None
+    _tunnel_host: str | None
+    _tunnel_port: int | None
+    _tunnel_scheme: str | None
+
+    def __init__(
+        self,
+        host: str,
+        port: int | None = None,
+        *,
+        timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+        source_address: tuple[str, int] | None = None,
+        blocksize: int = 16384,
+        socket_options: None
+        | (connection._TYPE_SOCKET_OPTIONS) = default_socket_options,
+        proxy: Url | None = None,
+        proxy_config: ProxyConfig | None = None,
+    ) -> None:
+        super().__init__(
+            host=host,
+            port=port,
+            timeout=Timeout.resolve_default_timeout(timeout),
+            source_address=source_address,
+            blocksize=blocksize,
+        )
+        self.socket_options = socket_options
+        self.proxy = proxy
+        self.proxy_config = proxy_config
+
+        self._has_connected_to_proxy = False
+        self._response_options = None
+        self._tunnel_host: str | None = None
+        self._tunnel_port: int | None = None
+        self._tunnel_scheme: str | None = None
+
+    # https://github.com/python/mypy/issues/4125
+    # Mypy treats this as LSP violation, which is considered a bug.
+    # If `host` is made a property it violates LSP, because a writeable attribute is overridden with a read-only one.
+    # However, there is also a `host` setter so LSP is not violated.
+    # Potentially, a `@host.deleter` might be needed depending on how this issue will be fixed.
+    @property
+    def host(self) -> str:
+        """
+        Getter method to remove any trailing dots that indicate the hostname is an FQDN.
+
+        In general, SSL certificates don't include the trailing dot indicating a
+        fully-qualified domain name, and thus, they don't validate properly when
+        checked against a domain name that includes the dot. In addition, some
+        servers may not expect to receive the trailing dot when provided.
+
+        However, the hostname with trailing dot is critical to DNS resolution; doing a
+        lookup with the trailing dot will properly only resolve the appropriate FQDN,
+        whereas a lookup without a trailing dot will search the system's search domain
+        list. Thus, it's important to keep the original host around for use only in
+        those cases where it's appropriate (i.e., when doing DNS lookup to establish the
+        actual TCP connection across which we're going to send HTTP requests).
+        """
+        return self._dns_host.rstrip(".")
+
+    @host.setter
+    def host(self, value: str) -> None:
+        """
+        Setter for the `host` property.
+
+        We assume that only urllib3 uses the _dns_host attribute; httplib itself
+        only uses `host`, and it seems reasonable that other libraries follow suit.
+        """
+        self._dns_host = value
+
+    def _new_conn(self) -> socket.socket:
+        """Establish a socket connection and set nodelay settings on it.
+
+        :return: New socket connection.
+        """
+        try:
+            sock = connection.create_connection(
+                (self._dns_host, self.port),
+                self.timeout,
+                source_address=self.source_address,
+                socket_options=self.socket_options,
+            )
+        except socket.gaierror as e:
+            raise NameResolutionError(self.host, self, e) from e
+        except SocketTimeout as e:
+            raise ConnectTimeoutError(
+                self,
+                f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
+            ) from e
+
+        except OSError as e:
+            raise NewConnectionError(
+                self, f"Failed to establish a new connection: {e}"
+            ) from e
+
+        # Audit hooks are only available in Python 3.8+
+        if _HAS_SYS_AUDIT:
+            sys.audit("http.client.connect", self, self.host, self.port)
+
+        return sock
+
+    def set_tunnel(
+        self,
+        host: str,
+        port: int | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        scheme: str = "http",
+    ) -> None:
+        if scheme not in ("http", "https"):
+            raise ValueError(
+                f"Invalid proxy scheme for tunneling: {scheme!r}, must be either 'http' or 'https'"
+            )
+        super().set_tunnel(host, port=port, headers=headers)
+        self._tunnel_scheme = scheme
+
+    def connect(self) -> None:
+        self.sock = self._new_conn()
+        if self._tunnel_host:
+            # If we're tunneling it means we're connected to our proxy.
+            self._has_connected_to_proxy = True
+
+            # TODO: Fix tunnel so it doesn't depend on self.sock state.
+            self._tunnel()  # type: ignore[attr-defined]
+
+        # If there's a proxy to be connected to we are fully connected.
+        # This is set twice (once above and here) due to forwarding proxies
+        # not using tunnelling.
+        self._has_connected_to_proxy = bool(self.proxy)
+
+    @property
+    def is_closed(self) -> bool:
+        return self.sock is None
+
+    @property
+    def is_connected(self) -> bool:
+        if self.sock is None:
+            return False
+        return not wait_for_read(self.sock, timeout=0.0)
+
+    @property
+    def has_connected_to_proxy(self) -> bool:
+        return self._has_connected_to_proxy
+
+    def close(self) -> None:
+        try:
+            super().close()
+        finally:
+            # Reset all stateful properties so connection
+            # can be re-used without leaking prior configs.
+            self.sock = None
+            self.is_verified = False
+            self.proxy_is_verified = None
+            self._has_connected_to_proxy = False
+            self._response_options = None
+            self._tunnel_host = None
+            self._tunnel_port = None
+            self._tunnel_scheme = None
+
+    def putrequest(
+        self,
+        method: str,
+        url: str,
+        skip_host: bool = False,
+        skip_accept_encoding: bool = False,
+    ) -> None:
+        """"""
+        # Empty docstring because the indentation of CPython's implementation
+        # is broken but we don't want this method in our documentation.
+        match = _CONTAINS_CONTROL_CHAR_RE.search(method)
+        if match:
+            raise ValueError(
+                f"Method cannot contain non-token characters {method!r} (found at least {match.group()!r})"
+            )
+
+        return super().putrequest(
+            method, url, skip_host=skip_host, skip_accept_encoding=skip_accept_encoding
+        )
+
+    def putheader(self, header: str, *values: str) -> None:
+        """"""
+        if not any(isinstance(v, str) and v == SKIP_HEADER for v in values):
+            super().putheader(header, *values)
+        elif to_str(header.lower()) not in SKIPPABLE_HEADERS:
+            skippable_headers = "', '".join(
+                [str.title(header) for header in sorted(SKIPPABLE_HEADERS)]
+            )
+            raise ValueError(
+                f"urllib3.util.SKIP_HEADER only supports '{skippable_headers}'"
+            )
+
+    # `request` method's signature intentionally violates LSP.
+    # urllib3's API is different from `http.client.HTTPConnection` and the subclassing is only incidental.
+    def request(  # type: ignore[override]
+        self,
+        method: str,
+        url: str,
+        body: _TYPE_BODY | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        *,
+        chunked: bool = False,
+        preload_content: bool = True,
+        decode_content: bool = True,
+        enforce_content_length: bool = True,
+    ) -> None:
+        # Update the inner socket's timeout value to send the request.
+        # This only triggers if the connection is re-used.
+        if self.sock is not None:
+            self.sock.settimeout(self.timeout)
+
+        # Store these values to be fed into the HTTPResponse
+        # object later. TODO: Remove this in favor of a real
+        # HTTP lifecycle mechanism.
+
+        # We have to store these before we call .request()
+        # because sometimes we can still salvage a response
+        # off the wire even if we aren't able to completely
+        # send the request body.
+        self._response_options = _ResponseOptions(
+            request_method=method,
+            request_url=url,
+            preload_content=preload_content,
+            decode_content=decode_content,
+            enforce_content_length=enforce_content_length,
+        )
+
+        if headers is None:
+            headers = {}
+        header_keys = frozenset(to_str(k.lower()) for k in headers)
+        skip_accept_encoding = "accept-encoding" in header_keys
+        skip_host = "host" in header_keys
+        self.putrequest(
+            method, url, skip_accept_encoding=skip_accept_encoding, skip_host=skip_host
+        )
+
+        # Transform the body into an iterable of sendall()-able chunks
+        # and detect if an explicit Content-Length is doable.
+        chunks_and_cl = body_to_chunks(body, method=method, blocksize=self.blocksize)
+        chunks = chunks_and_cl.chunks
+        content_length = chunks_and_cl.content_length
+
+        # When chunked is explicit set to 'True' we respect that.
+        if chunked:
+            if "transfer-encoding" not in header_keys:
+                self.putheader("Transfer-Encoding", "chunked")
+        else:
+            # Detect whether a framing mechanism is already in use. If so
+            # we respect that value, otherwise we pick chunked vs content-length
+            # depending on the type of 'body'.
+            if "content-length" in header_keys:
+                chunked = False
+            elif "transfer-encoding" in header_keys:
+                chunked = True
+
+            # Otherwise we go off the recommendation of 'body_to_chunks()'.
+            else:
+                chunked = False
+                if content_length is None:
+                    if chunks is not None:
+                        chunked = True
+                        self.putheader("Transfer-Encoding", "chunked")
+                else:
+                    self.putheader("Content-Length", str(content_length))
+
+        # Now that framing headers are out of the way we send all the other headers.
+        if "user-agent" not in header_keys:
+            self.putheader("User-Agent", _get_default_user_agent())
+        for header, value in headers.items():
+            self.putheader(header, value)
+        self.endheaders()
+
+        # If we're given a body we start sending that in chunks.
+        if chunks is not None:
+            for chunk in chunks:
+                # Sending empty chunks isn't allowed for TE: chunked
+                # as it indicates the end of the body.
+                if not chunk:
+                    continue
+                if isinstance(chunk, str):
+                    chunk = chunk.encode("utf-8")
+                if chunked:
+                    self.send(b"%x\r\n%b\r\n" % (len(chunk), chunk))
+                else:
+                    self.send(chunk)
+
+        # Regardless of whether we have a body or not, if we're in
+        # chunked mode we want to send an explicit empty chunk.
+        if chunked:
+            self.send(b"0\r\n\r\n")
+
+    def request_chunked(
+        self,
+        method: str,
+        url: str,
+        body: _TYPE_BODY | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+    ) -> None:
+        """
+        Alternative to the common request method, which sends the
+        body with chunked encoding and not as one block
+        """
+        warnings.warn(
+            "HTTPConnection.request_chunked() is deprecated and will be removed "
+            "in urllib3 v2.1.0. Instead use HTTPConnection.request(..., chunked=True).",
+            category=DeprecationWarning,
+            stacklevel=2,
+        )
+        self.request(method, url, body=body, headers=headers, chunked=True)
+
+    def getresponse(  # type: ignore[override]
+        self,
+    ) -> HTTPResponse:
+        """
+        Get the response from the server.
+
+        If the HTTPConnection is in the correct state, returns an instance of HTTPResponse or of whatever object is returned by the response_class variable.
+
+        If a request has not been sent or if a previous response has not be handled, ResponseNotReady is raised. If the HTTP response indicates that the connection should be closed, then it will be closed before the response is returned. When the connection is closed, the underlying socket is closed.
+        """
+        # Raise the same error as http.client.HTTPConnection
+        if self._response_options is None:
+            raise ResponseNotReady()
+
+        # Reset this attribute for being used again.
+        resp_options = self._response_options
+        self._response_options = None
+
+        # Since the connection's timeout value may have been updated
+        # we need to set the timeout on the socket.
+        self.sock.settimeout(self.timeout)
+
+        # This is needed here to avoid circular import errors
+        from .response import HTTPResponse
+
+        # Get the response from http.client.HTTPConnection
+        httplib_response = super().getresponse()
+
+        try:
+            assert_header_parsing(httplib_response.msg)
+        except (HeaderParsingError, TypeError) as hpe:
+            log.warning(
+                "Failed to parse headers (url=%s): %s",
+                _url_from_connection(self, resp_options.request_url),
+                hpe,
+                exc_info=True,
+            )
+
+        headers = HTTPHeaderDict(httplib_response.msg.items())
+
+        response = HTTPResponse(
+            body=httplib_response,
+            headers=headers,
+            status=httplib_response.status,
+            version=httplib_response.version,
+            reason=httplib_response.reason,
+            preload_content=resp_options.preload_content,
+            decode_content=resp_options.decode_content,
+            original_response=httplib_response,
+            enforce_content_length=resp_options.enforce_content_length,
+            request_method=resp_options.request_method,
+            request_url=resp_options.request_url,
+        )
+        return response
+
+
+class HTTPSConnection(HTTPConnection):
+    """
+    Many of the parameters to this constructor are passed to the underlying SSL
+    socket by means of :py:func:`urllib3.util.ssl_wrap_socket`.
+    """
+
+    default_port = port_by_scheme["https"]  # type: ignore[misc]
+
+    cert_reqs: int | str | None = None
+    ca_certs: str | None = None
+    ca_cert_dir: str | None = None
+    ca_cert_data: None | str | bytes = None
+    ssl_version: int | str | None = None
+    ssl_minimum_version: int | None = None
+    ssl_maximum_version: int | None = None
+    assert_fingerprint: str | None = None
+
+    def __init__(
+        self,
+        host: str,
+        port: int | None = None,
+        *,
+        timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+        source_address: tuple[str, int] | None = None,
+        blocksize: int = 16384,
+        socket_options: None
+        | (connection._TYPE_SOCKET_OPTIONS) = HTTPConnection.default_socket_options,
+        proxy: Url | None = None,
+        proxy_config: ProxyConfig | None = None,
+        cert_reqs: int | str | None = None,
+        assert_hostname: None | str | Literal[False] = None,
+        assert_fingerprint: str | None = None,
+        server_hostname: str | None = None,
+        ssl_context: ssl.SSLContext | None = None,
+        ca_certs: str | None = None,
+        ca_cert_dir: str | None = None,
+        ca_cert_data: None | str | bytes = None,
+        ssl_minimum_version: int | None = None,
+        ssl_maximum_version: int | None = None,
+        ssl_version: int | str | None = None,  # Deprecated
+        cert_file: str | None = None,
+        key_file: str | None = None,
+        key_password: str | None = None,
+    ) -> None:
+        super().__init__(
+            host,
+            port=port,
+            timeout=timeout,
+            source_address=source_address,
+            blocksize=blocksize,
+            socket_options=socket_options,
+            proxy=proxy,
+            proxy_config=proxy_config,
+        )
+
+        self.key_file = key_file
+        self.cert_file = cert_file
+        self.key_password = key_password
+        self.ssl_context = ssl_context
+        self.server_hostname = server_hostname
+        self.assert_hostname = assert_hostname
+        self.assert_fingerprint = assert_fingerprint
+        self.ssl_version = ssl_version
+        self.ssl_minimum_version = ssl_minimum_version
+        self.ssl_maximum_version = ssl_maximum_version
+        self.ca_certs = ca_certs and os.path.expanduser(ca_certs)
+        self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir)
+        self.ca_cert_data = ca_cert_data
+
+        # cert_reqs depends on ssl_context so calculate last.
+        if cert_reqs is None:
+            if self.ssl_context is not None:
+                cert_reqs = self.ssl_context.verify_mode
+            else:
+                cert_reqs = resolve_cert_reqs(None)
+        self.cert_reqs = cert_reqs
+
+    def set_cert(
+        self,
+        key_file: str | None = None,
+        cert_file: str | None = None,
+        cert_reqs: int | str | None = None,
+        key_password: str | None = None,
+        ca_certs: str | None = None,
+        assert_hostname: None | str | Literal[False] = None,
+        assert_fingerprint: str | None = None,
+        ca_cert_dir: str | None = None,
+        ca_cert_data: None | str | bytes = None,
+    ) -> None:
+        """
+        This method should only be called once, before the connection is used.
+        """
+        warnings.warn(
+            "HTTPSConnection.set_cert() is deprecated and will be removed "
+            "in urllib3 v2.1.0. Instead provide the parameters to the "
+            "HTTPSConnection constructor.",
+            category=DeprecationWarning,
+            stacklevel=2,
+        )
+
+        # If cert_reqs is not provided we'll assume CERT_REQUIRED unless we also
+        # have an SSLContext object in which case we'll use its verify_mode.
+        if cert_reqs is None:
+            if self.ssl_context is not None:
+                cert_reqs = self.ssl_context.verify_mode
+            else:
+                cert_reqs = resolve_cert_reqs(None)
+
+        self.key_file = key_file
+        self.cert_file = cert_file
+        self.cert_reqs = cert_reqs
+        self.key_password = key_password
+        self.assert_hostname = assert_hostname
+        self.assert_fingerprint = assert_fingerprint
+        self.ca_certs = ca_certs and os.path.expanduser(ca_certs)
+        self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir)
+        self.ca_cert_data = ca_cert_data
+
+    def connect(self) -> None:
+        sock: socket.socket | ssl.SSLSocket
+        self.sock = sock = self._new_conn()
+        server_hostname: str = self.host
+        tls_in_tls = False
+
+        # Do we need to establish a tunnel?
+        if self._tunnel_host is not None:
+            # We're tunneling to an HTTPS origin so need to do TLS-in-TLS.
+            if self._tunnel_scheme == "https":
+                self.sock = sock = self._connect_tls_proxy(self.host, sock)
+                tls_in_tls = True
+
+            # If we're tunneling it means we're connected to our proxy.
+            self._has_connected_to_proxy = True
+
+            self._tunnel()  # type: ignore[attr-defined]
+            # Override the host with the one we're requesting data from.
+            server_hostname = self._tunnel_host
+
+        if self.server_hostname is not None:
+            server_hostname = self.server_hostname
+
+        is_time_off = datetime.date.today() < RECENT_DATE
+        if is_time_off:
+            warnings.warn(
+                (
+                    f"System time is way off (before {RECENT_DATE}). This will probably "
+                    "lead to SSL verification errors"
+                ),
+                SystemTimeWarning,
+            )
+
+        sock_and_verified = _ssl_wrap_socket_and_match_hostname(
+            sock=sock,
+            cert_reqs=self.cert_reqs,
+            ssl_version=self.ssl_version,
+            ssl_minimum_version=self.ssl_minimum_version,
+            ssl_maximum_version=self.ssl_maximum_version,
+            ca_certs=self.ca_certs,
+            ca_cert_dir=self.ca_cert_dir,
+            ca_cert_data=self.ca_cert_data,
+            cert_file=self.cert_file,
+            key_file=self.key_file,
+            key_password=self.key_password,
+            server_hostname=server_hostname,
+            ssl_context=self.ssl_context,
+            tls_in_tls=tls_in_tls,
+            assert_hostname=self.assert_hostname,
+            assert_fingerprint=self.assert_fingerprint,
+        )
+        self.sock = sock_and_verified.socket
+        self.is_verified = sock_and_verified.is_verified
+
+        # If there's a proxy to be connected to we are fully connected.
+        # This is set twice (once above and here) due to forwarding proxies
+        # not using tunnelling.
+        self._has_connected_to_proxy = bool(self.proxy)
+
+    def _connect_tls_proxy(self, hostname: str, sock: socket.socket) -> ssl.SSLSocket:
+        """
+        Establish a TLS connection to the proxy using the provided SSL context.
+        """
+        # `_connect_tls_proxy` is called when self._tunnel_host is truthy.
+        proxy_config = typing.cast(ProxyConfig, self.proxy_config)
+        ssl_context = proxy_config.ssl_context
+        sock_and_verified = _ssl_wrap_socket_and_match_hostname(
+            sock,
+            cert_reqs=self.cert_reqs,
+            ssl_version=self.ssl_version,
+            ssl_minimum_version=self.ssl_minimum_version,
+            ssl_maximum_version=self.ssl_maximum_version,
+            ca_certs=self.ca_certs,
+            ca_cert_dir=self.ca_cert_dir,
+            ca_cert_data=self.ca_cert_data,
+            server_hostname=hostname,
+            ssl_context=ssl_context,
+            assert_hostname=proxy_config.assert_hostname,
+            assert_fingerprint=proxy_config.assert_fingerprint,
+            # Features that aren't implemented for proxies yet:
+            cert_file=None,
+            key_file=None,
+            key_password=None,
+            tls_in_tls=False,
+        )
+        self.proxy_is_verified = sock_and_verified.is_verified
+        return sock_and_verified.socket  # type: ignore[return-value]
+
+
+class _WrappedAndVerifiedSocket(typing.NamedTuple):
+    """
+    Wrapped socket and whether the connection is
+    verified after the TLS handshake
+    """
+
+    socket: ssl.SSLSocket | SSLTransport
+    is_verified: bool
+
+
+def _ssl_wrap_socket_and_match_hostname(
+    sock: socket.socket,
+    *,
+    cert_reqs: None | str | int,
+    ssl_version: None | str | int,
+    ssl_minimum_version: int | None,
+    ssl_maximum_version: int | None,
+    cert_file: str | None,
+    key_file: str | None,
+    key_password: str | None,
+    ca_certs: str | None,
+    ca_cert_dir: str | None,
+    ca_cert_data: None | str | bytes,
+    assert_hostname: None | str | Literal[False],
+    assert_fingerprint: str | None,
+    server_hostname: str | None,
+    ssl_context: ssl.SSLContext | None,
+    tls_in_tls: bool = False,
+) -> _WrappedAndVerifiedSocket:
+    """Logic for constructing an SSLContext from all TLS parameters, passing
+    that down into ssl_wrap_socket, and then doing certificate verification
+    either via hostname or fingerprint. This function exists to guarantee
+    that both proxies and targets have the same behavior when connecting via TLS.
+    """
+    default_ssl_context = False
+    if ssl_context is None:
+        default_ssl_context = True
+        context = create_urllib3_context(
+            ssl_version=resolve_ssl_version(ssl_version),
+            ssl_minimum_version=ssl_minimum_version,
+            ssl_maximum_version=ssl_maximum_version,
+            cert_reqs=resolve_cert_reqs(cert_reqs),
+        )
+    else:
+        context = ssl_context
+
+    context.verify_mode = resolve_cert_reqs(cert_reqs)
+
+    # In some cases, we want to verify hostnames ourselves
+    if (
+        # `ssl` can't verify fingerprints or alternate hostnames
+        assert_fingerprint
+        or assert_hostname
+        # assert_hostname can be set to False to disable hostname checking
+        or assert_hostname is False
+        # We still support OpenSSL 1.0.2, which prevents us from verifying
+        # hostnames easily: https://github.com/pyca/pyopenssl/pull/933
+        or ssl_.IS_PYOPENSSL
+        or not ssl_.HAS_NEVER_CHECK_COMMON_NAME
+    ):
+        context.check_hostname = False
+
+    # Try to load OS default certs if none are given. We need to do the hasattr() check
+    # for custom pyOpenSSL SSLContext objects because they don't support
+    # load_default_certs().
+    if (
+        not ca_certs
+        and not ca_cert_dir
+        and not ca_cert_data
+        and default_ssl_context
+        and hasattr(context, "load_default_certs")
+    ):
+        context.load_default_certs()
+
+    # Ensure that IPv6 addresses are in the proper format and don't have a
+    # scope ID. Python's SSL module fails to recognize scoped IPv6 addresses
+    # and interprets them as DNS hostnames.
+    if server_hostname is not None:
+        normalized = server_hostname.strip("[]")
+        if "%" in normalized:
+            normalized = normalized[: normalized.rfind("%")]
+        if is_ipaddress(normalized):
+            server_hostname = normalized
+
+    ssl_sock = ssl_wrap_socket(
+        sock=sock,
+        keyfile=key_file,
+        certfile=cert_file,
+        key_password=key_password,
+        ca_certs=ca_certs,
+        ca_cert_dir=ca_cert_dir,
+        ca_cert_data=ca_cert_data,
+        server_hostname=server_hostname,
+        ssl_context=context,
+        tls_in_tls=tls_in_tls,
+    )
+
+    try:
+        if assert_fingerprint:
+            _assert_fingerprint(
+                ssl_sock.getpeercert(binary_form=True), assert_fingerprint
+            )
+        elif (
+            context.verify_mode != ssl.CERT_NONE
+            and not context.check_hostname
+            and assert_hostname is not False
+        ):
+            cert: _TYPE_PEER_CERT_RET_DICT = ssl_sock.getpeercert()  # type: ignore[assignment]
+
+            # Need to signal to our match_hostname whether to use 'commonName' or not.
+            # If we're using our own constructed SSLContext we explicitly set 'False'
+            # because PyPy hard-codes 'True' from SSLContext.hostname_checks_common_name.
+            if default_ssl_context:
+                hostname_checks_common_name = False
+            else:
+                hostname_checks_common_name = (
+                    getattr(context, "hostname_checks_common_name", False) or False
+                )
+
+            _match_hostname(
+                cert,
+                assert_hostname or server_hostname,  # type: ignore[arg-type]
+                hostname_checks_common_name,
+            )
+
+        return _WrappedAndVerifiedSocket(
+            socket=ssl_sock,
+            is_verified=context.verify_mode == ssl.CERT_REQUIRED
+            or bool(assert_fingerprint),
+        )
+    except BaseException:
+        ssl_sock.close()
+        raise
+
+
+def _match_hostname(
+    cert: _TYPE_PEER_CERT_RET_DICT | None,
+    asserted_hostname: str,
+    hostname_checks_common_name: bool = False,
+) -> None:
+    # Our upstream implementation of ssl.match_hostname()
+    # only applies this normalization to IP addresses so it doesn't
+    # match DNS SANs so we do the same thing!
+    stripped_hostname = asserted_hostname.strip("[]")
+    if is_ipaddress(stripped_hostname):
+        asserted_hostname = stripped_hostname
+
+    try:
+        match_hostname(cert, asserted_hostname, hostname_checks_common_name)
+    except CertificateError as e:
+        log.warning(
+            "Certificate did not match expected hostname: %s. Certificate: %s",
+            asserted_hostname,
+            cert,
+        )
+        # Add cert to exception and reraise so client code can inspect
+        # the cert when catching the exception, if they want to
+        e._peer_cert = cert  # type: ignore[attr-defined]
+        raise
+
+
+def _wrap_proxy_error(err: Exception, proxy_scheme: str | None) -> ProxyError:
+    # Look for the phrase 'wrong version number', if found
+    # then we should warn the user that we're very sure that
+    # this proxy is HTTP-only and they have a configuration issue.
+    error_normalized = " ".join(re.split("[^a-z]", str(err).lower()))
+    is_likely_http_proxy = (
+        "wrong version number" in error_normalized
+        or "unknown protocol" in error_normalized
+    )
+    http_proxy_warning = (
+        ". Your proxy appears to only use HTTP and not HTTPS, "
+        "try changing your proxy URL to be HTTP. See: "
+        "https://urllib3.readthedocs.io/en/latest/advanced-usage.html"
+        "#https-proxy-error-http-proxy"
+    )
+    new_err = ProxyError(
+        f"Unable to connect to proxy"
+        f"{http_proxy_warning if is_likely_http_proxy and proxy_scheme == 'https' else ''}",
+        err,
+    )
+    new_err.__cause__ = err
+    return new_err
+
+
+def _get_default_user_agent() -> str:
+    return f"python-urllib3/{__version__}"
+
+
+class DummyConnection:
+    """Used to detect a failed ConnectionCls import."""
+
+
+if not ssl:
+    HTTPSConnection = DummyConnection  # type: ignore[misc, assignment] # noqa: F811
+
+
+VerifiedHTTPSConnection = HTTPSConnection
+
+
+def _url_from_connection(
+    conn: HTTPConnection | HTTPSConnection, path: str | None = None
+) -> str:
+    """Returns the URL from a given connection. This is mainly used for testing and logging."""
+
+    scheme = "https" if isinstance(conn, HTTPSConnection) else "http"
+
+    return Url(scheme=scheme, host=conn.host, port=conn.port, path=path).url

Modules/urllib3/contrib/pyopenssl.py

@@ -0,0 +1,548 @@
+"""
+Module for using pyOpenSSL as a TLS backend. This module was relevant before
+the standard library ``ssl`` module supported SNI, but now that we've dropped
+support for Python 2.7 all relevant Python versions support SNI so
+**this module is no longer recommended**.
+
+This needs the following packages installed:
+
+* `pyOpenSSL`_ (tested with 16.0.0)
+* `cryptography`_ (minimum 1.3.4, from pyopenssl)
+* `idna`_ (minimum 2.0)
+
+However, pyOpenSSL depends on cryptography, so while we use all three directly here we
+end up having relatively few packages required.
+
+You can install them with the following command:
+
+.. code-block:: bash
+
+    $ python -m pip install pyopenssl cryptography idna
+
+To activate certificate checking, call
+:func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code
+before you begin making HTTP requests. This can be done in a ``sitecustomize``
+module, or at any other time before your application begins using ``urllib3``,
+like this:
+
+.. code-block:: python
+
+    try:
+        import urllib3.contrib.pyopenssl
+        urllib3.contrib.pyopenssl.inject_into_urllib3()
+    except ImportError:
+        pass
+
+.. _pyopenssl: https://www.pyopenssl.org
+.. _cryptography: https://cryptography.io
+.. _idna: https://github.com/kjd/idna
+"""
+
+from __future__ import annotations
+
+import OpenSSL.SSL  # type: ignore[import]
+from cryptography import x509
+
+try:
+    from cryptography.x509 import UnsupportedExtension  # type: ignore[attr-defined]
+except ImportError:
+    # UnsupportedExtension is gone in cryptography >= 2.1.0
+    class UnsupportedExtension(Exception):  # type: ignore[no-redef]
+        pass
+
+
+import logging
+import ssl
+import typing
+from io import BytesIO
+from socket import socket as socket_cls
+from socket import timeout
+
+from .. import util
+
+if typing.TYPE_CHECKING:
+    from OpenSSL.crypto import X509  # type: ignore[import]
+
+
+__all__ = ["inject_into_urllib3", "extract_from_urllib3"]
+
+# Map from urllib3 to PyOpenSSL compatible parameter-values.
+_openssl_versions = {
+    util.ssl_.PROTOCOL_TLS: OpenSSL.SSL.SSLv23_METHOD,  # type: ignore[attr-defined]
+    util.ssl_.PROTOCOL_TLS_CLIENT: OpenSSL.SSL.SSLv23_METHOD,  # type: ignore[attr-defined]
+    ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
+}
+
+if hasattr(ssl, "PROTOCOL_TLSv1_1") and hasattr(OpenSSL.SSL, "TLSv1_1_METHOD"):
+    _openssl_versions[ssl.PROTOCOL_TLSv1_1] = OpenSSL.SSL.TLSv1_1_METHOD
+
+if hasattr(ssl, "PROTOCOL_TLSv1_2") and hasattr(OpenSSL.SSL, "TLSv1_2_METHOD"):
+    _openssl_versions[ssl.PROTOCOL_TLSv1_2] = OpenSSL.SSL.TLSv1_2_METHOD
+
+
+_stdlib_to_openssl_verify = {
+    ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
+    ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
+    ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER
+    + OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
+}
+_openssl_to_stdlib_verify = {v: k for k, v in _stdlib_to_openssl_verify.items()}
+
+# The SSLvX values are the most likely to be missing in the future
+# but we check them all just to be sure.
+_OP_NO_SSLv2_OR_SSLv3: int = getattr(OpenSSL.SSL, "OP_NO_SSLv2", 0) | getattr(
+    OpenSSL.SSL, "OP_NO_SSLv3", 0
+)
+_OP_NO_TLSv1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1", 0)
+_OP_NO_TLSv1_1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_1", 0)
+_OP_NO_TLSv1_2: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_2", 0)
+_OP_NO_TLSv1_3: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_3", 0)
+
+_openssl_to_ssl_minimum_version: dict[int, int] = {
+    ssl.TLSVersion.MINIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
+    ssl.TLSVersion.TLSv1: _OP_NO_SSLv2_OR_SSLv3,
+    ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1,
+    ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1,
+    ssl.TLSVersion.TLSv1_3: (
+        _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
+    ),
+    ssl.TLSVersion.MAXIMUM_SUPPORTED: (
+        _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
+    ),
+}
+_openssl_to_ssl_maximum_version: dict[int, int] = {
+    ssl.TLSVersion.MINIMUM_SUPPORTED: (
+        _OP_NO_SSLv2_OR_SSLv3
+        | _OP_NO_TLSv1
+        | _OP_NO_TLSv1_1
+        | _OP_NO_TLSv1_2
+        | _OP_NO_TLSv1_3
+    ),
+    ssl.TLSVersion.TLSv1: (
+        _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3
+    ),
+    ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3,
+    ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_3,
+    ssl.TLSVersion.TLSv1_3: _OP_NO_SSLv2_OR_SSLv3,
+    ssl.TLSVersion.MAXIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
+}
+
+# OpenSSL will only write 16K at a time
+SSL_WRITE_BLOCKSIZE = 16384
+
+orig_util_SSLContext = util.ssl_.SSLContext
+
+
+log = logging.getLogger(__name__)
+
+
+def inject_into_urllib3() -> None:
+    "Monkey-patch urllib3 with PyOpenSSL-backed SSL-support."
+
+    _validate_dependencies_met()
+
+    util.SSLContext = PyOpenSSLContext  # type: ignore[assignment]
+    util.ssl_.SSLContext = PyOpenSSLContext  # type: ignore[assignment]
+    util.IS_PYOPENSSL = True
+    util.ssl_.IS_PYOPENSSL = True
+
+
+def extract_from_urllib3() -> None:
+    "Undo monkey-patching by :func:`inject_into_urllib3`."
+
+    util.SSLContext = orig_util_SSLContext
+    util.ssl_.SSLContext = orig_util_SSLContext
+    util.IS_PYOPENSSL = False
+    util.ssl_.IS_PYOPENSSL = False
+
+
+def _validate_dependencies_met() -> None:
+    """
+    Verifies that PyOpenSSL's package-level dependencies have been met.
+    Throws `ImportError` if they are not met.
+    """
+    # Method added in `cryptography==1.1`; not available in older versions
+    from cryptography.x509.extensions import Extensions
+
+    if getattr(Extensions, "get_extension_for_class", None) is None:
+        raise ImportError(
+            "'cryptography' module missing required functionality.  "
+            "Try upgrading to v1.3.4 or newer."
+        )
+
+    # pyOpenSSL 0.14 and above use cryptography for OpenSSL bindings. The _x509
+    # attribute is only present on those versions.
+    from OpenSSL.crypto import X509
+
+    x509 = X509()
+    if getattr(x509, "_x509", None) is None:
+        raise ImportError(
+            "'pyOpenSSL' module missing required functionality. "
+            "Try upgrading to v0.14 or newer."
+        )
+
+
+def _dnsname_to_stdlib(name: str) -> str | None:
+    """
+    Converts a dNSName SubjectAlternativeName field to the form used by the
+    standard library on the given Python version.
+
+    Cryptography produces a dNSName as a unicode string that was idna-decoded
+    from ASCII bytes. We need to idna-encode that string to get it back, and
+    then on Python 3 we also need to convert to unicode via UTF-8 (the stdlib
+    uses PyUnicode_FromStringAndSize on it, which decodes via UTF-8).
+
+    If the name cannot be idna-encoded then we return None signalling that
+    the name given should be skipped.
+    """
+
+    def idna_encode(name: str) -> bytes | None:
+        """
+        Borrowed wholesale from the Python Cryptography Project. It turns out
+        that we can't just safely call `idna.encode`: it can explode for
+        wildcard names. This avoids that problem.
+        """
+        import idna
+
+        try:
+            for prefix in ["*.", "."]:
+                if name.startswith(prefix):
+                    name = name[len(prefix) :]
+                    return prefix.encode("ascii") + idna.encode(name)
+            return idna.encode(name)
+        except idna.core.IDNAError:
+            return None
+
+    # Don't send IPv6 addresses through the IDNA encoder.
+    if ":" in name:
+        return name
+
+    encoded_name = idna_encode(name)
+    if encoded_name is None:
+        return None
+    return encoded_name.decode("utf-8")
+
+
+def get_subj_alt_name(peer_cert: X509) -> list[tuple[str, str]]:
+    """
+    Given an PyOpenSSL certificate, provides all the subject alternative names.
+    """
+    cert = peer_cert.to_cryptography()
+
+    # We want to find the SAN extension. Ask Cryptography to locate it (it's
+    # faster than looping in Python)
+    try:
+        ext = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value
+    except x509.ExtensionNotFound:
+        # No such extension, return the empty list.
+        return []
+    except (
+        x509.DuplicateExtension,
+        UnsupportedExtension,
+        x509.UnsupportedGeneralNameType,
+        UnicodeError,
+    ) as e:
+        # A problem has been found with the quality of the certificate. Assume
+        # no SAN field is present.
+        log.warning(
+            "A problem was encountered with the certificate that prevented "
+            "urllib3 from finding the SubjectAlternativeName field. This can "
+            "affect certificate validation. The error was %s",
+            e,
+        )
+        return []
+
+    # We want to return dNSName and iPAddress fields. We need to cast the IPs
+    # back to strings because the match_hostname function wants them as
+    # strings.
+    # Sadly the DNS names need to be idna encoded and then, on Python 3, UTF-8
+    # decoded. This is pretty frustrating, but that's what the standard library
+    # does with certificates, and so we need to attempt to do the same.
+    # We also want to skip over names which cannot be idna encoded.
+    names = [
+        ("DNS", name)
+        for name in map(_dnsname_to_stdlib, ext.get_values_for_type(x509.DNSName))
+        if name is not None
+    ]
+    names.extend(
+        ("IP Address", str(name)) for name in ext.get_values_for_type(x509.IPAddress)
+    )
+
+    return names
+
+
+class WrappedSocket:
+    """API-compatibility wrapper for Python OpenSSL's Connection-class."""
+
+    def __init__(
+        self,
+        connection: OpenSSL.SSL.Connection,
+        socket: socket_cls,
+        suppress_ragged_eofs: bool = True,
+    ) -> None:
+        self.connection = connection
+        self.socket = socket
+        self.suppress_ragged_eofs = suppress_ragged_eofs
+        self._io_refs = 0
+        self._closed = False
+
+    def fileno(self) -> int:
+        return self.socket.fileno()
+
+    # Copy-pasted from Python 3.5 source code
+    def _decref_socketios(self) -> None:
+        if self._io_refs > 0:
+            self._io_refs -= 1
+        if self._closed:
+            self.close()
+
+    def recv(self, *args: typing.Any, **kwargs: typing.Any) -> bytes:
+        try:
+            data = self.connection.recv(*args, **kwargs)
+        except OpenSSL.SSL.SysCallError as e:
+            if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
+                return b""
+            else:
+                raise OSError(e.args[0], str(e)) from e
+        except OpenSSL.SSL.ZeroReturnError:
+            if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
+                return b""
+            else:
+                raise
+        except OpenSSL.SSL.WantReadError as e:
+            if not util.wait_for_read(self.socket, self.socket.gettimeout()):
+                raise timeout("The read operation timed out") from e
+            else:
+                return self.recv(*args, **kwargs)
+
+        # TLS 1.3 post-handshake authentication
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError(f"read error: {e!r}") from e
+        else:
+            return data  # type: ignore[no-any-return]
+
+    def recv_into(self, *args: typing.Any, **kwargs: typing.Any) -> int:
+        try:
+            return self.connection.recv_into(*args, **kwargs)  # type: ignore[no-any-return]
+        except OpenSSL.SSL.SysCallError as e:
+            if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
+                return 0
+            else:
+                raise OSError(e.args[0], str(e)) from e
+        except OpenSSL.SSL.ZeroReturnError:
+            if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
+                return 0
+            else:
+                raise
+        except OpenSSL.SSL.WantReadError as e:
+            if not util.wait_for_read(self.socket, self.socket.gettimeout()):
+                raise timeout("The read operation timed out") from e
+            else:
+                return self.recv_into(*args, **kwargs)
+
+        # TLS 1.3 post-handshake authentication
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError(f"read error: {e!r}") from e
+
+    def settimeout(self, timeout: float) -> None:
+        return self.socket.settimeout(timeout)
+
+    def _send_until_done(self, data: bytes) -> int:
+        while True:
+            try:
+                return self.connection.send(data)  # type: ignore[no-any-return]
+            except OpenSSL.SSL.WantWriteError as e:
+                if not util.wait_for_write(self.socket, self.socket.gettimeout()):
+                    raise timeout() from e
+                continue
+            except OpenSSL.SSL.SysCallError as e:
+                raise OSError(e.args[0], str(e)) from e
+
+    def sendall(self, data: bytes) -> None:
+        total_sent = 0
+        while total_sent < len(data):
+            sent = self._send_until_done(
+                data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]
+            )
+            total_sent += sent
+
+    def shutdown(self) -> None:
+        # FIXME rethrow compatible exceptions should we ever use this
+        self.connection.shutdown()
+
+    def close(self) -> None:
+        self._closed = True
+        if self._io_refs <= 0:
+            self._real_close()
+
+    def _real_close(self) -> None:
+        try:
+            return self.connection.close()  # type: ignore[no-any-return]
+        except OpenSSL.SSL.Error:
+            return
+
+    def getpeercert(
+        self, binary_form: bool = False
+    ) -> dict[str, list[typing.Any]] | None:
+        x509 = self.connection.get_peer_certificate()
+
+        if not x509:
+            return x509  # type: ignore[no-any-return]
+
+        if binary_form:
+            return OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_ASN1, x509)  # type: ignore[no-any-return]
+
+        return {
+            "subject": ((("commonName", x509.get_subject().CN),),),  # type: ignore[dict-item]
+            "subjectAltName": get_subj_alt_name(x509),
+        }
+
+    def version(self) -> str:
+        return self.connection.get_protocol_version_name()  # type: ignore[no-any-return]
+
+
+WrappedSocket.makefile = socket_cls.makefile  # type: ignore[attr-defined]
+
+
+class PyOpenSSLContext:
+    """
+    I am a wrapper class for the PyOpenSSL ``Context`` object. I am responsible
+    for translating the interface of the standard library ``SSLContext`` object
+    to calls into PyOpenSSL.
+    """
+
+    def __init__(self, protocol: int) -> None:
+        self.protocol = _openssl_versions[protocol]
+        self._ctx = OpenSSL.SSL.Context(self.protocol)
+        self._options = 0
+        self.check_hostname = False
+        self._minimum_version: int = ssl.TLSVersion.MINIMUM_SUPPORTED
+        self._maximum_version: int = ssl.TLSVersion.MAXIMUM_SUPPORTED
+
+    @property
+    def options(self) -> int:
+        return self._options
+
+    @options.setter
+    def options(self, value: int) -> None:
+        self._options = value
+        self._set_ctx_options()
+
+    @property
+    def verify_mode(self) -> int:
+        return _openssl_to_stdlib_verify[self._ctx.get_verify_mode()]
+
+    @verify_mode.setter
+    def verify_mode(self, value: ssl.VerifyMode) -> None:
+        self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
+
+    def set_default_verify_paths(self) -> None:
+        self._ctx.set_default_verify_paths()
+
+    def set_ciphers(self, ciphers: bytes | str) -> None:
+        if isinstance(ciphers, str):
+            ciphers = ciphers.encode("utf-8")
+        self._ctx.set_cipher_list(ciphers)
+
+    def load_verify_locations(
+        self,
+        cafile: str | None = None,
+        capath: str | None = None,
+        cadata: bytes | None = None,
+    ) -> None:
+        if cafile is not None:
+            cafile = cafile.encode("utf-8")  # type: ignore[assignment]
+        if capath is not None:
+            capath = capath.encode("utf-8")  # type: ignore[assignment]
+        try:
+            self._ctx.load_verify_locations(cafile, capath)
+            if cadata is not None:
+                self._ctx.load_verify_locations(BytesIO(cadata))
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError(f"unable to load trusted certificates: {e!r}") from e
+
+    def load_cert_chain(
+        self,
+        certfile: str,
+        keyfile: str | None = None,
+        password: str | None = None,
+    ) -> None:
+        try:
+            self._ctx.use_certificate_chain_file(certfile)
+            if password is not None:
+                if not isinstance(password, bytes):
+                    password = password.encode("utf-8")  # type: ignore[assignment]
+                self._ctx.set_passwd_cb(lambda *_: password)
+            self._ctx.use_privatekey_file(keyfile or certfile)
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError(f"Unable to load certificate chain: {e!r}") from e
+
+    def set_alpn_protocols(self, protocols: list[bytes | str]) -> None:
+        protocols = [util.util.to_bytes(p, "ascii") for p in protocols]
+        return self._ctx.set_alpn_protos(protocols)  # type: ignore[no-any-return]
+
+    def wrap_socket(
+        self,
+        sock: socket_cls,
+        server_side: bool = False,
+        do_handshake_on_connect: bool = True,
+        suppress_ragged_eofs: bool = True,
+        server_hostname: bytes | str | None = None,
+    ) -> WrappedSocket:
+        cnx = OpenSSL.SSL.Connection(self._ctx, sock)
+
+        # If server_hostname is an IP, don't use it for SNI, per RFC6066 Section 3
+        if server_hostname and not util.ssl_.is_ipaddress(server_hostname):
+            if isinstance(server_hostname, str):
+                server_hostname = server_hostname.encode("utf-8")
+            cnx.set_tlsext_host_name(server_hostname)
+
+        cnx.set_connect_state()
+
+        while True:
+            try:
+                cnx.do_handshake()
+            except OpenSSL.SSL.WantReadError as e:
+                if not util.wait_for_read(sock, sock.gettimeout()):
+                    raise timeout("select timed out") from e
+                continue
+            except OpenSSL.SSL.Error as e:
+                raise ssl.SSLError(f"bad handshake: {e!r}") from e
+            break
+
+        return WrappedSocket(cnx, sock)
+
+    def _set_ctx_options(self) -> None:
+        self._ctx.set_options(
+            self._options
+            | _openssl_to_ssl_minimum_version[self._minimum_version]
+            | _openssl_to_ssl_maximum_version[self._maximum_version]
+        )
+
+    @property
+    def minimum_version(self) -> int:
+        return self._minimum_version
+
+    @minimum_version.setter
+    def minimum_version(self, minimum_version: int) -> None:
+        self._minimum_version = minimum_version
+        self._set_ctx_options()
+
+    @property
+    def maximum_version(self) -> int:
+        return self._maximum_version
+
+    @maximum_version.setter
+    def maximum_version(self, maximum_version: int) -> None:
+        self._maximum_version = maximum_version
+        self._set_ctx_options()
+
+
+def _verify_callback(
+    cnx: OpenSSL.SSL.Connection,
+    x509: X509,
+    err_no: int,
+    err_depth: int,
+    return_code: int,
+) -> bool:
+    return err_no == 0

Modules/urllib3/contrib/socks.py

@@ -0,0 +1,230 @@
+"""
+This module contains provisional support for SOCKS proxies from within
+urllib3. This module supports SOCKS4, SOCKS4A (an extension of SOCKS4), and
+SOCKS5. To enable its functionality, either install PySocks or install this
+module with the ``socks`` extra.
+
+The SOCKS implementation supports the full range of urllib3 features. It also
+supports the following SOCKS features:
+
+- SOCKS4A (``proxy_url='socks4a://...``)
+- SOCKS4 (``proxy_url='socks4://...``)
+- SOCKS5 with remote DNS (``proxy_url='socks5h://...``)
+- SOCKS5 with local DNS (``proxy_url='socks5://...``)
+- Usernames and passwords for the SOCKS proxy
+
+.. note::
+   It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in
+   your ``proxy_url`` to ensure that DNS resolution is done from the remote
+   server instead of client-side when connecting to a domain name.
+
+SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5
+supports IPv4, IPv6, and domain names.
+
+When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url``
+will be sent as the ``userid`` section of the SOCKS request:
+
+.. code-block:: python
+
+    proxy_url="socks4a://<userid>@proxy-host"
+
+When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion
+of the ``proxy_url`` will be sent as the username/password to authenticate
+with the proxy:
+
+.. code-block:: python
+
+    proxy_url="socks5h://<username>:<password>@proxy-host"
+
+"""
+
+from __future__ import annotations
+
+try:
+    import socks  # type: ignore[import]
+except ImportError:
+    import warnings
+
+    from ..exceptions import DependencyWarning
+
+    warnings.warn(
+        (
+            "SOCKS support in urllib3 requires the installation of optional "
+            "dependencies: specifically, PySocks.  For more information, see "
+            "https://urllib3.readthedocs.io/en/latest/contrib.html#socks-proxies"
+        ),
+        DependencyWarning,
+    )
+    raise
+
+import typing
+from socket import timeout as SocketTimeout
+
+from ..connection import HTTPConnection, HTTPSConnection
+from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool
+from ..exceptions import ConnectTimeoutError, NewConnectionError
+from ..poolmanager import PoolManager
+from ..util.url import parse_url
+
+try:
+    import ssl
+except ImportError:
+    ssl = None  # type: ignore[assignment]
+
+from typing import TypedDict
+
+
+class _TYPE_SOCKS_OPTIONS(TypedDict):
+    socks_version: int
+    proxy_host: str | None
+    proxy_port: str | None
+    username: str | None
+    password: str | None
+    rdns: bool
+
+
+class SOCKSConnection(HTTPConnection):
+    """
+    A plain-text HTTP connection that connects via a SOCKS proxy.
+    """
+
+    def __init__(
+        self,
+        _socks_options: _TYPE_SOCKS_OPTIONS,
+        *args: typing.Any,
+        **kwargs: typing.Any,
+    ) -> None:
+        self._socks_options = _socks_options
+        super().__init__(*args, **kwargs)
+
+    def _new_conn(self) -> socks.socksocket:
+        """
+        Establish a new connection via the SOCKS proxy.
+        """
+        extra_kw: dict[str, typing.Any] = {}
+        if self.source_address:
+            extra_kw["source_address"] = self.source_address
+
+        if self.socket_options:
+            extra_kw["socket_options"] = self.socket_options
+
+        try:
+            conn = socks.create_connection(
+                (self.host, self.port),
+                proxy_type=self._socks_options["socks_version"],
+                proxy_addr=self._socks_options["proxy_host"],
+                proxy_port=self._socks_options["proxy_port"],
+                proxy_username=self._socks_options["username"],
+                proxy_password=self._socks_options["password"],
+                proxy_rdns=self._socks_options["rdns"],
+                timeout=self.timeout,
+                **extra_kw,
+            )
+
+        except SocketTimeout as e:
+            raise ConnectTimeoutError(
+                self,
+                f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
+            ) from e
+
+        except socks.ProxyError as e:
+            # This is fragile as hell, but it seems to be the only way to raise
+            # useful errors here.
+            if e.socket_err:
+                error = e.socket_err
+                if isinstance(error, SocketTimeout):
+                    raise ConnectTimeoutError(
+                        self,
+                        f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
+                    ) from e
+                else:
+                    # Adding `from e` messes with coverage somehow, so it's omitted.
+                    # See #2386.
+                    raise NewConnectionError(
+                        self, f"Failed to establish a new connection: {error}"
+                    )
+            else:
+                raise NewConnectionError(
+                    self, f"Failed to establish a new connection: {e}"
+                ) from e
+
+        except OSError as e:  # Defensive: PySocks should catch all these.
+            raise NewConnectionError(
+                self, f"Failed to establish a new connection: {e}"
+            ) from e
+
+        return conn
+
+
+# We don't need to duplicate the Verified/Unverified distinction from
+# urllib3/connection.py here because the HTTPSConnection will already have been
+# correctly set to either the Verified or Unverified form by that module. This
+# means the SOCKSHTTPSConnection will automatically be the correct type.
+class SOCKSHTTPSConnection(SOCKSConnection, HTTPSConnection):
+    pass
+
+
+class SOCKSHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = SOCKSConnection
+
+
+class SOCKSHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = SOCKSHTTPSConnection
+
+
+class SOCKSProxyManager(PoolManager):
+    """
+    A version of the urllib3 ProxyManager that routes connections via the
+    defined SOCKS proxy.
+    """
+
+    pool_classes_by_scheme = {
+        "http": SOCKSHTTPConnectionPool,
+        "https": SOCKSHTTPSConnectionPool,
+    }
+
+    def __init__(
+        self,
+        proxy_url: str,
+        username: str | None = None,
+        password: str | None = None,
+        num_pools: int = 10,
+        headers: typing.Mapping[str, str] | None = None,
+        **connection_pool_kw: typing.Any,
+    ):
+        parsed = parse_url(proxy_url)
+
+        if username is None and password is None and parsed.auth is not None:
+            split = parsed.auth.split(":")
+            if len(split) == 2:
+                username, password = split
+        if parsed.scheme == "socks5":
+            socks_version = socks.PROXY_TYPE_SOCKS5
+            rdns = False
+        elif parsed.scheme == "socks5h":
+            socks_version = socks.PROXY_TYPE_SOCKS5
+            rdns = True
+        elif parsed.scheme == "socks4":
+            socks_version = socks.PROXY_TYPE_SOCKS4
+            rdns = False
+        elif parsed.scheme == "socks4a":
+            socks_version = socks.PROXY_TYPE_SOCKS4
+            rdns = True
+        else:
+            raise ValueError(f"Unable to determine SOCKS version from {proxy_url}")
+
+        self.proxy_url = proxy_url
+
+        socks_options = {
+            "socks_version": socks_version,
+            "proxy_host": parsed.host,
+            "proxy_port": parsed.port,
+            "username": username,
+            "password": password,
+            "rdns": rdns,
+        }
+        connection_pool_kw["_socks_options"] = socks_options
+
+        super().__init__(num_pools, headers, **connection_pool_kw)
+
+        self.pool_classes_by_scheme = SOCKSProxyManager.pool_classes_by_scheme

Modules/urllib3/exceptions.py

@@ -0,0 +1,318 @@
+from __future__ import annotations
+
+import socket
+import typing
+import warnings
+from email.errors import MessageDefect
+from http.client import IncompleteRead as httplib_IncompleteRead
+
+if typing.TYPE_CHECKING:
+    from .connection import HTTPConnection
+    from .connectionpool import ConnectionPool
+    from .response import HTTPResponse
+    from .util.retry import Retry
+
+# Base Exceptions
+
+
+class HTTPError(Exception):
+    """Base exception used by this module."""
+
+
+class HTTPWarning(Warning):
+    """Base warning used by this module."""
+
+
+_TYPE_REDUCE_RESULT = typing.Tuple[
+    typing.Callable[..., object], typing.Tuple[object, ...]
+]
+
+
+class PoolError(HTTPError):
+    """Base exception for errors caused within a pool."""
+
+    def __init__(self, pool: ConnectionPool, message: str) -> None:
+        self.pool = pool
+        super().__init__(f"{pool}: {message}")
+
+    def __reduce__(self) -> _TYPE_REDUCE_RESULT:
+        # For pickling purposes.
+        return self.__class__, (None, None)
+
+
+class RequestError(PoolError):
+    """Base exception for PoolErrors that have associated URLs."""
+
+    def __init__(self, pool: ConnectionPool, url: str, message: str) -> None:
+        self.url = url
+        super().__init__(pool, message)
+
+    def __reduce__(self) -> _TYPE_REDUCE_RESULT:
+        # For pickling purposes.
+        return self.__class__, (None, self.url, None)
+
+
+class SSLError(HTTPError):
+    """Raised when SSL certificate fails in an HTTPS connection."""
+
+
+class ProxyError(HTTPError):
+    """Raised when the connection to a proxy fails."""
+
+    # The original error is also available as __cause__.
+    original_error: Exception
+
+    def __init__(self, message: str, error: Exception) -> None:
+        super().__init__(message, error)
+        self.original_error = error
+
+
+class DecodeError(HTTPError):
+    """Raised when automatic decoding based on Content-Type fails."""
+
+
+class ProtocolError(HTTPError):
+    """Raised when something unexpected happens mid-request/response."""
+
+
+#: Renamed to ProtocolError but aliased for backwards compatibility.
+ConnectionError = ProtocolError
+
+
+# Leaf Exceptions
+
+
+class MaxRetryError(RequestError):
+    """Raised when the maximum number of retries is exceeded.
+
+    :param pool: The connection pool
+    :type pool: :class:`~urllib3.connectionpool.HTTPConnectionPool`
+    :param str url: The requested Url
+    :param reason: The underlying error
+    :type reason: :class:`Exception`
+
+    """
+
+    def __init__(
+        self, pool: ConnectionPool, url: str, reason: Exception | None = None
+    ) -> None:
+        self.reason = reason
+
+        message = f"Max retries exceeded with url: {url} (Caused by {reason!r})"
+
+        super().__init__(pool, url, message)
+
+
+class HostChangedError(RequestError):
+    """Raised when an existing pool gets a request for a foreign host."""
+
+    def __init__(
+        self, pool: ConnectionPool, url: str, retries: Retry | int = 3
+    ) -> None:
+        message = f"Tried to open a foreign host with url: {url}"
+        super().__init__(pool, url, message)
+        self.retries = retries
+
+
+class TimeoutStateError(HTTPError):
+    """Raised when passing an invalid state to a timeout"""
+
+
+class TimeoutError(HTTPError):
+    """Raised when a socket timeout error occurs.
+
+    Catching this error will catch both :exc:`ReadTimeoutErrors
+    <ReadTimeoutError>` and :exc:`ConnectTimeoutErrors <ConnectTimeoutError>`.
+    """
+
+
+class ReadTimeoutError(TimeoutError, RequestError):
+    """Raised when a socket timeout occurs while receiving data from a server"""
+
+
+# This timeout error does not have a URL attached and needs to inherit from the
+# base HTTPError
+class ConnectTimeoutError(TimeoutError):
+    """Raised when a socket timeout occurs while connecting to a server"""
+
+
+class NewConnectionError(ConnectTimeoutError, HTTPError):
+    """Raised when we fail to establish a new connection. Usually ECONNREFUSED."""
+
+    def __init__(self, conn: HTTPConnection, message: str) -> None:
+        self.conn = conn
+        super().__init__(f"{conn}: {message}")
+
+    @property
+    def pool(self) -> HTTPConnection:
+        warnings.warn(
+            "The 'pool' property is deprecated and will be removed "
+            "in urllib3 v2.1.0. Use 'conn' instead.",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+
+        return self.conn
+
+
+class NameResolutionError(NewConnectionError):
+    """Raised when host name resolution fails."""
+
+    def __init__(self, host: str, conn: HTTPConnection, reason: socket.gaierror):
+        message = f"Failed to resolve '{host}' ({reason})"
+        super().__init__(conn, message)
+
+
+class EmptyPoolError(PoolError):
+    """Raised when a pool runs out of connections and no more are allowed."""
+
+
+class FullPoolError(PoolError):
+    """Raised when we try to add a connection to a full pool in blocking mode."""
+
+
+class ClosedPoolError(PoolError):
+    """Raised when a request enters a pool after the pool has been closed."""
+
+
+class LocationValueError(ValueError, HTTPError):
+    """Raised when there is something wrong with a given URL input."""
+
+
+class LocationParseError(LocationValueError):
+    """Raised when get_host or similar fails to parse the URL input."""
+
+    def __init__(self, location: str) -> None:
+        message = f"Failed to parse: {location}"
+        super().__init__(message)
+
+        self.location = location
+
+
+class URLSchemeUnknown(LocationValueError):
+    """Raised when a URL input has an unsupported scheme."""
+
+    def __init__(self, scheme: str):
+        message = f"Not supported URL scheme {scheme}"
+        super().__init__(message)
+
+        self.scheme = scheme
+
+
+class ResponseError(HTTPError):
+    """Used as a container for an error reason supplied in a MaxRetryError."""
+
+    GENERIC_ERROR = "too many error responses"
+    SPECIFIC_ERROR = "too many {status_code} error responses"
+
+
+class SecurityWarning(HTTPWarning):
+    """Warned when performing security reducing actions"""
+
+
+class InsecureRequestWarning(SecurityWarning):
+    """Warned when making an unverified HTTPS request."""
+
+
+class NotOpenSSLWarning(SecurityWarning):
+    """Warned when using unsupported SSL library"""
+
+
+class SystemTimeWarning(SecurityWarning):
+    """Warned when system time is suspected to be wrong"""
+
+
+class InsecurePlatformWarning(SecurityWarning):
+    """Warned when certain TLS/SSL configuration is not available on a platform."""
+
+
+class DependencyWarning(HTTPWarning):
+    """
+    Warned when an attempt is made to import a module with missing optional
+    dependencies.
+    """
+
+
+class ResponseNotChunked(ProtocolError, ValueError):
+    """Response needs to be chunked in order to read it as chunks."""
+
+
+class BodyNotHttplibCompatible(HTTPError):
+    """
+    Body should be :class:`http.client.HTTPResponse` like
+    (have an fp attribute which returns raw chunks) for read_chunked().
+    """
+
+
+class IncompleteRead(HTTPError, httplib_IncompleteRead):
+    """
+    Response length doesn't match expected Content-Length
+
+    Subclass of :class:`http.client.IncompleteRead` to allow int value
+    for ``partial`` to avoid creating large objects on streamed reads.
+    """
+
+    def __init__(self, partial: int, expected: int) -> None:
+        self.partial = partial  # type: ignore[assignment]
+        self.expected = expected
+
+    def __repr__(self) -> str:
+        return "IncompleteRead(%i bytes read, %i more expected)" % (
+            self.partial,  # type: ignore[str-format]
+            self.expected,
+        )
+
+
+class InvalidChunkLength(HTTPError, httplib_IncompleteRead):
+    """Invalid chunk length in a chunked response."""
+
+    def __init__(self, response: HTTPResponse, length: bytes) -> None:
+        self.partial: int = response.tell()  # type: ignore[assignment]
+        self.expected: int | None = response.length_remaining
+        self.response = response
+        self.length = length
+
+    def __repr__(self) -> str:
+        return "InvalidChunkLength(got length %r, %i bytes read)" % (
+            self.length,
+            self.partial,
+        )
+
+
+class InvalidHeader(HTTPError):
+    """The header provided was somehow invalid."""
+
+
+class ProxySchemeUnknown(AssertionError, URLSchemeUnknown):
+    """ProxyManager does not support the supplied scheme"""
+
+    # TODO(t-8ch): Stop inheriting from AssertionError in v2.0.
+
+    def __init__(self, scheme: str | None) -> None:
+        # 'localhost' is here because our URL parser parses
+        # localhost:8080 -> scheme=localhost, remove if we fix this.
+        if scheme == "localhost":
+            scheme = None
+        if scheme is None:
+            message = "Proxy URL had no scheme, should start with http:// or https://"
+        else:
+            message = f"Proxy URL had unsupported scheme {scheme}, should use http:// or https://"
+        super().__init__(message)
+
+
+class ProxySchemeUnsupported(ValueError):
+    """Fetching HTTPS resources through HTTPS proxies is unsupported"""
+
+
+class HeaderParsingError(HTTPError):
+    """Raised by assert_header_parsing, but we convert it to a log.warning statement."""
+
+    def __init__(
+        self, defects: list[MessageDefect], unparsed_data: bytes | str | None
+    ) -> None:
+        message = f"{defects or 'Unknown'}, unparsed data: {unparsed_data!r}"
+        super().__init__(message)
+
+
+class UnrewindableBodyError(HTTPError):
+    """urllib3 encountered an error when trying to rewind a body"""

Modules/urllib3/fields.py

@@ -0,0 +1,345 @@
+from __future__ import annotations
+
+import email.utils
+import mimetypes
+import typing
+
+_TYPE_FIELD_VALUE = typing.Union[str, bytes]
+_TYPE_FIELD_VALUE_TUPLE = typing.Union[
+    _TYPE_FIELD_VALUE,
+    typing.Tuple[str, _TYPE_FIELD_VALUE],
+    typing.Tuple[str, _TYPE_FIELD_VALUE, str],
+]
+
+
+def guess_content_type(
+    filename: str | None, default: str = "application/octet-stream"
+) -> str:
+    """
+    Guess the "Content-Type" of a file.
+
+    :param filename:
+        The filename to guess the "Content-Type" of using :mod:`mimetypes`.
+    :param default:
+        If no "Content-Type" can be guessed, default to `default`.
+    """
+    if filename:
+        return mimetypes.guess_type(filename)[0] or default
+    return default
+
+
+def format_header_param_rfc2231(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    Helper function to format and quote a single header parameter using the
+    strategy defined in RFC 2231.
+
+    Particularly useful for header parameters which might contain
+    non-ASCII values, like file names. This follows
+    `RFC 2388 Section 4.4 <https://tools.ietf.org/html/rfc2388#section-4.4>`_.
+
+    :param name:
+        The name of the parameter, a string expected to be ASCII only.
+    :param value:
+        The value of the parameter, provided as ``bytes`` or `str``.
+    :returns:
+        An RFC-2231-formatted unicode string.
+
+    .. deprecated:: 2.0.0
+        Will be removed in urllib3 v2.1.0. This is not valid for
+        ``multipart/form-data`` header parameters.
+    """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param_rfc2231' is deprecated and will be "
+        "removed in urllib3 v2.1.0. This is not valid for "
+        "multipart/form-data header parameters.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+
+    if isinstance(value, bytes):
+        value = value.decode("utf-8")
+
+    if not any(ch in value for ch in '"\\\r\n'):
+        result = f'{name}="{value}"'
+        try:
+            result.encode("ascii")
+        except (UnicodeEncodeError, UnicodeDecodeError):
+            pass
+        else:
+            return result
+
+    value = email.utils.encode_rfc2231(value, "utf-8")
+    value = f"{name}*={value}"
+
+    return value
+
+
+def format_multipart_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    Format and quote a single multipart header parameter.
+
+    This follows the `WHATWG HTML Standard`_ as of 2021/06/10, matching
+    the behavior of current browser and curl versions. Values are
+    assumed to be UTF-8. The ``\\n``, ``\\r``, and ``"`` characters are
+    percent encoded.
+
+    .. _WHATWG HTML Standard:
+        https://html.spec.whatwg.org/multipage/
+        form-control-infrastructure.html#multipart-form-data
+
+    :param name:
+        The name of the parameter, an ASCII-only ``str``.
+    :param value:
+        The value of the parameter, a ``str`` or UTF-8 encoded
+        ``bytes``.
+    :returns:
+        A string ``name="value"`` with the escaped value.
+
+    .. versionchanged:: 2.0.0
+        Matches the WHATWG HTML Standard as of 2021/06/10. Control
+        characters are no longer percent encoded.
+
+    .. versionchanged:: 2.0.0
+        Renamed from ``format_header_param_html5`` and
+        ``format_header_param``. The old names will be removed in
+        urllib3 v2.1.0.
+    """
+    if isinstance(value, bytes):
+        value = value.decode("utf-8")
+
+    # percent encode \n \r "
+    value = value.translate({10: "%0A", 13: "%0D", 34: "%22"})
+    return f'{name}="{value}"'
+
+
+def format_header_param_html5(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    .. deprecated:: 2.0.0
+        Renamed to :func:`format_multipart_header_param`. Will be
+        removed in urllib3 v2.1.0.
+    """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param_html5' has been renamed to "
+        "'format_multipart_header_param'. The old name will be "
+        "removed in urllib3 v2.1.0.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return format_multipart_header_param(name, value)
+
+
+def format_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    .. deprecated:: 2.0.0
+        Renamed to :func:`format_multipart_header_param`. Will be
+        removed in urllib3 v2.1.0.
+    """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param' has been renamed to "
+        "'format_multipart_header_param'. The old name will be "
+        "removed in urllib3 v2.1.0.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return format_multipart_header_param(name, value)
+
+
+class RequestField:
+    """
+    A data container for request body parameters.
+
+    :param name:
+        The name of this request field. Must be unicode.
+    :param data:
+        The data/value body.
+    :param filename:
+        An optional filename of the request field. Must be unicode.
+    :param headers:
+        An optional dict-like object of headers to initially use for the field.
+
+    .. versionchanged:: 2.0.0
+        The ``header_formatter`` parameter is deprecated and will
+        be removed in urllib3 v2.1.0.
+    """
+
+    def __init__(
+        self,
+        name: str,
+        data: _TYPE_FIELD_VALUE,
+        filename: str | None = None,
+        headers: typing.Mapping[str, str] | None = None,
+        header_formatter: typing.Callable[[str, _TYPE_FIELD_VALUE], str] | None = None,
+    ):
+        self._name = name
+        self._filename = filename
+        self.data = data
+        self.headers: dict[str, str | None] = {}
+        if headers:
+            self.headers = dict(headers)
+
+        if header_formatter is not None:
+            import warnings
+
+            warnings.warn(
+                "The 'header_formatter' parameter is deprecated and "
+                "will be removed in urllib3 v2.1.0.",
+                DeprecationWarning,
+                stacklevel=2,
+            )
+            self.header_formatter = header_formatter
+        else:
+            self.header_formatter = format_multipart_header_param
+
+    @classmethod
+    def from_tuples(
+        cls,
+        fieldname: str,
+        value: _TYPE_FIELD_VALUE_TUPLE,
+        header_formatter: typing.Callable[[str, _TYPE_FIELD_VALUE], str] | None = None,
+    ) -> RequestField:
+        """
+        A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters.
+
+        Supports constructing :class:`~urllib3.fields.RequestField` from
+        parameter of key/value strings AND key/filetuple. A filetuple is a
+        (filename, data, MIME type) tuple where the MIME type is optional.
+        For example::
+
+            'foo': 'bar',
+            'fakefile': ('foofile.txt', 'contents of foofile'),
+            'realfile': ('barfile.txt', open('realfile').read()),
+            'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'),
+            'nonamefile': 'contents of nonamefile field',
+
+        Field names and filenames must be unicode.
+        """
+        filename: str | None
+        content_type: str | None
+        data: _TYPE_FIELD_VALUE
+
+        if isinstance(value, tuple):
+            if len(value) == 3:
+                filename, data, content_type = typing.cast(
+                    typing.Tuple[str, _TYPE_FIELD_VALUE, str], value
+                )
+            else:
+                filename, data = typing.cast(
+                    typing.Tuple[str, _TYPE_FIELD_VALUE], value
+                )
+                content_type = guess_content_type(filename)
+        else:
+            filename = None
+            content_type = None
+            data = value
+
+        request_param = cls(
+            fieldname, data, filename=filename, header_formatter=header_formatter
+        )
+        request_param.make_multipart(content_type=content_type)
+
+        return request_param
+
+    def _render_part(self, name: str, value: _TYPE_FIELD_VALUE) -> str:
+        """
+        Override this method to change how each multipart header
+        parameter is formatted. By default, this calls
+        :func:`format_multipart_header_param`.
+
+        :param name:
+            The name of the parameter, an ASCII-only ``str``.
+        :param value:
+            The value of the parameter, a ``str`` or UTF-8 encoded
+            ``bytes``.
+
+        :meta public:
+        """
+        return self.header_formatter(name, value)
+
+    def _render_parts(
+        self,
+        header_parts: (
+            dict[str, _TYPE_FIELD_VALUE | None]
+            | typing.Sequence[tuple[str, _TYPE_FIELD_VALUE | None]]
+        ),
+    ) -> str:
+        """
+        Helper function to format and quote a single header.
+
+        Useful for single headers that are composed of multiple items. E.g.,
+        'Content-Disposition' fields.
+
+        :param header_parts:
+            A sequence of (k, v) tuples or a :class:`dict` of (k, v) to format
+            as `k1="v1"; k2="v2"; ...`.
+        """
+        iterable: typing.Iterable[tuple[str, _TYPE_FIELD_VALUE | None]]
+
+        parts = []
+        if isinstance(header_parts, dict):
+            iterable = header_parts.items()
+        else:
+            iterable = header_parts
+
+        for name, value in iterable:
+            if value is not None:
+                parts.append(self._render_part(name, value))
+
+        return "; ".join(parts)
+
+    def render_headers(self) -> str:
+        """
+        Renders the headers for this request field.
+        """
+        lines = []
+
+        sort_keys = ["Content-Disposition", "Content-Type", "Content-Location"]
+        for sort_key in sort_keys:
+            if self.headers.get(sort_key, False):
+                lines.append(f"{sort_key}: {self.headers[sort_key]}")
+
+        for header_name, header_value in self.headers.items():
+            if header_name not in sort_keys:
+                if header_value:
+                    lines.append(f"{header_name}: {header_value}")
+
+        lines.append("\r\n")
+        return "\r\n".join(lines)
+
+    def make_multipart(
+        self,
+        content_disposition: str | None = None,
+        content_type: str | None = None,
+        content_location: str | None = None,
+    ) -> None:
+        """
+        Makes this request field into a multipart request field.
+
+        This method overrides "Content-Disposition", "Content-Type" and
+        "Content-Location" headers to the request parameter.
+
+        :param content_disposition:
+            The 'Content-Disposition' of the request body. Defaults to 'form-data'
+        :param content_type:
+            The 'Content-Type' of the request body.
+        :param content_location:
+            The 'Content-Location' of the request body.
+
+        """
+        content_disposition = (content_disposition or "form-data") + "; ".join(
+            [
+                "",
+                self._render_parts(
+                    (("name", self._name), ("filename", self._filename))
+                ),
+            ]
+        )
+
+        self.headers["Content-Disposition"] = content_disposition
+        self.headers["Content-Type"] = content_type
+        self.headers["Content-Location"] = content_location

Modules/urllib3/filepost.py

@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+import binascii
+import codecs
+import os
+import typing
+from io import BytesIO
+
+from .fields import _TYPE_FIELD_VALUE_TUPLE, RequestField
+
+writer = codecs.lookup("utf-8")[3]
+
+_TYPE_FIELDS_SEQUENCE = typing.Sequence[
+    typing.Union[typing.Tuple[str, _TYPE_FIELD_VALUE_TUPLE], RequestField]
+]
+_TYPE_FIELDS = typing.Union[
+    _TYPE_FIELDS_SEQUENCE,
+    typing.Mapping[str, _TYPE_FIELD_VALUE_TUPLE],
+]
+
+
+def choose_boundary() -> str:
+    """
+    Our embarrassingly-simple replacement for mimetools.choose_boundary.
+    """
+    return binascii.hexlify(os.urandom(16)).decode()
+
+
+def iter_field_objects(fields: _TYPE_FIELDS) -> typing.Iterable[RequestField]:
+    """
+    Iterate over fields.
+
+    Supports list of (k, v) tuples and dicts, and lists of
+    :class:`~urllib3.fields.RequestField`.
+
+    """
+    iterable: typing.Iterable[RequestField | tuple[str, _TYPE_FIELD_VALUE_TUPLE]]
+
+    if isinstance(fields, typing.Mapping):
+        iterable = fields.items()
+    else:
+        iterable = fields
+
+    for field in iterable:
+        if isinstance(field, RequestField):
+            yield field
+        else:
+            yield RequestField.from_tuples(*field)
+
+
+def encode_multipart_formdata(
+    fields: _TYPE_FIELDS, boundary: str | None = None
+) -> tuple[bytes, str]:
+    """
+    Encode a dictionary of ``fields`` using the multipart/form-data MIME format.
+
+    :param fields:
+        Dictionary of fields or list of (key, :class:`~urllib3.fields.RequestField`).
+        Values are processed by :func:`urllib3.fields.RequestField.from_tuples`.
+
+    :param boundary:
+        If not specified, then a random boundary will be generated using
+        :func:`urllib3.filepost.choose_boundary`.
+    """
+    body = BytesIO()
+    if boundary is None:
+        boundary = choose_boundary()
+
+    for field in iter_field_objects(fields):
+        body.write(f"--{boundary}\r\n".encode("latin-1"))
+
+        writer(body).write(field.render_headers())
+        data = field.data
+
+        if isinstance(data, int):
+            data = str(data)  # Backwards compatibility
+
+        if isinstance(data, str):
+            writer(body).write(data)
+        else:
+            body.write(data)
+
+        body.write(b"\r\n")
+
+    body.write(f"--{boundary}--\r\n".encode("latin-1"))
+
+    content_type = f"multipart/form-data; boundary={boundary}"
+
+    return body.getvalue(), content_type

Modules/urllib3/poolmanager.py

@@ -0,0 +1,638 @@
+from __future__ import annotations
+
+import functools
+import logging
+import typing
+import warnings
+from types import TracebackType
+from urllib.parse import urljoin
+
+from ._collections import HTTPHeaderDict, RecentlyUsedContainer
+from ._request_methods import RequestMethods
+from .connection import ProxyConfig
+from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, port_by_scheme
+from .exceptions import (
+    LocationValueError,
+    MaxRetryError,
+    ProxySchemeUnknown,
+    URLSchemeUnknown,
+)
+from .response import BaseHTTPResponse
+from .util.connection import _TYPE_SOCKET_OPTIONS
+from .util.proxy import connection_requires_http_tunnel
+from .util.retry import Retry
+from .util.timeout import Timeout
+from .util.url import Url, parse_url
+
+if typing.TYPE_CHECKING:
+    import ssl
+    from typing import Literal
+
+__all__ = ["PoolManager", "ProxyManager", "proxy_from_url"]
+
+
+log = logging.getLogger(__name__)
+
+SSL_KEYWORDS = (
+    "key_file",
+    "cert_file",
+    "cert_reqs",
+    "ca_certs",
+    "ca_cert_data",
+    "ssl_version",
+    "ssl_minimum_version",
+    "ssl_maximum_version",
+    "ca_cert_dir",
+    "ssl_context",
+    "key_password",
+    "server_hostname",
+)
+# Default value for `blocksize` - a new parameter introduced to
+# http.client.HTTPConnection & http.client.HTTPSConnection in Python 3.7
+_DEFAULT_BLOCKSIZE = 16384
+
+_SelfT = typing.TypeVar("_SelfT")
+
+
+class PoolKey(typing.NamedTuple):
+    """
+    All known keyword arguments that could be provided to the pool manager, its
+    pools, or the underlying connections.
+
+    All custom key schemes should include the fields in this key at a minimum.
+    """
+
+    key_scheme: str
+    key_host: str
+    key_port: int | None
+    key_timeout: Timeout | float | int | None
+    key_retries: Retry | bool | int | None
+    key_block: bool | None
+    key_source_address: tuple[str, int] | None
+    key_key_file: str | None
+    key_key_password: str | None
+    key_cert_file: str | None
+    key_cert_reqs: str | None
+    key_ca_certs: str | None
+    key_ca_cert_data: str | bytes | None
+    key_ssl_version: int | str | None
+    key_ssl_minimum_version: ssl.TLSVersion | None
+    key_ssl_maximum_version: ssl.TLSVersion | None
+    key_ca_cert_dir: str | None
+    key_ssl_context: ssl.SSLContext | None
+    key_maxsize: int | None
+    key_headers: frozenset[tuple[str, str]] | None
+    key__proxy: Url | None
+    key__proxy_headers: frozenset[tuple[str, str]] | None
+    key__proxy_config: ProxyConfig | None
+    key_socket_options: _TYPE_SOCKET_OPTIONS | None
+    key__socks_options: frozenset[tuple[str, str]] | None
+    key_assert_hostname: bool | str | None
+    key_assert_fingerprint: str | None
+    key_server_hostname: str | None
+    key_blocksize: int | None
+
+
+def _default_key_normalizer(
+    key_class: type[PoolKey], request_context: dict[str, typing.Any]
+) -> PoolKey:
+    """
+    Create a pool key out of a request context dictionary.
+
+    According to RFC 3986, both the scheme and host are case-insensitive.
+    Therefore, this function normalizes both before constructing the pool
+    key for an HTTPS request. If you wish to change this behaviour, provide
+    alternate callables to ``key_fn_by_scheme``.
+
+    :param key_class:
+        The class to use when constructing the key. This should be a namedtuple
+        with the ``scheme`` and ``host`` keys at a minimum.
+    :type  key_class: namedtuple
+    :param request_context:
+        A dictionary-like object that contain the context for a request.
+    :type  request_context: dict
+
+    :return: A namedtuple that can be used as a connection pool key.
+    :rtype:  PoolKey
+    """
+    # Since we mutate the dictionary, make a copy first
+    context = request_context.copy()
+    context["scheme"] = context["scheme"].lower()
+    context["host"] = context["host"].lower()
+
+    # These are both dictionaries and need to be transformed into frozensets
+    for key in ("headers", "_proxy_headers", "_socks_options"):
+        if key in context and context[key] is not None:
+            context[key] = frozenset(context[key].items())
+
+    # The socket_options key may be a list and needs to be transformed into a
+    # tuple.
+    socket_opts = context.get("socket_options")
+    if socket_opts is not None:
+        context["socket_options"] = tuple(socket_opts)
+
+    # Map the kwargs to the names in the namedtuple - this is necessary since
+    # namedtuples can't have fields starting with '_'.
+    for key in list(context.keys()):
+        context["key_" + key] = context.pop(key)
+
+    # Default to ``None`` for keys missing from the context
+    for field in key_class._fields:
+        if field not in context:
+            context[field] = None
+
+    # Default key_blocksize to _DEFAULT_BLOCKSIZE if missing from the context
+    if context.get("key_blocksize") is None:
+        context["key_blocksize"] = _DEFAULT_BLOCKSIZE
+
+    return key_class(**context)
+
+
+#: A dictionary that maps a scheme to a callable that creates a pool key.
+#: This can be used to alter the way pool keys are constructed, if desired.
+#: Each PoolManager makes a copy of this dictionary so they can be configured
+#: globally here, or individually on the instance.
+key_fn_by_scheme = {
+    "http": functools.partial(_default_key_normalizer, PoolKey),
+    "https": functools.partial(_default_key_normalizer, PoolKey),
+}
+
+pool_classes_by_scheme = {"http": HTTPConnectionPool, "https": HTTPSConnectionPool}
+
+
+class PoolManager(RequestMethods):
+    """
+    Allows for arbitrary requests while transparently keeping track of
+    necessary connection pools for you.
+
+    :param num_pools:
+        Number of connection pools to cache before discarding the least
+        recently used pool.
+
+    :param headers:
+        Headers to include with all requests, unless other headers are given
+        explicitly.
+
+    :param \\**connection_pool_kw:
+        Additional parameters are used to create fresh
+        :class:`urllib3.connectionpool.ConnectionPool` instances.
+
+    Example:
+
+    .. code-block:: python
+
+        import urllib3
+
+        http = urllib3.PoolManager(num_pools=2)
+
+        resp1 = http.request("GET", "https://google.com/")
+        resp2 = http.request("GET", "https://google.com/mail")
+        resp3 = http.request("GET", "https://yahoo.com/")
+
+        print(len(http.pools))
+        # 2
+
+    """
+
+    proxy: Url | None = None
+    proxy_config: ProxyConfig | None = None
+
+    def __init__(
+        self,
+        num_pools: int = 10,
+        headers: typing.Mapping[str, str] | None = None,
+        **connection_pool_kw: typing.Any,
+    ) -> None:
+        super().__init__(headers)
+        self.connection_pool_kw = connection_pool_kw
+
+        self.pools: RecentlyUsedContainer[PoolKey, HTTPConnectionPool]
+        self.pools = RecentlyUsedContainer(num_pools)
+
+        # Locally set the pool classes and keys so other PoolManagers can
+        # override them.
+        self.pool_classes_by_scheme = pool_classes_by_scheme
+        self.key_fn_by_scheme = key_fn_by_scheme.copy()
+
+    def __enter__(self: _SelfT) -> _SelfT:
+        return self
+
+    def __exit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> Literal[False]:
+        self.clear()
+        # Return False to re-raise any potential exceptions
+        return False
+
+    def _new_pool(
+        self,
+        scheme: str,
+        host: str,
+        port: int,
+        request_context: dict[str, typing.Any] | None = None,
+    ) -> HTTPConnectionPool:
+        """
+        Create a new :class:`urllib3.connectionpool.ConnectionPool` based on host, port, scheme, and
+        any additional pool keyword arguments.
+
+        If ``request_context`` is provided, it is provided as keyword arguments
+        to the pool class used. This method is used to actually create the
+        connection pools handed out by :meth:`connection_from_url` and
+        companion methods. It is intended to be overridden for customization.
+        """
+        pool_cls: type[HTTPConnectionPool] = self.pool_classes_by_scheme[scheme]
+        if request_context is None:
+            request_context = self.connection_pool_kw.copy()
+
+        # Default blocksize to _DEFAULT_BLOCKSIZE if missing or explicitly
+        # set to 'None' in the request_context.
+        if request_context.get("blocksize") is None:
+            request_context["blocksize"] = _DEFAULT_BLOCKSIZE
+
+        # Although the context has everything necessary to create the pool,
+        # this function has historically only used the scheme, host, and port
+        # in the positional args. When an API change is acceptable these can
+        # be removed.
+        for key in ("scheme", "host", "port"):
+            request_context.pop(key, None)
+
+        if scheme == "http":
+            for kw in SSL_KEYWORDS:
+                request_context.pop(kw, None)
+
+        return pool_cls(host, port, **request_context)
+
+    def clear(self) -> None:
+        """
+        Empty our store of pools and direct them all to close.
+
+        This will not affect in-flight connections, but they will not be
+        re-used after completion.
+        """
+        self.pools.clear()
+
+    def connection_from_host(
+        self,
+        host: str | None,
+        port: int | None = None,
+        scheme: str | None = "http",
+        pool_kwargs: dict[str, typing.Any] | None = None,
+    ) -> HTTPConnectionPool:
+        """
+        Get a :class:`urllib3.connectionpool.ConnectionPool` based on the host, port, and scheme.
+
+        If ``port`` isn't given, it will be derived from the ``scheme`` using
+        ``urllib3.connectionpool.port_by_scheme``. If ``pool_kwargs`` is
+        provided, it is merged with the instance's ``connection_pool_kw``
+        variable and used to create the new connection pool, if one is
+        needed.
+        """
+
+        if not host:
+            raise LocationValueError("No host specified.")
+
+        request_context = self._merge_pool_kwargs(pool_kwargs)
+        request_context["scheme"] = scheme or "http"
+        if not port:
+            port = port_by_scheme.get(request_context["scheme"].lower(), 80)
+        request_context["port"] = port
+        request_context["host"] = host
+
+        return self.connection_from_context(request_context)
+
+    def connection_from_context(
+        self, request_context: dict[str, typing.Any]
+    ) -> HTTPConnectionPool:
+        """
+        Get a :class:`urllib3.connectionpool.ConnectionPool` based on the request context.
+
+        ``request_context`` must at least contain the ``scheme`` key and its
+        value must be a key in ``key_fn_by_scheme`` instance variable.
+        """
+        if "strict" in request_context:
+            warnings.warn(
+                "The 'strict' parameter is no longer needed on Python 3+. "
+                "This will raise an error in urllib3 v2.1.0.",
+                DeprecationWarning,
+            )
+            request_context.pop("strict")
+
+        scheme = request_context["scheme"].lower()
+        pool_key_constructor = self.key_fn_by_scheme.get(scheme)
+        if not pool_key_constructor:
+            raise URLSchemeUnknown(scheme)
+        pool_key = pool_key_constructor(request_context)
+
+        return self.connection_from_pool_key(pool_key, request_context=request_context)
+
+    def connection_from_pool_key(
+        self, pool_key: PoolKey, request_context: dict[str, typing.Any]
+    ) -> HTTPConnectionPool:
+        """
+        Get a :class:`urllib3.connectionpool.ConnectionPool` based on the provided pool key.
+
+        ``pool_key`` should be a namedtuple that only contains immutable
+        objects. At a minimum it must have the ``scheme``, ``host``, and
+        ``port`` fields.
+        """
+        with self.pools.lock:
+            # If the scheme, host, or port doesn't match existing open
+            # connections, open a new ConnectionPool.
+            pool = self.pools.get(pool_key)
+            if pool:
+                return pool
+
+            # Make a fresh ConnectionPool of the desired type
+            scheme = request_context["scheme"]
+            host = request_context["host"]
+            port = request_context["port"]
+            pool = self._new_pool(scheme, host, port, request_context=request_context)
+            self.pools[pool_key] = pool
+
+        return pool
+
+    def connection_from_url(
+        self, url: str, pool_kwargs: dict[str, typing.Any] | None = None
+    ) -> HTTPConnectionPool:
+        """
+        Similar to :func:`urllib3.connectionpool.connection_from_url`.
+
+        If ``pool_kwargs`` is not provided and a new pool needs to be
+        constructed, ``self.connection_pool_kw`` is used to initialize
+        the :class:`urllib3.connectionpool.ConnectionPool`. If ``pool_kwargs``
+        is provided, it is used instead. Note that if a new pool does not
+        need to be created for the request, the provided ``pool_kwargs`` are
+        not used.
+        """
+        u = parse_url(url)
+        return self.connection_from_host(
+            u.host, port=u.port, scheme=u.scheme, pool_kwargs=pool_kwargs
+        )
+
+    def _merge_pool_kwargs(
+        self, override: dict[str, typing.Any] | None
+    ) -> dict[str, typing.Any]:
+        """
+        Merge a dictionary of override values for self.connection_pool_kw.
+
+        This does not modify self.connection_pool_kw and returns a new dict.
+        Any keys in the override dictionary with a value of ``None`` are
+        removed from the merged dictionary.
+        """
+        base_pool_kwargs = self.connection_pool_kw.copy()
+        if override:
+            for key, value in override.items():
+                if value is None:
+                    try:
+                        del base_pool_kwargs[key]
+                    except KeyError:
+                        pass
+                else:
+                    base_pool_kwargs[key] = value
+        return base_pool_kwargs
+
+    def _proxy_requires_url_absolute_form(self, parsed_url: Url) -> bool:
+        """
+        Indicates if the proxy requires the complete destination URL in the
+        request.  Normally this is only needed when not using an HTTP CONNECT
+        tunnel.
+        """
+        if self.proxy is None:
+            return False
+
+        return not connection_requires_http_tunnel(
+            self.proxy, self.proxy_config, parsed_url.scheme
+        )
+
+    def urlopen(  # type: ignore[override]
+        self, method: str, url: str, redirect: bool = True, **kw: typing.Any
+    ) -> BaseHTTPResponse:
+        """
+        Same as :meth:`urllib3.HTTPConnectionPool.urlopen`
+        with custom cross-host redirect logic and only sends the request-uri
+        portion of the ``url``.
+
+        The given ``url`` parameter must be absolute, such that an appropriate
+        :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it.
+        """
+        u = parse_url(url)
+
+        if u.scheme is None:
+            warnings.warn(
+                "URLs without a scheme (ie 'https://') are deprecated and will raise an error "
+                "in a future version of urllib3. To avoid this DeprecationWarning ensure all URLs "
+                "start with 'https://' or 'http://'. Read more in this issue: "
+                "https://github.com/urllib3/urllib3/issues/2920",
+                category=DeprecationWarning,
+                stacklevel=2,
+            )
+
+        conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
+
+        kw["assert_same_host"] = False
+        kw["redirect"] = False
+
+        if "headers" not in kw:
+            kw["headers"] = self.headers
+
+        if self._proxy_requires_url_absolute_form(u):
+            response = conn.urlopen(method, url, **kw)
+        else:
+            response = conn.urlopen(method, u.request_uri, **kw)
+
+        redirect_location = redirect and response.get_redirect_location()
+        if not redirect_location:
+            return response
+
+        # Support relative URLs for redirecting.
+        redirect_location = urljoin(url, redirect_location)
+
+        if response.status == 303:
+            # Change the method according to RFC 9110, Section 15.4.4.
+            method = "GET"
+            # And lose the body not to transfer anything sensitive.
+            kw["body"] = None
+            kw["headers"] = HTTPHeaderDict(kw["headers"])._prepare_for_method_change()
+
+        retries = kw.get("retries")
+        if not isinstance(retries, Retry):
+            retries = Retry.from_int(retries, redirect=redirect)
+
+        # Strip headers marked as unsafe to forward to the redirected location.
+        # Check remove_headers_on_redirect to avoid a potential network call within
+        # conn.is_same_host() which may use socket.gethostbyname() in the future.
+        if retries.remove_headers_on_redirect and not conn.is_same_host(
+            redirect_location
+        ):
+            new_headers = kw["headers"].copy()
+            for header in kw["headers"]:
+                if header.lower() in retries.remove_headers_on_redirect:
+                    new_headers.pop(header, None)
+            kw["headers"] = new_headers
+
+        try:
+            retries = retries.increment(method, url, response=response, _pool=conn)
+        except MaxRetryError:
+            if retries.raise_on_redirect:
+                response.drain_conn()
+                raise
+            return response
+
+        kw["retries"] = retries
+        kw["redirect"] = redirect
+
+        log.info("Redirecting %s -> %s", url, redirect_location)
+
+        response.drain_conn()
+        return self.urlopen(method, redirect_location, **kw)
+
+
+class ProxyManager(PoolManager):
+    """
+    Behaves just like :class:`PoolManager`, but sends all requests through
+    the defined proxy, using the CONNECT method for HTTPS URLs.
+
+    :param proxy_url:
+        The URL of the proxy to be used.
+
+    :param proxy_headers:
+        A dictionary containing headers that will be sent to the proxy. In case
+        of HTTP they are being sent with each request, while in the
+        HTTPS/CONNECT case they are sent only once. Could be used for proxy
+        authentication.
+
+    :param proxy_ssl_context:
+        The proxy SSL context is used to establish the TLS connection to the
+        proxy when using HTTPS proxies.
+
+    :param use_forwarding_for_https:
+        (Defaults to False) If set to True will forward requests to the HTTPS
+        proxy to be made on behalf of the client instead of creating a TLS
+        tunnel via the CONNECT method. **Enabling this flag means that request
+        and response headers and content will be visible from the HTTPS proxy**
+        whereas tunneling keeps request and response headers and content
+        private.  IP address, target hostname, SNI, and port are always visible
+        to an HTTPS proxy even when this flag is disabled.
+
+    :param proxy_assert_hostname:
+        The hostname of the certificate to verify against.
+
+    :param proxy_assert_fingerprint:
+        The fingerprint of the certificate to verify against.
+
+    Example:
+
+    .. code-block:: python
+
+        import urllib3
+
+        proxy = urllib3.ProxyManager("https://localhost:3128/")
+
+        resp1 = proxy.request("GET", "https://google.com/")
+        resp2 = proxy.request("GET", "https://httpbin.org/")
+
+        print(len(proxy.pools))
+        # 1
+
+        resp3 = proxy.request("GET", "https://httpbin.org/")
+        resp4 = proxy.request("GET", "https://twitter.com/")
+
+        print(len(proxy.pools))
+        # 3
+
+    """
+
+    def __init__(
+        self,
+        proxy_url: str,
+        num_pools: int = 10,
+        headers: typing.Mapping[str, str] | None = None,
+        proxy_headers: typing.Mapping[str, str] | None = None,
+        proxy_ssl_context: ssl.SSLContext | None = None,
+        use_forwarding_for_https: bool = False,
+        proxy_assert_hostname: None | str | Literal[False] = None,
+        proxy_assert_fingerprint: str | None = None,
+        **connection_pool_kw: typing.Any,
+    ) -> None:
+        if isinstance(proxy_url, HTTPConnectionPool):
+            str_proxy_url = f"{proxy_url.scheme}://{proxy_url.host}:{proxy_url.port}"
+        else:
+            str_proxy_url = proxy_url
+        proxy = parse_url(str_proxy_url)
+
+        if proxy.scheme not in ("http", "https"):
+            raise ProxySchemeUnknown(proxy.scheme)
+
+        if not proxy.port:
+            port = port_by_scheme.get(proxy.scheme, 80)
+            proxy = proxy._replace(port=port)
+
+        self.proxy = proxy
+        self.proxy_headers = proxy_headers or {}
+        self.proxy_ssl_context = proxy_ssl_context
+        self.proxy_config = ProxyConfig(
+            proxy_ssl_context,
+            use_forwarding_for_https,
+            proxy_assert_hostname,
+            proxy_assert_fingerprint,
+        )
+
+        connection_pool_kw["_proxy"] = self.proxy
+        connection_pool_kw["_proxy_headers"] = self.proxy_headers
+        connection_pool_kw["_proxy_config"] = self.proxy_config
+
+        super().__init__(num_pools, headers, **connection_pool_kw)
+
+    def connection_from_host(
+        self,
+        host: str | None,
+        port: int | None = None,
+        scheme: str | None = "http",
+        pool_kwargs: dict[str, typing.Any] | None = None,
+    ) -> HTTPConnectionPool:
+        if scheme == "https":
+            return super().connection_from_host(
+                host, port, scheme, pool_kwargs=pool_kwargs
+            )
+
+        return super().connection_from_host(
+            self.proxy.host, self.proxy.port, self.proxy.scheme, pool_kwargs=pool_kwargs  # type: ignore[union-attr]
+        )
+
+    def _set_proxy_headers(
+        self, url: str, headers: typing.Mapping[str, str] | None = None
+    ) -> typing.Mapping[str, str]:
+        """
+        Sets headers needed by proxies: specifically, the Accept and Host
+        headers. Only sets headers not provided by the user.
+        """
+        headers_ = {"Accept": "*/*"}
+
+        netloc = parse_url(url).netloc
+        if netloc:
+            headers_["Host"] = netloc
+
+        if headers:
+            headers_.update(headers)
+        return headers_
+
+    def urlopen(  # type: ignore[override]
+        self, method: str, url: str, redirect: bool = True, **kw: typing.Any
+    ) -> BaseHTTPResponse:
+        "Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute."
+        u = parse_url(url)
+        if not connection_requires_http_tunnel(self.proxy, self.proxy_config, u.scheme):
+            # For connections using HTTP CONNECT, httplib sets the necessary
+            # headers on the CONNECT to the proxy. If we're not using CONNECT,
+            # we'll definitely need to set 'Host' at the very least.
+            headers = kw.get("headers", self.headers)
+            kw["headers"] = self._set_proxy_headers(url, headers)
+
+        return super().urlopen(method, url, redirect=redirect, **kw)
+
+
+def proxy_from_url(url: str, **kw: typing.Any) -> ProxyManager:
+    return ProxyManager(proxy_url=url, **kw)

Modules/urllib3/py.typed

@@ -0,0 +1,2 @@
+# Instruct type checkers to look for inline type annotations in this package.
+# See PEP 561.

Modules/urllib3/util/__init__.py

@@ -0,0 +1,42 @@
+# For backwards compatibility, provide imports that used to be here.
+from __future__ import annotations
+
+from .connection import is_connection_dropped
+from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
+from .response import is_fp_closed
+from .retry import Retry
+from .ssl_ import (
+    ALPN_PROTOCOLS,
+    IS_PYOPENSSL,
+    SSLContext,
+    assert_fingerprint,
+    create_urllib3_context,
+    resolve_cert_reqs,
+    resolve_ssl_version,
+    ssl_wrap_socket,
+)
+from .timeout import Timeout
+from .url import Url, parse_url
+from .wait import wait_for_read, wait_for_write
+
+__all__ = (
+    "IS_PYOPENSSL",
+    "SSLContext",
+    "ALPN_PROTOCOLS",
+    "Retry",
+    "Timeout",
+    "Url",
+    "assert_fingerprint",
+    "create_urllib3_context",
+    "is_connection_dropped",
+    "is_fp_closed",
+    "parse_url",
+    "make_headers",
+    "resolve_cert_reqs",
+    "resolve_ssl_version",
+    "ssl_wrap_socket",
+    "wait_for_read",
+    "wait_for_write",
+    "SKIP_HEADER",
+    "SKIPPABLE_HEADERS",
+)

Modules/urllib3/util/connection.py

@@ -0,0 +1,137 @@
+from __future__ import annotations
+
+import socket
+import typing
+
+from ..exceptions import LocationParseError
+from .timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
+
+_TYPE_SOCKET_OPTIONS = typing.Sequence[typing.Tuple[int, int, typing.Union[int, bytes]]]
+
+if typing.TYPE_CHECKING:
+    from .._base_connection import BaseHTTPConnection
+
+
+def is_connection_dropped(conn: BaseHTTPConnection) -> bool:  # Platform-specific
+    """
+    Returns True if the connection is dropped and should be closed.
+    :param conn: :class:`urllib3.connection.HTTPConnection` object.
+    """
+    return not conn.is_connected
+
+
+# This function is copied from socket.py in the Python 2.7 standard
+# library test suite. Added to its signature is only `socket_options`.
+# One additional modification is that we avoid binding to IPv6 servers
+# discovered in DNS if the system doesn't have IPv6 functionality.
+def create_connection(
+    address: tuple[str, int],
+    timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+    source_address: tuple[str, int] | None = None,
+    socket_options: _TYPE_SOCKET_OPTIONS | None = None,
+) -> socket.socket:
+    """Connect to *address* and return the socket object.
+
+    Convenience function.  Connect to *address* (a 2-tuple ``(host,
+    port)``) and return the socket object.  Passing the optional
+    *timeout* parameter will set the timeout on the socket instance
+    before attempting to connect.  If no *timeout* is supplied, the
+    global default timeout setting returned by :func:`socket.getdefaulttimeout`
+    is used.  If *source_address* is set it must be a tuple of (host, port)
+    for the socket to bind as a source address before making the connection.
+    An host of '' or port 0 tells the OS to use the default.
+    """
+
+    host, port = address
+    if host.startswith("["):
+        host = host.strip("[]")
+    err = None
+
+    # Using the value from allowed_gai_family() in the context of getaddrinfo lets
+    # us select whether to work with IPv4 DNS records, IPv6 records, or both.
+    # The original create_connection function always returns all records.
+    family = allowed_gai_family()
+
+    try:
+        host.encode("idna")
+    except UnicodeError:
+        raise LocationParseError(f"'{host}', label empty or too long") from None
+
+    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
+        af, socktype, proto, canonname, sa = res
+        sock = None
+        try:
+            sock = socket.socket(af, socktype, proto)
+
+            # If provided, set socket level options before connecting.
+            _set_socket_options(sock, socket_options)
+
+            if timeout is not _DEFAULT_TIMEOUT:
+                sock.settimeout(timeout)
+            if source_address:
+                sock.bind(source_address)
+            sock.connect(sa)
+            # Break explicitly a reference cycle
+            err = None
+            return sock
+
+        except OSError as _:
+            err = _
+            if sock is not None:
+                sock.close()
+
+    if err is not None:
+        try:
+            raise err
+        finally:
+            # Break explicitly a reference cycle
+            err = None
+    else:
+        raise OSError("getaddrinfo returns an empty list")
+
+
+def _set_socket_options(
+    sock: socket.socket, options: _TYPE_SOCKET_OPTIONS | None
+) -> None:
+    if options is None:
+        return
+
+    for opt in options:
+        sock.setsockopt(*opt)
+
+
+def allowed_gai_family() -> socket.AddressFamily:
+    """This function is designed to work in the context of
+    getaddrinfo, where family=socket.AF_UNSPEC is the default and
+    will perform a DNS search for both IPv6 and IPv4 records."""
+
+    family = socket.AF_INET
+    if HAS_IPV6:
+        family = socket.AF_UNSPEC
+    return family
+
+
+def _has_ipv6(host: str) -> bool:
+    """Returns True if the system can bind an IPv6 address."""
+    sock = None
+    has_ipv6 = False
+
+    if socket.has_ipv6:
+        # has_ipv6 returns true if cPython was compiled with IPv6 support.
+        # It does not tell us if the system has IPv6 support enabled. To
+        # determine that we must bind to an IPv6 address.
+        # https://github.com/urllib3/urllib3/pull/611
+        # https://bugs.python.org/issue658327
+        try:
+            sock = socket.socket(socket.AF_INET6)
+            sock.bind((host, 0))
+            has_ipv6 = True
+        except Exception:
+            pass
+
+    if sock:
+        sock.close()
+    return has_ipv6
+
+
+HAS_IPV6 = _has_ipv6("::1")

Modules/urllib3/util/proxy.py

@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import typing
+
+from .url import Url
+
+if typing.TYPE_CHECKING:
+    from ..connection import ProxyConfig
+
+
+def connection_requires_http_tunnel(
+    proxy_url: Url | None = None,
+    proxy_config: ProxyConfig | None = None,
+    destination_scheme: str | None = None,
+) -> bool:
+    """
+    Returns True if the connection requires an HTTP CONNECT through the proxy.
+
+    :param URL proxy_url:
+        URL of the proxy.
+    :param ProxyConfig proxy_config:
+        Proxy configuration from poolmanager.py
+    :param str destination_scheme:
+        The scheme of the destination. (i.e https, http, etc)
+    """
+    # If we're not using a proxy, no way to use a tunnel.
+    if proxy_url is None:
+        return False
+
+    # HTTP destinations never require tunneling, we always forward.
+    if destination_scheme == "http":
+        return False
+
+    # Support for forwarding with HTTPS proxies and HTTPS destinations.
+    if (
+        proxy_url.scheme == "https"
+        and proxy_config
+        and proxy_config.use_forwarding_for_https
+    ):
+        return False
+
+    # Otherwise always use a tunnel.
+    return True

Modules/urllib3/util/request.py

@@ -0,0 +1,256 @@
+from __future__ import annotations
+
+import io
+import typing
+from base64 import b64encode
+from enum import Enum
+
+from ..exceptions import UnrewindableBodyError
+from .util import to_bytes
+
+if typing.TYPE_CHECKING:
+    from typing import Final
+
+# Pass as a value within ``headers`` to skip
+# emitting some HTTP headers that are added automatically.
+# The only headers that are supported are ``Accept-Encoding``,
+# ``Host``, and ``User-Agent``.
+SKIP_HEADER = "@@@SKIP_HEADER@@@"
+SKIPPABLE_HEADERS = frozenset(["accept-encoding", "host", "user-agent"])
+
+ACCEPT_ENCODING = "gzip,deflate"
+try:
+    try:
+        import brotlicffi as _unused_module_brotli  # type: ignore[import] # noqa: F401
+    except ImportError:
+        import brotli as _unused_module_brotli  # type: ignore[import] # noqa: F401
+except ImportError:
+    pass
+else:
+    ACCEPT_ENCODING += ",br"
+try:
+    import zstandard as _unused_module_zstd  # type: ignore[import] # noqa: F401
+except ImportError:
+    pass
+else:
+    ACCEPT_ENCODING += ",zstd"
+
+
+class _TYPE_FAILEDTELL(Enum):
+    token = 0
+
+
+_FAILEDTELL: Final[_TYPE_FAILEDTELL] = _TYPE_FAILEDTELL.token
+
+_TYPE_BODY_POSITION = typing.Union[int, _TYPE_FAILEDTELL]
+
+# When sending a request with these methods we aren't expecting
+# a body so don't need to set an explicit 'Content-Length: 0'
+# The reason we do this in the negative instead of tracking methods
+# which 'should' have a body is because unknown methods should be
+# treated as if they were 'POST' which *does* expect a body.
+_METHODS_NOT_EXPECTING_BODY = {"GET", "HEAD", "DELETE", "TRACE", "OPTIONS", "CONNECT"}
+
+
+def make_headers(
+    keep_alive: bool | None = None,
+    accept_encoding: bool | list[str] | str | None = None,
+    user_agent: str | None = None,
+    basic_auth: str | None = None,
+    proxy_basic_auth: str | None = None,
+    disable_cache: bool | None = None,
+) -> dict[str, str]:
+    """
+    Shortcuts for generating request headers.
+
+    :param keep_alive:
+        If ``True``, adds 'connection: keep-alive' header.
+
+    :param accept_encoding:
+        Can be a boolean, list, or string.
+        ``True`` translates to 'gzip,deflate'.  If either the ``brotli`` or
+        ``brotlicffi`` package is installed 'gzip,deflate,br' is used instead.
+        List will get joined by comma.
+        String will be used as provided.
+
+    :param user_agent:
+        String representing the user-agent you want, such as
+        "python-urllib3/0.6"
+
+    :param basic_auth:
+        Colon-separated username:password string for 'authorization: basic ...'
+        auth header.
+
+    :param proxy_basic_auth:
+        Colon-separated username:password string for 'proxy-authorization: basic ...'
+        auth header.
+
+    :param disable_cache:
+        If ``True``, adds 'cache-control: no-cache' header.
+
+    Example:
+
+    .. code-block:: python
+
+        import urllib3
+
+        print(urllib3.util.make_headers(keep_alive=True, user_agent="Batman/1.0"))
+        # {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
+        print(urllib3.util.make_headers(accept_encoding=True))
+        # {'accept-encoding': 'gzip,deflate'}
+    """
+    headers: dict[str, str] = {}
+    if accept_encoding:
+        if isinstance(accept_encoding, str):
+            pass
+        elif isinstance(accept_encoding, list):
+            accept_encoding = ",".join(accept_encoding)
+        else:
+            accept_encoding = ACCEPT_ENCODING
+        headers["accept-encoding"] = accept_encoding
+
+    if user_agent:
+        headers["user-agent"] = user_agent
+
+    if keep_alive:
+        headers["connection"] = "keep-alive"
+
+    if basic_auth:
+        headers[
+            "authorization"
+        ] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}"
+
+    if proxy_basic_auth:
+        headers[
+            "proxy-authorization"
+        ] = f"Basic {b64encode(proxy_basic_auth.encode('latin-1')).decode()}"
+
+    if disable_cache:
+        headers["cache-control"] = "no-cache"
+
+    return headers
+
+
+def set_file_position(
+    body: typing.Any, pos: _TYPE_BODY_POSITION | None
+) -> _TYPE_BODY_POSITION | None:
+    """
+    If a position is provided, move file to that point.
+    Otherwise, we'll attempt to record a position for future use.
+    """
+    if pos is not None:
+        rewind_body(body, pos)
+    elif getattr(body, "tell", None) is not None:
+        try:
+            pos = body.tell()
+        except OSError:
+            # This differentiates from None, allowing us to catch
+            # a failed `tell()` later when trying to rewind the body.
+            pos = _FAILEDTELL
+
+    return pos
+
+
+def rewind_body(body: typing.IO[typing.AnyStr], body_pos: _TYPE_BODY_POSITION) -> None:
+    """
+    Attempt to rewind body to a certain position.
+    Primarily used for request redirects and retries.
+
+    :param body:
+        File-like object that supports seek.
+
+    :param int pos:
+        Position to seek to in file.
+    """
+    body_seek = getattr(body, "seek", None)
+    if body_seek is not None and isinstance(body_pos, int):
+        try:
+            body_seek(body_pos)
+        except OSError as e:
+            raise UnrewindableBodyError(
+                "An error occurred when rewinding request body for redirect/retry."
+            ) from e
+    elif body_pos is _FAILEDTELL:
+        raise UnrewindableBodyError(
+            "Unable to record file position for rewinding "
+            "request body during a redirect/retry."
+        )
+    else:
+        raise ValueError(
+            f"body_pos must be of type integer, instead it was {type(body_pos)}."
+        )
+
+
+class ChunksAndContentLength(typing.NamedTuple):
+    chunks: typing.Iterable[bytes] | None
+    content_length: int | None
+
+
+def body_to_chunks(
+    body: typing.Any | None, method: str, blocksize: int
+) -> ChunksAndContentLength:
+    """Takes the HTTP request method, body, and blocksize and
+    transforms them into an iterable of chunks to pass to
+    socket.sendall() and an optional 'Content-Length' header.
+
+    A 'Content-Length' of 'None' indicates the length of the body
+    can't be determined so should use 'Transfer-Encoding: chunked'
+    for framing instead.
+    """
+
+    chunks: typing.Iterable[bytes] | None
+    content_length: int | None
+
+    # No body, we need to make a recommendation on 'Content-Length'
+    # based on whether that request method is expected to have
+    # a body or not.
+    if body is None:
+        chunks = None
+        if method.upper() not in _METHODS_NOT_EXPECTING_BODY:
+            content_length = 0
+        else:
+            content_length = None
+
+    # Bytes or strings become bytes
+    elif isinstance(body, (str, bytes)):
+        chunks = (to_bytes(body),)
+        content_length = len(chunks[0])
+
+    # File-like object, TODO: use seek() and tell() for length?
+    elif hasattr(body, "read"):
+
+        def chunk_readable() -> typing.Iterable[bytes]:
+            nonlocal body, blocksize
+            encode = isinstance(body, io.TextIOBase)
+            while True:
+                datablock = body.read(blocksize)
+                if not datablock:
+                    break
+                if encode:
+                    datablock = datablock.encode("iso-8859-1")
+                yield datablock
+
+        chunks = chunk_readable()
+        content_length = None
+
+    # Otherwise we need to start checking via duck-typing.
+    else:
+        try:
+            # Check if the body implements the buffer API.
+            mv = memoryview(body)
+        except TypeError:
+            try:
+                # Check if the body is an iterable
+                chunks = iter(body)
+                content_length = None
+            except TypeError:
+                raise TypeError(
+                    f"'body' must be a bytes-like object, file-like "
+                    f"object, or iterable. Instead was {body!r}"
+                ) from None
+        else:
+            # Since it implements the buffer API can be passed directly to socket.sendall()
+            chunks = (body,)
+            content_length = mv.nbytes
+
+    return ChunksAndContentLength(chunks=chunks, content_length=content_length)

Modules/urllib3/util/response.py

@@ -0,0 +1,101 @@
+from __future__ import annotations
+
+import http.client as httplib
+from email.errors import MultipartInvariantViolationDefect, StartBoundaryNotFoundDefect
+
+from ..exceptions import HeaderParsingError
+
+
+def is_fp_closed(obj: object) -> bool:
+    """
+    Checks whether a given file-like object is closed.
+
+    :param obj:
+        The file-like object to check.
+    """
+
+    try:
+        # Check `isclosed()` first, in case Python3 doesn't set `closed`.
+        # GH Issue #928
+        return obj.isclosed()  # type: ignore[no-any-return, attr-defined]
+    except AttributeError:
+        pass
+
+    try:
+        # Check via the official file-like-object way.
+        return obj.closed  # type: ignore[no-any-return, attr-defined]
+    except AttributeError:
+        pass
+
+    try:
+        # Check if the object is a container for another file-like object that
+        # gets released on exhaustion (e.g. HTTPResponse).
+        return obj.fp is None  # type: ignore[attr-defined]
+    except AttributeError:
+        pass
+
+    raise ValueError("Unable to determine whether fp is closed.")
+
+
+def assert_header_parsing(headers: httplib.HTTPMessage) -> None:
+    """
+    Asserts whether all headers have been successfully parsed.
+    Extracts encountered errors from the result of parsing headers.
+
+    Only works on Python 3.
+
+    :param http.client.HTTPMessage headers: Headers to verify.
+
+    :raises urllib3.exceptions.HeaderParsingError:
+        If parsing errors are found.
+    """
+
+    # This will fail silently if we pass in the wrong kind of parameter.
+    # To make debugging easier add an explicit check.
+    if not isinstance(headers, httplib.HTTPMessage):
+        raise TypeError(f"expected httplib.Message, got {type(headers)}.")
+
+    unparsed_data = None
+
+    # get_payload is actually email.message.Message.get_payload;
+    # we're only interested in the result if it's not a multipart message
+    if not headers.is_multipart():
+        payload = headers.get_payload()
+
+        if isinstance(payload, (bytes, str)):
+            unparsed_data = payload
+
+    # httplib is assuming a response body is available
+    # when parsing headers even when httplib only sends
+    # header data to parse_headers() This results in
+    # defects on multipart responses in particular.
+    # See: https://github.com/urllib3/urllib3/issues/800
+
+    # So we ignore the following defects:
+    # - StartBoundaryNotFoundDefect:
+    #     The claimed start boundary was never found.
+    # - MultipartInvariantViolationDefect:
+    #     A message claimed to be a multipart but no subparts were found.
+    defects = [
+        defect
+        for defect in headers.defects
+        if not isinstance(
+            defect, (StartBoundaryNotFoundDefect, MultipartInvariantViolationDefect)
+        )
+    ]
+
+    if defects or unparsed_data:
+        raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
+
+
+def is_response_to_head(response: httplib.HTTPResponse) -> bool:
+    """
+    Checks whether the request of a response has been a HEAD-request.
+
+    :param http.client.HTTPResponse response:
+        Response to check if the originating request
+        used 'HEAD' as a method.
+    """
+    # FIXME: Can we do this somehow without accessing private httplib _method?
+    method_str = response._method  # type: str  # type: ignore[attr-defined]
+    return method_str.upper() == "HEAD"

Modules/urllib3/util/retry.py

@@ -0,0 +1,529 @@
+from __future__ import annotations
+
+import email
+import logging
+import random
+import re
+import time
+import typing
+from itertools import takewhile
+from types import TracebackType
+
+from ..exceptions import (
+    ConnectTimeoutError,
+    InvalidHeader,
+    MaxRetryError,
+    ProtocolError,
+    ProxyError,
+    ReadTimeoutError,
+    ResponseError,
+)
+from .util import reraise
+
+if typing.TYPE_CHECKING:
+    from ..connectionpool import ConnectionPool
+    from ..response import BaseHTTPResponse
+
+log = logging.getLogger(__name__)
+
+
+# Data structure for representing the metadata of requests that result in a retry.
+class RequestHistory(typing.NamedTuple):
+    method: str | None
+    url: str | None
+    error: Exception | None
+    status: int | None
+    redirect_location: str | None
+
+
+class Retry:
+    """Retry configuration.
+
+    Each retry attempt will create a new Retry object with updated values, so
+    they can be safely reused.
+
+    Retries can be defined as a default for a pool:
+
+    .. code-block:: python
+
+        retries = Retry(connect=5, read=2, redirect=5)
+        http = PoolManager(retries=retries)
+        response = http.request("GET", "https://example.com/")
+
+    Or per-request (which overrides the default for the pool):
+
+    .. code-block:: python
+
+        response = http.request("GET", "https://example.com/", retries=Retry(10))
+
+    Retries can be disabled by passing ``False``:
+
+    .. code-block:: python
+
+        response = http.request("GET", "https://example.com/", retries=False)
+
+    Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless
+    retries are disabled, in which case the causing exception will be raised.
+
+    :param int total:
+        Total number of retries to allow. Takes precedence over other counts.
+
+        Set to ``None`` to remove this constraint and fall back on other
+        counts.
+
+        Set to ``0`` to fail on the first retry.
+
+        Set to ``False`` to disable and imply ``raise_on_redirect=False``.
+
+    :param int connect:
+        How many connection-related errors to retry on.
+
+        These are errors raised before the request is sent to the remote server,
+        which we assume has not triggered the server to process the request.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+    :param int read:
+        How many times to retry on read errors.
+
+        These errors are raised after the request was sent to the server, so the
+        request may have side-effects.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+    :param int redirect:
+        How many redirects to perform. Limit this to avoid infinite redirect
+        loops.
+
+        A redirect is a HTTP response with a status code 301, 302, 303, 307 or
+        308.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+        Set to ``False`` to disable and imply ``raise_on_redirect=False``.
+
+    :param int status:
+        How many times to retry on bad status codes.
+
+        These are retries made on responses, where status code matches
+        ``status_forcelist``.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+    :param int other:
+        How many times to retry on other errors.
+
+        Other errors are errors that are not connect, read, redirect or status errors.
+        These errors might be raised after the request was sent to the server, so the
+        request might have side-effects.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+        If ``total`` is not set, it's a good idea to set this to 0 to account
+        for unexpected edge cases and avoid infinite retry loops.
+
+    :param Collection allowed_methods:
+        Set of uppercased HTTP method verbs that we should retry on.
+
+        By default, we only retry on methods which are considered to be
+        idempotent (multiple requests with the same parameters end with the
+        same state). See :attr:`Retry.DEFAULT_ALLOWED_METHODS`.
+
+        Set to a ``None`` value to retry on any verb.
+
+    :param Collection status_forcelist:
+        A set of integer HTTP status codes that we should force a retry on.
+        A retry is initiated if the request method is in ``allowed_methods``
+        and the response status code is in ``status_forcelist``.
+
+        By default, this is disabled with ``None``.
+
+    :param float backoff_factor:
+        A backoff factor to apply between attempts after the second try
+        (most errors are resolved immediately by a second try without a
+        delay). urllib3 will sleep for::
+
+            {backoff factor} * (2 ** ({number of previous retries}))
+
+        seconds. If `backoff_jitter` is non-zero, this sleep is extended by::
+
+            random.uniform(0, {backoff jitter})
+
+        seconds. For example, if the backoff_factor is 0.1, then :func:`Retry.sleep` will
+        sleep for [0.0s, 0.2s, 0.4s, 0.8s, ...] between retries. No backoff will ever
+        be longer than `backoff_max`.
+
+        By default, backoff is disabled (factor set to 0).
+
+    :param bool raise_on_redirect: Whether, if the number of redirects is
+        exhausted, to raise a MaxRetryError, or to return a response with a
+        response code in the 3xx range.
+
+    :param bool raise_on_status: Similar meaning to ``raise_on_redirect``:
+        whether we should raise an exception, or return a response,
+        if status falls in ``status_forcelist`` range and retries have
+        been exhausted.
+
+    :param tuple history: The history of the request encountered during
+        each call to :meth:`~Retry.increment`. The list is in the order
+        the requests occurred. Each list item is of class :class:`RequestHistory`.
+
+    :param bool respect_retry_after_header:
+        Whether to respect Retry-After header on status codes defined as
+        :attr:`Retry.RETRY_AFTER_STATUS_CODES` or not.
+
+    :param Collection remove_headers_on_redirect:
+        Sequence of headers to remove from the request when a response
+        indicating a redirect is returned before firing off the redirected
+        request.
+    """
+
+    #: Default methods to be used for ``allowed_methods``
+    DEFAULT_ALLOWED_METHODS = frozenset(
+        ["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"]
+    )
+
+    #: Default status codes to be used for ``status_forcelist``
+    RETRY_AFTER_STATUS_CODES = frozenset([413, 429, 503])
+
+    #: Default headers to be used for ``remove_headers_on_redirect``
+    DEFAULT_REMOVE_HEADERS_ON_REDIRECT = frozenset(["Cookie", "Authorization"])
+
+    #: Default maximum backoff time.
+    DEFAULT_BACKOFF_MAX = 120
+
+    # Backward compatibility; assigned outside of the class.
+    DEFAULT: typing.ClassVar[Retry]
+
+    def __init__(
+        self,
+        total: bool | int | None = 10,
+        connect: int | None = None,
+        read: int | None = None,
+        redirect: bool | int | None = None,
+        status: int | None = None,
+        other: int | None = None,
+        allowed_methods: typing.Collection[str] | None = DEFAULT_ALLOWED_METHODS,
+        status_forcelist: typing.Collection[int] | None = None,
+        backoff_factor: float = 0,
+        backoff_max: float = DEFAULT_BACKOFF_MAX,
+        raise_on_redirect: bool = True,
+        raise_on_status: bool = True,
+        history: tuple[RequestHistory, ...] | None = None,
+        respect_retry_after_header: bool = True,
+        remove_headers_on_redirect: typing.Collection[
+            str
+        ] = DEFAULT_REMOVE_HEADERS_ON_REDIRECT,
+        backoff_jitter: float = 0.0,
+    ) -> None:
+        self.total = total
+        self.connect = connect
+        self.read = read
+        self.status = status
+        self.other = other
+
+        if redirect is False or total is False:
+            redirect = 0
+            raise_on_redirect = False
+
+        self.redirect = redirect
+        self.status_forcelist = status_forcelist or set()
+        self.allowed_methods = allowed_methods
+        self.backoff_factor = backoff_factor
+        self.backoff_max = backoff_max
+        self.raise_on_redirect = raise_on_redirect
+        self.raise_on_status = raise_on_status
+        self.history = history or ()
+        self.respect_retry_after_header = respect_retry_after_header
+        self.remove_headers_on_redirect = frozenset(
+            h.lower() for h in remove_headers_on_redirect
+        )
+        self.backoff_jitter = backoff_jitter
+
+    def new(self, **kw: typing.Any) -> Retry:
+        params = dict(
+            total=self.total,
+            connect=self.connect,
+            read=self.read,
+            redirect=self.redirect,
+            status=self.status,
+            other=self.other,
+            allowed_methods=self.allowed_methods,
+            status_forcelist=self.status_forcelist,
+            backoff_factor=self.backoff_factor,
+            backoff_max=self.backoff_max,
+            raise_on_redirect=self.raise_on_redirect,
+            raise_on_status=self.raise_on_status,
+            history=self.history,
+            remove_headers_on_redirect=self.remove_headers_on_redirect,
+            respect_retry_after_header=self.respect_retry_after_header,
+            backoff_jitter=self.backoff_jitter,
+        )
+
+        params.update(kw)
+        return type(self)(**params)  # type: ignore[arg-type]
+
+    @classmethod
+    def from_int(
+        cls,
+        retries: Retry | bool | int | None,
+        redirect: bool | int | None = True,
+        default: Retry | bool | int | None = None,
+    ) -> Retry:
+        """Backwards-compatibility for the old retries format."""
+        if retries is None:
+            retries = default if default is not None else cls.DEFAULT
+
+        if isinstance(retries, Retry):
+            return retries
+
+        redirect = bool(redirect) and None
+        new_retries = cls(retries, redirect=redirect)
+        log.debug("Converted retries value: %r -> %r", retries, new_retries)
+        return new_retries
+
+    def get_backoff_time(self) -> float:
+        """Formula for computing the current backoff
+
+        :rtype: float
+        """
+        # We want to consider only the last consecutive errors sequence (Ignore redirects).
+        consecutive_errors_len = len(
+            list(
+                takewhile(lambda x: x.redirect_location is None, reversed(self.history))
+            )
+        )
+        if consecutive_errors_len <= 1:
+            return 0
+
+        backoff_value = self.backoff_factor * (2 ** (consecutive_errors_len - 1))
+        if self.backoff_jitter != 0.0:
+            backoff_value += random.random() * self.backoff_jitter
+        return float(max(0, min(self.backoff_max, backoff_value)))
+
+    def parse_retry_after(self, retry_after: str) -> float:
+        seconds: float
+        # Whitespace: https://tools.ietf.org/html/rfc7230#section-3.2.4
+        if re.match(r"^\s*[0-9]+\s*$", retry_after):
+            seconds = int(retry_after)
+        else:
+            retry_date_tuple = email.utils.parsedate_tz(retry_after)
+            if retry_date_tuple is None:
+                raise InvalidHeader(f"Invalid Retry-After header: {retry_after}")
+
+            retry_date = email.utils.mktime_tz(retry_date_tuple)
+            seconds = retry_date - time.time()
+
+        seconds = max(seconds, 0)
+
+        return seconds
+
+    def get_retry_after(self, response: BaseHTTPResponse) -> float | None:
+        """Get the value of Retry-After in seconds."""
+
+        retry_after = response.headers.get("Retry-After")
+
+        if retry_after is None:
+            return None
+
+        return self.parse_retry_after(retry_after)
+
+    def sleep_for_retry(self, response: BaseHTTPResponse) -> bool:
+        retry_after = self.get_retry_after(response)
+        if retry_after:
+            time.sleep(retry_after)
+            return True
+
+        return False
+
+    def _sleep_backoff(self) -> None:
+        backoff = self.get_backoff_time()
+        if backoff <= 0:
+            return
+        time.sleep(backoff)
+
+    def sleep(self, response: BaseHTTPResponse | None = None) -> None:
+        """Sleep between retry attempts.
+
+        This method will respect a server's ``Retry-After`` response header
+        and sleep the duration of the time requested. If that is not present, it
+        will use an exponential backoff. By default, the backoff factor is 0 and
+        this method will return immediately.
+        """
+
+        if self.respect_retry_after_header and response:
+            slept = self.sleep_for_retry(response)
+            if slept:
+                return
+
+        self._sleep_backoff()
+
+    def _is_connection_error(self, err: Exception) -> bool:
+        """Errors when we're fairly sure that the server did not receive the
+        request, so it should be safe to retry.
+        """
+        if isinstance(err, ProxyError):
+            err = err.original_error
+        return isinstance(err, ConnectTimeoutError)
+
+    def _is_read_error(self, err: Exception) -> bool:
+        """Errors that occur after the request has been started, so we should
+        assume that the server began processing it.
+        """
+        return isinstance(err, (ReadTimeoutError, ProtocolError))
+
+    def _is_method_retryable(self, method: str) -> bool:
+        """Checks if a given HTTP method should be retried upon, depending if
+        it is included in the allowed_methods
+        """
+        if self.allowed_methods and method.upper() not in self.allowed_methods:
+            return False
+        return True
+
+    def is_retry(
+        self, method: str, status_code: int, has_retry_after: bool = False
+    ) -> bool:
+        """Is this method/status code retryable? (Based on allowlists and control
+        variables such as the number of total retries to allow, whether to
+        respect the Retry-After header, whether this header is present, and
+        whether the returned status code is on the list of status codes to
+        be retried upon on the presence of the aforementioned header)
+        """
+        if not self._is_method_retryable(method):
+            return False
+
+        if self.status_forcelist and status_code in self.status_forcelist:
+            return True
+
+        return bool(
+            self.total
+            and self.respect_retry_after_header
+            and has_retry_after
+            and (status_code in self.RETRY_AFTER_STATUS_CODES)
+        )
+
+    def is_exhausted(self) -> bool:
+        """Are we out of retries?"""
+        retry_counts = [
+            x
+            for x in (
+                self.total,
+                self.connect,
+                self.read,
+                self.redirect,
+                self.status,
+                self.other,
+            )
+            if x
+        ]
+        if not retry_counts:
+            return False
+
+        return min(retry_counts) < 0
+
+    def increment(
+        self,
+        method: str | None = None,
+        url: str | None = None,
+        response: BaseHTTPResponse | None = None,
+        error: Exception | None = None,
+        _pool: ConnectionPool | None = None,
+        _stacktrace: TracebackType | None = None,
+    ) -> Retry:
+        """Return a new Retry object with incremented retry counters.
+
+        :param response: A response object, or None, if the server did not
+            return a response.
+        :type response: :class:`~urllib3.response.BaseHTTPResponse`
+        :param Exception error: An error encountered during the request, or
+            None if the response was received successfully.
+
+        :return: A new ``Retry`` object.
+        """
+        if self.total is False and error:
+            # Disabled, indicate to re-raise the error.
+            raise reraise(type(error), error, _stacktrace)
+
+        total = self.total
+        if total is not None:
+            total -= 1
+
+        connect = self.connect
+        read = self.read
+        redirect = self.redirect
+        status_count = self.status
+        other = self.other
+        cause = "unknown"
+        status = None
+        redirect_location = None
+
+        if error and self._is_connection_error(error):
+            # Connect retry?
+            if connect is False:
+                raise reraise(type(error), error, _stacktrace)
+            elif connect is not None:
+                connect -= 1
+
+        elif error and self._is_read_error(error):
+            # Read retry?
+            if read is False or method is None or not self._is_method_retryable(method):
+                raise reraise(type(error), error, _stacktrace)
+            elif read is not None:
+                read -= 1
+
+        elif error:
+            # Other retry?
+            if other is not None:
+                other -= 1
+
+        elif response and response.get_redirect_location():
+            # Redirect retry?
+            if redirect is not None:
+                redirect -= 1
+            cause = "too many redirects"
+            response_redirect_location = response.get_redirect_location()
+            if response_redirect_location:
+                redirect_location = response_redirect_location
+            status = response.status
+
+        else:
+            # Incrementing because of a server error like a 500 in
+            # status_forcelist and the given method is in the allowed_methods
+            cause = ResponseError.GENERIC_ERROR
+            if response and response.status:
+                if status_count is not None:
+                    status_count -= 1
+                cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status)
+                status = response.status
+
+        history = self.history + (
+            RequestHistory(method, url, error, status, redirect_location),
+        )
+
+        new_retry = self.new(
+            total=total,
+            connect=connect,
+            read=read,
+            redirect=redirect,
+            status=status_count,
+            other=other,
+            history=history,
+        )
+
+        if new_retry.is_exhausted():
+            reason = error or ResponseError(cause)
+            raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
+
+        log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
+
+        return new_retry
+
+    def __repr__(self) -> str:
+        return (
+            f"{type(self).__name__}(total={self.total}, connect={self.connect}, "
+            f"read={self.read}, redirect={self.redirect}, status={self.status})"
+        )
+
+
+# For backwards compatibility (equivalent to pre-v1.9):
+Retry.DEFAULT = Retry(3)

Modules/urllib3/util/ssl_.py

@@ -0,0 +1,514 @@
+from __future__ import annotations
+
+import hmac
+import os
+import socket
+import sys
+import typing
+import warnings
+from binascii import unhexlify
+from hashlib import md5, sha1, sha256
+
+from ..exceptions import ProxySchemeUnsupported, SSLError
+from .url import _BRACELESS_IPV6_ADDRZ_RE, _IPV4_RE
+
+SSLContext = None
+SSLTransport = None
+HAS_NEVER_CHECK_COMMON_NAME = False
+IS_PYOPENSSL = False
+ALPN_PROTOCOLS = ["http/1.1"]
+
+_TYPE_VERSION_INFO = typing.Tuple[int, int, int, str, int]
+
+# Maps the length of a digest to a possible hash function producing this digest
+HASHFUNC_MAP = {32: md5, 40: sha1, 64: sha256}
+
+
+def _is_bpo_43522_fixed(
+    implementation_name: str,
+    version_info: _TYPE_VERSION_INFO,
+    pypy_version_info: _TYPE_VERSION_INFO | None,
+) -> bool:
+    """Return True for CPython 3.8.9+, 3.9.3+ or 3.10+ and PyPy 7.3.8+ where
+    setting SSLContext.hostname_checks_common_name to False works.
+
+    Outside of CPython and PyPy we don't know which implementations work
+    or not so we conservatively use our hostname matching as we know that works
+    on all implementations.
+
+    https://github.com/urllib3/urllib3/issues/2192#issuecomment-821832963
+    https://foss.heptapod.net/pypy/pypy/-/issues/3539
+    """
+    if implementation_name == "pypy":
+        # https://foss.heptapod.net/pypy/pypy/-/issues/3129
+        return pypy_version_info >= (7, 3, 8)  # type: ignore[operator]
+    elif implementation_name == "cpython":
+        major_minor = version_info[:2]
+        micro = version_info[2]
+        return (
+            (major_minor == (3, 8) and micro >= 9)
+            or (major_minor == (3, 9) and micro >= 3)
+            or major_minor >= (3, 10)
+        )
+    else:  # Defensive:
+        return False
+
+
+def _is_has_never_check_common_name_reliable(
+    openssl_version: str,
+    openssl_version_number: int,
+    implementation_name: str,
+    version_info: _TYPE_VERSION_INFO,
+    pypy_version_info: _TYPE_VERSION_INFO | None,
+) -> bool:
+    # As of May 2023, all released versions of LibreSSL fail to reject certificates with
+    # only common names, see https://github.com/urllib3/urllib3/pull/3024
+    is_openssl = openssl_version.startswith("OpenSSL ")
+    # Before fixing OpenSSL issue #14579, the SSL_new() API was not copying hostflags
+    # like X509_CHECK_FLAG_NEVER_CHECK_SUBJECT, which tripped up CPython.
+    # https://github.com/openssl/openssl/issues/14579
+    # This was released in OpenSSL 1.1.1l+ (>=0x101010cf)
+    is_openssl_issue_14579_fixed = openssl_version_number >= 0x101010CF
+
+    return is_openssl and (
+        is_openssl_issue_14579_fixed
+        or _is_bpo_43522_fixed(implementation_name, version_info, pypy_version_info)
+    )
+
+
+if typing.TYPE_CHECKING:
+    from ssl import VerifyMode
+    from typing import Literal, TypedDict
+
+    from .ssltransport import SSLTransport as SSLTransportType
+
+    class _TYPE_PEER_CERT_RET_DICT(TypedDict, total=False):
+        subjectAltName: tuple[tuple[str, str], ...]
+        subject: tuple[tuple[tuple[str, str], ...], ...]
+        serialNumber: str
+
+
+# Mapping from 'ssl.PROTOCOL_TLSX' to 'TLSVersion.X'
+_SSL_VERSION_TO_TLS_VERSION: dict[int, int] = {}
+
+try:  # Do we have ssl at all?
+    import ssl
+    from ssl import (  # type: ignore[assignment]
+        CERT_REQUIRED,
+        HAS_NEVER_CHECK_COMMON_NAME,
+        OP_NO_COMPRESSION,
+        OP_NO_TICKET,
+        OPENSSL_VERSION,
+        OPENSSL_VERSION_NUMBER,
+        PROTOCOL_TLS,
+        PROTOCOL_TLS_CLIENT,
+        OP_NO_SSLv2,
+        OP_NO_SSLv3,
+        SSLContext,
+        TLSVersion,
+    )
+
+    PROTOCOL_SSLv23 = PROTOCOL_TLS
+
+    # Setting SSLContext.hostname_checks_common_name = False didn't work before CPython
+    # 3.8.9, 3.9.3, and 3.10 (but OK on PyPy) or OpenSSL 1.1.1l+
+    if HAS_NEVER_CHECK_COMMON_NAME and not _is_has_never_check_common_name_reliable(
+        OPENSSL_VERSION,
+        OPENSSL_VERSION_NUMBER,
+        sys.implementation.name,
+        sys.version_info,
+        sys.pypy_version_info if sys.implementation.name == "pypy" else None,  # type: ignore[attr-defined]
+    ):
+        HAS_NEVER_CHECK_COMMON_NAME = False
+
+    # Need to be careful here in case old TLS versions get
+    # removed in future 'ssl' module implementations.
+    for attr in ("TLSv1", "TLSv1_1", "TLSv1_2"):
+        try:
+            _SSL_VERSION_TO_TLS_VERSION[getattr(ssl, f"PROTOCOL_{attr}")] = getattr(
+                TLSVersion, attr
+            )
+        except AttributeError:  # Defensive:
+            continue
+
+    from .ssltransport import SSLTransport  # type: ignore[assignment]
+except ImportError:
+    OP_NO_COMPRESSION = 0x20000  # type: ignore[assignment]
+    OP_NO_TICKET = 0x4000  # type: ignore[assignment]
+    OP_NO_SSLv2 = 0x1000000  # type: ignore[assignment]
+    OP_NO_SSLv3 = 0x2000000  # type: ignore[assignment]
+    PROTOCOL_SSLv23 = PROTOCOL_TLS = 2  # type: ignore[assignment]
+    PROTOCOL_TLS_CLIENT = 16  # type: ignore[assignment]
+
+
+_TYPE_PEER_CERT_RET = typing.Union["_TYPE_PEER_CERT_RET_DICT", bytes, None]
+
+
+def assert_fingerprint(cert: bytes | None, fingerprint: str) -> None:
+    """
+    Checks if given fingerprint matches the supplied certificate.
+
+    :param cert:
+        Certificate as bytes object.
+    :param fingerprint:
+        Fingerprint as string of hexdigits, can be interspersed by colons.
+    """
+
+    if cert is None:
+        raise SSLError("No certificate for the peer.")
+
+    fingerprint = fingerprint.replace(":", "").lower()
+    digest_length = len(fingerprint)
+    hashfunc = HASHFUNC_MAP.get(digest_length)
+    if not hashfunc:
+        raise SSLError(f"Fingerprint of invalid length: {fingerprint}")
+
+    # We need encode() here for py32; works on py2 and p33.
+    fingerprint_bytes = unhexlify(fingerprint.encode())
+
+    cert_digest = hashfunc(cert).digest()
+
+    if not hmac.compare_digest(cert_digest, fingerprint_bytes):
+        raise SSLError(
+            f'Fingerprints did not match. Expected "{fingerprint}", got "{cert_digest.hex()}"'
+        )
+
+
+def resolve_cert_reqs(candidate: None | int | str) -> VerifyMode:
+    """
+    Resolves the argument to a numeric constant, which can be passed to
+    the wrap_socket function/method from the ssl module.
+    Defaults to :data:`ssl.CERT_REQUIRED`.
+    If given a string it is assumed to be the name of the constant in the
+    :mod:`ssl` module or its abbreviation.
+    (So you can specify `REQUIRED` instead of `CERT_REQUIRED`.
+    If it's neither `None` nor a string we assume it is already the numeric
+    constant which can directly be passed to wrap_socket.
+    """
+    if candidate is None:
+        return CERT_REQUIRED
+
+    if isinstance(candidate, str):
+        res = getattr(ssl, candidate, None)
+        if res is None:
+            res = getattr(ssl, "CERT_" + candidate)
+        return res  # type: ignore[no-any-return]
+
+    return candidate  # type: ignore[return-value]
+
+
+def resolve_ssl_version(candidate: None | int | str) -> int:
+    """
+    like resolve_cert_reqs
+    """
+    if candidate is None:
+        return PROTOCOL_TLS
+
+    if isinstance(candidate, str):
+        res = getattr(ssl, candidate, None)
+        if res is None:
+            res = getattr(ssl, "PROTOCOL_" + candidate)
+        return typing.cast(int, res)
+
+    return candidate
+
+
+def create_urllib3_context(
+    ssl_version: int | None = None,
+    cert_reqs: int | None = None,
+    options: int | None = None,
+    ciphers: str | None = None,
+    ssl_minimum_version: int | None = None,
+    ssl_maximum_version: int | None = None,
+) -> ssl.SSLContext:
+    """Creates and configures an :class:`ssl.SSLContext` instance for use with urllib3.
+
+    :param ssl_version:
+        The desired protocol version to use. This will default to
+        PROTOCOL_SSLv23 which will negotiate the highest protocol that both
+        the server and your installation of OpenSSL support.
+
+        This parameter is deprecated instead use 'ssl_minimum_version'.
+    :param ssl_minimum_version:
+        The minimum version of TLS to be used. Use the 'ssl.TLSVersion' enum for specifying the value.
+    :param ssl_maximum_version:
+        The maximum version of TLS to be used. Use the 'ssl.TLSVersion' enum for specifying the value.
+        Not recommended to set to anything other than 'ssl.TLSVersion.MAXIMUM_SUPPORTED' which is the
+        default value.
+    :param cert_reqs:
+        Whether to require the certificate verification. This defaults to
+        ``ssl.CERT_REQUIRED``.
+    :param options:
+        Specific OpenSSL options. These default to ``ssl.OP_NO_SSLv2``,
+        ``ssl.OP_NO_SSLv3``, ``ssl.OP_NO_COMPRESSION``, and ``ssl.OP_NO_TICKET``.
+    :param ciphers:
+        Which cipher suites to allow the server to select. Defaults to either system configured
+        ciphers if OpenSSL 1.1.1+, otherwise uses a secure default set of ciphers.
+    :returns:
+        Constructed SSLContext object with specified options
+    :rtype: SSLContext
+    """
+    if SSLContext is None:
+        raise TypeError("Can't create an SSLContext object without an ssl module")
+
+    # This means 'ssl_version' was specified as an exact value.
+    if ssl_version not in (None, PROTOCOL_TLS, PROTOCOL_TLS_CLIENT):
+        # Disallow setting 'ssl_version' and 'ssl_minimum|maximum_version'
+        # to avoid conflicts.
+        if ssl_minimum_version is not None or ssl_maximum_version is not None:
+            raise ValueError(
+                "Can't specify both 'ssl_version' and either "
+                "'ssl_minimum_version' or 'ssl_maximum_version'"
+            )
+
+        # 'ssl_version' is deprecated and will be removed in the future.
+        else:
+            # Use 'ssl_minimum_version' and 'ssl_maximum_version' instead.
+            ssl_minimum_version = _SSL_VERSION_TO_TLS_VERSION.get(
+                ssl_version, TLSVersion.MINIMUM_SUPPORTED
+            )
+            ssl_maximum_version = _SSL_VERSION_TO_TLS_VERSION.get(
+                ssl_version, TLSVersion.MAXIMUM_SUPPORTED
+            )
+
+            # This warning message is pushing users to use 'ssl_minimum_version'
+            # instead of both min/max. Best practice is to only set the minimum version and
+            # keep the maximum version to be it's default value: 'TLSVersion.MAXIMUM_SUPPORTED'
+            warnings.warn(
+                "'ssl_version' option is deprecated and will be "
+                "removed in urllib3 v2.1.0. Instead use 'ssl_minimum_version'",
+                category=DeprecationWarning,
+                stacklevel=2,
+            )
+
+    # PROTOCOL_TLS is deprecated in Python 3.10 so we always use PROTOCOL_TLS_CLIENT
+    context = SSLContext(PROTOCOL_TLS_CLIENT)
+
+    if ssl_minimum_version is not None:
+        context.minimum_version = ssl_minimum_version
+    else:  # Python <3.10 defaults to 'MINIMUM_SUPPORTED' so explicitly set TLSv1.2 here
+        context.minimum_version = TLSVersion.TLSv1_2
+
+    if ssl_maximum_version is not None:
+        context.maximum_version = ssl_maximum_version
+
+    # Unless we're given ciphers defer to either system ciphers in
+    # the case of OpenSSL 1.1.1+ or use our own secure default ciphers.
+    if ciphers:
+        context.set_ciphers(ciphers)
+
+    # Setting the default here, as we may have no ssl module on import
+    cert_reqs = ssl.CERT_REQUIRED if cert_reqs is None else cert_reqs
+
+    if options is None:
+        options = 0
+        # SSLv2 is easily broken and is considered harmful and dangerous
+        options |= OP_NO_SSLv2
+        # SSLv3 has several problems and is now dangerous
+        options |= OP_NO_SSLv3
+        # Disable compression to prevent CRIME attacks for OpenSSL 1.0+
+        # (issue #309)
+        options |= OP_NO_COMPRESSION
+        # TLSv1.2 only. Unless set explicitly, do not request tickets.
+        # This may save some bandwidth on wire, and although the ticket is encrypted,
+        # there is a risk associated with it being on wire,
+        # if the server is not rotating its ticketing keys properly.
+        options |= OP_NO_TICKET
+
+    context.options |= options
+
+    # Enable post-handshake authentication for TLS 1.3, see GH #1634. PHA is
+    # necessary for conditional client cert authentication with TLS 1.3.
+    # The attribute is None for OpenSSL <= 1.1.0 or does not exist in older
+    # versions of Python. We only enable if certificate verification is enabled to work
+    # around Python issue #37428
+    # See: https://bugs.python.org/issue37428
+    if (
+        cert_reqs == ssl.CERT_REQUIRED
+        and getattr(context, "post_handshake_auth", None) is not None
+    ):
+        context.post_handshake_auth = True
+
+    # The order of the below lines setting verify_mode and check_hostname
+    # matter due to safe-guards SSLContext has to prevent an SSLContext with
+    # check_hostname=True, verify_mode=NONE/OPTIONAL.
+    # We always set 'check_hostname=False' for pyOpenSSL so we rely on our own
+    # 'ssl.match_hostname()' implementation.
+    if cert_reqs == ssl.CERT_REQUIRED and not IS_PYOPENSSL:
+        context.verify_mode = cert_reqs
+        context.check_hostname = True
+    else:
+        context.check_hostname = False
+        context.verify_mode = cert_reqs
+
+    try:
+        context.hostname_checks_common_name = False
+    except AttributeError:  # Defensive: for CPython < 3.8.9 and 3.9.3; for PyPy < 7.3.8
+        pass
+
+    # Enable logging of TLS session keys via defacto standard environment variable
+    # 'SSLKEYLOGFILE', if the feature is available (Python 3.8+). Skip empty values.
+    if hasattr(context, "keylog_filename"):
+        sslkeylogfile = os.environ.get("SSLKEYLOGFILE")
+        if sslkeylogfile:
+            context.keylog_filename = sslkeylogfile
+
+    return context
+
+
+@typing.overload
+def ssl_wrap_socket(
+    sock: socket.socket,
+    keyfile: str | None = ...,
+    certfile: str | None = ...,
+    cert_reqs: int | None = ...,
+    ca_certs: str | None = ...,
+    server_hostname: str | None = ...,
+    ssl_version: int | None = ...,
+    ciphers: str | None = ...,
+    ssl_context: ssl.SSLContext | None = ...,
+    ca_cert_dir: str | None = ...,
+    key_password: str | None = ...,
+    ca_cert_data: None | str | bytes = ...,
+    tls_in_tls: Literal[False] = ...,
+) -> ssl.SSLSocket:
+    ...
+
+
+@typing.overload
+def ssl_wrap_socket(
+    sock: socket.socket,
+    keyfile: str | None = ...,
+    certfile: str | None = ...,
+    cert_reqs: int | None = ...,
+    ca_certs: str | None = ...,
+    server_hostname: str | None = ...,
+    ssl_version: int | None = ...,
+    ciphers: str | None = ...,
+    ssl_context: ssl.SSLContext | None = ...,
+    ca_cert_dir: str | None = ...,
+    key_password: str | None = ...,
+    ca_cert_data: None | str | bytes = ...,
+    tls_in_tls: bool = ...,
+) -> ssl.SSLSocket | SSLTransportType:
+    ...
+
+
+def ssl_wrap_socket(
+    sock: socket.socket,
+    keyfile: str | None = None,
+    certfile: str | None = None,
+    cert_reqs: int | None = None,
+    ca_certs: str | None = None,
+    server_hostname: str | None = None,
+    ssl_version: int | None = None,
+    ciphers: str | None = None,
+    ssl_context: ssl.SSLContext | None = None,
+    ca_cert_dir: str | None = None,
+    key_password: str | None = None,
+    ca_cert_data: None | str | bytes = None,
+    tls_in_tls: bool = False,
+) -> ssl.SSLSocket | SSLTransportType:
+    """
+    All arguments except for server_hostname, ssl_context, tls_in_tls, ca_cert_data and
+    ca_cert_dir have the same meaning as they do when using
+    :func:`ssl.create_default_context`, :meth:`ssl.SSLContext.load_cert_chain`,
+    :meth:`ssl.SSLContext.set_ciphers` and :meth:`ssl.SSLContext.wrap_socket`.
+
+    :param server_hostname:
+        When SNI is supported, the expected hostname of the certificate
+    :param ssl_context:
+        A pre-made :class:`SSLContext` object. If none is provided, one will
+        be created using :func:`create_urllib3_context`.
+    :param ciphers:
+        A string of ciphers we wish the client to support.
+    :param ca_cert_dir:
+        A directory containing CA certificates in multiple separate files, as
+        supported by OpenSSL's -CApath flag or the capath argument to
+        SSLContext.load_verify_locations().
+    :param key_password:
+        Optional password if the keyfile is encrypted.
+    :param ca_cert_data:
+        Optional string containing CA certificates in PEM format suitable for
+        passing as the cadata parameter to SSLContext.load_verify_locations()
+    :param tls_in_tls:
+        Use SSLTransport to wrap the existing socket.
+    """
+    context = ssl_context
+    if context is None:
+        # Note: This branch of code and all the variables in it are only used in tests.
+        # We should consider deprecating and removing this code.
+        context = create_urllib3_context(ssl_version, cert_reqs, ciphers=ciphers)
+
+    if ca_certs or ca_cert_dir or ca_cert_data:
+        try:
+            context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
+        except OSError as e:
+            raise SSLError(e) from e
+
+    elif ssl_context is None and hasattr(context, "load_default_certs"):
+        # try to load OS default certs; works well on Windows.
+        context.load_default_certs()
+
+    # Attempt to detect if we get the goofy behavior of the
+    # keyfile being encrypted and OpenSSL asking for the
+    # passphrase via the terminal and instead error out.
+    if keyfile and key_password is None and _is_key_file_encrypted(keyfile):
+        raise SSLError("Client private key is encrypted, password is required")
+
+    if certfile:
+        if key_password is None:
+            context.load_cert_chain(certfile, keyfile)
+        else:
+            context.load_cert_chain(certfile, keyfile, key_password)
+
+    try:
+        context.set_alpn_protocols(ALPN_PROTOCOLS)
+    except NotImplementedError:  # Defensive: in CI, we always have set_alpn_protocols
+        pass
+
+    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
+    return ssl_sock
+
+
+def is_ipaddress(hostname: str | bytes) -> bool:
+    """Detects whether the hostname given is an IPv4 or IPv6 address.
+    Also detects IPv6 addresses with Zone IDs.
+
+    :param str hostname: Hostname to examine.
+    :return: True if the hostname is an IP address, False otherwise.
+    """
+    if isinstance(hostname, bytes):
+        # IDN A-label bytes are ASCII compatible.
+        hostname = hostname.decode("ascii")
+    return bool(_IPV4_RE.match(hostname) or _BRACELESS_IPV6_ADDRZ_RE.match(hostname))
+
+
+def _is_key_file_encrypted(key_file: str) -> bool:
+    """Detects if a key file is encrypted or not."""
+    with open(key_file) as f:
+        for line in f:
+            # Look for Proc-Type: 4,ENCRYPTED
+            if "ENCRYPTED" in line:
+                return True
+
+    return False
+
+
+def _ssl_wrap_socket_impl(
+    sock: socket.socket,
+    ssl_context: ssl.SSLContext,
+    tls_in_tls: bool,
+    server_hostname: str | None = None,
+) -> ssl.SSLSocket | SSLTransportType:
+    if tls_in_tls:
+        if not SSLTransport:
+            # Import error, ssl is not available.
+            raise ProxySchemeUnsupported(
+                "TLS in TLS requires support for the 'ssl' module"
+            )
+
+        SSLTransport._validate_ssl_context_for_tls_in_tls(ssl_context)
+        return SSLTransport(sock, ssl_context, server_hostname)
+
+    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)

Modules/urllib3/util/ssl_match_hostname.py

@@ -0,0 +1,159 @@
+"""The match_hostname() function from Python 3.5, essential when using SSL."""
+
+# Note: This file is under the PSF license as the code comes from the python
+# stdlib.   http://docs.python.org/3/license.html
+# It is modified to remove commonName support.
+
+from __future__ import annotations
+
+import ipaddress
+import re
+import typing
+from ipaddress import IPv4Address, IPv6Address
+
+if typing.TYPE_CHECKING:
+    from .ssl_ import _TYPE_PEER_CERT_RET_DICT
+
+__version__ = "3.5.0.1"
+
+
+class CertificateError(ValueError):
+    pass
+
+
+def _dnsname_match(
+    dn: typing.Any, hostname: str, max_wildcards: int = 1
+) -> typing.Match[str] | None | bool:
+    """Matching according to RFC 6125, section 6.4.3
+
+    http://tools.ietf.org/html/rfc6125#section-6.4.3
+    """
+    pats = []
+    if not dn:
+        return False
+
+    # Ported from python3-syntax:
+    # leftmost, *remainder = dn.split(r'.')
+    parts = dn.split(r".")
+    leftmost = parts[0]
+    remainder = parts[1:]
+
+    wildcards = leftmost.count("*")
+    if wildcards > max_wildcards:
+        # Issue #17980: avoid denials of service by refusing more
+        # than one wildcard per fragment.  A survey of established
+        # policy among SSL implementations showed it to be a
+        # reasonable choice.
+        raise CertificateError(
+            "too many wildcards in certificate DNS name: " + repr(dn)
+        )
+
+    # speed up common case w/o wildcards
+    if not wildcards:
+        return bool(dn.lower() == hostname.lower())
+
+    # RFC 6125, section 6.4.3, subitem 1.
+    # The client SHOULD NOT attempt to match a presented identifier in which
+    # the wildcard character comprises a label other than the left-most label.
+    if leftmost == "*":
+        # When '*' is a fragment by itself, it matches a non-empty dotless
+        # fragment.
+        pats.append("[^.]+")
+    elif leftmost.startswith("xn--") or hostname.startswith("xn--"):
+        # RFC 6125, section 6.4.3, subitem 3.
+        # The client SHOULD NOT attempt to match a presented identifier
+        # where the wildcard character is embedded within an A-label or
+        # U-label of an internationalized domain name.
+        pats.append(re.escape(leftmost))
+    else:
+        # Otherwise, '*' matches any dotless string, e.g. www*
+        pats.append(re.escape(leftmost).replace(r"\*", "[^.]*"))
+
+    # add the remaining fragments, ignore any wildcards
+    for frag in remainder:
+        pats.append(re.escape(frag))
+
+    pat = re.compile(r"\A" + r"\.".join(pats) + r"\Z", re.IGNORECASE)
+    return pat.match(hostname)
+
+
+def _ipaddress_match(ipname: str, host_ip: IPv4Address | IPv6Address) -> bool:
+    """Exact matching of IP addresses.
+
+    RFC 9110 section 4.3.5: "A reference identity of IP-ID contains the decoded
+    bytes of the IP address. An IP version 4 address is 4 octets, and an IP
+    version 6 address is 16 octets. [...] A reference identity of type IP-ID
+    matches if the address is identical to an iPAddress value of the
+    subjectAltName extension of the certificate."
+    """
+    # OpenSSL may add a trailing newline to a subjectAltName's IP address
+    # Divergence from upstream: ipaddress can't handle byte str
+    ip = ipaddress.ip_address(ipname.rstrip())
+    return bool(ip.packed == host_ip.packed)
+
+
+def match_hostname(
+    cert: _TYPE_PEER_CERT_RET_DICT | None,
+    hostname: str,
+    hostname_checks_common_name: bool = False,
+) -> None:
+    """Verify that *cert* (in decoded format as returned by
+    SSLSocket.getpeercert()) matches the *hostname*.  RFC 2818 and RFC 6125
+    rules are followed, but IP addresses are not accepted for *hostname*.
+
+    CertificateError is raised on failure. On success, the function
+    returns nothing.
+    """
+    if not cert:
+        raise ValueError(
+            "empty or no certificate, match_hostname needs a "
+            "SSL socket or SSL context with either "
+            "CERT_OPTIONAL or CERT_REQUIRED"
+        )
+    try:
+        # Divergence from upstream: ipaddress can't handle byte str
+        #
+        # The ipaddress module shipped with Python < 3.9 does not support
+        # scoped IPv6 addresses so we unconditionally strip the Zone IDs for
+        # now. Once we drop support for Python 3.9 we can remove this branch.
+        if "%" in hostname:
+            host_ip = ipaddress.ip_address(hostname[: hostname.rfind("%")])
+        else:
+            host_ip = ipaddress.ip_address(hostname)
+
+    except ValueError:
+        # Not an IP address (common case)
+        host_ip = None
+    dnsnames = []
+    san: tuple[tuple[str, str], ...] = cert.get("subjectAltName", ())
+    key: str
+    value: str
+    for key, value in san:
+        if key == "DNS":
+            if host_ip is None and _dnsname_match(value, hostname):
+                return
+            dnsnames.append(value)
+        elif key == "IP Address":
+            if host_ip is not None and _ipaddress_match(value, host_ip):
+                return
+            dnsnames.append(value)
+
+    # We only check 'commonName' if it's enabled and we're not verifying
+    # an IP address. IP addresses aren't valid within 'commonName'.
+    if hostname_checks_common_name and host_ip is None and not dnsnames:
+        for sub in cert.get("subject", ()):
+            for key, value in sub:
+                if key == "commonName":
+                    if _dnsname_match(value, hostname):
+                        return
+                    dnsnames.append(value)
+
+    if len(dnsnames) > 1:
+        raise CertificateError(
+            "hostname %r "
+            "doesn't match either of %s" % (hostname, ", ".join(map(repr, dnsnames)))
+        )
+    elif len(dnsnames) == 1:
+        raise CertificateError(f"hostname {hostname!r} doesn't match {dnsnames[0]!r}")
+    else:
+        raise CertificateError("no appropriate subjectAltName fields were found")

Modules/urllib3/util/ssltransport.py

@@ -0,0 +1,280 @@
+from __future__ import annotations
+
+import io
+import socket
+import ssl
+import typing
+
+from ..exceptions import ProxySchemeUnsupported
+
+if typing.TYPE_CHECKING:
+    from typing import Literal
+
+    from .ssl_ import _TYPE_PEER_CERT_RET, _TYPE_PEER_CERT_RET_DICT
+
+
+_SelfT = typing.TypeVar("_SelfT", bound="SSLTransport")
+_WriteBuffer = typing.Union[bytearray, memoryview]
+_ReturnValue = typing.TypeVar("_ReturnValue")
+
+SSL_BLOCKSIZE = 16384
+
+
+class SSLTransport:
+    """
+    The SSLTransport wraps an existing socket and establishes an SSL connection.
+
+    Contrary to Python's implementation of SSLSocket, it allows you to chain
+    multiple TLS connections together. It's particularly useful if you need to
+    implement TLS within TLS.
+
+    The class supports most of the socket API operations.
+    """
+
+    @staticmethod
+    def _validate_ssl_context_for_tls_in_tls(ssl_context: ssl.SSLContext) -> None:
+        """
+        Raises a ProxySchemeUnsupported if the provided ssl_context can't be used
+        for TLS in TLS.
+
+        The only requirement is that the ssl_context provides the 'wrap_bio'
+        methods.
+        """
+
+        if not hasattr(ssl_context, "wrap_bio"):
+            raise ProxySchemeUnsupported(
+                "TLS in TLS requires SSLContext.wrap_bio() which isn't "
+                "available on non-native SSLContext"
+            )
+
+    def __init__(
+        self,
+        socket: socket.socket,
+        ssl_context: ssl.SSLContext,
+        server_hostname: str | None = None,
+        suppress_ragged_eofs: bool = True,
+    ) -> None:
+        """
+        Create an SSLTransport around socket using the provided ssl_context.
+        """
+        self.incoming = ssl.MemoryBIO()
+        self.outgoing = ssl.MemoryBIO()
+
+        self.suppress_ragged_eofs = suppress_ragged_eofs
+        self.socket = socket
+
+        self.sslobj = ssl_context.wrap_bio(
+            self.incoming, self.outgoing, server_hostname=server_hostname
+        )
+
+        # Perform initial handshake.
+        self._ssl_io_loop(self.sslobj.do_handshake)
+
+    def __enter__(self: _SelfT) -> _SelfT:
+        return self
+
+    def __exit__(self, *_: typing.Any) -> None:
+        self.close()
+
+    def fileno(self) -> int:
+        return self.socket.fileno()
+
+    def read(self, len: int = 1024, buffer: typing.Any | None = None) -> int | bytes:
+        return self._wrap_ssl_read(len, buffer)
+
+    def recv(self, buflen: int = 1024, flags: int = 0) -> int | bytes:
+        if flags != 0:
+            raise ValueError("non-zero flags not allowed in calls to recv")
+        return self._wrap_ssl_read(buflen)
+
+    def recv_into(
+        self,
+        buffer: _WriteBuffer,
+        nbytes: int | None = None,
+        flags: int = 0,
+    ) -> None | int | bytes:
+        if flags != 0:
+            raise ValueError("non-zero flags not allowed in calls to recv_into")
+        if nbytes is None:
+            nbytes = len(buffer)
+        return self.read(nbytes, buffer)
+
+    def sendall(self, data: bytes, flags: int = 0) -> None:
+        if flags != 0:
+            raise ValueError("non-zero flags not allowed in calls to sendall")
+        count = 0
+        with memoryview(data) as view, view.cast("B") as byte_view:
+            amount = len(byte_view)
+            while count < amount:
+                v = self.send(byte_view[count:])
+                count += v
+
+    def send(self, data: bytes, flags: int = 0) -> int:
+        if flags != 0:
+            raise ValueError("non-zero flags not allowed in calls to send")
+        return self._ssl_io_loop(self.sslobj.write, data)
+
+    def makefile(
+        self,
+        mode: str,
+        buffering: int | None = None,
+        *,
+        encoding: str | None = None,
+        errors: str | None = None,
+        newline: str | None = None,
+    ) -> typing.BinaryIO | typing.TextIO | socket.SocketIO:
+        """
+        Python's httpclient uses makefile and buffered io when reading HTTP
+        messages and we need to support it.
+
+        This is unfortunately a copy and paste of socket.py makefile with small
+        changes to point to the socket directly.
+        """
+        if not set(mode) <= {"r", "w", "b"}:
+            raise ValueError(f"invalid mode {mode!r} (only r, w, b allowed)")
+
+        writing = "w" in mode
+        reading = "r" in mode or not writing
+        assert reading or writing
+        binary = "b" in mode
+        rawmode = ""
+        if reading:
+            rawmode += "r"
+        if writing:
+            rawmode += "w"
+        raw = socket.SocketIO(self, rawmode)  # type: ignore[arg-type]
+        self.socket._io_refs += 1  # type: ignore[attr-defined]
+        if buffering is None:
+            buffering = -1
+        if buffering < 0:
+            buffering = io.DEFAULT_BUFFER_SIZE
+        if buffering == 0:
+            if not binary:
+                raise ValueError("unbuffered streams must be binary")
+            return raw
+        buffer: typing.BinaryIO
+        if reading and writing:
+            buffer = io.BufferedRWPair(raw, raw, buffering)  # type: ignore[assignment]
+        elif reading:
+            buffer = io.BufferedReader(raw, buffering)
+        else:
+            assert writing
+            buffer = io.BufferedWriter(raw, buffering)
+        if binary:
+            return buffer
+        text = io.TextIOWrapper(buffer, encoding, errors, newline)
+        text.mode = mode  # type: ignore[misc]
+        return text
+
+    def unwrap(self) -> None:
+        self._ssl_io_loop(self.sslobj.unwrap)
+
+    def close(self) -> None:
+        self.socket.close()
+
+    @typing.overload
+    def getpeercert(
+        self, binary_form: Literal[False] = ...
+    ) -> _TYPE_PEER_CERT_RET_DICT | None:
+        ...
+
+    @typing.overload
+    def getpeercert(self, binary_form: Literal[True]) -> bytes | None:
+        ...
+
+    def getpeercert(self, binary_form: bool = False) -> _TYPE_PEER_CERT_RET:
+        return self.sslobj.getpeercert(binary_form)  # type: ignore[return-value]
+
+    def version(self) -> str | None:
+        return self.sslobj.version()
+
+    def cipher(self) -> tuple[str, str, int] | None:
+        return self.sslobj.cipher()
+
+    def selected_alpn_protocol(self) -> str | None:
+        return self.sslobj.selected_alpn_protocol()
+
+    def selected_npn_protocol(self) -> str | None:
+        return self.sslobj.selected_npn_protocol()
+
+    def shared_ciphers(self) -> list[tuple[str, str, int]] | None:
+        return self.sslobj.shared_ciphers()
+
+    def compression(self) -> str | None:
+        return self.sslobj.compression()
+
+    def settimeout(self, value: float | None) -> None:
+        self.socket.settimeout(value)
+
+    def gettimeout(self) -> float | None:
+        return self.socket.gettimeout()
+
+    def _decref_socketios(self) -> None:
+        self.socket._decref_socketios()  # type: ignore[attr-defined]
+
+    def _wrap_ssl_read(self, len: int, buffer: bytearray | None = None) -> int | bytes:
+        try:
+            return self._ssl_io_loop(self.sslobj.read, len, buffer)
+        except ssl.SSLError as e:
+            if e.errno == ssl.SSL_ERROR_EOF and self.suppress_ragged_eofs:
+                return 0  # eof, return 0.
+            else:
+                raise
+
+    # func is sslobj.do_handshake or sslobj.unwrap
+    @typing.overload
+    def _ssl_io_loop(self, func: typing.Callable[[], None]) -> None:
+        ...
+
+    # func is sslobj.write, arg1 is data
+    @typing.overload
+    def _ssl_io_loop(self, func: typing.Callable[[bytes], int], arg1: bytes) -> int:
+        ...
+
+    # func is sslobj.read, arg1 is len, arg2 is buffer
+    @typing.overload
+    def _ssl_io_loop(
+        self,
+        func: typing.Callable[[int, bytearray | None], bytes],
+        arg1: int,
+        arg2: bytearray | None,
+    ) -> bytes:
+        ...
+
+    def _ssl_io_loop(
+        self,
+        func: typing.Callable[..., _ReturnValue],
+        arg1: None | bytes | int = None,
+        arg2: bytearray | None = None,
+    ) -> _ReturnValue:
+        """Performs an I/O loop between incoming/outgoing and the socket."""
+        should_loop = True
+        ret = None
+
+        while should_loop:
+            errno = None
+            try:
+                if arg1 is None and arg2 is None:
+                    ret = func()
+                elif arg2 is None:
+                    ret = func(arg1)
+                else:
+                    ret = func(arg1, arg2)
+            except ssl.SSLError as e:
+                if e.errno not in (ssl.SSL_ERROR_WANT_READ, ssl.SSL_ERROR_WANT_WRITE):
+                    # WANT_READ, and WANT_WRITE are expected, others are not.
+                    raise e
+                errno = e.errno
+
+            buf = self.outgoing.read()
+            self.socket.sendall(buf)
+
+            if errno is None:
+                should_loop = False
+            elif errno == ssl.SSL_ERROR_WANT_READ:
+                buf = self.socket.recv(SSL_BLOCKSIZE)
+                if buf:
+                    self.incoming.write(buf)
+                else:
+                    self.incoming.write_eof()
+        return typing.cast(_ReturnValue, ret)

Modules/urllib3/util/timeout.py

@@ -0,0 +1,279 @@
+from __future__ import annotations
+
+import time
+import typing
+from enum import Enum
+from socket import getdefaulttimeout
+
+from ..exceptions import TimeoutStateError
+
+if typing.TYPE_CHECKING:
+    from typing import Final
+
+
+class _TYPE_DEFAULT(Enum):
+    # This value should never be passed to socket.settimeout() so for safety we use a -1.
+    # socket.settimout() raises a ValueError for negative values.
+    token = -1
+
+
+_DEFAULT_TIMEOUT: Final[_TYPE_DEFAULT] = _TYPE_DEFAULT.token
+
+_TYPE_TIMEOUT = typing.Optional[typing.Union[float, _TYPE_DEFAULT]]
+
+
+class Timeout:
+    """Timeout configuration.
+
+    Timeouts can be defined as a default for a pool:
+
+    .. code-block:: python
+
+        import urllib3
+
+        timeout = urllib3.util.Timeout(connect=2.0, read=7.0)
+
+        http = urllib3.PoolManager(timeout=timeout)
+
+        resp = http.request("GET", "https://example.com/")
+
+        print(resp.status)
+
+    Or per-request (which overrides the default for the pool):
+
+    .. code-block:: python
+
+       response = http.request("GET", "https://example.com/", timeout=Timeout(10))
+
+    Timeouts can be disabled by setting all the parameters to ``None``:
+
+    .. code-block:: python
+
+       no_timeout = Timeout(connect=None, read=None)
+       response = http.request("GET", "https://example.com/", timeout=no_timeout)
+
+
+    :param total:
+        This combines the connect and read timeouts into one; the read timeout
+        will be set to the time leftover from the connect attempt. In the
+        event that both a connect timeout and a total are specified, or a read
+        timeout and a total are specified, the shorter timeout will be applied.
+
+        Defaults to None.
+
+    :type total: int, float, or None
+
+    :param connect:
+        The maximum amount of time (in seconds) to wait for a connection
+        attempt to a server to succeed. Omitting the parameter will default the
+        connect timeout to the system default, probably `the global default
+        timeout in socket.py
+        <http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
+        None will set an infinite timeout for connection attempts.
+
+    :type connect: int, float, or None
+
+    :param read:
+        The maximum amount of time (in seconds) to wait between consecutive
+        read operations for a response from the server. Omitting the parameter
+        will default the read timeout to the system default, probably `the
+        global default timeout in socket.py
+        <http://hg.python.org/cpython/file/603b4d593758/Lib/socket.py#l535>`_.
+        None will set an infinite timeout.
+
+    :type read: int, float, or None
+
+    .. note::
+
+        Many factors can affect the total amount of time for urllib3 to return
+        an HTTP response.
+
+        For example, Python's DNS resolver does not obey the timeout specified
+        on the socket. Other factors that can affect total request time include
+        high CPU load, high swap, the program running at a low priority level,
+        or other behaviors.
+
+        In addition, the read and total timeouts only measure the time between
+        read operations on the socket connecting the client and the server,
+        not the total amount of time for the request to return a complete
+        response. For most requests, the timeout is raised because the server
+        has not sent the first byte in the specified time. This is not always
+        the case; if a server streams one byte every fifteen seconds, a timeout
+        of 20 seconds will not trigger, even though the request will take
+        several minutes to complete.
+
+        If your goal is to cut off any request after a set amount of wall clock
+        time, consider having a second "watcher" thread to cut off a slow
+        request.
+    """
+
+    #: A sentinel object representing the default timeout value
+    DEFAULT_TIMEOUT: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT
+
+    def __init__(
+        self,
+        total: _TYPE_TIMEOUT = None,
+        connect: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+        read: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
+    ) -> None:
+        self._connect = self._validate_timeout(connect, "connect")
+        self._read = self._validate_timeout(read, "read")
+        self.total = self._validate_timeout(total, "total")
+        self._start_connect: float | None = None
+
+    def __repr__(self) -> str:
+        return f"{type(self).__name__}(connect={self._connect!r}, read={self._read!r}, total={self.total!r})"
+
+    # __str__ provided for backwards compatibility
+    __str__ = __repr__
+
+    @staticmethod
+    def resolve_default_timeout(timeout: _TYPE_TIMEOUT) -> float | None:
+        return getdefaulttimeout() if timeout is _DEFAULT_TIMEOUT else timeout
+
+    @classmethod
+    def _validate_timeout(cls, value: _TYPE_TIMEOUT, name: str) -> _TYPE_TIMEOUT:
+        """Check that a timeout attribute is valid.
+
+        :param value: The timeout value to validate
+        :param name: The name of the timeout attribute to validate. This is
+            used to specify in error messages.
+        :return: The validated and casted version of the given value.
+        :raises ValueError: If it is a numeric value less than or equal to
+            zero, or the type is not an integer, float, or None.
+        """
+        if value is None or value is _DEFAULT_TIMEOUT:
+            return value
+
+        if isinstance(value, bool):
+            raise ValueError(
+                "Timeout cannot be a boolean value. It must "
+                "be an int, float or None."
+            )
+        try:
+            float(value)
+        except (TypeError, ValueError):
+            raise ValueError(
+                "Timeout value %s was %s, but it must be an "
+                "int, float or None." % (name, value)
+            ) from None
+
+        try:
+            if value <= 0:
+                raise ValueError(
+                    "Attempted to set %s timeout to %s, but the "
+                    "timeout cannot be set to a value less "
+                    "than or equal to 0." % (name, value)
+                )
+        except TypeError:
+            raise ValueError(
+                "Timeout value %s was %s, but it must be an "
+                "int, float or None." % (name, value)
+            ) from None
+
+        return value
+
+    @classmethod
+    def from_float(cls, timeout: _TYPE_TIMEOUT) -> Timeout:
+        """Create a new Timeout from a legacy timeout value.
+
+        The timeout value used by httplib.py sets the same timeout on the
+        connect(), and recv() socket requests. This creates a :class:`Timeout`
+        object that sets the individual timeouts to the ``timeout`` value
+        passed to this function.
+
+        :param timeout: The legacy timeout value.
+        :type timeout: integer, float, :attr:`urllib3.util.Timeout.DEFAULT_TIMEOUT`, or None
+        :return: Timeout object
+        :rtype: :class:`Timeout`
+        """
+        return Timeout(read=timeout, connect=timeout)
+
+    def clone(self) -> Timeout:
+        """Create a copy of the timeout object
+
+        Timeout properties are stored per-pool but each request needs a fresh
+        Timeout object to ensure each one has its own start/stop configured.
+
+        :return: a copy of the timeout object
+        :rtype: :class:`Timeout`
+        """
+        # We can't use copy.deepcopy because that will also create a new object
+        # for _GLOBAL_DEFAULT_TIMEOUT, which socket.py uses as a sentinel to
+        # detect the user default.
+        return Timeout(connect=self._connect, read=self._read, total=self.total)
+
+    def start_connect(self) -> float:
+        """Start the timeout clock, used during a connect() attempt
+
+        :raises urllib3.exceptions.TimeoutStateError: if you attempt
+            to start a timer that has been started already.
+        """
+        if self._start_connect is not None:
+            raise TimeoutStateError("Timeout timer has already been started.")
+        self._start_connect = time.monotonic()
+        return self._start_connect
+
+    def get_connect_duration(self) -> float:
+        """Gets the time elapsed since the call to :meth:`start_connect`.
+
+        :return: Elapsed time in seconds.
+        :rtype: float
+        :raises urllib3.exceptions.TimeoutStateError: if you attempt
+            to get duration for a timer that hasn't been started.
+        """
+        if self._start_connect is None:
+            raise TimeoutStateError(
+                "Can't get connect duration for timer that has not started."
+            )
+        return time.monotonic() - self._start_connect
+
+    @property
+    def connect_timeout(self) -> _TYPE_TIMEOUT:
+        """Get the value to use when setting a connection timeout.
+
+        This will be a positive float or integer, the value None
+        (never timeout), or the default system timeout.
+
+        :return: Connect timeout.
+        :rtype: int, float, :attr:`Timeout.DEFAULT_TIMEOUT` or None
+        """
+        if self.total is None:
+            return self._connect
+
+        if self._connect is None or self._connect is _DEFAULT_TIMEOUT:
+            return self.total
+
+        return min(self._connect, self.total)  # type: ignore[type-var]
+
+    @property
+    def read_timeout(self) -> float | None:
+        """Get the value for the read timeout.
+
+        This assumes some time has elapsed in the connection timeout and
+        computes the read timeout appropriately.
+
+        If self.total is set, the read timeout is dependent on the amount of
+        time taken by the connect timeout. If the connection time has not been
+        established, a :exc:`~urllib3.exceptions.TimeoutStateError` will be
+        raised.
+
+        :return: Value to use for the read timeout.
+        :rtype: int, float or None
+        :raises urllib3.exceptions.TimeoutStateError: If :meth:`start_connect`
+            has not yet been called on this object.
+        """
+        if (
+            self.total is not None
+            and self.total is not _DEFAULT_TIMEOUT
+            and self._read is not None
+            and self._read is not _DEFAULT_TIMEOUT
+        ):
+            # In case the connect timeout has not yet been established.
+            if self._start_connect is None:
+                return self._read
+            return max(0, min(self.total - self.get_connect_duration(), self._read))
+        elif self.total is not None and self.total is not _DEFAULT_TIMEOUT:
+            return max(0, self.total - self.get_connect_duration())
+        else:
+            return self.resolve_default_timeout(self._read)

Modules/urllib3/util/url.py

@@ -0,0 +1,471 @@
+from __future__ import annotations
+
+import re
+import typing
+
+from ..exceptions import LocationParseError
+from .util import to_str
+
+# We only want to normalize urls with an HTTP(S) scheme.
+# urllib3 infers URLs without a scheme (None) to be http.
+_NORMALIZABLE_SCHEMES = ("http", "https", None)
+
+# Almost all of these patterns were derived from the
+# 'rfc3986' module: https://github.com/python-hyper/rfc3986
+_PERCENT_RE = re.compile(r"%[a-fA-F0-9]{2}")
+_SCHEME_RE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+-]*:|/)")
+_URI_RE = re.compile(
+    r"^(?:([a-zA-Z][a-zA-Z0-9+.-]*):)?"
+    r"(?://([^\\/?#]*))?"
+    r"([^?#]*)"
+    r"(?:\?([^#]*))?"
+    r"(?:#(.*))?$",
+    re.UNICODE | re.DOTALL,
+)
+
+_IPV4_PAT = r"(?:[0-9]{1,3}\.){3}[0-9]{1,3}"
+_HEX_PAT = "[0-9A-Fa-f]{1,4}"
+_LS32_PAT = "(?:{hex}:{hex}|{ipv4})".format(hex=_HEX_PAT, ipv4=_IPV4_PAT)
+_subs = {"hex": _HEX_PAT, "ls32": _LS32_PAT}
+_variations = [
+    #                            6( h16 ":" ) ls32
+    "(?:%(hex)s:){6}%(ls32)s",
+    #                       "::" 5( h16 ":" ) ls32
+    "::(?:%(hex)s:){5}%(ls32)s",
+    # [               h16 ] "::" 4( h16 ":" ) ls32
+    "(?:%(hex)s)?::(?:%(hex)s:){4}%(ls32)s",
+    # [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
+    "(?:(?:%(hex)s:)?%(hex)s)?::(?:%(hex)s:){3}%(ls32)s",
+    # [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
+    "(?:(?:%(hex)s:){0,2}%(hex)s)?::(?:%(hex)s:){2}%(ls32)s",
+    # [ *3( h16 ":" ) h16 ] "::"    h16 ":"   ls32
+    "(?:(?:%(hex)s:){0,3}%(hex)s)?::%(hex)s:%(ls32)s",
+    # [ *4( h16 ":" ) h16 ] "::"              ls32
+    "(?:(?:%(hex)s:){0,4}%(hex)s)?::%(ls32)s",
+    # [ *5( h16 ":" ) h16 ] "::"              h16
+    "(?:(?:%(hex)s:){0,5}%(hex)s)?::%(hex)s",
+    # [ *6( h16 ":" ) h16 ] "::"
+    "(?:(?:%(hex)s:){0,6}%(hex)s)?::",
+]
+
+_UNRESERVED_PAT = r"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._\-~"
+_IPV6_PAT = "(?:" + "|".join([x % _subs for x in _variations]) + ")"
+_ZONE_ID_PAT = "(?:%25|%)(?:[" + _UNRESERVED_PAT + "]|%[a-fA-F0-9]{2})+"
+_IPV6_ADDRZ_PAT = r"\[" + _IPV6_PAT + r"(?:" + _ZONE_ID_PAT + r")?\]"
+_REG_NAME_PAT = r"(?:[^\[\]%:/?#]|%[a-fA-F0-9]{2})*"
+_TARGET_RE = re.compile(r"^(/[^?#]*)(?:\?([^#]*))?(?:#.*)?$")
+
+_IPV4_RE = re.compile("^" + _IPV4_PAT + "$")
+_IPV6_RE = re.compile("^" + _IPV6_PAT + "$")
+_IPV6_ADDRZ_RE = re.compile("^" + _IPV6_ADDRZ_PAT + "$")
+_BRACELESS_IPV6_ADDRZ_RE = re.compile("^" + _IPV6_ADDRZ_PAT[2:-2] + "$")
+_ZONE_ID_RE = re.compile("(" + _ZONE_ID_PAT + r")\]$")
+
+_HOST_PORT_PAT = ("^(%s|%s|%s)(?::0*?(|0|[1-9][0-9]{0,4}))?$") % (
+    _REG_NAME_PAT,
+    _IPV4_PAT,
+    _IPV6_ADDRZ_PAT,
+)
+_HOST_PORT_RE = re.compile(_HOST_PORT_PAT, re.UNICODE | re.DOTALL)
+
+_UNRESERVED_CHARS = set(
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-~"
+)
+_SUB_DELIM_CHARS = set("!$&'()*+,;=")
+_USERINFO_CHARS = _UNRESERVED_CHARS | _SUB_DELIM_CHARS | {":"}
+_PATH_CHARS = _USERINFO_CHARS | {"@", "/"}
+_QUERY_CHARS = _FRAGMENT_CHARS = _PATH_CHARS | {"?"}
+
+
+class Url(
+    typing.NamedTuple(
+        "Url",
+        [
+            ("scheme", typing.Optional[str]),
+            ("auth", typing.Optional[str]),
+            ("host", typing.Optional[str]),
+            ("port", typing.Optional[int]),
+            ("path", typing.Optional[str]),
+            ("query", typing.Optional[str]),
+            ("fragment", typing.Optional[str]),
+        ],
+    )
+):
+    """
+    Data structure for representing an HTTP URL. Used as a return value for
+    :func:`parse_url`. Both the scheme and host are normalized as they are
+    both case-insensitive according to RFC 3986.
+    """
+
+    def __new__(  # type: ignore[no-untyped-def]
+        cls,
+        scheme: str | None = None,
+        auth: str | None = None,
+        host: str | None = None,
+        port: int | None = None,
+        path: str | None = None,
+        query: str | None = None,
+        fragment: str | None = None,
+    ):
+        if path and not path.startswith("/"):
+            path = "/" + path
+        if scheme is not None:
+            scheme = scheme.lower()
+        return super().__new__(cls, scheme, auth, host, port, path, query, fragment)
+
+    @property
+    def hostname(self) -> str | None:
+        """For backwards-compatibility with urlparse. We're nice like that."""
+        return self.host
+
+    @property
+    def request_uri(self) -> str:
+        """Absolute path including the query string."""
+        uri = self.path or "/"
+
+        if self.query is not None:
+            uri += "?" + self.query
+
+        return uri
+
+    @property
+    def authority(self) -> str | None:
+        """
+        Authority component as defined in RFC 3986 3.2.
+        This includes userinfo (auth), host and port.
+
+        i.e.
+            userinfo@host:port
+        """
+        userinfo = self.auth
+        netloc = self.netloc
+        if netloc is None or userinfo is None:
+            return netloc
+        else:
+            return f"{userinfo}@{netloc}"
+
+    @property
+    def netloc(self) -> str | None:
+        """
+        Network location including host and port.
+
+        If you need the equivalent of urllib.parse's ``netloc``,
+        use the ``authority`` property instead.
+        """
+        if self.host is None:
+            return None
+        if self.port:
+            return f"{self.host}:{self.port}"
+        return self.host
+
+    @property
+    def url(self) -> str:
+        """
+        Convert self into a url
+
+        This function should more or less round-trip with :func:`.parse_url`. The
+        returned url may not be exactly the same as the url inputted to
+        :func:`.parse_url`, but it should be equivalent by the RFC (e.g., urls
+        with a blank port will have : removed).
+
+        Example:
+
+        .. code-block:: python
+
+            import urllib3
+
+            U = urllib3.util.parse_url("https://google.com/mail/")
+
+            print(U.url)
+            # "https://google.com/mail/"
+
+            print( urllib3.util.Url("https", "username:password",
+                                    "host.com", 80, "/path", "query", "fragment"
+                                    ).url
+                )
+            # "https://username:password@host.com:80/path?query#fragment"
+        """
+        scheme, auth, host, port, path, query, fragment = self
+        url = ""
+
+        # We use "is not None" we want things to happen with empty strings (or 0 port)
+        if scheme is not None:
+            url += scheme + "://"
+        if auth is not None:
+            url += auth + "@"
+        if host is not None:
+            url += host
+        if port is not None:
+            url += ":" + str(port)
+        if path is not None:
+            url += path
+        if query is not None:
+            url += "?" + query
+        if fragment is not None:
+            url += "#" + fragment
+
+        return url
+
+    def __str__(self) -> str:
+        return self.url
+
+
+@typing.overload
+def _encode_invalid_chars(
+    component: str, allowed_chars: typing.Container[str]
+) -> str:  # Abstract
+    ...
+
+
+@typing.overload
+def _encode_invalid_chars(
+    component: None, allowed_chars: typing.Container[str]
+) -> None:  # Abstract
+    ...
+
+
+def _encode_invalid_chars(
+    component: str | None, allowed_chars: typing.Container[str]
+) -> str | None:
+    """Percent-encodes a URI component without reapplying
+    onto an already percent-encoded component.
+    """
+    if component is None:
+        return component
+
+    component = to_str(component)
+
+    # Normalize existing percent-encoded bytes.
+    # Try to see if the component we're encoding is already percent-encoded
+    # so we can skip all '%' characters but still encode all others.
+    component, percent_encodings = _PERCENT_RE.subn(
+        lambda match: match.group(0).upper(), component
+    )
+
+    uri_bytes = component.encode("utf-8", "surrogatepass")
+    is_percent_encoded = percent_encodings == uri_bytes.count(b"%")
+    encoded_component = bytearray()
+
+    for i in range(0, len(uri_bytes)):
+        # Will return a single character bytestring
+        byte = uri_bytes[i : i + 1]
+        byte_ord = ord(byte)
+        if (is_percent_encoded and byte == b"%") or (
+            byte_ord < 128 and byte.decode() in allowed_chars
+        ):
+            encoded_component += byte
+            continue
+        encoded_component.extend(b"%" + (hex(byte_ord)[2:].encode().zfill(2).upper()))
+
+    return encoded_component.decode()
+
+
+def _remove_path_dot_segments(path: str) -> str:
+    # See http://tools.ietf.org/html/rfc3986#section-5.2.4 for pseudo-code
+    segments = path.split("/")  # Turn the path into a list of segments
+    output = []  # Initialize the variable to use to store output
+
+    for segment in segments:
+        # '.' is the current directory, so ignore it, it is superfluous
+        if segment == ".":
+            continue
+        # Anything other than '..', should be appended to the output
+        if segment != "..":
+            output.append(segment)
+        # In this case segment == '..', if we can, we should pop the last
+        # element
+        elif output:
+            output.pop()
+
+    # If the path starts with '/' and the output is empty or the first string
+    # is non-empty
+    if path.startswith("/") and (not output or output[0]):
+        output.insert(0, "")
+
+    # If the path starts with '/.' or '/..' ensure we add one more empty
+    # string to add a trailing '/'
+    if path.endswith(("/.", "/..")):
+        output.append("")
+
+    return "/".join(output)
+
+
+@typing.overload
+def _normalize_host(host: None, scheme: str | None) -> None:
+    ...
+
+
+@typing.overload
+def _normalize_host(host: str, scheme: str | None) -> str:
+    ...
+
+
+def _normalize_host(host: str | None, scheme: str | None) -> str | None:
+    if host:
+        if scheme in _NORMALIZABLE_SCHEMES:
+            is_ipv6 = _IPV6_ADDRZ_RE.match(host)
+            if is_ipv6:
+                # IPv6 hosts of the form 'a::b%zone' are encoded in a URL as
+                # such per RFC 6874: 'a::b%25zone'. Unquote the ZoneID
+                # separator as necessary to return a valid RFC 4007 scoped IP.
+                match = _ZONE_ID_RE.search(host)
+                if match:
+                    start, end = match.span(1)
+                    zone_id = host[start:end]
+
+                    if zone_id.startswith("%25") and zone_id != "%25":
+                        zone_id = zone_id[3:]
+                    else:
+                        zone_id = zone_id[1:]
+                    zone_id = _encode_invalid_chars(zone_id, _UNRESERVED_CHARS)
+                    return f"{host[:start].lower()}%{zone_id}{host[end:]}"
+                else:
+                    return host.lower()
+            elif not _IPV4_RE.match(host):
+                return to_str(
+                    b".".join([_idna_encode(label) for label in host.split(".")]),
+                    "ascii",
+                )
+    return host
+
+
+def _idna_encode(name: str) -> bytes:
+    if not name.isascii():
+        try:
+            import idna
+        except ImportError:
+            raise LocationParseError(
+                "Unable to parse URL without the 'idna' module"
+            ) from None
+
+        try:
+            return idna.encode(name.lower(), strict=True, std3_rules=True)
+        except idna.IDNAError:
+            raise LocationParseError(
+                f"Name '{name}' is not a valid IDNA label"
+            ) from None
+
+    return name.lower().encode("ascii")
+
+
+def _encode_target(target: str) -> str:
+    """Percent-encodes a request target so that there are no invalid characters
+
+    Pre-condition for this function is that 'target' must start with '/'.
+    If that is the case then _TARGET_RE will always produce a match.
+    """
+    match = _TARGET_RE.match(target)
+    if not match:  # Defensive:
+        raise LocationParseError(f"{target!r} is not a valid request URI")
+
+    path, query = match.groups()
+    encoded_target = _encode_invalid_chars(path, _PATH_CHARS)
+    if query is not None:
+        query = _encode_invalid_chars(query, _QUERY_CHARS)
+        encoded_target += "?" + query
+    return encoded_target
+
+
+def parse_url(url: str) -> Url:
+    """
+    Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is
+    performed to parse incomplete urls. Fields not provided will be None.
+    This parser is RFC 3986 and RFC 6874 compliant.
+
+    The parser logic and helper functions are based heavily on
+    work done in the ``rfc3986`` module.
+
+    :param str url: URL to parse into a :class:`.Url` namedtuple.
+
+    Partly backwards-compatible with :mod:`urllib.parse`.
+
+    Example:
+
+    .. code-block:: python
+
+        import urllib3
+
+        print( urllib3.util.parse_url('http://google.com/mail/'))
+        # Url(scheme='http', host='google.com', port=None, path='/mail/', ...)
+
+        print( urllib3.util.parse_url('google.com:80'))
+        # Url(scheme=None, host='google.com', port=80, path=None, ...)
+
+        print( urllib3.util.parse_url('/foo?bar'))
+        # Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
+    """
+    if not url:
+        # Empty
+        return Url()
+
+    source_url = url
+    if not _SCHEME_RE.search(url):
+        url = "//" + url
+
+    scheme: str | None
+    authority: str | None
+    auth: str | None
+    host: str | None
+    port: str | None
+    port_int: int | None
+    path: str | None
+    query: str | None
+    fragment: str | None
+
+    try:
+        scheme, authority, path, query, fragment = _URI_RE.match(url).groups()  # type: ignore[union-attr]
+        normalize_uri = scheme is None or scheme.lower() in _NORMALIZABLE_SCHEMES
+
+        if scheme:
+            scheme = scheme.lower()
+
+        if authority:
+            auth, _, host_port = authority.rpartition("@")
+            auth = auth or None
+            host, port = _HOST_PORT_RE.match(host_port).groups()  # type: ignore[union-attr]
+            if auth and normalize_uri:
+                auth = _encode_invalid_chars(auth, _USERINFO_CHARS)
+            if port == "":
+                port = None
+        else:
+            auth, host, port = None, None, None
+
+        if port is not None:
+            port_int = int(port)
+            if not (0 <= port_int <= 65535):
+                raise LocationParseError(url)
+        else:
+            port_int = None
+
+        host = _normalize_host(host, scheme)
+
+        if normalize_uri and path:
+            path = _remove_path_dot_segments(path)
+            path = _encode_invalid_chars(path, _PATH_CHARS)
+        if normalize_uri and query:
+            query = _encode_invalid_chars(query, _QUERY_CHARS)
+        if normalize_uri and fragment:
+            fragment = _encode_invalid_chars(fragment, _FRAGMENT_CHARS)
+
+    except (ValueError, AttributeError) as e:
+        raise LocationParseError(source_url) from e
+
+    # For the sake of backwards compatibility we put empty
+    # string values for path if there are any defined values
+    # beyond the path in the URL.
+    # TODO: Remove this when we break backwards compatibility.
+    if not path:
+        if query is not None or fragment is not None:
+            path = ""
+        else:
+            path = None
+
+    return Url(
+        scheme=scheme,
+        auth=auth,
+        host=host,
+        port=port_int,
+        path=path,
+        query=query,
+        fragment=fragment,
+    )

Modules/urllib3/util/util.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+import typing
+from types import TracebackType
+
+
+def to_bytes(
+    x: str | bytes, encoding: str | None = None, errors: str | None = None
+) -> bytes:
+    if isinstance(x, bytes):
+        return x
+    elif not isinstance(x, str):
+        raise TypeError(f"not expecting type {type(x).__name__}")
+    if encoding or errors:
+        return x.encode(encoding or "utf-8", errors=errors or "strict")
+    return x.encode()
+
+
+def to_str(
+    x: str | bytes, encoding: str | None = None, errors: str | None = None
+) -> str:
+    if isinstance(x, str):
+        return x
+    elif not isinstance(x, bytes):
+        raise TypeError(f"not expecting type {type(x).__name__}")
+    if encoding or errors:
+        return x.decode(encoding or "utf-8", errors=errors or "strict")
+    return x.decode()
+
+
+def reraise(
+    tp: type[BaseException] | None,
+    value: BaseException,
+    tb: TracebackType | None = None,
+) -> typing.NoReturn:
+    try:
+        if value.__traceback__ is not tb:
+            raise value.with_traceback(tb)
+        raise value
+    finally:
+        value = None  # type: ignore[assignment]
+        tb = None

Modules/urllib3/util/wait.py

@@ -0,0 +1,124 @@
+from __future__ import annotations
+
+import select
+import socket
+from functools import partial
+
+__all__ = ["wait_for_read", "wait_for_write"]
+
+
+# How should we wait on sockets?
+#
+# There are two types of APIs you can use for waiting on sockets: the fancy
+# modern stateful APIs like epoll/kqueue, and the older stateless APIs like
+# select/poll. The stateful APIs are more efficient when you have a lots of
+# sockets to keep track of, because you can set them up once and then use them
+# lots of times. But we only ever want to wait on a single socket at a time
+# and don't want to keep track of state, so the stateless APIs are actually
+# more efficient. So we want to use select() or poll().
+#
+# Now, how do we choose between select() and poll()? On traditional Unixes,
+# select() has a strange calling convention that makes it slow, or fail
+# altogether, for high-numbered file descriptors. The point of poll() is to fix
+# that, so on Unixes, we prefer poll().
+#
+# On Windows, there is no poll() (or at least Python doesn't provide a wrapper
+# for it), but that's OK, because on Windows, select() doesn't have this
+# strange calling convention; plain select() works fine.
+#
+# So: on Windows we use select(), and everywhere else we use poll(). We also
+# fall back to select() in case poll() is somehow broken or missing.
+
+
+def select_wait_for_socket(
+    sock: socket.socket,
+    read: bool = False,
+    write: bool = False,
+    timeout: float | None = None,
+) -> bool:
+    if not read and not write:
+        raise RuntimeError("must specify at least one of read=True, write=True")
+    rcheck = []
+    wcheck = []
+    if read:
+        rcheck.append(sock)
+    if write:
+        wcheck.append(sock)
+    # When doing a non-blocking connect, most systems signal success by
+    # marking the socket writable. Windows, though, signals success by marked
+    # it as "exceptional". We paper over the difference by checking the write
+    # sockets for both conditions. (The stdlib selectors module does the same
+    # thing.)
+    fn = partial(select.select, rcheck, wcheck, wcheck)
+    rready, wready, xready = fn(timeout)
+    return bool(rready or wready or xready)
+
+
+def poll_wait_for_socket(
+    sock: socket.socket,
+    read: bool = False,
+    write: bool = False,
+    timeout: float | None = None,
+) -> bool:
+    if not read and not write:
+        raise RuntimeError("must specify at least one of read=True, write=True")
+    mask = 0
+    if read:
+        mask |= select.POLLIN
+    if write:
+        mask |= select.POLLOUT
+    poll_obj = select.poll()
+    poll_obj.register(sock, mask)
+
+    # For some reason, poll() takes timeout in milliseconds
+    def do_poll(t: float | None) -> list[tuple[int, int]]:
+        if t is not None:
+            t *= 1000
+        return poll_obj.poll(t)
+
+    return bool(do_poll(timeout))
+
+
+def _have_working_poll() -> bool:
+    # Apparently some systems have a select.poll that fails as soon as you try
+    # to use it, either due to strange configuration or broken monkeypatching
+    # from libraries like eventlet/greenlet.
+    try:
+        poll_obj = select.poll()
+        poll_obj.poll(0)
+    except (AttributeError, OSError):
+        return False
+    else:
+        return True
+
+
+def wait_for_socket(
+    sock: socket.socket,
+    read: bool = False,
+    write: bool = False,
+    timeout: float | None = None,
+) -> bool:
+    # We delay choosing which implementation to use until the first time we're
+    # called. We could do it at import time, but then we might make the wrong
+    # decision if someone goes wild with monkeypatching select.poll after
+    # we're imported.
+    global wait_for_socket
+    if _have_working_poll():
+        wait_for_socket = poll_wait_for_socket
+    elif hasattr(select, "select"):
+        wait_for_socket = select_wait_for_socket
+    return wait_for_socket(sock, read, write, timeout)
+
+
+def wait_for_read(sock: socket.socket, timeout: float | None = None) -> bool:
+    """Waits for reading to be available on a given socket.
+    Returns True if the socket is readable, or False if the timeout expired.
+    """
+    return wait_for_socket(sock, read=True, timeout=timeout)
+
+
+def wait_for_write(sock: socket.socket, timeout: float | None = None) -> bool:
+    """Waits for writing to be available on a given socket.
+    Returns True if the socket is readable, or False if the timeout expired.
+    """
+    return wait_for_socket(sock, write=True, timeout=timeout)

Modules/vdf/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2015 Rossen Georgiev <rossen@rgp.io>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Modules/vdf/__init__.py

@@ -0,0 +1,519 @@
+"""
+Module for deserializing/serializing to and from VDF
+"""
+__version__ = "3.4"
+__author__ = "Rossen Georgiev"
+
+import re
+import sys
+import struct
+from binascii import crc32
+from io import BytesIO
+from io import StringIO as unicodeIO
+
+try:
+    from collections.abc import Mapping
+except:
+    from collections import Mapping
+
+# Py2 & Py3 compatibility
+if sys.version_info[0] >= 3:
+    string_type = str
+    int_type = int
+    BOMS = '\ufffe\ufeff'
+
+    def strip_bom(line):
+        return line.lstrip(BOMS)
+else:
+    from StringIO import StringIO as strIO
+    string_type = basestring
+    int_type = long
+    BOMS = '\xef\xbb\xbf\xff\xfe\xfe\xff'
+    BOMS_UNICODE = '\\ufffe\\ufeff'.decode('unicode-escape')
+
+    def strip_bom(line):
+        return line.lstrip(BOMS if isinstance(line, str) else BOMS_UNICODE)
+
+# string escaping
+_unescape_char_map = {
+    r"\n": "\n",
+    r"\t": "\t",
+    r"\v": "\v",
+    r"\b": "\b",
+    r"\r": "\r",
+    r"\f": "\f",
+    r"\a": "\a",
+    r"\\": "\\",
+    r"\?": "?",
+    r"\"": "\"",
+    r"\'": "\'",
+}
+_escape_char_map = {v: k for k, v in _unescape_char_map.items()}
+
+def _re_escape_match(m):
+    return _escape_char_map[m.group()]
+
+def _re_unescape_match(m):
+    return _unescape_char_map[m.group()]
+
+def _escape(text):
+    return re.sub(r"[\n\t\v\b\r\f\a\\\?\"']", _re_escape_match, text)
+
+def _unescape(text):
+    return re.sub(r"(\\n|\\t|\\v|\\b|\\r|\\f|\\a|\\\\|\\\?|\\\"|\\')", _re_unescape_match, text)
+
+# parsing and dumping for KV1
+def parse(fp, mapper=dict, merge_duplicate_keys=True, escaped=True):
+    """
+    Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a VDF)
+    to a Python object.
+
+    ``mapper`` specifies the Python object used after deserializetion. ``dict` is
+    used by default. Alternatively, ``collections.OrderedDict`` can be used if you
+    wish to preserve key order. Or any object that acts like a ``dict``.
+
+    ``merge_duplicate_keys`` when ``True`` will merge multiple KeyValue lists with the
+    same key into one instead of overwriting. You can se this to ``False`` if you are
+    using ``VDFDict`` and need to preserve the duplicates.
+    """
+    if not issubclass(mapper, Mapping):
+        raise TypeError("Expected mapper to be subclass of dict, got %s" % type(mapper))
+    if not hasattr(fp, 'readline'):
+        raise TypeError("Expected fp to be a file-like object supporting line iteration")
+
+    stack = [mapper()]
+    expect_bracket = False
+
+    re_keyvalue = re.compile(r'^("(?P<qkey>(?:\\.|[^\\"])*)"|(?P<key>#?[a-z0-9\-\_\\\?$%<>]+))'
+                             r'([ \t]*('
+                             r'"(?P<qval>(?:\\.|[^\\"])*)(?P<vq_end>")?'
+                             r'|(?P<val>(?:(?<!/)/(?!/)|[a-z0-9\-\_\\\?\*\.$<> ])+)'
+                             r'|(?P<sblock>{[ \t]*)(?P<eblock>})?'
+                             r'))?',
+                             flags=re.I)
+
+    for lineno, line in enumerate(fp, 1):
+        if lineno == 1:
+            line = strip_bom(line)
+
+        line = line.lstrip()
+
+        # skip empty and comment lines
+        if line == "" or line[0] == '/':
+            continue
+
+        # one level deeper
+        if line[0] == "{":
+            expect_bracket = False
+            continue
+
+        if expect_bracket:
+            raise SyntaxError("vdf.parse: expected openning bracket",
+                              (getattr(fp, 'name', '<%s>' % fp.__class__.__name__), lineno, 1, line))
+
+        # one level back
+        if line[0] == "}":
+            if len(stack) > 1:
+                stack.pop()
+                continue
+
+            raise SyntaxError("vdf.parse: one too many closing parenthasis",
+                              (getattr(fp, 'name', '<%s>' % fp.__class__.__name__), lineno, 0, line))
+
+        # parse keyvalue pairs
+        while True:
+            match = re_keyvalue.match(line)
+
+            if not match:
+                try:
+                    line += next(fp)
+                    continue
+                except StopIteration:
+                    raise SyntaxError("vdf.parse: unexpected EOF (open key quote?)",
+                                      (getattr(fp, 'name', '<%s>' % fp.__class__.__name__), lineno, 0, line))
+
+            key = match.group('key') if match.group('qkey') is None else match.group('qkey')
+            val = match.group('qval')
+            if val is None:
+                val = match.group('val')
+                if val is not None:
+                    val = val.rstrip()
+                    if val == "":
+                        val = None
+
+            if escaped:
+                key = _unescape(key)
+
+            # we have a key with value in parenthesis, so we make a new dict obj (level deeper)
+            if val is None:
+                if merge_duplicate_keys and key in stack[-1]:
+                    _m = stack[-1][key]
+                    # we've descended a level deeper, if value is str, we have to overwrite it to mapper
+                    if not isinstance(_m, mapper):
+                        _m = stack[-1][key] = mapper()
+                else:
+                    _m = mapper()
+                    stack[-1][key] = _m
+
+                if match.group('eblock') is None:
+                    # only expect a bracket if it's not already closed or on the same line
+                    stack.append(_m)
+                    if match.group('sblock') is None:
+                        expect_bracket = True
+
+            # we've matched a simple keyvalue pair, map it to the last dict obj in the stack
+            else:
+                # if the value is line consume one more line and try to match again,
+                # until we get the KeyValue pair
+                if match.group('vq_end') is None and match.group('qval') is not None:
+                    try:
+                        line += next(fp)
+                        continue
+                    except StopIteration:
+                        raise SyntaxError("vdf.parse: unexpected EOF (open quote for value?)",
+                                          (getattr(fp, 'name', '<%s>' % fp.__class__.__name__), lineno, 0, line))
+
+                stack[-1][key] = _unescape(val) if escaped else val
+
+            # exit the loop
+            break
+
+    if len(stack) != 1:
+        raise SyntaxError("vdf.parse: unclosed parenthasis or quotes (EOF)",
+                           (getattr(fp, 'name', '<%s>' % fp.__class__.__name__), lineno, 0, line))
+
+    return stack.pop()
+
+
+def loads(s, **kwargs):
+    """
+    Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a JSON
+    document) to a Python object.
+    """
+    if not isinstance(s, string_type):
+        raise TypeError("Expected s to be a str, got %s" % type(s))
+
+    try:
+        fp = unicodeIO(s)
+    except TypeError:
+        fp = strIO(s)
+
+    return parse(fp, **kwargs)
+
+
+def load(fp, **kwargs):
+    """
+    Deserialize ``fp`` (a ``.readline()``-supporting file-like object containing
+    a JSON document) to a Python object.
+    """
+    return parse(fp, **kwargs)
+
+
+def dumps(obj, pretty=False, escaped=True):
+    """
+    Serialize ``obj`` to a VDF formatted ``str``.
+    """
+    if not isinstance(obj, Mapping):
+        raise TypeError("Expected data to be an instance of``dict``")
+    if not isinstance(pretty, bool):
+        raise TypeError("Expected pretty to be of type bool")
+    if not isinstance(escaped, bool):
+        raise TypeError("Expected escaped to be of type bool")
+
+    return ''.join(_dump_gen(obj, pretty, escaped))
+
+
+def dump(obj, fp, pretty=False, escaped=True):
+    """
+    Serialize ``obj`` as a VDF formatted stream to ``fp`` (a
+    ``.write()``-supporting file-like object).
+    """
+    if not isinstance(obj, Mapping):
+        raise TypeError("Expected data to be an instance of``dict``")
+    if not hasattr(fp, 'write'):
+        raise TypeError("Expected fp to have write() method")
+    if not isinstance(pretty, bool):
+        raise TypeError("Expected pretty to be of type bool")
+    if not isinstance(escaped, bool):
+        raise TypeError("Expected escaped to be of type bool")
+
+    for chunk in _dump_gen(obj, pretty, escaped):
+        fp.write(chunk)
+
+
+def _dump_gen(data, pretty=False, escaped=True, level=0):
+    indent = "\t"
+    line_indent = ""
+
+    if pretty:
+        line_indent = indent * level
+
+    for key, value in data.items():
+        if escaped and isinstance(key, string_type):
+            key = _escape(key)
+
+        if isinstance(value, Mapping):
+            yield '%s"%s"\n%s{\n' % (line_indent, key, line_indent)
+            for chunk in _dump_gen(value, pretty, escaped, level+1):
+                yield chunk
+            yield "%s}\n" % line_indent
+        else:
+            if escaped and isinstance(value, string_type):
+                value = _escape(value)
+
+            yield '%s"%s" "%s"\n' % (line_indent, key, value)
+
+
+# binary VDF
+class BASE_INT(int_type):
+    def __repr__(self):
+        return "%s(%d)" % (self.__class__.__name__, self)
+
+class UINT_64(BASE_INT):
+    pass
+
+class INT_64(BASE_INT):
+    pass
+
+class POINTER(BASE_INT):
+    pass
+
+class COLOR(BASE_INT):
+    pass
+
+BIN_NONE        = b'\x00'
+BIN_STRING      = b'\x01'
+BIN_INT32       = b'\x02'
+BIN_FLOAT32     = b'\x03'
+BIN_POINTER     = b'\x04'
+BIN_WIDESTRING  = b'\x05'
+BIN_COLOR       = b'\x06'
+BIN_UINT64      = b'\x07'
+BIN_END         = b'\x08'
+BIN_INT64       = b'\x0A'
+BIN_END_ALT     = b'\x0B'
+
+def binary_loads(b, mapper=dict, merge_duplicate_keys=True, alt_format=False, raise_on_remaining=True):
+    """
+    Deserialize ``b`` (``bytes`` containing a VDF in "binary form")
+    to a Python object.
+
+    ``mapper`` specifies the Python object used after deserializetion. ``dict` is
+    used by default. Alternatively, ``collections.OrderedDict`` can be used if you
+    wish to preserve key order. Or any object that acts like a ``dict``.
+
+    ``merge_duplicate_keys`` when ``True`` will merge multiple KeyValue lists with the
+    same key into one instead of overwriting. You can se this to ``False`` if you are
+    using ``VDFDict`` and need to preserve the duplicates.
+    """
+    if not isinstance(b, bytes):
+        raise TypeError("Expected s to be bytes, got %s" % type(b))
+
+    return binary_load(BytesIO(b), mapper, merge_duplicate_keys, alt_format, raise_on_remaining)
+
+def binary_load(fp, mapper=dict, merge_duplicate_keys=True, alt_format=False, raise_on_remaining=False):
+    """
+    Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
+    binary VDF) to a Python object.
+
+    ``mapper`` specifies the Python object used after deserializetion. ``dict` is
+    used by default. Alternatively, ``collections.OrderedDict`` can be used if you
+    wish to preserve key order. Or any object that acts like a ``dict``.
+
+    ``merge_duplicate_keys`` when ``True`` will merge multiple KeyValue lists with the
+    same key into one instead of overwriting. You can se this to ``False`` if you are
+    using ``VDFDict`` and need to preserve the duplicates.
+    """
+    if not hasattr(fp, 'read') or not hasattr(fp, 'tell') or not hasattr(fp, 'seek'):
+        raise TypeError("Expected fp to be a file-like object with tell()/seek() and read() returning bytes")
+    if not issubclass(mapper, Mapping):
+        raise TypeError("Expected mapper to be subclass of dict, got %s" % type(mapper))
+
+    # helpers
+    int32 = struct.Struct('<i')
+    uint64 = struct.Struct('<Q')
+    int64 = struct.Struct('<q')
+    float32 = struct.Struct('<f')
+
+    def read_string(fp, wide=False):
+        buf, end = b'', -1
+        offset = fp.tell()
+
+        # locate string end
+        while end == -1:
+            chunk = fp.read(64)
+
+            if chunk == b'':
+                raise SyntaxError("Unterminated cstring (offset: %d)" % offset)
+
+            buf += chunk
+            end = buf.find(b'\x00\x00' if wide else b'\x00')
+
+        if wide:
+            end += end % 2
+
+        # rewind fp
+        fp.seek(end - len(buf) + (2 if wide else 1), 1)
+
+        # decode string
+        result = buf[:end]
+
+        if wide:
+            result = result.decode('utf-16')
+        elif bytes is not str:
+            result = result.decode('utf-8', 'replace')
+        else:
+            try:
+                result.decode('ascii')
+            except:
+                result = result.decode('utf-8', 'replace')
+
+        return result
+
+    stack = [mapper()]
+    CURRENT_BIN_END = BIN_END if not alt_format else BIN_END_ALT
+
+    for t in iter(lambda: fp.read(1), b''):
+        if t == CURRENT_BIN_END:
+            if len(stack) > 1:
+                stack.pop()
+                continue
+            break
+
+        key = read_string(fp)
+
+        if t == BIN_NONE:
+            if merge_duplicate_keys and key in stack[-1]:
+                _m = stack[-1][key]
+            else:
+                _m = mapper()
+                stack[-1][key] = _m
+            stack.append(_m)
+        elif t == BIN_STRING:
+            stack[-1][key] = read_string(fp)
+        elif t == BIN_WIDESTRING:
+            stack[-1][key] = read_string(fp, wide=True)
+        elif t in (BIN_INT32, BIN_POINTER, BIN_COLOR):
+            val = int32.unpack(fp.read(int32.size))[0]
+
+            if t == BIN_POINTER:
+                val = POINTER(val)
+            elif t == BIN_COLOR:
+                val = COLOR(val)
+
+            stack[-1][key] = val
+        elif t == BIN_UINT64:
+            stack[-1][key] = UINT_64(uint64.unpack(fp.read(int64.size))[0])
+        elif t == BIN_INT64:
+            stack[-1][key] = INT_64(int64.unpack(fp.read(int64.size))[0])
+        elif t == BIN_FLOAT32:
+            stack[-1][key] = float32.unpack(fp.read(float32.size))[0]
+        else:
+            raise SyntaxError("Unknown data type at offset %d: %s" % (fp.tell() - 1, repr(t)))
+
+    if len(stack) != 1:
+        raise SyntaxError("Reached EOF, but Binary VDF is incomplete")
+    if raise_on_remaining and fp.read(1) != b'':
+        fp.seek(-1, 1)
+        raise SyntaxError("Binary VDF ended at offset %d, but there is more data remaining" % (fp.tell() - 1))
+
+    return stack.pop()
+
+def binary_dumps(obj, alt_format=False):
+    """
+    Serialize ``obj`` to a binary VDF formatted ``bytes``.
+    """
+    buf = BytesIO()
+    binary_dump(obj, buf, alt_format)
+    return buf.getvalue()
+
+def binary_dump(obj, fp, alt_format=False):
+    """
+    Serialize ``obj`` to a binary VDF formatted ``bytes`` and write it to ``fp`` filelike object
+    """
+    if not isinstance(obj, Mapping):
+        raise TypeError("Expected obj to be type of Mapping")
+    if not hasattr(fp, 'write'):
+        raise TypeError("Expected fp to have write() method")
+
+    for chunk in _binary_dump_gen(obj, alt_format=alt_format):
+        fp.write(chunk)
+
+def _binary_dump_gen(obj, level=0, alt_format=False):
+    if level == 0 and len(obj) == 0:
+        return
+
+    int32 = struct.Struct('<i')
+    uint64 = struct.Struct('<Q')
+    int64 = struct.Struct('<q')
+    float32 = struct.Struct('<f')
+
+    for key, value in obj.items():
+        if isinstance(key, string_type):
+            key = key.encode('utf-8')
+        else:
+            raise TypeError("dict keys must be of type str, got %s" % type(key))
+
+        if isinstance(value, Mapping):
+            yield BIN_NONE + key + BIN_NONE
+            for chunk in _binary_dump_gen(value, level+1, alt_format=alt_format):
+                yield chunk
+        elif isinstance(value, UINT_64):
+            yield BIN_UINT64 + key + BIN_NONE + uint64.pack(value)
+        elif isinstance(value, INT_64):
+            yield BIN_INT64 + key + BIN_NONE + int64.pack(value)
+        elif isinstance(value, string_type):
+            try:
+                value = value.encode('utf-8') + BIN_NONE
+                yield BIN_STRING
+            except:
+                value = value.encode('utf-16') + BIN_NONE*2
+                yield BIN_WIDESTRING
+            yield key + BIN_NONE + value
+        elif isinstance(value, float):
+            yield BIN_FLOAT32 + key + BIN_NONE + float32.pack(value)
+        elif isinstance(value, (COLOR, POINTER, int, int_type)):
+            if isinstance(value, COLOR):
+                yield BIN_COLOR
+            elif isinstance(value, POINTER):
+                yield BIN_POINTER
+            else:
+                yield BIN_INT32
+            yield key + BIN_NONE
+            yield int32.pack(value)
+        else:
+            raise TypeError("Unsupported type: %s" % type(value))
+
+    yield BIN_END if not alt_format else BIN_END_ALT
+
+
+def vbkv_loads(s, mapper=dict, merge_duplicate_keys=True):
+    """
+    Deserialize ``s`` (``bytes`` containing a VBKV to a Python object.
+
+    ``mapper`` specifies the Python object used after deserializetion. ``dict` is
+    used by default. Alternatively, ``collections.OrderedDict`` can be used if you
+    wish to preserve key order. Or any object that acts like a ``dict``.
+
+    ``merge_duplicate_keys`` when ``True`` will merge multiple KeyValue lists with the
+    same key into one instead of overwriting. You can se this to ``False`` if you are
+    using ``VDFDict`` and need to preserve the duplicates.
+    """
+    if s[:4] != b'VBKV':
+        raise ValueError("Invalid header")
+
+    checksum, = struct.unpack('<i', s[4:8])
+
+    if checksum != crc32(s[8:]):
+        raise ValueError("Invalid checksum")
+
+    return binary_loads(s[8:], mapper, merge_duplicate_keys, alt_format=True)
+
+def vbkv_dumps(obj):
+    """
+    Serialize ``obj`` to a VBKV formatted ``bytes``.
+    """
+    data =  b''.join(_binary_dump_gen(obj, alt_format=True))
+    checksum = crc32(data)
+
+    return b'VBKV' + struct.pack('<i', checksum) + data

NSLPlugin.desktop

@@ -0,0 +1,19 @@
+[Desktop Entry]
+Comment[en_US]=
+Comment=
+Exec=/bin/bash -c 'export logged_in_home=$(eval echo "~$(whoami)"); curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/refs/heads/main/NSLPluginInstaller.sh | nohup /bin/bash'
+GenericName[en_US]=
+GenericName=
+Icon=uav
+MimeType=
+Name[en_US]=NSL Decky Plugin
+Name=NSL Decky Plugin
+Path=
+StartupNotify=true
+Terminal=false
+TerminalOptions=
+Type=Application
+X-DBUS-ServiceName=
+X-DBUS-StartupType=
+X-KDE-SubstituteUID=false
+X-KDE-Username=

NSLPluginInstaller.sh

@@ -0,0 +1,150 @@
+#!/bin/bash
+
+# ENVIRONMENT VARIABLES
+logged_in_user=$(logname 2>/dev/null || whoami)
+logged_in_home=$(eval echo "~${logged_in_user}")
+
+# Function to prompt for sudo password
+prompt_for_sudo() {
+  password=$(zenity --password --title="Authentication Required" --text="Please enter your password to proceed with installation/update.")
+
+  # Validate password
+  echo "$password" | sudo -S -v >/dev/null 2>&1
+  if [ $? -ne 0 ]; then
+    zenity --error --text="Incorrect password or sudo failed. Exiting."
+    exit 1
+  fi
+}
+
+# Function to switch to Game Mode
+switch_to_game_mode() {
+  echo "Switching to Game Mode..."
+  rm -rf "${logged_in_home}/.config/systemd/user/nslgamescanner.service"
+  unlink "${logged_in_home}/.config/systemd/user/default.target.wants/nslgamescanner.service"
+  systemctl --user daemon-reload
+  qdbus org.kde.Shutdown /Shutdown org.kde.Shutdown.logout
+}
+
+# Function to display Zenity messages
+show_message() {
+  zenity --notification --text="$1" --timeout=1
+}
+
+show_update_message() {
+  zenity --notification --text="Updating from $1 to $2..." --timeout=5
+}
+
+# Set URLs and paths
+REPO_URL="https://github.com/moraroy/NonSteamLaunchersDecky/archive/refs/heads/main.zip"
+GITHUB_URL="https://raw.githubusercontent.com/moraroy/NonSteamLaunchersDecky/main/package.json"
+LOCAL_DIR="${logged_in_home}/homebrew/plugins/NonSteamLaunchers"
+
+# Ask the user
+zenity --question --text="Would you like to install or update the NonSteamLaunchers Decky Plugin?" --title="Install/Update Plugin" --ok-label="Yes" --cancel-label="No"
+if [ $? -eq 1 ]; then
+  echo "User canceled the installation/update."
+  exit 0
+fi
+
+# Prompt for sudo once
+prompt_for_sudo
+
+# Check for existing directories
+DECKY_LOADER_EXISTS=false
+NSL_PLUGIN_EXISTS=false
+
+if [ -d "${logged_in_home}/homebrew/plugins" ]; then
+  DECKY_LOADER_EXISTS=true
+fi
+
+if [ -d "$LOCAL_DIR" ] && [ -n "$(ls -A "$LOCAL_DIR")" ]; then
+  NSL_PLUGIN_EXISTS=true
+fi
+
+# Version extraction from JSON (no jq)
+extract_version() {
+  grep -o '"version": *"[^"]*"' "$1" | sed 's/.*"version": *"\([^"]*\)".*/\1/'
+}
+
+fetch_github_version() {
+  version=$(curl -s "$GITHUB_URL" | grep -o '"version": *"[^"]*"' | sed 's/.*"version": *"\([^"]*\)".*/\1/')
+  echo "$version"
+}
+
+fetch_local_version() {
+  if [ -f "$LOCAL_DIR/package.json" ]; then
+    version=$(extract_version "$LOCAL_DIR/package.json")
+    echo "$version"
+  fi
+}
+
+compare_versions() {
+  if [ ! -f "$LOCAL_DIR/package.json" ]; then
+    return 1
+  fi
+
+  local_version=$(fetch_local_version)
+  github_version=$(fetch_github_version)
+
+  if [ "$local_version" == "$github_version" ]; then
+    return 0
+  else
+    return 1
+  fi
+}
+
+# Main logic
+set +x
+
+# Sanity checks
+if $DECKY_LOADER_EXISTS; then
+  if ! $NSL_PLUGIN_EXISTS; then
+    zenity --info --text="Decky Loader is detected but no NSL plugin found. It will now be injected into Game Mode."
+  fi
+else
+  zenity --error --text="Decky Loader not found. Please install it and re-run the script."
+  exit 1
+fi
+
+# Version check
+compare_versions
+if [ $? -eq 0 ]; then
+  show_message "No update needed. The plugin is already up-to-date."
+else
+  local_version=$(fetch_local_version)
+  github_version=$(fetch_github_version)
+  show_update_message "$local_version" "$github_version"
+
+  if $NSL_PLUGIN_EXISTS; then
+    show_message "NSL Plugin detected. Deleting and updating..."
+    echo "$password" | sudo -S rm -rf "$LOCAL_DIR"
+  fi
+
+  show_message "Creating base directory and setting permissions..."
+
+  echo "$password" | sudo -S mkdir -p "$LOCAL_DIR"
+  echo "$password" | sudo -S chmod -R u+rw "$LOCAL_DIR"
+  echo "$password" | sudo -S chown -R "$logged_in_user:$logged_in_user" "$LOCAL_DIR"
+
+  curl -L "$REPO_URL" -o /tmp/NonSteamLaunchersDecky.zip
+  unzip -o /tmp/NonSteamLaunchersDecky.zip -d /tmp/
+  cp -r /tmp/NonSteamLaunchersDecky-main/* "$LOCAL_DIR"
+
+  rm -rf /tmp/NonSteamLaunchersDecky*
+fi
+
+set -x
+cd "$LOCAL_DIR"
+
+
+
+
+
+
+# Ask to switch to Game Mode
+zenity --question --text="Plugin installed or updated. Do you want to switch to Game Mode now?" --title="Switch to Game Mode?" --ok-label="Yes" --cancel-label="No"
+if [ $? -eq 0 ]; then
+  switch_to_game_mode
+else
+  show_message "Remaining in Desktop Mode."
+fi

NonSteamLaunchers.desktop

@@ -0,0 +1,19 @@
+[Desktop Entry]
+Comment[en_US]=
+Comment=
+Exec=/bin/bash -c 'export logged_in_home=$(eval echo "~$(whoami)"); curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash'
+GenericName[en_US]=
+GenericName=
+Icon=uav
+MimeType=
+Name[en_US]=NonSteamLaunchers
+Name=NonSteamLaunchers
+Path=
+StartupNotify=true
+Terminal=false
+TerminalOptions=
+Type=Application
+X-DBUS-ServiceName=
+X-DBUS-StartupType=
+X-KDE-SubstituteUID=false
+X-KDE-Username=

README.md

@@ -1,46 +1,513 @@
-# NonSteamLaunchers
-Installs the latest GE-Proton and Installs Non Steam Launchers under 1 Proton prefix folder in your compatdata folder path called "NonSteamLaunchers".
+<p align="center"><em>“Who hath ascended up into heaven, or descended? who hath gathered the wind in his fists? who hath bound the waters in a garment? who hath established all the ends of the earth? what is his name, and what is his son's name, if thou canst tell?”</em><br><em>- Proverbs 30:4 (KJV)</em></p>
 
-No files are left in your "Downloads" they are deleted after installation. 
- 
-Currently Only Installs... 👀 
+<p align="center"><em>“For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life.”</em><br><em>“For God sent not his Son into the world to condemn the world; but that the world through him might be saved.”</em><br><em>- John 3:16-17 (KJV)</em></p>
 
-  ~Battle.net ✔️ 
+
+
+<p align="center">
+  <img
+    src="https://github-stats-extended.vercel.app/api?username=moraroy&theme=transparent&show_icons=true&hide_border=true&count_private=true"
+    width="48%"
+  />
+  <img
+    src="https://streak-stats.demolab.com?user=moraroy&theme=transparent&hide_border=true"
+    width="48%"
+  />
+  <img
+    src="https://github-stats-extended.vercel.app/api/top-langs/?username=moraroy&theme=transparent&hide_border=true&layout=compact"
+    width="48%"
+  />
+</p>
+
+
+
+<p align="center">
+  <img src="https://github.com/cchrkk/NSLOSD-DL/raw/main/logo.svg" width=40% height=auto
+</p>
+
+<h1 align="center">
+NonSteamLaunchers 🚀
+</h1>
+
+This script installs the latest UMU & GE-Proton and installs NonSteamLaunchers under one unique Proton prefix folder in your compatdata folder path called "NonSteamLaunchers" and adds them to your Steam Library. It will also add the games automatically in real time and will attempt to remove the games from your library in real time when you uninstall a game from a launcher. Collections for your games, launchers and web shortcuts will also be created/removed per launcher. Play time is tracked for all non steam games as well as boot videos downloaded per shortcut if any exist. Non-Downloadable Game theme music is also applied to your library. Metadata cards are also applied to your library automatically to give your non steam pages some life including Player Count if there are any on Steam!
+Special ".desktop" files will be created per shortcut allowing you to run the games and launchers outside of Steam as well.
+NSL can be used on Desktop or in Game Mode, and don't you worry,
+Local Saves and Cloud saves are supported, as well as multiplayer/online support (because you're using the launchers). Obviously, certain anticheat games will not work on linux enviroments; this is on a game to game basis.
+
+<h1 align="center">
+Features  ✅
+</h1>
+
+- Automatic installation of the most popular launchers for your Steam Deck and Steam Machine on SteamOS 🎮
+
+- Handle automatically the download and installation of your chosen launchers and the games, artwork included! ⌚️ 
+
+- MicroSD Support 💾 This script supports moving the entire prefix to a microSD. The script will install launchers and games to your SD card, and the launchers in Steam will point to the SD card installation. This allows you to save internal storage space on your Steam Deck!
+
+- ProtonTricks is compatible with NonSteamLaunchers default installation (one prefix). This will add a NonSteamLaunchers shorcut in your library...this shortcut doesnt do anything. All you have to do is simply "Hide this Game" in your Library. Right click its properties and choose "Manage" and "Hide this Game". You never have to worry about it again! If you were to open up ProtonTricks to fix any game or launcher it is now accessible! 
+
+- In case you didnt know, you can also choose to check mark "Separate App Id's" when installing a launcher, this will install all launchers in each of their own prefix. Automatically working with ProtonTricks!
+
+- Command Line Ready, you can call it from online, heres an example of installing a launcher ``` /bin/bash -c 'curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash -s -- "Epic Games"' ```
+
+- NSL can in fact be installed on many linux distros, feel free to try, here are some examples of some... Ubuntu LTS, ChimeraOS, Nobara and Arch Linux as well as any KDE Environments such as this opensuse - tumbleweed - wayland , if for any reason you find that NonSteamLaunchers installs perfectly or not, let me know!
+
+- RemotePlayWhatever is also bundled with NSL to allow for local and co-op play between non steam games, this is created by m4Engi, here is the repo [here](https://github.com/m4dEngi/RemotePlayWhatever)
+
+- Ludusavi is also pre-installed and setup for NSL for your games save backups. Not all games will work with this yet so bear this in mind when deleted or uninstalling games that are arent backed up yet, here is the repo [here](https://github.com/mtkennerly/ludusavi)
+
+In both versions of NonSteamLaunchers, Desktop or Decky, NSL will back up your games saves here automatically ```/home/deck/NSLGameSaves``` The Desktop Version only does this once, at the start of when the script is opened and you see the main options list. The decky plugin version does this on every manual scan that you do.
+
+
+-Pressing "Update Proton GE" in both the Desktop version or the Decky Plugin version, will give you the latest version of Proton GE and UMU. A patch will also be applied to allow Game Streaming from Discord in Game Mode.
+
+- Pressing the "Music Button" on the top left of your game page will enable/disble the Game Theme Feature. This simply attempts to play your games theme music in the client! You can even change the music if you dont like it with the paste button!. Clicking the button will hide and disable the feature.
+
+- [UMU Launcher](https://github.com/Open-Wine-Components/umu-launcher) is automatically used and is processed for each game and Launcher. Proton GE will be used where necessary.
+
+### Notes
+- With NSL youre able to send notes to each other and communicate to other NSL users via a hashtag in your note at the beginning, write #nsl and leave a space, and then type your actual note. The script will then look for that note and send it through the api and spit it back out for that non-steam game. Everyone who uses NSL will then receive it and it will be added to the "NSL Community Note". This is to allow people to have first hand information about their games right in front of them from others! Currently you can participate only if you send a note! Once you created a note, open up NonSteamLaunchers and press the ❤️. This is an expiremental feature so keep that in mind!
+
+### Logs
+- The logs for NSL are located here ``` /home/deck/Downloads/NonSteamLaunchers-install.log ``` and here ``` /home/deck/homebrew/logs/NonSteamLaunchers/ ``` if using the Decky Plugin Version.
+
+
+# As Seen on
+just to name a few!...there are much more videos and articles out there just wanted to share some resources on how to install and how the program works.
+## Videos
+- [Linus Tech Tips](https://www.youtube.com/watch?v=tdR-bxvQKN8&t=885s) (starting at 14:45)
+- [GameTechPlanet](https://www.youtube.com/watch?v=jE1qD3yzrks)
+- [NerdZap](https://www.youtube.com/watch?v=t2EzbKkbS1Q)
+- [Joserra y sus cosicas](https://www.youtube.com/watch?v=6ETxmbzRODQ)
+- [Steam Deck In Hand](https://www.youtube.com/watch?v=_j3HV6yyGjI)
+- [Steam Deck Gaming](https://www.youtube.com/watch?v=svOj4MTEAVc)
+- [BakaKuma](https://www.youtube.com/watch?v=QluZ3UGYoKo)
+- [SteamFlow](https://www.youtube.com/watch?v=aud5F6iwA0s)
+- [Hooandee - 6 Hour Video](https://www.youtube.com/watch?v=OGmwtSS-zoE&t=7023s) (starting at 1:57:23)
+- [Central Deck](https://www.youtube.com/watch?v=oIKL1JRn4cw)
+- [Goldenoptic Gaming](https://www.youtube.com/watch?v=dMnUn3U0dPE)
+- [Deck Ready](https://www.youtube.com/watch?v=9Ap_suofBV8&t=196s) (starting at 3:16)
+- [Steam Deck Checker](https://www.youtube.com/watch?v=vFRllG15jjs)
+- [SteamDeckHQ](https://www.tiktok.com/@steamdeckhq/video/7579970230265384223)
+- [ChoiTech](https://www.youtube.com/watch?v=ucrVWJNQ2rc)
+
+## Articles
+- [Gaming On Linux - Non-Steam Launchers Tool for Installing Popular Game Stores](https://www.gamingonlinux.com/2025/01/nonsteamlaunchers-tool-for-installing-popular-game-stores-working-on-better-desktop-linux-support/)
+- [Steam Deck HQ - Non-Steam Launchers New Update Community Notes](https://steamdeckhq.com/news/nonsteamlaunchers-new-update-community-notes/)
+- [Windows Central - How to Install Decky Loader on Steam Deck](https://www.windowscentral.com/gaming/how-to-install-decky-loader-on-steam-deck)
+- [Dexerto - Non-Steam Launchers on Steam Deck](https://www.dexerto.com/tech/nonsteamlaunchers-steam-deck-2808063/)
+- [MSN - Steam Deck: How to Install Epic Games Launcher with Decky Loader](https://www.msn.com/en-ca/news/technology/steam-deck-how-to-install-epic-games-launcher-with-decky-loader/ar-BB1pW1Ht)
+- [PCMAG - How to Install Third-Party Game Launchers on Steam Deck](https://www.pcmag.com/how-to/steam-deck-install-third-party-game-launchers)
+- [dadwithadeck - How to install Non-Steam Game Launchers on Steam Deck with NonSteamLaunchers](https://dadwithadeck.com/2025/11/24/how-to-install-non-steam-game-launchers-on-steam-deck-with-nonsteamlaunchers/)
+
+<p align="center">
+    ▶️ <b>YouTube Tutorials</b> 🡺🡺🡺
+    <a href="https://youtu.be/sxMmI8I9G_g">Decky Plugin</a> |
+    <a href="https://www.youtube.com/watch?v=HHdw4u5enrc">GamingOnSteam</a>
+    🡸🡸🡸 ▶️
+</p>
+
+<p align="center">
+    📖 <b>Step-by-step Articles</b> 🡺🡺🡺
+    <a href="https://steamdeckhq.com/news/nonsteamlaunchers-adds-scan-support-launchers">SteamDeckHQ Guide</a> |
+    <a href="https://gamingonsteam.com/2026/04/30/the-ultimate-guide-to-installing-any-launcher-on-your-steam-deck-nonsteamlaunchers/">GamingOnSteam Guide</a>
+    🡸🡸🡸 📖
+</p>
+
+---
+
+
+
+
+
+
+<h1 align="center">
+Currently Working On 👷‍♂️
+</h1>
+
+* Decky Loader Plugin is available [here](https://github.com/moraroy/NonSteamLaunchersDecky) and the pull request for it [here](https://github.com/SteamDeckHomebrew/decky-plugin-database/pull/677) and can be installed with this big button, only press this button if you have Decky Loader installed already
+* Working on Flatpak version
+
+ <p align="center">
+  <a name="download button" href="https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/releases/download/v3.9.6/NSLPlugin.desktop"><img src="https://user-images.githubusercontent.com/98482469/242361563-33f31d3d-9a69-4fca-a928-207a5d17a98f.png"  alt="Download NSL Decky Plugin" width="350px" style="padding-top: 15px;"></a>
+
+---
+
+**Windows Installation Steps**:
+
+1. Download the current Windows version of Decky Loader here [link](https://nightly.link/SteamDeckHomebrew/decky-loader/workflows/build-win/main/PluginLoader%20Win.zip)
+
+2. Download **NSLPluginWindows.exe** from [here](https://github.com/moraroy/NonSteamLaunchersDecky/releases).
+
+3. Run **NSLPluginWindows.exe** first. This will also create the necessary cef debugging file for Decky Loader.
+
+4. Run either No_console.exe or Plugin Loader.exe. You can also press Win + R, type shell:startup, and press Enter, then place the .exe there to launch Decky Loader on boot.
+5. Go into **Game Mode** or **Big Picture Mode** to see the Decky Loader plugin and NonSteamLaunchers.
+
+
+This setup will automatically add all your non-Steam games with artwork, correctly formatted for Windows. Only scanning will work; nothing else will function, so you can either auto-scan or manually scan your games.
+
+<h1 align="center">
+How to Install the Desktop Version 🔧
+</h1>
+
+<p align="center">
+  <a name="download button" href="https://github.com/moraroy/NonSteamLaunchers-On-Steam-Deck/releases/download/v3.9.1/NonSteamLaunchers.desktop"><img src="https://user-images.githubusercontent.com/98482469/242361563-33f31d3d-9a69-4fca-a928-207a5d17a98f.png"  alt="Download NonSteamLaunchers" width="350px" style="padding-top: 15px;"></a>
+</p>
+<!--- Thanks https://github.com/Heus-Sueh -->
+
+* Go to desktop mode, click the download button above and it should download the .desktop file in your Downloads folder.
+* Go to your downloads folder, click the NonSteamLaunchers icon, it will download and run the latest NonSteamLaunchers.sh from this repository and run it.
+* You will simply have to choose which launcher to install and let the script handle the rest. 💻 No files are left in your "Downloads" they are deleted after installation.
+* After running the script, you'll find the new launchers in your library under the non-steam tab. Click a launcher to see your installed games from that store, and launch them directly from Steam! If you have downloaded a game inside of your launcher, the scanner should have added it to our library, if not simply run a scan. Restart your Deck or quit and reopen Steam and the NSLGameScanner.service should add it to your library, even in gamemode! 🥳
+
+<p align="center">
+
+<h1 align="center">
+How to Run 🏃‍♂️
+</h1>
+
++ Select your launchers and hit OK. This will install your selected launchers. (Optionally, check "separate app ids", to install each launcher in its own prefix)
++ ❤️ = this will send and recieves any notes you have created to the community using the ```#nsl``` tag at the beginning of your note.
++ Uninstall = uninstalls the specific launchers and possibly its games too, each launcher uninstallation is different.
++ 🔍 = Pressing the magnifying glass will stop the NSL Scanner and prompt you to restart it if needed. When you open NSL the scanner will update from online then auto scan. So restarting it activates the real time service file for active scanning.
++ Start Fresh = Wipes all of NSL, all the prefixes, launchers, games, etc. Shortcuts will remain, and your game save backups, if any, at ```/home/deck/NSLGameSaves``` will not be deleted. This essentially "Uninstalls" NonSteamLaunchers.
++ Move to SD Card = moves each prefix to your SD Card, this is legacy code and probably still needs work.
++ Update Proton GE = this will update and install Proton GE and UMU if you dont have it already, the script attempts to do this on each launcher install but you can do it manually and help the script before hand if you want.
++ 🖥️ Off = this simply turns off your screen, useful if your doing long downloads to save battery.
++ NSLGameSaves = this will inject your game saves from ```/home/deck/NSLGameSaves``` into its correct locations using ludusavi into your launchers. Use this if you pressed "Start Fresh" and have downloaded your launchers again, dont download your games until you have pressed this button.
++ README = opens up this read me file.
+
+
+
+To stop the NSLGameScanner.service, open up NSL and hit "🔍" it will then ask you if you want to restart it, click no, and that's it.
+
+
+<h1 align="center">
+Command Lines 🫡
+</h1>
+The NSL script can be called from online via bash, heres an example of it installing a launcher
+
+```/bin/bash -c 'curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash -s -- "Epic Games"'```
+
+
+- All launchers can be installed by calling their name like this  ```"Epic Games"``` ```"Ubisoft Connect"``` etc.
+- All launchers can be uninstalled by calling their name like this ```"Uninstall Epic Games"``` ```"Uninstall Ubisoft Connect"``` etc.
+- Here is the list of commands that can also be called
   
-  ~Epic Games ✔️ 
+- ```"Start Fresh"``` ```"Update Proton-GE"``` ```"Stop NSLGameScanner"``` ```"Move to SD Card"```
+
+- The "Move to SD Card" function can only be called in this format
   
-  ~Gog ✔️ 
-  
-  ~Origin (only working way) ✔️ 
-  
-  ~Ubisoft ✔️ 
+```/bin/bash -c 'curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash -s -- "Move to SD Card" "EpicGamesLauncher"'```
 
-  ~The EA App ✔️ 
-  
-  ~Amazon Games Launcher ✔️ 
-  
-  ~Itch.io ✔️ 
+- The format of "EpicGamesLauncher" comes from the user choosing to either "Separate App ID's" or use the default installation prefix "NonSteamLaunchers" in the compatdata folder. This would be named differently for each launcher. Otherwise the command line would then only be 
 
-  ~Legacy Games ✔️ 
-
-
-Currently Working On These Things 👷‍♂️
-
-  
- ~Adding them to steam as shortcuts. 
+```/bin/bash -c 'curl -Ls https://raw.githubusercontent.com/moraroy/NonSteamLaunchers-On-Steam-Deck/main/NonSteamLaunchers.sh | nohup /bin/bash -s -- "Move to SD Card" "NonSteamLaunchers"'```
 
 
 
-INSTRUCTIONS 📝 
-In order to run it, grab the NonSteamLaunchers.sh file and throw it on the Desktop. 
 
-Please right click and go to properties > permissions and check mark "Is Executable". Double Click it and it should run.
+
+<h1 align="center">
+Supported Stores 🛍
+</h1>
+
+- Unreal Engine (via Epic Games) ✔️
+- Amazon Games Launcher ✔️
+- Battle.net ✔️
+- EA App ✔️
+- Epic Games ✔️
+- GOG Galaxy ✔️
+- Humble Games Collection ✔️
+- IndieGala ✔️
+- itch.io ✔️
+- Legacy Games ✔️
+- Rockstar Games Launcher ✔️
+- Ubisoft Connect ✔️
+- Glyph ✔️
+- Playstation Plus ✔️
+- VK Play ✔️
+- HoYoPlay ✔️
+- Nexon Launcher ✔️
+- Game Jolt Client ✔️
+- Artix Game Launcher ✔️
+- ARC Launcher ✔️
+- Pokémon Trading Card Game Live ✔️
+- Minecraft Launcher(Legacy) (Java Edition doesnt work but its for Dungeons) ✔️
+- PURPLE Launcher ✔️
+- Plarium Play ✔️
+- VFUN Launcher ✔️
+- Tempo Launcher ✔️
+- Antstream Arcade ✔️
+- Hytale ✔️
+- Big Fish Games Manager ✔️
+- Gryphlink ✔️
+- RemotePlayWhatever ✔️
+- NVIDIA GeForce NOW (Native Linux) ✔️
+- STOVE Client ✔️
+- Moonlight Game Streaming ✔️
+
+<h1 align="center">
+Supported Streaming Sites for games and as well as any website. 🌐
+</h1>
+
+- Website Shortcut Creator ✔️
+- Fortnite ✔️
+- Venge ✔️
+- PokéRogue ✔️
+- Xbox Game Pass ✔️
+- Better xCloud ✔️
+- GeForce Now ✔️
+- Amazon Luna ✔️
+- Boosteroid Cloud Gaming ✔️
+- Stim.io ✔️
+- WebRcade ✔️
+- WebRcade Editor ✔️
+- Afterplay.io ✔️
+- OnePlay ✔️
+- AirGPU ✔️
+- CloudDeck ✔️
+- JioGamesCloud ✔️
+- WatchParty ✔️
+- Rocketcrab ✔️
+- Netflix ✔️
+- Amazon Prime Video ✔️
+- Disney+ ✔️
+- Hulu ✔️
+- Tubi ✔️
+- Youtube ✔️
+- Twitch ✔️
+- Plex ✔️
+- Apple TV+ ✔️
+- Crunchyroll ✔️
+- Super Monkey Ball Online ✔️
+
+
+<h1 align="left">
+Finds Games Automatically
+</h1> 
+
+"NSLGameScanner.service" is also live when you use this script and continues after the script is closed and even works after your Steam Deck has restarted. This works in the background as a service file to automatically add your games to your library on every Steam restart. Currently adds:
+- Epic Games 🎮          💾 Full SD Card Support
+- Ubisoft Connect 🎮     💾 Full SD Card Support
+- EA App 🎮              💾 Full SD Card Support not sure 
+- Gog Galaxy 🎮          💾 Full SD Card Support
+- Battle.net 🎮
+- Amazon Games 🎮        💾 Full SD Card Support
+- Itch.io 🎮
+- Legacy Games 🎮
+- VK Play 🎮             💾 Full SD Card Support
+- HoYoPlay 🎮            💾 Full SD Card Support
+- Game Jolt Client 🎮    💾 Full SD Card Support
+- Minecraft Launcher 🎮
+- Waydroid Apps 🎮       📜 Your own script Support
+- Humble Games Collection 🎮 💾 Full SD Card Support
+- NVIDIA GeForce NOW (Native Linux App) - You must "Favorite" the game with the heart. Old favorites will not be picked up, you need to re-favorite.
+
+## Chrome Bookmarks
+The scanner will pick these up automatically. But in Chrome for Geforce Now and Boosteroid Cloud Gaming you need to change the name of the bookmark to your actual game name. Or you can press "Play" then Use "Ctrl + D" to edit the bookmarks for the Chrome Browser. As long as the game name is in the Bookmark Name.
+- Xbox Game Pass
+- GeForce Now
+- Amazon Luna
+- Boosteroid Cloud Gaming
+
+## Waydroid Detection
+If you're not in Official SteamOS or not using [this script](https://github.com/ryanrudolfoba/SteamOS-Waydroid-Installer) to install your Waydroid, you can put your own script for the App launch with the name "waydroid-cage.sh" in your own home directory and the scanner will do the rest. A good references for writing your own Script is in [here](https://github.com/SwallowKOR/cachyos-waydroid-gamemode).
 
 
 
-PLEASE NOTE!
-This does not add the Launchers to Steam as a non steam shortcut yet. This just installs them in your compatdata folder titled "NonSteamLaunchers" or their own "App Id".
+<!--- TODO: handful of broken icons (cf. 🡺🡺🡺 ); probably should remove or replace them with more common font to handle unicode-->
 
 
-⚠️ 
-DONT TOUCH THE GOG OR BATTLENET INSTALLS LET THEM INSTALL ON THEIR OWN THEY WILL REMOVE THEMSELVES.
+
+
+
+<h1 align="center">
+Contributing 🤝
+</h1>
+
+If you have any suggestions or improvements for this script, feel free to open an issue or submit a pull request.
+
+You can donate to me on [ko-fi](https://ko-fi.com/moraroy), [liberapay](https://liberapay.com/moraroy), or [sponsor me on github](https://github.com/sponsors/moraroy) or [patreon](https://patreon.com/moraroy)
+
+## Development Environment
+
+### Dev Container
+
+Install [Docker](https://docs.docker.com/compose/install/). Once installed, a clean dev environment with a Docker container [native to VSCode](https://code.visualstudio.com/docs/devcontainers/create-dev-container#_dockerfile) is spun up automatically. 
+
+* [Command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) (⇧⌘P) > Dev Containers: Reopen in Container
+* F5 for debug
+    * May need to select interpreter (e.g., `/opt/venv/bin/python`) first
+
+**VSCode Extensions (Dev Container)**
+
+* [Atom Keymap](https://marketplace.visualstudio.com/items?itemName=ms-vscode.atom-keybindings)
+* [Bash IDE](https://marketplace.visualstudio.com/items?itemName=mads-hartmann.bash-ide-vscode)
+* [Better Comments](https://marketplace.visualstudio.com/items?itemName=aaron-bond.better-comments)
+* [Docker](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
+* [EditorConfig](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig)
+* [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot)
+* [GitHub Copilot Chat](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat)
+* [gitignore](https://marketplace.visualstudio.com/items?itemName=codezombiech.gitignore)
+* [GitLens](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens)
+* [Markdown All in One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one)
+* [MS Visual Studio Live Share](https://marketplace.visualstudio.com/items?itemName=MS-vsliveshare.vsliveshare)
+* [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+* [Shellcheck](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck)
+
+### Manual Docker Instance
+
+If VSCode isn't present or only the python portion (cf. `__init__.py`) is being worked on, it's possible to just run a Docker container on its own. The container installs the correct version of python and any dependencies (e.g., ipython, rich) in `requirements.txt`.
+
+```bash
+# navigate to directory with Dockerfile
+cd .devcontainer/
+
+# build image
+docker build -t nonsteamlaunchers .
+
+# run container
+docker run -it --rm --name=mynonsteamlaunchers --workdir=/app -v $(pwd):/app nonsteamlaunchers bash
+
+# exit container
+exit
+```
+
+### Python virtual environment
+
+Useful for the python module(s), but extra compared to the [dev container](#dev-container) portion that covers the core shell script.
+
+```bash
+# create virtual environment
+python -m venv .venv
+
+# activate virtual environment
+source .venv/bin/activate
+
+# install dependencies
+python -m pip install -r requirements.txt 
+```
+
+### Pre-commit hooks
+
+Pre-commit hooks are installed via `pre-commit` and are run automatically on `git commit`. 
+
+Most importantly, `ruff` is used to lint all python code.
+
+* Install [pre-commit](https://pre-commit.com/#install)
+* Install pre-commit hooks
+    ```bash
+    pre-commit install
+    ```
+* Trigger pre-commit hooks automatically on `git commit`
+    ```bash
+    git add .
+    git commit -m "commit message"
+    ```
+* Bypass pre-commit hooks
+  * Sometimes, it's necessary to bypass pre-commit hooks. This can be done with the `--no-verify` flag.
+    ```bash
+    git commit -m "commit message" --no-verify
+    ```
+
+### Conventional Commits
+
+While not currently enforced, by using [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/#summary), it's possible to automatically generate changelogs and version numbers via [release-please](https://github.com/googleapis/release-please).
+
+To help with that, the [commitizen](https://commitizen-tools.github.io/commitizen/) tool can be installed. 
+
+To reduce some of the non-essential tooling, it's been removed from the repo, but feel free to install globally without checking into source control.
+
+### Formatting
+
+> **TL;DR**: The [Ruff formatter](https://astral.sh/blog/the-ruff-formatter) is an extremely fast Python formatter, written in Rust. It’s over 30x faster than Black and 100x faster than YAPF, formatting large-scale Python projects in milliseconds — all while achieving >99.9% Black compatibility.
+
+* While it runs automatically on commits, it can also be run manually
+    ```bash
+    # check for errors
+    ruff check .
+
+    # fix (some) errors automatically
+    ruff check . --fix
+    ```
+
+### Additional tooling
+
+#### mise
+
+* Install [mise](https://mise.jdx.dev/getting-started.html)
+  * `curl https://mise.run | sh`
+* Usage
+
+```bash
+# Install tools from repo
+mise install
+
+# List tools
+mise list uv
+
+# Install specific plugins
+mise use uv 0.8.8
+
+# set stable to system python
+mise use -g python
+```
+
+#### uv
+
+[uv](https://docs.astral.sh/uv/) is a faster drop-in replacement for `poetry` among other dependency and virtual environment managers written in rust.
+
+Common operations after [installing](https://docs.astral.sh/uv/getting-started/installation/):
+
+```bash
+# create venv w/pinned python version
+uv venv --python ">=3.11,<3.13"
+
+# activate venv
+source .venv/bin/activate
+
+# add deps
+uv add redis
+uv add --optional dev ruff
+
+# install optional (extra) deps
+uv pip install -r pyproject.toml --all-extras
+
+# pip freeze
+uv pip freeze > requirements.txt
+```
+
+#### Ruff
+
+[Ruff](https://docs.astral.sh/ruff/) is a linter and formatter compatible with various other tools like black, flake8, isort, etc. Like [uv](#uv), it's by Astral and written in rust.
+
+Install is handled by `uv`. These are a few of my favorite things 🎵:
+
+```bash
+# run linter
+ruff check .
+
+# formatter
+ruff format .
+
+# run tests
+ruff
+
+# run tests with coverage
+ruff --coverage
+
+# run tests with coverage and open in browser
+ruff --coverage --open
+```
+
+#### shellcheck
+
+`.shellcheckrc` excludes various [bash language rules](https://github.com/koalaman/shellcheck/wiki/Ignore#ignoring-one-or-more-types-of-errors-forever). Useful to control noise vs. legitimate warnings/errors when using the shellcheck extension.
+
+<h1 align="center">
+License 📝
+</h1>
+
+This project is licensed under the MIT License. See the `LICENSE` file for more information.