Accept-Encoding: gzip (take 2)

[shamilbi]

use "Accept-Encoding": "gzip" to reduce traffic
(take1: #86)

[grische]

Suggested change

	def download_req(index_url: str, req_name: str) -> tuple[dict, str]:
	def fetch_package_metadata(index_url: str, req_name: str) -> tuple[dict, str]:

Please use a more descriptive name, like fetch_package_metadata

[grische]

As this method is only used by Mirrorer, please make it a Mirrorer method, not a global function

[grische]

Suggested change

response_url = ""

This line here is unnecessary

[grische]

The comparison should be case-insensitive (GZIP is a valid encoding), and you could strip whitespaces in the process.

[grische]

A broken server or a corrupted or truncated body can raise a gzip.BadGzipFile or EOFError. We need to have a better error handling in case

A suggestion:

class MetadataError(RuntimeError):
    """The index returned an unusable metadata response."""

# and then inside the function's `with urllib.request.urlopen(...)` block:
try:
    if response.headers.get("Content-Encoding") == "gzip":
        with gzip.GzipFile(fileobj=response) as gzip_response:
            data = json.load(gzip_response)
    else:
        data = json.load(response)
except (gzip.BadGzipFile, EOFError, zlib.error, json.JSONDecodeError) as err:
    msg = f"Failed decoding metadata from {response.url}: {err}"
    raise MetadataError(msg) from err

and then expanding except (urllib.error.HTTPError, MetadataError) as err: in mirror().