module documentation

network related helper functions, such as fetching from URLs

Function download Mostly just requests.get(), for byte-data download, with some options that make it a little more specifically useful for downloading.
def download(url: str, tofile_path: str = None, show_progress=None, chunk_size=131072, params=None, timeout=10): ΒΆ

Mostly just requests.get(), for byte-data download, with some options that make it a little more specifically useful for downloading.

The main addition is the option to stream-download to filesystem:

  • if tofile is not None, we stream-save to that file path, by name (and return None)
  • if tofile is None we return the data as a bytes object (which means we kept it in RAM, which may not be wise for huge downloads)

uses requests's stream=True, which seems chunked HTTP transfer, or just a TCP window? TOCHECK

if the HTTP response code is >=400 (actually if !response.ok, see requests's documentation), we raise a ValueError

Parameters
url:strthe URL to fetch data from
tofile_path:strIf this is non-None, we open it as a filename and _stream_ the download to that if we can.
show_progresswhether to print/show output on stderr while streadming downloads - default is not to.
chunk_sizechunk byte size when trying to stream.
paramspassed through to requests.get(): a dictionary, list of tuples or bytes to send as a query string.
timeouttimeout to pass on to requests.get
Returns
if tofile_path is given, we return None; if it is not given, we return a bytes object