module documentation
network related helper functions, such as fetching from URLs
| Function | download |
Mostly just requests.get(), for byte-data download, with some options that make it a little more specifically useful for downloading. |
def download(url, tofile_path=None, show_progress=None, chunk_size=131072, params=None, timeout=10):
¶
Mostly just requests.get(), for byte-data download, with some options that make it a little more specifically useful for downloading.
The main addition is the option to stream-download to filesystem:
- if tofile is not None, we stream-save to that file path, by name (and return None)
- if tofile is None we return the data as a bytes object (which means we kept it in RAM, which may not be wise for huge downloads)
uses requests's stream=True, which seems chunked HTTP transfer, or just a TCP window? TOCHECK
| Parameters | |
url:str | the URL to fetch data from |
tofilestr | If this is non-None, we open it as a filename and _stream_ the download to that if we can. |
| show | whether to print/show output on stderr while downloading. |
| chunk | chunk byte size when trying to stream. |
| params | passed through to requests.get(): a dictionary, list of tuples or bytes to send as a query string. |
| timeout | timeout to pass on to requests.get |
| Returns | |
| byte if the HTTP response code is >=400 (actually if !response.ok, see requests's documentation), we raise a ValueError | |