Improved caching for urls

- When there is no file at the destination yet, we have no modification time for the `If-Modified-Since`-Header. In this case trust the cache to make the right decision to either serve a cached version or to refresh from origin. This should help with mass-deployment scenarios where you want to use a local cache to relieve your uplink.
- If you don't trust the cache to make the right decision you can still force it to refresh by providing the `force: yes` option.
This commit is contained in:
Michael Riss 2016-08-30 11:54:22 +02:00 committed by Brian Coca
parent 931939e2f8
commit c05bad9f74

View file

@ -848,13 +848,14 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
if http_agent: if http_agent:
request.add_header('User-agent', http_agent) request.add_header('User-agent', http_agent)
# if we're ok with getting a 304, set the timestamp in the # Cache control
# header, otherwise make sure we don't get a cached copy # Either we directly force a cache refresh
if last_mod_time and not force: if force:
request.add_header('cache-control', 'no-cache')
# or we do it if the original is more recent than our copy
elif last_mod_time:
tstamp = last_mod_time.strftime('%a, %d %b %Y %H:%M:%S +0000') tstamp = last_mod_time.strftime('%a, %d %b %Y %H:%M:%S +0000')
request.add_header('If-Modified-Since', tstamp) request.add_header('If-Modified-Since', tstamp)
else:
request.add_header('cache-control', 'no-cache')
# user defined headers now, which may override things we've set above # user defined headers now, which may override things we've set above
if headers: if headers: