Catch exceptions, back off and retry when creating annotations

[dave-finlay]

When starting promtimer against many cbcollects, the machine can get overwhelmed causing annotation creation to fail (see the backtrace below.) Promtimer should catch this exception, back-off and retry (and log to indicate that this is happening.)

In addition it would be good to have an option to bypass annotation creation exposed as a command-line argument.

/dev/workspace/promtimer/bin/promtimer -g /usr/local/Cellar/grafana/9.1.7/bin/ -p /usr/local/Cellar/prometheus/2.39.1/bin/prometheus
using grafana home path:/usr/local/Cellar/grafana/9.1.7/bin/
starting prometheus server on 0.0.0.0:13301 against cbcollect_info_ns_1@10.57.0.109_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.57.0.109.log
starting prometheus server on 0.0.0.0:13302 against cbcollect_info_ns_1@10.57.0.119_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.57.0.119.log

....snip... (to save space -- the rest complete like these ones)

starting prometheus server on 0.0.0.0:13374 against cbcollect_info_ns_1@10.78.101.0_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.78.101.0.log
starting grafana server (on localhost:13300; logging to .promtimer/logs/grafana.log)
starting browser using http://localhost:13300/dashboards
Potential annotations: 34
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 941, in connect
    self.sock = self._create_connection(
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/joe/dev/workspace/promtimer/bin/../promtimer/promtimer.py", line 308, in <module>
    main()
  File "/Users/joe/dev/workspace/promtimer/bin/../promtimer/promtimer.py", line 304, in main
    annotations.get_and_create_annotations(grafana_port, stats_sources, not args.cluster)
  File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 314, in get_and_create_annotations
    create_annotations(grafana_port, events)
  File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 297, in create_annotations
    existing = get_existing_annotations(top_level_url)
  File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 95, in get_existing_annotations
    response = util.execute_request(host_url, ANNOTATIONS_API_PATH, retries=5)
  File "/Users/joe/dev/workspace/promtimer/promtimer/util.py", line 100, in execute_request
    response = opener.open(request)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1377, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>