Skip to content

Command line utility to ingest old events#11

Open
sfoale wants to merge 3 commits intomasterfrom
ingest-old-events
Open

Command line utility to ingest old events#11
sfoale wants to merge 3 commits intomasterfrom
ingest-old-events

Conversation

@sfoale
Copy link
Copy Markdown

@sfoale sfoale commented Nov 14, 2025

This PR adds a new selective ingestion tool.

The selective ingestion tool (selective_ingest.py) adds a script for catching up on historical alerts with three modes: list, ingest and GraceDB. Supports dry-run and is documented in the README.

Listener Improvements

  • Force IPv4 for Kafka connections to fix IPv6 unreachability issues
  • Add KAFKA_OFFSET_RESET environment variable support for controlling consumer offset behaviour

Configuration

  • Backwards-compatible config path resolution: tries production path (/home/azureuser/cron/listener_config.json) first, then local path (./listener_config.json), then falls back to environment variables

All changes are designed to be backwards compatible with existing production deployments.

…available alert files for each event and ingest them.

Also document how to use this utility in the README.
@sfoale sfoale requested review from cmccully and swyatt7 November 14, 2025 10:52
Copy link
Copy Markdown
Collaborator

@swyatt7 swyatt7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything lgtm.

Should probably use datetime.datetime.fromisoformat(self.gwalert_dict["time_of_signal"]) instead of the custom time. It's pretty robust.

That's it.

Comment on lines +17 to +35
def parse_datetime_flexible(datetime_str):
"""
Parse datetime string with or without fractional seconds.
Handles both formats: '2023-01-01T12:34:56.123456' and '2023-01-01T12:34:56'
"""
formats = [
"%Y-%m-%dT%H:%M:%S.%f", # With fractional seconds
"%Y-%m-%dT%H:%M:%S", # Without fractional seconds
]

for fmt in formats:
try:
return datetime.datetime.strptime(datetime_str, fmt)
except ValueError:
continue

# If none work, raise the error
raise ValueError(f"Unable to parse datetime string: {datetime_str}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There has been a function added to datetime api that does this:

mydate = datetime.datetime.fromisoformat(date_string)


try:
tos = datetime.datetime.strptime(self.gwalert_dict["time_of_signal"], "%Y-%m-%dT%H:%M:%S.%f")
tos = parse_datetime_flexible(self.gwalert_dict["time_of_signal"])
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datetime.datetime.fromisoformat(self.gwalert_dict["time_of_signal"])

print('Calculating LAT contours')

tos = datetime.datetime.strptime(self.gwalert_dict["time_of_signal"], "%Y-%m-%dT%H:%M:%S.%f")
tos = parse_datetime_flexible(self.gwalert_dict["time_of_signal"])
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datetime.datetime.fromisoformat(self.gwalert_dict["time_of_signal"])

sfoale added a commit that referenced this pull request Dec 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants