ti_misp: add daily refetch option to threat attributes

[chemamartinez]

Proposed commit message

Added a new configuration option daily_refetch that, when enabled:

• Performs a daily full refetch: Every 24 hours, the cursor is reset and all attributes from the configured Initial Interval are re-fetched from MISP.
• Ensures decay state synchronization: The next daily refresh cycle will re-ingest attributes with their current decay scores from MISP and removed decayed attributes from destination indices.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes ti_misp: fixed expiration date isn't accurate enough for MISP attributes #15439

Screenshots

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chrisberkhout]

The daily refetch seems good, but setting a 24-hour expiration is confusing.

Since we already have a latest transform, can't we just make sure the transform picks up the latest version of each item (this may require adding an ingest time or something) and leave the decay score and decayed at the same?

Things still get updated as their decay info changes in MISP, and we have the same rules as usual for orphans.

[chemamartinez]

The daily refetch seems good, but setting a 24-hour expiration is confusing.

Since we already have a latest transform, can't we just make sure the transform picks up the latest version of each item (this may require adding an ingest time or something) and leave the decay score and decayed at the same?

Things still get updated as their decay info changes in MISP, and we have the same rules as usual for orphans.

I have a doubt with that approach, the idea for forcing a 24-hour expiration for ingested indicators with this new daily refresh mode, is that we make sure that indicators that have been decayed by MISP are dropped from the destination indices every day.

If we leave the decayed_at the same, it would mean every indicator that is ingested and not decayed it would be indexed with a decayed_at defined by ioc_expiration_duration which is 90 days by default. How could we remove that indicator before that date if MISP marked it as decayed in later iterations?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2371dca

History

• 💚 Build #35347 succeeded a6a046a

cc @chemamartinez

[elastic-vault-github-plugin-prod]

Package ti_misp - 1.41.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.41.0/