Skip to content

global ephemeris data cache#165

Open
nicHoch wants to merge 7 commits intoTCDSolar:mainfrom
nicHoch:anc_file_skip_auto_fido_search
Open

global ephemeris data cache#165
nicHoch wants to merge 7 commits intoTCDSolar:mainfrom
nicHoch:anc_file_skip_auto_fido_search

Conversation

@nicHoch
Copy link
Copy Markdown
Contributor

@nicHoch nicHoch commented Apr 10, 2025

add a internal ephemeris data cache. This allows to bypass hidden internal FIDO searches and downloads of the same data multiple times

All coordinate transform and imaging methods might internally search for correct ephemeris data for higher precession.
To get the ephemeris data a FIDO search (defaults to our stix data server) is uses and data gets downloaded.

The ephemeris data search utilises a global cache in order to avoid unnecessary network traffic or file loading. ANC ephemeris fits files are organised as daily files. The entire day is put to cache if touched once.

A user can prefetch or pinpoint specific ephemeris data files with the `load_ephemeris_fits_to_cache(filename)' method.

    from stixpy.coordinates.transforms import load_ephemeris_fits_to_cache

    start_time = Time("2023-01-01T12:00:00")
    end_time = Time("2023-01-01T12:30:00")
    query = Fido.search(
        a.Time(start_time, end_time),
        a.Instrument.stix,
        a.Level.anc,
        a.stix.DataType.asp,
        a.stix.DataProduct.asp_ephemeris,
    )

    aux_files = Fido.fetch(query["stix"])
    for file in aux_files:
        load_ephemeris_fits_to_cache(file)

@nicHoch nicHoch self-assigned this Apr 10, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 10, 2025

Codecov Report

❌ Patch coverage is 92.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.89%. Comparing base (c46bfcd) to head (8d9287d).

Files with missing lines Patch % Lines
stixpy/utils/table_lru.py 91.93% 5 Missing ⚠️
stixpy/product/sources/anc.py 87.09% 4 Missing ⚠️
stixpy/coordinates/transforms.py 94.11% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #165      +/-   ##
==========================================
+ Coverage   81.54%   81.89%   +0.35%     
==========================================
  Files          35       37       +2     
  Lines        2439     2558     +119     
==========================================
+ Hits         1989     2095     +106     
- Misses        450      463      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Member

@samaloney samaloney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea this is very different from what I was thinking but that not a bad thing, any idea on performance? As the reason I added the cache in the first place was because this code is call a lot during plotting operations.

It would be nice to be disable the cache maybe add a module level variable. Needs some narrative documentation could be added to top of the rst file and also needs a corresponding change log.

In the future if we have different version of the anc-ephem files and I wanted to compare the results from V0X to V0Y is this in the table or does the user have to remember which one the loaded last so will be in the cache? It would probably be useful to know which file the data came from anyway for debugging purposes anyway

Comment thread stixpy/coordinates/transforms.py Outdated
Comment thread stixpy/product/product_factory.py Outdated
Comment thread stixpy/utils/tests/test_table_lru_cache.py
Comment thread stixpy/utils/table_lru.py Outdated
@samaloney
Copy link
Copy Markdown
Member

@nicHoch I force pushed, sorry, so don't forget to pull the changes back down to your local branch.

@samaloney
Copy link
Copy Markdown
Member

pre-commit.ci autofix

@nicHoch nicHoch force-pushed the anc_file_skip_auto_fido_search branch from 6258511 to 8d9287d Compare January 30, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants