Issues with fermipy when using 16YFL catalog

[marcesiles]

Hello, I am trying to use the 16YFL catalog (gll_psc_v41.fit) with:

• fermipy 1.4.1
• Fermitools / ScienceTools 2.5.1
• Python 3.11

During GTAnalysis initialization, fermipy crashes with:

KeyError: 'Spatial_Filename'

The traceback points to:

table['Spatial_Filename'][table['Spatial_Filename'] == 'None'] = ''

inside fermipy/catalog.py.

I checked the catalog directly with Astropy:

from astropy.table import Table

t = Table.read('gll_psc_v41.fit')
print('Spatial_Filename' in t.colnames)

and it returns:

False

So it seems the FL16Y schema does not contain the Spatial_Filename column expected by fermipy.

Has anyone else encountered this with FL16Y? Is there an officially recommended workaround about this?

At the moment I can manually patch the FITS table by adding an empty Spatial_Filename column, but I am not sure if that is the issue actually, and I consider it has something to do with fermipy.

Thanks.

[ndilalla]

Hi @marcesiles, the FL16Y source list is not yet officially supported in fermipy version 1.4.1. We are about to release a new version (likely 1.4.2 or 1.5.0). However, if you’d like to access this feature early, you can install the latest version from GitHub using pip:

pip install -U git+https://github.com/fermiPy/fermipy.git

After that, you should be able to use the FL16Y source list in the same way you used 4FGL-DR4 catalog. There is no need to download any additional data, as these are already included within fermipy.

[marcesiles]

Thanks, updating to the development version resolved the previous Spatial_Filename issue.

However, I now run into a different error during catalog loading:

astropy.table.np_utils.TableMergeError:
Left key column 'Extended_Source_Name' has missing values

The traceback points to _load_extended_tables() and join_tables() inside fermipy/catalog.py, during the join between the LAT_Point_Source_Catalog table and the ExtendedSources table.

The FITS structure itself seems correct and includes the ExtendedSources HDU.

Environment:

• fermipy 1.4.1+5.g4046
• ScienceTools 2.5.1
• Python 3.11

[ndilalla]

Thanks @marcesiles for the response. Could you please provide the code you are running in order to reproduce the error?

I don't run into any issues by loading the catalog in this way:

from fermipy.catalog import CatalogFL16Y
fl16y = CatalogFL16Y()

[marcesiles]

Thanks for checking.

The issue appears during GTAnalysis initialization rather than when directly loading CatalogFL16Y.

My script is:

from fermipy.gtanalysis import GTAnalysis

gta = GTAnalysis(
    '/home/marcelo.siles/regions/16YFL/config.yaml',
    logging={'verbosity': 3}
)

gta.setup()
gta.optimize()
gta.free_sources(distance=10.0, pars='norm')

gta.fit()

model = {
    'SpatialModel': 'PointSource',
    'Index': 2.0,
    'SpectrumType': 'PowerLaw'
}

m1 = gta.tsmap(
    'tsmap',
    model=model,
    make_plots=True,
    multithread=True
)

gta.plotter.make_tsmap_plots(m1, roi=gta.roi)

gta.write_roi('oneGeV_fit', make_plots=True)

and in the YAML:

model:
  catalogs:
    - '/home/marcelo.siles/regions/16YFL/gll_psc_v41.fit'

The traceback occurs during ROI model creation inside _load_extended_tables() / join_tables() in fermipy/catalog.py.

The relevant analysis settings are:

selection:
  emin    : 1000
  emax    : 1000000
  evclass : 128
  evtype  : 1

gtlike:
  irfs : 'P8R3_SOURCE_V3'
  edisp : True
  edisp_disable : ['isodiff']

[jirenliu]

This could be a conflict between different (newer, 7.2.0)version of astropy.
For my case, I reinstall astropy version 7.0.1 in the fermipy envirement:
pip install astropy==7.0.1
and also numpy
pip install numpy=1.26.4 .
And it solve my problem on "Left key column 'Extended_Source_Name' has missing values".