access coordinage manager without going through checks?

[eileenrmartin]

Hi folks,
Any tips on how I can access the coordinate manager to update the times without directly having to access the patch?
Thanks,
Eileen

Some background:
I'm trying to read a series of Sintela files (some Onyx and some Peta, stored in separate folders), which have a file format that is (at least mostly) prodml compliant. I can get it to index the spool and recognize that it's prodml without issue, and it gets almost all headers right (usually all headers). However, on the files that had a force end (just end of experiment, so recording was stopped), there's a slight rounding error mismatch on the end time and part end time. This also leads to errors in the coordinate manager for that patch (seems to always be 1 more sample than the data array has), and the time_step also prints out with a somewhat large error (e.g. might be 0.0005 seconds ideally, but prints out as 0.00058 seconds when I check the spool get_contents output).

As a solution, I'd like to set up an error try/catch that would update the time coordinates to have 1 fewer time sample if I encounter an error. Whenever I try to access any data or apply any methods to the patch (e.g. if I even try to do something as simple as print myspool[0]) it sets off the checks of the _validate_coords method of the coordinate manager. Because _validate_coords finds that the coordinates say there should be 1 more time sample than the data array really has, it won't let me move ahead with any additional actions directly related to the patch.

[d-chambers]

Hey @eileenrmartin,

Two thoughts. First, we should make the prodml parser more robust to correctly handle the file that is causing the issues. If you try to read the file by itself (eg dascore.spool('the_bad_file.hdf5')[0]), you get the same error no? If so, can you share it with so I can debug it? Conversely, I can probably set aside some time later this week or early next if you want to meet online to debug it.

Second, it might make sense to add a configuration somewhere to skip the coord manager checks. The only problem with doing that if the CoordManager is not fixed it will raise very cryptic indexing errors later on when the array isn't the expected shape. I have thought about a global configuration of some sort for DASCore, so that could go in there I suppose.

[gloverha]

@d-chambers Thanks for the quick response! I agree that it probably makes sense to make the prodml parser more robust for this type of issue. Im also talking to Sintela - I think this is an issue they should fix with their file writer, but its likely that people will still try to use dascore with these slightly off files.

We can send you examples of a peta file that works and one that fails. If you open the file with an h5 reader, you'll be able to see that the PartEndTime does not match the time of the last entry in the time variable for the file that fails to read. It is 1 sample interval higher. For example, this file was recorded at 2kHz and the PartEndTime metadata is '2025-06-18T19:43:35.646500', but the final entry in the actual RawDataTime is '2025-06-18 19:43:35.646000'

[d-chambers]

Hey @gloverha, sounds great. Please send me the files and I will see what I can do.

[gloverha]

https://drive.google.com/drive/folders/1xtZfjYL35LW_ZJcORnOZMcAeb8L0C735?usp=sharing

[d-chambers]

It should be fixed now, but please try the current master on your dataset. You will probably have to re-index since the old index will have the incorrect sampling rate on the problematic files.

spool.inder.index_path.unlink()
spool.update_index()

[gloverha]

Thanks! Will do. From: Derrick Chambers ***@***.***> Date: Friday, August 22, 2025 at 9:27 AM To: DASDAE/dascore ***@***.***> Cc: Glover, Hannah E ***@***.***>, Mention ***@***.***> Subject: Re: [DASDAE/dascore] access coordinage manager without going through checks? (Discussion #513) [This email originated from outside of OSU. Use caution with links and attachments.] It should be fixed now, but please try the current master on your dataset. You will probably have to re-index since the old index will have the incorrect sampling rate on the problematic files. spool.inder.index_path.unlink() spool.update_index() — Reply to this email directly, view it on GitHub<#513 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZDLDZ5TSRKOKLI4XJT5EM33O4ZFNAVCNFSM6AAAAACEM335RGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMJZGEYDOOA>. You are receiving this because you were mentioned.

[gloverha]

I confirmed that the current master works with the files that failed before. Thanks!