Implement Geometric 3D Reconstruction + 3D CNNs

[bryankim96]

See here for a general description of the corresponding Google Summer of Code (GSoC) project created through OpenAstronomy: here

This issue is for discussing the development/implementation of the features described in the project. The project as described above will probably include the following tasks:

1. Modifications to the DataLoader class (and the data storage format) to handle truly 3D data. (see ctlearn/data_loader.py).
2. Modifications to the DataProcessor class (see ctlearn/data_processing.py) to adapt the existing functionality (cropping, cleaning, normalization) to 3D inputs.
3. Implementation of an entirely new module implementing a geometric 3D reconstruction method (from multiple shower images to a single 3D shower representation). A memo outlining some ideas for a proposed approach can be provided on request.
4. Implementation of a 3D CNN model in Tensorflow in the supported model format (see ctlearn/default_models.py for examples).
5. Implementation of any scripts required for visualization of results/verification of data/etc.

Any discussion of ideas/implementation for this task can go below!

[orionpax00]

Hi @aribrill and @bryankim96, I would love to contribute to this issue.

After going through the Idea page, I started searching for 3D Reconstruction algorithm and ended up with a research paper describing some of the reconstruction method (you can find it here).

I need help in Understanding the point,

Read and understand the 3D geometric reconstruction method laid out in the Event Reconstruction for VERITAS note.

If I'm not wrong the mentioned VERITAS is an ground-based gamma-ray observatory, so where to look for it's 3D geometric reconstruction method

[qi-feng]

@orionpax00 Hi and thanks for your interest.
If you search in the paper you found, you an see VERITAS mentioned in there. It's an array of telescopes: https://veritas.sao.arizona.edu
One paper on 3D reconstruction is http://hal.in2p3.fr/in2p3-00127358/document

[orionpax00]

Thanks @qi-feng,
I'll try to implement the method and soon share the results.

[dreammify]

Greetings, travellers

I've been looking through the OpenAstronomy ideas for this year's GSoC and this project piqued my interest in particular. I've read through the paper mentioned above and I can say that I have some grasp of the concepts mentioned (Although the physics still elude me, being a computer sciences student, I'm still willing to learn!). Do you have any pointers about how to proceed?

[bryankim96]

Hi @orionpax00 and @panosis,

Thanks for the interest!

Read and understand the 3D geometric reconstruction method laid out in the Event Reconstruction for VERITAS note.

I was referring to a memo written for VERITAS a while ago that describes several event reconstruction methods (one of which I thought would be a good starting point for the ray back-propagation approach proposed for this project). Because it's an internal document and not part of a publication, it was suggested to me that I shouldn't post it anywhere public (I don't think it would be a big deal but I'm trying to stay on the safe side). However, if you post your email address (or another way to reach you), I can send you the memo as a PDF.

Do you have any pointers about how to proceed?

My general suggestion would be the following:

If you have a general idea of what the project is about and what you need to do, I would encourage you to look into the literature (such as the paper @qi-feng linked or the one @orionpax00 found) for ideas about how you could approach the problem. Another thing you should do is take a close look at the existing code in the repo so you can see how things are organized and get a concrete idea about how you would implement your approach within the existing framework. Once you have a reasonably well thought-out approach and a plan for implementation, you should prepare and submit an application through the official Google Summer of Code process (which opened today).

As you're doing the stuff mentioned above, if you come up with any specific questions about the package or the project (i.e. details about what the data looks like, questions about specific parts of the code relevant to the project, questions about the science background), feel free to post them here and we'll try our best to answer them.

[orionpax00]

Hi @bryankim96,
Thanks for your clarification,
dkumar@ce.iitr.ac.in is my email id. You can send that memo here.

[dreammify]

Hello~
My e-mail is panoslantavos@gmail.com, you can send any reference there :)

[orionpax00]

Hello @bryankim96, @aribrill, @qi-feng

I read the memo on Event Reconstruction and Parameterization for VERITAS, especially the 3D reconstruction part. I got the mathematics behind the algorithms but I'm still confused for the input vector and output 3D matrix on computational ground. Like what will be the input for the function.

As far as I got Inputs will be different images captured by different telescopes along with the coordinates of telescope.

please help

[bryankim96]

@orionpax00 Yeah, that's correct. The inputs will be telescope images (2D matrix) showing the amount of charge deposited in each camera pixel. The telescope positions can be represented by vectors. In theory you could also have access to the full time-evolution of the camera image (so the pulse shape as a function of time) which would be a 3D matrix.

The output will be a 3D matrix (volume) representing the shower in 3D space. You have a lot of flexibility about what this 3D matrix is/how it will be constructed. It certainly won't be an accurate, "true" 3D model of the shower. However, it should ideally preserve as much of the spatial and morphological structure of the shower as possible, so that it serves as a good input to a 3D CNN.

[Amanbhandula]

Hi mentors! @bryankim96 @aribrill @qi-feng I am Aman. I have been in touch with @bryankim96 regarding this project. I am from Planetary Science background and doing my minors in Applied Machine Learning. I am very interested in Astrodynamics and would like to do MS in Astrophysics. I have done some projects related to Planetary science and Deep Learning. Also, I had done an intern, in which I have implemented 3D Pose Estimation. I have been following this project since the beginning of GSoC'19. I have read the memo sent by @bryankim96 , the paper mentioned by @qi-feng and some other papers on 3D Event Reconstruction. I have worked with the telescope data in past and I believe that I have a very clear understanding of what we will be implementing in this project. I am writing my proposal and will complete it by tonight. I have read different papers and I have doubt on which method we should use. @bryankim96 @aribrill @qi-feng Please tell me where should I discuss about these methods of 3D Event Reconstruction. Since my doubts are very specific so it would be better to discuss them on slack channel or on email.
Thank You in advance.