Event reconstruction (DL1 parameters): Crashes

[Elisa-Visentin]

Hi,
I think I still have some problems with (simulated) event reconstruction (and I can't solve them by myself, sorry):

Some events raise this error:

Traceback (most recent call last):
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 486, in <module>
    main()
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 477, in main
    mc_dl0_to_dl1(args.input_file, args.output_dir, config)
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 329, in mc_dl0_to_dl1
    timing_params = timing_parameters(
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/ctapipe/image/timing.py", line 82, in timing_parameters
    beta, error = lts_linear_regression(x=longi, y=peak_time, samples=5)
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/numba/np/linalg.py", line 751, in _check_finite_matrix
    raise np.linalg.LinAlgError(
numpy.linalg.LinAlgError: Array must not contain infs or NaNs.

This seems to be a non-handled exception, but I don't know how I made it happen...

Other events cause a more 'exotic' crash:

Traceback (most recent call last):
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 486, in <module>
    main()
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 477, in main
    mc_dl0_to_dl1(args.input_file, args.output_dir, config)
  File "/home/evisentin/Desktop/lstchain/magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_mc_dl0_to_dl1.py", line 220, in mc_dl0_to_dl1
    for event in event_source:
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/ctapipe/io/eventsource.py", line 323, in __iter__
    for event in self._generator():
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/ctapipe/io/simteleventsource.py", line 705, in _generator
    yield from self._generate_events()
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/ctapipe/io/simteleventsource.py", line 717, in _generate_events
    for counter, array_event in enumerate(self.file_):
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/simtel/simtelfile.py", line 291, in iter_array_events
    self.next_low_level()
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/simtel/simtelfile.py", line 163, in next_low_level
    self.current_array_event = parse_array_event(
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/simtel/simtelfile.py", line 428, in parse_array_event
    telescope_events[o.telescope_id] = parse_telescope_event(o)
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/simtel/simtelfile.py", line 487, in parse_telescope_event
    event['pixel_timing'] = o.parse()
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/simtel/objects.py", line 1280, in parse
    pixel_timing['pixel_list'] = read_array(
  File "/home/evisentin/mambaforge/envs/lstchain/lib/python3.9/site-packages/eventio/tools.py", line 41, in read_array
    return np.frombuffer(f.read(count * dt.itemsize), count=count, dtype=dt)
ValueError: buffer is smaller than requested size

The script used to process them is the same and they belong to the same MC production. Only some of the events in a few simulation runs cause them. Both ctapipe 0.12 and 0.19 exhibit the same crashes.
What I'm doing wrong?

Thank you

[maxnoe]

The second error is most likely a broken simulation file (simtel array crashed and didn't finish writing the file), you should report that to whomever created that file, there is nothing really we can do about it.

The second one is something else though. It seems you are computing timing parameters on an event where the hillas parameters are invalid (length or psi nan?).

You need to skip those events.

Note that ctapipe-process does this automatically by using a QualityCriteria component.

You seem to re-implement ctapipe-process in another package, something that has happened before (e.g. in lstchain, protopipe, ....) and you then keep running in all the edge cases that we already solved when implementing that Tool here in ctapipe.

This is not what should be done. If you for some reason can't use ctapipe-process, please open issues with the reasons why that is not possible so we can improve.

[Elisa-Visentin]

Thank you.
As for the timing parameters, I will try (temporarily) to fix a bit the scripts (maybe I can fix it for all of my runs): some updates have to be implemented before upgrading the pipeline to ctapipe-process...
As for broken simulations, I must find out who created the files.
Thank you really much, I will work on your suggestions

[Elisa-Visentin]

Yes, timing parameters where 'computed' on events without psi 🤦. Now we managed to fix it .
Thank you (and sorry for bothering you)