Add full beamline tof lookup table

[nvaytet]

We compute full-beamline lookup tables and add them to the data registry.

Needs scipp/essreduce#308

[nvaytet]

In principle, this should not change existing results.
I will double-check and if that is indeed the case, maybe we don't need to make new files, but we can replace the old ones?

[nvaytet]

Update: In the new full beamline tables, we are more lenient with the variance threshold for masking. We therefore are keeping more events between the two frames, where neutron paths overlap.
This means that the results change a little.

Because of this, I will not replace the old ones with the new ones.

[SimonHeybrock]

Does that mean in practice we would not actually want to use a unified table? Or does the threshold need adjusting, e.g., balancing relative vs absolute variance, or factoring in some other parameter?

[nvaytet]

Sorry, I think this required more context.
When I first made the new tables, I got this

where I could see that at low distances, the values were being masked because of large uncertainties, which makes sense; the further you are away from the instrument, the more rays spread out and the lower the uncertainty.

So I changed the (single) threshold I had from 0.02 to 1.0, to obtain

Looks better at low distances, but the gap between the two frames towards the top has now been filled.

Maybe the threshold should be distance-dependent? Can we come up with a good rule/expression/dimensionality argument for a distance-dependent uncertainty threshold?

[nvaytet]

Does that mean in practice we would not actually want to use a unified table?

I think we definitely want to try and use a unified table if possible, for the simplicity it provides.

[SimonHeybrock]

Yes, that is what I understood and was referring to. My question is basically: We probably want to discard events in the gap (right?), while still supporting lookup at small distances. How can we reconcile the two?

[nvaytet]

Here is the difference in results between the 2 tables: as expected, only in the region between the two frames to we see differences.

[SimonHeybrock]

What is your conclusion?

[nvaytet]

The conclusion from in-person discussion was that we will leave it to the scientists to decide on the threshold, instead of hard-coding it in our tables.
So by default, we shouldn't mask anything, but return the table with the uncertainties (as it already done).

The threshold parameter should then become a parameter of the main reduction workflow, instead of being a parameter of the one that builds the LUT.

[nvaytet]

Update: after getting the latest changes from essreduce, the tables stored on disk are no longer masked according to uncertainty. The masking is done on-the-fly in the workflow, according to a user-set threshold parameter.

To not change the existing results of the DreamGeant4Workflow, the threshold has been set to 0.02 by default on the workflow. This currently means that computing a tof for the bunker monitor will likely yield mostly NaNs, as the table at low distances from the source is mostly masked out due to large uncertainties.

This will remain the case until we come up with alternatives to having a single threshold for the entire table.
Options could be:

• a threshold that depends on distance, either as a 1d scipp variable with distance dimension, or a function that return different values according to an input distance
• a threshold that depends on component: make the LookupTableRelativeErrorThreshold a generic that scopes RunType and MonitorType, and can be different for each component. (users will most likely not care about the distances, but know more what wavelength uncertaintly they are prepared to live with for the Mantle and for the Bunker monitor).

A default has not been set for the DreamWorkflow (apart from the Inf in the GenericTofWorkflow), so any notebook using the DreamWorkflow should adapt accordingly.

[nvaytet]

In the end, I decided to replace the tables and set the default threshold on the Dreamgeant4workflow to 0.02, so that the results are the same as before.