Jobsub ID 92176.144@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 92176.144@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3936 |
| Stage ID | 1 |
| User name | imawby@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 43200 (12 hours) |
| Submitted time | 2024-11-06 16:48:03 |
| Site | UK_Edinburgh |
| Entry | DUNE_UK_SGridECDF_ce1 |
| Last heartbeat | 2024-11-06 17:34:21 |
| From worker node | Hostname | node2a07.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-06 16:55:51 |
| Input files | fardet-hd:nu_dune10kt_1x2x6_1098_768_20230826T013826Z_gen_g4_detsim_hitreco__20240227T210314Z_reco2.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-11-06 17:34:21 |
| Saved logs | justin-logs:92176.144-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
block should be empty ------------------
---------------------------------------------------------
Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/51/f0/nu_dune10kt_1x2x6_1098_768_20230826T013826Z_gen_g4_detsim_hitreco__20240227T210314Z_reco2.root
../justin-jobscript: line 64: 1398 Aborted (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 134
=== Start last 100 lines of lar log file ===
[2024-11-06 16:56:26.518915 +0000][Debug ][ExDbgMsg ][ 1398] [msg: 0x135f16f0] Assigned MsgHandler: 0x135f0410.
[2024-11-06 16:56:26.518988 +0000][Debug ][ExDbgMsg ][ 1398] [handler: 0x135f0410] Removed MsgHandler: 0x135f0410 from the in-queue.
[2024-11-06 16:56:26.519150 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Calling MsgHandler: 0x135f0410 (message: kXR_read (handle: 0x00000000, offset: 1363843816, size: 1918) ) with status: [SUCCESS] .
[2024-11-06 16:56:26.519211 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Destroying MsgHandler: 0x135f0410.
[2024-11-06 16:56:26.521701 +0000][Debug ][File ][ 1398] [0x5f67060@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/51/f0/nu_dune10kt_1x2x6_1098_768_20230826T013826Z_gen_g4_detsim_hitreco__20240227T210314Z_reco2.root?xrdcl.requuid=486faed5-ba0e-4c79-ae82-16755748322c] Sending a read command for handle 0x0 to dpmpool21.farm.particle.cz:21133
[2024-11-06 16:56:26.521783 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] MsgHandler created: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962368, size: 102) ).
[2024-11-06 16:56:26.521940 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Moving MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962368, size: 102) ) from out-queu to in-queue.
[2024-11-06 16:56:26.553064 +0000][Debug ][ExDbgMsg ][ 1398] [msg: 0x136694d0] Assigned MsgHandler: 0x13679450.
[2024-11-06 16:56:26.553156 +0000][Debug ][ExDbgMsg ][ 1398] [handler: 0x13679450] Removed MsgHandler: 0x13679450 from the in-queue.
[2024-11-06 16:56:26.553274 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Calling MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962368, size: 102) ) with status: [SUCCESS] .
[2024-11-06 16:56:26.553332 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Destroying MsgHandler: 0x13679450.
[2024-11-06 16:56:26.553409 +0000][Debug ][File ][ 1398] [0x5f67060@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/51/f0/nu_dune10kt_1x2x6_1098_768_20230826T013826Z_gen_g4_detsim_hitreco__20240227T210314Z_reco2.root?xrdcl.requuid=486faed5-ba0e-4c79-ae82-16755748322c] Sending a read command for handle 0x0 to dpmpool21.farm.particle.cz:21133
[2024-11-06 16:56:26.553474 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] MsgHandler created: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962470, size: 108) ).
[2024-11-06 16:56:26.553619 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Moving MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962470, size: 108) ) from out-queu to in-queue.
[2024-11-06 16:56:26.584659 +0000][Debug ][ExDbgMsg ][ 1398] [msg: 0x136695e0] Assigned MsgHandler: 0x13679450.
[2024-11-06 16:56:26.584728 +0000][Debug ][ExDbgMsg ][ 1398] [handler: 0x13679450] Removed MsgHandler: 0x13679450 from the in-queue.
[2024-11-06 16:56:26.584839 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Calling MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312962470, size: 108) ) with status: [SUCCESS] .
[2024-11-06 16:56:26.584895 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Destroying MsgHandler: 0x13679450.
[2024-11-06 16:56:26.584977 +0000][Debug ][File ][ 1398] [0x5f67060@root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/51/f0/nu_dune10kt_1x2x6_1098_768_20230826T013826Z_gen_g4_detsim_hitreco__20240227T210314Z_reco2.root?xrdcl.requuid=486faed5-ba0e-4c79-ae82-16755748322c] Sending a read command for handle 0x0 to dpmpool21.farm.particle.cz:21133
[2024-11-06 16:56:26.585041 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] MsgHandler created: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312937150, size: 25218) ).
[2024-11-06 16:56:26.585193 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Moving MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312937150, size: 25218) ) from out-queu to in-queue.
[2024-11-06 16:56:26.616252 +0000][Debug ][ExDbgMsg ][ 1398] [msg: 0x136694b0] Assigned MsgHandler: 0x13679450.
[2024-11-06 16:56:26.616318 +0000][Debug ][ExDbgMsg ][ 1398] [handler: 0x13679450] Removed MsgHandler: 0x13679450 from the in-queue.
[2024-11-06 16:56:26.616440 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Calling MsgHandler: 0x13679450 (message: kXR_read (handle: 0x00000000, offset: 1312937150, size: 25218) ) with status: [SUCCESS] .
[2024-11-06 16:56:26.616496 +0000][Debug ][ExDbgMsg ][ 1398] [dpmpool21.farm.particle.cz:21133] Destroying MsgHandler: 0x13679450.
daughterClusterListU.size(): 41
daughterClusterListV.size(): 32
daughterClusterListW.size(): 38
daughterClusterListU.size(): 41
daughterClusterListV.size(): 32
daughterClusterListW.size(): 38
daughterClusterListU.size(): 41
daughterClusterListV.size(): 32
daughterClusterListW.size(): 38
daughterClusterListU.size(): 41
daughterClusterListV.size(): 32
daughterClusterListW.size(): 38
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
ElectronInitialRegionRefinementAlgorithm: unable to find shower pfo list ShowerParticles3D
%MSG-w ShowerPCADirection: LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule 06-Nov-2024 16:56:28 GMT run: 1098 subRun: 1 event: 76802
0 spacepoints in shower, not calculating direction
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule 06-Nov-2024 16:56:28 GMT run: 1098 subRun: 1 event: 76802
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule 06-Nov-2024 16:56:28 GMT run: 1098 subRun: 1 event: 76802
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule 06-Nov-2024 16:56:28 GMT run: 1098 subRun: 1 event: 76802
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule 06-Nov-2024 16:56:28 GMT run: 1098 subRun: 1 event: 76802
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
: Rebuilding Dataset Default
<ERROR> : 0-th variable of the event is NaN --> return MVA value -999,
<ERROR> : that's all I can do, please fix or remove this event.
<ERROR> : 0-th variable of the event is NaN --> return MVA value -999,
<ERROR> : that's all I can do, please fix or remove this event.
<ERROR> : 0-th variable of the event is NaN --> return MVA value -999,
<ERROR> : that's all I can do, please fix or remove this event.
<ERROR> : 0-th variable of the event is NaN --> return MVA value -999,
<ERROR> : that's all I can do, please fix or remove this event.
<ERROR> : 0-th variable of the event is NaN --> return MVA value -999,
<ERROR> : that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 181, 195, 185
minwire 0: 1102
minwire 1: 975
minwire 2: 1126
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.358619,
Output 1: 0.125886, 0.000288111, 0.00962322, 0.864202,
Output 2: 0.0620438, 0.667463, 0.127696, 0.142798,
Output 3: 0.106737, 0.230003, 0.366444, 0.296816,
Output 4: 0.269002, 0.646356, 0.0834417, 0.00120019,
Output 5: 0.999219, 0.000686255, 7.69752e-05, 1.78327e-05,
Output 6: 0.996171, 0.00338615, 0.00032365, 0.000119225,
%MSG-e DUNEAna: dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule 06-Nov-2024 16:56:35 GMT run: 1098 subRun: 1 event: 76802
Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna: dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule 06-Nov-2024 16:56:35 GMT run: 1098 subRun: 1 event: 76802
Failed to find product with label caldata ... returning empty vector
%MSG
: Rebuilding Dataset Default
: Rebuilding Dataset Default
: Rebuilding Dataset Default
HT SHOWER: NOT ALL ORIENTATIONS COVERED
terminate called without an active exception
=== End last 100 lines of lar log file === 