justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 86787.91@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID86787.91@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3843
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-25 16:30:53
SiteCA_Victoria
EntryDUNE_CA_Victoria_dune-condor_whole
Last heartbeat2024-10-27 10:47:24
From worker nodeHostnamedune--heprc--3456932580--269724142550093
cpuinfoIntel Core Processor (Broadwell, IBRS)
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-25 19:09:25
Input filesmonte-carlo-003843-000238
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-27 10:47:24
Saved logsjustin-logs:86787.91-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

1 XVs -> 10 XUVs
102 XUVs total
73 collection wire objects
102 potential space points
Neighbour search...
246 tests to find 144 neighbours
Iterating with no regularization...
Begin: 243003
0 200565
1 200098
2 200087
Now with regularization...
Begin: 133488
0 133482
Begin processing the 43rd record. run: 3843 subRun: 0 event: 11893 at 27-Oct-2024 10:42:43 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 9 XUs and 11 XVs -> 7 XUVs
C:0 T:3 17 XUs and 21 XVs -> 11 XUVs
C:0 T:5 10 XUs and 10 XVs -> 7 XUVs
C:0 T:7 9 XUs and 11 XVs -> 7 XUVs
C:0 T:9 15 XUs and 18 XVs -> 8 XUVs
C:0 T:11 14 XUs and 17 XVs -> 10 XUVs
C:0 T:13 13 XUs and 19 XVs -> 10 XUVs
C:0 T:15 3 XUs and 2 XVs -> 1 XUVs
C:0 T:17 11 XUs and 12 XVs -> 8 XUVs
C:0 T:19 6 XUs and 7 XVs -> 4 XUVs
C:0 T:21 10 XUs and 5 XVs -> 5 XUVs
C:0 T:23 26 XUs and 29 XVs -> 16 XUVs
94 XUVs total
64 collection wire objects
94 potential space points
Neighbour search...
200 tests to find 106 neighbours
Iterating with no regularization...
Begin: 267113
0 240694
1 240626
Now with regularization...
Begin: 176148
0 176144
Begin processing the 44th record. run: 3843 subRun: 0 event: 11894 at 27-Oct-2024 10:42:54 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 32 XUs and 29 XVs -> 18 XUVs
C:0 T:3 12 XUs and 12 XVs -> 6 XUVs
C:0 T:5 12 XUs and 13 XVs -> 8 XUVs
C:0 T:7 4 XUs and 4 XVs -> 2 XUVs
C:0 T:9 17 XUs and 17 XVs -> 7 XUVs
C:0 T:11 10 XUs and 8 XVs -> 5 XUVs
C:0 T:13 8 XUs and 8 XVs -> 4 XUVs
C:0 T:15 15 XUs and 15 XVs -> 9 XUVs
C:0 T:17 4 XUs and 2 XVs -> 2 XUVs
C:0 T:19 14 XUs and 11 XVs -> 8 XUVs
C:0 T:21 29 XUs and 23 XVs -> 13 XUVs
C:0 T:23 7 XUs and 5 XVs -> 3 XUVs
85 XUVs total
69 collection wire objects
85 potential space points
Neighbour search...
143 tests to find 58 neighbours
Iterating with no regularization...
Begin: 344483
0 319532
1 319532
Now with regularization...
Begin: 255175
0 255171
Begin processing the 45th record. run: 3843 subRun: 0 event: 11895 at 27-Oct-2024 10:43:08 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 19 XUs and 23 XVs -> 13 XUVs
C:0 T:3 18 XUs and 21 XVs -> 12 XUVs
C:0 T:5 15 XUs and 15 XVs -> 8 XUVs
C:0 T:7 5 XUs and 3 XVs -> 3 XUVs
C:0 T:9 5 XUs and 9 XVs -> 4 XUVs
C:0 T:11 12 XUs and 16 XVs -> 7 XUVs
C:0 T:13 6 XUs and 7 XVs -> 4 XUVs
C:0 T:15 6 XUs and 5 XVs -> 4 XUVs
C:0 T:17 9 XUs and 11 XVs -> 5 XUVs
C:0 T:19 20 XUs and 19 XVs -> 10 XUVs
C:0 T:21 21 XUs and 24 XVs -> 14 XUVs
C:0 T:23 14 XUs and 13 XVs -> 8 XUVs
92 XUVs total
74 collection wire objects
92 potential space points
Neighbour search...
182 tests to find 90 neighbours
Iterating with no regularization...
Begin: 359880
0 342009
1 341918
Now with regularization...
Begin: 274459
0 274454
Begin processing the 46th record. run: 3843 subRun: 0 event: 11896 at 27-Oct-2024 10:43:21 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 23 XUs and 29 XVs -> 18 XUVs
C:0 T:3 22 XUs and 25 XVs -> 13 XUVs
C:0 T:5 9 XUs and 9 XVs -> 8 XUVs
C:0 T:7 6 XUs and 7 XVs -> 3 XUVs
C:0 T:9 18 XUs and 14 XVs -> 11 XUVs
C:0 T:11 8 XUs and 11 XVs -> 4 XUVs
C:0 T:13 13 XUs and 11 XVs -> 8 XUVs
C:0 T:15 5 XUs and 4 XVs -> 4 XUVs
C:0 T:17 4 XUs and 3 XVs -> 2 XUVs
C:0 T:19 9 XUs and 12 XVs -> 3 XUVs
C:0 T:21 35 XUs and 41 XVs -> 21 XUVs
C:0 T:23 18 XUs and 18 XVs -> 11 XUVs
106 XUVs total
75 collection wire objects
106 potential space points
Neighbour search...
224 tests to find 118 neighbours
Iterating with no regularization...
Begin: 325142
0 249696
1 248777
2 248762
Now with regularization...
Begin: 172746
0 172736
Begin processing the 47th record. run: 3843 subRun: 0 event: 11897 at 27-Oct-2024 10:43:32 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 24 XUs and 27 XVs -> 14 XUVs
C:0 T:3 27 XUs and 29 XVs -> 17 XUVs
C:0 T:5 18 XUs and 14 XVs -> 9 XUVs
C:0 T:7 17 XUs and 17 XVs -> 10 XUVs
C:0 T:9 6 XUs and 5 XVs -> 3 XUVs
C:0 T:11 6 XUs and 5 XVs -> 4 XUVs
C:0 T:13 14 XUs and 18 XVs -> 7 XUVs
C:0 T:15 2 XUs and 0 XVs -> 0 XUVs
C:0 T:17 20 XUs and 22 XVs -> 15 XUVs
C:0 T:19 9 XUs and 10 XVs -> 6 XUVs
C:0 T:21 22 XUs and 27 XVs -> 13 XUVs
C:0 T:23 29 XUs and 25 XVs -> 13 XUVs
111 XUVs total
85 collection wire objects
111 potential space points
Neighbour search...
221 tests to find 110 neighbours
Iterating with no regularization...
Begin: 329141
0 307987
1 307969
Now with regularization...
Begin: 229731
0 229728
Begin processing the 48th record. run: 3843 subRun: 0 event: 11898 at 27-Oct-2024 10:43:45 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 21 XUs and 22 XVs -> 12 XUVs
C:0 T:3 27 XUs and 31 XVs -> 19 XUVs
C:0 T:5 6 XUs and 6 XVs -> 4 XUVs
C:0 T:7 2 XUs and 3 XVs -> 0 XUVs
C:0 T:9 15 XUs and 14 XVs -> 8 XUVs
C:0 T:11 6 XUs and 9 XVs -> 5 XUVs
C:0 T:13 15 XUs and 16 XVs -> 12 XUVs
C:0 T:15 8 XUs and 9 XVs -> 6 XUVs
C:0 T:17 3 XUs and 4 XVs -> 2 XUVs
C:0 T:19 6 XUs and 9 XVs -> 3 XUVs
C:0 T:21 21 XUs and 25 XVs -> 12 XUVs
C:0 T:23 22 XUs and 17 XVs -> 11 XUVs
94 XUVs total
70 collection wire objects
94 potential space points
Neighbour search...
204 tests to find 100 neighbours
Iterating with no regularization...
Begin: 220770
0 197180
1 196594
2 196562
Now with regularization...
Begin: 139679
0 139671
Begin processing the 49th record. run: 3843 subRun: 0 event: 11899 at 27-Oct-2024 10:43:55 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 18 XUs and 22 XVs -> 13 XUVs
C:0 T:3 25 XUs and 22 XVs -> 12 XUVs
C:0 T:5 9 XUs and 13 XVs -> 5 XUVs
C:0 T:7 1 XUs and 0 XVs -> 0 XUVs
C:0 T:9 6 XUs and 7 XVs -> 4 XUVs
C:0 T:11 11 XUs and 17 XVs -> 9 XUVs
C:0 T:13 10 XUs and 9 XVs -> 7 XUVs
C:0 T:15 21 XUs and 24 XVs -> 14 XUVs
C:0 T:17 5 XUs and 6 XVs -> 3 XUVs
C:0 T:19 1 XUs and 2 XVs -> 1 XUVs
C:0 T:21 24 XUs and 29 XVs -> 16 XUVs
C:0 T:23 46 XUs and 44 XVs -> 23 XUVs
107 XUVs total
83 collection wire objects
107 potential space points
Neighbour search...
217 tests to find 110 neighbours
Iterating with no regularization...
Begin: 344734
0 290280
1 289938
2 289928
Now with regularization...
Begin: 212323
0 212315
Begin processing the 50th record. run: 3843 subRun: 0 event: 11900 at 27-Oct-2024 10:44:07 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 25 XUs and 25 XVs -> 13 XUVs
C:0 T:3 17 XUs and 15 XVs -> 9 XUVs
C:0 T:5 21 XUs and 17 XVs -> 11 XUVs
C:0 T:7 12 XUs and 11 XVs -> 9 XUVs
C:0 T:9 1 XUs and 3 XVs -> 0 XUVs
C:0 T:11 0 XUs and 0 XVs -> 0 XUVs
C:0 T:13 5 XUs and 6 XVs -> 5 XUVs
C:0 T:15 15 XUs and 13 XVs -> 4 XUVs
C:0 T:17 6 XUs and 6 XVs -> 4 XUVs
C:0 T:19 19 XUs and 18 XVs -> 12 XUVs
C:0 T:21 11 XUs and 14 XVs -> 6 XUVs
C:0 T:23 25 XUs and 30 XVs -> 17 XUVs
90 XUVs total
66 collection wire objects
90 potential space points
Neighbour search...
182 tests to find 92 neighbours
Iterating with no regularization...
Begin: 203136
0 174693
1 174540
Now with regularization...
Begin: 109940
0 109927
terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TFile::Flush
  error flushing file ./RootOutput-2f09-acbf-4da3-c66e.root Input/output error
  ROOT severity: 5000
---- FatalRootError END

../justin-jobscript: line 167:  1383 Aborted                 lar -c reco1_supernova_dune10kt_1x2x6.fcl ${detsim_name}.root -o ${reco_name}.root -n -1
ERROR: lar (reco) exit code: 51134 
output files size: 
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238.root 109989334
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol.root 720374328
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim.root 1833641901
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim_hist.root 12750
../justin-jobscript: line 181: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
python: can't open file '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f6498424e95c5bbeffa30d19fd7ec57044da6683/pdjson2metadata': [Errno 2] No such file or directory
ERROR: metadata writing  52002 
output files size: 
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238.root 109989334
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol.root 720374328
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim.root 1833641901
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim_hist.root 12750
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root.ext.json 0
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241025T190932Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root.json 0
../justin-jobscript: line 219: return: can only `return' from a function or sourced script
Exiting with error
../justin-jobscript: line 226: return: can only `return' from a function or sourced script
justIN time: 2024-11-23 16:58:49 UTC       justIN version: 01.01.09