justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 87780.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID87780.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3843
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-27 18:34:08
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2024-10-27 21:35:13
From worker nodeHostnamedune001-2863503.0-lcg2713.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit216000 (60 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-27 18:35:25
Input filesmonte-carlo-003843-000238
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-27 21:35:13
Saved logsjustin-logs:87780.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

s -> 11 XUVs
C:0 T:23 13 XUs and 16 XVs -> 8 XUVs
107 XUVs total
83 collection wire objects
107 potential space points
Neighbour search...
211 tests to find 104 neighbours
Iterating with no regularization...
Begin: 283586
0 248325
1 248289
Now with regularization...
Begin: 178417
0 178409
Begin processing the 44th record. run: 3843 subRun: 0 event: 11894 at 27-Oct-2024 21:34:48 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 9 XUs and 10 XVs -> 3 XUVs
C:0 T:3 23 XUs and 25 XVs -> 15 XUVs
C:0 T:5 8 XUs and 8 XVs -> 5 XUVs
C:0 T:7 15 XUs and 15 XVs -> 10 XUVs
C:0 T:9 5 XUs and 9 XVs -> 4 XUVs
C:0 T:11 16 XUs and 14 XVs -> 11 XUVs
C:0 T:13 11 XUs and 15 XVs -> 8 XUVs
C:0 T:15 10 XUs and 8 XVs -> 5 XUVs
C:0 T:17 9 XUs and 7 XVs -> 5 XUVs
C:0 T:19 20 XUs and 12 XVs -> 8 XUVs
C:0 T:21 14 XUs and 18 XVs -> 11 XUVs
C:0 T:23 20 XUs and 15 XVs -> 10 XUVs
95 XUVs total
68 collection wire objects
95 potential space points
Neighbour search...
187 tests to find 92 neighbours
Iterating with no regularization...
Begin: 199542
0 160751
1 160232
2 160224
Now with regularization...
Begin: 100536
0 100531
Begin processing the 45th record. run: 3843 subRun: 0 event: 11895 at 27-Oct-2024 21:34:49 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 21 XUs and 17 XVs -> 11 XUVs
C:0 T:3 14 XUs and 15 XVs -> 9 XUVs
C:0 T:5 8 XUs and 12 XVs -> 7 XUVs
C:0 T:7 13 XUs and 7 XVs -> 4 XUVs
C:0 T:9 18 XUs and 17 XVs -> 11 XUVs
C:0 T:11 12 XUs and 13 XVs -> 9 XUVs
C:0 T:13 9 XUs and 10 XVs -> 5 XUVs
C:0 T:15 21 XUs and 17 XVs -> 11 XUVs
C:0 T:17 31 XUs and 32 XVs -> 21 XUVs
C:0 T:19 10 XUs and 10 XVs -> 5 XUVs
C:0 T:21 19 XUs and 16 XVs -> 13 XUVs
C:0 T:23 31 XUs and 28 XVs -> 14 XUVs
120 XUVs total
89 collection wire objects
120 potential space points
Neighbour search...
240 tests to find 118 neighbours
Iterating with no regularization...
Begin: 409993
0 366599
1 366437
Now with regularization...
Begin: 279690
0 279683
Begin processing the 46th record. run: 3843 subRun: 0 event: 11896 at 27-Oct-2024 21:34:49 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 51 XUs and 50 XVs -> 34 XUVs
C:0 T:3 17 XUs and 17 XVs -> 10 XUVs
C:0 T:5 11 XUs and 8 XVs -> 6 XUVs
C:0 T:7 5 XUs and 8 XVs -> 3 XUVs
C:0 T:9 6 XUs and 8 XVs -> 4 XUVs
C:0 T:11 10 XUs and 12 XVs -> 6 XUVs
C:0 T:13 8 XUs and 10 XVs -> 5 XUVs
C:0 T:15 2 XUs and 2 XVs -> 2 XUVs
C:0 T:17 26 XUs and 22 XVs -> 12 XUVs
C:0 T:19 9 XUs and 6 XVs -> 6 XUVs
C:0 T:21 31 XUs and 28 XVs -> 20 XUVs
C:0 T:23 24 XUs and 26 XVs -> 16 XUVs
124 XUVs total
88 collection wire objects
124 potential space points
Neighbour search...
282 tests to find 158 neighbours
Iterating with no regularization...
Begin: 255376
0 221912
1 221316
2 221279
Now with regularization...
Begin: 133494
0 133484
Begin processing the 47th record. run: 3843 subRun: 0 event: 11897 at 27-Oct-2024 21:34:49 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 15 XUs and 12 XVs -> 6 XUVs
C:0 T:3 27 XUs and 22 XVs -> 15 XUVs
C:0 T:5 3 XUs and 2 XVs -> 2 XUVs
C:0 T:7 11 XUs and 10 XVs -> 8 XUVs
C:0 T:9 7 XUs and 10 XVs -> 6 XUVs
C:0 T:11 12 XUs and 13 XVs -> 11 XUVs
C:0 T:13 8 XUs and 7 XVs -> 5 XUVs
C:0 T:15 7 XUs and 7 XVs -> 5 XUVs
C:0 T:17 2 XUs and 1 XVs -> 1 XUVs
C:0 T:19 17 XUs and 14 XVs -> 7 XUVs
C:0 T:21 25 XUs and 27 XVs -> 18 XUVs
C:0 T:23 21 XUs and 17 XVs -> 10 XUVs
94 XUVs total
68 collection wire objects
94 potential space points
Neighbour search...
190 tests to find 96 neighbours
Iterating with no regularization...
Begin: 373678
0 323905
1 323003
2 322989
Now with regularization...
Begin: 258527
0 258518
Begin processing the 48th record. run: 3843 subRun: 0 event: 11898 at 27-Oct-2024 21:34:50 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 17 XUs and 23 XVs -> 11 XUVs
C:0 T:3 23 XUs and 21 XVs -> 9 XUVs
C:0 T:5 10 XUs and 8 XVs -> 5 XUVs
C:0 T:7 11 XUs and 9 XVs -> 6 XUVs
C:0 T:9 0 XUs and 0 XVs -> 0 XUVs
C:0 T:11 7 XUs and 8 XVs -> 3 XUVs
C:0 T:13 6 XUs and 7 XVs -> 5 XUVs
C:0 T:15 10 XUs and 13 XVs -> 7 XUVs
C:0 T:17 3 XUs and 4 XVs -> 2 XUVs
C:0 T:19 10 XUs and 8 XVs -> 6 XUVs
C:0 T:21 16 XUs and 14 XVs -> 9 XUVs
C:0 T:23 12 XUs and 15 XVs -> 5 XUVs
68 XUVs total
50 collection wire objects
68 potential space points
Neighbour search...
144 tests to find 72 neighbours
Iterating with no regularization...
Begin: 235617
0 223954
1 223894
Now with regularization...
Begin: 170173
0 170167
Begin processing the 49th record. run: 3843 subRun: 0 event: 11899 at 27-Oct-2024 21:34:50 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 31 XUs and 27 XVs -> 15 XUVs
C:0 T:3 40 XUs and 43 XVs -> 24 XUVs
C:0 T:5 10 XUs and 10 XVs -> 7 XUVs
C:0 T:7 12 XUs and 14 XVs -> 8 XUVs
C:0 T:9 11 XUs and 10 XVs -> 8 XUVs
C:0 T:11 27 XUs and 27 XVs -> 18 XUVs
C:0 T:13 14 XUs and 12 XVs -> 7 XUVs
C:0 T:15 3 XUs and 2 XVs -> 2 XUVs
C:0 T:17 8 XUs and 10 XVs -> 6 XUVs
C:0 T:19 4 XUs and 3 XVs -> 2 XUVs
C:0 T:21 15 XUs and 15 XVs -> 10 XUVs
C:0 T:23 18 XUs and 21 XVs -> 11 XUVs
118 XUVs total
83 collection wire objects
118 potential space points
Neighbour search...
278 tests to find 156 neighbours
Iterating with no regularization...
Begin: 384001
0 331754
1 331049
2 331038
Now with regularization...
Begin: 234849
0 234842
Begin processing the 50th record. run: 3843 subRun: 0 event: 11900 at 27-Oct-2024 21:34:50 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 20 XUs and 19 XVs -> 16 XUVs
C:0 T:3 33 XUs and 38 XVs -> 22 XUVs
C:0 T:5 10 XUs and 10 XVs -> 3 XUVs
C:0 T:7 7 XUs and 7 XVs -> 3 XUVs
C:0 T:9 14 XUs and 11 XVs -> 8 XUVs
C:0 T:11 17 XUs and 15 XVs -> 11 XUVs
C:0 T:13 14 XUs and 17 XVs -> 10 XUVs
C:0 T:15 11 XUs and 13 XVs -> 6 XUVs
C:0 T:17 5 XUs and 8 XVs -> 4 XUVs
C:0 T:19 8 XUs and 6 XVs -> 5 XUVs
C:0 T:21 19 XUs and 22 XVs -> 11 XUVs
C:0 T:23 40 XUs and 39 XVs -> 19 XUVs
118 XUVs total
88 collection wire objects
118 potential space points
Neighbour search...
226 tests to find 108 neighbours
Iterating with no regularization...
Begin: 337493
0 294490
1 294398
Now with regularization...
Begin: 203146
0 203141
27-Oct-2024 21:34:52 UTC  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root"
27-Oct-2024 21:34:52 UTC  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0349795     0.0388306     0.0496591     0.0385863    0.00232958       50     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000320682   0.000431988   0.000837389   0.000431063   7.7282e-05       50     
reco:gaushit:GausHitFinder                     0.0280809     0.0305318     0.0329957     0.0304725    0.00113572       50     
reco:spsolve:SpacePointSolver                 0.00179504    0.00236206    0.00587346    0.00228416    0.000559422      50     
reco:hitfd:DisambigFromSpacePoints            0.000589836   0.000788108   0.00243305    0.000748475   0.000248577      50     
reco:rns:RandomNumberSaver                    1.7195e-05    2.17707e-05   0.000140552   1.88635e-05   1.72216e-05      50     
[art]:TriggerResults:TriggerResultInserter     8.824e-06    1.09061e-05   3.5914e-05     1.008e-05    4.19405e-06      50     
end_path:out1:RootOutput                       2.995e-06    3.71946e-06   1.6956e-05    3.4055e-06    1.91409e-06      50     
end_path:out1:RootOutput(write)               0.00375716    0.00445256    0.00663986     0.0043577    0.000499018      50     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1771.61 MB
  Peak resident set size usage (VmHWM): 862.282 MB
====================================================================================================
Art has completed and will exit with status 0.
python: can't open file '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f6498424e95c5bbeffa30d19fd7ec57044da6683/pdjson2metadata': [Errno 2] No such file or directory
ERROR: metadata writing  52002 
output files size: 
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238.root 109962974
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol.root 721434182
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim.root 1836687093
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim_hist.root 12750
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root 1840101146
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root.ext.json 44
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241027T183529Z_gen_000238_supernova_g4_halfActiveVol_detsim_reco.root.json 0
../justin-jobscript: line 219: return: can only `return' from a function or sourced script
Exiting with error
../justin-jobscript: line 226: return: can only `return' from a function or sourced script
justIN time: 2024-11-23 16:39:01 UTC       justIN version: 01.01.09