Jobsub ID 152330.47@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	152330.47@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	5345
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-03-03 01:17:10
Site	CH_UNIBE-LHEP
Entry	OSG_CH_UNIBE_LHEP_ce02
Last heartbeat	2025-03-03 01:28:55
From worker node	Hostname	wn-1-34.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-03-03 01:18:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_112_20231121T150911Z_gen_g4_detsim_hitreco.root
Outputting started	2025-03-03 01:20:54
Output files
Finished	2025-03-03 02:30:56
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/45169c08e824fec56611e2d7afb7427458b57090
printing env
Justin processors: 1
/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/45169c08e824fec56611e2d7afb7427458b57090/atmo_v2_maps.fcl
Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b6/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_112_20231121T150911Z_gen_g4_detsim_hitreco.root
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 2: 2441
Particle counts: 1, 0, 0, 0
Begin processing the 190th record. run: 65840069 subRun: 1 event: 22590 at 03-Mar-2025 02:20:49 CET
Boundary wire vector sizes: 174, 166, 131
minwire 0: 319
minwire 1: 2638
minwire 2: 4
Particle counts: 1, 0, 0, 0
Begin processing the 191st record. run: 65840069 subRun: 1 event: 22591 at 03-Mar-2025 02:20:49 CET
Boundary wire vector sizes: 226, 94, 185
minwire 0: 2184
minwire 1: 968
minwire 2: 2114
Particle counts: 1, 0, 0, 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 192nd record. run: 65840069 subRun: 1 event: 22592 at 03-Mar-2025 02:20:49 CET
Boundary wire vector sizes: 52, 61, 46
minwire 0: 1646
minwire 1: 840
minwire 2: 1845
Particle counts: 0, 0, 0, 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 1, 300
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 193rd record. run: 65840069 subRun: 1 event: 22593 at 03-Mar-2025 02:20:49 CET
Boundary wire vector sizes: 127, 123, 114
minwire 0: 792
minwire 1: 2363
minwire 2: 452
Particle counts: 1, 0, 0, 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 194th record. run: 65840069 subRun: 1 event: 22594 at 03-Mar-2025 02:20:49 CET
Boundary wire vector sizes: 6493, 7444, 4120
minwire 0: 605
minwire 1: 695
minwire 2: 371
Particle counts: 5, 4, 4, 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 195th record. run: 65840069 subRun: 1 event: 22595 at 03-Mar-2025 02:20:52 CET
Begin processing the 196th record. run: 65840069 subRun: 1 event: 22596 at 03-Mar-2025 02:20:52 CET
Boundary wire vector sizes: 3068, 2216, 1256
minwire 0: 1972
minwire 1: 100
minwire 2: 2367
Particle counts: 1, 1, 1, 0
Begin processing the 197th record. run: 65840069 subRun: 1 event: 22597 at 03-Mar-2025 02:20:53 CET
Boundary wire vector sizes: 633, 571, 501
minwire 0: 895
minwire 1: 910
minwire 2: 872
Particle counts: 1, 2, 0, 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 198th record. run: 65840069 subRun: 1 event: 22598 at 03-Mar-2025 02:20:53 CET
Boundary wire vector sizes: 281, 412, 355
minwire 0: 1972
minwire 1: 752
minwire 2: 1951
Particle counts: 0, 0, 0, 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 199th record. run: 65840069 subRun: 1 event: 22599 at 03-Mar-2025 02:20:53 CET
Boundary wire vector sizes: 211, 195, 223
minwire 0: 1221
minwire 1: 1694
minwire 2: 942
Particle counts: 1, 0, 0, 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Begin processing the 200th record. run: 65840069 subRun: 1 event: 22600 at 03-Mar-2025 02:20:53 CET
03-Mar-2025 02:20:53 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b6/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_112_20231121T150911Z_gen_g4_detsim_hitreco.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0205793     0.100509      0.565591      0.0912675     0.0870882       200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00513567    0.00772158     0.0305633    0.00584005    0.00482979       200    
map:cvnmap:CVNMapper                           0.0149843     0.0445787      0.21964      0.0400481     0.0317541       200    
[art]:TriggerResults:TriggerResultInserter    1.2201e-05    2.19305e-05   9.2525e-05    2.07345e-05   7.57288e-06      200    
end_path:zlib:CVNZlibMaker                    1.1743e-05     0.0479422     0.410959      0.0427142     0.0600618       200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2677.03 MB
  Peak resident set size usage (VmHWM): 1008.36 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0