justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108702.25@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108702.25@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:04:38
SiteUS_UConn-HPC
EntryGLUEX_US_UConn-HPC_osgce
Last heartbeat2024-11-23 04:01:09
From worker nodeHostnamecn425
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:12:31
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root
JobscriptExit code0
Real time48m (2905s)
CPU time45m (2748s = 94%)
Outputting started2024-11-23 04:00:57
Output files
Finished2024-11-23 04:01:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

wer@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:00:52 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:00:53 EST run: 6406796 subRun: 1 event: 56500
Comparing two wires in the same plane: return failure
%MSG
247
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 569!
22-Nov-2024 23:00:54 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_data_2024-11-23T_031235Z.root"
22-Nov-2024 23:00:54 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/8c/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.317409       27.1053       258.709       11.3394       41.7078        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00843373     0.0585822     0.150915      0.0554981     0.0293022       100    
reco:linecluster:LineCluster                           0.0403049     0.147064      0.918311      0.117688      0.123459        100    
reco:trajcluster:TrajCluster                           0.0202687     0.337477       9.34286      0.0322853      1.12597        100    
reco:pandora:StandardPandora                           0.0342589      2.0041        11.8563       1.68862       1.3696         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000147849   0.00556913     0.084673     0.00184077     0.0111763       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000234744    0.0215281     0.612954     0.00359095     0.076162        100    
reco:pandoracalo:Calorimetry                          0.000101772    0.0121892     0.221468     0.00128751     0.0352413       100    
reco:pandorapid:Chi2ParticleID                        4.0016e-05    0.000212104   0.00187179    0.000162112   0.000208869      100    
reco:trkshowersplit:TrackShowerHits                   0.000112483    0.0109729     0.189832     0.000911513    0.0310282       100    
reco:pmtrack:PMAlgTrackMaker                          0.00152794      1.95285       32.888        0.25253       4.84686        100    
reco:pmtrackcalo:Calorimetry                          7.9641e-05    0.00740833     0.0955946    0.00116595     0.0167069       100    
reco:pmtrackpid:Chi2ParticleID                        3.2111e-05    0.000316027   0.00200195    0.000191807   0.000379701      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000258308     1.56936       13.6122      0.408066       2.77393        100    
reco:pmtrajfitcalo:Calorimetry                         7.418e-05     0.0129674     0.235808     0.00198529     0.0362842       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5658e-05    0.000235111   0.00102759    0.000202713   0.000177793      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00119548      2.14494       43.056       0.281705        5.74          100    
reco:pmtracktccalo:Calorimetry                        7.5202e-05    0.00634555      0.10566     0.00104811     0.0143913       100    
reco:pmtracktcpid:Chi2ParticleID                       3.209e-05    0.000299656   0.00160443    0.000198329   0.000320446      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000204767     1.53575       21.1488      0.389784       3.17173        100    
reco:pmtrajfittccalo:Calorimetry                      7.5392e-05     0.010722       0.21527      0.0015986     0.0298821       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.1961e-05    0.000380934   0.00258439    0.000208734   0.00046571       100    
reco:blurredcluster:BlurredClustering                 8.3307e-05      1.18938       6.58023      0.663021       1.44211        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000127721     12.9117       137.345       4.01513       21.5745        100    
reco:emshower:EMShower                                0.00020108     0.0306309     0.375814     0.00443204     0.056171        100    
reco:cvnmap:CVNMapper                                 2.2032e-05     0.0147307     0.0427771     0.0146811     0.0110688       100    
reco:cvneva:CVNEvaluator                              1.8355e-05     0.821232       1.2992        1.09608      0.504732        100    
reco:energyrecnumu:EnergyReco                         0.000325996    0.0252389      0.3883       0.004843       0.06388        100    
reco:energyrecnue:EnergyReco                          0.000130917   0.00177355     0.0256107    0.000793475   0.00321548       100    
reco:energyrecnc:EnergyReco                           0.00190053    0.00424911     0.0299211    0.00320828    0.00353367       100    
reco:energyrecnumurange:EnergyReco                    0.000163339    0.0226485      0.40514     0.00283764     0.0607908       100    
reco:energyrecnumumcs:EnergyReco                      0.00138794     0.023962      0.397069     0.00412208     0.0598902       100    
reco:opdec:Deconvolution                               0.0463789     0.288329      0.615366       0.26589      0.146497        100    
reco:ophitspe:OpHitFinderDeco                          0.0598062      1.66522       3.18104       1.73169      0.742632        100    
reco:opflash:OpFlashFinder                            0.000252286   0.00176117    0.00653439    0.00167879    0.00090373       100    
reco:rns:RandomNumberSaver                            2.0639e-05    5.01929e-05   0.000417799   3.9269e-05    4.65952e-05      100    
reco:anglereconue:NuAngularReco                       0.000182135   0.00134609     0.021481     0.000552745   0.00267438       100    
reco:anglereconumu:NuAngularReco                      7.9401e-05    0.00104491     0.0198436    0.000290669   0.00252665       100    
reco:anglereconuepfps:NuAngularReco                   0.000134504   0.00122218     0.0202607    0.000483559   0.00253598       100    
reco:anglereconumupfps:NuAngularReco                  0.000104457   0.00112418     0.0200896    0.000368106   0.00256034       100    
[art]:TriggerResults:TriggerResultInserter             9.318e-06    1.65179e-05   4.8932e-05    1.5384e-05    5.38517e-06      100    
end_path:out1:RootOutput                               3.297e-06    7.47622e-06   2.0639e-05     6.763e-06    3.57583e-06      100    
end_path:out1:RootOutput(write)                       0.00893071     0.257968      0.910559      0.273368      0.166472        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21018.9 MB
  Peak resident set size usage (VmHWM): 2130.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_2024-11-23T_031235Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_564_20231202T134417Z_gen_g4_detsim_hitreco__20240507T223703Z_reco2_reco_data_2024-11-23T_031235Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 12:24:11 UTC       justIN version: 01.01.09