justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108730.120@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108730.120@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:04:08
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2024-11-23 05:20:06
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:06:30
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2.root
JobscriptExit code0
Real time1h (4395s)
CPU time1h (5096s = 115%)
Outputting started2024-11-23 05:19:46
Output files
Finished2024-11-23 05:20:06
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
1st wire C:0 T:5 P:2 W:517 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:19:42 GMT run: 74494003 subRun: 1 event: 53200
Comparing two wires in the same plane: return failure
%MSG
282
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 589!
23-Nov-2024 05:19:44 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2_reco_data_2024-11-23T_040638Z.root"
23-Nov-2024 05:19:44 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/d5/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.374487       42.0407       524.296       13.9662       90.8938        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00467587     0.0315263     0.301208      0.0169016     0.0474708       100    
reco:linecluster:LineCluster                           0.0311011     0.739284       33.2904      0.0935206      3.55179        100    
reco:trajcluster:TrajCluster                           0.0126328      1.38584       35.7744      0.0318653      5.58207        100    
reco:pandora:StandardPandora                           0.0485549      3.52107       70.428        1.52427       8.54962        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000189164    0.0085579     0.0894784    0.00251896     0.0173253       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000261129    0.0141038     0.140603      0.0053097     0.0222553       100    
reco:pandoracalo:Calorimetry                          0.000151824    0.0319221      0.71224     0.00219555     0.104847        100    
reco:pandorapid:Chi2ParticleID                         5.325e-05    0.000284576   0.00205786    0.000223518   0.000261281      100    
reco:trkshowersplit:TrackShowerHits                   0.000148398    0.0432839      1.1083      0.00145374     0.154097        100    
reco:pmtrack:PMAlgTrackMaker                          0.00137825      4.18649       79.5841      0.332377       12.7003        100    
reco:pmtrackcalo:Calorimetry                          7.2836e-05     0.0160233     0.455649     0.00132802     0.0540108       100    
reco:pmtrackpid:Chi2ParticleID                        3.1118e-05    0.000484148    0.0045095    0.00021628    0.000744863      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000214682     3.26348       58.4528      0.501125       9.25505        100    
reco:pmtrajfitcalo:Calorimetry                        6.7476e-05     0.0353676     0.745549     0.00277576     0.116465        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8183e-05    0.000300009   0.00127037    0.000242779   0.000227776      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00142156      4.92989       90.6195      0.370361       15.427         100    
reco:pmtracktccalo:Calorimetry                        9.7122e-05     0.0104772     0.294381     0.00154059     0.0332943       100    
reco:pmtracktcpid:Chi2ParticleID                      4.1788e-05    0.000400425    0.0038993    0.000209723   0.000561895      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000250819     2.68955       46.0792      0.375665       7.1417         100    
reco:pmtrajfittccalo:Calorimetry                      9.5018e-05     0.0238998     0.538151     0.00206304     0.0759145       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9765e-05    0.000568304    0.0050065    0.000270206   0.000882749      100    
reco:blurredcluster:BlurredClustering                 0.000269915     1.39466       10.2802      0.642765       1.91953        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000182262     15.384        169.712       4.75732       29.5951        100    
reco:emshower:EMShower                                0.000254657    0.038319      0.371158     0.00727427     0.0708131       100    
reco:cvnmap:CVNMapper                                 3.5166e-05     0.0169183     0.0581216     0.0186697     0.0128802       100    
reco:cvneva:CVNEvaluator                              2.5968e-05      1.08571       1.89392       1.38498      0.640619        100    
reco:energyrecnumu:EnergyReco                         0.000599944    0.034626      0.592932     0.00647244     0.0844164       100    
reco:energyrecnue:EnergyReco                          0.000198301   0.00274649     0.0387618    0.000739174   0.00568282       100    
reco:energyrecnc:EnergyReco                           0.00203395    0.00568356     0.0435355    0.00387709    0.00609413       100    
reco:energyrecnumurange:EnergyReco                    0.00024027     0.0308417     0.573926     0.00351755     0.0827326       100    
reco:energyrecnumumcs:EnergyReco                      0.00210241     0.0330035     0.584349     0.00469658     0.082809        100    
reco:opdec:Deconvolution                               0.0192475     0.276865       1.03977      0.231475      0.169797        100    
reco:ophitspe:OpHitFinderDeco                          0.0408318      2.46675       4.52723       2.33011       0.9802         100    
reco:opflash:OpFlashFinder                            0.000228397   0.00214259    0.00576296    0.00202553    0.000954431      100    
reco:rns:RandomNumberSaver                            2.1681e-05    6.90782e-05   0.000506709   6.54275e-05   4.63467e-05      100    
reco:anglereconue:NuAngularReco                       0.000245239   0.00232187     0.0342973    0.000682748   0.00522809       100    
reco:anglereconumu:NuAngularReco                      9.8044e-05    0.00187885     0.0334701    0.000354052   0.00486894       100    
reco:anglereconuepfps:NuAngularReco                   0.000193702   0.00217007     0.0338681    0.000564852   0.00501827       100    
reco:anglereconumupfps:NuAngularReco                  0.000135624   0.00200627     0.0339849    0.000422802   0.00499256       100    
[art]:TriggerResults:TriggerResultInserter            1.1452e-05    2.15593e-05   5.4743e-05    1.9381e-05    8.67629e-06      100    
end_path:out1:RootOutput                               5.35e-06     1.14879e-05   3.4004e-05    1.11455e-05   3.53368e-06      100    
end_path:out1:RootOutput(write)                        0.0144234     0.323879       1.34704      0.312329      0.227837        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39515.2 MB
  Peak resident set size usage (VmHWM): 2316.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2_reco_2024-11-23T_040638Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_531_20231201T170331Z_gen_g4_detsim_hitreco__20240507T200949Z_reco2_reco_data_2024-11-23T_040638Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:26:59 UTC       justIN version: 01.01.09