justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108744.175@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108744.175@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:38:45
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2024-11-23 05:38:48
From worker nodeHostnamen259.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:44:21
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root
JobscriptExit code0
Real time54m (3249s)
CPU time48m (2896s = 89%)
Outputting started2024-11-23 05:38:31
Output files
Finished2024-11-23 05:38:48
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:38:22 GMT run: 6471483 subRun: 1 event: 23800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:38:22 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 245, 164, 202
minwire 0: 808
minwire 1: 2637
minwire 2: 264
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.305248, 0.600603, 0.0941493, 
Output 1: 0.0161795, 0.503121, 0.4745, 0.00619923, 
Output 2: 0.875958, 0.124042, 

475
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1614!
23-Nov-2024 05:38:30 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_data_2024-11-23T_044426Z.root"
23-Nov-2024 05:38:30 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/6e/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.34702       28.6899       472.836       8.17217       77.7639        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179069     0.0324575     0.0825704     0.0353351     0.0130964       100    
reco:linecluster:LineCluster                           0.174667       0.4136        10.4346      0.183042       1.17949        100    
reco:trajcluster:TrajCluster                           0.0510294     0.862102       40.3737      0.0577064      4.5653         100    
reco:pandora:StandardPandora                            1.0324        2.52099       46.4562        1.167        6.64554        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000172117   0.00586124      0.14783     0.00141583     0.0183239       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000217312    0.0104703     0.250459     0.00275919     0.0273665       100    
reco:pandoracalo:Calorimetry                          0.000125047    0.0299175     0.955738     0.000936736     0.13859        100    
reco:pandorapid:Chi2ParticleID                        4.1298e-05    0.000163586   0.00119355    0.000135487   0.000151613      100    
reco:trkshowersplit:TrackShowerHits                   0.000120418    0.0248095     0.710587     0.000638541    0.107975        100    
reco:pmtrack:PMAlgTrackMaker                          0.00131423      3.03929       76.8302       0.11687       12.0047        100    
reco:pmtrackcalo:Calorimetry                          7.8499e-05     0.0120458     0.401996     0.000887332    0.0542529       100    
reco:pmtrackpid:Chi2ParticleID                        3.0048e-05    0.000305074   0.00371297    0.000148507   0.000599669      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000215909     2.50172       55.3727      0.275683       8.45696        100    
reco:pmtrajfitcalo:Calorimetry                        7.5894e-05     0.0306239     0.978345     0.00135798     0.140005        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9526e-05    0.000200454   0.00111024    0.000169943   0.000177555      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00111922      3.52278       85.7546       0.14398       14.4589        100    
reco:pmtracktccalo:Calorimetry                        7.2117e-05    0.00723854     0.172964     0.000873647    0.0250455       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9947e-05    0.000271081   0.00257577    0.000146598   0.000425895      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000170003     1.77539       42.0732      0.232912       6.11045        100    
reco:pmtrajfittccalo:Calorimetry                      7.5032e-05     0.020448      0.581349     0.000928264    0.0879937       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.0888e-05    0.000394565   0.00515522    0.000152209   0.000793863      100    
reco:blurredcluster:BlurredClustering                 0.00563135     0.936472       8.93591      0.315693       1.54112        100    
reco:emtrkmichelid:EmTrackMichelId                     0.281589       9.84874       144.847       2.24268       24.6203        100    
reco:emshower:EMShower                                0.000328232    0.0268164     0.410756     0.00486153     0.0555993       100    
reco:cvnmap:CVNMapper                                 2.9055e-05    0.00928665     0.0450299    0.00858106    0.00909176       100    
reco:cvneva:CVNEvaluator                              2.0349e-05     0.529741       1.27581      0.758581      0.385122        100    
reco:energyrecnumu:EnergyReco                         0.00037466     0.0189039     0.387959     0.00402884     0.0515389       100    
reco:energyrecnue:EnergyReco                          0.000156036   0.00191689     0.0549585    0.000409306   0.00639987       100    
reco:energyrecnc:EnergyReco                           0.00216397    0.00442175     0.0600279    0.00290053    0.00680164       100    
reco:energyrecnumurange:EnergyReco                    0.000172597    0.0170246     0.387018     0.00264242     0.0511701       100    
reco:energyrecnumumcs:EnergyReco                      0.000976963    0.0183528     0.389326     0.00311686     0.0505453       100    
reco:opdec:Deconvolution                               0.116374      0.278748      0.685352      0.279181      0.0942085       100    
reco:ophitspe:OpHitFinderDeco                          0.486728       1.95109       3.82211       1.96637      0.788372        100    
reco:opflash:OpFlashFinder                            0.000396823   0.00145864    0.00618675    0.00134551    0.000806956      100    
reco:rns:RandomNumberSaver                             2.16e-05     4.2826e-05    0.000441788   3.9565e-05    4.08889e-05      100    
reco:anglereconue:NuAngularReco                       0.00019511    0.00148407     0.0403501    0.00040894    0.00472571       100    
reco:anglereconumu:NuAngularReco                      0.000104267   0.00121464     0.0360779    0.000243812   0.00428367       100    
reco:anglereconuepfps:NuAngularReco                   0.000149824   0.00139354     0.0371356    0.000385862   0.00438157       100    
reco:anglereconumupfps:NuAngularReco                  0.00012638    0.00128023     0.0367384    0.000299467    0.0043577       100    
[art]:TriggerResults:TriggerResultInserter             1.069e-05    1.44083e-05   6.0514e-05    1.34955e-05   5.12381e-06      100    
end_path:out1:RootOutput                               3.466e-06    7.06228e-06   2.5639e-05     6.818e-06    3.3854e-06       100    
end_path:out1:RootOutput(write)                        0.0247805     0.228429       1.03646      0.237686      0.182782        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38463.4 MB
  Peak resident set size usage (VmHWM): 2402.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_2024-11-23T_044426Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_data_2024-11-23T_044426Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:14:05 UTC       justIN version: 01.01.09