justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108662.108@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108662.108@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 01:16:51
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 02:27:38
From worker nodeHostnamecomp19-18
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 01:42:29
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2.root
JobscriptExit code0
Real time44m (2695s)
CPU time42m (2544s = 94%)
Outputting started2024-11-23 02:27:24
Output files
Finished2024-11-23 02:27:38
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

aring two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:13 GMT run: 50538503 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:13 GMT run: 50538503 subRun: 1 event: 70500
1st wire C:0 T:1 P:2 W:1010 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:13 GMT run: 50538503 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:13 GMT run: 50538503 subRun: 1 event: 70500
1st wire C:0 T:1 P:2 W:782 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:14 GMT run: 50538503 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:27:14 GMT run: 50538503 subRun: 1 event: 70500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 208, 207, 204
minwire 0: 705
minwire 1: 2916
minwire 2: 16
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.637621, 0.017901, 0.344478, 
Output 1: 0.0606967, 0.708659, 0.212885, 0.0177597, 
Output 2: 0.885518, 0.114482, 

429
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1414!
23-Nov-2024 02:27:23 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2_reco_data_2024-11-23T_014232Z.root"
23-Nov-2024 02:27:23 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/bb/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.58227       25.2832       350.486       9.86717       49.4829        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0142643     0.0377569     0.0753573     0.0364825     0.0128876       100    
reco:linecluster:LineCluster                           0.128008      0.260377       5.90605      0.141245      0.628159        100    
reco:trajcluster:TrajCluster                           0.0412475     0.607273       16.3657      0.0515683      2.19372        100    
reco:pandora:StandardPandora                            1.11068       1.94933       30.5567       1.19144       3.57682        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139572   0.00595487     0.170625     0.00155193     0.0181142       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000177481   0.00908569     0.155912     0.00278263     0.0212444       100    
reco:pandoracalo:Calorimetry                          9.8789e-05     0.0165962     0.478257     0.00108068     0.0641572       100    
reco:pandorapid:Chi2ParticleID                        3.6596e-05    0.000138435   0.00131406    9.6864e-05    0.000148514      100    
reco:trkshowersplit:TrackShowerHits                   0.00011921     0.0160322      0.38939     0.000645205    0.0563189       100    
reco:pmtrack:PMAlgTrackMaker                           0.0011941      1.71914       41.6907       0.19963       5.84006        100    
reco:pmtrackcalo:Calorimetry                           7.807e-05    0.00856493     0.378827     0.000913497    0.0389056       100    
reco:pmtrackpid:Chi2ParticleID                        3.0393e-05    0.000230134   0.00269621    0.000119421   0.000349921      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000209502     1.70093       27.5691      0.300429       4.50847        100    
reco:pmtrajfitcalo:Calorimetry                        6.3333e-05     0.0175961     0.468906     0.00160975     0.0676601       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7566e-05    0.000160461   0.00106137    0.000129772   0.000132598      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00109493      1.90347       41.3152      0.197906       5.67456        100    
reco:pmtracktccalo:Calorimetry                        7.6337e-05    0.00627256     0.255492     0.00100597     0.0262056       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9924e-05    0.000217125   0.00227892    0.000128318   0.000290417      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000202238     1.23421       25.778        0.27535       3.29987        100    
reco:pmtrajfittccalo:Calorimetry                      6.7617e-05      0.01476      0.509168     0.00116124     0.0592212       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7436e-05    0.000301281   0.00322827    0.000143846   0.00049708       100    
reco:blurredcluster:BlurredClustering                 0.00266472     0.973265       8.98412      0.576379       1.30448        100    
reco:emtrkmichelid:EmTrackMichelId                     0.254534       11.3646       161.37        3.5175        22.8419        100    
reco:emshower:EMShower                                0.000250052    0.0246845     0.421351     0.00464439     0.0527459       100    
reco:cvnmap:CVNMapper                                 2.2614e-05     0.0141568     0.0471404     0.0127955     0.0111073       100    
reco:cvneva:CVNEvaluator                              1.4831e-05     0.514799      0.836027      0.669677      0.299008        100    
reco:energyrecnumu:EnergyReco                         0.000313223    0.0235355     0.290456     0.00428675     0.0538085       100    
reco:energyrecnue:EnergyReco                          0.000138068   0.00184648     0.0532984    0.00046969    0.00567455       100    
reco:energyrecnc:EnergyReco                           0.00189395    0.00364998     0.0557148    0.00223628    0.00577585       100    
reco:energyrecnumurange:EnergyReco                    0.000177342    0.0214325     0.287638      0.0021442     0.053642        100    
reco:energyrecnumumcs:EnergyReco                      0.00197865     0.0227802     0.288382     0.00265424     0.0535538       100    
reco:opdec:Deconvolution                               0.106117      0.293436      0.479522      0.285972      0.0822556       100    
reco:ophitspe:OpHitFinderDeco                          0.467874       2.29041       4.0306        2.21485      0.815286        100    
reco:opflash:OpFlashFinder                            0.000323758   0.00131849    0.00414334    0.00124095    0.000559373      100    
reco:rns:RandomNumberSaver                            1.8219e-05    3.11354e-05   0.000277126   2.7618e-05    2.60714e-05      100    
reco:anglereconue:NuAngularReco                       0.000156938   0.00147359     0.0527396    0.000326351   0.00546328       100    
reco:anglereconumu:NuAngularReco                      6.8168e-05    0.00129093     0.0510885    0.000179184   0.00529882       100    
reco:anglereconuepfps:NuAngularReco                   0.000115556   0.00140995     0.0510104    0.000288972   0.00529143       100    
reco:anglereconumupfps:NuAngularReco                  9.6552e-05    0.00136313     0.0509646    0.000252157   0.00528848       100    
[art]:TriggerResults:TriggerResultInserter             8.954e-06    1.11819e-05    4.848e-05    1.0394e-05    4.02352e-06      100    
end_path:out1:RootOutput                               2.903e-06    5.50358e-06   1.9323e-05     4.884e-06    2.66797e-06      100    
end_path:out1:RootOutput(write)                        0.0214559     0.217653      0.949157      0.218491      0.141486        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9474.05 MB
  Peak resident set size usage (VmHWM): 2401.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2_reco_2024-11-23T_014232Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_704_20231201T174448Z_gen_g4_detsim_hitreco__20240507T191638Z_reco2_reco_data_2024-11-23T_014232Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:39:13 UTC       justIN version: 01.01.09