justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301679.121@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301679.121@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:41:13
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2024-11-23 04:44:02
From worker nodeHostnamen256.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:43:16
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2.root
JobscriptExit code0
Real time1h (3629s)
CPU time54m (3242s = 89%)
Outputting started2024-11-23 04:43:46
Output files
Finished2024-11-23 04:44:02
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

51900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
1st wire C:0 T:21 P:2 W:746 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
1st wire C:0 T:21 P:2 W:560 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:43:34 GMT run: 66004350 subRun: 1 event: 51900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 231, 132, 96
minwire 0: 2473
minwire 1: 246
minwire 2: 2435
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.010431, 6.36069e-05, 0.989505, 
Output 1: 0.651923, 0.332217, 0.0151038, 0.000756015, 
Output 2: 0.995051, 0.00494943, 

487
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1593!
23-Nov-2024 04:43:44 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2_reco_data_2024-11-23T_034325Z.root"
23-Nov-2024 04:43:44 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/4e/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.50566       32.1999       405.627       13.9556       54.1757        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179943     0.0396168     0.0752796     0.0378742     0.0145858       100    
reco:linecluster:LineCluster                           0.176533       0.45512       15.5796      0.200804       1.61704        100    
reco:trajcluster:TrajCluster                           0.0511708     0.859913       35.7857      0.0745155      3.79032        100    
reco:pandora:StandardPandora                            1.04612       2.12026       21.2126       1.45138       2.40268        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00016455    0.00708617     0.0869977    0.00226866     0.013184        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000250912    0.0131053     0.119287      0.0048387     0.021772        100    
reco:pandoracalo:Calorimetry                          0.000139332    0.0174991     0.512055     0.00215772     0.0574423       100    
reco:pandorapid:Chi2ParticleID                        5.1237e-05    0.00022135    0.00206548    0.000169219   0.000228718      100    
reco:trkshowersplit:TrackShowerHits                   0.000139873    0.0294303      0.91113     0.00174058     0.107419        100    
reco:pmtrack:PMAlgTrackMaker                          0.00112926      2.7662        58.9764      0.313877       7.23398        100    
reco:pmtrackcalo:Calorimetry                          7.5352e-05    0.00985447     0.161558     0.00169089     0.0231349       100    
reco:pmtrackpid:Chi2ParticleID                        2.9295e-05    0.000396051   0.00276083    0.000204089   0.000502896      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00023891      2.17578       27.2019      0.458923       4.30425        100    
reco:pmtrajfitcalo:Calorimetry                        8.6212e-05     0.0185535     0.520927     0.00265337     0.0584139       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.723e-05    0.000256989   0.000965337   0.000206139   0.000197119      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00133095      3.46904       75.6257      0.310154       9.30329        100    
reco:pmtracktccalo:Calorimetry                        8.8717e-05    0.00790535     0.0908622     0.0017746     0.0159367       100    
reco:pmtracktcpid:Chi2ParticleID                      3.7501e-05    0.000373582   0.00222819    0.000217143   0.00042181       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000257474     2.01352       30.4582      0.448082       4.07805        100    
reco:pmtrajfittccalo:Calorimetry                      9.1622e-05     0.0152968     0.304806     0.00215202     0.0397601       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6138e-05    0.000488554   0.00382708    0.000236049   0.000643604      100    
reco:blurredcluster:BlurredClustering                  0.010873       1.41633       7.79385      0.664804       1.71576        100    
reco:emtrkmichelid:EmTrackMichelId                     0.272163       12.7582       125.086       5.10424       20.5344        100    
reco:emshower:EMShower                                0.000343677    0.0574057     0.407849     0.00663797     0.0895645       100    
reco:cvnmap:CVNMapper                                 3.1489e-05     0.0126081     0.0441926     0.0105583     0.0109152       100    
reco:cvneva:CVNEvaluator                              2.0269e-05     0.712243       1.14265      0.901089      0.392404        100    
reco:energyrecnumu:EnergyReco                         0.000399832    0.039931      0.634087     0.00520301     0.104409        100    
reco:energyrecnue:EnergyReco                          0.000151014    0.0023931     0.0268538    0.000854694   0.00410573       100    
reco:energyrecnc:EnergyReco                           0.00202537     0.0054639     0.032746     0.00387214    0.00452945       100    
reco:energyrecnumurange:EnergyReco                    0.000209765    0.0376525     0.629419      0.0035639     0.103663        100    
reco:energyrecnumumcs:EnergyReco                      0.00102742     0.0395712     0.627906     0.00418407     0.103481        100    
reco:opdec:Deconvolution                               0.122024      0.325067      0.617212      0.328604      0.114368        100    
reco:ophitspe:OpHitFinderDeco                          0.586662       2.46217       4.56286       2.54096       1.04797        100    
reco:opflash:OpFlashFinder                            0.000365998   0.00171953    0.00582409    0.00160796    0.000858042      100    
reco:rns:RandomNumberSaver                            2.0899e-05    4.35064e-05   0.000350359   3.7611e-05    3.45003e-05      100    
reco:anglereconue:NuAngularReco                       0.000182193   0.00181351     0.0220323    0.000617728    0.0033478       100    
reco:anglereconumu:NuAngularReco                      9.7404e-05    0.00158176     0.0216827    0.00037208    0.00331485       100    
reco:anglereconuepfps:NuAngularReco                   0.000161675    0.0017454     0.0219413    0.000545106   0.00333496       100    
reco:anglereconumupfps:NuAngularReco                  0.000138712   0.00167891     0.0220353    0.000445644   0.00336673       100    
[art]:TriggerResults:TriggerResultInserter            1.1602e-05    1.6564e-05    5.5123e-05    1.50535e-05   6.33769e-06      100    
end_path:out1:RootOutput                               3.307e-06    7.12942e-06   3.3483e-05    6.5775e-06    4.45199e-06      100    
end_path:out1:RootOutput(write)                        0.031416       0.29989      0.962755      0.299735      0.182703        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38276 MB
  Peak resident set size usage (VmHWM): 2212.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2_reco_2024-11-23T_034325Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_518_20231202T160914Z_gen_g4_detsim_hitreco__20240508T041733Z_reco2_reco_data_2024-11-23T_034325Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:07:26 UTC       justIN version: 01.01.09