justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108725.93@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108725.93@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:51:13
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2024-11-23 05:11:05
From worker nodeHostnamen256.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:55:46
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2.root
JobscriptExit code0
Real time1h (4503s)
CPU time1h (3786s = 84%)
Outputting started2024-11-23 05:10:50
Output files
Finished2024-11-23 05:11:05
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
1st wire C:0 T:18 P:2 W:598 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
1st wire C:0 T:18 P:2 W:697 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
1st wire C:0 T:18 P:2 W:738 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
1st wire C:0 T:18 P:2 W:842 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:10:38 GMT run: 6422149 subRun: 1 event: 107000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 159, 147, 133
minwire 0: 1872
minwire 1: 706
minwire 2: 1736
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.489413, 0.000363918, 0.510223, 
Output 1: 0.0168932, 0.975934, 0.00710023, 7.20932e-05, 
Output 2: 0.623487, 0.376513, 

571
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1936!
23-Nov-2024 05:10:48 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2_reco_data_2024-11-23T_035555Z.root"
23-Nov-2024 05:10:48 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cf/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.02879       38.1451       1285.23       10.1616       135.155        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0350473     0.0676431     0.105216      0.0732122     0.0176337       100    
reco:linecluster:LineCluster                           0.202106       1.7656        125.147      0.364323       12.4256        100    
reco:trajcluster:TrajCluster                            0.1013        1.72132       126.053      0.109945       12.5857        100    
reco:pandora:StandardPandora                           0.108941       2.99327       111.017       1.45479       11.2182        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00013777    0.00691734     0.208654      0.0013396     0.0222882       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000178056    0.0166891      0.58957      0.0033259     0.0612234       100    
reco:pandoracalo:Calorimetry                          0.000105648    0.0298383      1.43056     0.00103909     0.152678        100    
reco:pandorapid:Chi2ParticleID                        3.6409e-05    0.00019287    0.00237595    0.000137068   0.000258405      100    
reco:trkshowersplit:TrackShowerHits                   5.9462e-05     0.0427395      2.59971     0.00062969     0.264807        100    
reco:pmtrack:PMAlgTrackMaker                          0.00108646      4.65931       239.388      0.173843       24.5854        100    
reco:pmtrackcalo:Calorimetry                           6.909e-05     0.0120106     0.443752     0.000902468    0.0494759       100    
reco:pmtrackpid:Chi2ParticleID                         2.625e-05    0.000311086   0.00450896    0.000150483   0.000578993      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000171013     3.9244        206.245      0.225812       21.2145        100    
reco:pmtrajfitcalo:Calorimetry                        6.0404e-05     0.0297282      1.43443     0.00137478     0.152433        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3685e-05    0.000206252   0.00142413    0.000142719   0.000181718      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000840242     4.20039       174.996       0.19982       18.9883        100    
reco:pmtracktccalo:Calorimetry                        5.9963e-05    0.00712432     0.149114     0.000937303    0.0209781       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4476e-05    0.000320411   0.00382116    0.000159801   0.000544546      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000136147     2.27136       67.3339      0.188324       8.23624        100    
reco:pmtrajfittccalo:Calorimetry                      5.6847e-05     0.0207634     0.872794     0.00101133     0.0957495       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2652e-05    0.000404091    0.0059879    0.00017303    0.000799829      100    
reco:blurredcluster:BlurredClustering                  6.348e-05      1.13583       11.555       0.621624       1.70552        100    
reco:emtrkmichelid:EmTrackMichelId                    8.9218e-05      11.5965       206.116       2.49315       26.9032        100    
reco:emshower:EMShower                                0.000133822    0.0295049     0.479911     0.00800347     0.0609125       100    
reco:cvnmap:CVNMapper                                  1.609e-05     0.0101389     0.0562343    0.00705903     0.0096308       100    
reco:cvneva:CVNEvaluator                              1.2524e-05     0.583761       1.20671      0.647384       0.38906        100    
reco:energyrecnumu:EnergyReco                         0.000298001    0.0598583      3.02186     0.00453872     0.312865        100    
reco:energyrecnue:EnergyReco                          0.000115838   0.00212071     0.0612495    0.000474498   0.00656571       100    
reco:energyrecnc:EnergyReco                           0.00141763    0.00454231     0.0648303    0.00338911    0.00677753       100    
reco:energyrecnumurange:EnergyReco                    0.000128532    0.0576588      3.00342     0.00331775     0.311076        100    
reco:energyrecnumumcs:EnergyReco                       0.0014058     0.0593915      3.01391     0.00414758     0.311913        100    
reco:opdec:Deconvolution                               0.103597       0.41724      0.771558      0.417371       0.11303        100    
reco:ophitspe:OpHitFinderDeco                           0.30293       2.16153       5.44584       1.93759       1.02397        100    
reco:opflash:OpFlashFinder                            0.000145113   0.00148914    0.00631854    0.00132179    0.000858482      100    
reco:rns:RandomNumberSaver                            2.0198e-05    4.13343e-05   0.000421704   3.71345e-05   4.02282e-05      100    
reco:anglereconue:NuAngularReco                       0.000155593   0.00175487     0.053254     0.000439192   0.00567654       100    
reco:anglereconumu:NuAngularReco                      6.0173e-05    0.00146795     0.0513905    0.000221627   0.00545474       100    
reco:anglereconuepfps:NuAngularReco                   0.000117371   0.00163781     0.0517516    0.000374028   0.00549539       100    
reco:anglereconumupfps:NuAngularReco                   8.994e-05    0.00153979     0.0517327    0.000290957   0.00547875       100    
[art]:TriggerResults:TriggerResultInserter             8.676e-06    1.43069e-05   7.3959e-05    1.3931e-05    6.89066e-06      100    
end_path:out1:RootOutput                               3.136e-06    6.6894e-06    2.9335e-05     5.876e-06    3.49092e-06      100    
end_path:out1:RootOutput(write)                       0.00823593     0.246402       1.4514       0.241015       0.20095        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38529.6 MB
  Peak resident set size usage (VmHWM): 2475.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2_reco_2024-11-23T_035555Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1069_20231204T120519Z_gen_g4_detsim_hitreco__20240509T214019Z_reco2_reco_data_2024-11-23T_035555Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:44:07 UTC       justIN version: 01.01.09