justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108837.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108837.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 05:50:25
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 06:48:09
From worker nodeHostnamecomp07-22
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 05:51:27
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root
JobscriptExit code0
Real time56m (3389s)
CPU time52m (3142s = 92%)
Outputting started2024-11-23 06:47:57
Output files
Finished2024-11-23 06:48:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

T:12 P:2 W:600 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:47:45 GMT run: 50515154 subRun: 1 event: 26700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:47:45 GMT run: 50515154 subRun: 1 event: 26700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:47:45 GMT run: 50515154 subRun: 1 event: 26700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:47:50 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:47:50 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:47:50 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:47:50 GMT run: 50515154 subRun: 1 event: 26700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 239, 254, 128
minwire 0: 1643
minwire 1: 1624
minwire 2: 1366
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0227035, 0.976552, 0.00074478, 
Output 1: 0.828446, 0.163168, 0.00781768, 0.000568324, 
Output 2: 0.947701, 0.0522988, 

634
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2354!
23-Nov-2024 06:47:55 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_data_2024-11-23T_055129Z.root"
23-Nov-2024 06:47:55 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/12/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.492199       31.2962       652.601       8.67319       80.4851        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0101816     0.025631      0.186019      0.0221034     0.0185294       100    
reco:linecluster:LineCluster                           0.0550904     0.477185       15.8903      0.115387       1.72109        100    
reco:trajcluster:TrajCluster                           0.0266355      1.69204       75.5707      0.038273       8.31694        100    
reco:pandora:StandardPandora                           0.0428769      2.2579        48.2105       1.16831       5.48067        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000126472   0.00665988     0.168638     0.00121881     0.0198315       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00017954     0.0106752     0.177782      0.002148      0.0251975       100    
reco:pandoracalo:Calorimetry                          9.6015e-05     0.033679       1.81114     0.000772277    0.187409        100    
reco:pandorapid:Chi2ParticleID                        3.4696e-05    0.000132152   0.00150918    9.2447e-05    0.000167246      100    
reco:trkshowersplit:TrackShowerHits                   5.7362e-05     0.0274711     0.789472     0.000488817    0.109691        100    
reco:pmtrack:PMAlgTrackMaker                          0.00112275      2.28881       58.2155      0.0866404      7.51102        100    
reco:pmtrackcalo:Calorimetry                          7.2724e-05      0.01501      0.625574     0.000700756    0.0671294       100    
reco:pmtrackpid:Chi2ParticleID                        2.9244e-05    0.00024673    0.00327844    0.000104947   0.00046088       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000167117     3.0395        132.126      0.237485       14.1002        100    
reco:pmtrajfitcalo:Calorimetry                        6.1754e-05     0.0328811      1.74699     0.00114061      0.18013        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6465e-05    0.000157721   0.00121605    0.000115262   0.000163927      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000969649     2.90979       78.2162       0.13295       9.89917        100    
reco:pmtracktccalo:Calorimetry                         7.063e-05     0.010745      0.442711     0.00068794     0.0468754       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7842e-05    0.000227057   0.00278942    0.000114466   0.000379132      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000159606     1.48346       30.8375       0.18324       4.21175        100    
reco:pmtrajfittccalo:Calorimetry                      5.9765e-05     0.0197987     0.682593     0.000860133    0.0806796       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.537e-05    0.000301219   0.00364569    0.000121187   0.000566279      100    
reco:blurredcluster:BlurredClustering                 6.2063e-05     0.986241       10.8607        0.432        1.57847        100    
reco:emtrkmichelid:EmTrackMichelId                    9.9476e-05       12.71        198.496       2.95344       28.8675        100    
reco:emshower:EMShower                                0.000149356    0.0243955     0.271503     0.00358851     0.049224        100    
reco:cvnmap:CVNMapper                                 1.6844e-05     0.013651      0.0495235     0.012247      0.0112816       100    
reco:cvneva:CVNEvaluator                              1.3913e-05     0.463337      0.684194      0.609417        0.268         100    
reco:energyrecnumu:EnergyReco                         0.000236099    0.0459545      1.32549     0.00341482     0.166945        100    
reco:energyrecnue:EnergyReco                          0.000123812   0.00219803     0.0584626    0.00041184    0.00677086       100    
reco:energyrecnc:EnergyReco                           0.00176245    0.00386486     0.0606858    0.00210326    0.00680741       100    
reco:energyrecnumurange:EnergyReco                    0.000180918    0.0441233      1.32152     0.00196029     0.166358        100    
reco:energyrecnumumcs:EnergyReco                      0.00142807     0.045239       1.32082     0.00256506     0.165979        100    
reco:opdec:Deconvolution                               0.0399384     0.279062       1.28676      0.264428      0.146259        100    
reco:ophitspe:OpHitFinderDeco                          0.0844324      2.11986       3.97939       2.19427      0.900237        100    
reco:opflash:OpFlashFinder                            0.000205272   0.00120107    0.00411811    0.00116123     0.0006062       100    
reco:rns:RandomNumberSaver                            1.6754e-05    2.71232e-05   0.000244178   2.3675e-05    2.30311e-05      100    
reco:anglereconue:NuAngularReco                       0.000143431   0.00173925     0.050281     0.000307882    0.0057242       100    
reco:anglereconumu:NuAngularReco                      6.6455e-05    0.00155526     0.0497472    0.000176626   0.00564853       100    
reco:anglereconuepfps:NuAngularReco                   0.000114403   0.00166569     0.0502692    0.00026303     0.0056966       100    
reco:anglereconumupfps:NuAngularReco                  9.5803e-05    0.00161432     0.0501024    0.000229305   0.00567559       100    
[art]:TriggerResults:TriggerResultInserter             8.025e-06    1.03167e-05   4.9879e-05    9.3815e-06    4.35682e-06      100    
end_path:out1:RootOutput                               2.309e-06    4.5805e-06    2.9301e-05     3.266e-06    3.26193e-06      100    
end_path:out1:RootOutput(write)                       0.00854061     0.216696       1.18719      0.207778      0.168394        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9548.15 MB
  Peak resident set size usage (VmHWM): 2475.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_2024-11-23T_055129Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_266_20231201T071049Z_gen_g4_detsim_hitreco__20240507T183548Z_reco2_reco_data_2024-11-23T_055129Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 12:21:19 UTC       justIN version: 01.01.09