justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108721.75@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108721.75@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:40:08
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 04:39:23
From worker nodeHostnamecomp23-06
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:48:22
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2.root
JobscriptExit code0
Real time50m (3040s)
CPU time46m (2803s = 92%)
Outputting started2024-11-23 04:39:03
Output files
Finished2024-11-23 04:39:23
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

bRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
1st wire C:0 T:7 P:2 W:506 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:38:53 GMT run: 66049673 subRun: 1 event: 29600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 266, 162, 86
minwire 0: 910
minwire 1: 1823
minwire 2: 485
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0240044, 0.000446857, 0.975549, 
Output 1: 0.834157, 0.152341, 0.0104816, 0.00301971, 
Output 2: 0.963875, 0.0361249, 

505
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1600!
23-Nov-2024 04:39:02 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2_reco_data_2024-11-23T_034826Z.root"
23-Nov-2024 04:39:02 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/ae/a9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.35148       28.3389       441.865       10.1008       56.9941        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0121795     0.117217       0.4983       0.0925283     0.086385        100    
reco:linecluster:LineCluster                            0.13835      0.413581       3.28852      0.263872      0.523802        100    
reco:trajcluster:TrajCluster                           0.0368156      1.27904       44.1427      0.0955521      5.80381        100    
reco:pandora:StandardPandora                            1.04738       1.88414       28.3859       1.1853        2.96583        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000132788    0.0064724     0.152357     0.00170872     0.0177101       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000177013    0.0159918     0.667053     0.00272225     0.0691331       100    
reco:pandoracalo:Calorimetry                          9.6522e-05     0.016519      0.591703     0.00146499     0.0641025       100    
reco:pandorapid:Chi2ParticleID                        3.8398e-05    0.000145039   0.00132642    0.000116618   0.000152095      100    
reco:trkshowersplit:TrackShowerHits                   0.000103766    0.0175892     0.418488     0.000712588    0.0640079       100    
reco:pmtrack:PMAlgTrackMaker                          0.00121868      2.12045       53.0635      0.142274       6.62588        100    
reco:pmtrackcalo:Calorimetry                          8.1111e-05     0.0107847     0.313498     0.000793359    0.0394838       100    
reco:pmtrackpid:Chi2ParticleID                        3.0128e-05    0.00024397    0.00227735    0.000122148   0.000356669      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000179724     1.41562       19.711       0.190922       3.1395         100    
reco:pmtrajfitcalo:Calorimetry                        6.0336e-05     0.0175219     0.591839     0.00170921     0.0654438       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6805e-05    0.000165356   0.00082999    0.000132544   0.000138471      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00107999      2.50118       65.1267      0.196005       7.99742        100    
reco:pmtracktccalo:Calorimetry                        7.0817e-05    0.00791693     0.177795     0.000902311    0.0250285       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7274e-05    0.000233401   0.00184735    0.000128804   0.000304574      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000157738     1.37423       24.0882      0.306807       3.2727         100    
reco:pmtrajfittccalo:Calorimetry                      5.9797e-05     0.014504      0.424976     0.00111621     0.0513368       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7494e-05    0.000304721   0.00247395    0.00014714    0.000444004      100    
reco:blurredcluster:BlurredClustering                 0.00433139      0.97312       8.96964      0.371233       1.41381        100    
reco:emtrkmichelid:EmTrackMichelId                     0.203999       12.7414       182.59        3.82671       25.4276        100    
reco:emshower:EMShower                                0.000237914    0.0191004     0.179788     0.00453842     0.0321236       100    
reco:cvnmap:CVNMapper                                 2.0689e-05     0.0126411     0.0464917     0.0124387     0.010169        100    
reco:cvneva:CVNEvaluator                              1.5313e-05     0.546524      0.848717      0.759711      0.358774        100    
reco:energyrecnumu:EnergyReco                         0.000281096    0.0237114     0.623728     0.00492517     0.086595        100    
reco:energyrecnue:EnergyReco                          0.000136334   0.00207235     0.0494032    0.000582851   0.00555501       100    
reco:energyrecnc:EnergyReco                           0.00174953    0.00364008     0.0507495    0.00220358    0.00557189       100    
reco:energyrecnumurange:EnergyReco                    0.000215757    0.0214532     0.618084     0.00198855     0.0869291       100    
reco:energyrecnumumcs:EnergyReco                      0.00104802     0.0224353     0.623162     0.00252725     0.0872197       100    
reco:opdec:Deconvolution                               0.112375      0.379394      0.895201      0.355811      0.150262        100    
reco:ophitspe:OpHitFinderDeco                          0.276079       2.16925       4.12633       2.19758        0.848         100    
reco:opflash:OpFlashFinder                            0.000287907   0.00129458    0.00339831    0.00124527    0.000545385      100    
reco:rns:RandomNumberSaver                             1.902e-05    2.95764e-05   0.000269672   2.5303e-05    2.50245e-05      100    
reco:anglereconue:NuAngularReco                       0.000139248   0.00153689     0.0439368    0.00035175    0.00485663       100    
reco:anglereconumu:NuAngularReco                      6.7066e-05    0.00133984     0.0428285    0.000183501   0.00474772       100    
reco:anglereconuepfps:NuAngularReco                   0.000114037   0.00143526     0.0421987    0.000276234   0.00468393       100    
reco:anglereconumupfps:NuAngularReco                  9.4412e-05    0.00139579     0.0419156    0.000252401   0.00467173       100    
[art]:TriggerResults:TriggerResultInserter             8.695e-06    1.10688e-05   4.2667e-05    1.02425e-05   3.49751e-06      100    
end_path:out1:RootOutput                               2.627e-06    4.65097e-06    1.563e-05     3.616e-06    2.42793e-06      100    
end_path:out1:RootOutput(write)                        0.013188      0.201615      0.949433      0.202312       0.14936        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9511.3 MB
  Peak resident set size usage (VmHWM): 2355.48 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2_reco_2024-11-23T_034826Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_295_20231207T153544Z_gen_g4_detsim_hitreco__20240510T044104Z_reco2_reco_data_2024-11-23T_034826Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 10:36:38 UTC       justIN version: 01.01.09