justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108702.75@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108702.75@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:04:38
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2024-11-23 04:31:59
From worker nodeHostnamen211.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:12:33
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2.root
JobscriptExit code0
Real time1h (4748s)
CPU time1h (4370s = 92%)
Outputting started2024-11-23 04:31:42
Output files
Finished2024-11-23 04:31:59
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
1st wire C:0 T:20 P:2 W:510 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:31:32 GMT run: 6414162 subRun: 1 event: 3500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 100, 63, 36
minwire 0: 2520
minwire 1: 384
minwire 2: 2675
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0380485, 0.000616618, 0.961335, 
Output 1: 0.0114098, 0.969024, 0.0192785, 0.000288207, 
Output 2: 0.954051, 0.0459486, 

351
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 963!
23-Nov-2024 04:31:41 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2_reco_data_2024-11-23T_031237Z.root"
23-Nov-2024 04:31:41 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/da/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.586787       43.4204       486.439       15.5095       80.8045        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0177056     0.0377716     0.0775157     0.035823      0.0122927       100    
reco:linecluster:LineCluster                           0.104297      0.572751       12.3726      0.188986       1.61264        100    
reco:trajcluster:TrajCluster                           0.0502873      1.22131       31.3156      0.063592       4.05546        100    
reco:pandora:StandardPandora                           0.0707198      2.99127       34.0564       1.6697        4.92513        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000170985   0.00837886     0.109908     0.00195824     0.0173595       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000265594    0.0141254      0.19683      0.0038516     0.0301556       100    
reco:pandoracalo:Calorimetry                          0.000144006    0.0331072     0.878541     0.00138294     0.115507        100    
reco:pandorapid:Chi2ParticleID                        5.0705e-05    0.00021324    0.00162251    0.000162338   0.000197074      100    
reco:trkshowersplit:TrackShowerHits                   7.3165e-05     0.0336106     0.832239     0.000871672    0.110518        100    
reco:pmtrack:PMAlgTrackMaker                          0.00162268      3.51955       53.828       0.304398       9.10964        100    
reco:pmtrackcalo:Calorimetry                          0.000101808    0.0125868      0.19944     0.00124541     0.0337965       100    
reco:pmtrackpid:Chi2ParticleID                        3.8166e-05    0.000432913   0.00301528    0.000204691   0.00059137       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000249424     3.39055       58.0522      0.408212       9.5639         100    
reco:pmtrajfitcalo:Calorimetry                        0.000100844    0.0344489     0.865686     0.00172366     0.116277        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.1177e-05    0.000249258   0.000825089   0.000201338   0.000175623      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00139273      4.52252       77.8048       0.2477        12.1864        100    
reco:pmtracktccalo:Calorimetry                        0.00010107    0.00799217     0.130957     0.00133953     0.0186436       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6969e-05    0.00038083    0.00244516    0.000211611   0.000461921      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000235342     2.50246       32.0464      0.470497       5.72655        100    
reco:pmtrajfittccalo:Calorimetry                      9.0905e-05     0.0260448     0.435004     0.00162502     0.0765838       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.8104e-05    0.000569138   0.00449903    0.000228024   0.000890426      100    
reco:blurredcluster:BlurredClustering                 7.3098e-05      1.27354       11.1372      0.713871       1.73372        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000130144     17.7887       170.495       4.9912        32.4739        100    
reco:emshower:EMShower                                0.000209041    0.0457321     0.391173      0.0118259     0.0727778       100    
reco:cvnmap:CVNMapper                                 2.5871e-05     0.0166452     0.0479333     0.0198531     0.0116051       100    
reco:cvneva:CVNEvaluator                              1.9453e-05      0.90825       1.53823       1.17567      0.546202        100    
reco:energyrecnumu:EnergyReco                         0.000422079    0.0622514      1.11532     0.00569972     0.178895        100    
reco:energyrecnue:EnergyReco                           0.0002096    0.00273778     0.0356534    0.000738504   0.00555853       100    
reco:energyrecnc:EnergyReco                           0.00257534    0.00530265     0.0386596    0.00334307    0.00571971       100    
reco:energyrecnumurange:EnergyReco                    0.000243877    0.0569882     0.792253      0.0030635     0.161327        100    
reco:energyrecnumumcs:EnergyReco                      0.00198033     0.0581437     0.788724     0.00400428     0.158618        100    
reco:opdec:Deconvolution                               0.0864875     0.344976      0.639186      0.324541      0.116712        100    
reco:ophitspe:OpHitFinderDeco                          0.149555       3.61253       6.86946       3.44537       1.47067        100    
reco:opflash:OpFlashFinder                            0.000162964   0.00186864    0.00607898    0.00169877    0.000883494      100    
reco:rns:RandomNumberSaver                            2.5027e-05    4.58968e-05   0.000366305   4.16345e-05   3.39111e-05      100    
reco:anglereconue:NuAngularReco                       0.000177785   0.00214559     0.0333728    0.000518197   0.00495472       100    
reco:anglereconumu:NuAngularReco                       9.285e-05    0.00181203     0.0303748    0.000345939   0.00453171       100    
reco:anglereconuepfps:NuAngularReco                   0.000145611   0.00197377     0.0302614    0.000477761   0.00451567       100    
reco:anglereconumupfps:NuAngularReco                  0.000142905   0.00189293     0.0302957    0.000433287   0.00443875       100    
[art]:TriggerResults:TriggerResultInserter            1.0827e-05    1.57957e-05   4.8606e-05    1.5327e-05    3.94384e-06      100    
end_path:out1:RootOutput                               2.446e-06    6.27812e-06   1.8784e-05     6.079e-06    2.1278e-06       100    
end_path:out1:RootOutput(write)                       0.00907452     0.302373       1.1411       0.290836      0.210616        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13909.3 MB
  Peak resident set size usage (VmHWM): 2459.43 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2_reco_2024-11-23T_031237Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_34_20231203T130243Z_gen_g4_detsim_hitreco__20240508T072944Z_reco2_reco_data_2024-11-23T_031237Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:26:41 UTC       justIN version: 01.01.09