justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108722.50@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108722.50@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:45:32
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2024-11-23 04:42:15
From worker nodeHostnamen260.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:49:41
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root
JobscriptExit code0
Real time52m (3137s)
CPU time45m (2739s = 87%)
Outputting started2024-11-23 04:41:59
Output files
Finished2024-11-23 04:42:15
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:41:52 GMT run: 6471483 subRun: 1 event: 23800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:41:52 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:41:55 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:41:55 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:41:55 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:41:55 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:41:55 GMT run: 6471483 subRun: 1 event: 23800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 245, 164, 202
minwire 0: 808
minwire 1: 2637
minwire 2: 264
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.305248, 0.600603, 0.0941493, 
Output 1: 0.0161795, 0.503121, 0.4745, 0.00619923, 
Output 2: 0.875958, 0.124042, 

475
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1614!
23-Nov-2024 04:41:58 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_data_2024-11-23T_034945Z.root"
23-Nov-2024 04:41:58 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/6e/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.15165       27.2533       476.702       7.76672       72.9429        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0182355     0.0369659     0.071104      0.0361492     0.0127313       100    
reco:linecluster:LineCluster                           0.174088      0.409053       10.8588      0.185984       1.18644        100    
reco:trajcluster:TrajCluster                           0.0508993      0.77704       34.4252      0.0581382      3.97421        100    
reco:pandora:StandardPandora                           0.869086       2.45999       44.5458       1.24024       6.22771        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000126318   0.00599162      0.15024     0.00130682     0.0186439       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000193615    0.0107853      0.27639     0.00247854     0.029899        100    
reco:pandoracalo:Calorimetry                          9.4779e-05     0.0308674     0.998814     0.000848589    0.143571        100    
reco:pandorapid:Chi2ParticleID                         3.687e-05    0.000163836   0.000802903   0.000121799   0.000136632      100    
reco:trkshowersplit:TrackShowerHits                   0.000113384    0.0254342     0.734664     0.000503474     0.11157        100    
reco:pmtrack:PMAlgTrackMaker                          0.00129864      2.90958       82.1549      0.106445       11.7184        100    
reco:pmtrackcalo:Calorimetry                          7.4681e-05     0.0116601     0.429784     0.000796878    0.0543745       100    
reco:pmtrackpid:Chi2ParticleID                        3.2021e-05    0.00029888    0.00434206    0.000138146   0.000611555      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000214234     2.36507       57.8251      0.235928       8.07404        100    
reco:pmtrajfitcalo:Calorimetry                        7.5332e-05     0.029318       1.01247     0.00146858     0.138391        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2672e-05    0.000204218   0.00128569    0.000160608   0.000192625      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00107056      3.24625       88.4914      0.139063       13.3893        100    
reco:pmtracktccalo:Calorimetry                        7.6965e-05    0.00644397     0.160253     0.000813033    0.0226066       100    
reco:pmtracktcpid:Chi2ParticleID                      3.4586e-05    0.000266966   0.00279152    0.000143882   0.000430898      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000224794     1.60328       44.0661      0.216102       5.6233         100    
reco:pmtrajfittccalo:Calorimetry                      8.1313e-05     0.0154261     0.438074     0.000938177    0.0621933       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.9416e-05    0.000340429   0.00327635    0.000154211   0.000591806      100    
reco:blurredcluster:BlurredClustering                 0.00588218     0.898632       7.44196      0.299273       1.44876        100    
reco:emtrkmichelid:EmTrackMichelId                     0.183254       9.48567       155.488       2.13838       23.4699        100    
reco:emshower:EMShower                                0.000338148    0.0246415     0.264177     0.00487248     0.0444721       100    
reco:cvnmap:CVNMapper                                 2.8955e-05    0.00855906     0.0426162    0.00758961     0.0090254       100    
reco:cvneva:CVNEvaluator                              1.8745e-05     0.548204       1.12536      0.767542      0.403623        100    
reco:energyrecnumu:EnergyReco                         0.000380037    0.0175938     0.415336     0.00391677     0.0512919       100    
reco:energyrecnue:EnergyReco                          0.000140845   0.00152815     0.0284203    0.000385983   0.00408704       100    
reco:energyrecnc:EnergyReco                           0.00160792    0.00391089     0.0310722    0.00289156    0.00437646       100    
reco:energyrecnumurange:EnergyReco                    0.000136838    0.0156455     0.407034     0.00272503     0.050547        100    
reco:energyrecnumumcs:EnergyReco                      0.00107311     0.0172807      0.41198       0.00308      0.050753        100    
reco:opdec:Deconvolution                               0.117674      0.272632      0.527397      0.274933      0.0838978       100    
reco:ophitspe:OpHitFinderDeco                          0.494161       1.7866        3.97583       1.78502      0.753483        100    
reco:opflash:OpFlashFinder                            0.00036056    0.00143914    0.00560426    0.00134209    0.00073296       100    
reco:rns:RandomNumberSaver                             2.104e-05    4.40032e-05   0.000450529   3.95695e-05   4.23369e-05      100    
reco:anglereconue:NuAngularReco                       0.000177214   0.00143523     0.0417353    0.000406376   0.00472297       100    
reco:anglereconumu:NuAngularReco                      7.2416e-05    0.00123732     0.0428865    0.000226693   0.00480932       100    
reco:anglereconuepfps:NuAngularReco                   0.000107483   0.00139709     0.0406802    0.000369747   0.00460117       100    
reco:anglereconumupfps:NuAngularReco                  8.1914e-05    0.00125726     0.040066     0.000285418   0.00455381       100    
[art]:TriggerResults:TriggerResultInserter             8.936e-06    1.42195e-05   5.3752e-05    1.33705e-05   5.0968e-06       100    
end_path:out1:RootOutput                               2.786e-06    7.0097e-06    1.9447e-05    7.1985e-06    2.51606e-06      100    
end_path:out1:RootOutput(write)                        0.0259646     0.219134       1.17195      0.219892      0.183088        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38463.7 MB
  Peak resident set size usage (VmHWM): 2406.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_2024-11-23T_034945Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_237_20231208T035034Z_gen_g4_detsim_hitreco__20240510T034436Z_reco2_reco_data_2024-11-23T_034945Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 10:49:57 UTC       justIN version: 01.01.09