Jobsub ID 301641.142@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301641.142@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:12:27
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 03:27:27
From worker node	Hostname	tdm004.pic.es
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:22:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3906s)
	CPU time	1h (5808s = 148%)
Outputting started	2024-11-23 03:27:09
Output files
Finished	2024-11-23 03:27:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

0572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:26:56 CET run: 50572032 subRun: 1 event: 29100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 235, 220, 130
minwire 0: 309
minwire 1: 1607
minwire 2: 72
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.281058, 0.0298128, 0.689129, 
Output 1: 0.0201707, 0.27068, 0.6493, 0.0598496, 
Output 2: 0.551776, 0.448224, 

581
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1924!
23-Nov-2024 04:27:07 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2_reco_data_2024-11-23T_022207Z.root"
23-Nov-2024 04:27:08 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/32/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.67942       35.7229       1268.17       10.6725       144.416        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0334209     0.061001      0.0939641     0.057577      0.0179034       100    
reco:linecluster:LineCluster                           0.304116       1.25468       82.6732      0.320223       8.24135        100    
reco:trajcluster:TrajCluster                           0.0989226      3.82978       307.807      0.109509       30.9498        100    
reco:pandora:StandardPandora                            1.28679       3.38943       100.47        1.39717       13.1185        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143243   0.00733964      0.23344     0.00183758     0.028944        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000190103   0.00955485     0.0951755    0.00343598     0.0175894       100    
reco:pandoracalo:Calorimetry                          0.000103882    0.0336152      1.47929     0.00109553      0.20562        100    
reco:pandorapid:Chi2ParticleID                        4.0951e-05    0.000160415   0.000836086   0.000120342   0.000134457      100    
reco:trkshowersplit:TrackShowerHits                   0.000110252    0.0401939      2.53685     0.000698307    0.270115        100    
reco:pmtrack:PMAlgTrackMaker                           0.0013676      3.06557       146.346      0.225237       16.3396        100    
reco:pmtrackcalo:Calorimetry                          7.4031e-05     0.0146548     0.650271     0.000935172    0.0816507       100    
reco:pmtrackpid:Chi2ParticleID                        3.0811e-05    0.000294613   0.00501405    0.000142283   0.000658474      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000183674     3.24639        148.9       0.320186       17.5902        100    
reco:pmtrajfitcalo:Calorimetry                        6.6131e-05     0.0379734      1.7586      0.00151104     0.233551        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7661e-05    0.000185772   0.00116091    0.000136297   0.000180969      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114852      3.5164        156.713      0.243783       18.0874        100    
reco:pmtracktccalo:Calorimetry                        6.7131e-05    0.00850519     0.331155     0.000879881    0.0415066       100    
reco:pmtracktcpid:Chi2ParticleID                       2.819e-05    0.000256761   0.00359528    0.000140912   0.000484955      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000191364     2.3964        87.0083      0.308322       11.7326        100    
reco:pmtrajfittccalo:Calorimetry                      6.6071e-05     0.0230914      1.03894     0.00108473     0.133999        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7261e-05    0.000340367   0.00592591    0.000156608   0.000830811      100    
reco:blurredcluster:BlurredClustering                  0.0109303      1.1345        10.8508      0.628407       1.67196        100    
reco:emtrkmichelid:EmTrackMichelId                     0.232353       10.0645       221.65        2.74726       30.0209        100    
reco:emshower:EMShower                                0.000284445    0.0302787     0.596578     0.00475234     0.0836371       100    
reco:cvnmap:CVNMapper                                 2.1891e-05    0.00953123     0.0568317     0.0073301    0.00958745       100    
reco:cvneva:CVNEvaluator                               1.714e-05     0.985693       1.56396       1.23587      0.559534        100    
reco:energyrecnumu:EnergyReco                         0.000358256    0.0556043      2.6934      0.00480198     0.285846        100    
reco:energyrecnue:EnergyReco                          0.000142163   0.00243228     0.0836312    0.000479664   0.00999287       100    
reco:energyrecnc:EnergyReco                           0.00240002    0.00499246     0.0868248     0.0029992     0.0102553       100    
reco:energyrecnumurange:EnergyReco                    0.000195603    0.0523287      2.66404     0.00266735     0.282303        100    
reco:energyrecnumumcs:EnergyReco                       0.0015308     0.0549725      2.7575      0.00344855     0.290606        100    
reco:opdec:Deconvolution                               0.118748       0.42636      0.728029      0.434344       0.10285        100    
reco:ophitspe:OpHitFinderDeco                          0.403774       1.68457       2.97887       1.67982      0.525623        100    
reco:opflash:OpFlashFinder                            0.000258954   0.00136279    0.00381068    0.00132704    0.000594504      100    
reco:rns:RandomNumberSaver                            1.8101e-05    3.03494e-05   0.000289335   2.35255e-05   2.80368e-05      100    
reco:anglereconue:NuAngularReco                       0.000155483   0.00204252     0.0800751    0.000363732   0.00922878       100    
reco:anglereconumu:NuAngularReco                      7.3282e-05    0.00188589     0.0799686    0.000204614   0.00921637       100    
reco:anglereconuepfps:NuAngularReco                   0.000126002   0.00199983     0.0785426    0.000331806   0.00904276       100    
reco:anglereconumupfps:NuAngularReco                  0.000107702   0.00197579     0.0816052    0.000291005   0.00935648       100    
[art]:TriggerResults:TriggerResultInserter             9.861e-06     1.364e-05    5.0431e-05    1.15555e-05   7.63774e-06      100    
end_path:out1:RootOutput                               2.96e-06     5.54299e-06   3.6921e-05     3.73e-06     3.95176e-06      100    
end_path:out1:RootOutput(write)                        0.0247373     0.270101       1.64735      0.264694       0.22303        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39798.9 MB
  Peak resident set size usage (VmHWM): 2634.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2_reco_2024-11-23T_022207Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_290_20231203T045733Z_gen_g4_detsim_hitreco__20240508T073956Z_reco2_reco_data_2024-11-23T_022207Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work