Jobsub ID 108742.55@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.55@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 05:27:12
From worker node	Hostname	b9p12p1803.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2.root
Jobscript	Exit code	0
	Real time	46m (2764s)
	CPU time	46m (2816s = 101%)
Outputting started	2024-11-23 05:26:54
Output files
Finished	2024-11-23 05:27:12
List job events Wrapper job log
Jobscript log (last 10,000 characters)

SG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:26:46 CET run: 50604274 subRun: 1 event: 55800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:26:46 CET run: 50604274 subRun: 1 event: 55800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:46 CET run: 50604274 subRun: 1 event: 55800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:46 CET run: 50604274 subRun: 1 event: 55800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:26:48 CET run: 50604274 subRun: 1 event: 55800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 46, 44, 40
minwire 0: 921
minwire 1: 2335
minwire 2: 328
Used alternate method to get min and max wires due to vertex determination failure: 128, 327
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 229, 428
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 135, 334
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.99241, 0.000213033, 0.0073769, 
Output 1: 0.263198, 0.671507, 0.0613804, 0.00391424, 
Output 2: 0.993248, 0.00675165, 

474
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1429!
23-Nov-2024 06:26:52 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2_reco_data_2024-11-23T_044055Z.root"
23-Nov-2024 06:26:52 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/5a/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.63682       25.6189       421.884       9.8479        50.8643        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0170403     0.0433095     0.158857      0.0429337     0.0180994       100    
reco:linecluster:LineCluster                           0.0941934      0.27601       4.44533      0.166955      0.554628        100    
reco:trajcluster:TrajCluster                           0.0478459     0.613698       24.4237       0.05774       2.58289        100    
reco:pandora:StandardPandora                           0.059655       2.3999        44.9561       1.28693       5.1586         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000152712   0.00660559     0.222767     0.00146688     0.0227975       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000194492    0.0121887     0.185847     0.00366167     0.0259828       100    
reco:pandoracalo:Calorimetry                          9.6601e-05     0.0214518     0.879032     0.00117719     0.0916523       100    
reco:pandorapid:Chi2ParticleID                        3.5631e-05    0.000177277   0.00158206    0.000128152   0.00019339       100    
reco:trkshowersplit:TrackShowerHits                   9.9731e-05     0.0255295      0.94673     0.000751484    0.102079        100    
reco:pmtrack:PMAlgTrackMaker                          0.00101852      2.14751       53.2489      0.198196       6.17958        100    
reco:pmtrackcalo:Calorimetry                          5.8261e-05    0.00890196     0.277973     0.00111283     0.0300135       100    
reco:pmtrackpid:Chi2ParticleID                         2.399e-05    0.000291179    0.0022977    0.000157177   0.000393483      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000168172     1.87121       39.7412       0.31456       4.84765        100    
reco:pmtrajfitcalo:Calorimetry                         6.327e-05     0.0205632     0.779579     0.00174421     0.0837141       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.551e-05    0.00020155    0.00151494    0.000155896   0.000187519      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00085432      2.65078       78.6879      0.233626       8.71973        100    
reco:pmtracktccalo:Calorimetry                        5.7921e-05    0.00591672     0.143913     0.000979197    0.0168698       100    
reco:pmtracktcpid:Chi2ParticleID                       2.439e-05    0.000273546   0.00268919    0.000154052   0.000370065      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000142401     1.45819       28.3474      0.303558       3.61333        100    
reco:pmtrajfittccalo:Calorimetry                       6.673e-05     0.0143363      0.47655      0.0011543     0.052484        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.72e-05     0.000374249   0.00434428    0.000163367   0.000609616      100    
reco:blurredcluster:BlurredClustering                 0.000146511    0.909246       9.18263      0.348448       1.38063        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000168912     9.6147        129.519       2.68509       18.3317        100    
reco:emshower:EMShower                                0.000233103    0.0266918     0.174799     0.00461959     0.0395102       100    
reco:cvnmap:CVNMapper                                 2.4151e-05     0.0109756     0.0351844     0.0141089    0.00848115       100    
reco:cvneva:CVNEvaluator                              1.6541e-05     0.688368       1.27752      0.936091      0.472667        100    
reco:energyrecnumu:EnergyReco                         0.000355344    0.0314564      1.09317     0.00463782     0.113305        100    
reco:energyrecnue:EnergyReco                          0.000120352   0.00187625     0.045842     0.000526246   0.00476171       100    
reco:energyrecnc:EnergyReco                           0.00172965    0.00444042     0.0485558    0.00335097    0.00486774       100    
reco:energyrecnumurange:EnergyReco                    0.000167282    0.0304022      1.19502     0.00304288     0.122757        100    
reco:energyrecnumumcs:EnergyReco                      0.00132116     0.0287009     0.870293     0.00391911     0.0922622       100    
reco:opdec:Deconvolution                               0.0576419     0.273328      0.454858      0.279268      0.0880301       100    
reco:ophitspe:OpHitFinderDeco                          0.231527       2.1594        4.05398       2.10481       0.95365        100    
reco:opflash:OpFlashFinder                            0.000298513   0.00154898    0.00689915    0.00131827    0.000846674      100    
reco:rns:RandomNumberSaver                            2.0911e-05    4.36611e-05   0.000403155   4.10305e-05   3.71922e-05      100    
reco:anglereconue:NuAngularReco                       0.000192282   0.00154722     0.0511707    0.000393805   0.00518323       100    
reco:anglereconumu:NuAngularReco                      7.8791e-05    0.00130829     0.0515304    0.000218203   0.00521489       100    
reco:anglereconuepfps:NuAngularReco                   0.000160152   0.00153186     0.0560284    0.000371294   0.00564384       100    
reco:anglereconumupfps:NuAngularReco                  0.000130402   0.00138982     0.0514076    0.000288324   0.00519488       100    
[art]:TriggerResults:TriggerResultInserter              8.8e-06     1.54216e-05   5.1701e-05     1.473e-05    5.45198e-06      100    
end_path:out1:RootOutput                               3.11e-06     7.15727e-06    2.518e-05      7.4e-06     2.41612e-06      100    
end_path:out1:RootOutput(write)                        0.0185982      0.25235      0.997089       0.2622       0.176027        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21664.5 MB
  Peak resident set size usage (VmHWM): 2251.63 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2_reco_2024-11-23T_044055Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_557_20231207T025521Z_gen_g4_detsim_hitreco__20240510T050826Z_reco2_reco_data_2024-11-23T_044055Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work