Jobsub ID 301836.148@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301836.148@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:10:11
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-23 07:39:29
From worker node	Hostname	mff-magic10.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:12:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5202s)
	CPU time	1h (5238s = 100%)
Outputting started	2024-11-23 07:39:15
Output files
Finished	2024-11-23 07:39:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

8 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:39:08 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:39:08 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:39:08 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:39:08 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:39:11 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:39:11 CET run: 66009532 subRun: 1 event: 73400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 61, 74, 62
minwire 0: 17
minwire 1: 2340
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.987328, 0.00540369, 0.00726816, 
Output 1: 0.123856, 0.506627, 0.320526, 0.0489907, 
Output 2: 0.982645, 0.0173549, 

293
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 720!
23-Nov-2024 08:39:14 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2_reco_data_2024-11-23T_061237Z.root"
23-Nov-2024 08:39:14 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/df/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.36228       50.7183       1311.49       11.8504       162.191        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00167542     0.0177265     0.122456      0.0141513     0.0165387       100    
reco:linecluster:LineCluster                           0.0131462      0.58653       32.1722      0.0371594      3.58729        100    
reco:trajcluster:TrajCluster                          0.00177369      2.43595       93.3991      0.0216485      12.397         100    
reco:pandora:StandardPandora                            1.47423       8.48055       544.345       1.73315       54.492         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00017199     0.0117855     0.259552     0.00267181     0.0347459       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000238976    0.0247757     0.625604     0.00454978     0.077088        100    
reco:pandoracalo:Calorimetry                          0.000124187    0.0673935      2.69534      0.0018434     0.350029        100    
reco:pandorapid:Chi2ParticleID                        5.0414e-05    0.000223923   0.00237217    0.000166187   0.000258122      100    
reco:trkshowersplit:TrackShowerHits                   0.000152658    0.041411       1.44513     0.00112739     0.190913        100    
reco:pmtrack:PMAlgTrackMaker                          0.00162935      4.16836       114.392      0.314187       15.7362        100    
reco:pmtrackcalo:Calorimetry                          0.000104144    0.0277862     0.721576     0.00138824     0.109185        100    
reco:pmtrackpid:Chi2ParticleID                        4.8347e-05    0.000452434   0.00489928    0.000184159   0.000792318      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000287525     3.35851       106.229      0.472948       12.4189        100    
reco:pmtrajfitcalo:Calorimetry                        0.000101869    0.0678678      2.6659      0.00213084     0.348847        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.8819e-05    0.000256397   0.00121139    0.000190933   0.000212182      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00141027      5.76271       175.963      0.304405       22.8461        100    
reco:pmtracktccalo:Calorimetry                        0.000107117    0.0145466     0.272234     0.00145158     0.044863        100    
reco:pmtracktcpid:Chi2ParticleID                      4.5617e-05    0.000376027   0.00334054    0.000200155   0.000527883      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000244358     2.58831       50.0332      0.441366       7.6182         100    
reco:pmtrajfittccalo:Calorimetry                      9.6875e-05     0.0363494      1.0905       0.0016193     0.153608        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.1913e-05    0.000506962   0.00593748    0.000222558   0.000900638      100    
reco:blurredcluster:BlurredClustering                  0.0136111      1.48468       11.235       0.612614       2.17355        100    
reco:emtrkmichelid:EmTrackMichelId                     0.331674       18.1472       254.543       4.5257        41.8746        100    
reco:emshower:EMShower                                0.000319714    0.0479484     0.451852     0.00728196     0.0876492       100    
reco:cvnmap:CVNMapper                                 3.0456e-05     0.0185212     0.0541107     0.017426      0.0120831       100    
reco:cvneva:CVNEvaluator                              2.2754e-05     0.863958       4.66499      0.954346      0.540496        100    
reco:energyrecnumu:EnergyReco                         0.000371504    0.0851423      3.46492     0.00880515     0.362731        100    
reco:energyrecnue:EnergyReco                          0.000225465   0.00376045     0.0717433    0.000869987    0.0104854       100    
reco:energyrecnc:EnergyReco                            0.0027168    0.00625729     0.071072     0.00364784    0.00952912       100    
reco:energyrecnumurange:EnergyReco                    0.000256912    0.0794328      3.54393     0.00332039     0.366103        100    
reco:energyrecnumumcs:EnergyReco                      0.00102118     0.0778657      3.29697     0.00502772     0.342314        100    
reco:opdec:Deconvolution                               0.0347614     0.235284      0.622683      0.239121      0.102749        100    
reco:ophitspe:OpHitFinderDeco                          0.231467       1.61947       3.59997       1.72319      0.667672        100    
reco:opflash:OpFlashFinder                            0.000350301   0.00181133    0.00632808    0.00173578    0.000863947      100    
reco:rns:RandomNumberSaver                            2.1631e-05    4.20211e-05   0.000551644   3.87875e-05   5.22472e-05      100    
reco:anglereconue:NuAngularReco                       0.000186524    0.0027638     0.0585851    0.000473161   0.00808239       100    
reco:anglereconumu:NuAngularReco                      0.000103751   0.00251832     0.0576787    0.000326979   0.00792259       100    
reco:anglereconuepfps:NuAngularReco                   0.00015681     0.0027169     0.0575902    0.00043293    0.00803456       100    
reco:anglereconumupfps:NuAngularReco                  0.000150607   0.00270542     0.0579887    0.000408053   0.00806285       100    
[art]:TriggerResults:TriggerResultInserter             1.244e-05    1.73713e-05   0.000118097   1.57325e-05   1.08417e-05      100    
end_path:out1:RootOutput                               2.739e-06    5.85884e-06   3.5662e-05    6.1615e-06    3.66354e-06      100    
end_path:out1:RootOutput(write)                        0.0224135     0.341445       1.52033      0.308121      0.242476        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9710.31 MB
  Peak resident set size usage (VmHWM): 2499.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2_reco_2024-11-23T_061237Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_733_20231203T233702Z_gen_g4_detsim_hitreco__20240509T210942Z_reco2_reco_data_2024-11-23T_061237Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work