Jobsub ID 108736.87@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108736.87@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:19:19
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 06:25:26
From worker node	Hostname	b9g37p7552.cern.ch
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:21:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7410s)
	CPU time	2h (7236s = 97%)
Outputting started	2024-11-23 06:25:08
Output files
Finished	2024-11-23 06:25:26
List job events Wrapper job log
Jobscript log (last 10,000 characters)

max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00844868, 0.991452, 9.88507e-05, 
Output 1: 0.992116, 0.00758139, 0.000243247, 5.93374e-05, 
Output 2: 0.993144, 0.00685576, 

715
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2611!
Begin processing the 100th record. run: 6389847 subRun: 1 event: 58200 at 23-Nov-2024 07:25:01 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:25:03 CET run: 6389847 subRun: 1 event: 58200
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:25:03 CET run: 6389847 subRun: 1 event: 58200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:03 CET run: 6389847 subRun: 1 event: 58200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:25:04 CET run: 6389847 subRun: 1 event: 58200
Comparing two wires in the same plane: return failure
%MSG
322
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 677!
23-Nov-2024 07:25:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2_reco_data_2024-11-23T_042141Z.root"
23-Nov-2024 07:25:06 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/06/b7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.578969       71.6247       1719.85       13.4428       225.224        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0189878     0.0716868      1.21876      0.0484022     0.136747        100    
reco:linecluster:LineCluster                           0.0936876      1.53519       66.752       0.178997       7.35624        100    
reco:trajcluster:TrajCluster                           0.0471301      4.45484       138.304      0.0689237      19.1408        100    
reco:pandora:StandardPandora                           0.0687952      6.70203       229.201       1.82957       25.2259        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000171699     0.02404      0.754616     0.00215765     0.0902768       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000235066    0.0222095     0.439792     0.00384127     0.0529544       100    
reco:pandoracalo:Calorimetry                          0.000128492    0.0816116      1.82652     0.00138695     0.308635        100    
reco:pandorapid:Chi2ParticleID                        5.1072e-05    0.000239151   0.00317112    0.000160839   0.000341839      100    
reco:trkshowersplit:TrackShowerHits                   0.000121529    0.0735068      2.50098     0.00118802     0.312849        100    
reco:pmtrack:PMAlgTrackMaker                          0.00164462      6.62918       146.539      0.282858       23.0814        100    
reco:pmtrackcalo:Calorimetry                          0.000103158    0.0462833      1.80287      0.0011772     0.211514        100    
reco:pmtrackpid:Chi2ParticleID                        4.2928e-05    0.000520386    0.0059038    0.000178512   0.000992255      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000266023     11.1907       761.621      0.405899       76.0745        100    
reco:pmtrajfitcalo:Calorimetry                        9.9476e-05     0.0837539      1.70038     0.00217245     0.307721        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.1816e-05    0.000273589   0.00168086    0.000189418   0.000286644      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0013748      8.39012       192.471      0.301651       29.2538        100    
reco:pmtracktccalo:Calorimetry                        0.000106026    0.0265617      1.03418     0.00137092     0.118437        100    
reco:pmtracktcpid:Chi2ParticleID                      4.3657e-05    0.00048125     0.0068632    0.000195994   0.000930595      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000236628     4.03651       80.758       0.396211       12.7008        100    
reco:pmtrajfittccalo:Calorimetry                      9.8447e-05     0.0721727      2.49922     0.00143275     0.311289        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9256e-05    0.000691757    0.0075902    0.000201355   0.00139021       100    
reco:blurredcluster:BlurredClustering                 9.6693e-05      1.65693       14.4857      0.719176       2.58448        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000141506     22.8605       322.597       5.10798       53.0443        100    
reco:emshower:EMShower                                0.00025532     0.056082      0.654983     0.00842115     0.107852        100    
reco:cvnmap:CVNMapper                                 2.7069e-05     0.0166446     0.0774286     0.0168258     0.0143677       100    
reco:cvneva:CVNEvaluator                              2.0473e-05     0.746199       1.60823      0.924955      0.492715        100    
reco:energyrecnumu:EnergyReco                         0.000348212    0.0553229     0.955471     0.00555079     0.145933        100    
reco:energyrecnue:EnergyReco                          0.000204091   0.00616911     0.137967     0.000899823    0.0196444       100    
reco:energyrecnc:EnergyReco                           0.00258679    0.00855163     0.129684     0.00353114     0.0187782       100    
reco:energyrecnumurange:EnergyReco                    0.000273942    0.0574327      1.50364     0.00326732     0.183756        100    
reco:energyrecnumumcs:EnergyReco                      0.00261578     0.0529985      1.01151     0.00496956     0.145408        100    
reco:opdec:Deconvolution                               0.0550258     0.358272       1.13445      0.350948      0.147493        100    
reco:ophitspe:OpHitFinderDeco                          0.141213        1.935        3.70688       1.91039      0.823671        100    
reco:opflash:OpFlashFinder                            0.000200757   0.00185649    0.00585724    0.00174398    0.000906577      100    
reco:rns:RandomNumberSaver                             2.096e-05    3.93874e-05   0.000379022   3.57535e-05   3.52646e-05      100    
reco:anglereconue:NuAngularReco                       0.000169045   0.00480684     0.120363     0.000487975    0.0167668       100    
reco:anglereconumu:NuAngularReco                      0.000104192   0.00466848     0.124934     0.000316512    0.0170469       100    
reco:anglereconuepfps:NuAngularReco                   0.000156315   0.00462462     0.110196     0.000470231    0.0159818       100    
reco:anglereconumupfps:NuAngularReco                  0.000147335   0.00463603      0.11304     0.000411129    0.0164048       100    
[art]:TriggerResults:TriggerResultInserter            1.2791e-05    1.83354e-05   9.0797e-05    1.7149e-05    8.21437e-06      100    
end_path:out1:RootOutput                               3.267e-06    6.72675e-06   2.6182e-05    6.8095e-06    3.39756e-06      100    
end_path:out1:RootOutput(write)                        0.0145303     0.349002       1.76882      0.310517      0.309807        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9721.47 MB
  Peak resident set size usage (VmHWM): 2518.17 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2_reco_2024-11-23T_042141Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_581_20231201T191500Z_gen_g4_detsim_hitreco__20240507T214433Z_reco2_reco_data_2024-11-23T_042141Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work