Jobsub ID 108742.16@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.16@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 05:22:59
From worker node	Hostname	tds211.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:38:01
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2684s)
	CPU time	45m (2735s = 101%)
Outputting started	2024-11-23 05:22:45
Output files
Finished	2024-11-23 05:22:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

2500 at 23-Nov-2024 06:22:41 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:22:42 CET run: 74574126 subRun: 1 event: 2500
1st wire C:0 T:6 P:2 W:752 does not exist (max wire number: 480)
%MSG
338
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 870!
23-Nov-2024 06:22:44 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2_reco_data_2024-11-23T_043803Z.root"
23-Nov-2024 06:22:44 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/7e/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.78656       25.9306       408.55        7.72779       61.5608        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00100695    0.00184072     0.0246004    0.00152655    0.00231987       100    
reco:linecluster:LineCluster                           0.011074      0.247419       13.6115      0.0165397      1.45879        100    
reco:trajcluster:TrajCluster                          0.00109802     0.942351       37.1675     0.00935779      4.52692        100    
reco:pandora:StandardPandora                            1.07812       2.33789       48.7051       1.15311       5.85784        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130386   0.00665453     0.100669     0.00116786     0.0169114       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000175229    0.0128163     0.193083     0.00230971     0.0302379       100    
reco:pandoracalo:Calorimetry                          0.000100736    0.0248005     0.611556     0.000826137    0.0931847       100    
reco:pandorapid:Chi2ParticleID                        4.1792e-05     0.0001477    0.00112316    0.000106643   0.000142318      100    
reco:trkshowersplit:TrackShowerHits                   0.000105973    0.016634      0.566383     0.000502906    0.0673651       100    
reco:pmtrack:PMAlgTrackMaker                           0.0012322      2.20905       52.3832      0.117572       7.27382        100    
reco:pmtrackcalo:Calorimetry                          8.1364e-05     0.0139561     0.336568     0.00065906     0.0494828       100    
reco:pmtrackpid:Chi2ParticleID                        3.4345e-05    0.000250285   0.00294802    0.000119009   0.000453511      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000191054     1.67089       27.5284      0.207668       4.43715        100    
reco:pmtrajfitcalo:Calorimetry                        7.5594e-05     0.0258966     0.626854     0.00103218     0.0969747       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3454e-05    0.000165675   0.00115513    0.000118425   0.00016857       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102705      2.69672       70.3622      0.139871       9.32866        100    
reco:pmtracktccalo:Calorimetry                        8.1027e-05    0.00965254     0.221497     0.000630553    0.0317422       100    
reco:pmtracktcpid:Chi2ParticleID                      3.4342e-05    0.00023384    0.00206549    0.000123607   0.000355757      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000182375     1.49258       26.6727      0.240833       4.11875        100    
reco:pmtrajfittccalo:Calorimetry                      7.7039e-05     0.0192224     0.504466     0.00092318     0.070901        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3095e-05    0.000316978   0.00345011    0.000144471   0.000576513      100    
reco:blurredcluster:BlurredClustering                 0.00472441     0.956802       6.42587      0.431998       1.33137        100    
reco:emtrkmichelid:EmTrackMichelId                     0.251537       11.1255       140.526       2.61145       24.5788        100    
reco:emshower:EMShower                                 0.0002465     0.0216324     0.319756     0.00424432     0.0445089       100    
reco:cvnmap:CVNMapper                                 2.5859e-05    0.00972738     0.0380917    0.00957667     0.0088236       100    
reco:cvneva:CVNEvaluator                              1.8501e-05     0.443777      0.713336      0.653609      0.311792        100    
reco:energyrecnumu:EnergyReco                         0.000264808    0.0641732      3.98497     0.00312296     0.403713        100    
reco:energyrecnue:EnergyReco                          0.000155336   0.00237309     0.0325269    0.000397794   0.00599179       100    
reco:energyrecnc:EnergyReco                           0.00200764    0.00436467     0.0365879    0.00231477    0.00641297       100    
reco:energyrecnumurange:EnergyReco                    0.000161042    0.0630586      3.96154     0.00213855     0.402048        100    
reco:energyrecnumumcs:EnergyReco                      0.000591352    0.063868       3.96026     0.00267844     0.401171        100    
reco:opdec:Deconvolution                               0.0434041     0.158612      0.528014      0.154913      0.0688079       100    
reco:ophitspe:OpHitFinderDeco                          0.265606       1.07353       2.19056       1.07226      0.430131        100    
reco:opflash:OpFlashFinder                            0.000352575    0.0012181    0.00373037    0.00115774     0.0005567       100    
reco:rns:RandomNumberSaver                            1.8125e-05    2.78155e-05   0.000273088   2.2853e-05    2.55611e-05      100    
reco:anglereconue:NuAngularReco                       0.000113012   0.00188717     0.029915     0.000257194   0.00517473       100    
reco:anglereconumu:NuAngularReco                      8.3037e-05    0.00173789     0.0297291    0.000174835   0.00503469       100    
reco:anglereconuepfps:NuAngularReco                   0.000126447   0.00186819     0.0298894    0.000299445   0.00508525       100    
reco:anglereconumupfps:NuAngularReco                  0.000118714   0.00180451     0.0297307    0.000249393   0.00503399       100    
[art]:TriggerResults:TriggerResultInserter            1.0139e-05    1.24594e-05   3.9715e-05    1.14525e-05   3.4109e-06       100    
end_path:out1:RootOutput                               2.615e-06    4.28435e-06   1.5741e-05     3.476e-06    2.00487e-06      100    
end_path:out1:RootOutput(write)                        0.0241683     0.203755      0.839613      0.208893       0.15902        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8374.26 MB
  Peak resident set size usage (VmHWM): 2278.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2_reco_2024-11-23T_043803Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_24_20231207T233514Z_gen_g4_detsim_hitreco__20240510T035158Z_reco2_reco_data_2024-11-23T_043803Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work