Jobsub ID 108678.94@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108678.94@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:53:36
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:18:45
From worker node	Hostname	comp11-05
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:21:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root
Jobscript	Exit code	0
	Real time	56m (3415s)
	CPU time	52m (3169s = 92%)
Outputting started	2024-11-23 03:18:30
Output files
Finished	2024-11-23 03:18:45
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ax tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.366895, 0.150907, 0.482198, 
Output 1: 0.175463, 0.749367, 0.0731487, 0.00202133, 
Output 2: 0.629757, 0.370243, 

390
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1009!
Begin processing the 100th record. run: 50640579 subRun: 1 event: 16400 at 23-Nov-2024 03:18:23 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 03:18:25 GMT run: 50640579 subRun: 1 event: 16400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 03:18:25 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 03:18:25 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:18:25 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:18:25 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
334
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 802!
23-Nov-2024 03:18:28 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_data_2024-11-23T_022137Z.root"
23-Nov-2024 03:18:29 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/95/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.60319       32.1315       500.439       10.1962       73.0574        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0119326     0.105281      0.347804      0.0862432     0.0690898       100    
reco:linecluster:LineCluster                           0.129156      0.649324       15.2453       0.33633       1.90278        100    
reco:trajcluster:TrajCluster                           0.0354564      1.38254       48.7054      0.132114       6.32164        100    
reco:pandora:StandardPandora                            1.27144       2.29962       34.6791       1.41769       4.32442        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136824   0.00753964     0.162866     0.00117662     0.0229852       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174329    0.0100327     0.255533     0.00217992     0.0282401       100    
reco:pandoracalo:Calorimetry                          0.000103824    0.0204136     0.648162     0.000725645    0.0845836       100    
reco:pandorapid:Chi2ParticleID                        3.8518e-05    0.000151705    0.0016157    0.000102198   0.000189259      100    
reco:trkshowersplit:TrackShowerHits                   0.000101773    0.0237475     0.745243     0.000528513    0.105862        100    
reco:pmtrack:PMAlgTrackMaker                           0.0012407      2.12522       56.9274      0.0993802      7.36239        100    
reco:pmtrackcalo:Calorimetry                          8.5034e-05     0.0120812     0.287513     0.000747604    0.0460896       100    
reco:pmtrackpid:Chi2ParticleID                        3.2427e-05    0.000251037   0.00264827    0.000109866   0.000413272      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000188925     1.96688       63.3462      0.135328       7.38016        100    
reco:pmtrajfitcalo:Calorimetry                        6.7342e-05     0.0225541     0.708835     0.000899866    0.0924859       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8709e-05    0.000172048   0.000949965   0.000124433   0.000167481      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00108181      2.62517       52.9775      0.145458       8.43615        100    
reco:pmtracktccalo:Calorimetry                        7.7769e-05    0.00816001     0.226505     0.000764118    0.0300277       100    
reco:pmtracktcpid:Chi2ParticleID                       3.003e-05    0.000229812   0.00207847    0.000113318   0.000329333      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000183366     1.5012        30.8983      0.216494       4.34679        100    
reco:pmtrajfittccalo:Calorimetry                      6.6322e-05     0.0200804     0.551616     0.000885027    0.081207        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7442e-05    0.000325615   0.00366499    0.000119487   0.000582703      100    
reco:blurredcluster:BlurredClustering                  0.0133266      1.01722       8.7115       0.412838       1.54145        100    
reco:emtrkmichelid:EmTrackMichelId                     0.257743       14.832        189.243       3.34896       32.0932        100    
reco:emshower:EMShower                                0.00024758     0.0238582     0.333874      0.0062063     0.048152        100    
reco:cvnmap:CVNMapper                                 2.2383e-05     0.0129642     0.0457533     0.014241      0.0100876       100    
reco:cvneva:CVNEvaluator                              1.6267e-05     0.564942      0.924039      0.775491      0.361959        100    
reco:energyrecnumu:EnergyReco                         0.000277132    0.0451198     0.758817     0.00478949     0.124918        100    
reco:energyrecnue:EnergyReco                          0.000144754   0.00250383     0.0522286    0.000452427    0.0076228       100    
reco:energyrecnc:EnergyReco                           0.00191867    0.00424952     0.0545694    0.00217715    0.00806167       100    
reco:energyrecnumurange:EnergyReco                    0.000197309    0.042658       0.75493     0.00206497     0.123875        100    
reco:energyrecnumumcs:EnergyReco                      0.000751715    0.0437345     0.753968     0.00274225     0.123557        100    
reco:opdec:Deconvolution                               0.122419      0.388723      0.823169      0.381002      0.148201        100    
reco:ophitspe:OpHitFinderDeco                           0.50105       2.15405       4.49668       2.05595      0.904597        100    
reco:opflash:OpFlashFinder                            0.000329872   0.00125609    0.00399749    0.00109454    0.000603208      100    
reco:rns:RandomNumberSaver                            1.8741e-05    2.89415e-05   0.000243761    2.446e-05    2.25673e-05      100    
reco:anglereconue:NuAngularReco                       0.000157059    0.0019593     0.0479242    0.000273376    0.006696        100    
reco:anglereconumu:NuAngularReco                      7.0289e-05    0.00174863     0.0464089    0.000138903   0.00649076       100    
reco:anglereconuepfps:NuAngularReco                   0.000110517   0.00185383     0.0456327    0.000239407   0.00648916       100    
reco:anglereconumupfps:NuAngularReco                  0.000101181    0.0018088     0.0454002    0.000205797   0.00645496       100    
[art]:TriggerResults:TriggerResultInserter             8.588e-06    1.09491e-05   4.6994e-05    1.0145e-05    3.96612e-06      100    
end_path:out1:RootOutput                               2.291e-06    4.30795e-06   1.9948e-05    3.1845e-06    2.42098e-06      100    
end_path:out1:RootOutput(write)                        0.0192264      0.20818      0.912074      0.199646      0.160099        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9601.18 MB
  Peak resident set size usage (VmHWM): 2446.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_2024-11-23T_022137Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_data_2024-11-23T_022137Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work