Jobsub ID 301824.23@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301824.23@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:51:28
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:27:29
From worker node	Hostname	heplnc155.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:54:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5581s)
	CPU time	1h (6136s = 109%)
Outputting started	2024-11-23 07:27:04
Output files
Finished	2024-11-23 07:27:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ubRun: 1 event: 79699
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:53 GMT run: 6492179 subRun: 1 event: 79699
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:53 GMT run: 6492179 subRun: 1 event: 79699
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:53 GMT run: 6492179 subRun: 1 event: 79699
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:53 GMT run: 6492179 subRun: 1 event: 79699
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1927, 2137, 1698
minwire 0: 726
minwire 1: 1039
minwire 2: 791
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0236734, 0.0396766, 0.93665, 
Output 1: 0.496051, 0.354575, 0.118088, 0.0312866, 
Output 2: 0.0459273, 0.954073, 

718
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2725!
Begin processing the 100th record. run: 6492179 subRun: 1 event: 79700 at 23-Nov-2024 07:26:55 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
127
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 173!
23-Nov-2024 07:27:01 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2_reco_data_2024-11-23T_055409Z.root"
23-Nov-2024 07:27:01 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/60/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.752871       52.6746       753.317       16.8398       117.682        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0241046     0.0572588     0.276714      0.0565049     0.0274048       100    
reco:linecluster:LineCluster                            0.13099      0.903155       33.3329      0.240362       3.6074         100    
reco:trajcluster:TrajCluster                           0.0659376      2.30624       115.403      0.0912727      12.1969        100    
reco:pandora:StandardPandora                           0.0897513      5.06486       65.7506       2.54919       9.91074        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000226053    0.0150283      0.24397     0.00393419     0.0387118       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000311613    0.0188996     0.313257     0.00506774     0.0419862       100    
reco:pandoracalo:Calorimetry                          0.000195677    0.0404181      1.1259      0.00274024     0.142443        100    
reco:pandorapid:Chi2ParticleID                        7.0031e-05    0.000393401   0.00382449    0.00027264    0.000444303      100    
reco:trkshowersplit:TrackShowerHits                   0.000145502    0.0441092     0.930208     0.00202484     0.154447        100    
reco:pmtrack:PMAlgTrackMaker                          0.00153289      4.25437       82.2296      0.312385       12.8077        100    
reco:pmtrackcalo:Calorimetry                          7.6954e-05     0.0187167     0.402048      0.0024013     0.0569759       100    
reco:pmtrackpid:Chi2ParticleID                        3.3313e-05    0.000587917   0.00418799    0.000322428   0.000777658      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000269945     3.57318       61.5212      0.431277       9.87798        100    
reco:pmtrajfitcalo:Calorimetry                        7.3588e-05     0.0406248     0.914231     0.00392154     0.134197        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3091e-05    0.000411233    0.0016305    0.000319477   0.000324473      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000985313     5.14684       123.894      0.395529       16.7039        100    
reco:pmtracktccalo:Calorimetry                        8.1112e-05     0.0119325     0.235684     0.00243131     0.0326658       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3493e-05    0.00055975    0.00589001    0.000335498   0.000788655      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000211646     2.46272       40.929       0.453285       6.53783        100    
reco:pmtrajfittccalo:Calorimetry                      7.6162e-05     0.0267962     0.698423     0.00284343     0.0902229       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.2771e-05    0.000682168   0.00646479    0.000356322   0.000980889      100    
reco:blurredcluster:BlurredClustering                 0.000242203     2.14537       16.016       0.975685       3.28661        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000158346     21.3571       246.334       5.86376       46.5548        100    
reco:emshower:EMShower                                0.00027267     0.0408794     0.554457     0.00972941     0.0779174       100    
reco:cvnmap:CVNMapper                                 2.3624e-05     0.0392937     0.146612      0.033173      0.0327978       100    
reco:cvneva:CVNEvaluator                              2.3654e-05       1.715        3.9213        1.94368      0.997627        100    
reco:energyrecnumu:EnergyReco                         0.000771253    0.0670208      1.98555     0.00959525     0.245401        100    
reco:energyrecnue:EnergyReco                          0.000231313   0.00377091     0.0498259     0.0010552    0.00798778       100    
reco:energyrecnc:EnergyReco                           0.00201652    0.00671902     0.0511096    0.00462484    0.00769809       100    
reco:energyrecnumurange:EnergyReco                    0.000237624    0.0563377      1.71516     0.00303664     0.220542        100    
reco:energyrecnumumcs:EnergyReco                      0.00174411     0.0543096      1.48229     0.00544405     0.197459        100    
reco:opdec:Deconvolution                               0.0943442     0.363266      0.634253       0.35424      0.107534        100    
reco:ophitspe:OpHitFinderDeco                          0.197022       2.44464       4.52702       2.3599       0.969462        100    
reco:opflash:OpFlashFinder                            0.000240921    0.0031005     0.010358     0.00278297    0.00152879       100    
reco:rns:RandomNumberSaver                            3.6408e-05    9.65505e-05   0.000832517   8.2414e-05    8.50293e-05      100    
reco:anglereconue:NuAngularReco                       0.000324517   0.00341123     0.0366149    0.00109051    0.00701667       100    
reco:anglereconumu:NuAngularReco                      0.000103844    0.002461      0.0352303    0.00049999    0.00632651       100    
reco:anglereconuepfps:NuAngularReco                   0.000226624   0.00278919     0.0350765    0.000811162   0.00645948       100    
reco:anglereconumupfps:NuAngularReco                  0.000190646   0.00255466     0.0356144    0.000559913   0.00643883       100    
[art]:TriggerResults:TriggerResultInserter            1.2183e-05    2.91962e-05   9.3054e-05    2.63195e-05   1.25689e-05      100    
end_path:out1:RootOutput                               4.819e-06    1.40125e-05   5.1977e-05    1.2037e-05    8.56965e-06      100    
end_path:out1:RootOutput(write)                        0.0277155     0.373343       1.44974       0.35652      0.262728        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39453.7 MB
  Peak resident set size usage (VmHWM): 2280.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2_reco_2024-11-23T_055409Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_796_20231208T050858Z_gen_g4_detsim_hitreco__20240510T042501Z_reco2_reco_data_2024-11-23T_055409Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work