Jobsub ID 301744.45@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301744.45@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:56:37
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 06:23:24
From worker node	Hostname	heplnc147.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:58:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5080s)
	CPU time	1h (4862s = 95%)
Outputting started	2024-11-23 06:22:59
Output files
Finished	2024-11-23 06:23:24
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00816722, 0.000868186, 0.990965, 
Output 1: 0.452997, 0.328541, 0.136842, 0.0816198, 
Output 2: 0.572348, 0.427652, 

727
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2639!
Begin processing the 100th record. run: 50624901 subRun: 1 event: 1900 at 23-Nov-2024 06:22:51 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:22:54 GMT run: 50624901 subRun: 1 event: 1900
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:22:54 GMT run: 50624901 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:22:54 GMT run: 50624901 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:22:54 GMT run: 50624901 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:22:55 GMT run: 50624901 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
142
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 205!
23-Nov-2024 06:22:58 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2_reco_data_2024-11-23T_045824Z.root"
23-Nov-2024 06:22:58 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1b/91/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.26099       48.5823       1325.99       14.4831       148.315        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00520437     0.0992167     0.464277      0.0472984     0.104353        100    
reco:linecluster:LineCluster                           0.0359031      1.50496       112.439      0.0885184      11.1803        100    
reco:trajcluster:TrajCluster                           0.0112431      3.61996       286.083      0.0270591      28.642         100    
reco:pandora:StandardPandora                            1.27137       3.83744       62.7117       2.31357       7.65666        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000183226   0.00858485     0.101542     0.00252158     0.0163453       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000261603    0.0144602     0.140917     0.00409879     0.0223594       100    
reco:pandoracalo:Calorimetry                          0.000131568    0.0268217     0.957578     0.00143958     0.111256        100    
reco:pandorapid:Chi2ParticleID                        5.0836e-05    0.000325104   0.00479386    0.000197593   0.000541843      100    
reco:trkshowersplit:TrackShowerHits                   0.000131638    0.0344921      1.35303     0.00107637     0.158922        100    
reco:pmtrack:PMAlgTrackMaker                          0.00152507      4.31418       194.684      0.190609       20.7584        100    
reco:pmtrackcalo:Calorimetry                          6.9892e-05     0.0149178     0.556285     0.00135398     0.0637583       100    
reco:pmtrackpid:Chi2ParticleID                        3.8563e-05    0.000471184   0.00529588    0.000215232   0.000816052      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000276532     3.57717       109.694      0.294667       13.8048        100    
reco:pmtrajfitcalo:Calorimetry                        7.6845e-05     0.0322632     0.946882     0.00196382     0.137111        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5066e-05    0.000326079   0.00217183    0.000257892   0.000305087      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00150794      4.74991       203.741      0.222232       21.9663        100    
reco:pmtracktccalo:Calorimetry                        9.8546e-05    0.00843707     0.184388     0.00139064     0.0262442       100    
reco:pmtracktcpid:Chi2ParticleID                      3.4735e-05    0.000434546   0.00397995    0.000241164   0.000606111      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000254531     1.87503       40.6222      0.244045       5.30275        100    
reco:pmtrajfittccalo:Calorimetry                      9.7494e-05     0.0213619     0.760488     0.00167666     0.0891417       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.0807e-05    0.000536805   0.00578718    0.000257782   0.000918388      100    
reco:blurredcluster:BlurredClustering                 0.000340102     1.34974       11.879       0.659264       1.78822        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000279187     19.439        292.013       4.8795        44.829         100    
reco:emshower:EMShower                                0.000357154    0.0310148     0.240377      0.0056716     0.0527517       100    
reco:cvnmap:CVNMapper                                 2.5378e-05     0.0199376     0.0767545     0.0221322     0.0168327       100    
reco:cvneva:CVNEvaluator                              2.0568e-05      1.16073       2.93784       1.38152       0.90836        100    
reco:energyrecnumu:EnergyReco                         0.000322529    0.0367461     0.858402     0.00688052     0.0993767       100    
reco:energyrecnue:EnergyReco                          0.000164591   0.00289953     0.0534874    0.000905219   0.00682601       100    
reco:energyrecnc:EnergyReco                           0.00211604    0.00569401     0.0557371    0.00387114    0.00674311       100    
reco:energyrecnumurange:EnergyReco                    0.000126349    0.0330557     0.881387     0.00381666     0.100724        100    
reco:energyrecnumumcs:EnergyReco                      0.00136258     0.0451879      1.85967     0.00458176     0.190564        100    
reco:opdec:Deconvolution                               0.0421077     0.405798       2.45293      0.253134      0.468258        100    
reco:ophitspe:OpHitFinderDeco                          0.273479       2.00273       5.70749       2.03988      0.957168        100    
reco:opflash:OpFlashFinder                            0.000537815   0.00233561     0.0118396    0.00215749    0.00138269       100    
reco:rns:RandomNumberSaver                            2.4546e-05    7.54839e-05   0.00070996    5.59705e-05   7.45165e-05      100    
reco:anglereconue:NuAngularReco                       0.000239813   0.00211844     0.0467932    0.000654996   0.00575203       100    
reco:anglereconumu:NuAngularReco                      9.5861e-05     0.0016205     0.0403114    0.000291255   0.00491097       100    
reco:anglereconuepfps:NuAngularReco                   0.000155885   0.00195504     0.0387341    0.000538526   0.00478416       100    
reco:anglereconumupfps:NuAngularReco                  0.000142769   0.00175272     0.0434066    0.000417688   0.00507101       100    
[art]:TriggerResults:TriggerResultInserter            1.1452e-05    2.23802e-05   0.000111871   1.82695e-05   1.4021e-05       100    
end_path:out1:RootOutput                               4.188e-06    1.28517e-05    5.298e-05    1.0575e-05    8.51319e-06      100    
end_path:out1:RootOutput(write)                        0.0303993     0.294582       1.44693      0.301499      0.224577        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21756 MB
  Peak resident set size usage (VmHWM): 2333.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2_reco_2024-11-23T_045824Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_18_20231207T122716Z_gen_g4_detsim_hitreco__20240510T042404Z_reco2_reco_data_2024-11-23T_045824Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work