Jobsub ID 108715.28@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108715.28@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:31:31
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 04:16:23
From worker node	Hostname	heplnc157.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:33:24
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2.root
Jobscript	Exit code	0
	Real time	42m (2557s)
	CPU time	49m (2971s = 116%)
Outputting started	2024-11-23 04:16:02
Output files
Finished	2024-11-23 04:16:23
List job events Wrapper job log
Jobscript log (last 10,000 characters)

to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0198234, 0.96314, 0.0170364, 
Output 1: 0.240016, 0.519751, 0.211318, 0.0289156, 
Output 2: 0.12361, 0.87639, 

680
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2652!
Begin processing the 100th record. run: 50602933 subRun: 1 event: 33400 at 23-Nov-2024 04:15:56 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:15:58 GMT run: 50602933 subRun: 1 event: 33400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:15:58 GMT run: 50602933 subRun: 1 event: 33400
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:15:59 GMT run: 50602933 subRun: 1 event: 33400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:15:59 GMT run: 50602933 subRun: 1 event: 33400
Comparing two wires in the same plane: return failure
%MSG
230
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 525!
23-Nov-2024 04:16:01 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2_reco_data_2024-11-23T_033331Z.root"
23-Nov-2024 04:16:01 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/e1/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.198443       22.8271       268.478       13.5672       34.1422        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00535447     0.160615       1.45821      0.0885562     0.222737        100    
reco:linecluster:LineCluster                           0.0258882     0.614692       4.86638      0.221088      0.942138        100    
reco:trajcluster:TrajCluster                           0.0119606     0.201799       5.21851      0.0269947     0.610377        100    
reco:pandora:StandardPandora                           0.0232456      2.07169       26.3778       1.51947       2.91509        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000164068   0.00451285     0.0595765    0.00200667    0.00796398       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000223269    0.011394      0.365907     0.00399603     0.0374032       100    
reco:pandoracalo:Calorimetry                          0.00011184      0.01112      0.304836     0.00149802     0.0363235       100    
reco:pandorapid:Chi2ParticleID                        3.7921e-05    0.000211483   0.00229563    0.000189551   0.000244009      100    
reco:trkshowersplit:TrackShowerHits                   9.7954e-05    0.00990378     0.265165     0.00112074     0.0316493       100    
reco:pmtrack:PMAlgTrackMaker                          0.00135607      1.37955       26.2378      0.301593       3.43842        100    
reco:pmtrackcalo:Calorimetry                           9.05e-05     0.00565073     0.157145     0.00134511     0.0175912       100    
reco:pmtrackpid:Chi2ParticleID                        3.6338e-05    0.000307685   0.00297783    0.000214819   0.000385869      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000220594     1.27939       18.3859      0.259018       2.76304        100    
reco:pmtrajfitcalo:Calorimetry                         7.454e-05     0.0105531     0.229515     0.00190523     0.0302534       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9566e-05    0.000225157   0.00074567    0.00020796    0.000146075      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00103615      1.83404       40.7112      0.290215       5.06572        100    
reco:pmtracktccalo:Calorimetry                        9.7012e-05    0.00607768     0.150654     0.00128431     0.0188341       100    
reco:pmtracktcpid:Chi2ParticleID                      4.0355e-05    0.000324791   0.00283997    0.000211392   0.000373373      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000217097     1.17093       22.4874      0.387533       2.70277        100    
reco:pmtrajfittccalo:Calorimetry                      9.6872e-05    0.00869969     0.240161     0.00161332     0.0285569       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6068e-05    0.000389935   0.00348658    0.000242822   0.000497498      100    
reco:blurredcluster:BlurredClustering                 0.000172644     1.13248       7.23598      0.802682       1.2766         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000163407     8.30352       107.974        3.497        14.637         100    
reco:emshower:EMShower                                0.00024495     0.0291698     0.185472     0.00696305     0.0431092       100    
reco:cvnmap:CVNMapper                                 3.0938e-05     0.0153185     0.0427541     0.0183486     0.0112914       100    
reco:cvneva:CVNEvaluator                              2.2502e-05     0.983334       1.47873       1.28411      0.561183        100    
reco:energyrecnumu:EnergyReco                         0.00051704     0.0244112     0.790311     0.00493401     0.0844469       100    
reco:energyrecnue:EnergyReco                          0.00017563    0.00159539     0.0188794    0.000659102   0.00250161       100    
reco:energyrecnc:EnergyReco                           0.00202267    0.00441821     0.0239204    0.00371235    0.00275133       100    
reco:energyrecnumurange:EnergyReco                    0.000200246    0.0225636     0.773561     0.00344946     0.0832985       100    
reco:energyrecnumumcs:EnergyReco                      0.00135164     0.0237385     0.786222     0.00450738     0.082591        100    
reco:opdec:Deconvolution                               0.0159005      0.58973       2.38033      0.332994      0.566675        100    
reco:ophitspe:OpHitFinderDeco                          0.0371161      2.62009       5.92163       2.52883       1.17686        100    
reco:opflash:OpFlashFinder                            0.000172464   0.00175711    0.00574207    0.00173587    0.000785446      100    
reco:rns:RandomNumberSaver                            2.0057e-05    5.46251e-05   0.000396735   5.1482e-05    3.74711e-05      100    
reco:anglereconue:NuAngularReco                       0.000149189   0.00103897     0.0140061    0.000531347   0.00177672       100    
reco:anglereconumu:NuAngularReco                      6.9841e-05    0.000788271    0.013641     0.000293105   0.00172714       100    
reco:anglereconuepfps:NuAngularReco                   0.000142698   0.000973629    0.0137427    0.000480051   0.00174311       100    
reco:anglereconumupfps:NuAngularReco                  9.8575e-05    0.000861016    0.0138497    0.000365376   0.00173908       100    
[art]:TriggerResults:TriggerResultInserter             9.238e-06    2.02558e-05   0.000116479   1.88805e-05   1.17071e-05      100    
end_path:out1:RootOutput                               3.015e-06    6.96176e-06   2.3645e-05     7.033e-06    2.6268e-06       100    
end_path:out1:RootOutput(write)                        0.0161575     0.286474      0.947626      0.303882      0.161127        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39229.1 MB
  Peak resident set size usage (VmHWM): 2064.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2_reco_2024-11-23T_033331Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_333_20231205T093452Z_gen_g4_detsim_hitreco__20240510T035020Z_reco2_reco_data_2024-11-23T_033331Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work