Jobsub ID 301815.47@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301815.47@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:31:15
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:20:22
From worker node	Hostname	heplnc114.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:33:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6370s)
	CPU time	1h (6011s = 94%)
Outputting started	2024-11-23 07:20:02
Output files
Finished	2024-11-23 07:20:22
List job events Wrapper job log
Jobscript log (last 10,000 characters)

PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:19:36 GMT run: 74494002 subRun: 1 event: 40199
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:19:36 GMT run: 74494002 subRun: 1 event: 40199
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:19:48 GMT run: 74494002 subRun: 1 event: 40199
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:19:48 GMT run: 74494002 subRun: 1 event: 40199
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 244, 276, 224
minwire 0: 946
minwire 1: 1317
minwire 2: 988
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0011443, 0.99879, 6.5345e-05, 
Output 1: 0.958107, 0.041671, 0.000216726, 5.46322e-06, 
Output 2: 0.999142, 0.00085802, 

545
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1758!
Begin processing the 100th record. run: 74494002 subRun: 1 event: 40200 at 23-Nov-2024 07:19:53 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
227
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 565!
23-Nov-2024 07:20:00 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2_reco_data_2024-11-23T_053356Z.root"
23-Nov-2024 07:20:00 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a1/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.222343       61.2283       1542.27       16.1081       168.979        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00837006     0.0459574     0.531944      0.0166965     0.0841495       100    
reco:linecluster:LineCluster                           0.0246793      1.11517       80.5586      0.101694       8.01566        100    
reco:trajcluster:TrajCluster                           0.0119885      3.49918       220.897      0.0368432      22.2211        100    
reco:pandora:StandardPandora                           0.028406       4.47615       94.0049       2.37742       10.0468        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000236198    0.017683      0.359537     0.00350451     0.0459585       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000314931    0.0221634     0.164897     0.00825055     0.0351298       100    
reco:pandoracalo:Calorimetry                          0.000190282    0.0530843      2.16221     0.00265336     0.226934        100    
reco:pandorapid:Chi2ParticleID                         6.897e-05    0.000330315   0.00315889    0.000251908   0.000372401      100    
reco:trkshowersplit:TrackShowerHits                   0.00010917     0.0639795      2.62966     0.00152013      0.28357        100    
reco:pmtrack:PMAlgTrackMaker                          0.00251295      6.65143       258.46        0.38487       27.2136        100    
reco:pmtrackcalo:Calorimetry                          0.000180677    0.0284372      1.04492     0.00187156     0.113089        100    
reco:pmtrackpid:Chi2ParticleID                        5.9599e-05    0.000604103   0.00702609    0.00026423    0.00097789       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000350667     4.98766       182.784      0.694061       19.118         100    
reco:pmtrajfitcalo:Calorimetry                        0.000138429    0.0553889      2.24234     0.00309425     0.236119        100    
reco:pmtrajfitpid:Chi2ParticleID                      5.4006e-05    0.000349849   0.00189937    0.000253576   0.000333839      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00220173      7.97356       326.359      0.382909       34.3462        100    
reco:pmtracktccalo:Calorimetry                        0.000180668    0.0159099     0.275996     0.00207545     0.040215        100    
reco:pmtracktcpid:Chi2ParticleID                      6.7682e-05    0.000531539   0.00415268    0.000282749   0.000702938      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00039604      3.18232       55.493       0.600412       7.2581         100    
reco:pmtrajfittccalo:Calorimetry                      0.000143599    0.0480772      1.74334     0.00200647     0.191354        100    
reco:pmtrajfittcpid:Chi2ParticleID                     5.486e-05    0.000756591   0.00733797    0.000282661   0.00120369       100    
reco:blurredcluster:BlurredClustering                 0.000121114     1.62336       10.8186      0.624413       2.19882        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000213983     22.8678       287.148       5.32739       43.5844        100    
reco:emshower:EMShower                                0.000316092    0.0583844     0.461901      0.0117813     0.0922857       100    
reco:cvnmap:CVNMapper                                 3.5116e-05     0.0197796     0.0699871     0.0222117     0.0153853       100    
reco:cvneva:CVNEvaluator                              2.8925e-05      1.12426       2.14813       1.49411      0.742265        100    
reco:energyrecnumu:EnergyReco                         0.000515034    0.0861079      2.66041     0.00813685     0.313362        100    
reco:energyrecnue:EnergyReco                          0.000274395   0.00524811     0.104577     0.00130761     0.0131253       100    
reco:energyrecnc:EnergyReco                           0.00257221    0.00788433     0.103027     0.00398535     0.012845        100    
reco:energyrecnumurange:EnergyReco                    0.000404944    0.080156       2.61785     0.00409214     0.296922        100    
reco:energyrecnumumcs:EnergyReco                       0.0012343     0.0820113      2.54832     0.00624762     0.291444        100    
reco:opdec:Deconvolution                               0.0187718     0.358744       2.46707      0.295131      0.296112        100    
reco:ophitspe:OpHitFinderDeco                          0.0345981      2.24565       4.52402       2.17965       1.0409         100    
reco:opflash:OpFlashFinder                            0.000187206   0.00242233    0.00827005     0.0022517    0.00130444       100    
reco:rns:RandomNumberSaver                            2.4864e-05    5.28037e-05   0.00059653    4.7711e-05    5.57604e-05      100    
reco:anglereconue:NuAngularReco                       0.000209594   0.00399166     0.0916506    0.000703259    0.0116383       100    
reco:anglereconumu:NuAngularReco                      0.000101834   0.00363982     0.084843     0.000474943    0.0110231       100    
reco:anglereconuepfps:NuAngularReco                   0.000154639   0.00387051     0.0850554    0.000633388    0.0110511       100    
reco:anglereconumupfps:NuAngularReco                  0.000140549   0.00373562     0.0803092    0.000571462    0.0106622       100    
[art]:TriggerResults:TriggerResultInserter             1.313e-05    2.3531e-05    0.000114354   2.1489e-05    1.08104e-05      100    
end_path:out1:RootOutput                               3.332e-06    7.73862e-06   3.6612e-05    8.0035e-06    3.57527e-06      100    
end_path:out1:RootOutput(write)                        0.0159714     0.409314       1.93909      0.401571      0.313727        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8412.42 MB
  Peak resident set size usage (VmHWM): 2315.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2_reco_2024-11-23T_053356Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_401_20231201T170143Z_gen_g4_detsim_hitreco__20240507T195309Z_reco2_reco_data_2024-11-23T_053356Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work