Jobsub ID 108727.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108727.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:57:40
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-23 05:12:40
From worker node	Hostname	heplnc164.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:01:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4266s)
	CPU time	1h (4560s = 106%)
Outputting started	2024-11-23 05:12:15
Output files
Finished	2024-11-23 05:12:40
List job events Wrapper job log
Jobscript log (last 10,000 characters)

3-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:03 GMT run: 74494002 subRun: 1 event: 27200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 41, 75, 40
minwire 0: 512
minwire 1: 1379
minwire 2: 490
Used alternate method to get min and max wires due to vertex determination failure: 525, 724
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 565, 764
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0371879, 0.000237161, 0.962575, 
Output 1: 0.92806, 0.0590981, 0.00941315, 0.00342836, 
Output 2: 0.942576, 0.0574242, 

389
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1135!
23-Nov-2024 05:12:13 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2_reco_data_2024-11-23T_040116Z.root"
23-Nov-2024 05:12:13 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/85/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.514694       39.7003       636.129       17.8119       76.6505        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00597717     0.156171      0.771268      0.0814602     0.183616        100    
reco:linecluster:LineCluster                           0.0481453     0.701657       26.5674      0.137161       2.67731        100    
reco:trajcluster:TrajCluster                           0.0123928     0.655831       41.0398      0.0342559      4.12321        100    
reco:pandora:StandardPandora                           0.0636446      3.04796       31.4688       2.49055       3.11314        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000202571   0.00712731     0.092982     0.00305522     0.0128656       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000308349    0.0155744     0.264679     0.00487169     0.0343139       100    
reco:pandoracalo:Calorimetry                          0.000168055    0.0151618     0.427922     0.00153671     0.0490035       100    
reco:pandorapid:Chi2ParticleID                        5.4943e-05    0.000342363   0.00417117    0.000281148   0.000421522      100    
reco:trkshowersplit:TrackShowerHits                   0.000170029    0.0164604     0.565993     0.00115205     0.0650349       100    
reco:pmtrack:PMAlgTrackMaker                          0.00155742      2.0687        68.3984      0.164506       7.49723        100    
reco:pmtrackcalo:Calorimetry                          8.9598e-05    0.00833957     0.223826     0.00168193     0.0261444       100    
reco:pmtrackpid:Chi2ParticleID                        3.5166e-05    0.000453316    0.0039988    0.000299728   0.000564531      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000226676     1.99701       53.3028      0.347693       6.58039        100    
reco:pmtrajfitcalo:Calorimetry                         6.914e-05     0.0203741     0.893922     0.00270541     0.0911308       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2401e-05    0.000358646   0.00154569    0.000314145   0.000263159      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000974611     2.57139       109.807      0.201018       11.3575        100    
reco:pmtracktccalo:Calorimetry                        9.4177e-05    0.00691108     0.130072     0.00182559     0.0163358       100    
reco:pmtracktcpid:Chi2ParticleID                      4.0496e-05    0.000459226   0.00325487    0.000355528   0.000499148      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000193353     1.40436       32.4659      0.291167       3.75026        100    
reco:pmtrajfittccalo:Calorimetry                      6.7547e-05     0.0118985     0.330936     0.00191982     0.0390084       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.9426e-05    0.000577036   0.00522902    0.000345533   0.000764843      100    
reco:blurredcluster:BlurredClustering                 0.000449033     1.34732       7.95882      0.803844       1.50714        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000376197     20.6655       255.66        6.85208       38.8768        100    
reco:emshower:EMShower                                0.000503677    0.027146      0.217833      0.006037      0.0432703       100    
reco:cvnmap:CVNMapper                                 6.0394e-05     0.0234639     0.0850707     0.0297985     0.0192447       100    
reco:cvneva:CVNEvaluator                              3.7961e-05      1.44948       3.59662       1.92791       1.08995        100    
reco:energyrecnumu:EnergyReco                         0.00058522     0.062887       2.16583     0.00848733     0.252956        100    
reco:energyrecnue:EnergyReco                          0.000257223    0.0024828     0.0307985    0.000982275   0.00420998       100    
reco:energyrecnc:EnergyReco                           0.00231709    0.00544784     0.0369565    0.00429503    0.00442909       100    
reco:energyrecnumurange:EnergyReco                    0.000268374    0.0502781      1.83039     0.00372778     0.206459        100    
reco:energyrecnumumcs:EnergyReco                      0.00194786     0.0472687      1.36456     0.00482329     0.166476        100    
reco:opdec:Deconvolution                               0.0201918     0.666134       3.95358      0.335302      0.719103        100    
reco:ophitspe:OpHitFinderDeco                          0.0880769      2.33279       4.88195       2.41028       1.05969        100    
reco:opflash:OpFlashFinder                            0.000524034   0.00295736     0.017612     0.00262649    0.00193213       100    
reco:rns:RandomNumberSaver                            2.7903e-05    0.00010077    0.00137202    8.55855e-05   0.000130814      100    
reco:anglereconue:NuAngularReco                       0.000240722   0.00179624     0.0220839    0.000834563   0.00302198       100    
reco:anglereconumu:NuAngularReco                      0.000115978    0.0011836     0.0218971    0.000442101   0.00270887       100    
reco:anglereconuepfps:NuAngularReco                   0.000198714   0.00149941     0.0222825    0.000732697    0.0028669       100    
reco:anglereconumupfps:NuAngularReco                  0.000120817   0.00122368     0.0225602    0.000496658   0.00266332       100    
[art]:TriggerResults:TriggerResultInserter            1.4717e-05    3.03471e-05   0.000171322    2.668e-05    1.74811e-05      100    
end_path:out1:RootOutput                               5.249e-06    1.58684e-05   6.7867e-05    1.51735e-05   8.32818e-06      100    
end_path:out1:RootOutput(write)                        0.0278323     0.299897      0.943039      0.338857      0.201178        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 39206.5 MB
  Peak resident set size usage (VmHWM): 2094.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2_reco_2024-11-23T_040116Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_271_20231201T170009Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2_reco_data_2024-11-23T_040116Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work