Jobsub ID 108712.74@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108712.74@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:22:55
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 04:48:20
From worker node	Hostname	heplnc064.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:28:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4783s)
	CPU time	1h (4535s = 94%)
Outputting started	2024-11-23 04:48:02
Output files
Finished	2024-11-23 04:48:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

c@BeginModule  23-Nov-2024 04:46:00 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:00 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:00 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:00 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:01 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:46:01 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:47:54 GMT run: 6407968 subRun: 1 event: 5800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 4680, 3666, 2848
minwire 0: 1062
minwire 1: 309
minwire 2: 813
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0401931, 0.776958, 0.182848, 
Output 1: 0.189028, 0.60776, 0.172462, 0.0307494, 
Output 2: 0.0634987, 0.936501, 

784
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2921!
23-Nov-2024 04:48:01 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2_reco_data_2024-11-23T_032824Z.root"
23-Nov-2024 04:48:01 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/af/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.765627       44.8071       535.74        15.5535       91.5306        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00537943     0.143271      0.867859      0.0748997     0.171727        100    
reco:linecluster:LineCluster                           0.0227914     0.630927       10.5385      0.123912       1.4427         100    
reco:trajcluster:TrajCluster                           0.0111856      1.63621       66.0594      0.0331428      7.40613        100    
reco:pandora:StandardPandora                           0.0361373      4.63592       81.9892       3.2901        8.44671        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00017965     0.0106852     0.141824     0.00234123     0.0237437       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000228713    0.0201992     0.330024     0.00479903     0.0414515       100    
reco:pandoracalo:Calorimetry                          0.000150359    0.0350466     0.888276     0.00212888     0.124877        100    
reco:pandorapid:Chi2ParticleID                        5.9671e-05    0.000288111   0.00294726    0.000216187   0.000318236      100    
reco:trkshowersplit:TrackShowerHits                   0.000128064    0.029127      0.637874     0.00112563     0.0978462       100    
reco:pmtrack:PMAlgTrackMaker                           0.0016214      3.77535       74.793       0.251522       11.414         100    
reco:pmtrackcalo:Calorimetry                          0.000106248    0.021933      0.698124     0.00155604     0.0817656       100    
reco:pmtrackpid:Chi2ParticleID                        4.7206e-05    0.000544316    0.006114     0.00023723    0.000994818      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000259341     4.29048       110.066      0.408234       13.9877        100    
reco:pmtrajfitcalo:Calorimetry                        0.00010071     0.0355446     0.912361     0.00262718     0.125371        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.3969e-05    0.000311254   0.00144514    0.000239873   0.000250885      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00139544      4.56879       113.642      0.325639       15.1198        100    
reco:pmtracktccalo:Calorimetry                        0.000119241    0.0112246     0.157564      0.0014758     0.0270802       100    
reco:pmtracktcpid:Chi2ParticleID                      4.3405e-05    0.000473931   0.00509487    0.000246903   0.000759372      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000267457     2.24005       28.0286      0.426455       4.94733        100    
reco:pmtrajfittccalo:Calorimetry                      0.000106705    0.0242611     0.392021     0.00224292     0.0716803       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.6548e-05    0.000645656   0.00533046    0.000271287   0.00104101       100    
reco:blurredcluster:BlurredClustering                 0.000136574     1.16996        9.821       0.449202       1.88551        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000232494     16.8155       170.26        4.19502       32.1721        100    
reco:emshower:EMShower                                0.000311593    0.0445745      1.01364     0.00626015     0.119195        100    
reco:cvnmap:CVNMapper                                 3.2481e-05     0.0229055     0.112909      0.0263034     0.0195779       100    
reco:cvneva:CVNEvaluator                              2.4969e-05      1.15238       2.38267       1.27544      0.803771        100    
reco:energyrecnumu:EnergyReco                         0.000795794    0.0585707      1.24878     0.00557562     0.166973        100    
reco:energyrecnue:EnergyReco                          0.000224131    0.003729      0.0653293    0.00101436     0.0085056       100    
reco:energyrecnc:EnergyReco                           0.00250266    0.00741594     0.0801239    0.00410551     0.0106052       100    
reco:energyrecnumurange:EnergyReco                    0.000292036    0.0582806      1.31591      0.0035668      0.17403        100    
reco:energyrecnumumcs:EnergyReco                      0.00173219     0.0580838      1.22355     0.00460343     0.163975        100    
reco:opdec:Deconvolution                               0.0609121     0.543764       3.24638      0.309925      0.562939        100    
reco:ophitspe:OpHitFinderDeco                          0.0533988      2.38817       4.74992       2.29596       1.03759        100    
reco:opflash:OpFlashFinder                            0.000202193   0.00211849    0.00804441    0.00199103    0.00103221       100    
reco:rns:RandomNumberSaver                            2.5803e-05    5.32573e-05   0.000633602   4.57815e-05   5.98706e-05      100    
reco:anglereconue:NuAngularReco                       0.000185813   0.00225239     0.0319851    0.000566628    0.0046904       100    
reco:anglereconumu:NuAngularReco                      0.000109435   0.00197525     0.0317255    0.000359473   0.00457433       100    
reco:anglereconuepfps:NuAngularReco                   0.000166575   0.00224314     0.0329049    0.000535751   0.00467484       100    
reco:anglereconumupfps:NuAngularReco                  0.000161197   0.00216856     0.0320335    0.000506296   0.00466686       100    
[art]:TriggerResults:TriggerResultInserter            1.3753e-05    2.52285e-05   0.000122619   2.2792e-05    1.39107e-05      100    
end_path:out1:RootOutput                               3.443e-06    6.21464e-06   3.6987e-05    5.2085e-06    4.22293e-06      100    
end_path:out1:RootOutput(write)                        0.0188724     0.358621      0.977611      0.386246      0.228707        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8322.55 MB
  Peak resident set size usage (VmHWM): 2229.49 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2_reco_2024-11-23T_032824Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_57_20231203T021122Z_gen_g4_detsim_hitreco__20240508T051015Z_reco2_reco_data_2024-11-23T_032824Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work