Jobsub ID 108735.88@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108735.88@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:18:15
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:28:57
From worker node	Hostname	b9p10p2531.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:25:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3800s)
	CPU time	1h (3961s = 104%)
Outputting started	2024-11-23 05:28:41
Output files
Finished	2024-11-23 05:28:57
List job events Wrapper job log
Jobscript log (last 10,000 characters)

480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
1st wire C:0 T:20 P:2 W:519 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:28:32 CET run: 6413332 subRun: 1 event: 7000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:28:35 CET run: 6413332 subRun: 1 event: 7000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 170, 144, 148
minwire 0: 2505
minwire 1: 538
minwire 2: 2496
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0882644, 0.910772, 0.000964027, 
Output 1: 0.932535, 0.0604552, 0.00563731, 0.00137214, 
Output 2: 0.990431, 0.00956909, 

386
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1170!
23-Nov-2024 06:28:39 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2_reco_data_2024-11-23T_042525Z.root"
23-Nov-2024 06:28:39 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/7a/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.35235       36.0536       1490.89       9.26536       153.066        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.016509       0.04974      0.288745      0.044697      0.0386528       100    
reco:linecluster:LineCluster                           0.146003      0.603617       25.6788      0.168796       2.92726        100    
reco:trajcluster:TrajCluster                           0.0473326      1.26598       66.1855      0.0611696      7.22105        100    
reco:pandora:StandardPandora                           0.763395       9.20939       756.288       1.14603       75.1559        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00010852     0.0062027     0.168308      0.0015994     0.0187049       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000156451    0.0117247     0.194829     0.00309519     0.0257999       100    
reco:pandoracalo:Calorimetry                           8.763e-05     0.033259       1.83451     0.00105954     0.193658        100    
reco:pandorapid:Chi2ParticleID                         3.445e-05    0.000162344   0.00101405    0.000121876   0.000144704      100    
reco:trkshowersplit:TrackShowerHits                    7.053e-05     0.0296709      1.41945     0.000739763    0.156756        100    
reco:pmtrack:PMAlgTrackMaker                          0.000973115     3.16087       140.35       0.241338        15.13         100    
reco:pmtrackcalo:Calorimetry                           6.213e-05     0.0124957     0.594547     0.000966749    0.0623535       100    
reco:pmtrackpid:Chi2ParticleID                         2.364e-05    0.000266708   0.00350757    0.000140866   0.000442578      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000137291     2.53911       73.9752      0.341088       10.2127        100    
reco:pmtrajfitcalo:Calorimetry                         5.488e-05     0.0329117      1.68471      0.0016188      0.18729        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.325e-05    0.000183836   0.00113494    0.000140431   0.000164712      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000759524     3.54045       149.776      0.318444       16.254         100    
reco:pmtracktccalo:Calorimetry                         5.699e-05    0.00698582     0.239471     0.000928635    0.0262361       100    
reco:pmtracktcpid:Chi2ParticleID                       2.268e-05    0.000242571   0.00242452    0.000142816   0.000342654      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000122411     1.83819       72.2809      0.293731       7.50795        100    
reco:pmtrajfittccalo:Calorimetry                       5.236e-05     0.0201386      1.05903     0.00115541     0.110478        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2121e-05    0.000366769   0.00856449    0.000153121   0.000902656      100    
reco:blurredcluster:BlurredClustering                  6.453e-05      1.05724       9.86855      0.498639       1.48201        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0760622      9.60605       185.191       2.55148       22.573         100    
reco:emshower:EMShower                                0.000218261    0.0240885     0.355917     0.00480223     0.0498131       100    
reco:cvnmap:CVNMapper                                 1.8011e-05     0.0102273     0.0462032     0.0104333    0.00864653       100    
reco:cvneva:CVNEvaluator                               1.464e-05     0.652028       5.40857      0.750106      0.604969        100    
reco:energyrecnumu:EnergyReco                         0.000275962    0.0457999      2.0586      0.00405986     0.213474        100    
reco:energyrecnue:EnergyReco                          0.000109271   0.00202712     0.0522483    0.000505787   0.00608171       100    
reco:energyrecnc:EnergyReco                           0.00149372    0.00414766     0.0543036    0.00284748    0.00623352       100    
reco:energyrecnumurange:EnergyReco                    0.000118041    0.039732       1.76513      0.0020978     0.184774        100    
reco:energyrecnumumcs:EnergyReco                      0.000940684    0.046393       2.2631      0.00373416     0.232023        100    
reco:opdec:Deconvolution                               0.110287      0.269414      0.443036      0.257336      0.0765538       100    
reco:ophitspe:OpHitFinderDeco                          0.464752       1.70821       3.70938       1.59019      0.742509        100    
reco:opflash:OpFlashFinder                            0.000444122   0.00135669    0.00332544    0.00130618    0.000578709      100    
reco:rns:RandomNumberSaver                             1.501e-05    3.82471e-05   0.000253761   3.2615e-05    2.49255e-05      100    
reco:anglereconue:NuAngularReco                       0.000128861   0.00158955     0.0419591    0.000429357   0.00468359       100    
reco:anglereconumu:NuAngularReco                       5.577e-05    0.00135185     0.0414107    0.000258701   0.00461934       100    
reco:anglereconuepfps:NuAngularReco                    9.558e-05    0.00147005     0.0416879    0.000372737   0.00456947       100    
reco:anglereconumupfps:NuAngularReco                  8.2221e-05    0.00141148     0.0410836    0.000299946   0.00460623       100    
[art]:TriggerResults:TriggerResultInserter             7.98e-06     1.22294e-05    3.51e-05     1.07655e-05   4.19952e-06      100    
end_path:out1:RootOutput                               2.19e-06     5.64732e-06    1.543e-05     5.23e-06     2.17667e-06      100    
end_path:out1:RootOutput(write)                        0.0289303      0.21723       1.11205      0.189002      0.168675        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22095.4 MB
  Peak resident set size usage (VmHWM): 2671.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2_reco_2024-11-23T_042525Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_69_20231203T041014Z_gen_g4_detsim_hitreco__20240508T064202Z_reco2_reco_data_2024-11-23T_042525Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work