Jobsub ID 108744.107@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.107@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 05:44:39
From worker node	Hostname	td810.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:42:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3717s)
	CPU time	1h (3700s = 99%)
Outputting started	2024-11-23 05:44:20
Output files
Finished	2024-11-23 05:44:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

n the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:10 CET run: 66006611 subRun: 1 event: 54700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:10 CET run: 66006611 subRun: 1 event: 54700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:10 CET run: 66006611 subRun: 1 event: 54700
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:10 CET run: 66006611 subRun: 1 event: 54700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:10 CET run: 66006611 subRun: 1 event: 54700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:44:13 CET run: 66006611 subRun: 1 event: 54700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:44:13 CET run: 66006611 subRun: 1 event: 54700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 108, 91, 79
minwire 0: 1454
minwire 1: 1003
minwire 2: 1690
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.411409, 0.566609, 0.0219817, 
Output 1: 0.79326, 0.193427, 0.0122478, 0.00106534, 
Output 2: 0.981885, 0.0181154, 

465
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1407!
23-Nov-2024 06:44:18 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2_reco_data_2024-11-23T_044227Z.root"
23-Nov-2024 06:44:18 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/35/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0772916      36.1533       801.315       13.2419       89.8024        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00119378    0.00239495    0.00454781     0.0022013    0.000781523      100    
reco:linecluster:LineCluster                          0.00247476     0.573033       46.0383      0.0254999      4.59342        100    
reco:trajcluster:TrajCluster                          0.00126781      1.88048       119.404      0.0190292      12.2821        100    
reco:pandora:StandardPandora                           0.0111353      2.73501       47.6399       1.67242       5.06173        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000159381   0.00803133     0.137777     0.00203599     0.0181858       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000180772    0.0141036     0.178237     0.00419586     0.0263669       100    
reco:pandoracalo:Calorimetry                          0.000103411    0.0222316     0.869468     0.00162053     0.0930689       100    
reco:pandorapid:Chi2ParticleID                        4.0751e-05    0.000205181   0.00181196    0.000152017   0.000210502      100    
reco:trkshowersplit:TrackShowerHits                   6.8921e-05     0.0294424      1.09378     0.00121046      0.12555        100    
reco:pmtrack:PMAlgTrackMaker                          0.00131035      3.49595       135.37       0.357788       14.253         100    
reco:pmtrackcalo:Calorimetry                          7.1471e-05     0.0120022     0.447506     0.00160704     0.047598        100    
reco:pmtrackpid:Chi2ParticleID                         3.003e-05    0.000367523   0.00414723    0.000203187   0.00052968       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000194482     2.74645       69.4679      0.486314       8.69209        100    
reco:pmtrajfitcalo:Calorimetry                         6.387e-05     0.0223706      0.92367     0.00201258     0.0976519       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8271e-05    0.000232288   0.000805398   0.000198771   0.000148065      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114105      3.97633       139.214      0.425808       15.0589        100    
reco:pmtracktccalo:Calorimetry                         6.898e-05    0.00822562     0.161409     0.00147757     0.0218976       100    
reco:pmtracktcpid:Chi2ParticleID                        2.9e-05     0.000349821    0.0029349    0.000202777   0.000438109      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000219482     2.20861       48.377       0.627007       5.75353        100    
reco:pmtrajfittccalo:Calorimetry                      6.6161e-05     0.0191139     0.694017     0.00166475     0.0752841       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.777e-05    0.000467254   0.00551515    0.000214957   0.00073751       100    
reco:blurredcluster:BlurredClustering                 0.000116811     1.24965       8.08343      0.553236       1.63808        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000137121     13.4689       172.646       4.29375       25.4394        100    
reco:emshower:EMShower                                0.000207053    0.0454077     0.577415      0.0185119     0.0778501       100    
reco:cvnmap:CVNMapper                                  2.489e-05    0.00993059     0.0468196    0.00814216    0.00882497       100    
reco:cvneva:CVNEvaluator                                1.9e-05      0.793675       1.25923      0.964699      0.420262        100    
reco:energyrecnumu:EnergyReco                         0.000374523    0.0553699      2.15158     0.00559068     0.226764        100    
reco:energyrecnue:EnergyReco                          0.000159601   0.00267584     0.0549185    0.000687447   0.00676883       100    
reco:energyrecnc:EnergyReco                           0.00222842    0.00585325     0.0616525    0.00384879    0.00740231       100    
reco:energyrecnumurange:EnergyReco                    0.000201382    0.0525964      2.12194     0.00368733     0.223977        100    
reco:energyrecnumumcs:EnergyReco                      0.00123168     0.0553069      2.23765     0.00477903     0.234204        100    
reco:opdec:Deconvolution                              0.00600075     0.230572      0.486999      0.230087      0.0913999       100    
reco:ophitspe:OpHitFinderDeco                          0.003426       2.06621       4.98805        2.132       0.917348        100    
reco:opflash:OpFlashFinder                            0.000172452   0.00164915    0.00449784    0.00158811    0.000722492      100    
reco:rns:RandomNumberSaver                             1.758e-05    3.50812e-05   0.000246422   3.08905e-05   2.45964e-05      100    
reco:anglereconue:NuAngularReco                       0.000152762   0.00201654     0.0480004    0.000429569   0.00557089       100    
reco:anglereconumu:NuAngularReco                      7.1661e-05    0.00179555     0.0459513    0.000288138   0.00541442       100    
reco:anglereconuepfps:NuAngularReco                   0.000122541   0.00195042     0.0467361    0.00041266    0.00547134       100    
reco:anglereconumupfps:NuAngularReco                  0.000124392    0.0018979     0.046499     0.000352353   0.00547841       100    
[art]:TriggerResults:TriggerResultInserter             9.209e-06    1.38587e-05    4.085e-05    1.30555e-05   3.68027e-06      100    
end_path:out1:RootOutput                               2.93e-06     6.02935e-06    3.793e-05    4.7055e-06    4.11457e-06      100    
end_path:out1:RootOutput(write)                        0.0149341     0.350094       2.16631       0.32511      0.290669        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21284.7 MB
  Peak resident set size usage (VmHWM): 2140.53 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2_reco_2024-11-23T_044227Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2_reco_data_2024-11-23T_044227Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work