Jobsub ID 108742.73@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.73@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:27:27
From worker node	Hostname	b9p08p5272.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:37:04
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2.root
Jobscript	Exit code	0
	Real time	50m (3002s)
	CPU time	55m (3338s = 111%)
Outputting started	2024-11-23 05:27:07
Output files
Finished	2024-11-23 05:27:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

IDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:26 CET run: 50602933 subRun: 1 event: 50800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:26 CET run: 50602933 subRun: 1 event: 50800
1st wire C:0 T:22 P:2 W:1106 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:26 CET run: 50602933 subRun: 1 event: 50800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:27 CET run: 50602933 subRun: 1 event: 50800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:27 CET run: 50602933 subRun: 1 event: 50800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:27 CET run: 50602933 subRun: 1 event: 50800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:26:27 CET run: 50602933 subRun: 1 event: 50800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1462, 1468, 1475
minwire 0: 1553
minwire 1: 860
minwire 2: 1381
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00169854, 0.00660331, 0.991698, 
Output 1: 0.415281, 0.45725, 0.0972872, 0.0301822, 
Output 2: 0.853918, 0.146082, 

767
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2963!
23-Nov-2024 06:27:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2_reco_data_2024-11-23T_043709Z.root"
23-Nov-2024 06:27:06 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d9/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.90971       29.0927       384.74        9.71375       58.1575        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00468045    0.00761741     0.0116077    0.00729764    0.00220889       100    
reco:linecluster:LineCluster                           0.0406486     0.342225       16.7664      0.0516157      1.79654        100    
reco:trajcluster:TrajCluster                           0.0117466      0.92076       28.361       0.0253539      3.82072        100    
reco:pandora:StandardPandora                           0.790582       2.13378       37.3325       1.1934        4.24857        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000129351   0.00706223     0.096087      0.0016831     0.0157663       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000183311    0.0124498     0.154479     0.00341788     0.0225322       100    
reco:pandoracalo:Calorimetry                          0.000115661    0.0243852     0.683286     0.00131299     0.0851889       100    
reco:pandorapid:Chi2ParticleID                         3.722e-05    0.00017667     0.0021723    0.000121456   0.000230973      100    
reco:trkshowersplit:TrackShowerHits                   0.000109141    0.0300501     0.962253     0.000851796    0.114631        100    
reco:pmtrack:PMAlgTrackMaker                          0.00105988      2.69049       68.6721      0.272299       8.28661        100    
reco:pmtrackcalo:Calorimetry                          6.7521e-05    0.00984487     0.225755     0.000914497    0.0286398       100    
reco:pmtrackpid:Chi2ParticleID                         2.431e-05    0.000289827   0.00218174    0.000147556   0.000388779      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000177221     2.61163       53.7489      0.388842       7.78108        100    
reco:pmtrajfitcalo:Calorimetry                         5.844e-05     0.0223528     0.629224     0.00171928     0.0774239       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.401e-05    0.000188385   0.000937758   0.000150721   0.000150321      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00105938      2.9105        52.0068      0.269589       8.34459        100    
reco:pmtracktccalo:Calorimetry                         8.434e-05     0.0065414     0.113001     0.000997618    0.0155698       100    
reco:pmtracktcpid:Chi2ParticleID                       3.452e-05    0.000258887   0.00170635    0.000154842   0.000301835      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000222202     1.62055       23.5935      0.333461       3.63688        100    
reco:pmtrajfittccalo:Calorimetry                      8.6331e-05     0.0151244     0.265291     0.00122526     0.0431952       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.509e-05    0.000364571   0.00353333    0.000175656   0.000534927      100    
reco:blurredcluster:BlurredClustering                 0.00930201      1.25795       7.97959      0.664432       1.62871        100    
reco:emtrkmichelid:EmTrackMichelId                     0.174319       11.2338       99.4779       3.3548        20.2374        100    
reco:emshower:EMShower                                0.000270932    0.0310924     0.279119     0.00759825     0.0542777       100    
reco:cvnmap:CVNMapper                                  2.639e-05     0.0110876     0.0358817     0.0100193    0.00813013       100    
reco:cvneva:CVNEvaluator                               1.637e-05     0.757878       1.24174      0.831893      0.366372        100    
reco:energyrecnumu:EnergyReco                         0.000358282    0.0427127      0.92105      0.0046646     0.134774        100    
reco:energyrecnue:EnergyReco                          0.000140851   0.00217988     0.0254676    0.00051595    0.00446976       100    
reco:energyrecnc:EnergyReco                           0.00143853    0.00467358     0.0405259    0.00327584    0.00560343       100    
reco:energyrecnumurange:EnergyReco                    0.000171391    0.0433348      1.01644     0.00304661     0.147684        100    
reco:energyrecnumumcs:EnergyReco                      0.000898627    0.0427382     0.911286     0.00369749     0.137919        100    
reco:opdec:Deconvolution                               0.0486621     0.193876      0.370103      0.185997      0.0741543       100    
reco:ophitspe:OpHitFinderDeco                          0.381746       1.84759       4.29259       1.70516      0.903323        100    
reco:opflash:OpFlashFinder                            0.000396314   0.00146108    0.00553932    0.00140614    0.00072987       100    
reco:rns:RandomNumberSaver                             1.648e-05    3.83731e-05   0.000361623   3.55405e-05   3.45251e-05      100    
reco:anglereconue:NuAngularReco                       0.000136081   0.00170067     0.0260654    0.000437394   0.00386775       100    
reco:anglereconumu:NuAngularReco                      6.7311e-05    0.00153747     0.0256069    0.000270057   0.00383978       100    
reco:anglereconuepfps:NuAngularReco                   0.000111261   0.00166735     0.0254496    0.000377723   0.00379059       100    
reco:anglereconumupfps:NuAngularReco                  9.1821e-05    0.00161076     0.0252948    0.000307488   0.00384664       100    
[art]:TriggerResults:TriggerResultInserter             8.29e-06     1.24883e-05   5.6441e-05    1.1775e-05    5.19597e-06      100    
end_path:out1:RootOutput                               2.59e-06     6.21624e-06    2.596e-05     5.985e-06    3.64925e-06      100    
end_path:out1:RootOutput(write)                        0.0228304     0.247208      0.813637      0.229206      0.150402        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21833.7 MB
  Peak resident set size usage (VmHWM): 2403.95 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2_reco_2024-11-23T_043709Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_507_20231205T101415Z_gen_g4_detsim_hitreco__20240510T051315Z_reco2_reco_data_2024-11-23T_043709Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work