Jobsub ID 108807.5@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108807.5@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:23:22
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 07:08:32
From worker node	Hostname	b9p08p5554.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:24:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6198s)
	CPU time	1h (6092s = 98%)
Outputting started	2024-11-23 07:08:11
Output files
Finished	2024-11-23 07:08:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

3 CET run: 65840069 subRun: 1 event: 68000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
1st wire C:0 T:5 P:2 W:483 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:07:53 CET run: 65840069 subRun: 1 event: 68000
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 819, 181, 476
minwire 0: 303
minwire 1: 2042
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00261186, 7.15169e-05, 0.997317, 
Output 1: 0.160318, 0.738344, 0.0973384, 0.00399954, 
Output 2: 0.981815, 0.018185, 

545
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1838!
23-Nov-2024 08:08:09 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2_reco_data_2024-11-23T_052458Z.root"
23-Nov-2024 08:08:09 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f1/94/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.539108       29.1851       590.69        9.11451       69.5695        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0174445     0.0415308     0.213056      0.0411124     0.0174498       200    
reco:linecluster:LineCluster                           0.0944398     0.583812        21.71       0.174999       2.40235        200    
reco:trajcluster:TrajCluster                           0.0473388     0.932477       42.6044      0.0575355      4.40361        200    
reco:pandora:StandardPandora                           0.0560585      2.47673       62.1462       1.27686       6.61205        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000126292   0.00576304     0.121994     0.00118992     0.0138224       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000164732    0.0176944     0.503581     0.00255838     0.0589061       200    
reco:pandoracalo:Calorimetry                          9.5301e-05     0.0236925     0.659715     0.000768031    0.0892387       200    
reco:pandorapid:Chi2ParticleID                        3.7511e-05    0.000177311   0.00190712    0.000133682   0.000185281      200    
reco:trkshowersplit:TrackShowerHits                   8.6271e-05     0.0331075      1.27012     0.000571787    0.142531        200    
reco:pmtrack:PMAlgTrackMaker                          0.000962763     3.01474       96.6694      0.134937       10.6459        200    
reco:pmtrackcalo:Calorimetry                          6.2801e-05    0.00974543     0.410078     0.000788375    0.0377919       200    
reco:pmtrackpid:Chi2ParticleID                         2.547e-05    0.000281686   0.00484942    0.000141792   0.000542984      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166752     2.63637       105.959       0.22767       9.43124        200    
reco:pmtrajfitcalo:Calorimetry                        5.9281e-05     0.0205158     0.586924     0.00109492     0.0732857       200    
reco:pmtrajfitpid:Chi2ParticleID                       2.53e-05     0.000180706   0.000918682   0.000139587   0.000144996      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000837501     3.3426        89.4888      0.198832       11.7763        200    
reco:pmtracktccalo:Calorimetry                        6.6171e-05    0.00632276     0.105084     0.000699339    0.0166573       200    
reco:pmtracktcpid:Chi2ParticleID                       2.635e-05    0.000254993   0.00295839    0.000132952   0.000374731      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000165812     1.74168       40.9393      0.197557       5.1196         200    
reco:pmtrajfittccalo:Calorimetry                      5.8161e-05     0.0163325     0.522171     0.000902332    0.0624635       200    
reco:pmtrajfittcpid:Chi2ParticleID                     2.398e-05    0.000367202   0.00467726    0.000148786   0.000673868      200    
reco:blurredcluster:BlurredClustering                 0.000115051    0.934751       8.18977      0.376895       1.36916        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000145972     9.88508       126.092       2.12719       20.3259        200    
reco:emshower:EMShower                                0.000211193    0.0260557     0.332016     0.00329632     0.0489703       200    
reco:cvnmap:CVNMapper                                  2.062e-05     0.0106051     0.0516832    0.00642371     0.0105568       200    
reco:cvneva:CVNEvaluator                               1.538e-05     0.704671       5.94514      0.926982      0.615422        200    
reco:energyrecnumu:EnergyReco                         0.000316124    0.0420445      2.04515     0.00382926     0.169018        200    
reco:energyrecnue:EnergyReco                          0.000126112   0.00191781     0.0326481    0.000654459   0.00346211       200    
reco:energyrecnc:EnergyReco                           0.00177247    0.00447361     0.0396647    0.00336306     0.0038991       200    
reco:energyrecnumurange:EnergyReco                    0.000168252    0.0395228      2.01099     0.00318324     0.164604        200    
reco:energyrecnumumcs:EnergyReco                       0.0007503     0.0410355      2.03093     0.00368763     0.165163        200    
reco:opdec:Deconvolution                               0.0500156     0.286379       1.08383      0.274532      0.128654        200    
reco:ophitspe:OpHitFinderDeco                          0.142566       2.04973       4.84519       1.97863      0.956907        200    
reco:opflash:OpFlashFinder                            0.000146002   0.00146464    0.00622631    0.00138533    0.000746736      200    
reco:rns:RandomNumberSaver                            1.8851e-05    4.2547e-05    0.000399165   4.10805e-05   2.8538e-05       200    
reco:anglereconue:NuAngularReco                       0.000171372   0.00134706     0.0294911    0.000445255   0.00299414       200    
reco:anglereconumu:NuAngularReco                      7.0271e-05     0.0010773     0.0287236    0.000220918   0.00280586       200    
reco:anglereconuepfps:NuAngularReco                   0.000110632   0.00124574     0.0290497    0.00037729     0.0028719       200    
reco:anglereconumupfps:NuAngularReco                  9.6111e-05    0.00115706     0.0288999    0.000305454   0.00281147       200    
[art]:TriggerResults:TriggerResultInserter             9.14e-06     1.60129e-05    5.535e-05    1.5375e-05    4.80719e-06      200    
end_path:out1:RootOutput                               3.21e-06      7.638e-06     2.432e-05     7.66e-06     3.38982e-06      200    
end_path:out1:RootOutput(write)                        0.0118463     0.245834       1.05674      0.247784      0.185883        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21804 MB
  Peak resident set size usage (VmHWM): 2370.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2_reco_2024-11-23T_052458Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_339_20231121T155759Z_gen_g4_detsim_hitreco__20240503T065745Z_reco2_reco_data_2024-11-23T_052458Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work