Jobsub ID 108593.74@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108593.74@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:25:00
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2024-11-23 00:57:07
From worker node	Hostname	cdr52.int.cedar.computecanada.ca
	cpuinfo	Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	84598 (23 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:36:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4850s)
	CPU time	1h (3945s = 81%)
Outputting started	2024-11-23 00:56:52
Output files
Finished	2024-11-23 00:57:07
List job events Wrapper job log
Jobscript log (last 10,000 characters)

e: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 16:56:42 PST run: 50572032 subRun: 1 event: 112800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 16:56:42 PST run: 50572032 subRun: 1 event: 112800
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:56:42 PST run: 50572032 subRun: 1 event: 112800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:56:42 PST run: 50572032 subRun: 1 event: 112800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:56:42 PST run: 50572032 subRun: 1 event: 112800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 16:56:45 PST run: 50572032 subRun: 1 event: 112800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 113, 69, 64
minwire 0: 216
minwire 1: 2290
minwire 2: 198
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000462182, 0.998238, 0.00129974, 
Output 1: 0.326224, 0.567771, 0.104053, 0.00195104, 
Output 2: 0.995568, 0.00443168, 

331
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 936!
22-Nov-2024 16:56:50 PST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2_reco_data_2024-11-22T_233614Z.root"
22-Nov-2024 16:56:50 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/09/97/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4.29182       41.0267       1403.58       16.0731       140.027        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0736163     0.161874      0.278399      0.160054      0.0478952       100    
reco:linecluster:LineCluster                           0.662937       1.42683       70.2328      0.708462       6.9157         100    
reco:trajcluster:TrajCluster                            0.21756       1.96267       147.12       0.233928       14.6055        100    
reco:pandora:StandardPandora                            1.75135       4.13952       191.371       1.90834       18.8391        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000172867   0.00903708     0.317898      0.0022333     0.0322683       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000238902    0.0180877     0.200506     0.00444363     0.0342958       100    
reco:pandoracalo:Calorimetry                          0.000134046    0.0331436      2.31827      0.001589      0.230593        100    
reco:pandorapid:Chi2ParticleID                        5.4494e-05    0.000210619   0.00212015    0.000168373    0.0002269       100    
reco:trkshowersplit:TrackShowerHits                   0.000155797    0.0382706      2.92675     0.00118061      0.29098        100    
reco:pmtrack:PMAlgTrackMaker                          0.00168945      3.76712       221.92       0.321656       22.151         100    
reco:pmtrackcalo:Calorimetry                          0.000110706    0.0212033      1.42431     0.00146302     0.141684        100    
reco:pmtrackpid:Chi2ParticleID                        4.6727e-05    0.000364821   0.00751542    0.000184087   0.000775806      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000283489     2.7445        139.264      0.586384       13.9162        100    
reco:pmtrajfitcalo:Calorimetry                        0.000105273    0.0334248      2.29171     0.00209976      0.22792        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.6072e-05    0.000221068   0.00113071    0.000185641   0.000156482      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00151198      4.02024       224.205      0.376539       22.398         100    
reco:pmtracktccalo:Calorimetry                        0.000114438    0.0116975     0.542593     0.00158064     0.0547066       100    
reco:pmtracktcpid:Chi2ParticleID                      4.6735e-05    0.000344094   0.00534572    0.000206411   0.000569787      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000270167     1.98666       70.597       0.549003       7.19661        100    
reco:pmtrajfittccalo:Calorimetry                      0.000105914    0.0245061      1.51062     0.00183037     0.150562        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.5841e-05    0.000429704   0.00765287    0.000223218   0.000816371      100    
reco:blurredcluster:BlurredClustering                  0.0225992      1.35468       12.5257      0.771851       1.73432        100    
reco:emtrkmichelid:EmTrackMichelId                     0.299392       14.8048       304.389       5.2546        33.1576        100    
reco:emshower:EMShower                                0.000309112    0.0342246     0.491161     0.00943371     0.061743        100    
reco:cvnmap:CVNMapper                                 3.3777e-05     0.0129877     0.0599705     0.0132226    0.00989439       100    
reco:cvneva:CVNEvaluator                              2.5524e-05      0.87686       1.39278       1.12771      0.481165        100    
reco:energyrecnumu:EnergyReco                         0.000378853    0.0300342     0.473249     0.00593226     0.063866        100    
reco:energyrecnue:EnergyReco                          0.000222922   0.00306627     0.0935424     0.0013187    0.00951649       100    
reco:energyrecnc:EnergyReco                           0.00298014    0.00562241     0.0976275    0.00352529    0.00967542       100    
reco:energyrecnumurange:EnergyReco                    0.000259368    0.0280658     0.469442      0.0035638     0.0633072       100    
reco:energyrecnumumcs:EnergyReco                      0.00297147     0.0299597     0.484233     0.00545037     0.0640269       100    
reco:opdec:Deconvolution                               0.322185      0.764032       1.15706      0.749599      0.184547        100    
reco:ophitspe:OpHitFinderDeco                          0.855726        2.356        3.87928       2.39182      0.668014        100    
reco:opflash:OpFlashFinder                            0.000337645   0.00169878    0.00465095    0.00167589    0.000723028      100    
reco:rns:RandomNumberSaver                            2.1673e-05    3.56467e-05   0.000313311   3.44485e-05   2.92491e-05      100    
reco:anglereconue:NuAngularReco                       0.00017491    0.00210651     0.0817526    0.000457188   0.00828812       100    
reco:anglereconumu:NuAngularReco                      0.000111359   0.00197455     0.0805644    0.000362237   0.00818094       100    
reco:anglereconuepfps:NuAngularReco                   0.00017087    0.00212431     0.0815607    0.00046971    0.00826823       100    
reco:anglereconumupfps:NuAngularReco                  0.000162939   0.00209222     0.0815447    0.000449551   0.00827594       100    
[art]:TriggerResults:TriggerResultInserter            1.3889e-05    1.60545e-05   5.1644e-05    1.5269e-05    3.89964e-06      100    
end_path:out1:RootOutput                               2.78e-06     5.28376e-06    1.584e-05    3.5715e-06    2.52055e-06      100    
end_path:out1:RootOutput(write)                        0.0230788      0.31395       1.69364      0.319711      0.207669        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8543.13 MB
  Peak resident set size usage (VmHWM): 2559.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2_reco_2024-11-22T_233614Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1127_20231203T073300Z_gen_g4_detsim_hitreco__20240508T072558Z_reco2_reco_data_2024-11-22T_233614Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work