Jobsub ID 301707.101@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301707.101@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:33:23
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 06:40:05
From worker node	Hostname	b9p08p5323.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:34:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7490s)
	CPU time	2h (7842s = 104%)
Outputting started	2024-11-23 06:39:45
Output files
Finished	2024-11-23 06:40:05
List job events Wrapper job log
Jobscript log (last 10,000 characters)

bRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:39:30 CET run: 6209058 subRun: 1 event: 47000
1st wire C:0 T:1 P:2 W:671 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:39:30 CET run: 6209058 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:39:30 CET run: 6209058 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:39:30 CET run: 6209058 subRun: 1 event: 47000
1st wire C:0 T:1 P:2 W:627 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:39:31 CET run: 6209058 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:39:38 CET run: 6209058 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 317, 343, 306
minwire 0: 581
minwire 1: 2749
minwire 2: 11
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000152926, 0.999671, 0.000175895, 
Output 1: 0.0113845, 0.298851, 0.659998, 0.0297665, 
Output 2: 0.904844, 0.0951564, 

460
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1641!
23-Nov-2024 07:39:42 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2_reco_data_2024-11-23T_043500Z.root"
23-Nov-2024 07:39:42 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/76/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.207498       36.7952       1165.77       8.63285       122.179        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00347489    0.00604982     0.0144057    0.00629545    0.00186805       200    
reco:linecluster:LineCluster                           0.017824      0.914201       80.6105      0.0434655      6.59677        200    
reco:trajcluster:TrajCluster                          0.00924171      1.97164       127.526      0.020913       11.5504        200    
reco:pandora:StandardPandora                           0.0221628      3.98398       235.97        1.17546       19.0655        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112711   0.00749507     0.151668     0.00136573     0.0195563       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000166292    0.0142286      0.24914     0.00297668     0.0326897       200    
reco:pandoracalo:Calorimetry                          8.5321e-05     0.0356199      1.89187     0.000925663    0.180531        200    
reco:pandorapid:Chi2ParticleID                         3.232e-05    0.000163675   0.00127663    0.000124461   0.00014908       200    
reco:trkshowersplit:TrackShowerHits                   8.7571e-05     0.0431677      1.43342     0.000687561    0.185883        200    
reco:pmtrack:PMAlgTrackMaker                          0.000985729     4.38008       219.958      0.184637       20.1493        200    
reco:pmtrackcalo:Calorimetry                           6.258e-05     0.0165298     0.803739     0.000830482    0.0768582       200    
reco:pmtrackpid:Chi2ParticleID                         2.32e-05     0.000318738   0.00373945    0.000128506   0.000549695      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000153821     3.29953       174.384      0.231858       14.5711        200    
reco:pmtrajfitcalo:Calorimetry                        5.4041e-05     0.034733       1.76611     0.00125203      0.16948        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.277e-05    0.000194141    0.0010218    0.000138166   0.000176362      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000797367     4.62327       185.605      0.224893       19.8202        200    
reco:pmtracktccalo:Calorimetry                        6.5391e-05    0.00961394     0.371042     0.000849857    0.0374764       200    
reco:pmtracktcpid:Chi2ParticleID                       2.629e-05    0.000265274    0.0038918    0.000135641   0.000415464      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000132591     2.04866       76.5157      0.246507       7.4084         200    
reco:pmtrajfittccalo:Calorimetry                      5.5011e-05     0.0240663      1.16362     0.00101905     0.112522        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3891e-05    0.000403016   0.00691682    0.000148437   0.000816452      200    
reco:blurredcluster:BlurredClustering                 0.000142491     1.05721       8.54778      0.439546        1.593         200    
reco:emtrkmichelid:EmTrackMichelId                    0.000145012     11.1836       199.403       2.43653       26.2948        200    
reco:emshower:EMShower                                0.000214602    0.0290869     0.267825     0.00469825     0.0529257       200    
reco:cvnmap:CVNMapper                                  2.382e-05     0.0103223     0.0403363    0.00873803    0.00901831       200    
reco:cvneva:CVNEvaluator                              1.4601e-05     0.720483       4.4543       0.882619      0.516202        200    
reco:energyrecnumu:EnergyReco                         0.000285362    0.0489446      2.23657     0.00489296      0.19852        200    
reco:energyrecnue:EnergyReco                          0.000109691   0.00270763     0.0810142    0.000571335    0.0077199       200    
reco:energyrecnc:EnergyReco                           0.00137277    0.00511135     0.0902887    0.00317201    0.00835383       200    
reco:energyrecnumurange:EnergyReco                    0.000141801    0.0452281      2.24689     0.00285782     0.193296        200    
reco:energyrecnumumcs:EnergyReco                      0.000707446    0.0459252       2.206      0.00408783     0.188138        200    
reco:opdec:Deconvolution                               0.0149904     0.176794      0.339262      0.177567      0.0688598       200    
reco:ophitspe:OpHitFinderDeco                          0.0753523      1.80325       4.03131       1.77994      0.855348        200    
reco:opflash:OpFlashFinder                            0.000241112   0.00138917    0.00397285    0.00135284    0.000626556      200    
reco:rns:RandomNumberSaver                             1.723e-05    3.64313e-05   0.000306883   3.4171e-05    2.16452e-05      200    
reco:anglereconue:NuAngularReco                       0.000137751   0.00204304     0.0623723    0.000401168    0.006202        200    
reco:anglereconumu:NuAngularReco                       6.013e-05    0.00182342     0.0545666    0.000228802   0.00582831       200    
reco:anglereconuepfps:NuAngularReco                   9.9241e-05    0.00193838     0.0518891    0.000374138    0.0056724       200    
reco:anglereconumupfps:NuAngularReco                   8.274e-05     0.0018761     0.0527294    0.000300523   0.00569996       200    
[art]:TriggerResults:TriggerResultInserter             7.94e-06     1.33827e-05   4.6031e-05     1.277e-05    4.28031e-06      200    
end_path:out1:RootOutput                               2.24e-06     6.08065e-06    2.194e-05    6.1705e-06    2.52302e-06      200    
end_path:out1:RootOutput(write)                        0.0158129     0.241285       1.30044      0.240098      0.191565        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21931.6 MB
  Peak resident set size usage (VmHWM): 2498.93 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2_reco_2024-11-23T_043500Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_234_20231122T105139Z_gen_g4_detsim_hitreco__20240507T181900Z_reco2_reco_data_2024-11-23T_043500Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work