Jobsub ID 108779.87@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.87@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:16:35
From worker node	Hostname	comp23-24
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:04:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4295s)
	CPU time	1h (3682s = 85%)
Outputting started	2024-11-23 06:16:20
Output files
Finished	2024-11-23 06:16:35
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ailure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
1st wire C:0 T:0 P:2 W:495 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:16:08 GMT run: 6207207 subRun: 1 event: 31200
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 176, 274, 207
minwire 0: 582
minwire 1: 2274
minwire 2: 288
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00447015, 3.01137e-05, 0.9955, 
Output 1: 0.137193, 0.783439, 0.0772621, 0.00210533, 
Output 2: 0.993577, 0.00642268, 

434
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1331!
23-Nov-2024 06:16:18 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2_reco_data_2024-11-23T_050448Z.root"
23-Nov-2024 06:16:18 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c5/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.922806       18.9553       312.214       9.12206        33.19         200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0133772     0.0969968     0.450891      0.0785812     0.0664257       200    
reco:linecluster:LineCluster                           0.0840188     0.406961       11.2571      0.278787      0.820561        200    
reco:trajcluster:TrajCluster                           0.0311308     0.571955       42.383       0.109786       3.21449        200    
reco:pandora:StandardPandora                           0.0662533      1.68607       14.9516       1.44689       1.21685        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000137632   0.00362205     0.055237     0.00127003    0.00726101       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000172621    0.009533      0.385368     0.00194736     0.0357803       200    
reco:pandoracalo:Calorimetry                          9.8409e-05    0.00715768     0.340398     0.000637365    0.0288172       200    
reco:pandorapid:Chi2ParticleID                        3.6386e-05    0.000124047   0.00108222    0.000103186   9.87791e-05      200    
reco:trkshowersplit:TrackShowerHits                   0.000109373   0.00892529     0.412162     0.000404652    0.0386256       200    
reco:pmtrack:PMAlgTrackMaker                          0.00119004     0.968394       41.9393      0.0798617      3.58817        200    
reco:pmtrackcalo:Calorimetry                          8.3826e-05    0.00465031     0.136755     0.000595409    0.0151876       200    
reco:pmtrackpid:Chi2ParticleID                        2.9566e-05    0.000194725   0.00247206    0.000109672   0.000293124      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000179524    0.871683       19.8705      0.137973       2.24258        200    
reco:pmtrajfitcalo:Calorimetry                        6.1655e-05    0.00782864     0.362974     0.00089196     0.0307031       200    
reco:pmtrajfitpid:Chi2ParticleID                      2.8473e-05    0.000142358   0.000670798   0.000119558   0.000106143      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00104667      1.23223       59.2874      0.113583       5.03902        200    
reco:pmtracktccalo:Calorimetry                        7.4219e-05    0.00360136     0.0856795    0.000623432   0.00993863       200    
reco:pmtracktcpid:Chi2ParticleID                      3.0039e-05    0.000179143    0.001568     0.000115159   0.000209254      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000168535     0.74885       16.1872       0.1773        1.84102        200    
reco:pmtrajfittccalo:Calorimetry                      6.1358e-05    0.00719602      0.27521     0.000695561    0.0274577       200    
reco:pmtrajfittcpid:Chi2ParticleID                     2.788e-05    0.000228623   0.00257959    0.000120284   0.000335404      200    
reco:blurredcluster:BlurredClustering                 7.7076e-05     0.935409       6.95013      0.435498       1.19898        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000127586     7.99433       107.726       2.42558       15.3256        200    
reco:emshower:EMShower                                0.000194041    0.0182463     0.209787     0.00288076     0.0323136       200    
reco:cvnmap:CVNMapper                                 2.0252e-05     0.0134437     0.0619674     0.0124194     0.011423        200    
reco:cvneva:CVNEvaluator                              1.5721e-05     0.572989       1.03723      0.800288      0.376342        200    
reco:energyrecnumu:EnergyReco                         0.000272037    0.0233253      1.03279      0.0052509     0.0878763       200    
reco:energyrecnue:EnergyReco                          0.000137962   0.00125972     0.0193423    0.000447382   0.00236424       200    
reco:energyrecnc:EnergyReco                           0.00180298    0.00292567     0.0223235    0.00216633    0.00245283       200    
reco:energyrecnumurange:EnergyReco                    0.000185034    0.0206609      1.03145     0.00197289     0.0875004       200    
reco:energyrecnumumcs:EnergyReco                      0.00101812     0.0217006      1.02466     0.00242434     0.0870722       200    
reco:opdec:Deconvolution                               0.0790231     0.377529       1.82363      0.344251      0.172267        200    
reco:ophitspe:OpHitFinderDeco                          0.183309       2.13441       3.92618       2.09763      0.838813        200    
reco:opflash:OpFlashFinder                            0.000159506   0.00122548    0.00293324    0.00121384    0.000477862      200    
reco:rns:RandomNumberSaver                            1.8386e-05    2.94828e-05   0.000204815   2.75115e-05   1.48042e-05      200    
reco:anglereconue:NuAngularReco                       0.000144673   0.000892489    0.0160176    0.000305622   0.00194687       200    
reco:anglereconumu:NuAngularReco                      6.7838e-05    0.000715297    0.0151892    0.000161754   0.00186043       200    
reco:anglereconuepfps:NuAngularReco                   0.000115655   0.000816427    0.0152953    0.000260733   0.00187558       200    
reco:anglereconumupfps:NuAngularReco                  9.8661e-05    0.000782652    0.0154243    0.000223488   0.00188568       200    
[art]:TriggerResults:TriggerResultInserter             8.364e-06    1.1431e-05    3.7836e-05    1.03985e-05   2.99438e-06      200    
end_path:out1:RootOutput                                2.1e-06     4.20393e-06   1.4803e-05    3.3185e-06    2.10456e-06      200    
end_path:out1:RootOutput(write)                       0.00721454     0.196492      0.702362      0.220845      0.121799        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9398.71 MB
  Peak resident set size usage (VmHWM): 2239.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2_reco_2024-11-23T_050448Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2_reco_data_2024-11-23T_050448Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work