Jobsub ID 108730.51@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108730.51@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:04:08
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 05:13:04
From worker node	Hostname	heplnc064.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:06:16
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3988s)
	CPU time	1h (3928s = 98%)
Outputting started	2024-11-23 05:12:44
Output files
Finished	2024-11-23 05:13:04
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:12:26 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:26 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:12:26 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:26 GMT run: 74517882 subRun: 1 event: 4000
1st wire C:0 T:4 P:2 W:747 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:26 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:27 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:12:27 GMT run: 74517882 subRun: 1 event: 4000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 247, 203, 117
minwire 0: 851
minwire 1: 1202
minwire 2: 485
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0805368, 0.000359688, 0.919104, 
Output 1: 0.0156678, 0.217716, 0.733039, 0.0335767, 
Output 2: 0.90666, 0.0933401, 

606
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2143!
23-Nov-2024 05:12:43 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2_reco_data_2024-11-23T_040621Z.root"
23-Nov-2024 05:12:43 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/51/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.510222       37.7628       912.095        17.57        97.6539        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00515087     0.0440137      1.08089      0.0165425      0.12044        100    
reco:linecluster:LineCluster                           0.0478878     0.341095       17.028       0.0863201      1.69841        100    
reco:trajcluster:TrajCluster                           0.0117424      1.0901        65.4521      0.0427853      6.81165        100    
reco:pandora:StandardPandora                           0.0417287      3.72071       60.8838       3.03975       5.90816        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000184811    0.0073191     0.114869     0.00311133     0.0149688       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000254714    0.018645      0.242357     0.00578114     0.0351795       100    
reco:pandoracalo:Calorimetry                          0.000142207    0.0282152      1.22228     0.00259562     0.135078        100    
reco:pandorapid:Chi2ParticleID                        5.8198e-05    0.000274666   0.00347623    0.000219327   0.000344857      100    
reco:trkshowersplit:TrackShowerHits                   0.000136791    0.0228996     0.993711     0.00158702     0.113081        100    
reco:pmtrack:PMAlgTrackMaker                          0.00160777      2.5547        83.5314      0.465492       9.27464        100    
reco:pmtrackcalo:Calorimetry                          0.000108618    0.0127595     0.477238      0.0020602     0.0505775       100    
reco:pmtrackpid:Chi2ParticleID                        4.6525e-05    0.000411241    0.003576     0.000276743   0.000532223      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000292405     5.71185       343.634      0.581859       34.5575        100    
reco:pmtrajfitcalo:Calorimetry                        0.000105455     0.02943       1.24332     0.00334352     0.136622        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.4093e-05    0.000305058   0.00104956    0.000262569   0.000186147      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00143514      3.05952       104.218      0.507961       11.4529        100    
reco:pmtracktccalo:Calorimetry                        0.000121157   0.00884594     0.157838     0.00203623     0.023253        100    
reco:pmtracktcpid:Chi2ParticleID                      6.0732e-05    0.000384709   0.00266564    0.00028011    0.000412001      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000245502     1.68381       25.0734      0.666651       3.43369        100    
reco:pmtrajfittccalo:Calorimetry                      0.000104866    0.0151977     0.573655     0.00285342     0.0600463       100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.3169e-05    0.000489243   0.00450837    0.000327262   0.000597011      100    
reco:blurredcluster:BlurredClustering                 0.00013564      1.12629       9.35247      0.590461       1.4409         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000253623     13.8608       189.966       5.99911       26.0246        100    
reco:emshower:EMShower                                0.000319403    0.0436472     0.349328      0.0102698     0.0645602       100    
reco:cvnmap:CVNMapper                                 3.6237e-05     0.0244236     0.0668808     0.027835      0.0160746       100    
reco:cvneva:CVNEvaluator                              2.4442e-05      1.24098       2.28569       1.40607      0.743592        100    
reco:energyrecnumu:EnergyReco                         0.000517777    0.0780369      3.32748     0.00618316     0.351994        100    
reco:energyrecnue:EnergyReco                          0.000252913   0.00263162     0.0421003     0.0011384    0.00484843       100    
reco:energyrecnc:EnergyReco                           0.00250804    0.00621368     0.0604122    0.00421324    0.00672433       100    
reco:energyrecnumurange:EnergyReco                    0.000321813    0.0887172      4.02952     0.00361527     0.428108        100    
reco:energyrecnumumcs:EnergyReco                      0.00250168     0.0992199      4.77554     0.00549068     0.497714        100    
reco:opdec:Deconvolution                               0.0307082     0.287253        1.761       0.259892      0.197311        100    
reco:ophitspe:OpHitFinderDeco                           0.05092       2.19818       4.49484       2.20658      0.945717        100    
reco:opflash:OpFlashFinder                            0.00022433    0.00206022    0.00809232    0.00190856    0.00114516       100    
reco:rns:RandomNumberSaver                            2.5787e-05    5.42704e-05   0.000612927   4.63645e-05   5.80098e-05      100    
reco:anglereconue:NuAngularReco                       0.000215379   0.00175337     0.0329712    0.000646374   0.00390105       100    
reco:anglereconumu:NuAngularReco                      0.000120475    0.0015492     0.0310187    0.000477714   0.00383116       100    
reco:anglereconuepfps:NuAngularReco                   0.000188438   0.00175153     0.0298299    0.000647973   0.00369314       100    
reco:anglereconumupfps:NuAngularReco                  0.000173497   0.00165402     0.0303959    0.000619837   0.00368494       100    
[art]:TriggerResults:TriggerResultInserter            1.5152e-05    2.40663e-05   0.000116222   2.25255e-05   1.13039e-05      100    
end_path:out1:RootOutput                               3.363e-06    6.30947e-06   3.7862e-05     5.278e-06    3.80048e-06      100    
end_path:out1:RootOutput(write)                        0.0188265     0.343565       1.0834       0.364645      0.211847        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8309.5 MB
  Peak resident set size usage (VmHWM): 2211.14 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2_reco_2024-11-23T_040621Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_39_20231204T140253Z_gen_g4_detsim_hitreco__20240509T194055Z_reco2_reco_data_2024-11-23T_040621Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work