Jobsub ID 301653.41@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301653.41@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:35:32
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:45:39
From worker node	Hostname	comp23-06
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:42:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3761s)
	CPU time	58m (3528s = 93%)
Outputting started	2024-11-23 03:45:25
Output files
Finished	2024-11-23 03:45:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ginModule  23-Nov-2024 03:45:11 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:45:11 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:45:11 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:45:11 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:45:11 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:45:19 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:45:19 GMT run: 6492237 subRun: 1 event: 8900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 219, 330, 255
minwire 0: 757
minwire 1: 2653
minwire 2: 61
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00376463, 0.995905, 0.000330216, 
Output 1: 0.386672, 0.570739, 0.0419481, 0.000640688, 
Output 2: 0.994834, 0.00516595, 

498
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1613!
23-Nov-2024 03:45:23 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2_reco_data_2024-11-23T_024246Z.root"
23-Nov-2024 03:45:23 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/0a/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.63345       35.5932       296.152       12.5556       57.5659        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0129429     0.101099      0.351807      0.0740487     0.0648623       100    
reco:linecluster:LineCluster                           0.138027      0.682599       13.0451      0.278208       1.84247        100    
reco:trajcluster:TrajCluster                           0.0356249      1.34159       24.1706      0.114932       4.14348        100    
reco:pandora:StandardPandora                            1.1462        2.23997       21.1378       1.35484       3.01208        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000145326   0.00726694     0.047273     0.00252827     0.010856        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00018356     0.0137806     0.113084     0.00398188     0.022444        100    
reco:pandoracalo:Calorimetry                          0.000105484    0.025806      0.417141     0.00140289     0.0721464       100    
reco:pandorapid:Chi2ParticleID                        3.9072e-05    0.000179661    0.0021746    0.000145696   0.000224543      100    
reco:trkshowersplit:TrackShowerHits                   0.000114256    0.0250492     0.572325     0.000962818    0.0785649       100    
reco:pmtrack:PMAlgTrackMaker                          0.00132361      2.65905       46.2229      0.303542       6.91299        100    
reco:pmtrackcalo:Calorimetry                          8.8401e-05     0.0102812     0.0929802    0.00129087     0.020286        100    
reco:pmtrackpid:Chi2ParticleID                        3.3675e-05    0.000305529   0.00138976    0.00014734    0.000364388      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000228297     2.16025       27.9941      0.473568       4.68515        100    
reco:pmtrajfitcalo:Calorimetry                        6.7808e-05     0.0264153     0.415116      0.0020305     0.0725262       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.9047e-05    0.000190481   0.000629871   0.00015245    0.000137146      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00113975      3.30688       45.6787      0.262964       7.78208        100    
reco:pmtracktccalo:Calorimetry                        7.8811e-05    0.00762841     0.0770229    0.00130953     0.0139703       100    
reco:pmtracktcpid:Chi2ParticleID                      2.9781e-05    0.000278855    0.0012902    0.000151017   0.00029886       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000209247     1.69107       19.4111      0.386622       3.35319        100    
reco:pmtrajfittccalo:Calorimetry                      6.6465e-05     0.0196381     0.272785     0.00163794     0.0501682       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8267e-05    0.000396013   0.00244384    0.000164173   0.00053197       100    
reco:blurredcluster:BlurredClustering                 0.00986592      1.29552       7.15828       0.64659       1.57273        100    
reco:emtrkmichelid:EmTrackMichelId                      0.25735       16.0847       112.187       4.38375       25.1916        100    
reco:emshower:EMShower                                0.000259252    0.0246758     0.266763     0.00604541     0.0409216       100    
reco:cvnmap:CVNMapper                                 2.3581e-05     0.0138483     0.0408797     0.0130028     0.0106433       100    
reco:cvneva:CVNEvaluator                              1.7282e-05     0.605604      0.925783      0.808356       0.3696         100    
reco:energyrecnumu:EnergyReco                         0.000294707    0.0601424      2.24619     0.00788444     0.240721        100    
reco:energyrecnue:EnergyReco                          0.000141731   0.00214311     0.0146322    0.000695978   0.00328606       100    
reco:energyrecnc:EnergyReco                           0.00198507    0.00393067     0.0165247    0.00243514    0.00332387       100    
reco:energyrecnumurange:EnergyReco                    0.000209471    0.0572149      2.2367       0.0024805     0.239855        100    
reco:energyrecnumumcs:EnergyReco                      0.00137577     0.0585966      2.26246     0.00390462     0.242076        100    
reco:opdec:Deconvolution                               0.0903652      0.40345      0.927434      0.413975       0.14699        100    
reco:ophitspe:OpHitFinderDeco                          0.474463       2.4232        4.17059       2.49335       0.89343        100    
reco:opflash:OpFlashFinder                            0.000404231   0.00143685     0.0050633     0.0014266    0.00062005       100    
reco:rns:RandomNumberSaver                            2.0629e-05    3.29016e-05   0.000312697   2.74025e-05   2.93757e-05      100    
reco:anglereconue:NuAngularReco                       0.000160674   0.00158721     0.0128042    0.000472822   0.00259847       100    
reco:anglereconumu:NuAngularReco                      7.4752e-05    0.00136105     0.0117343    0.00026016    0.00243228       100    
reco:anglereconuepfps:NuAngularReco                   0.00012892    0.00148375     0.0131954    0.000389265   0.00255386       100    
reco:anglereconumupfps:NuAngularReco                  0.000109461   0.00143905     0.0118862    0.000333297   0.00245325       100    
[art]:TriggerResults:TriggerResultInserter             8.992e-06    1.24701e-05   8.3764e-05    1.09455e-05   7.60253e-06      100    
end_path:out1:RootOutput                               2.778e-06    5.82956e-06   2.7964e-05     4.757e-06    3.52473e-06      100    
end_path:out1:RootOutput(write)                        0.0152177     0.231299      0.689496      0.224783      0.151137        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9469.37 MB
  Peak resident set size usage (VmHWM): 2314.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2_reco_2024-11-23T_024246Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_88_20231208T051221Z_gen_g4_detsim_hitreco__20240510T051858Z_reco2_reco_data_2024-11-23T_024246Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work