Jobsub ID 301440.63@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301440.63@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 22:12:02
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce04
Last heartbeat	2024-11-22 23:27:49
From worker node	Hostname	dune001-2995000.0-lcg2706.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 22:38:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2.root
Outputting started	2024-11-22 23:20:51
Output files
Finished	2024-11-22 23:27:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

trajfittc@BeginModule  22-Nov-2024 23:20:36 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:36 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:36 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:36 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:20:36 UTC run: 6422149 subRun: 1 event: 59100
Cannot attach back to inner node of other track.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:20:46 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:20:46 UTC run: 6422149 subRun: 1 event: 59100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 329, 352, 303
minwire 0: 1567
minwire 1: 309
minwire 2: 1773
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.130634, 0.8592, 0.0101661, 
Output 1: 0.0590099, 0.612836, 0.309804, 0.0183505, 
Output 2: 0.195929, 0.804071, 

606
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2062!
22-Nov-2024 23:20:50 UTC  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2_reco_data_2024-11-22T_223811Z.root"
22-Nov-2024 23:20:50 UTC  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/df/a0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.85216       24.3989       406.59        9.58862       50.0306        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00126943     0.0995484     0.698242      0.0808082     0.0866989       100    
reco:linecluster:LineCluster                           0.0248149     0.710949       38.1708      0.258866       3.81943        100    
reco:trajcluster:TrajCluster                          0.00689601     0.893579       38.0895      0.108065       4.42557        100    
reco:pandora:StandardPandora                           0.970321       1.95853       36.281        1.2458        3.76708        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000141904   0.00508332     0.0350368    0.00210648    0.00666471       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000217298    0.0111207     0.119315     0.00364401     0.0200453       100    
reco:pandoracalo:Calorimetry                          0.000102254    0.0197402     0.654337     0.00130341     0.0743368       100    
reco:pandorapid:Chi2ParticleID                        3.7226e-05    0.000172584   0.00200181    0.000132735   0.000203424      100    
reco:trkshowersplit:TrackShowerHits                   0.000141934    0.0233265     0.822213     0.00100248     0.0958123       100    
reco:pmtrack:PMAlgTrackMaker                          0.00125298      2.43327       72.5009      0.258502       8.63598        100    
reco:pmtrackcalo:Calorimetry                           7.305e-05    0.00722909     0.165437     0.00118046     0.0211701       100    
reco:pmtrackpid:Chi2ParticleID                        2.6399e-05    0.000276231   0.00240444    0.000169185   0.000344318      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00022593      1.91969       29.7476      0.389749       4.69472        100    
reco:pmtrajfitcalo:Calorimetry                        6.0602e-05     0.0186509     0.657316      0.0020081     0.0720011       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5338e-05    0.000188749   0.00080248    0.000160832   0.000117996      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00129175       2.683        71.1614      0.299668       9.35699        100    
reco:pmtracktccalo:Calorimetry                        7.1187e-05    0.00463895     0.054928     0.00116926    0.00882678       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7421e-05    0.000244999   0.00134884     0.0001554    0.000232664      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000233371     1.28274       22.339       0.387834       2.80342        100    
reco:pmtrajfittccalo:Calorimetry                      6.0791e-05     0.0102948     0.276401     0.00150429     0.032863        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4527e-05    0.000326218   0.00238879    0.000176311   0.000381622      100    
reco:blurredcluster:BlurredClustering                 0.00857013     0.881715       4.10365      0.464999      0.972934        100    
reco:emtrkmichelid:EmTrackMichelId                     0.236259       8.50699       88.2337       3.02608       13.5227        100    
reco:emshower:EMShower                                0.000310919    0.0231598     0.202174     0.00706115     0.035576        100    
reco:cvnmap:CVNMapper                                 2.4386e-05    0.00621833     0.0353645    0.00441692    0.00574748       100    
reco:cvneva:CVNEvaluator                              1.5824e-05     0.616073      0.996784      0.618858      0.252127        100    
reco:energyrecnumu:EnergyReco                         0.000295115    0.0377416     0.651463     0.00790176     0.113442        100    
reco:energyrecnue:EnergyReco                          0.00010612    0.00129629     0.0144754    0.000458005   0.00213438       100    
reco:energyrecnc:EnergyReco                            0.0013093    0.00302999     0.0134801    0.00242162    0.00209668       100    
reco:energyrecnumurange:EnergyReco                    0.000152911    0.0328766     0.646042     0.00227916     0.113669        100    
reco:energyrecnumumcs:EnergyReco                      0.00113751     0.0330619      0.64172     0.00307507     0.111109        100    
reco:opdec:Deconvolution                               0.0408725     0.281255      0.563419      0.278088      0.104064        100    
reco:ophitspe:OpHitFinderDeco                          0.190136       1.65411       3.2642        1.64939      0.638052        100    
reco:opflash:OpFlashFinder                            0.000287424    0.0014241    0.00950405    0.00132525    0.00098564       100    
reco:rns:RandomNumberSaver                            1.7476e-05    2.84308e-05   0.000300353   2.3916e-05    2.8032e-05       100    
reco:anglereconue:NuAngularReco                       0.000162024   0.00105256     0.0091294    0.000435983   0.00149334       100    
reco:anglereconumu:NuAngularReco                      6.0221e-05    0.000839985    0.0092138    0.000230302   0.00143521       100    
reco:anglereconuepfps:NuAngularReco                   0.00010627    0.000962422   0.00912432    0.000337734    0.0014355       100    
reco:anglereconumupfps:NuAngularReco                  7.6124e-05    0.000902291   0.00904916    0.000297078   0.00143541       100    
[art]:TriggerResults:TriggerResultInserter             7.612e-06    1.12406e-05   4.3075e-05    1.0846e-05    3.93705e-06      100    
end_path:out1:RootOutput                               2.724e-06    4.3597e-06    2.3535e-05     3.516e-06    2.58169e-06      100    
end_path:out1:RootOutput(write)                        0.0220294     0.231726      0.786102      0.216051      0.118802        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 40350.3 MB
  Peak resident set size usage (VmHWM): 2540.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2_reco_2024-11-22T_223811Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_590_20231204T120430Z_gen_g4_detsim_hitreco__20240509T202147Z_reco2_reco_data_2024-11-22T_223811Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work