Jobsub ID 108746.63@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108746.63@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:39:54
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:59:54
From worker node	Hostname	b9p10p4272.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:42:24
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4631s)
	CPU time	1h (4678s = 101%)
Outputting started	2024-11-23 05:59:35
Output files
Finished	2024-11-23 05:59:54
List job events Wrapper job log
Jobscript log (last 10,000 characters)

@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:59:23 CET run: 74551443 subRun: 1 event: 74600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 4645, 4810, 4807
minwire 0: 312
minwire 1: 512
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0394596, 0.671501, 0.289039, 
Output 1: 0.0774644, 0.421613, 0.327899, 0.173023, 
Output 2: 0.0805687, 0.919431, 

833
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3143!
23-Nov-2024 06:59:34 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2_reco_data_2024-11-23T_044228Z.root"
23-Nov-2024 06:59:34 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/9b/6b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.798393       44.0054       1146.41       7.87122       161.659        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0172409     0.0444685     0.214783      0.041313      0.0276128       100    
reco:linecluster:LineCluster                           0.0940033      2.14446       112.529      0.172368       13.2473        100    
reco:trajcluster:TrajCluster                           0.0472718      4.91452       282.723      0.063192       31.5302        100    
reco:pandora:StandardPandora                           0.0571957      3.53645       81.6869       1.16312       12.1518        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000116192    0.0092928      0.23261     0.00106821     0.0340804       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000173493    0.0105748     0.127732      0.0016587     0.0240662       100    
reco:pandoracalo:Calorimetry                          9.1151e-05     0.048696       1.9509      0.000609797    0.247283        100    
reco:pandorapid:Chi2ParticleID                         3.465e-05    0.000149024   0.00117914    9.4916e-05    0.000184455      100    
reco:trkshowersplit:TrackShowerHits                   7.4251e-05     0.069456       3.04048     0.000650402    0.367214        100    
reco:pmtrack:PMAlgTrackMaker                          0.000946831     4.65835       144.836      0.125347       19.7225        100    
reco:pmtrackcalo:Calorimetry                          6.0161e-05     0.0247375     0.775337     0.000630302    0.107979        100    
reco:pmtrackpid:Chi2ParticleID                         2.354e-05    0.000317954   0.00453337    0.000120676   0.000693323      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000156172     4.17875       238.944      0.206243       24.4545        100    
reco:pmtrajfitcalo:Calorimetry                         5.582e-05     0.0425473      1.79738     0.00100129     0.217144        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3621e-05    0.000188086   0.00232113    0.000117317   0.000281897      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00077843      5.87037       187.552      0.132287       25.6781        100    
reco:pmtracktccalo:Calorimetry                        5.7441e-05     0.0135448     0.312736     0.000733799    0.0509106       100    
reco:pmtracktcpid:Chi2ParticleID                       2.341e-05    0.000295932   0.00379929    0.000121006   0.000626726      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000145352     2.5135        57.1225      0.258564       9.17982        100    
reco:pmtrajfittccalo:Calorimetry                       5.283e-05     0.0283319     0.813633     0.00106934     0.123306        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.222e-05    0.00036441     0.0058119    0.000145247   0.000790339      100    
reco:blurredcluster:BlurredClustering                 0.000111562     1.01963       10.1658      0.403027       1.74704        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000119971     11.8292       190.658       2.09656       32.9731        100    
reco:emshower:EMShower                                0.000202683    0.0349182     0.697563     0.00721077     0.0879431       100    
reco:cvnmap:CVNMapper                                  2.084e-05    0.00876055     0.055684     0.00825828    0.00901953       100    
reco:cvneva:CVNEvaluator                               1.54e-05      0.578928       1.18672      0.760505      0.410681        100    
reco:energyrecnumu:EnergyReco                         0.000301763    0.0713185      2.06324     0.00347169     0.285592        100    
reco:energyrecnue:EnergyReco                          0.000124101   0.00307594     0.0657929    0.000423965    0.0102818       100    
reco:energyrecnc:EnergyReco                           0.00136972     0.0053376     0.0710558    0.00301494     0.0111534       100    
reco:energyrecnumurange:EnergyReco                    0.000167252    0.0726763      2.14781     0.00244487     0.298947        100    
reco:energyrecnumumcs:EnergyReco                      0.00145742     0.0772426      2.15285     0.00317962     0.315087        100    
reco:opdec:Deconvolution                               0.0564372     0.271722      0.476758      0.277818      0.0869701       100    
reco:ophitspe:OpHitFinderDeco                          0.184589       1.69167       3.72256       1.63574      0.749139        100    
reco:opflash:OpFlashFinder                            0.000154132   0.00127696    0.00504288    0.00114775     0.0007177       100    
reco:rns:RandomNumberSaver                             1.714e-05    3.58595e-05   0.000356534   3.15605e-05   3.41755e-05      100    
reco:anglereconue:NuAngularReco                       0.000152672   0.00247877     0.0748706    0.000347629   0.00931161       100    
reco:anglereconumu:NuAngularReco                       6.512e-05    0.00231184     0.0755949    0.000166362   0.00948663       100    
reco:anglereconuepfps:NuAngularReco                   0.000103601   0.00237671     0.0745841    0.000287898   0.00914299       100    
reco:anglereconumupfps:NuAngularReco                  9.0971e-05    0.00236644     0.0745905    0.000219477   0.00935425       100    
[art]:TriggerResults:TriggerResultInserter              8.3e-06     1.3023e-05    4.6131e-05    1.15355e-05   6.86203e-06      100    
end_path:out1:RootOutput                                2.8e-06     6.29673e-06    2.977e-05     6.035e-06    3.36948e-06      100    
end_path:out1:RootOutput(write)                        0.011321      0.218779       1.24766       0.19676      0.211609        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21994.4 MB
  Peak resident set size usage (VmHWM): 2562.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2_reco_2024-11-23T_044228Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_745_20231207T113335Z_gen_g4_detsim_hitreco__20240510T054536Z_reco2_reco_data_2024-11-23T_044228Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work