Jobsub ID 108744.124@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.124@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:39:27
From worker node	Hostname	b9p10p3098.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:43:09
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3359s)
	CPU time	57m (3456s = 102%)
Outputting started	2024-11-23 05:39:09
Output files
Finished	2024-11-23 05:39:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ent: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:39:02 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:39:02 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:39:02 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:39:02 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:39:02 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:39:05 CET run: 66006611 subRun: 1 event: 111100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 158, 205, 178
minwire 0: 320
minwire 1: 2403
minwire 2: 38
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00156288, 0.998277, 0.000159893, 
Output 1: 0.214419, 0.756272, 0.0288047, 0.000503957, 
Output 2: 0.976012, 0.0239879, 

340
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 945!
23-Nov-2024 06:39:08 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2_reco_data_2024-11-23T_044314Z.root"
23-Nov-2024 06:39:08 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/9f/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.541012       31.5609       684.628       7.27039       87.9711        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0164588     0.0388711     0.0712439     0.0395773     0.0108486       100    
reco:linecluster:LineCluster                           0.0940546     0.728991       32.7078      0.171738       3.37923        100    
reco:trajcluster:TrajCluster                           0.0470203      2.00757       110.187      0.0581623      11.4735        100    
reco:pandora:StandardPandora                           0.0547822      2.64218       55.8333       1.00801       7.12754        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000117531   0.00753934     0.125583     0.00117725     0.0210765       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000165081    0.0160069     0.638081     0.00173343     0.0654496       100    
reco:pandoracalo:Calorimetry                           8.502e-05     0.0326303     0.957671     0.000948444    0.129695        100    
reco:pandorapid:Chi2ParticleID                        3.2441e-05    0.000139964   0.00119176    8.7625e-05    0.000161473      100    
reco:trkshowersplit:TrackShowerHits                   0.000102071    0.0361348      1.14734     0.000624633    0.144783        100    
reco:pmtrack:PMAlgTrackMaker                          0.000878954     3.49822       106.751      0.135977       12.9945        100    
reco:pmtrackcalo:Calorimetry                           5.964e-05     0.0158995     0.401028     0.000637773    0.0599323       100    
reco:pmtrackpid:Chi2ParticleID                         2.283e-05    0.000264617    0.0029606    0.000108471   0.00050399       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000163461     2.49597       56.0596      0.198338       7.96174        100    
reco:pmtrajfitcalo:Calorimetry                         5.724e-05     0.0344157      1.09216     0.00128944     0.139128        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.31e-05     0.000157739   0.00105345    0.000106796   0.000169053      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000765244     4.16252       125.474      0.141862       15.5682        100    
reco:pmtracktccalo:Calorimetry                         5.445e-05    0.00924267     0.171985     0.000638163    0.030522        100    
reco:pmtracktcpid:Chi2ParticleID                       2.148e-05    0.000223322   0.00244199    0.000114591   0.000366188      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000154511     1.83605       34.9021      0.191252       5.33681        100    
reco:pmtrajfittccalo:Calorimetry                       5.673e-05     0.0202969     0.516804     0.000945954    0.0755923       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.393e-05    0.000320419   0.00323344    0.000119005   0.000616134      100    
reco:blurredcluster:BlurredClustering                 0.00012741     0.906432       7.92251       0.51424       1.25619        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000135641     10.4868       142.682       2.07448       24.2184        100    
reco:emshower:EMShower                                0.000181711    0.026253      0.388539     0.00417185     0.0584839       100    
reco:cvnmap:CVNMapper                                  2.237e-05    0.00921497     0.0460606    0.00721459    0.00900221       100    
reco:cvneva:CVNEvaluator                               1.443e-05     0.559574       1.16873      0.709912      0.380234        100    
reco:energyrecnumu:EnergyReco                         0.000239691    0.0537817      1.61969     0.00306542     0.226003        100    
reco:energyrecnue:EnergyReco                          0.00011066    0.00237347     0.0387893    0.000418807   0.00645211       100    
reco:energyrecnc:EnergyReco                           0.00155935    0.00414746     0.0424971    0.00211026    0.00683462       100    
reco:energyrecnumurange:EnergyReco                    0.000117391    0.0521288      1.62896     0.00176609     0.224675        100    
reco:energyrecnumumcs:EnergyReco                      0.00101291     0.0537761      1.60125     0.00247308     0.223164        100    
reco:opdec:Deconvolution                               0.0676847     0.254108      0.666936      0.247885      0.0910591       100    
reco:ophitspe:OpHitFinderDeco                          0.140862       1.36381       3.88886       1.26289       0.62786        100    
reco:opflash:OpFlashFinder                            0.00015263    0.00112747    0.00419961    0.00109505    0.000565209      100    
reco:rns:RandomNumberSaver                             1.69e-05     3.35215e-05   0.000310061   3.1411e-05    2.93641e-05      100    
reco:anglereconue:NuAngularReco                       0.00012732    0.00181592     0.0343217    0.000314701   0.00528768       100    
reco:anglereconumu:NuAngularReco                       6.27e-05     0.00164261     0.0338561    0.00017199    0.00530799       100    
reco:anglereconuepfps:NuAngularReco                   0.000108651    0.0017633     0.0340464    0.000273896   0.00534127       100    
reco:anglereconumupfps:NuAngularReco                  8.5751e-05    0.00169778     0.0339763    0.000215691   0.00530185       100    
[art]:TriggerResults:TriggerResultInserter              8.1e-06     1.1062e-05    3.9271e-05     9.765e-06    4.17886e-06      100    
end_path:out1:RootOutput                               2.34e-06     4.76333e-06    1.848e-05     4.98e-06     2.11409e-06      100    
end_path:out1:RootOutput(write)                        0.0115326     0.195204      0.906426      0.178017      0.165752        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21886.8 MB
  Peak resident set size usage (VmHWM): 2448.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2_reco_2024-11-23T_044314Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1110_20231203T051723Z_gen_g4_detsim_hitreco__20240508T070906Z_reco2_reco_data_2024-11-23T_044314Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work