Jobsub ID 108737.133@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108737.133@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:21:28
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:11:20
From worker node	Hostname	b9p11p3251.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:28:48
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2.root
Jobscript	Exit code	0
	Real time	42m (2535s)
	CPU time	44m (2685s = 105%)
Outputting started	2024-11-23 05:11:03
Output files
Finished	2024-11-23 05:11:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

event: 26300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
1st wire C:0 T:19 P:2 W:891 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:10:56 CET run: 6420351 subRun: 1 event: 26300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 152, 94, 143
minwire 0: 2492
minwire 1: 1250
minwire 2: 2142
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00435827, 0.000170303, 0.995471, 
Output 1: 0.0194656, 0.969425, 0.0106422, 0.000466915, 
Output 2: 0.994075, 0.00592534, 

507
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1749!
23-Nov-2024 06:11:02 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2_reco_data_2024-11-23T_042853Z.root"
23-Nov-2024 06:11:02 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/78/f6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.44079       23.4238       401.75        8.72985       54.261         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0165084     0.0464287     0.0955354     0.0472474     0.0145887       100    
reco:linecluster:LineCluster                           0.150868      0.342618       6.66923      0.182015      0.855467        100    
reco:trajcluster:TrajCluster                           0.0481104     0.774336       25.874       0.0602047      3.38554        100    
reco:pandora:StandardPandora                           0.877331       2.02597       32.6547       1.27066       3.79993        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124651   0.00547872     0.117359     0.00142354     0.0160447       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000162812    0.0112161     0.196933     0.00292008     0.0266051       100    
reco:pandoracalo:Calorimetry                          9.8331e-05     0.0187619     0.834465     0.00096795     0.0912028       100    
reco:pandorapid:Chi2ParticleID                         3.356e-05    0.000161336   0.00124515    0.000127716   0.000143395      100    
reco:trkshowersplit:TrackShowerHits                   0.000104151    0.0199076     0.704422     0.000693293    0.0857107       100    
reco:pmtrack:PMAlgTrackMaker                           0.0009968      2.1437        50.4414       0.22792       7.56359        100    
reco:pmtrackcalo:Calorimetry                          6.4861e-05    0.00862172     0.276956     0.000869724    0.0348074       100    
reco:pmtrackpid:Chi2ParticleID                        2.5161e-05    0.000246256   0.00204801    0.000143756   0.000340625      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000164071     1.78291       49.4356      0.268333       5.89728        100    
reco:pmtrajfitcalo:Calorimetry                         5.362e-05     0.0195082     0.699227     0.00124918     0.082854        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3031e-05    0.00018524    0.00097138    0.000135691   0.000163139      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00116492      2.57272       76.5784      0.185716       9.75936        100    
reco:pmtracktccalo:Calorimetry                        7.8781e-05    0.00622018     0.207944     0.000855134    0.0246284       100    
reco:pmtracktcpid:Chi2ParticleID                       2.576e-05    0.000229501   0.00214139    0.000136716   0.000316067      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000222242     1.30497       30.9639      0.286054       3.81702        100    
reco:pmtrajfittccalo:Calorimetry                       9.058e-05     0.0133704      0.56942     0.00123385     0.0626277       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.506e-05    0.000311435    0.0045231    0.000161342   0.000544877      100    
reco:blurredcluster:BlurredClustering                  0.010262      0.829012       6.64309      0.473775       1.11314        100    
reco:emtrkmichelid:EmTrackMichelId                     0.195774       8.33429       120.398       2.19615       18.6826        100    
reco:emshower:EMShower                                0.000273283    0.0262109      0.31009     0.00458363     0.0506173       100    
reco:cvnmap:CVNMapper                                  2.137e-05    0.00925269     0.0322353     0.0104382    0.00755473       100    
reco:cvneva:CVNEvaluator                                1.6e-05      0.621113       1.13425      0.784529      0.419373        100    
reco:energyrecnumu:EnergyReco                         0.000298633    0.0197204     0.320273     0.00380196     0.0471209       100    
reco:energyrecnue:EnergyReco                          0.000122912   0.00192577     0.0409502    0.00050005    0.00560515       100    
reco:energyrecnc:EnergyReco                           0.00143679    0.00436438     0.0440387    0.00319657    0.00578011       100    
reco:energyrecnumurange:EnergyReco                    0.000128671    0.0181222     0.309888      0.0024907     0.0462818       100    
reco:energyrecnumumcs:EnergyReco                      0.00108502     0.0200802     0.311169     0.00342318     0.0470241       100    
reco:opdec:Deconvolution                                0.12479      0.287948      0.529592      0.288768      0.0861834       100    
reco:ophitspe:OpHitFinderDeco                          0.686178       1.92421       4.06372       1.87955      0.790792        100    
reco:opflash:OpFlashFinder                            0.000401254   0.00140842    0.00390457    0.00130157    0.000617571      100    
reco:rns:RandomNumberSaver                             1.702e-05    3.91079e-05   0.000315643   3.55705e-05   2.98033e-05      100    
reco:anglereconue:NuAngularReco                       0.000158942   0.00142891     0.0327038    0.000367754   0.00441669       100    
reco:anglereconumu:NuAngularReco                      6.5531e-05    0.00117717     0.0301257    0.000196657   0.00421125       100    
reco:anglereconuepfps:NuAngularReco                   0.000111951   0.00133183     0.0308225    0.000349998    0.0042049       100    
reco:anglereconumupfps:NuAngularReco                  0.000102901    0.0012346     0.0304323    0.000254778   0.00416227       100    
[art]:TriggerResults:TriggerResultInserter             8.93e-06     1.23727e-05   3.8951e-05    1.1855e-05    3.6816e-06       100    
end_path:out1:RootOutput                               2.73e-06     5.89757e-06    1.64e-05        6e-06      2.15908e-06      100    
end_path:out1:RootOutput(write)                        0.0306365     0.221181       1.02381      0.220396      0.159637        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21813.5 MB
  Peak resident set size usage (VmHWM): 2402.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2_reco_2024-11-23T_042853Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_262_20231204T033000Z_gen_g4_detsim_hitreco__20240509T194219Z_reco2_reco_data_2024-11-23T_042853Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work