Jobsub ID 108742.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 05:53:58
From worker node	Hostname	tds286.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:37:03
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4601s)
	CPU time	1h (3870s = 84%)
Outputting started	2024-11-23 05:53:44
Output files
Finished	2024-11-23 05:53:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

T:17 P:2 W:574 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
1st wire C:0 T:17 P:2 W:715 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:53:14 CET run: 50642276 subRun: 1 event: 51500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1413, 891, 694
minwire 0: 1965
minwire 1: 789
minwire 2: 1736
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00541001, 0.000961449, 0.993629, 
Output 1: 0.417117, 0.392354, 0.123492, 0.0670373, 
Output 2: 0.794071, 0.205929, 

729
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2807!
23-Nov-2024 06:53:43 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2_reco_data_2024-11-23T_043706Z.root"
23-Nov-2024 06:53:43 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/8b/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.81545       38.927        473.235       14.2607       71.1251        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.046133      0.0884112     0.290467      0.0970565     0.0356639       100    
reco:linecluster:LineCluster                           0.411862      0.723954       7.13986      0.466739      0.913574        100    
reco:trajcluster:TrajCluster                           0.134367       1.33801         26         0.169346       4.01065        100    
reco:pandora:StandardPandora                           0.150208       3.45758       103.533       1.57463       10.5011        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140202   0.00843554     0.0828624    0.00220662     0.0159437       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187274    0.0138757     0.117637     0.00452293     0.0217642       100    
reco:pandoracalo:Calorimetry                          0.000111265    0.0437026      1.06568     0.00213032     0.146189        100    
reco:pandorapid:Chi2ParticleID                        4.4693e-05    0.000187503   0.00258771    0.000146238   0.000258827      100    
reco:trkshowersplit:TrackShowerHits                   9.2695e-05     0.0264224     0.487507     0.00133376     0.0772272       100    
reco:pmtrack:PMAlgTrackMaker                          0.00123503      2.55724       33.5836      0.356164       5.9774         100    
reco:pmtrackcalo:Calorimetry                          8.6408e-05     0.0182159      0.36213      0.0013259     0.0556294       100    
reco:pmtrackpid:Chi2ParticleID                        3.6699e-05    0.00036203    0.00288676    0.000168792   0.000549429      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000215839     3.0714        61.2074      0.486335       8.24657        100    
reco:pmtrajfitcalo:Calorimetry                        8.3197e-05     0.0440851      1.0612      0.00271191     0.144772        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5472e-05    0.000200855   0.000921593   0.000154345   0.000151139      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00110635      3.34823       47.7979      0.453205       8.20024        100    
reco:pmtracktccalo:Calorimetry                        8.6585e-05     0.0125328     0.265436     0.00139548     0.0372043       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5466e-05    0.000329338   0.00254422    0.000171117   0.00043154       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000204755     2.15536       35.1353      0.593429       4.90101        100    
reco:pmtrajfittccalo:Calorimetry                      8.3389e-05     0.0277328     0.662972     0.00216477     0.0869078       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4962e-05    0.000459607   0.00428155    0.000199981   0.000706141      100    
reco:blurredcluster:BlurredClustering                 0.00346486      1.31985       8.26025      0.748134       1.65542        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0469465      17.0658       162.916       5.93913       29.7159        100    
reco:emshower:EMShower                                0.000243159    0.0294599     0.239396      0.0029293     0.0474301       100    
reco:cvnmap:CVNMapper                                 2.5179e-05     0.0121579     0.0470011     0.0118397    0.00780261       100    
reco:cvneva:CVNEvaluator                              1.9455e-05     0.636761      0.981401      0.754477      0.302578        100    
reco:energyrecnumu:EnergyReco                         0.000284632    0.0862683      1.33057     0.00546338     0.226333        100    
reco:energyrecnue:EnergyReco                          0.000186693   0.00271267     0.0240522    0.00084117    0.00468576       100    
reco:energyrecnc:EnergyReco                            0.0022558     0.0047792     0.0267954    0.00289528    0.00474919       100    
reco:energyrecnumurange:EnergyReco                    0.000184526    0.0799179      1.32677     0.00373919     0.221078        100    
reco:energyrecnumumcs:EnergyReco                      0.00227513     0.0837287      1.3399      0.00561736      0.2297         100    
reco:opdec:Deconvolution                               0.183293      0.549513      0.855069      0.536423      0.120672        100    
reco:ophitspe:OpHitFinderDeco                          0.567299       1.82983       3.10746       1.76116      0.515187        100    
reco:opflash:OpFlashFinder                            0.000356094   0.00149988     0.0065046    0.00137874    0.000737297      100    
reco:rns:RandomNumberSaver                            1.8038e-05    2.92879e-05   0.000356008   2.52355e-05   3.3512e-05       100    
reco:anglereconue:NuAngularReco                       0.000145268   0.00191617     0.0203815    0.000452984   0.00384082       100    
reco:anglereconumu:NuAngularReco                      8.8856e-05    0.00175887     0.020083     0.000342627   0.00370601       100    
reco:anglereconuepfps:NuAngularReco                   0.000140213   0.00187692     0.0199623    0.000466869   0.00378759       100    
reco:anglereconumupfps:NuAngularReco                  0.000129785    0.0018353     0.0201112    0.000437859    0.0037232       100    
[art]:TriggerResults:TriggerResultInserter            1.1135e-05    1.39279e-05   7.9874e-05    1.23155e-05   7.03633e-06      100    
end_path:out1:RootOutput                               2.743e-06    5.21962e-06   2.5904e-05    3.5515e-06    3.17137e-06      100    
end_path:out1:RootOutput(write)                        0.0215794     0.278747       1.24197      0.251512      0.184174        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11779.1 MB
  Peak resident set size usage (VmHWM): 2469.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2_reco_2024-11-23T_043706Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_514_20231208T031938Z_gen_g4_detsim_hitreco__20240510T064555Z_reco2_reco_data_2024-11-23T_043706Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work