Jobsub ID 108847.13@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108847.13@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:59:03
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce508
Last heartbeat	2024-11-23 06:36:30
From worker node	Hostname	b9p11p1596.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:00:36
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2.root
Jobscript	Exit code	0
	Real time	35m (2129s)
	CPU time	38m (2296s = 107%)
Outputting started	2024-11-23 06:36:06
Output files
Finished	2024-11-23 06:36:30
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:35:56 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:35:56 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:35:56 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:35:56 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:35:59 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:35:59 CET run: 6457551 subRun: 1 event: 47400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 202, 192, 121
minwire 0: 871
minwire 1: 1789
minwire 2: 801
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0072492, 0.992375, 0.000375737, 
Output 1: 0.955577, 0.0442927, 0.000117462, 1.2611e-05, 
Output 2: 0.999421, 0.000579549, 

473
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1536!
23-Nov-2024 07:36:04 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2_reco_data_2024-11-23T_060042Z.root"
23-Nov-2024 07:36:04 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/25/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.14719       19.2281       254.625       8.90572       31.9165        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0181285     0.0494443     0.685505      0.0434308     0.0656834       100    
reco:linecluster:LineCluster                           0.151229       0.2211         1.645       0.176791      0.195035        100    
reco:trajcluster:TrajCluster                           0.048185       0.33048       10.3489      0.0630963      1.21636        100    
reco:pandora:StandardPandora                           0.901112       1.78748       39.6489       1.2447        3.85916        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000153271    0.0045531     0.0932605    0.00127906     0.010625        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000201581    0.0167843     0.216212     0.00302746     0.0388172       100    
reco:pandoracalo:Calorimetry                          0.000117841    0.0119181     0.383198     0.00104912     0.0430634       100    
reco:pandorapid:Chi2ParticleID                         4.295e-05    0.000163324   0.00142499    0.000112256   0.000160937      100    
reco:trkshowersplit:TrackShowerHits                   0.000112001    0.0137726     0.506201     0.000746894    0.0570657       100    
reco:pmtrack:PMAlgTrackMaker                          0.00131282      1.41893       25.9722      0.179306       3.46209        100    
reco:pmtrackcalo:Calorimetry                          6.7771e-05    0.00673384     0.199862     0.000659123    0.0220074       100    
reco:pmtrackpid:Chi2ParticleID                         2.475e-05    0.000233045   0.00164793    0.000125101   0.000297698      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000201891     1.45357       16.6815       0.23873       3.09066        100    
reco:pmtrajfitcalo:Calorimetry                         5.582e-05    0.00983194     0.259654     0.00125432     0.0321573       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.375e-05    0.000160958   0.000590123   0.000136135   0.000106882      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00105801      1.62002       33.8746      0.166247       4.4828         100    
reco:pmtracktccalo:Calorimetry                        8.0301e-05    0.00477431     0.119096     0.000724589     0.01315        100    
reco:pmtracktcpid:Chi2ParticleID                       3.29e-05     0.000209227   0.00135706    0.000128425   0.000206521      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000194501     1.07488       20.4906      0.237393       2.47737        100    
reco:pmtrajfittccalo:Calorimetry                      7.6451e-05    0.00920386     0.321312     0.00105514     0.0349333       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.132e-05    0.000291639   0.00285024    0.000153871   0.000396019      100    
reco:blurredcluster:BlurredClustering                 0.00477546     0.909676       6.29775      0.508686       1.09324        100    
reco:emtrkmichelid:EmTrackMichelId                     0.180765       7.10189       90.7154       2.26593       12.6173        100    
reco:emshower:EMShower                                0.000264752     0.01772      0.306535     0.00349276     0.0376079       100    
reco:cvnmap:CVNMapper                                  2.548e-05    0.00904417     0.034145      0.0108206     0.007874        100    
reco:cvneva:CVNEvaluator                               1.61e-05      0.590153       1.21115      0.733361      0.410447        100    
reco:energyrecnumu:EnergyReco                         0.000299432    0.0285729      1.12933     0.00423055     0.121559        100    
reco:energyrecnue:EnergyReco                          0.000116721   0.00135952     0.0170735    0.000482118   0.00254998       100    
reco:energyrecnc:EnergyReco                           0.00170569    0.00380182     0.0218067    0.00322273    0.00282081       100    
reco:energyrecnumurange:EnergyReco                    0.000175241    0.026333       1.13418     0.00305403     0.120173        100    
reco:energyrecnumumcs:EnergyReco                      0.00168726     0.0261085      1.14502     0.00338192     0.117281        100    
reco:opdec:Deconvolution                               0.0820365     0.277184      0.449629      0.282043      0.0767703       100    
reco:ophitspe:OpHitFinderDeco                          0.315179       1.9741        3.60771       1.84867      0.826209        100    
reco:opflash:OpFlashFinder                            0.000277992   0.00143988    0.00558463    0.00139801    0.000675864      100    
reco:rns:RandomNumberSaver                             1.704e-05    3.89974e-05   0.000400132   3.5635e-05    3.72402e-05      100    
reco:anglereconue:NuAngularReco                       0.00016724    0.00104185     0.0132391    0.000456007   0.00178852       100    
reco:anglereconumu:NuAngularReco                       7.198e-05    0.000781678    0.0127745    0.000199641   0.00171252       100    
reco:anglereconuepfps:NuAngularReco                   0.000123581    0.0009341     0.012908     0.000375927    0.0017111       100    
reco:anglereconumupfps:NuAngularReco                  0.000102021   0.000859603    0.0128806    0.000309892   0.00174168       100    
[art]:TriggerResults:TriggerResultInserter             8.52e-06     1.3303e-05     4.286e-05    1.2835e-05    3.86474e-06      100    
end_path:out1:RootOutput                               3.25e-06     8.38936e-06    2.22e-05      8.83e-06     2.67319e-06      100    
end_path:out1:RootOutput(write)                        0.0201531     0.220565       0.8927       0.245086      0.142136        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21669 MB
  Peak resident set size usage (VmHWM): 2240.93 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2_reco_2024-11-23T_060042Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_473_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061427Z_reco2_reco_data_2024-11-23T_060042Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work