Jobsub ID 108827.10@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108827.10@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:25:34
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 06:25:03
From worker node	Hostname	b9p11p9400.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:26:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2.root
Jobscript	Exit code	0
	Real time	57m (3462s)
	CPU time	56m (3418s = 98%)
Outputting started	2024-11-23 06:24:38
Output files
Finished	2024-11-23 06:25:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

trajfittc@BeginModule  23-Nov-2024 07:23:58 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 6483936 subRun: 1 event: 43500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1835, 1595, 1190
minwire 0: 94
minwire 1: 1805
minwire 2: 20
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0201141, 0.962625, 0.0172613, 
Output 1: 0.494764, 0.428974, 0.0656384, 0.0106237, 
Output 2: 0.0598476, 0.940152, 

615
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2032!
23-Nov-2024 07:24:36 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2_reco_data_2024-11-23T_052701Z.root"
23-Nov-2024 07:24:36 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/4a/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.897073       32.4247       336.183       9.05633       67.6847        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0174978     0.0417036     0.258696      0.0412953     0.0241214       100    
reco:linecluster:LineCluster                           0.095767       0.60151       14.0835      0.176442       1.98751        100    
reco:trajcluster:TrajCluster                           0.0480697     0.963502       19.7653      0.0556618      3.16761        100    
reco:pandora:StandardPandora                           0.0691346      2.47859       24.154        1.34171       4.0557         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000168252   0.00734823     0.152689     0.00124767     0.0191682       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000237233    0.0140618     0.300661     0.00277701     0.0341749       100    
reco:pandoracalo:Calorimetry                          0.000150032    0.0268914      0.55167     0.000726585    0.0856318       100    
reco:pandorapid:Chi2ParticleID                        5.0941e-05    0.000207054   0.00266191    0.000147952   0.000274697      100    
reco:trkshowersplit:TrackShowerHits                   0.000102681    0.0383564     0.706135     0.000542302    0.125848        100    
reco:pmtrack:PMAlgTrackMaker                          0.00154687      3.49326       56.4002      0.132772       10.2896        100    
reco:pmtrackcalo:Calorimetry                          9.0161e-05     0.0133228     0.401929     0.00081023     0.0486833       100    
reco:pmtrackpid:Chi2ParticleID                        3.5051e-05    0.000335617   0.00380674    0.000133327   0.000606173      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000270643     2.57759       46.5564       0.20177       7.4169         100    
reco:pmtrajfitcalo:Calorimetry                        8.8861e-05     0.0257777     0.450884     0.000974833    0.0785757       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6061e-05    0.000216855   0.00157752    0.000152912   0.00022723       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00134493      3.60303       56.2658      0.158962       10.4061        100    
reco:pmtracktccalo:Calorimetry                        8.5072e-05    0.00875457     0.201787      0.0007501     0.0283336       100    
reco:pmtracktcpid:Chi2ParticleID                       3.289e-05    0.000303689   0.00281022    0.000145597   0.000457276      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000256003     2.07548       28.2824       0.18312       5.24284        100    
reco:pmtrajfittccalo:Calorimetry                      9.9961e-05     0.0207594     0.339526     0.000854566    0.0634125       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.852e-05     0.0004508     0.004169     0.000147741   0.000815982      100    
reco:blurredcluster:BlurredClustering                 0.000124461     1.28987       9.59368      0.452897       1.83264        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000141081     11.4344       116.112       2.18981       23.9318        100    
reco:emshower:EMShower                                0.000212822    0.0216873     0.173998     0.00461223     0.0323072       100    
reco:cvnmap:CVNMapper                                  2.33e-05      0.0113513     0.0534124    0.00955363     0.0107847       100    
reco:cvneva:CVNEvaluator                              1.6751e-05     0.736426       1.30495      0.990528      0.481569        100    
reco:energyrecnumu:EnergyReco                         0.000274973     0.05789       2.99189     0.00376528     0.304734        100    
reco:energyrecnue:EnergyReco                          0.000129682    0.0024208     0.035716     0.000492186   0.00540245       100    
reco:energyrecnc:EnergyReco                            0.0016552     0.0050675     0.0378859    0.00352144     0.0055047       100    
reco:energyrecnumurange:EnergyReco                    0.000135032    0.0528384      2.77448     0.00240036     0.283386        100    
reco:energyrecnumumcs:EnergyReco                      0.000715769    0.054958       2.86755     0.00383163     0.291897        100    
reco:opdec:Deconvolution                               0.0830716     0.288916      0.573062      0.281787      0.0958896       100    
reco:ophitspe:OpHitFinderDeco                          0.443834       2.19446       4.43528       2.1169        0.99819        100    
reco:opflash:OpFlashFinder                            0.000351785   0.00157368     0.0063856    0.00145384    0.000836474      100    
reco:rns:RandomNumberSaver                            2.1341e-05    4.15901e-05   0.000334294    3.891e-05    3.15206e-05      100    
reco:anglereconue:NuAngularReco                       0.000162312   0.00178879     0.047697     0.000424546    0.0051696       100    
reco:anglereconumu:NuAngularReco                      7.4871e-05    0.00153505     0.0475139    0.000254228   0.00512106       100    
reco:anglereconuepfps:NuAngularReco                   0.000137452    0.0017634     0.0474646    0.000403381   0.00533033       100    
reco:anglereconumupfps:NuAngularReco                  0.000104382   0.00168475     0.0476352    0.000327044   0.00529885       100    
[art]:TriggerResults:TriggerResultInserter              9.5e-06     1.57868e-05    6.085e-05     1.524e-05    5.55724e-06      100    
end_path:out1:RootOutput                               3.49e-06     8.32533e-06    2.571e-05      8.3e-06     3.65273e-06      100    
end_path:out1:RootOutput(write)                        0.0262233     0.272296       1.14719      0.269891      0.209412        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21698.3 MB
  Peak resident set size usage (VmHWM): 2272.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2_reco_2024-11-23T_052701Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_434_20231207T085645Z_gen_g4_detsim_hitreco__20240510T040505Z_reco2_reco_data_2024-11-23T_052701Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work