Jobsub ID 108749.29@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108749.29@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:42:02
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:44:29
From worker node	Hostname	b9p27p9785.cern.ch
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:47:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2.root
Jobscript	Exit code	0
	Real time	56m (3416s)
	CPU time	54m (3271s = 95%)
Outputting started	2024-11-23 05:44:08
Output files
Finished	2024-11-23 05:44:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

700
1st wire C:0 T:9 P:2 W:899 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:56 CET run: 50557588 subRun: 1 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:56 CET run: 50557588 subRun: 1 event: 48700
1st wire C:0 T:9 P:2 W:579 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:56 CET run: 50557588 subRun: 1 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:56 CET run: 50557588 subRun: 1 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:44:02 CET run: 50557588 subRun: 1 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:44:02 CET run: 50557588 subRun: 1 event: 48700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 223, 231, 236
minwire 0: 1218
minwire 1: 2036
minwire 2: 901
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 4.24608e-05, 0.999938, 1.98602e-05, 
Output 1: 0.00202131, 0.99782, 0.000158338, 5.83173e-07, 
Output 2: 0.999, 0.000999816, 

585
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2081!
23-Nov-2024 06:44:07 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_reco_data_2024-11-23T_044717Z.root"
23-Nov-2024 06:44:07 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/1b/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.540496       32.0406       1201.49       11.4422       120.861        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0191365     0.0449555     0.146365      0.0435788     0.0180843       100    
reco:linecluster:LineCluster                           0.094573      0.590432       37.0808      0.182269       3.67081        100    
reco:trajcluster:TrajCluster                           0.0475729      2.01279       173.601      0.0680077      17.259         100    
reco:pandora:StandardPandora                           0.0576503      3.04744       152.083       1.40921       15.0125        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000132992    0.0071017     0.288712     0.00206586     0.0291637       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000182643    0.0125817     0.144804      0.0038357     0.0228911       100    
reco:pandoracalo:Calorimetry                          9.6062e-05     0.0335842      2.47931     0.00193934     0.247021        100    
reco:pandorapid:Chi2ParticleID                        3.6151e-05    0.000184035    0.0024271    0.000129676   0.000270572      100    
reco:trkshowersplit:TrackShowerHits                    5.747e-05     0.0289474       1.939       0.0010475     0.194101        100    
reco:pmtrack:PMAlgTrackMaker                           0.0011913      2.95618       156.754      0.248665       15.7092        100    
reco:pmtrackcalo:Calorimetry                          5.8011e-05     0.0155884     0.981897     0.00134192     0.0981185       100    
reco:pmtrackpid:Chi2ParticleID                         2.455e-05    0.000302941   0.00479462    0.000150257   0.000560719      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000184732     3.28264       191.227       0.34006       19.0706        100    
reco:pmtrajfitcalo:Calorimetry                        5.4671e-05     0.0265393      1.72163     0.00209317     0.172051        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3251e-05    0.000191074   0.00103787    0.000153822   0.000155125      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00101676      3.21542       160.325      0.305122       16.1708        100    
reco:pmtracktccalo:Calorimetry                        5.9181e-05    0.00872776     0.345018     0.00140262     0.0359856       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5261e-05    0.000275708   0.00344177    0.000159392   0.000414583      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000186052     1.77403       64.6796      0.416743       6.68563        100    
reco:pmtrajfittccalo:Calorimetry                      6.9311e-05     0.0204026      1.11818     0.00157458     0.114521        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.38e-05     0.000374909   0.00588728    0.000182672   0.000726677      100    
reco:blurredcluster:BlurredClustering                 0.000120301    0.891751       7.64227      0.452061       1.29944        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000117372     11.1652       238.948       3.70785       26.8193        100    
reco:emshower:EMShower                                0.000172672    0.0307222     0.342266     0.00429196     0.0576583       100    
reco:cvnmap:CVNMapper                                   2.2e-05      0.0116706     0.0364838     0.0133669    0.00895806       100    
reco:cvneva:CVNEvaluator                               1.613e-05     0.639228       4.86868      0.784063      0.583848        100    
reco:energyrecnumu:EnergyReco                         0.000345925    0.0325889      1.47176     0.00503142     0.148439        100    
reco:energyrecnue:EnergyReco                          0.000125242   0.00236964     0.0670567    0.000830716   0.00706091       100    
reco:energyrecnc:EnergyReco                           0.00164821    0.00459037     0.0732552    0.00305427    0.00760288       100    
reco:energyrecnumurange:EnergyReco                    0.000157042    0.0305562      1.47803      0.0028026     0.149185        100    
reco:energyrecnumumcs:EnergyReco                      0.00129305     0.0307155      1.4401      0.00366703     0.145101        100    
reco:opdec:Deconvolution                               0.0524233     0.289479      0.591033      0.291745      0.102833        100    
reco:ophitspe:OpHitFinderDeco                          0.145497       1.58998       3.31205       1.55842      0.751845        100    
reco:opflash:OpFlashFinder                            0.000151852    0.0013653    0.00485264    0.00130254    0.000710029      100    
reco:rns:RandomNumberSaver                            2.2451e-05    4.37754e-05   0.000361575   4.04455e-05   3.32722e-05      100    
reco:anglereconue:NuAngularReco                       0.000157902   0.00170936     0.0594237    0.000424881   0.00620792       100    
reco:anglereconumu:NuAngularReco                      6.0801e-05    0.00148155     0.0580603    0.000220473   0.00606406       100    
reco:anglereconuepfps:NuAngularReco                   0.000120672   0.00165732     0.0594388    0.000385346   0.00619245       100    
reco:anglereconumupfps:NuAngularReco                  9.3591e-05    0.00155543     0.0582166    0.000287074    0.0060836       100    
[art]:TriggerResults:TriggerResultInserter             8.71e-06     1.25061e-05    5.093e-05    1.1765e-05    4.6316e-06       100    
end_path:out1:RootOutput                               3.41e-06     6.37369e-06    2.374e-05     6.59e-06     2.23838e-06      100    
end_path:out1:RootOutput(write)                        0.0121764     0.233032       1.41115      0.229722      0.193032        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12981.9 MB
  Peak resident set size usage (VmHWM): 2437.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_reco_2024-11-23T_044717Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_486_20231202T102703Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_reco_data_2024-11-23T_044717Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work