Jobsub ID 108752.17@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108752.17@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:46:20
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:27:59
From worker node	Hostname	b9p08p0363.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:47:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root
Jobscript	Exit code	0
	Real time	39m (2396s)
	CPU time	42m (2545s = 106%)
Outputting started	2024-11-23 05:27:42
Output files
Finished	2024-11-23 05:27:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.347794, 0.00722755, 0.644978, 
Output 1: 0.188673, 0.447089, 0.256448, 0.107789, 
Output 2: 0.0440435, 0.955956, 

790
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3314!
Begin processing the 100th record. run: 66050399 subRun: 1 event: 60100 at 23-Nov-2024 06:27:36 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:27:38 CET run: 66050399 subRun: 1 event: 60100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:27:38 CET run: 66050399 subRun: 1 event: 60100
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:27:38 CET run: 66050399 subRun: 1 event: 60100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:27:38 CET run: 66050399 subRun: 1 event: 60100
Comparing two wires in the same plane: return failure
%MSG
233
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 541!
23-Nov-2024 06:27:40 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_data_2024-11-23T_044751Z.root"
23-Nov-2024 06:27:40 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a6/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.550109       21.8796       224.055       9.08014       38.4136        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0174305     0.060035       1.4588       0.0443074     0.142514        100    
reco:linecluster:LineCluster                           0.104929      0.231699       1.21233      0.167546      0.208745        100    
reco:trajcluster:TrajCluster                           0.0472007     0.446555       8.4046       0.0568301      1.2839         100    
reco:pandora:StandardPandora                           0.0579993      1.76732       15.9762       1.24908       2.33297        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000120141   0.00553049     0.0833741    0.00146586     0.0130588       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000157752    0.0115935     0.320823     0.00295273     0.0348664       100    
reco:pandoracalo:Calorimetry                          9.8341e-05     0.0153801     0.297789     0.000947926    0.0487977       100    
reco:pandorapid:Chi2ParticleID                         3.422e-05    0.000161851   0.00130144    0.000123982   0.000158336      100    
reco:trkshowersplit:TrackShowerHits                   0.000109432    0.0199694     0.529935     0.000583896    0.0710808       100    
reco:pmtrack:PMAlgTrackMaker                          0.00096374      1.89091       28.5004       0.17747       4.99203        100    
reco:pmtrackcalo:Calorimetry                           5.816e-05    0.00772808      0.18115     0.000752653    0.0256186       100    
reco:pmtrackpid:Chi2ParticleID                        2.4291e-05    0.000269191   0.00247149    0.000145967   0.000380704      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000163132     1.54192       18.9161      0.254409       3.57201        100    
reco:pmtrajfitcalo:Calorimetry                        5.6551e-05     0.012714      0.232466     0.00138043     0.0356546       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.352e-05    0.000178446   0.000681427   0.000143646   0.000131403      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000748648     2.09699       29.2441      0.234517       5.38895        100    
reco:pmtracktccalo:Calorimetry                         5.625e-05    0.00513207     0.125476     0.000839704    0.0160569       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3191e-05    0.000224872   0.00145935    0.000149532   0.000243564      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000160602     1.36252       24.3688      0.263954       3.29453        100    
reco:pmtrajfittccalo:Calorimetry                       5.359e-05     0.0106814     0.248153     0.00111616     0.0342899       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.129e-05    0.000312548   0.00256171    0.000171002   0.000445593      100    
reco:blurredcluster:BlurredClustering                 0.000123181    0.926413       7.2873       0.505936       1.20407        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000162232     8.14593       94.3692       2.34444       15.9629        100    
reco:emshower:EMShower                                0.000203643    0.0226664     0.180931     0.00441604     0.0387585       100    
reco:cvnmap:CVNMapper                                  2.225e-05    0.00951096     0.0387091    0.00789412    0.00790021       100    
reco:cvneva:CVNEvaluator                               1.528e-05     0.752201       6.34916      0.799697       0.67026        100    
reco:energyrecnumu:EnergyReco                         0.000289353    0.0372891      1.64857     0.00360419     0.173446        100    
reco:energyrecnue:EnergyReco                          0.000116022    0.0017077     0.0226099    0.000449765   0.00354448       100    
reco:energyrecnc:EnergyReco                           0.00150274    0.00396047     0.0270765    0.00309597     0.003808        100    
reco:energyrecnumurange:EnergyReco                    0.000156722    0.0331297      1.43825     0.00224799     0.153599        100    
reco:energyrecnumumcs:EnergyReco                      0.00153141     0.0341793      1.3653      0.00377319      0.14681        100    
reco:opdec:Deconvolution                               0.0629578     0.268695      0.428844      0.274842      0.0789622       100    
reco:ophitspe:OpHitFinderDeco                           0.14244       1.91933       4.3161        1.9421        0.82563        100    
reco:opflash:OpFlashFinder                            9.5671e-05     0.0013248    0.00501567    0.00125665    0.000664202      100    
reco:rns:RandomNumberSaver                             1.652e-05    3.99368e-05   0.000376524   3.4115e-05    4.33874e-05      100    
reco:anglereconue:NuAngularReco                       0.000136382   0.00126367     0.0182879    0.000382359   0.00294732       100    
reco:anglereconumu:NuAngularReco                      6.1041e-05    0.00105693     0.0177825    0.000202512   0.00284798       100    
reco:anglereconuepfps:NuAngularReco                   0.000100881   0.00119246     0.0175786    0.000341213   0.00281022       100    
reco:anglereconumupfps:NuAngularReco                  8.7141e-05     0.0010528     0.0172977    0.000279098   0.00252325       100    
[art]:TriggerResults:TriggerResultInserter             8.19e-06     1.26738e-05   5.2281e-05    1.20455e-05   5.03819e-06      100    
end_path:out1:RootOutput                               2.81e-06     6.74694e-06    2.43e-05      6.775e-06    3.24836e-06      100    
end_path:out1:RootOutput(write)                       0.00817126     0.228851       0.91536      0.252847      0.139114        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21668 MB
  Peak resident set size usage (VmHWM): 2249.08 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_2024-11-23T_044751Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_data_2024-11-23T_044751Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work