Jobsub ID 108741.14@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108741.14@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:27:52
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 05:56:06
From worker node	Hostname	heplnc149.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:33:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4934s)
	CPU time	1h (4830s = 97%)
Outputting started	2024-11-23 05:55:44
Output files
Finished	2024-11-23 05:56:06
List job events Wrapper job log
Jobscript log (last 10,000 characters)

itialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:55:38 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:55:39 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:55:39 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:55:39 GMT run: 50577091 subRun: 1 event: 67500
Comparing two wires in the same plane: return failure
%MSG
358
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 936!
23-Nov-2024 05:55:42 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2_reco_data_2024-11-23T_043334Z.root"
23-Nov-2024 05:55:42 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/83/eb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.97316       47.5037       923.547       12.6449       116.762        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00513195     0.0500485     0.904656      0.0167153     0.119178        100    
reco:linecluster:LineCluster                           0.0413634     0.905266       48.1271      0.0881466      4.8571         100    
reco:trajcluster:TrajCluster                           0.0116871      2.05372       93.0185      0.0266736      10.057         100    
reco:pandora:StandardPandora                            1.05464       4.03018       133.714       1.67132       13.5544        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000144283    0.0101621     0.142502     0.00233425     0.0231024       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000210637    0.0226321     0.426231     0.00421833     0.0526016       100    
reco:pandoracalo:Calorimetry                          0.000117812    0.0311268      0.86582     0.00161312     0.108082        100    
reco:pandorapid:Chi2ParticleID                        3.9274e-05    0.000230096   0.00165472    0.000173302   0.000228397      100    
reco:trkshowersplit:TrackShowerHits                   0.000164551    0.0500779      2.01679     0.000944648    0.221201        100    
reco:pmtrack:PMAlgTrackMaker                          0.00129248      5.0112        148.534       0.27788       17.4908        100    
reco:pmtrackcalo:Calorimetry                          7.2737e-05     0.0198463     0.351897     0.00124886     0.0613598       100    
reco:pmtrackpid:Chi2ParticleID                        2.7031e-05    0.000436636   0.00372001    0.000189823   0.000711612      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000269608      3.483        94.6963       0.4334        11.5923        100    
reco:pmtrajfitcalo:Calorimetry                         7.426e-05     0.0328328     0.943327     0.00195649     0.115055        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2201e-05    0.000261085   0.00122686    0.000195659   0.000223712      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00107481      5.26769       110.853      0.311648       16.323         100    
reco:pmtracktccalo:Calorimetry                         6.35e-05      0.0144943     0.227096     0.00127424     0.0396318       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5849e-05    0.000411294   0.00342549    0.000181838   0.000617902      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000202652     2.82002       60.8018      0.410187       7.70574        100    
reco:pmtrajfittccalo:Calorimetry                      7.0613e-05     0.0339699      0.80699     0.00160009     0.113045        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5909e-05    0.000618843   0.00686011    0.000209981   0.00115956       100    
reco:blurredcluster:BlurredClustering                 0.00901119      1.34571       9.69912      0.482775       1.99902        100    
reco:emtrkmichelid:EmTrackMichelId                     0.268583       18.3863       213.324       3.7939        37.3763        100    
reco:emshower:EMShower                                0.000335162    0.042943      0.384882     0.00802143     0.0752573       100    
reco:cvnmap:CVNMapper                                 2.5779e-05     0.0122305     0.0522014     0.013051      0.0111966       100    
reco:cvneva:CVNEvaluator                              1.7482e-05     0.906028       2.77425       1.21557      0.641216        100    
reco:energyrecnumu:EnergyReco                          0.0002755     0.112826       3.43468     0.00669645     0.423703        100    
reco:energyrecnue:EnergyReco                          0.000135575   0.00354132     0.0453439    0.00073429    0.00794399       100    
reco:energyrecnc:EnergyReco                           0.00208015    0.00652172     0.0513138    0.00365669    0.00856795       100    
reco:energyrecnumurange:EnergyReco                    0.000127651    0.111747       3.66694     0.00323317      0.44165        100    
reco:energyrecnumumcs:EnergyReco                      0.00134171     0.108524       3.31851     0.00450843     0.410794        100    
reco:opdec:Deconvolution                               0.0277783      0.3003        1.65857      0.216844      0.270879        100    
reco:ophitspe:OpHitFinderDeco                          0.166381       2.01425       3.86714       1.91583      0.979529        100    
reco:opflash:OpFlashFinder                            0.000282794   0.00176092    0.00541472    0.00167291    0.000897515      100    
reco:rns:RandomNumberSaver                            1.8424e-05    4.23449e-05   0.000401758   3.76615e-05   3.86931e-05      100    
reco:anglereconue:NuAngularReco                       0.000168027   0.00263578     0.0397568    0.000551786   0.00637755       100    
reco:anglereconumu:NuAngularReco                       7.959e-05    0.00237632     0.0378456    0.000310276   0.00624573       100    
reco:anglereconuepfps:NuAngularReco                   0.000131488   0.00255174     0.038187     0.000481198   0.00625919       100    
reco:anglereconumupfps:NuAngularReco                  0.00011127    0.00245379     0.0380401    0.000417482   0.00626783       100    
[art]:TriggerResults:TriggerResultInserter             9.498e-06    1.6128e-05    6.2618e-05    1.4252e-05    7.54347e-06      100    
end_path:out1:RootOutput                               2.875e-06    6.81083e-06   2.1992e-05     6.582e-06    3.09438e-06      100    
end_path:out1:RootOutput(write)                        0.0206165     0.300133       1.41925      0.294571      0.248522        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21690.8 MB
  Peak resident set size usage (VmHWM): 2275.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2_reco_2024-11-23T_043334Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_674_20231204T002619Z_gen_g4_detsim_hitreco__20240509T210519Z_reco2_reco_data_2024-11-23T_043334Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work