Jobsub ID 108784.12@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108784.12@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:15:53
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:55:19
From worker node	Hostname	comp23-13
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:18:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5793s)
	CPU time	1h (5165s = 89%)
Outputting started	2024-11-23 06:55:03
Output files
Finished	2024-11-23 06:55:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:54:58 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:54:59 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:54:59 GMT run: 6207422 subRun: 1 event: 41200
Comparing two wires in the same plane: return failure
%MSG
259
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 554!
23-Nov-2024 06:55:01 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2_reco_data_2024-11-23T_051833Z.root"
23-Nov-2024 06:55:01 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/a5/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.668092       26.2565       679.68        9.1162        69.637         200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0107237     0.0990459     0.637553      0.0704277     0.0792672       200    
reco:linecluster:LineCluster                           0.078026      0.531704       20.3257      0.239853       2.03971        200    
reco:trajcluster:TrajCluster                           0.0311942      1.36356       93.5914      0.0911881      8.49771        200    
reco:pandora:StandardPandora                           0.0669276      1.9864        63.6761       1.22599       5.14491        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000129548   0.00548208     0.201965     0.00119022     0.0173781       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00016775     0.0110561     0.517591     0.00225601     0.0405163       200    
reco:pandoracalo:Calorimetry                          9.6722e-05     0.0242376      2.20455     0.000676094    0.168618        200    
reco:pandorapid:Chi2ParticleID                        3.4868e-05    0.000137256   0.000619577   0.000107191   0.000104826      200    
reco:trkshowersplit:TrackShowerHits                   0.000102597    0.0170908     0.944986     0.000422207    0.0878543       200    
reco:pmtrack:PMAlgTrackMaker                          0.00115077      1.89665       77.0358      0.097364       8.18938        200    
reco:pmtrackcalo:Calorimetry                          8.2915e-05    0.00955248      0.42559     0.000691627    0.044094        200    
reco:pmtrackpid:Chi2ParticleID                        3.1531e-05    0.000229319   0.00290873    0.000113925   0.000371657      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000178855     1.56689       100.782      0.173861       7.70067        200    
reco:pmtrajfitcalo:Calorimetry                        6.3353e-05     0.0241025      2.09726     0.000881775    0.160849        200    
reco:pmtrajfitpid:Chi2ParticleID                      2.6168e-05    0.000150499   0.000718725   0.000119377   0.000115243      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00100987      2.23546       97.5753      0.130262       9.63354        200    
reco:pmtracktccalo:Calorimetry                        7.1738e-05    0.00630065     0.194289     0.000707114    0.0231431       200    
reco:pmtracktcpid:Chi2ParticleID                      2.7984e-05    0.000213545   0.00235874    0.000117829   0.000295457      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000168869     1.14616       39.6446      0.183386       3.82199        200    
reco:pmtrajfittccalo:Calorimetry                      6.2851e-05     0.0145653     0.825506     0.00078265     0.0730572       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6064e-05    0.000270832   0.00428212    0.000122286   0.000465955      200    
reco:blurredcluster:BlurredClustering                 6.4438e-05     0.918842       8.51218      0.486724       1.22747        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000106629     11.0011       208.085       2.65364       25.2863        200    
reco:emshower:EMShower                                0.000164047    0.0199188      0.23012     0.00259874     0.037544        200    
reco:cvnmap:CVNMapper                                 1.9209e-05     0.0131346     0.0461999     0.0129739    0.00906104       200    
reco:cvneva:CVNEvaluator                              1.5338e-05     0.606666      0.856662      0.782042       0.33686        200    
reco:energyrecnumu:EnergyReco                         0.000270433    0.0374499      2.71421     0.00542307     0.215399        200    
reco:energyrecnue:EnergyReco                          0.000131953   0.00172587     0.0537181    0.000459571   0.00489193       200    
reco:energyrecnc:EnergyReco                           0.00186784     0.0034228     0.0556745    0.00218369    0.00497203       200    
reco:energyrecnumurange:EnergyReco                    0.000192042    0.0352833      2.79216     0.00203868     0.220319        200    
reco:energyrecnumumcs:EnergyReco                      0.000766087    0.0359541      2.73134     0.00257892     0.215741        200    
reco:opdec:Deconvolution                               0.0871927     0.361918       0.9171       0.334242      0.140141        200    
reco:ophitspe:OpHitFinderDeco                           0.12145       2.07302       4.15671       2.00662      0.831378        200    
reco:opflash:OpFlashFinder                            0.000260474   0.00121062    0.00286133    0.00114731    0.000493802      200    
reco:rns:RandomNumberSaver                            1.7726e-05    3.05347e-05   0.000242827   2.72065e-05   1.78109e-05      200    
reco:anglereconue:NuAngularReco                       0.000141671   0.00127883     0.0475157    0.000294718   0.00417834       200    
reco:anglereconumu:NuAngularReco                      6.6386e-05    0.00109464     0.0461711    0.000157261   0.00403718       200    
reco:anglereconuepfps:NuAngularReco                   0.000109423   0.00118736     0.0460423    0.000260489   0.00401077       200    
reco:anglereconumupfps:NuAngularReco                  9.4374e-05    0.00116438     0.0482729    0.000224711    0.0041446       200    
[art]:TriggerResults:TriggerResultInserter             8.483e-06    1.10884e-05   4.3018e-05    1.00595e-05   3.12413e-06      200    
end_path:out1:RootOutput                               2.45e-06     4.62385e-06   1.8845e-05    3.3525e-06    2.19017e-06      200    
end_path:out1:RootOutput(write)                        0.011159      0.201092       1.08062      0.195588      0.139265        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9788.06 MB
  Peak resident set size usage (VmHWM): 2624.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2_reco_2024-11-23T_051833Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_205_20231121T150908Z_gen_g4_detsim_hitreco__20240503T061206Z_reco2_reco_data_2024-11-23T_051833Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work