Jobsub ID 301686.119@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301686.119@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:54:28
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 04:52:49
From worker node	Hostname	comp19-19
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:56:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3350s)
	CPU time	47m (2874s = 85%)
Outputting started	2024-11-23 04:52:34
Output files
Finished	2024-11-23 04:52:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

geDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:52:17 GMT run: 50607041 subRun: 1 event: 28700
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:52:17 GMT run: 50607041 subRun: 1 event: 28700
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:52:17 GMT run: 50607041 subRun: 1 event: 28700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:52:17 GMT run: 50607041 subRun: 1 event: 28700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:52:27 GMT run: 50607041 subRun: 1 event: 28700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:52:27 GMT run: 50607041 subRun: 1 event: 28700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:52:27 GMT run: 50607041 subRun: 1 event: 28700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 379, 296, 305
minwire 0: 447
minwire 1: 1833
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.333366, 0.663008, 0.00362608, 
Output 1: 0.210024, 0.746485, 0.042806, 0.000685598, 
Output 2: 0.98689, 0.0131098, 

486
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1584!
23-Nov-2024 04:52:32 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2_reco_data_2024-11-23T_035647Z.root"
23-Nov-2024 04:52:32 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/bf/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.719956       30.1178       618.678       13.9214       67.8608        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0105385     0.138428       1.07618      0.0859453     0.147706        100    
reco:linecluster:LineCluster                           0.108776       0.68166       27.6642      0.324667       2.74468        100    
reco:trajcluster:TrajCluster                           0.0297825     0.893423       48.3492      0.115408       5.02104        100    
reco:pandora:StandardPandora                            0.14289       2.45763       62.709        1.42498       6.41743        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130462   0.00614284     0.113298     0.00181607     0.0139918       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000178482    0.0183054     0.257852     0.00372552     0.0414874       100    
reco:pandoracalo:Calorimetry                           9.75e-05      0.025536       1.61882     0.00185331     0.162592        100    
reco:pandorapid:Chi2ParticleID                         3.742e-05    0.00015429    0.00132638    0.000120116   0.000147829      100    
reco:trkshowersplit:TrackShowerHits                   0.000103134    0.0167065     0.714867      0.0011586     0.0837436       100    
reco:pmtrack:PMAlgTrackMaker                          0.00118669      2.21671       93.3498       0.28384       9.66312        100    
reco:pmtrackcalo:Calorimetry                          8.0146e-05     0.0114702     0.547541     0.00132148     0.055536        100    
reco:pmtrackpid:Chi2ParticleID                        3.2563e-05    0.000250046   0.00290899    0.000150952   0.000368775      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000195274     1.79745       41.5467      0.462822       4.99992        100    
reco:pmtrajfitcalo:Calorimetry                        6.3729e-05     0.025741       1.59945     0.00206814     0.160491        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8073e-05    0.000177327   0.00101157    0.000136471   0.000143969      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00105631      2.18077       88.312       0.324625       9.13266        100    
reco:pmtracktccalo:Calorimetry                         7.286e-05    0.00762264     0.130272     0.00120049     0.0193833       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0578e-05    0.000226997   0.00199277    0.00014618    0.000270099      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00017782      1.33853       32.0891      0.305884       3.68044        100    
reco:pmtrajfittccalo:Calorimetry                      6.2819e-05     0.0147066     0.632014     0.00179973     0.0664943       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8035e-05    0.000292664    0.0037688    0.000161254   0.000470353      100    
reco:blurredcluster:BlurredClustering                 6.8706e-05      1.18191       7.45711      0.736251       1.42335        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104299     13.0779       197.549       5.22941       25.2323        100    
reco:emshower:EMShower                                0.000151109    0.0247154     0.266784     0.00434195     0.0419619       100    
reco:cvnmap:CVNMapper                                 1.8478e-05     0.0156564     0.0593676     0.0138056     0.0118744       100    
reco:cvneva:CVNEvaluator                              1.5634e-05     0.666257       1.47245      0.824196       0.39192        100    
reco:energyrecnumu:EnergyReco                         0.000342391    0.069994       2.63003     0.00543665     0.326297        100    
reco:energyrecnue:EnergyReco                          0.000133947    0.0018481     0.0343188    0.000646214   0.00422828       100    
reco:energyrecnc:EnergyReco                           0.00197486    0.00395205     0.0363562    0.00251638    0.00457187       100    
reco:energyrecnumurange:EnergyReco                    0.000250403    0.0664528      2.53823     0.00265459     0.319011        100    
reco:energyrecnumumcs:EnergyReco                      0.00168588     0.0677325      2.53927     0.00450975     0.318799        100    
reco:opdec:Deconvolution                               0.128729      0.439784      0.980351      0.411705      0.170241        100    
reco:ophitspe:OpHitFinderDeco                          0.106388       2.42712       5.14189       2.3798       0.891587        100    
reco:opflash:OpFlashFinder                            0.000144553    0.0012691    0.00419963    0.00121974    0.000545272      100    
reco:rns:RandomNumberSaver                            1.8131e-05    3.17035e-05   0.000274628   2.79585e-05   2.59022e-05      100    
reco:anglereconue:NuAngularReco                       0.000151854   0.00145424     0.0269304    0.000410683   0.00336746       100    
reco:anglereconumu:NuAngularReco                      7.3989e-05     0.0012218     0.0261588    0.000245251   0.00326767       100    
reco:anglereconuepfps:NuAngularReco                   0.000118308   0.00147736     0.0261504    0.000347569   0.00353361       100    
reco:anglereconumupfps:NuAngularReco                  0.000100331   0.00130204     0.026006     0.000320694   0.00326514       100    
[art]:TriggerResults:TriggerResultInserter             9.794e-06    1.17265e-05   4.6505e-05    1.11715e-05   3.6617e-06       100    
end_path:out1:RootOutput                               2.66e-06     5.58688e-06   1.8905e-05     4.599e-06    2.86881e-06      100    
end_path:out1:RootOutput(write)                       0.00742754     0.233817       1.04343      0.225659      0.151705        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9305.13 MB
  Peak resident set size usage (VmHWM): 2238.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2_reco_2024-11-23T_035647Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_286_20231207T035355Z_gen_g4_detsim_hitreco__20240510T054727Z_reco2_reco_data_2024-11-23T_035647Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work