Jobsub ID 92473.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 92473.0@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3920 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-07 00:53:05 |
| Site | US_UConn-HPC |
| Entry | GLUEX_US_UConn-HPC_osgce |
| Last heartbeat | 2024-11-07 02:57:02 |
| From worker node | Hostname | cn438 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-07 00:54:48 |
| Input files | fardet-hd:anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7319s) |
| CPU time | 1h (6662s = 91%) |
| Outputting started | 2024-11-07 02:56:48 |
| Output files | |
| Finished | 2024-11-07 02:57:02 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ne: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 06-Nov-2024 21:56:32 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
Shower cones:
3
9
10
11
16
22
1
6
8
18
19
Event 16600 track shower separation:
Shower initial tracks are:
2
4
Track tracks are:
0
5
7
13
14
15
17
20
21
Boundary wire vector sizes: 481, 562, 533
minwire 0: 2357
minwire 1: 55
minwire 2: 2603
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.113989,
Output 1: 0.000695361, 0.00257896, 0.125435, 0.871291,
Output 2: 0.798496, 0.0326562, 0.0671356, 0.101712,
Output 3: 0.927665, 0.0705996, 0.00151292, 0.000222514,
Output 4: 0.0211922, 0.967965, 0.0102652, 0.000577384,
Output 5: 0.0103312, 0.985763, 0.00368777, 0.000217958,
Output 6: 0.107973, 0.82175, 0.0623337, 0.00794264,
421
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1228!
06-Nov-2024 21:56:47 EST Closed output file "anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_data_2024-11-07T_005452Z.root"
06-Nov-2024 21:56:47 EST Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/4e/f8/anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.785985 67.7432 670.102 20.5641 116.355 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0308842 0.0612685 0.285382 0.0575085 0.024378 100
reco:linecluster:LineCluster 0.144172 0.754775 11.7238 0.259535 1.76475 100
reco:trajcluster:TrajCluster 0.0722781 5.14917 97.1329 0.144967 15.4458 100
reco:pandora:StandardPandora 0.0811415 5.01317 144.484 1.20862 16.1283 100
reco:pandoraTrack:LArPandoraTrackCreation 0.000139443 0.0178645 0.215538 0.00417595 0.0331128 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000178577 0.0321262 0.423319 0.00745644 0.0645643 100
reco:pandoracalo:Calorimetry 8.8487e-05 0.0512719 0.692141 0.00432175 0.11602 100
reco:pandorapid:Chi2ParticleID 3.5337e-05 0.000243941 0.00319499 0.000178447 0.000340106 100
reco:trkshowersplit:TrackShowerHits 7.8608e-05 0.0582271 0.881225 0.00368139 0.140151 100
reco:pmtrack:PMAlgTrackMaker 0.00136817 6.97688 76.424 1.11106 13.9137 100
reco:pmtrackcalo:Calorimetry 7.0463e-05 0.0437932 0.659693 0.00282908 0.104486 100
reco:pmtrackpid:Chi2ParticleID 2.7281e-05 0.000604167 0.00365707 0.000237498 0.000797789 100
reco:pmtrajfit:PMAlgTrajFitter 0.000168969 3.87964 34.525 0.982916 7.02668 100
reco:pmtrajfitcalo:Calorimetry 5.6807e-05 0.0539712 0.778403 0.0048217 0.123467 100
reco:pmtrajfitpid:Chi2ParticleID 2.4537e-05 0.000274212 0.00101573 0.000198294 0.000236784 100
reco:pmtracktc:PMAlgTrackMaker 0.00103348 8.72221 102.653 1.04122 17.8921 100
reco:pmtracktccalo:Calorimetry 6.1536e-05 0.0271255 0.253438 0.00379109 0.0511138 100
reco:pmtracktcpid:Chi2ParticleID 2.649e-05 0.000528791 0.00249229 0.000251609 0.000605831 100
reco:pmtrajfittc:PMAlgTrajFitter 0.000144313 4.16027 36.9168 0.959915 7.50286 100
reco:pmtrajfittccalo:Calorimetry 5.6277e-05 0.0515122 0.662282 0.00464935 0.114074 100
reco:pmtrajfittcpid:Chi2ParticleID 2.3544e-05 0.000688727 0.00400676 0.000313221 0.000878596 100
reco:blurredcluster:BlurredClustering 0.000167696 1.71748 7.8611 1.12353 1.6929 100
reco:emtrkmichelid:EmTrackMichelId 9.6242e-05 27.4903 191.066 9.75768 39.1965 100
reco:emshower:EMShower 0.000124324 0.0340969 0.223084 0.00523232 0.0481639 100
reco:cvnmap:CVNMapper 1.5169e-05 0.0244138 0.0565297 0.0215277 0.0142131 100
reco:cvneva:CVNEvaluator 1.3345e-05 0.711926 2.50795 0.800985 0.333583 100
reco:energyrecnumu:EnergyReco 0.000388332 0.161738 2.28177 0.0147589 0.401329 100
reco:energyrecnue:EnergyReco 0.000131208 0.00618479 0.0786267 0.00145496 0.0125013 100
reco:energyrecnc:EnergyReco 0.00162787 0.00810073 0.0800357 0.00298961 0.0125945 100
reco:energyrecnumurange:EnergyReco 0.000196431 0.153027 2.27045 0.00660586 0.389956 100
reco:energyrecnumumcs:EnergyReco 0.00196667 0.156871 2.27165 0.00917731 0.399807 100
reco:opdec:Deconvolution 0.0790963 0.348696 0.626477 0.344959 0.0920319 100
reco:ophitspe:OpHitFinderDeco 0.223915 1.42825 2.19246 1.46968 0.435882 100
reco:opflash:OpFlashFinder 0.000168087 0.00134807 0.00595606 0.00129632 0.000659064 100
reco:rns:RandomNumberSaver 1.8375e-05 3.2011e-05 0.000362163 2.9055e-05 3.41912e-05 100
[art]:TriggerResults:TriggerResultInserter 1.034e-05 1.33009e-05 4.6849e-05 1.2068e-05 4.99383e-06 100
end_path:out1:RootOutput 3.126e-06 5.55666e-06 2.4346e-05 5.3545e-06 2.4364e-06 100
end_path:out1:RootOutput(write) 0.00688846 0.442885 1.29076 0.389169 0.256403 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 21665.7 MB
Peak resident set size usage (VmHWM): 2801.15 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root
all-input-dids.txt
anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_2024-11-07T_005452Z.log
anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_data_2024-11-07T_005452Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
work 