Jobsub ID 92474.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 92474.0@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3920 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-07 03:00:23 |
| Site | US_UConn-HPC |
| Entry | GLUEX_US_UConn-HPC_osgce |
| Last heartbeat | 2024-11-07 05:05:45 |
| From worker node | Hostname | cn438 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-07 03:01:48 |
| Input files | fardet-hd:anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7338s) |
| CPU time | 1h (6800s = 92%) |
| Outputting started | 2024-11-07 05:05:32 |
| Output files | |
| Finished | 2024-11-07 05:05:45 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ne: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 07-Nov-2024 00:01:57 EST run: 1127 subRun: 1 event: 16600
TuneFullTree failed.
%MSG
Shower cones:
3
9
10
11
16
22
1
6
8
18
19
Event 16600 track shower separation:
Shower initial tracks are:
2
4
Track tracks are:
0
5
7
13
14
15
17
20
21
Boundary wire vector sizes: 481, 562, 533
minwire 0: 2357
minwire 1: 55
minwire 2: 2603
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.113989,
Output 1: 0.000695361, 0.00257896, 0.125435, 0.871291,
Output 2: 0.798496, 0.0326562, 0.0671356, 0.101712,
Output 3: 0.927665, 0.0705996, 0.00151292, 0.000222514,
Output 4: 0.0211922, 0.967965, 0.0102652, 0.000577384,
Output 5: 0.0103312, 0.985763, 0.00368777, 0.000217958,
Output 6: 0.107973, 0.82175, 0.0623337, 0.00794264,
421
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1228!
07-Nov-2024 00:04:05 EST Closed output file "anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_data_2024-11-07T_030150Z.root"
07-Nov-2024 00:04:05 EST Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/4e/f8/anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.744828 69.1304 667.268 22.0912 118.018 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0341035 0.0648656 0.118467 0.0625365 0.0145603 100
reco:linecluster:LineCluster 0.143421 0.702276 10.9912 0.252398 1.6806 100
reco:trajcluster:TrajCluster 0.0718247 5.13937 97.0794 0.144865 15.4423 100
reco:pandora:StandardPandora 0.078875 5.01976 140.327 1.38993 15.7123 100
reco:pandoraTrack:LArPandoraTrackCreation 0.00011659 0.0177258 0.221476 0.00407018 0.0341987 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000175601 0.0323937 0.417897 0.00721534 0.0652877 100
reco:pandoracalo:Calorimetry 8.8658e-05 0.0506155 0.73821 0.00383403 0.116952 100
reco:pandorapid:Chi2ParticleID 3.4686e-05 0.000220007 0.00185355 0.000160769 0.00023049 100
reco:trkshowersplit:TrackShowerHits 6.3841e-05 0.058241 0.881806 0.00361651 0.141118 100
reco:pmtrack:PMAlgTrackMaker 0.00115133 6.9249 79.3897 1.10841 13.9706 100
reco:pmtrackcalo:Calorimetry 6.0374e-05 0.0429982 0.68285 0.00296849 0.104417 100
reco:pmtrackpid:Chi2ParticleID 2.5688e-05 0.000599366 0.0038372 0.000236386 0.000808745 100
reco:pmtrajfit:PMAlgTrajFitter 0.000178888 3.78243 34.564 0.978644 6.85161 100
reco:pmtrajfitcalo:Calorimetry 5.7308e-05 0.0534283 0.703939 0.00490427 0.120468 100
reco:pmtrajfitpid:Chi2ParticleID 2.3675e-05 0.000269337 0.00113546 0.00019605 0.000245456 100
reco:pmtracktc:PMAlgTrackMaker 0.000946767 8.63523 102.909 1.05463 17.8267 100
reco:pmtracktccalo:Calorimetry 6.1046e-05 0.026401 0.251087 0.00363076 0.050036 100
reco:pmtracktcpid:Chi2ParticleID 2.4205e-05 0.000516111 0.00242252 0.000240634 0.000600633 100
reco:pmtrajfittc:PMAlgTrajFitter 0.000145114 4.101 36.5049 0.971844 7.41346 100
reco:pmtrajfittccalo:Calorimetry 5.5164e-05 0.0495916 0.691436 0.0048467 0.111314 100
reco:pmtrajfittcpid:Chi2ParticleID 2.3564e-05 0.000665679 0.00447649 0.00028898 0.000870065 100
reco:blurredcluster:BlurredClustering 0.000162446 1.70264 7.86284 1.10575 1.68568 100
reco:emtrkmichelid:EmTrackMichelId 9.1453e-05 29.2616 207.531 10.5692 41.6636 100
reco:emshower:EMShower 0.000123383 0.0345278 0.21999 0.00638235 0.0488348 100
reco:cvnmap:CVNMapper 1.591e-05 0.0222739 0.059183 0.0219362 0.0117745 100
reco:cvneva:CVNEvaluator 1.4778e-05 0.709572 2.18171 0.800157 0.317839 100
reco:energyrecnumu:EnergyReco 0.000319142 0.158096 2.23668 0.0163396 0.390533 100
reco:energyrecnue:EnergyReco 0.000131929 0.00598753 0.0766246 0.00141717 0.0120703 100
reco:energyrecnc:EnergyReco 0.00179923 0.00794255 0.0790718 0.00282282 0.0124699 100
reco:energyrecnumurange:EnergyReco 0.000160753 0.150805 2.22183 0.00652658 0.384412 100
reco:energyrecnumumcs:EnergyReco 0.00179518 0.151621 2.23263 0.00883663 0.381982 100
reco:opdec:Deconvolution 0.0765328 0.339899 0.467496 0.342722 0.0785493 100
reco:ophitspe:OpHitFinderDeco 0.214658 1.43355 2.19276 1.49041 0.437799 100
reco:opflash:OpFlashFinder 0.000113935 0.00132285 0.00552523 0.00132942 0.000628853 100
reco:rns:RandomNumberSaver 1.6651e-05 3.1733e-05 0.000287303 3.03575e-05 2.71441e-05 100
[art]:TriggerResults:TriggerResultInserter 8.987e-06 1.18183e-05 3.6659e-05 1.0721e-05 3.7479e-06 100
end_path:out1:RootOutput 2.726e-06 5.01825e-06 2.0659e-05 4.8395e-06 2.11745e-06 100
end_path:out1:RootOutput(write) 0.00573038 0.445362 1.28449 0.403264 0.253902 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 21665.7 MB
Peak resident set size usage (VmHWM): 2797.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2.root
all-input-dids.txt
anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_2024-11-07T_030150Z.log
anutau_dune10kt_1x2x6_1127_165_20230828T190254Z_gen_g4_detsim_hitreco__20240221T023848Z_reco2_reco_data_2024-11-07T_030150Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
work 