Jobsub ID 108521.7@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108521.7@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:47:01 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:18:57 |
| From worker node | Hostname | wn-choc-026.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 22:58:05 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_231_20231207T052258Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 15m (910s) |
| CPU time | 14m (897s = 98%) |
| Outputting started | 2024-11-22 23:13:16 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_231_20231207T052258Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root
|
| Finished | 2024-11-22 23:18:57 |
| Saved logs | justin-logs:108521.7-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:13 CET run: 50609765 subRun: 1 event: 23103
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:14 CET run: 50609765 subRun: 1 event: 23103
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:13:15 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/88/d8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_231_20231207T052258Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00121238 6.09992 9.62259 8.67597 4.32972 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00103857 0.00118177 0.00129435 0.00121238 0.000106644 3
reco:linecluster:LineCluster 0.0133782 3.61199 10.7389 0.0837131 5.03956 3
reco:trajcluster:TrajCluster 0.00594509 43.1366 129.387 0.0167874 60.9882 3
reco:pandora:StandardPandora 2.16667 142.616 422.628 3.05236 197.999 3
reco:pandoraTrack:LArPandoraTrackCreation 0.000314962 0.00178025 0.00430156 0.000724218 0.00179065 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.000682758 0.00508017 0.0131332 0.00142458 0.00570237 3
reco:pandoracalo:Calorimetry 0.000186059 0.00126702 0.00322576 0.000389248 0.00138752 3
reco:pandorapid:Chi2ParticleID 8.3356e-05 0.000498501 0.00130808 0.000104064 0.000572524 3
reco:trkshowersplit:TrackShowerHits 0.000496631 0.335206 1.00315 0.00196793 0.472311 3
reco:pmtrack:PMAlgTrackMaker 0.0433024 27.7809 83.2111 0.0883573 39.1951 3
reco:pmtrackcalo:Calorimetry 0.000543391 0.245093 0.734122 0.000614747 0.345795 3
reco:pmtrackpid:Chi2ParticleID 0.00011043 0.00172214 0.00492941 0.000126588 0.00226789 3
reco:pmtrajfit:PMAlgTrajFitter 0.00189729 0.0859552 0.182275 0.0736929 0.0741478 3
reco:pmtrajfitcalo:Calorimetry 0.000143402 0.000374188 0.000514952 0.00046421 0.0001645 3
reco:pmtrajfitpid:Chi2ParticleID 5.4216e-05 8.79783e-05 0.00010571 0.000104009 2.38837e-05 3
reco:pmtracktc:PMAlgTrackMaker 0.107766 41.9459 125.592 0.137598 59.147 3
reco:pmtracktccalo:Calorimetry 0.000534136 0.103562 0.309483 0.000667687 0.145609 3
reco:pmtracktcpid:Chi2ParticleID 0.000126197 0.00105625 0.00290462 0.000137932 0.001307 3
reco:pmtrajfittc:PMAlgTrajFitter 0.10957 13.7641 41.055 0.127703 19.2976 3
reco:pmtrajfittccalo:Calorimetry 0.000541405 0.298734 0.894986 0.000673455 0.421614 3
reco:pmtrajfittcpid:Chi2ParticleID 0.000114809 0.00167533 0.00477329 0.000137889 0.00219061 3
reco:blurredcluster:BlurredClustering 0.330027 0.759128 1.18823 0.759128 0.429101 2
reco:emtrkmichelid:EmTrackMichelId 1.91811 2.16193 2.40575 2.16193 0.24382 2
reco:emshower:EMShower 0.0125712 0.0193934 0.0262156 0.0193934 0.00682222 2
reco:cvnmap:CVNMapper 0.0119082 0.0167401 0.0215719 0.0167401 0.00483186 2
reco:cvneva:CVNEvaluator 1.18263 1.19712 1.21161 1.19712 0.0144873 2
reco:energyrecnumu:EnergyReco 0.00168895 0.00403705 0.00638515 0.00403705 0.0023481 2
reco:energyrecnue:EnergyReco 0.00026926 0.000356301 0.000443342 0.000356301 8.7041e-05 2
reco:energyrecnc:EnergyReco 0.00266542 0.00282254 0.00297966 0.00282254 0.000157119 2
reco:energyrecnumurange:EnergyReco 0.000716854 0.0010323 0.00134775 0.0010323 0.000315448 2
reco:energyrecnumumcs:EnergyReco 0.0031863 0.00359637 0.00400644 0.00359637 0.000410065 2
reco:opdec:Deconvolution 0.165233 0.187296 0.209359 0.187296 0.0220628 2
reco:ophitspe:OpHitFinderDeco 1.07518 1.34104 1.60691 1.34104 0.265867 2
reco:opflash:OpFlashFinder 0.00141061 0.00254 0.00366939 0.00254 0.00112939 2
reco:rns:RandomNumberSaver 2.2838e-05 0.000152402 0.000281966 0.000152402 0.000129564 2
reco:anglereconue:NuAngularReco 0.000280676 0.000693636 0.0011066 0.000693636 0.00041296 2
reco:anglereconumu:NuAngularReco 0.000156199 0.000207823 0.000259448 0.000207823 5.16245e-05 2
reco:anglereconuepfps:NuAngularReco 0.000274873 0.00126302 0.00225118 0.00126302 0.000988152 2
reco:anglereconumupfps:NuAngularReco 0.000240075 0.000269511 0.000298947 0.000269511 2.9436e-05 2
[art]:TriggerResults:TriggerResultInserter 1.2895e-05 2.96385e-05 4.6382e-05 2.96385e-05 1.67435e-05 2
end_path:out1:RootOutput 4.813e-06 9.8225e-06 1.4832e-05 9.8225e-06 5.0095e-06 2
end_path:out1:RootOutput(write) 0.270254 0.32579 0.381326 0.32579 0.0555357 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6425.11 MB
Peak resident set size usage (VmHWM): 1677.62 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:13:15 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50609765 subRun: 1 event: 23103
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-a89f-1c9e-e857-2c49.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_231_20231207T052258Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_231_20231207T052258Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_reco_2024-11-22T_225808Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 