Jobsub ID 108605.44@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108605.44@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:35:25 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:48:54 |
| From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:41:01 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_590_20231208T060051Z_gen_g4_detsim_hitreco__20240510T050241Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 1m (107s) |
| CPU time | 1m (100s = 93%) |
| Outputting started | 2024-11-22 23:42:49 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_590_20231208T060051Z_gen_g4_detsim_hitreco__20240510T050241Z_reco2.root
|
| Finished | 2024-11-22 23:48:54 |
| Saved logs | justin-logs:108605.44-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:42:47 CET run: 6492237 subRun: 1 event: 59003
TuneFullTree failed.
%MSG
23-Nov-2024 00:42:48 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5c/86/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_590_20231208T060051Z_gen_g4_detsim_hitreco__20240510T050241Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00122755 6.38401 10.1658 8.98502 4.53898 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00102551 0.00115292 0.00122755 0.00120571 9.05331e-05 3
reco:linecluster:LineCluster 0.0198236 0.031878 0.039897 0.0359133 0.00867752 3
reco:trajcluster:TrajCluster 0.0124656 0.0507945 0.11608 0.0238375 0.0463969 3
reco:pandora:StandardPandora 2.17988 2.50984 3.11315 2.23648 0.427234 3
reco:pandoraTrack:LArPandoraTrackCreation 0.00122945 0.00569819 0.00981833 0.0060468 0.00351505 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.00287086 0.0195586 0.0388633 0.0169415 0.0148099 3
reco:pandoracalo:Calorimetry 0.000986198 0.00576178 0.0116381 0.004661 0.00441775 3
reco:pandorapid:Chi2ParticleID 0.00010029 0.000647366 0.00165385 0.000187961 0.000712589 3
reco:trkshowersplit:TrackShowerHits 0.000564053 0.00368291 0.00760162 0.00288305 0.00292822 3
reco:pmtrack:PMAlgTrackMaker 0.0917843 0.472885 1.14043 0.186438 0.473607 3
reco:pmtrackcalo:Calorimetry 0.000477742 0.00315114 0.00785732 0.00111837 0.00333803 3
reco:pmtrackpid:Chi2ParticleID 0.000103241 0.00020548 0.000341226 0.000171974 0.000100004 3
reco:pmtrajfit:PMAlgTrajFitter 0.204536 1.12206 2.67118 0.490454 1.1016 3
reco:pmtrajfitcalo:Calorimetry 0.00109026 0.00532763 0.0131439 0.00174873 0.00553348 3
reco:pmtrajfitpid:Chi2ParticleID 0.000136929 0.000201726 0.000271189 0.000197061 5.49106e-05 3
reco:pmtracktc:PMAlgTrackMaker 0.113977 0.484829 1.14845 0.192057 0.470335 3
reco:pmtracktccalo:Calorimetry 0.000676802 0.00298271 0.00749438 0.000776956 0.00319049 3
reco:pmtracktcpid:Chi2ParticleID 0.000135245 0.000196637 0.000316635 0.000138031 8.4859e-05 3
reco:pmtrajfittc:PMAlgTrajFitter 0.167028 0.929129 2.135 0.485359 0.862526 3
reco:pmtrajfittccalo:Calorimetry 0.000899784 0.00407683 0.00995523 0.00137548 0.00416119 3
reco:pmtrajfittcpid:Chi2ParticleID 0.000145189 0.000259635 0.000439958 0.000193759 0.00012904 3
reco:blurredcluster:BlurredClustering 0.0761081 0.146489 0.216869 0.146489 0.0703807 2
reco:emtrkmichelid:EmTrackMichelId 2.56563 2.91844 3.27124 2.91844 0.352809 2
reco:emshower:EMShower 0.00765119 0.00851795 0.00938471 0.00851795 0.000866762 2
reco:cvnmap:CVNMapper 0.0120821 0.0198082 0.0275342 0.0198082 0.00772606 2
reco:cvneva:CVNEvaluator 1.18783 1.19309 1.19835 1.19309 0.0052598 2
reco:energyrecnumu:EnergyReco 0.00184986 0.00619457 0.0105393 0.00619457 0.00434471 2
reco:energyrecnue:EnergyReco 0.000488326 0.000564262 0.000640198 0.000564262 7.5936e-05 2
reco:energyrecnc:EnergyReco 0.00323354 0.00323982 0.0032461 0.00323982 6.281e-06 2
reco:energyrecnumurange:EnergyReco 0.00154924 0.00308472 0.00462021 0.00308472 0.00153549 2
reco:energyrecnumumcs:EnergyReco 0.00401969 0.00573107 0.00744245 0.00573107 0.00171138 2
reco:opdec:Deconvolution 0.150376 0.163002 0.175629 0.163002 0.0126266 2
reco:ophitspe:OpHitFinderDeco 1.12227 1.15998 1.19769 1.15998 0.0377101 2
reco:opflash:OpFlashFinder 0.000999943 0.0025332 0.00406647 0.0025332 0.00153326 2
reco:rns:RandomNumberSaver 1.9325e-05 0.000157674 0.000296023 0.000157674 0.000138349 2
reco:anglereconue:NuAngularReco 0.000379087 0.000796307 0.00121353 0.000796307 0.000417219 2
reco:anglereconumu:NuAngularReco 0.00028469 0.000318885 0.00035308 0.000318885 3.4195e-05 2
reco:anglereconuepfps:NuAngularReco 0.000542151 0.00153353 0.00252491 0.00153353 0.000991379 2
reco:anglereconumupfps:NuAngularReco 0.000382197 0.000416995 0.000451793 0.000416995 3.4798e-05 2
[art]:TriggerResults:TriggerResultInserter 1.2007e-05 2.73115e-05 4.2616e-05 2.73115e-05 1.53045e-05 2
end_path:out1:RootOutput 2.784e-06 8.4265e-06 1.4069e-05 8.4265e-06 5.6425e-06 2
end_path:out1:RootOutput(write) 0.251657 0.254522 0.257387 0.254522 0.00286524 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6422.84 MB
Peak resident set size usage (VmHWM): 1675.25 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:42:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6492237 subRun: 1 event: 59003
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-3d0e-9f77-2999-347e.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_590_20231208T060051Z_gen_g4_detsim_hitreco__20240510T050241Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_590_20231208T060051Z_gen_g4_detsim_hitreco__20240510T050241Z_reco2_reco_2024-11-22T_234105Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 