Jobsub ID 108550.162@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108550.162@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:15:23 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:38:51 |
| From worker node | Hostname | wn-choc-026.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:33:36 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_5_20231201T050653Z_gen_g4_detsim_hitreco__20240507T192950Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (161s) |
| CPU time | 2m (154s = 95%) |
| Outputting started | 2024-11-22 23:36:18 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_5_20231201T050653Z_gen_g4_detsim_hitreco__20240507T192950Z_reco2.root
|
| Finished | 2024-11-22 23:38:51 |
| Saved logs | justin-logs:108550.162-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:36:17 CET run: 74481093 subRun: 1 event: 508
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:36:18 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/33/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_5_20231201T050653Z_gen_g4_detsim_hitreco__20240507T192950Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00106406 9.31451 16.976 10.468 5.18544 8
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000926553 0.00111314 0.00126706 0.00110643 9.77835e-05 8
reco:linecluster:LineCluster 0.0114599 0.0335839 0.105865 0.0222229 0.0296861 8
reco:trajcluster:TrajCluster 0.00229725 0.0403315 0.162128 0.0187702 0.0501114 8
reco:pandora:StandardPandora 1.35335 1.78267 3.21334 1.52021 0.598797 8
reco:pandoraTrack:LArPandoraTrackCreation 0.000171058 0.00282873 0.00973431 0.00134301 0.00302179 8
reco:pandoraShower:LArPandoraModularShowerCreation 0.000483151 0.00471254 0.0173655 0.00299669 0.00507052 8
reco:pandoracalo:Calorimetry 0.000124151 0.00260721 0.00901265 0.000983595 0.00295063 8
reco:pandorapid:Chi2ParticleID 4.7929e-05 0.000289283 0.00147558 0.00013341 0.000449865 8
reco:trkshowersplit:TrackShowerHits 0.000229712 0.00234753 0.0102951 0.00102768 0.00315831 8
reco:pmtrack:PMAlgTrackMaker 0.00228881 0.379407 0.892503 0.347453 0.287873 8
reco:pmtrackcalo:Calorimetry 0.000251298 0.0016238 0.00414552 0.0011745 0.00122693 8
reco:pmtrackpid:Chi2ParticleID 7.8444e-05 0.000204989 0.000374742 0.000186211 8.91737e-05 8
reco:pmtrajfit:PMAlgTrajFitter 0.0304727 0.730682 3.31061 0.237014 1.03333 8
reco:pmtrajfitcalo:Calorimetry 0.000235679 0.00291572 0.010582 0.00171577 0.00312612 8
reco:pmtrajfitpid:Chi2ParticleID 8.5247e-05 0.000184449 0.000332118 0.000174731 7.09786e-05 8
reco:pmtracktc:PMAlgTrackMaker 0.00361072 0.421884 1.60107 0.252518 0.477815 8
reco:pmtracktccalo:Calorimetry 0.000358685 0.00180888 0.00521812 0.00141553 0.00151451 8
reco:pmtracktcpid:Chi2ParticleID 9.9056e-05 0.000220523 0.000435628 0.000202801 0.000105873 8
reco:pmtrajfittc:PMAlgTrajFitter 0.128844 0.497246 1.16994 0.458035 0.326157 8
reco:pmtrajfittccalo:Calorimetry 0.000364486 0.00290152 0.0107239 0.00194702 0.00312292 8
reco:pmtrajfittcpid:Chi2ParticleID 0.000100476 0.000271138 0.000625942 0.000232787 0.000153277 8
reco:blurredcluster:BlurredClustering 0.166186 0.763485 1.91223 0.487075 0.598455 7
reco:emtrkmichelid:EmTrackMichelId 0.747318 3.79646 6.25713 3.76386 1.96411 7
reco:emshower:EMShower 0.00913526 0.0367176 0.079674 0.0267715 0.0260111 7
reco:cvnmap:CVNMapper 4.9534e-05 0.0122209 0.0233351 0.0110361 0.00665362 7
reco:cvneva:CVNEvaluator 2.3346e-05 1.06037 1.35942 1.17603 0.440054 7
reco:energyrecnumu:EnergyReco 0.00168053 0.00416644 0.0108722 0.00305642 0.0028645 7
reco:energyrecnue:EnergyReco 0.000205642 0.000605923 0.00158896 0.000426302 0.000442958 7
reco:energyrecnc:EnergyReco 0.00257483 0.00294104 0.0036172 0.0028085 0.000330363 7
reco:energyrecnumurange:EnergyReco 0.000841732 0.00222076 0.00284522 0.00267164 0.000736422 7
reco:energyrecnumumcs:EnergyReco 0.00258168 0.00368521 0.00547921 0.00330263 0.00103316 7
reco:opdec:Deconvolution 0.133935 0.181435 0.247184 0.163535 0.0378425 7
reco:ophitspe:OpHitFinderDeco 0.94989 1.34483 1.89531 1.21234 0.295722 7
reco:opflash:OpFlashFinder 0.000870324 0.00163272 0.00411907 0.00115203 0.00104602 7
reco:rns:RandomNumberSaver 1.9679e-05 6.11517e-05 0.000276666 2.3082e-05 8.81669e-05 7
reco:anglereconue:NuAngularReco 0.000183022 0.000518713 0.00161591 0.000344017 0.000464517 7
reco:anglereconumu:NuAngularReco 0.000120292 0.000332034 0.000799838 0.000249505 0.000221397 7
reco:anglereconuepfps:NuAngularReco 0.000174362 0.000713599 0.00293246 0.000359086 0.000914555 7
reco:anglereconumupfps:NuAngularReco 0.000172039 0.000405286 0.000866646 0.000343057 0.000229143 7
[art]:TriggerResults:TriggerResultInserter 1.271e-05 1.80767e-05 4.6513e-05 1.3545e-05 1.16171e-05 7
end_path:out1:RootOutput 2.767e-06 5.23757e-06 1.3792e-05 2.967e-06 3.69908e-06 7
end_path:out1:RootOutput(write) 0.06759 0.239839 0.302352 0.262988 0.074219 7
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6406.34 MB
Peak resident set size usage (VmHWM): 1625.08 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:36:18 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74481093 subRun: 1 event: 508
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 7 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-ff55-b118-f330-cf06.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_5_20231201T050653Z_gen_g4_detsim_hitreco__20240507T192950Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_5_20231201T050653Z_gen_g4_detsim_hitreco__20240507T192950Z_reco2_reco_2024-11-22T_233340Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 