Jobsub ID 108520.158@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108520.158@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:45:48 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 22:58:53 |
| From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 22:55:34 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_185_20231208T061712Z_gen_g4_detsim_hitreco__20240510T043335Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (134s) |
| CPU time | 2m (129s = 96%) |
| Outputting started | 2024-11-22 22:57:49 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_185_20231208T061712Z_gen_g4_detsim_hitreco__20240510T043335Z_reco2.root
|
| Finished | 2024-11-22 22:58:53 |
| Saved logs | justin-logs:108520.158-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Run: 1 event: 18503
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:57:48 CET run: 6472619 subRun: 1 event: 18503
Comparing two wires in the same plane: return failure
%MSG
22-Nov-2024 23:57:48 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/a2/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_185_20231208T061712Z_gen_g4_detsim_hitreco__20240510T043335Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00113491 6.81459 16.8414 3.60123 7.24076 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00109637 0.00128249 0.00161618 0.00113491 0.000236482 3
reco:linecluster:LineCluster 0.0108041 0.0751143 0.160443 0.0540962 0.0628715 3
reco:trajcluster:TrajCluster 0.00176848 0.725803 2.10334 0.0723016 0.974491 3
reco:pandora:StandardPandora 2.14027 2.94744 3.53493 3.16712 0.590178 3
reco:pandoraTrack:LArPandoraTrackCreation 0.000199035 0.0117783 0.0285945 0.00654134 0.0121695 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.000614924 0.0454435 0.117147 0.0185683 0.0512292 3
reco:pandoracalo:Calorimetry 0.000134811 0.0184355 0.0499488 0.00522296 0.0223798 3
reco:pandorapid:Chi2ParticleID 5.1672e-05 0.000673081 0.00165514 0.000312436 0.00070253 3
reco:trkshowersplit:TrackShowerHits 0.000172992 0.0209793 0.0590201 0.00374473 0.0269384 3
reco:pmtrack:PMAlgTrackMaker 0.00434281 3.01974 8.497 0.55788 3.87959 3
reco:pmtrackcalo:Calorimetry 0.000274435 0.0222504 0.0651618 0.00131507 0.0303459 3
reco:pmtrackpid:Chi2ParticleID 9.4785e-05 0.000417763 0.000968506 0.000189999 0.000391369 3
reco:pmtrajfit:PMAlgTrajFitter 0.00607123 1.83991 3.6521 1.86156 1.48856 3
reco:pmtrajfitcalo:Calorimetry 0.000270365 0.0192792 0.0544882 0.00307915 0.0249229 3
reco:pmtrajfitpid:Chi2ParticleID 8.6043e-05 0.000234009 0.000362491 0.000253493 0.000113697 3
reco:pmtracktc:PMAlgTrackMaker 0.00445121 3.66139 9.96016 1.01955 4.47314 3
reco:pmtracktccalo:Calorimetry 0.000247548 0.0390194 0.11427 0.00254008 0.0532188 3
reco:pmtracktcpid:Chi2ParticleID 8.22e-05 0.000419095 0.000898716 0.000276369 0.000348284 3
reco:pmtrajfittc:PMAlgTrajFitter 0.0579006 3.31321 9.19385 0.687887 4.16618 3
reco:pmtrajfittccalo:Calorimetry 0.000281321 0.0231299 0.066541 0.00256744 0.0307105 3
reco:pmtrajfittcpid:Chi2ParticleID 8.5659e-05 0.000491308 0.00111786 0.000270402 0.000449416 3
reco:blurredcluster:BlurredClustering 0.166559 0.240632 0.314704 0.240632 0.0740722 2
reco:emtrkmichelid:EmTrackMichelId 0.337484 3.00242 5.66736 3.00242 2.66494 2
reco:emshower:EMShower 0.00184753 0.00580235 0.00975717 0.00580235 0.00395482 2
reco:cvnmap:CVNMapper 4.2842e-05 0.00621794 0.012393 0.00621794 0.0061751 2
reco:cvneva:CVNEvaluator 2.6237e-05 0.594756 1.18949 0.594756 0.59473 2
reco:energyrecnumu:EnergyReco 0.00265813 0.00635426 0.0100504 0.00635426 0.00369613 2
reco:energyrecnue:EnergyReco 0.000225727 0.000641158 0.00105659 0.000641158 0.000415431 2
reco:energyrecnc:EnergyReco 0.00253083 0.00307681 0.00362278 0.00307681 0.000545974 2
reco:energyrecnumurange:EnergyReco 0.00256253 0.00349636 0.00443019 0.00349636 0.000933827 2
reco:energyrecnumumcs:EnergyReco 0.00255426 0.00508644 0.00761861 0.00508644 0.00253217 2
reco:opdec:Deconvolution 0.0850919 0.169458 0.253824 0.169458 0.0843659 2
reco:ophitspe:OpHitFinderDeco 0.570881 1.14874 1.7266 1.14874 0.577861 2
reco:opflash:OpFlashFinder 0.000462995 0.00255065 0.0046383 0.00255065 0.00208765 2
reco:rns:RandomNumberSaver 1.994e-05 0.000300251 0.000580563 0.000300251 0.000280312 2
reco:anglereconue:NuAngularReco 0.000197444 0.000902171 0.0016069 0.000902171 0.000704726 2
reco:anglereconumu:NuAngularReco 0.000146495 0.000397395 0.000648295 0.000397395 0.0002509 2
reco:anglereconuepfps:NuAngularReco 0.000190958 0.00155693 0.0029229 0.00155693 0.00136597 2
reco:anglereconumupfps:NuAngularReco 0.000169959 0.000444876 0.000719792 0.000444876 0.000274917 2
[art]:TriggerResults:TriggerResultInserter 1.1872e-05 2.79685e-05 4.4065e-05 2.79685e-05 1.60965e-05 2
end_path:out1:RootOutput 2.891e-06 8.7185e-06 1.4546e-05 8.7185e-06 5.8275e-06 2
end_path:out1:RootOutput(write) 0.0486663 0.177829 0.306992 0.177829 0.129163 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6415.44 MB
Peak resident set size usage (VmHWM): 1666.67 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 22-Nov-2024 23:57:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6472619 subRun: 1 event: 18503
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-b7b7-4566-2b59-5217.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_185_20231208T061712Z_gen_g4_detsim_hitreco__20240510T043335Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_185_20231208T061712Z_gen_g4_detsim_hitreco__20240510T043335Z_reco2_reco_2024-11-22T_225537Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 