Jobsub ID 301489.68@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301489.68@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:05:17 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:22:47 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:19:27 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 1m (93s) |
| CPU time | 1m (88s = 94%) |
| Outputting started | 2024-11-22 23:21:00 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root
|
| Finished | 2024-11-22 23:22:47 |
| Saved logs | justin-logs:301489.68-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
BeginModule 23-Nov-2024 00:20:58 CET run: 50515283 subRun: 1 event: 23802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:58 CET run: 50515283 subRun: 1 event: 23802
1st wire C:0 T:11 P:2 W:822 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:59 CET run: 50515283 subRun: 1 event: 23802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:59 CET run: 50515283 subRun: 1 event: 23802
1st wire C:0 T:11 P:2 W:776 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:59 CET run: 50515283 subRun: 1 event: 23802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:59 CET run: 50515283 subRun: 1 event: 23802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:20:59 CET run: 50515283 subRun: 1 event: 23802
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 00:20:59 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/bb/ed/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.000987765 3.05503 6.10907 3.05503 3.05404 2
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000987765 0.00103155 0.00107533 0.00103155 4.37845e-05 2
reco:linecluster:LineCluster 0.0193431 0.0293961 0.0394491 0.0293961 0.010053 2
reco:trajcluster:TrajCluster 0.0135588 0.0595865 0.105614 0.0595865 0.0460277 2
reco:pandora:StandardPandora 2.52817 2.85149 3.17481 2.85149 0.323322 2
reco:pandoraTrack:LArPandoraTrackCreation 0.00636444 0.00676351 0.00716258 0.00676351 0.00039907 2
reco:pandoraShower:LArPandoraModularShowerCreation 0.0128856 0.0146995 0.0165133 0.0146995 0.00181387 2
reco:pandoracalo:Calorimetry 0.00432644 0.00536977 0.0064131 0.00536977 0.00104333 2
reco:pandorapid:Chi2ParticleID 0.000227311 0.000926104 0.0016249 0.000926104 0.000698793 2
reco:trkshowersplit:TrackShowerHits 0.00237319 0.00391798 0.00546276 0.00391798 0.00154478 2
reco:pmtrack:PMAlgTrackMaker 0.0144422 0.338434 0.662427 0.338434 0.323992 2
reco:pmtrackcalo:Calorimetry 0.00030894 0.00289912 0.0054893 0.00289912 0.00259018 2
reco:pmtrackpid:Chi2ParticleID 8.2152e-05 0.000239463 0.000396774 0.000239463 0.000157311 2
reco:pmtrajfit:PMAlgTrajFitter 0.0112311 0.7201 1.42897 0.7201 0.708869 2
reco:pmtrajfitcalo:Calorimetry 0.000315651 0.00394672 0.00757778 0.00394672 0.00363107 2
reco:pmtrajfitpid:Chi2ParticleID 8.6391e-05 0.000166908 0.000247425 0.000166908 8.0517e-05 2
reco:pmtracktc:PMAlgTrackMaker 0.00330315 0.496896 0.990489 0.496896 0.493593 2
reco:pmtracktccalo:Calorimetry 0.000190251 0.00343215 0.00667405 0.00343215 0.0032419 2
reco:pmtracktcpid:Chi2ParticleID 6.5098e-05 0.00026559 0.000466081 0.00026559 0.000200491 2
reco:pmtrajfittc:PMAlgTrajFitter 0.00731872 0.731262 1.4552 0.731262 0.723943 2
reco:pmtrajfittccalo:Calorimetry 0.000288076 0.00388242 0.00747675 0.00388242 0.00359434 2
reco:pmtrajfittcpid:Chi2ParticleID 8.2379e-05 0.000287588 0.000492798 0.000287588 0.000205209 2
reco:blurredcluster:BlurredClustering 0.976379 0.976379 0.976379 0.976379 0 1
reco:emtrkmichelid:EmTrackMichelId 0.815164 0.815164 0.815164 0.815164 0 1
reco:emshower:EMShower 0.013925 0.013925 0.013925 0.013925 0 1
reco:cvnmap:CVNMapper 0.000300175 0.000300175 0.000300175 0.000300175 0 1
reco:cvneva:CVNEvaluator 0.000251392 0.000251392 0.000251392 0.000251392 0 1
reco:energyrecnumu:EnergyReco 0.00602668 0.00602668 0.00602668 0.00602668 0 1
reco:energyrecnue:EnergyReco 0.000510781 0.000510781 0.000510781 0.000510781 0 1
reco:energyrecnc:EnergyReco 0.00305081 0.00305081 0.00305081 0.00305081 0 1
reco:energyrecnumurange:EnergyReco 0.000405731 0.000405731 0.000405731 0.000405731 0 1
reco:energyrecnumumcs:EnergyReco 0.00302912 0.00302912 0.00302912 0.00302912 0 1
reco:opdec:Deconvolution 0.113448 0.113448 0.113448 0.113448 0 1
reco:ophitspe:OpHitFinderDeco 0.822991 0.822991 0.822991 0.822991 0 1
reco:opflash:OpFlashFinder 0.00382624 0.00382624 0.00382624 0.00382624 0 1
reco:rns:RandomNumberSaver 0.000316686 0.000316686 0.000316686 0.000316686 0 1
reco:anglereconue:NuAngularReco 0.00131437 0.00131437 0.00131437 0.00131437 0 1
reco:anglereconumu:NuAngularReco 0.000288035 0.000288035 0.000288035 0.000288035 0 1
reco:anglereconuepfps:NuAngularReco 0.00314722 0.00314722 0.00314722 0.00314722 0 1
reco:anglereconumupfps:NuAngularReco 0.000390344 0.000390344 0.000390344 0.000390344 0 1
[art]:TriggerResults:TriggerResultInserter 6.0352e-05 6.0352e-05 6.0352e-05 6.0352e-05 0 1
end_path:out1:RootOutput 1.746e-05 1.746e-05 1.746e-05 1.746e-05 0 1
end_path:out1:RootOutput(write) 0.0635351 0.0635351 0.0635351 0.0635351 0 1
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6367.66 MB
Peak resident set size usage (VmHWM): 1583.75 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:21:00 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50515283 subRun: 1 event: 23802
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1223-964f-d38a-464f.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2_reco_2024-11-22T_231930Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 