Jobsub ID 301488.14@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301488.14@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:59:11 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:19:02 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:14:45 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_924_20231204T140424Z_gen_g4_detsim_hitreco__20240509T204604Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (172s) |
| CPU time | 2m (166s = 96%) |
| Outputting started | 2024-11-22 23:17:37 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_924_20231204T140424Z_gen_g4_detsim_hitreco__20240509T204604Z_reco2.root
|
| Finished | 2024-11-22 23:19:02 |
| Saved logs | justin-logs:301488.14-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
eIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
1st wire C:0 T:18 P:2 W:782 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:17:36 CET run: 74517882 subRun: 1 event: 92409
TuneFullTree failed.
%MSG
23-Nov-2024 00:17:36 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/a8/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_924_20231204T140424Z_gen_g4_detsim_hitreco__20240509T204604Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00138481 10.4337 26.2312 7.07921 7.62183 9
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000973308 0.00120609 0.00140923 0.001231 0.00014612 9
reco:linecluster:LineCluster 0.0118897 0.023838 0.0469852 0.0241217 0.0109734 9
reco:trajcluster:TrajCluster 0.0020909 0.024036 0.0662921 0.0144626 0.0219728 9
reco:pandora:StandardPandora 1.38016 1.74903 3.14668 1.50227 0.545816 9
reco:pandoraTrack:LArPandoraTrackCreation 0.000507286 0.00237157 0.00588389 0.00188376 0.00178256 9
reco:pandoraShower:LArPandoraModularShowerCreation 0.00061069 0.00715342 0.0263157 0.00290552 0.00772693 9
reco:pandoracalo:Calorimetry 0.000217179 0.0020681 0.00862087 0.00122193 0.00244156 9
reco:pandorapid:Chi2ParticleID 7.2411e-05 0.00019383 0.000648315 0.000153479 0.000164421 9
reco:trkshowersplit:TrackShowerHits 0.000186528 0.00147003 0.00401508 0.00143507 0.00122293 9
reco:pmtrack:PMAlgTrackMaker 0.0019857 0.427276 1.45891 0.1432 0.480042 9
reco:pmtrackcalo:Calorimetry 0.00011578 0.00166053 0.00629931 0.00110128 0.00181051 9
reco:pmtrackpid:Chi2ParticleID 4.3636e-05 0.000158884 0.000264861 0.000180055 7.13639e-05 9
reco:pmtrajfit:PMAlgTrajFitter 0.0261664 0.496096 2.32709 0.336009 0.677798 9
reco:pmtrajfitcalo:Calorimetry 0.000242241 0.00252894 0.00852627 0.00236988 0.00238644 9
reco:pmtrajfitpid:Chi2ParticleID 7.3583e-05 0.000237811 0.000840276 0.000177723 0.000219971 9
reco:pmtracktc:PMAlgTrackMaker 0.00537662 0.401453 1.2099 0.215358 0.405482 9
reco:pmtracktccalo:Calorimetry 0.000242001 0.00164139 0.0059576 0.000819492 0.00174579 9
reco:pmtracktcpid:Chi2ParticleID 7.2559e-05 0.000162138 0.000263589 0.000166782 6.56835e-05 9
reco:pmtrajfittc:PMAlgTrajFitter 0.00637376 0.524005 1.15755 0.401406 0.439859 9
reco:pmtrajfittccalo:Calorimetry 0.000198657 0.00182139 0.0050885 0.0011976 0.00164464 9
reco:pmtrajfittcpid:Chi2ParticleID 6.51e-05 0.000180319 0.000292996 0.000186707 8.55402e-05 9
reco:blurredcluster:BlurredClustering 0.0364154 0.599225 1.40132 0.469525 0.509372 8
reco:emtrkmichelid:EmTrackMichelId 0.752331 4.6209 14.2738 3.10588 4.49701 8
reco:emshower:EMShower 0.000452003 0.0180672 0.0730906 0.00145725 0.0251809 8
reco:cvnmap:CVNMapper 3.3977e-05 0.00955962 0.016957 0.0109566 0.0058431 8
reco:cvneva:CVNEvaluator 2.2134e-05 0.867242 1.1813 1.15093 0.500712 8
reco:energyrecnumu:EnergyReco 0.000384407 0.0235888 0.141033 0.00492268 0.044974 8
reco:energyrecnue:EnergyReco 0.00019126 0.000722604 0.00171957 0.000700423 0.000476666 8
reco:energyrecnc:EnergyReco 0.00259481 0.00301997 0.00411021 0.00281026 0.000469259 8
reco:energyrecnumurange:EnergyReco 0.000269742 0.02113 0.13372 0.00284369 0.043153 8
reco:energyrecnumumcs:EnergyReco 0.0026253 0.0227458 0.133749 0.00316746 0.0426276 8
reco:opdec:Deconvolution 0.114821 0.17654 0.214477 0.181369 0.031701 8
reco:ophitspe:OpHitFinderDeco 0.805254 1.32307 1.56364 1.41675 0.252599 8
reco:opflash:OpFlashFinder 0.000795635 0.0016425 0.00454872 0.00128784 0.00111941 8
reco:rns:RandomNumberSaver 2.0529e-05 6.15749e-05 0.000314758 2.41055e-05 9.58394e-05 8
reco:anglereconue:NuAngularReco 0.000194619 0.000559539 0.00109731 0.000488128 0.000341582 8
reco:anglereconumu:NuAngularReco 0.000119202 0.000369587 0.000936057 0.000265348 0.000264348 8
reco:anglereconuepfps:NuAngularReco 0.000238411 0.000839664 0.00204725 0.000519518 0.000639154 8
reco:anglereconumupfps:NuAngularReco 0.000208006 0.000440248 0.00104133 0.000317467 0.000270717 8
[art]:TriggerResults:TriggerResultInserter 1.2953e-05 1.85784e-05 4.9769e-05 1.3469e-05 1.1872e-05 8
end_path:out1:RootOutput 2.888e-06 5.574e-06 1.7316e-05 3.037e-06 4.6587e-06 8
end_path:out1:RootOutput(write) 0.0900361 0.230143 0.328686 0.257992 0.0848653 8
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6419.01 MB
Peak resident set size usage (VmHWM): 1637.94 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:17:37 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74517882 subRun: 1 event: 92409
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-99ee-4a7a-c94c-ce19.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_924_20231204T140424Z_gen_g4_detsim_hitreco__20240509T204604Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_924_20231204T140424Z_gen_g4_detsim_hitreco__20240509T204604Z_reco2_reco_2024-11-22T_231448Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 