Jobsub ID 108542.94@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108542.94@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:55:24 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 23:15:47 |
| From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:11:30 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_593_20231205T053827Z_gen_g4_detsim_hitreco__20240509T221831Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (124s) |
| CPU time | 1m (117s = 94%) |
| Outputting started | 2024-11-22 23:13:34 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_593_20231205T053827Z_gen_g4_detsim_hitreco__20240509T221831Z_reco2.root
|
| Finished | 2024-11-22 23:15:47 |
| Saved logs | justin-logs:108542.94-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
:pmtrajfittc@BeginModule 23-Nov-2024 00:13:32 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:32 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:33 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:33 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:33 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:33 CET run: 74521450 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:13:33 CET run: 74521450 subRun: 1 event: 59302
1st wire C:0 T:22 P:2 W:783 does not exist (max wire number: 480)
%MSG
23-Nov-2024 00:13:33 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b3/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_593_20231205T053827Z_gen_g4_detsim_hitreco__20240509T221831Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00123311 10.9962 21.9912 10.9962 10.995 2
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00123311 0.00139966 0.00156621 0.00139966 0.000166553 2
reco:linecluster:LineCluster 0.0623379 0.0645103 0.0666826 0.0645103 0.00217234 2
reco:trajcluster:TrajCluster 0.0783579 0.352383 0.626408 0.352383 0.274025 2
reco:pandora:StandardPandora 2.59163 2.93127 3.27091 2.93127 0.339638 2
reco:pandoraTrack:LArPandoraTrackCreation 0.00937823 0.0125571 0.015736 0.0125571 0.00317887 2
reco:pandoraShower:LArPandoraModularShowerCreation 0.0220168 0.0928216 0.163626 0.0928216 0.0708048 2
reco:pandoracalo:Calorimetry 0.00731933 0.0303319 0.0533445 0.0303319 0.0230126 2
reco:pandorapid:Chi2ParticleID 0.000236586 0.000942247 0.00164791 0.000942247 0.00070566 2
reco:trkshowersplit:TrackShowerHits 0.00515071 0.0127081 0.0202656 0.0127081 0.00755743 2
reco:pmtrack:PMAlgTrackMaker 0.450358 3.91282 7.37529 3.91282 3.46247 2
reco:pmtrackcalo:Calorimetry 0.0043397 0.0171674 0.0299951 0.0171674 0.0128277 2
reco:pmtrackpid:Chi2ParticleID 0.000331294 0.000358568 0.000385842 0.000358568 2.7274e-05 2
reco:pmtrajfit:PMAlgTrajFitter 0.880887 1.39671 1.91252 1.39671 0.515818 2
reco:pmtrajfitcalo:Calorimetry 0.0062481 0.0301195 0.053991 0.0301195 0.0238714 2
reco:pmtrajfitpid:Chi2ParticleID 0.000257523 0.000298268 0.000339014 0.000298268 4.07455e-05 2
reco:pmtracktc:PMAlgTrackMaker 0.71128 3.92956 7.14785 3.92956 3.21828 2
reco:pmtracktccalo:Calorimetry 0.00635209 0.0303969 0.0544418 0.0303969 0.0240448 2
reco:pmtracktcpid:Chi2ParticleID 0.000305794 0.000386886 0.000467977 0.000386886 8.10915e-05 2
reco:pmtrajfittc:PMAlgTrajFitter 1.05752 1.93929 2.82106 1.93929 0.881768 2
reco:pmtrajfittccalo:Calorimetry 0.00633685 0.0171038 0.0278707 0.0171038 0.0107669 2
reco:pmtrajfittcpid:Chi2ParticleID 0.000307352 0.000389469 0.000471585 0.000389469 8.21165e-05 2
reco:blurredcluster:BlurredClustering 1.09577 1.09577 1.09577 1.09577 0 1
reco:emtrkmichelid:EmTrackMichelId 10.0076 10.0076 10.0076 10.0076 0 1
reco:emshower:EMShower 0.281791 0.281791 0.281791 0.281791 0 1
reco:cvnmap:CVNMapper 0.0150417 0.0150417 0.0150417 0.0150417 0 1
reco:cvneva:CVNEvaluator 1.20656 1.20656 1.20656 1.20656 0 1
reco:energyrecnumu:EnergyReco 0.018826 0.018826 0.018826 0.018826 0 1
reco:energyrecnue:EnergyReco 0.00196907 0.00196907 0.00196907 0.00196907 0 1
reco:energyrecnc:EnergyReco 0.00496255 0.00496255 0.00496255 0.00496255 0 1
reco:energyrecnumurange:EnergyReco 0.0119008 0.0119008 0.0119008 0.0119008 0 1
reco:energyrecnumumcs:EnergyReco 0.0149711 0.0149711 0.0149711 0.0149711 0 1
reco:opdec:Deconvolution 0.263318 0.263318 0.263318 0.263318 0 1
reco:ophitspe:OpHitFinderDeco 2.13183 2.13183 2.13183 2.13183 0 1
reco:opflash:OpFlashFinder 0.00471975 0.00471975 0.00471975 0.00471975 0 1
reco:rns:RandomNumberSaver 0.000288996 0.000288996 0.000288996 0.000288996 0 1
reco:anglereconue:NuAngularReco 0.00229989 0.00229989 0.00229989 0.00229989 0 1
reco:anglereconumu:NuAngularReco 0.00134773 0.00134773 0.00134773 0.00134773 0 1
reco:anglereconuepfps:NuAngularReco 0.00376304 0.00376304 0.00376304 0.00376304 0 1
reco:anglereconumupfps:NuAngularReco 0.00145951 0.00145951 0.00145951 0.00145951 0 1
[art]:TriggerResults:TriggerResultInserter 5.0564e-05 5.0564e-05 5.0564e-05 5.0564e-05 0 1
end_path:out1:RootOutput 1.4744e-05 1.4744e-05 1.4744e-05 1.4744e-05 0 1
end_path:out1:RootOutput(write) 0.336689 0.336689 0.336689 0.336689 0 1
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6395.9 MB
Peak resident set size usage (VmHWM): 1647.67 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:13:34 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74521450 subRun: 1 event: 59302
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-d736-1922-877f-f37b.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_593_20231205T053827Z_gen_g4_detsim_hitreco__20240509T221831Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_593_20231205T053827Z_gen_g4_detsim_hitreco__20240509T221831Z_reco2_reco_2024-11-22T_231133Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 