Jobsub ID 108519.109@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 108519.109@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 22:43:08 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-22 22:55:21 |
| From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 22:52:29 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_307_20231208T062856Z_gen_g4_detsim_hitreco__20240510T053536Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (149s) |
| CPU time | 2m (143s = 95%) |
| Outputting started | 2024-11-22 22:55:00 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_307_20231208T062856Z_gen_g4_detsim_hitreco__20240510T053536Z_reco2.root
|
| Finished | 2024-11-22 22:55:21 |
| Saved logs | justin-logs:108519.109-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 22-Nov-2024 23:54:58 CET run: 74577407 subRun: 1 event: 30710
Flip, endpoint closer to vStart.
%MSG
22-Nov-2024 23:54:59 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/3a/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_307_20231208T062856Z_gen_g4_detsim_hitreco__20240510T053536Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.000943849 7.07958 10.5409 8.00009 3.38347 10
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000921221 0.00163654 0.00621993 0.00109648 0.00153905 10
reco:linecluster:LineCluster 0.0108215 0.0181539 0.0430608 0.0154199 0.00926927 10
reco:trajcluster:TrajCluster 0.00172729 0.00961671 0.023897 0.00863262 0.0071269 10
reco:pandora:StandardPandora 1.33663 1.6522 3.17048 1.40604 0.546514 10
reco:pandoraTrack:LArPandoraTrackCreation 0.000191616 0.00137558 0.00546288 0.000842209 0.00150076 10
reco:pandoraShower:LArPandoraModularShowerCreation 0.00046847 0.00379792 0.0146433 0.00161208 0.00433298 10
reco:pandoracalo:Calorimetry 0.000128732 0.0010167 0.00424783 0.000569535 0.00118047 10
reco:pandorapid:Chi2ParticleID 5.1443e-05 0.000255977 0.00159495 0.000113903 0.000447637 10
reco:trkshowersplit:TrackShowerHits 0.000181589 0.000642951 0.00249782 0.000487304 0.000648959 10
reco:pmtrack:PMAlgTrackMaker 0.00159072 0.167869 0.563058 0.0898152 0.204692 10
reco:pmtrackcalo:Calorimetry 9.7657e-05 0.000676496 0.00160758 0.00064231 0.000496195 10
reco:pmtrackpid:Chi2ParticleID 4.0558e-05 0.000118526 0.00017096 0.000130697 4.78556e-05 10
reco:pmtrajfit:PMAlgTrajFitter 0.0149743 0.175368 0.524014 0.128155 0.16442 10
reco:pmtrajfitcalo:Calorimetry 0.000273621 0.00087199 0.00158812 0.000726016 0.000570875 10
reco:pmtrajfitpid:Chi2ParticleID 8.3255e-05 0.000123586 0.000194552 0.000122324 3.67485e-05 10
reco:pmtracktc:PMAlgTrackMaker 0.00170604 0.161421 0.381411 0.147974 0.145036 10
reco:pmtracktccalo:Calorimetry 0.000103286 0.0008121 0.00160204 0.000861317 0.000488192 10
reco:pmtracktcpid:Chi2ParticleID 4.5711e-05 0.000132824 0.000185427 0.00014991 4.54698e-05 10
reco:pmtrajfittc:PMAlgTrajFitter 0.00406513 0.257868 0.733871 0.215018 0.21857 10
reco:pmtrajfittccalo:Calorimetry 0.000196034 0.000848621 0.00136333 0.00102863 0.000477723 10
reco:pmtrajfittcpid:Chi2ParticleID 6.5995e-05 0.000134486 0.000190109 0.000136665 4.55375e-05 10
reco:blurredcluster:BlurredClustering 0.0719545 0.536136 1.39871 0.334751 0.414522 9
reco:emtrkmichelid:EmTrackMichelId 0.594726 2.27313 3.61371 2.71553 1.14626 9
reco:emshower:EMShower 0.000705936 0.0177287 0.0990962 0.00327603 0.0301584 9
reco:cvnmap:CVNMapper 5.1724e-05 0.00955838 0.0186581 0.010685 0.00532463 9
reco:cvneva:CVNEvaluator 2.5383e-05 0.905964 1.20327 1.15801 0.484463 9
reco:energyrecnumu:EnergyReco 0.000365267 0.00499028 0.0163002 0.00296615 0.00459403 9
reco:energyrecnue:EnergyReco 0.000188775 0.000399829 0.000717125 0.000288999 0.000197641 9
reco:energyrecnc:EnergyReco 0.00252795 0.0028073 0.0032737 0.00276969 0.000216861 9
reco:energyrecnumurange:EnergyReco 0.00028755 0.00298479 0.0117101 0.0011672 0.00356128 9
reco:energyrecnumumcs:EnergyReco 0.00253849 0.00465853 0.0118222 0.0032778 0.00311858 9
reco:opdec:Deconvolution 0.105355 0.169977 0.24566 0.181324 0.0475932 9
reco:ophitspe:OpHitFinderDeco 0.723983 1.24935 1.86898 1.21966 0.390156 9
reco:opflash:OpFlashFinder 0.00071721 0.00148887 0.0041211 0.00122607 0.000993006 9
reco:rns:RandomNumberSaver 2.0333e-05 5.47622e-05 0.000294216 2.56e-05 8.47304e-05 9
reco:anglereconue:NuAngularReco 0.000182104 0.000398989 0.00136786 0.000238247 0.000358368 9
reco:anglereconumu:NuAngularReco 0.000113095 0.000227703 0.000484474 0.000158487 0.000128189 9
reco:anglereconuepfps:NuAngularReco 0.000179068 0.000600083 0.00283273 0.000270442 0.000806919 9
reco:anglereconumupfps:NuAngularReco 0.00016166 0.000302273 0.000531041 0.000253869 0.000132547 9
[art]:TriggerResults:TriggerResultInserter 1.21e-05 1.71121e-05 4.7146e-05 1.3148e-05 1.06943e-05 9
end_path:out1:RootOutput 2.638e-06 5.37244e-06 1.8223e-05 3.741e-06 4.6659e-06 9
end_path:out1:RootOutput(write) 0.053771 0.213011 0.294108 0.24464 0.0850055 9
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6415.53 MB
Peak resident set size usage (VmHWM): 1635.16 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 22-Nov-2024 23:54:59 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74577407 subRun: 1 event: 30710
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 9 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-9df2-933d-9a41-8d52.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_307_20231208T062856Z_gen_g4_detsim_hitreco__20240510T053536Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_307_20231208T062856Z_gen_g4_detsim_hitreco__20240510T053536Z_reco2_reco_2024-11-22T_225233Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 