Jobsub ID 301554.46@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301554.46@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:43:32 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-23 00:02:23 |
| From worker node | Hostname | wn-choc-026.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:55:54 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_236_20231207T035332Z_gen_g4_detsim_hitreco__20240510T053035Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (123s) |
| CPU time | 1m (116s = 94%) |
| Outputting started | 2024-11-22 23:57:57 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_236_20231207T035332Z_gen_g4_detsim_hitreco__20240510T053035Z_reco2.root
|
| Finished | 2024-11-23 00:02:23 |
| Saved logs | justin-logs:301554.46-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
bRun: 1 event: 23605
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
Track empty.
%MSG
%MSG-e pma::Track3D: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:56 CET run: 50607041 subRun: 1 event: 23605
TuneFullTree failed.
%MSG
23-Nov-2024 00:57:57 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/9d/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_236_20231207T035332Z_gen_g4_detsim_hitreco__20240510T053035Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00119798 8.38964 18.2808 9.01399 6.38287 5
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000967016 0.00112111 0.00119798 0.00113846 8.05767e-05 5
reco:linecluster:LineCluster 0.0107391 0.0221076 0.0347756 0.0224784 0.00853914 5
reco:trajcluster:TrajCluster 0.00171883 0.0318706 0.0564042 0.0379558 0.0218663 5
reco:pandora:StandardPandora 1.4038 2.31727 3.41271 2.15714 0.811492 5
reco:pandoraTrack:LArPandoraTrackCreation 0.000443058 0.00240199 0.00535831 0.00213539 0.0016722 5
reco:pandoraShower:LArPandoraModularShowerCreation 0.000719224 0.00500779 0.0160731 0.0023739 0.00563118 5
reco:pandoracalo:Calorimetry 0.000219271 0.00202017 0.00422715 0.00117662 0.00155685 5
reco:pandorapid:Chi2ParticleID 7.638e-05 0.000368128 0.00133536 0.000135441 0.000484783 5
reco:trkshowersplit:TrackShowerHits 0.000160835 0.00165836 0.00294236 0.00216252 0.0011441 5
reco:pmtrack:PMAlgTrackMaker 0.00310334 0.431526 1.52538 0.222168 0.553711 5
reco:pmtrackcalo:Calorimetry 0.000171439 0.000929166 0.00169475 0.000991143 0.000498155 5
reco:pmtrackpid:Chi2ParticleID 6.4301e-05 0.000148429 0.000228525 0.000148771 5.30531e-05 5
reco:pmtrajfit:PMAlgTrajFitter 0.0986621 0.391312 0.908955 0.38817 0.283828 5
reco:pmtrajfitcalo:Calorimetry 0.000292888 0.0019401 0.00425048 0.00141829 0.00132652 5
reco:pmtrajfitpid:Chi2ParticleID 8.3125e-05 0.000170086 0.000256094 0.000178696 6.15617e-05 5
reco:pmtracktc:PMAlgTrackMaker 0.0411907 0.332218 0.716035 0.256124 0.244179 5
reco:pmtracktccalo:Calorimetry 0.000229371 0.00108522 0.002238 0.000961653 0.000656824 5
reco:pmtracktcpid:Chi2ParticleID 7.6951e-05 0.000166581 0.000265829 0.000156161 6.11299e-05 5
reco:pmtrajfittc:PMAlgTrajFitter 0.0079802 0.29303 0.561794 0.371374 0.198078 5
reco:pmtrajfittccalo:Calorimetry 0.00020473 0.00131945 0.0024375 0.00119156 0.000891356 5
reco:pmtrajfittcpid:Chi2ParticleID 7.2608e-05 0.000184289 0.000273541 0.000190281 7.97756e-05 5
reco:blurredcluster:BlurredClustering 0.0433984 0.523185 0.771156 0.639093 0.288942 4
reco:emtrkmichelid:EmTrackMichelId 0.520493 3.11638 6.86903 2.53799 2.34673 4
reco:emshower:EMShower 0.000944854 0.020192 0.0605274 0.00964779 0.0238524 4
reco:cvnmap:CVNMapper 5.1198e-05 0.00965147 0.0143205 0.0121171 0.00561554 4
reco:cvneva:CVNEvaluator 2.3193e-05 0.943001 1.37077 1.20061 0.549193 4
reco:energyrecnumu:EnergyReco 0.000459572 0.0062503 0.0192626 0.00263951 0.00760864 4
reco:energyrecnue:EnergyReco 0.000197057 0.000501142 0.000707986 0.000549763 0.00020988 4
reco:energyrecnc:EnergyReco 0.00265701 0.00296563 0.00321121 0.00299715 0.000238777 4
reco:energyrecnumurange:EnergyReco 0.000306256 0.00359897 0.00991083 0.00208939 0.00379922 4
reco:energyrecnumumcs:EnergyReco 0.00270821 0.00499794 0.0103665 0.00345854 0.00311831 4
reco:opdec:Deconvolution 0.125309 0.193653 0.248572 0.200365 0.0486041 4
reco:ophitspe:OpHitFinderDeco 0.923708 1.4152 1.72729 1.50491 0.333258 4
reco:opflash:OpFlashFinder 0.000785219 0.0019548 0.00425784 0.00138807 0.00136112 4
reco:rns:RandomNumberSaver 2.0576e-05 8.2975e-05 0.000257719 2.68025e-05 0.000100921 4
reco:anglereconue:NuAngularReco 0.000192566 0.000495367 0.00113728 0.000325812 0.000374607 4
reco:anglereconumu:NuAngularReco 0.000117449 0.000251581 0.000370142 0.000259366 9.26829e-05 4
reco:anglereconuepfps:NuAngularReco 0.000210967 0.00085084 0.00238206 0.000405168 0.000888492 4
reco:anglereconumupfps:NuAngularReco 0.000193846 0.00031707 0.000421197 0.000326619 8.1092e-05 4
[art]:TriggerResults:TriggerResultInserter 1.2147e-05 2.0465e-05 4.3064e-05 1.33245e-05 1.30614e-05 4
end_path:out1:RootOutput 2.795e-06 6.72825e-06 1.5268e-05 4.425e-06 5.09125e-06 4
end_path:out1:RootOutput(write) 0.0647571 0.228084 0.295696 0.275942 0.0953836 4
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6409.22 MB
Peak resident set size usage (VmHWM): 1628.55 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:57:57 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50607041 subRun: 1 event: 23605
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-c9eb-c30d-aa5a-f2d8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_236_20231207T035332Z_gen_g4_detsim_hitreco__20240510T053035Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_236_20231207T035332Z_gen_g4_detsim_hitreco__20240510T053035Z_reco2_reco_2024-11-22T_235558Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 