Jobsub ID 301554.27@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 301554.27@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 4213 |
| Stage ID | 1 |
| User name | ismerio@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-22 23:43:32 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2024-11-23 00:03:20 |
| From worker node | Hostname | wn-choc-037.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| Inner Apptainer? | True |
| Job state | finished |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-11-22 23:55:19 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_358_20231205T052324Z_gen_g4_detsim_hitreco__20240509T222925Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 2m (137s) |
| CPU time | 2m (130s = 94%) |
| Outputting started | 2024-11-22 23:57:37 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_358_20231205T052324Z_gen_g4_detsim_hitreco__20240509T222925Z_reco2.root
|
| Finished | 2024-11-23 00:03:20 |
| Saved logs | justin-logs:301554.27-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
mtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
1st wire C:0 T:20 P:2 W:508 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck: PMAlgTrajFitter:pmtrajfittc@BeginModule 23-Nov-2024 00:57:36 CET run: 6430442 subRun: 1 event: 35803
1st wire C:0 T:20 P:2 W:489 does not exist (max wire number: 480)
%MSG
23-Nov-2024 00:57:37 CET Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/73/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_358_20231205T052324Z_gen_g4_detsim_hitreco__20240509T222925Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.00153323 14.773 39.0473 5.27019 17.2988 3
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000882762 0.0063715 0.0166985 0.00153323 0.00730712 3
reco:linecluster:LineCluster 0.0215498 0.0488048 0.0722603 0.0526042 0.0208761 3
reco:trajcluster:TrajCluster 0.0145103 0.140721 0.210028 0.197625 0.0893879 3
reco:pandora:StandardPandora 2.18826 2.65683 3.12447 2.65777 0.382206 3
reco:pandoraTrack:LArPandoraTrackCreation 0.00449543 0.00914124 0.0145034 0.00842486 0.00411703 3
reco:pandoraShower:LArPandoraModularShowerCreation 0.0141089 0.031319 0.0632625 0.0165855 0.0226101 3
reco:pandoracalo:Calorimetry 0.00344642 0.00883318 0.0123372 0.0107159 0.00386609 3
reco:pandorapid:Chi2ParticleID 0.000235094 0.000636354 0.00136371 0.000310261 0.00051523 3
reco:trkshowersplit:TrackShowerHits 0.00189154 0.00899286 0.0138211 0.0112659 0.0051286 3
reco:pmtrack:PMAlgTrackMaker 0.0142303 1.44247 2.8992 1.41397 1.17796 3
reco:pmtrackcalo:Calorimetry 0.000151039 0.00780939 0.013264 0.0100131 0.00557554 3
reco:pmtrackpid:Chi2ParticleID 5.0575e-05 0.000292549 0.000494844 0.000332228 0.000183529 3
reco:pmtrajfit:PMAlgTrajFitter 0.11784 1.88722 3.37301 2.17082 1.34396 3
reco:pmtrajfitcalo:Calorimetry 0.000461675 0.0096663 0.0142927 0.0142445 0.00650868 3
reco:pmtrajfitpid:Chi2ParticleID 0.000107564 0.00026743 0.000399567 0.000295159 0.000120811 3
reco:pmtracktc:PMAlgTrackMaker 0.0230526 1.53219 2.82832 1.74521 1.15511 3
reco:pmtracktccalo:Calorimetry 0.000282229 0.00777472 0.0125889 0.0104531 0.00536927 3
reco:pmtracktcpid:Chi2ParticleID 8.4281e-05 0.000318617 0.000451122 0.000420448 0.000166173 3
reco:pmtrajfittc:PMAlgTrajFitter 0.184282 1.67076 2.52145 2.30653 1.05475 3
reco:pmtrajfittccalo:Calorimetry 0.000372541 0.0142231 0.0311395 0.0111573 0.0127462 3
reco:pmtrajfittcpid:Chi2ParticleID 9.336e-05 0.000760603 0.00177639 0.000412061 0.000729957 3
reco:blurredcluster:BlurredClustering 0.0114284 1.24963 2.48784 1.24963 1.2382 2
reco:emtrkmichelid:EmTrackMichelId 0.613173 10.4948 20.3764 10.4948 9.8816 2
reco:emshower:EMShower 0.00477503 0.0374423 0.0701095 0.0374423 0.0326672 2
reco:cvnmap:CVNMapper 0.000246445 0.0183472 0.036448 0.0183472 0.0181008 2
reco:cvneva:CVNEvaluator 0.000238558 0.64592 1.2916 0.64592 0.645682 2
reco:energyrecnumu:EnergyReco 0.00771479 0.0683919 0.129069 0.0683919 0.0606771 2
reco:energyrecnue:EnergyReco 0.000845897 0.00173584 0.00262579 0.00173584 0.000889945 2
reco:energyrecnc:EnergyReco 0.00284926 0.00391154 0.00497383 0.00391154 0.00106229 2
reco:energyrecnumurange:EnergyReco 0.000700131 0.0613291 0.121958 0.0613291 0.0606289 2
reco:energyrecnumumcs:EnergyReco 0.00294431 0.0625603 0.122176 0.0625603 0.059616 2
reco:opdec:Deconvolution 0.11502 0.171764 0.228508 0.171764 0.0567439 2
reco:ophitspe:OpHitFinderDeco 0.890885 1.28642 1.68195 1.28642 0.395534 2
reco:opflash:OpFlashFinder 0.0016863 0.00276095 0.00383561 0.00276095 0.00107465 2
reco:rns:RandomNumberSaver 3.3976e-05 0.000178504 0.000323033 0.000178504 0.000144529 2
reco:anglereconue:NuAngularReco 0.00121532 0.00146508 0.00171485 0.00146508 0.000249762 2
reco:anglereconumu:NuAngularReco 0.000279177 0.000919123 0.00155907 0.000919123 0.000639946 2
reco:anglereconuepfps:NuAngularReco 0.0018263 0.00221421 0.00260212 0.00221421 0.000387912 2
reco:anglereconumupfps:NuAngularReco 0.000319454 0.000991994 0.00166453 0.000991994 0.00067254 2
[art]:TriggerResults:TriggerResultInserter 1.3368e-05 3.08025e-05 4.8237e-05 3.08025e-05 1.74345e-05 2
end_path:out1:RootOutput 6.352e-06 1.15195e-05 1.6687e-05 1.15195e-05 5.1675e-06 2
end_path:out1:RootOutput(write) 0.0650393 0.222423 0.379808 0.222423 0.157384 2
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6419.83 MB
Peak resident set size usage (VmHWM): 1672.87 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Nov-2024 00:57:37 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6430442 subRun: 1 event: 35803
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-68d6-df9c-aeaa-50ba.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_358_20231205T052324Z_gen_g4_detsim_hitreco__20240509T222925Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_358_20231205T052324Z_gen_g4_detsim_hitreco__20240509T222925Z_reco2_reco_2024-11-22T_235523Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work 