Jobsub ID 78438.2@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 78438.2@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3454 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:38:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2024-09-26 11:00:50 |
| From worker node | Hostname | ccwcondor0582 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 09:49:42 |
| Input files | hd-protodune:np04hd_raw_run029109_0014_dataflow4_datawriter_0_20240909T164052.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 11:00:50 |
| Saved logs | justin-logs:78438.2-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
:2 22094 XUs and 31656 XVs -> 905 XUVs
C:0 T:3 148 XUs and 188 XVs -> 9 XUVs
C:0 T:4 301 XUs and 335 XVs -> 21 XUVs
C:0 T:5 29423 XUs and 43982 XVs -> 1620 XUVs
C:0 T:6 36303 XUs and 52565 XVs -> 5082 XUVs
C:0 T:7 194 XUs and 242 XVs -> 3 XUVs
7648 XUVs total
2261 collection wire objects
7648 potential space points
Neighbour search...
1601138 tests to find 464358 neighbours
Iterating with no regularization...
Begin: 1.67977e+07
0 1.34912e+07
1 1.31567e+07
2 1.31107e+07
3 1.30985e+07
Now with regularization...
Begin: 1.22088e+07
0 1.21963e+07
1 1.21921e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29109 subRun: 1 event: 3636 at 26-Sep-2024 12:48:41 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 62951 traces, 62951 summary
Retagger: tagging trace set: gauss with 44995 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 44995 traces tagged "gauss"
FrameSaver: q=1.00082e+06 n=814614 tag=gauss
wclsFrameSaver: saving 62951 traces tagged "wiener"
FrameSaver: q=1.08896e+06 n=739123 tag=wiener
1042 hits in TPC 0
2086 hits in TPC 1
4233 hits in TPC 2
2658 hits in TPC 3
4210 hits in TPC 4
7463 hits in TPC 5
7552 hits in TPC 6
2995 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 158 XUs and 107 XVs -> 0 XUVs
C:0 T:2 24518 XUs and 34771 XVs -> 1046 XUVs
C:0 T:3 705 XUs and 648 XVs -> 17 XUVs
C:0 T:4 54 XUs and 85 XVs -> 3 XUVs
C:0 T:5 46094 XUs and 78929 XVs -> 3929 XUVs
C:0 T:6 42875 XUs and 59505 XVs -> 2197 XUVs
C:0 T:7 195 XUs and 180 XVs -> 11 XUVs
7203 XUVs total
1960 collection wire objects
7203 potential space points
Neighbour search...
684813 tests to find 344508 neighbours
Iterating with no regularization...
Begin: 1.77929e+07
0 1.65425e+07
1 1.64553e+07
2 1.64443e+07
Now with regularization...
Begin: 1.55943e+07
0 1.55905e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29109 subRun: 1 event: 3644 at 26-Sep-2024 12:50:19 CEST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 46615 traces, 46615 summary
Retagger: tagging trace set: gauss with 36627 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 36627 traces tagged "gauss"
FrameSaver: q=767839 n=681608 tag=gauss
wclsFrameSaver: saving 46615 traces tagged "wiener"
FrameSaver: q=824992 n=626821 tag=wiener
3341 hits in TPC 0
2912 hits in TPC 1
4562 hits in TPC 2
3500 hits in TPC 3
2516 hits in TPC 4
4410 hits in TPC 5
1735 hits in TPC 6
1167 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 5351 XUs and 4829 XVs -> 825 XUVs
C:0 T:2 16371 XUs and 16738 XVs -> 438 XUVs
C:0 T:3 65 XUs and 77 XVs -> 0 XUVs
C:0 T:4 124 XUs and 194 XVs -> 7 XUVs
C:0 T:5 11970 XUs and 14694 XVs -> 459 XUVs
C:0 T:6 2007 XUs and 2362 XVs -> 161 XUVs
C:0 T:7 239 XUs and 171 XVs -> 10 XUVs
1900 XUVs total
834 collection wire objects
1900 potential space points
Neighbour search...
73132 tests to find 44926 neighbours
Iterating with no regularization...
Begin: 5.41721e+06
0 4.59774e+06
1 4.5772e+06
2 4.56907e+06
3 4.56725e+06
Now with regularization...
Begin: 4.33893e+06
0 4.33808e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:51:35 CEST Closed output file "np04hd_raw_run029109_0014_dataflow4_datawriter_0_20240909T164052_reco_stage1_reco_stage2_20240926T105135_keepup.root"
26-Sep-2024 12:51:35 CEST Closed input file "np04hd_raw_run029109_0014_dataflow4_datawriter_0_20240909T164052_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 49.1578 90.2391 126.53 91.0062 15.6663 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000552248 0.00307579 0.00729684 0.00265212 0.00192567 30
produce:fembfilter:PDHDFEMBFilter 0.499121 0.792361 4.23885 0.603956 0.677328 30
produce:wclsdatahd:WireCellToolkit 33.001 37.2852 44.7653 36.8454 2.71402 30
produce:gaushit:GausHitFinder 0.308001 0.733094 2.3534 0.687903 0.343296 30
produce:nhitsfilter:NumberOfHitsFilter 0.000262924 0.000363766 0.000865477 0.000343871 0.000104962 30
produce:reco3d:SpacePointSolver 0.244741 1.02145 4.02107 0.705514 0.848084 29
produce:hitpdune:DisambigFromSpacePoints 0.168865 0.464465 1.18302 0.431622 0.208696 29
produce:pandora:StandardPandora 5.14682 11.5624 21.1677 10.9241 4.10827 29
produce:pandoraWriter:StandardPandora 0.0841565 0.15879 0.288101 0.146884 0.0495289 29
produce:pandoraTrack:LArPandoraTrackCreation 0.639328 1.76487 3.83988 1.64782 0.730773 29
produce:pandoraShower:LArPandoraShowerCreation 0.819706 2.31167 3.85091 2.46556 0.910409 29
produce:pandoracalo:Calorimetry 0.349761 0.840128 1.77675 0.875998 0.292926 29
produce:pandoracalonosce:Calorimetry 0.365065 0.731001 1.40646 0.696777 0.220968 29
produce:pandorapid:Chi2ParticleID 0.00115905 0.00197405 0.00330882 0.00179756 0.000523524 29
produce:pandoraShowercalo:ShowerCalorimetry 0.559174 1.92903 3.22674 2.11535 0.741862 29
produce:pandoraShowercalonosce:ShowerCalorimetry 0.468123 1.82221 3.41785 1.84113 0.738731 29
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0711123 0.134191 0.21016 0.121777 0.0388001 29
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0273002 0.0468263 0.0776027 0.044173 0.0123119 29
produce:emtrkmichelid:EmTrackMichelId 18.2468 28.6266 43.2932 28.172 6.01094 29
[art]:TriggerResults:TriggerResultInserter 4.1297e-05 7.40106e-05 0.000183645 7.0748e-05 2.50533e-05 30
end_path:out1:RootOutput 1.0831e-05 2.10718e-05 6.2418e-05 2.04785e-05 8.48038e-06 30
end_path:michelremoving:michelremoving 6.896e-05 0.126924 0.469545 0.0979882 0.104874 30
end_path:michelremovinggnocchi:michelremoving 4.8491e-05 0.121743 0.578706 0.0818838 0.117799 30
end_path:calibanagnocchi:CalibAnaTree 5.2318e-05 0.577149 1.1805 0.577164 0.2483 30
end_path:out1:RootOutput(write) 0.4154 0.676317 1.01906 0.663597 0.139713 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4940.07 MB
Peak resident set size usage (VmHWM): 2338.75 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029109_0014_dataflow4_datawriter_0_20240909T164052_reco_stage1_reco_stage2_20240926T105135_keepup.root
\tHists: np04hd_raw_run029109_0014_dataflow4_datawriter_0_20240909T164052_reco_stage1_reco_stage2_20240926T105135_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
