justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301835.3@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301835.3@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 06:09:05
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 07:13:16
From worker nodeHostnamecomp21-10
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 06:10:19
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2.root
JobscriptExit code0
Real time1h (3761s)
CPU time49m (2957s = 78%)
Outputting started2024-11-23 07:13:01
Output files
Finished2024-11-23 07:13:16
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:12:53 GMT run: 74511302 subRun: 1 event: 20400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 67, 79, 70
minwire 0: 2212
minwire 1: 62
minwire 2: 2634
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.474524, 0.391033, 0.134444, 
Output 1: 0.142356, 0.355959, 0.383443, 0.118241, 
Output 2: 0.771582, 0.228418, 

282
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 737!
23-Nov-2024 07:12:59 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2_reco_data_2024-11-23T_061023Z.root"
23-Nov-2024 07:12:59 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/9f/06/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.974847       30.0466       556.86        9.46745       67.8877        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0288474     0.063738      0.366875      0.0655546     0.0354128       100    
reco:linecluster:LineCluster                           0.162403      0.649543       15.5235      0.345554       1.57321        100    
reco:trajcluster:TrajCluster                           0.0817876      1.79009       88.6588      0.102497       9.2022         100    
reco:pandora:StandardPandora                           0.0997063      1.90484       32.1382       1.11568       3.47011        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124469   0.00708573     0.180847     0.00134664     0.0215642       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167385    0.0100472     0.233456     0.00201553     0.028298        100    
reco:pandoracalo:Calorimetry                          9.1868e-05     0.0222156     0.948399     0.000691947    0.0994256       100    
reco:pandorapid:Chi2ParticleID                        3.4736e-05    0.000145475   0.00188231    9.9755e-05    0.000202463      100    
reco:trkshowersplit:TrackShowerHits                   9.3408e-05     0.0247816     0.774813     0.000514402    0.0929613       100    
reco:pmtrack:PMAlgTrackMaker                          0.00113536      2.23562       58.2964      0.150526       6.96682        100    
reco:pmtrackcalo:Calorimetry                          7.2837e-05     0.0146677      0.60768     0.000669473    0.0657172       100    
reco:pmtrackpid:Chi2ParticleID                        2.8528e-05    0.000263138   0.00322365    0.000122923   0.000463237      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000175701     1.85615       38.7091       0.25863       5.34469        100    
reco:pmtrajfitcalo:Calorimetry                        6.1959e-05     0.0236913      1.00007     0.00110251     0.104721        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6025e-05    0.000159099   0.000814701   0.000128612   0.000140975      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000980579     2.93017       79.8275      0.142632       9.67841        100    
reco:pmtracktccalo:Calorimetry                        6.7029e-05    0.00880071     0.233848     0.000816836    0.029293        100    
reco:pmtracktcpid:Chi2ParticleID                      2.8157e-05    0.000233989   0.00216547    0.000130095   0.000333746      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000165654     1.40177       31.4837      0.218702       3.82281        100    
reco:pmtrajfittccalo:Calorimetry                      6.0901e-05     0.0192096     0.640198     0.00100677     0.0730062       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4407e-05    0.000332302   0.00331971    0.000148984   0.000543826      100    
reco:blurredcluster:BlurredClustering                 7.5237e-05      1.06139       7.69498      0.539272       1.5106         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000125641     12.4155       180.358       2.71995       26.1733        100    
reco:emshower:EMShower                                0.000190392    0.0248857     0.245711     0.00738879     0.043431        100    
reco:cvnmap:CVNMapper                                 2.0266e-05     0.0118255     0.0443597     0.0120413    0.00955409       100    
reco:cvneva:CVNEvaluator                              1.4868e-05      0.52269       1.51699      0.709237      0.354183        100    
reco:energyrecnumu:EnergyReco                         0.000273687    0.0650049      4.1896      0.00509622     0.419337        100    
reco:energyrecnue:EnergyReco                          0.000133923   0.00243954     0.0672921    0.00049814    0.00763104       100    
reco:energyrecnc:EnergyReco                           0.00166225    0.00405203     0.0714278    0.00207104    0.00790871       100    
reco:energyrecnumurange:EnergyReco                    0.000190251    0.0595161      3.98092     0.00189929     0.398621        100    
reco:energyrecnumumcs:EnergyReco                      0.00165259     0.0605415      3.97712     0.00230367      0.39811        100    
reco:opdec:Deconvolution                               0.0940504     0.442311      0.833406      0.436928      0.151144        100    
reco:ophitspe:OpHitFinderDeco                          0.315628       2.20031       3.80973       2.19461       0.83006        100    
reco:opflash:OpFlashFinder                            0.000191371   0.00123769    0.00435364    0.00117184    0.000583747      100    
reco:rns:RandomNumberSaver                            1.8298e-05    2.85834e-05   0.000254799   2.3301e-05    2.39416e-05      100    
reco:anglereconue:NuAngularReco                       0.00014151    0.00193828     0.0595114    0.000289632   0.00674358       100    
reco:anglereconumu:NuAngularReco                      6.3514e-05    0.00169925     0.057897     0.000169428    0.0065509       100    
reco:anglereconuepfps:NuAngularReco                   0.000113013   0.00182234     0.0588889    0.000259356   0.00666598       100    
reco:anglereconumupfps:NuAngularReco                  9.6978e-05    0.00176489     0.0576471    0.00022605     0.0065444       100    
[art]:TriggerResults:TriggerResultInserter             8.587e-06    1.09109e-05   4.1666e-05    9.9455e-06    3.44838e-06      100    
end_path:out1:RootOutput                               2.594e-06    5.02175e-06   1.9602e-05    3.7035e-06    2.66448e-06      100    
end_path:out1:RootOutput(write)                       0.00787955      0.20249       1.0592       0.197784      0.159593        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9633.81 MB
  Peak resident set size usage (VmHWM): 2477.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2_reco_2024-11-23T_061023Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2_reco_data_2024-11-23T_061023Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:52:48 UTC       justIN version: 01.01.09