justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301669.41@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301669.41@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:17:34
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 04:29:37
From worker nodeHostnamecomp19-18
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:19:23
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2.root
JobscriptExit code0
Real time1h (4202s)
CPU time1h (3831s = 91%)
Outputting started2024-11-23 04:29:25
Output files
Finished2024-11-23 04:29:37
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

4 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:29:18 GMT run: 74257901 subRun: 1 event: 300
1st wire C:0 T:11 P:2 W:509 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 135, 72, 60
minwire 0: 1048
minwire 1: 1543
minwire 2: 1077
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0601035, 0.00401344, 0.935883, 
Output 1: 0.00537474, 0.0495641, 0.225847, 0.719215, 
Output 2: 0.971464, 0.0285363, 

488
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1539!
23-Nov-2024 04:29:24 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2_reco_data_2024-11-23T_031926Z.root"
23-Nov-2024 04:29:24 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/4b/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.47183       38.7537       1782.9        11.3055       177.24         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                  0.03398      0.0890177     0.334852      0.0620051     0.0557131       100    
reco:linecluster:LineCluster                           0.132857       3.51125       315.104      0.282197       31.3175        100    
reco:trajcluster:TrajCluster                           0.0342524      4.0879        367.605      0.096572       36.5597        100    
reco:pandora:StandardPandora                            1.09707       2.62798       109.573       1.26443       10.7819        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000128518   0.00704571     0.284106     0.00181495     0.0286796       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000170845   0.00863031     0.107542     0.00285634     0.0167008       100    
reco:pandoracalo:Calorimetry                          9.8263e-05     0.0376945      3.01475     0.00116275      0.29982        100    
reco:pandorapid:Chi2ParticleID                        3.5904e-05    0.000141879   0.00172834    0.000107649   0.000183144      100    
reco:trkshowersplit:TrackShowerHits                   0.000121685    0.0398258      3.06691     0.000931619    0.305706        100    
reco:pmtrack:PMAlgTrackMaker                          0.00117285      3.69956       265.087      0.271975       26.3574        100    
reco:pmtrackcalo:Calorimetry                          7.4811e-05     0.0175359      1.28938     0.000993172    0.128271        100    
reco:pmtrackpid:Chi2ParticleID                        2.9511e-05    0.000258669   0.00509252    0.000131913   0.000533107      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000188678     2.79785       165.497      0.296221       16.5206        100    
reco:pmtrajfitcalo:Calorimetry                        5.9996e-05     0.0382558      3.01035     0.00187639     0.299353        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6443e-05    0.000161919   0.00113016    0.000141725   0.000135583      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102812      3.24707       195.42       0.302542       19.5392        100    
reco:pmtracktccalo:Calorimetry                        7.1386e-05    0.00661283     0.270581     0.00116299     0.0275179       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8458e-05    0.000231199   0.00260286    0.000143287   0.00030616       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000190094     1.38984       54.6225      0.372943       5.51571        100    
reco:pmtrajfittccalo:Calorimetry                      6.4192e-05     0.0199868      1.29504     0.00134048     0.129074        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6139e-05    0.000310478   0.00452661    0.00015426    0.000519355      100    
reco:blurredcluster:BlurredClustering                 0.00420326      1.08573       10.0904      0.593736       1.49906        100    
reco:emtrkmichelid:EmTrackMichelId                     0.262703       12.3221       276.402       4.2916        29.954         100    
reco:emshower:EMShower                                0.000242606    0.0248816     0.317021     0.00734962     0.0425482       100    
reco:cvnmap:CVNMapper                                 2.1869e-05     0.0141567     0.0595095     0.0134131     0.0105077       100    
reco:cvneva:CVNEvaluator                              1.5635e-05     0.527625      0.758536      0.653253      0.282217        100    
reco:energyrecnumu:EnergyReco                         0.000291874    0.0361618      1.13832     0.00441672      0.12998        100    
reco:energyrecnue:EnergyReco                          0.000142621   0.00228696     0.0931156    0.000589022   0.00938106       100    
reco:energyrecnc:EnergyReco                           0.00189448    0.00403967     0.0940726    0.00233265    0.00932359       100    
reco:energyrecnumurange:EnergyReco                    0.000211209    0.034211       1.1541      0.00210541     0.131186        100    
reco:energyrecnumumcs:EnergyReco                      0.000716531    0.035304       1.13945     0.00287434     0.129745        100    
reco:opdec:Deconvolution                               0.116803      0.431242       1.06077       0.40916      0.181511        100    
reco:ophitspe:OpHitFinderDeco                          0.432986       2.37609       4.23467       2.31753      0.879895        100    
reco:opflash:OpFlashFinder                            0.000334372   0.00132151     0.0048731    0.00123795    0.000597284      100    
reco:rns:RandomNumberSaver                            1.8467e-05    3.0172e-05    0.000284301   2.5552e-05    2.66479e-05      100    
reco:anglereconue:NuAngularReco                       0.000140411   0.00180972     0.0805801    0.000401411   0.00812533       100    
reco:anglereconumu:NuAngularReco                      6.9517e-05    0.00161736     0.0801973    0.000225973   0.00808407       100    
reco:anglereconuepfps:NuAngularReco                   0.000108702   0.00172239     0.0794203    0.000340265   0.00801271       100    
reco:anglereconumupfps:NuAngularReco                  9.5233e-05     0.0016846     0.0801651    0.000318268   0.00807521       100    
[art]:TriggerResults:TriggerResultInserter             8.509e-06    1.1103e-05    6.4962e-05    1.0305e-05    5.54406e-06      100    
end_path:out1:RootOutput                               2.626e-06    5.32983e-06   2.4826e-05     3.797e-06    3.29618e-06      100    
end_path:out1:RootOutput(write)                        0.0188015     0.222876       1.39409      0.219851      0.161452        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9477.3 MB
  Peak resident set size usage (VmHWM): 2399.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2_reco_2024-11-23T_031926Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_2_20231122T052821Z_gen_g4_detsim_hitreco__20240503T070126Z_reco2_reco_data_2024-11-23T_031926Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:32:50 UTC       justIN version: 01.01.09