justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301621.26@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301621.26@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 01:22:49
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat2024-11-23 02:57:54
From worker nodeHostnameiut2-c299.iu.edu
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 01:26:42
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2.root
JobscriptExit code0
Real time1h (5453s)
CPU time1h (5153s = 94%)
Outputting started2024-11-23 02:57:35
Output files
Finished2024-11-23 02:57:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rajfittc@BeginModule  22-Nov-2024 21:57:19 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:19 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:19 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:19 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:19 EST run: 6432497 subRun: 1 event: 31500
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:57:26 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:57:26 EST run: 6432497 subRun: 1 event: 31500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 128, 166, 154
minwire 0: 823
minwire 1: 1793
minwire 2: 592
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.438472, 0.558136, 0.00339259, 
Output 1: 0.895987, 0.102674, 0.00123597, 0.00010241, 
Output 2: 0.999186, 0.000813719, 

437
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1349!
22-Nov-2024 21:57:34 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2_reco_data_2024-11-23T_012648Z.root"
22-Nov-2024 21:57:34 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/0f/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.57196       51.0106       800.01        17.8396       118.137        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0274787     0.085656      0.184371      0.0820052     0.0321963       100    
reco:linecluster:LineCluster                           0.243785      0.501289       8.93774      0.298348       1.0687         100    
reco:trajcluster:TrajCluster                           0.0781228      1.82007       85.8831      0.0994259      9.87434        100    
reco:pandora:StandardPandora                            1.58062       3.95587       81.9683       2.14278       10.0063        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000219048    0.0139845     0.296311      0.0031257     0.0413504       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000323805    0.0313998     0.711241     0.00705305     0.0805567       100    
reco:pandoracalo:Calorimetry                          0.000185683    0.0333182      1.02899     0.00262316      0.13958        100    
reco:pandorapid:Chi2ParticleID                        6.2328e-05    0.000312719   0.00312364    0.000229496   0.000357919      100    
reco:trkshowersplit:TrackShowerHits                   0.000174294    0.0542662      2.00829     0.00182644     0.249469        100    
reco:pmtrack:PMAlgTrackMaker                          0.00210138      4.83044       128.204      0.587694       17.4899        100    
reco:pmtrackcalo:Calorimetry                          0.000136974    0.0253148     0.795962     0.00187634     0.106455        100    
reco:pmtrackpid:Chi2ParticleID                        4.9311e-05    0.000555588   0.00743948    0.000249388   0.00111525       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000373247     3.98769       58.9145      0.619916       10.8576        100    
reco:pmtrajfitcalo:Calorimetry                        0.000113635    0.0361156      1.02201     0.00323878     0.146205        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.3392e-05    0.000349537   0.00236473    0.000262166   0.000353242      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00200544      6.25128       170.355      0.576874       23.5209        100    
reco:pmtracktccalo:Calorimetry                        0.000118518    0.0171362     0.427316     0.00204196     0.057297        100    
reco:pmtracktcpid:Chi2ParticleID                      4.8759e-05    0.000500441   0.00506322    0.000255001   0.000787249      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000372498     3.62104       79.0571      0.662525       10.3786        100    
reco:pmtrajfittccalo:Calorimetry                       9.549e-05     0.0339398      1.15554     0.00260124      0.14813        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.9839e-05    0.000648558   0.00864499    0.000295469   0.00125426       100    
reco:blurredcluster:BlurredClustering                 0.00772055      1.68751       10.8756       0.86528       2.2647         100    
reco:emtrkmichelid:EmTrackMichelId                     0.409677       18.1764       211.334       5.20369       35.2288        100    
reco:emshower:EMShower                                0.000387697    0.0688399      1.5483      0.00588159     0.187641        100    
reco:cvnmap:CVNMapper                                 3.1371e-05     0.0255255     0.0886174      0.02975      0.0192863       100    
reco:cvneva:CVNEvaluator                              2.3453e-05      1.07568       2.58598       1.43106      0.675273        100    
reco:energyrecnumu:EnergyReco                         0.000448136    0.0739316      1.77661     0.00828784     0.233416        100    
reco:energyrecnue:EnergyReco                          0.00022303    0.00471952     0.105442     0.00109253     0.0144022       100    
reco:energyrecnc:EnergyReco                           0.00318671    0.00876768     0.111209     0.00565155     0.0147286       100    
reco:energyrecnumurange:EnergyReco                    0.00029626     0.0691944      1.64869     0.00508299     0.223398        100    
reco:energyrecnumumcs:EnergyReco                      0.00151188     0.0712401      1.64871      0.0066924      0.22201        100    
reco:opdec:Deconvolution                               0.123496      0.530924       1.23221      0.498285      0.230482        100    
reco:ophitspe:OpHitFinderDeco                          0.428675       3.53007       7.56987       3.36895       1.66654        100    
reco:opflash:OpFlashFinder                            0.000496792   0.00237947     0.0102187    0.00226486    0.00124195       100    
reco:rns:RandomNumberSaver                             2.678e-05    6.32777e-05   0.000718628   5.61015e-05   6.6903e-05       100    
reco:anglereconue:NuAngularReco                       0.00021595    0.00359155     0.094238      0.000772      0.0120448       100    
reco:anglereconumu:NuAngularReco                      0.000106296   0.00324744     0.0912286    0.000493493    0.0120524       100    
reco:anglereconuepfps:NuAngularReco                   0.000190126   0.00354667      0.09312     0.000691591    0.0124399       100    
reco:anglereconumupfps:NuAngularReco                  0.000175126   0.00345054     0.0922853    0.000619085    0.0124556       100    
[art]:TriggerResults:TriggerResultInserter            1.4185e-05    2.08589e-05   0.000114847   1.88615e-05   1.02547e-05      100    
end_path:out1:RootOutput                               3.838e-06    7.89193e-06   3.8459e-05     7.489e-06    3.48968e-06      100    
end_path:out1:RootOutput(write)                        0.0298976     0.367303       1.79598      0.350067       0.29134        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 15060.5 MB
  Peak resident set size usage (VmHWM): 2326 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2_reco_2024-11-23T_012648Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_314_20231206T032932Z_gen_g4_detsim_hitreco__20240510T035710Z_reco2_reco_data_2024-11-23T_012648Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:39:05 UTC       justIN version: 01.01.09