Jobsub ID 301641.169@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301641.169@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:12:27
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2024-11-23 03:34:37
From worker node	Hostname	iut2-c412.iu.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:22:56
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4285s)
	CPU time	1h (3925s = 91%)
Outputting started	2024-11-23 03:34:22
Output files
Finished	2024-11-23 03:34:37
List job events Wrapper job log
Jobscript log (last 10,000 characters)

subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:33:31 EST run: 74518431 subRun: 1 event: 19200
1st wire C:0 T:21 P:2 W:525 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:33:31 EST run: 74518431 subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:33:31 EST run: 74518431 subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:33:31 EST run: 74518431 subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:33:32 EST run: 74518431 subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 22:34:13 EST run: 74518431 subRun: 1 event: 19200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1814, 1753, 1565
minwire 0: 684
minwire 1: 590
minwire 2: 399
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0908203, 0.168587, 0.740593, 
Output 1: 0.3134, 0.458444, 0.180781, 0.0473744, 
Output 2: 0.0918848, 0.908115, 

780
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3041!
22-Nov-2024 22:34:20 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2_reco_data_2024-11-23T_022300Z.root"
22-Nov-2024 22:34:20 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/3d/95/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.994977       39.5325       407.423       15.4741       71.4744        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0272356     0.088923      0.175302      0.0887156     0.0311054       100    
reco:linecluster:LineCluster                           0.239616      0.710444       19.8471      0.288591       2.18146        100    
reco:trajcluster:TrajCluster                           0.0773659      1.2516        28.7886      0.108376       4.13047        100    
reco:pandora:StandardPandora                           0.0934856      2.51423       34.1837       1.40504       4.20814        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000156864   0.00875556     0.098787     0.00268473     0.0163234       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000216136    0.0256187      1.10337     0.00466668     0.110853        100    
reco:pandoracalo:Calorimetry                          0.000131166    0.0269156     0.415339     0.00232473     0.0711251       100    
reco:pandorapid:Chi2ParticleID                         4.76e-05     0.000235362    0.0023481    0.000180279   0.000253609      100    
reco:trkshowersplit:TrackShowerHits                   0.000114715    0.0432128     0.822195      0.0020846     0.130664        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122577      4.0645        72.7365      0.460073       11.1681        100    
reco:pmtrackcalo:Calorimetry                          9.7924e-05     0.012993       0.19707     0.00204154     0.0316892       100    
reco:pmtrackpid:Chi2ParticleID                        3.5376e-05    0.000394432   0.00279194    0.000216466   0.000505846      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000245561     3.39414       66.5378      0.547248       8.73844        100    
reco:pmtrajfitcalo:Calorimetry                        7.6013e-05     0.0277373     0.427694     0.00345216     0.0731753       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1248e-05    0.000256173   0.000899768   0.000198718   0.000179868      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00116041      4.60089       73.4662      0.541918       12.1602        100    
reco:pmtracktccalo:Calorimetry                         9.59e-05      0.0101251     0.196939      0.0023176     0.0256555       100    
reco:pmtracktcpid:Chi2ParticleID                      4.1318e-05    0.000381043   0.00235922    0.000238458   0.00043548       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000220714     2.26208       26.2344      0.591709       4.68134        100    
reco:pmtrajfittccalo:Calorimetry                      8.4569e-05     0.019886      0.363659     0.00253277     0.052953        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4564e-05    0.000510818   0.00399874    0.000244389   0.000709681      100    
reco:blurredcluster:BlurredClustering                 0.000127339     1.39467       10.5089      0.609014        1.847         100    
reco:emtrkmichelid:EmTrackMichelId                     0.0319984      14.9057       118.906       5.66997       24.5526        100    
reco:emshower:EMShower                                0.000306455    0.0358665     0.460757     0.00503484     0.0675117       100    
reco:cvnmap:CVNMapper                                  3.163e-05     0.0121765     0.0390868     0.0116019    0.00965544       100    
reco:cvneva:CVNEvaluator                              2.0399e-05      0.99709       13.5573      0.957989       1.77687        100    
reco:energyrecnumu:EnergyReco                         0.000353603    0.0408314     0.633986     0.00588215     0.0894898       100    
reco:energyrecnue:EnergyReco                          0.000166362   0.00272636     0.0307231    0.000972589   0.00505186       100    
reco:energyrecnc:EnergyReco                           0.00150527    0.00532699     0.0346228    0.00357554    0.00538818       100    
reco:energyrecnumurange:EnergyReco                    0.000163798    0.0379512     0.603687      0.0030309     0.087008        100    
reco:energyrecnumumcs:EnergyReco                      0.00154138     0.0389102     0.598965     0.00492948     0.0855224       100    
reco:opdec:Deconvolution                               0.0856145     0.409064      0.731121      0.389837      0.147876        100    
reco:ophitspe:OpHitFinderDeco                          0.296519       2.28805       4.14436       2.2744       0.865775        100    
reco:opflash:OpFlashFinder                            0.000270317   0.00162327    0.00536936    0.00164907    0.000723617      100    
reco:rns:RandomNumberSaver                            2.2862e-05    4.57289e-05   0.000399569   4.0486e-05    3.78412e-05      100    
reco:anglereconue:NuAngularReco                       0.00019676    0.00202044     0.0242794    0.000582232   0.00388008       100    
reco:anglereconumu:NuAngularReco                      9.2413e-05    0.00180352     0.0245706    0.000406181   0.00388498       100    
reco:anglereconuepfps:NuAngularReco                   0.000148889   0.00196647     0.0247234    0.000530174   0.00386796       100    
reco:anglereconumupfps:NuAngularReco                  0.000124394   0.00186955     0.025503     0.00046393     0.0038805       100    
[art]:TriggerResults:TriggerResultInserter             9.538e-06    1.62345e-05   5.8961e-05    1.5093e-05    5.49234e-06      100    
end_path:out1:RootOutput                               3.015e-06    6.20269e-06   2.5107e-05     6.322e-06    2.66001e-06      100    
end_path:out1:RootOutput(write)                        0.0135242     0.288399      0.903997      0.274946      0.189229        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17322 MB
  Peak resident set size usage (VmHWM): 2366.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2_reco_2024-11-23T_022300Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_191_20231204T205607Z_gen_g4_detsim_hitreco__20240510T025006Z_reco2_reco_data_2024-11-23T_022300Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work