Jobsub ID 108746.89@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108746.89@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:39:54
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce511
Last heartbeat	2024-11-23 05:14:09
From worker node	Hostname	b9p05p7101.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:41:39
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2.root
Jobscript	Exit code	0
	Real time	32m (1932s)
	CPU time	34m (2086s = 107%)
Outputting started	2024-11-23 05:13:51
Output files
Finished	2024-11-23 05:14:09
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:7 P:2 W:1030 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:24 CET run: 50610358 subRun: 1 event: 76400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:24 CET run: 50610358 subRun: 1 event: 76400
1st wire C:0 T:7 P:2 W:1021 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:24 CET run: 50610358 subRun: 1 event: 76400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:24 CET run: 50610358 subRun: 1 event: 76400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:24 CET run: 50610358 subRun: 1 event: 76400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:13:25 CET run: 50610358 subRun: 1 event: 76400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1236, 1135, 1361
minwire 0: 257
minwire 1: 1364
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0104491, 0.000416217, 0.989135, 
Output 1: 0.943432, 0.046445, 0.00670974, 0.00341328, 
Output 2: 0.930519, 0.0694809, 

717
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2793!
23-Nov-2024 06:13:50 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2_reco_data_2024-11-23T_044145Z.root"
23-Nov-2024 06:13:50 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b6/0f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.616141       17.3353       168.376       9.64452       24.7456        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0168462     0.0416305     0.0968619     0.0423992     0.0157712       100    
reco:linecluster:LineCluster                           0.111681      0.229695       2.53733      0.181658      0.296297        100    
reco:trajcluster:TrajCluster                           0.0477236     0.264981       7.45789      0.0674711     0.856155        100    
reco:pandora:StandardPandora                           0.0583004      1.44312       8.78867       1.2379        1.05231        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000116691   0.00390339     0.0421454     0.0016277    0.00655973       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000147601   0.00891073     0.119036     0.00334594     0.0176483       100    
reco:pandoracalo:Calorimetry                           8.916e-05    0.00847165     0.230527     0.00122205     0.0288309       100    
reco:pandorapid:Chi2ParticleID                         3.496e-05    0.000159907   0.00157544    0.000130276   0.000169861      100    
reco:trkshowersplit:TrackShowerHits                   9.6421e-05    0.00942543     0.308485     0.000933491    0.0346762       100    
reco:pmtrack:PMAlgTrackMaker                          0.000959886     1.2311        17.9091      0.232981       2.7367         100    
reco:pmtrackcalo:Calorimetry                           6.025e-05    0.00420041     0.0618026    0.00104532     0.0097484       100    
reco:pmtrackpid:Chi2ParticleID                         2.301e-05    0.000206232   0.00105625    0.000146596   0.000201679      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154441     1.13988       24.2359      0.358902       2.78313        100    
reco:pmtrajfitcalo:Calorimetry                         5.611e-05     0.0075622      0.19266     0.00156441     0.0225902       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.303e-05    0.000171491   0.000673695   0.000148211   0.000116585      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000846886     1.5074        31.4797      0.301631       4.0509         100    
reco:pmtracktccalo:Calorimetry                         6.312e-05    0.00377944     0.0424139    0.00113687    0.00786972       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4301e-05    0.000210868   0.00103862    0.000168776   0.000192858      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000148931     0.92526       11.8476      0.344107       1.73564        100    
reco:pmtrajfittccalo:Calorimetry                       5.947e-05    0.00673612      0.19268     0.00128399     0.0218948       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.377e-05    0.000275323   0.00278549    0.000173996   0.000343857      100    
reco:blurredcluster:BlurredClustering                 0.000117341    0.801693       5.0168       0.507124       0.98272        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000115521     6.63311       62.8682       3.09612       10.457         100    
reco:emshower:EMShower                                0.000174251    0.0195202     0.190986     0.00510629     0.0330211       100    
reco:cvnmap:CVNMapper                                  1.96e-05     0.00954445     0.0407216    0.00803453    0.00834296       100    
reco:cvneva:CVNEvaluator                               1.593e-05     0.624795       1.08516      0.776477      0.390109        100    
reco:energyrecnumu:EnergyReco                         0.000270182    0.0195446     0.294622     0.00396263     0.0485831       100    
reco:energyrecnue:EnergyReco                          0.000113281   0.00137472     0.0129162    0.000582394   0.00211422       100    
reco:energyrecnc:EnergyReco                             0.00146      0.003593      0.0168147    0.00307131    0.00249226       100    
reco:energyrecnumurange:EnergyReco                    0.000138661    0.0170499     0.284402     0.00279448     0.0459164       100    
reco:energyrecnumumcs:EnergyReco                      0.000658615    0.0181635     0.275678     0.00360794     0.0442272       100    
reco:opdec:Deconvolution                               0.0511992     0.272423      0.451735      0.276905      0.0786135       100    
reco:ophitspe:OpHitFinderDeco                          0.154454       1.86639       4.12224       1.78652      0.803665        100    
reco:opflash:OpFlashFinder                            0.000182481   0.00131995    0.00535588    0.00118433    0.000724624      100    
reco:rns:RandomNumberSaver                             1.656e-05    3.91855e-05   0.000378003   3.4036e-05    3.61858e-05      100    
reco:anglereconue:NuAngularReco                       0.000137681   0.000946482   0.00955346    0.000428632   0.00161738       100    
reco:anglereconumu:NuAngularReco                       6.027e-05    0.000745317   0.00916192    0.000266337   0.00154229       100    
reco:anglereconuepfps:NuAngularReco                   9.7981e-05    0.000881719   0.00967939    0.000366382   0.00158798       100    
reco:anglereconumupfps:NuAngularReco                  8.3671e-05    0.000782762   0.00909196    0.000305457    0.0015236       100    
[art]:TriggerResults:TriggerResultInserter             8.18e-06     1.26606e-05    4.915e-05     1.141e-05    4.84893e-06      100    
end_path:out1:RootOutput                               3.07e-06     6.14665e-06    2.165e-05     5.92e-06     2.59987e-06      100    
end_path:out1:RootOutput(write)                        0.0161177     0.204405      0.727102      0.215142       0.12647        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21592.1 MB
  Peak resident set size usage (VmHWM): 2186.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2_reco_2024-11-23T_044145Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_763_20231207T090008Z_gen_g4_detsim_hitreco__20240510T034943Z_reco2_reco_data_2024-11-23T_044145Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work