Jobsub ID 108742.57@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.57@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:38:48
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:38:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3609s)
	CPU time	48m (2919s = 80%)
Outputting started	2024-11-23 05:38:31
Output files
Finished	2024-11-23 05:38:48
List job events Wrapper job log
Jobscript log (last 10,000 characters)

64269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:38:23 GMT run: 6464269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:38:24 GMT run: 6464269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:38:24 GMT run: 6464269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:38:24 GMT run: 6464269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:38:24 GMT run: 6464269 subRun: 1 event: 53900
1st wire C:0 T:2 P:2 W:546 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:38:26 GMT run: 6464269 subRun: 1 event: 53900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 89, 57, 62
minwire 0: 277
minwire 1: 2442
minwire 2: 80
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0502571, 0.000508262, 0.949235, 
Output 1: 0.97552, 0.0237908, 0.000586729, 0.000102211, 
Output 2: 0.963918, 0.0360823, 

294
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 806!
23-Nov-2024 05:38:30 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2_reco_data_2024-11-23T_043829Z.root"
23-Nov-2024 05:38:30 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/bc/23/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.23803       29.6195       1085.74       9.71482       111.054        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0342204     0.0687647     0.182669      0.0692094     0.0261435       100    
reco:linecluster:LineCluster                            0.22356      0.885406       44.1758      0.369372       4.36364        100    
reco:trajcluster:TrajCluster                           0.101408       1.13907       81.9725      0.109787       8.17244        100    
reco:pandora:StandardPandora                           0.112841       3.69507       208.118       1.3468        20.5916        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000170242   0.00708073     0.289411      0.0013201     0.0297321       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000228774    0.0125171     0.228093     0.00285839     0.031366        100    
reco:pandoracalo:Calorimetry                          0.000140667    0.0191678     0.945297     0.000785715    0.100067        100    
reco:pandorapid:Chi2ParticleID                        4.8352e-05    0.00019222    0.00140218    0.000149208   0.000183228      100    
reco:trkshowersplit:TrackShowerHits                   0.000135477    0.0339415      2.25715     0.000760387     0.22862        100    
reco:pmtrack:PMAlgTrackMaker                          0.00115967      3.10588       172.627      0.147906       17.5216        100    
reco:pmtrackcalo:Calorimetry                          7.6505e-05     0.0133514     0.742593     0.000971422    0.0772135       100    
reco:pmtrackpid:Chi2ParticleID                        3.2972e-05    0.000306829   0.00556373    0.000166145   0.000621576      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000238542     1.97124       93.1395      0.157322       9.48024        100    
reco:pmtrajfitcalo:Calorimetry                        8.1094e-05     0.020362      0.999316     0.00113804     0.105428        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.4015e-05    0.000222961   0.00227741    0.000168775   0.000254127      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00126629      3.04436       162.709      0.170843       16.6351        100    
reco:pmtracktccalo:Calorimetry                        8.6925e-05    0.00763237     0.255207     0.000899671    0.0306818       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6859e-05    0.000276187   0.00332315    0.000174366   0.000414363      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000243902     1.70818       75.5517      0.240254       7.80744        100    
reco:pmtrajfittccalo:Calorimetry                      8.1584e-05     0.0172687     0.928573     0.000921252    0.0966958       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3533e-05    0.000369468   0.00662459    0.000166335   0.00076177       100    
reco:blurredcluster:BlurredClustering                 0.00010046     0.851756       9.23942      0.379645       1.37049        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000180863     9.45596       222.112       2.10117       26.3268        100    
reco:emshower:EMShower                                0.000243231    0.0341064     0.655774       0.00466      0.0780464       100    
reco:cvnmap:CVNMapper                                  3.137e-05     0.012705      0.0497093     0.0116587     0.0117331       100    
reco:cvneva:CVNEvaluator                              1.9748e-05     0.589278       1.17948      0.855915      0.418451        100    
reco:energyrecnumu:EnergyReco                         0.000402764    0.0247206     0.477813     0.00414215     0.0752192       100    
reco:energyrecnue:EnergyReco                          0.000138512   0.00225017     0.0822086    0.000505664   0.00845067       100    
reco:energyrecnc:EnergyReco                           0.00181501    0.00464946     0.090002     0.00307851    0.00900155       100    
reco:energyrecnumurange:EnergyReco                    0.000175382    0.0219821      0.4624      0.00282187     0.0726038       100    
reco:energyrecnumumcs:EnergyReco                      0.00168366     0.0237638      0.4702      0.00332765     0.0729619       100    
reco:opdec:Deconvolution                               0.116693      0.411451      0.756177      0.402407      0.127684        100    
reco:ophitspe:OpHitFinderDeco                          0.451427       2.17688        4.071        2.07279      0.928173        100    
reco:opflash:OpFlashFinder                            0.00044856     0.0018559     0.007163     0.00164265    0.00107675       100    
reco:rns:RandomNumberSaver                            1.9346e-05    6.00354e-05   0.000407653   5.30905e-05   5.15761e-05      100    
reco:anglereconue:NuAngularReco                       0.000187395   0.00192636      0.08009     0.000475141   0.00810573       100    
reco:anglereconumu:NuAngularReco                      9.2686e-05    0.00156889     0.0780842    0.000230947    0.0079168       100    
reco:anglereconuepfps:NuAngularReco                   0.000134835   0.00179036     0.0822114    0.000425437   0.00829553       100    
reco:anglereconumupfps:NuAngularReco                  0.000113114   0.00165047     0.0799113    0.000317452    0.0080896       100    
[art]:TriggerResults:TriggerResultInserter             9.568e-06    1.66236e-05   3.8994e-05     1.588e-05    4.37365e-06      100    
end_path:out1:RootOutput                               2.675e-06    9.51502e-06   2.9947e-05    9.2875e-06    4.42092e-06      100    
end_path:out1:RootOutput(write)                        0.0174351     0.248015       1.45703      0.256237      0.203348        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38757 MB
  Peak resident set size usage (VmHWM): 2689.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2_reco_2024-11-23T_043829Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_538_20231207T033601Z_gen_g4_detsim_hitreco__20240510T041136Z_reco2_reco_data_2024-11-23T_043829Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work