Jobsub ID 87162.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	87162.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	3836
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-10-26 05:16:41
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce01
Last heartbeat	2024-10-26 06:46:35
From worker node	Hostname	ccwcondor0695.in2p3.fr
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-10-26 05:17:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5324s)
	CPU time	1h (5007s = 94%)
Outputting started	2024-10-26 06:45:59
Output files
Finished	2024-10-26 06:46:35
List job events Wrapper job log
Jobscript log (last 10,000 characters)

CEST run: 74511854 subRun: 1 event: 31100
1st wire C:0 T:22 P:2 W:812 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:45 CEST run: 74511854 subRun: 1 event: 31100
1st wire C:0 T:22 P:2 W:998 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  26-Oct-2024 08:44:47 CEST run: 74511854 subRun: 1 event: 31100
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 2214, 1563, 2121
minwire 0: 1362
minwire 1: 673
minwire 2: 1104
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00146279, 0.000423276, 0.998114, 
Output 1: 0.381757, 0.424657, 0.130416, 0.06317, 
Output 2: 0.78256, 0.21744, 

781
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3086!
26-Oct-2024 08:45:55 CEST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2_reco_data_2024-10-26T_051719Z.root"
26-Oct-2024 08:45:55 CEST  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/ff/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.687804       50.0873       1190.79       11.7515       143.093        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0218917     0.0545092     0.309247      0.049988      0.0322457       100    
reco:linecluster:LineCluster                           0.127882       1.04877       58.3542      0.239512       5.98917        100    
reco:trajcluster:TrajCluster                           0.0603136      3.04555       133.823      0.0863213      15.4121        100    
reco:pandora:StandardPandora                           0.0845082      4.62609       172.351       1.48825       18.0607        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121229    0.0109767     0.163712     0.00209216     0.0274951       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000177234    0.0254005     0.624274     0.00494279     0.0728884       100    
reco:pandoracalo:Calorimetry                          9.2544e-05     0.0440498      1.93054     0.00151944     0.210194        100    
reco:pandorapid:Chi2ParticleID                        3.8503e-05    0.000229046    0.0021301    0.000164925   0.000263482      100    
reco:trkshowersplit:TrackShowerHits                   0.000107453    0.0443898      1.64724     0.00100126     0.186089        100    
reco:pmtrack:PMAlgTrackMaker                          0.00118262      5.78673        212.9       0.250021       23.6475        100    
reco:pmtrackcalo:Calorimetry                          6.6425e-05     0.0266267     0.784732     0.00123938     0.101309        100    
reco:pmtrackpid:Chi2ParticleID                        2.7452e-05    0.000446929   0.00443043    0.000169535   0.000826037      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000192263     3.70732       137.199      0.363229       14.8199        100    
reco:pmtrajfitcalo:Calorimetry                        5.6336e-05     0.0424079      1.66524     0.00217302     0.189532        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3434e-05    0.000243708   0.00125194     0.0001723    0.000228715      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000966703     6.02922       151.752      0.373532       20.8318        100    
reco:pmtracktccalo:Calorimetry                        5.6166e-05     0.015302      0.254511     0.00125376     0.0445468       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3805e-05    0.000360229   0.00260258    0.000179408   0.000524463      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000158128     2.97649       66.7367      0.353466       8.78571        100    
reco:pmtrajfittccalo:Calorimetry                       5.33e-05      0.0341315      1.07398     0.00165416     0.129787        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2382e-05    0.000516826    0.0051449    0.000202191   0.000890312      100    
reco:blurredcluster:BlurredClustering                 9.9849e-05      1.30903       8.25329      0.617387       1.68027        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00013802      17.6768       232.751       3.76311       38.3442        100    
reco:emshower:EMShower                                0.000195439    0.0343657     0.253391     0.00631469     0.0548089       100    
reco:cvnmap:CVNMapper                                  2.149e-05     0.0156199     0.0438598     0.0202034     0.0110434       100    
reco:cvneva:CVNEvaluator                              1.5299e-05      0.82019       1.59706       1.06414      0.554377        100    
reco:energyrecnumu:EnergyReco                         0.000378744    0.0913065      2.17595     0.00653818     0.298376        100    
reco:energyrecnue:EnergyReco                          0.000156355    0.0037011     0.0595416    0.000701097   0.00906199       100    
reco:energyrecnc:EnergyReco                           0.00195831    0.00680384     0.072799     0.00362725     0.0101643       100    
reco:energyrecnumurange:EnergyReco                    0.000186912    0.0861985      2.05704     0.00296434     0.294634        100    
reco:energyrecnumumcs:EnergyReco                      0.00114611     0.0892244      2.11755     0.00483666     0.301756        100    
reco:opdec:Deconvolution                               0.065096      0.327756      0.689804      0.318533      0.111319        100    
reco:ophitspe:OpHitFinderDeco                           0.21515       1.80366       3.45352       1.79513      0.780797        100    
reco:opflash:OpFlashFinder                            0.000262545   0.00186238    0.00562915    0.00172012    0.000913108      100    
reco:rns:RandomNumberSaver                             2.131e-05    4.88934e-05   0.000389925   4.5105e-05    3.56413e-05      100    
reco:anglereconue:NuAngularReco                       0.000180851   0.00318063     0.061301     0.000545413   0.00923415       100    
reco:anglereconumu:NuAngularReco                      7.7546e-05    0.00284462     0.0567456    0.000295007   0.00885835       100    
reco:anglereconuepfps:NuAngularReco                   0.000160493   0.00300284     0.0567109    0.000439694   0.00871602       100    
reco:anglereconumupfps:NuAngularReco                  0.000113934   0.00297005     0.0584812    0.000353346   0.00899551       100    
[art]:TriggerResults:TriggerResultInserter             8.947e-06    1.58617e-05    6.876e-05    1.4893e-05    7.06761e-06      100    
end_path:out1:RootOutput                               3.397e-06    7.92119e-06   2.5999e-05     7.449e-06    2.72794e-06      100    
end_path:out1:RootOutput(write)                        0.0126593     0.286315       1.43561       0.26825      0.257447        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12952.4 MB
  Peak resident set size usage (VmHWM): 2169.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2_reco_2024-10-26T_051719Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_310_20231203T165058Z_gen_g4_detsim_hitreco__20240509T194949Z_reco2_reco_data_2024-10-26T_051719Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
work