Jobsub ID 108744.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:30:36
From worker node	Hostname	b9p07p3554.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:44:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2.root
Jobscript	Exit code	0
	Real time	46m (2769s)
	CPU time	48m (2894s = 104%)
Outputting started	2024-11-23 05:30:22
Output files
Finished	2024-11-23 05:30:36
List job events Wrapper job log
Jobscript log (last 10,000 characters)

--------------------

Found: 2212 with KE=0.272144 GeV, E=1.21042 GeV, p2=0.584752
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:17 CET run: 74552621 subRun: 1 event: 10900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 37, 66, 62
minwire 0: 391
minwire 1: 2616
minwire 2: 78
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.997088, 0.00110805, 0.00180407, 
Output 1: 0.0165107, 0.983171, 0.000301343, 1.73973e-05, 
Output 2: 0.999368, 0.000631925, 

298
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 880!
23-Nov-2024 06:30:20 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2_reco_data_2024-11-23T_044417Z.root"
23-Nov-2024 06:30:21 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/25/4e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.525691       25.7464       384.582       7.24824       60.654         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.016425      0.0427205     0.0762004     0.0446313     0.0127822       100    
reco:linecluster:LineCluster                           0.0940468     0.311099       3.93467      0.166262      0.576206        100    
reco:trajcluster:TrajCluster                           0.0472623     0.862638       21.6831      0.0616042      3.07436        100    
reco:pandora:StandardPandora                           0.0574878      3.82023       178.284       1.00991       18.5895        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000109911   0.00527207     0.119024     0.00125334     0.0142906       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000147111    0.0080792     0.075356     0.00232368     0.0134698       100    
reco:pandoracalo:Calorimetry                          8.6221e-05     0.0197842     0.450479     0.000911198    0.069893        100    
reco:pandorapid:Chi2ParticleID                         3.212e-05    0.000134029   0.00146325    9.7671e-05    0.000156609      100    
reco:trkshowersplit:TrackShowerHits                   7.4001e-05     0.0251312     0.749347     0.000575285    0.0940202       100    
reco:pmtrack:PMAlgTrackMaker                          0.000904948     2.16781        45.18        0.15951       6.41246        100    
reco:pmtrackcalo:Calorimetry                           5.383e-05    0.00996966     0.309204     0.000737062    0.0371535       100    
reco:pmtrackpid:Chi2ParticleID                         2.283e-05    0.000238703   0.00249604    0.000111496   0.000395189      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000145601     2.1621        34.7253      0.223979       5.43416        100    
reco:pmtrajfitcalo:Calorimetry                        5.1321e-05     0.0207664     0.416961     0.00120331     0.0702498       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.226e-05    0.000151877   0.000901159   0.000111517   0.000137853      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000717457     2.82324       54.8348      0.181484       8.23987        100    
reco:pmtracktccalo:Calorimetry                        5.2261e-05    0.00604813     0.175146     0.000773018    0.0203078       100    
reco:pmtracktcpid:Chi2ParticleID                       2.235e-05    0.000211669   0.00197947    0.000116911   0.000308561      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000122211     1.29682       22.3604      0.199082       3.54079        100    
reco:pmtrajfittccalo:Calorimetry                      5.0471e-05     0.0141291     0.414865     0.00101534     0.0518856       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.136e-05    0.000284035   0.00315592    0.000131976   0.000486897      100    
reco:blurredcluster:BlurredClustering                 0.000109111    0.762931       7.50878      0.289725       1.28298        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000120681     8.90792       121.714       2.09142       18.9202        100    
reco:emshower:EMShower                                0.000192112    0.0214571     0.234351     0.00358481     0.0426851       100    
reco:cvnmap:CVNMapper                                  1.905e-05    0.00798895     0.0372879    0.00540538    0.00828747       100    
reco:cvneva:CVNEvaluator                               1.407e-05     0.567076       1.09686      0.735131      0.380711        100    
reco:energyrecnumu:EnergyReco                         0.000258813    0.0250744     0.573564     0.00384372     0.0794524       100    
reco:energyrecnue:EnergyReco                          0.000105901   0.00180841     0.0342897    0.000424864   0.00433807       100    
reco:energyrecnc:EnergyReco                           0.00145721    0.00365983     0.0373922    0.00235743    0.00466944       100    
reco:energyrecnumurange:EnergyReco                    0.000117851    0.0226971     0.573579     0.00184549     0.0778576       100    
reco:energyrecnumumcs:EnergyReco                      0.000892628    0.0241435     0.571604     0.00301445     0.0779231       100    
reco:opdec:Deconvolution                               0.0495077     0.245348       0.42959      0.247105      0.0701668       100    
reco:ophitspe:OpHitFinderDeco                          0.144022       1.37207       3.11188       1.33157      0.529833        100    
reco:opflash:OpFlashFinder                            0.000141731   0.00111216    0.00420264    0.00109159    0.000515668      100    
reco:rns:RandomNumberSaver                             1.612e-05    3.03708e-05   0.000331093   2.71805e-05   3.10959e-05      100    
reco:anglereconue:NuAngularReco                       0.000133711   0.00132325     0.0304247    0.000295438   0.00358678       100    
reco:anglereconumu:NuAngularReco                      5.8031e-05    0.00115358     0.0298859    0.000178532   0.00352758       100    
reco:anglereconuepfps:NuAngularReco                   0.000101571   0.00126662     0.0300426    0.000272222   0.00351729       100    
reco:anglereconumupfps:NuAngularReco                  8.2331e-05     0.0011894     0.029927     0.000229288   0.00351247       100    
[art]:TriggerResults:TriggerResultInserter             8.38e-06     1.0875e-05     3.945e-05     9.79e-06     3.74626e-06      100    
end_path:out1:RootOutput                               2.73e-06     4.98661e-06    1.88e-05      4.09e-06     2.09968e-06      100    
end_path:out1:RootOutput(write)                        0.0126608     0.179756      0.760635      0.171966      0.129547        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21895.2 MB
  Peak resident set size usage (VmHWM): 2466.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2_reco_2024-11-23T_044417Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_108_20231207T121507Z_gen_g4_detsim_hitreco__20240510T060138Z_reco2_reco_data_2024-11-23T_044417Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work