Jobsub ID 301744.60@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301744.60@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:56:37
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:49:43
From worker node	Hostname	b9p08p1455.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:58:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3082s)
	CPU time	52m (3154s = 102%)
Outputting started	2024-11-23 05:49:27
Output files
Finished	2024-11-23 05:49:43
List job events Wrapper job log
Jobscript log (last 10,000 characters)

96296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:49:20 CET run: 74496296 subRun: 1 event: 47200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 113, 110, 86
minwire 0: 2431
minwire 1: 45
minwire 2: 2830
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.64686, 0.00287837, 0.350262, 
Output 1: 0.201582, 0.66264, 0.132566, 0.00321223, 
Output 2: 0.243548, 0.756452, 

309
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 797!
23-Nov-2024 06:49:26 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2_reco_data_2024-11-23T_045811Z.root"
23-Nov-2024 06:49:26 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/c7/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.534722       28.7938       717.677       8.18591       82.0724        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175789     0.0420269     0.0963488     0.0398943      0.01428        100    
reco:linecluster:LineCluster                           0.0946449     0.578469       20.9634      0.167532       2.52487        100    
reco:trajcluster:TrajCluster                           0.0473572      1.03152       49.7098      0.0563394      5.45215        100    
reco:pandora:StandardPandora                           0.0565894      2.54841       58.9995       1.2955        6.74269        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000165882   0.00674344     0.211078     0.00129239     0.0226083       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000237772    0.0152046     0.316321     0.00275579     0.0412539       100    
reco:pandoracalo:Calorimetry                          0.000128142    0.0290775      1.28509     0.000874774    0.140225        100    
reco:pandorapid:Chi2ParticleID                        4.4581e-05    0.000193686   0.00217059    0.000142286   0.000241406      100    
reco:trkshowersplit:TrackShowerHits                   6.1301e-05     0.0364048      1.78894     0.000564686    0.194259        100    
reco:pmtrack:PMAlgTrackMaker                          0.00124424      2.86275       104.009      0.160764       11.6015        100    
reco:pmtrackcalo:Calorimetry                          7.1351e-05     0.0156326     0.897255     0.000771588    0.0904884       100    
reco:pmtrackpid:Chi2ParticleID                         2.706e-05    0.000297974   0.00591693    0.000135827   0.000641672      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000211743     2.49237       62.1568      0.222889       8.41644        100    
reco:pmtrajfitcalo:Calorimetry                         6.127e-05     0.0291357      1.38104     0.00114355      0.1459         100    
reco:pmtrajfitpid:Chi2ParticleID                       3.148e-05    0.000189346   0.00168759    0.000138392   0.000195808      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00085987      3.48837       135.91       0.168298       15.0465        100    
reco:pmtracktccalo:Calorimetry                        6.3911e-05    0.00677517     0.227555     0.000852764    0.0250224       100    
reco:pmtracktcpid:Chi2ParticleID                       2.86e-05     0.000243318   0.00307311    0.000134582   0.000378923      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000176452     1.69593       62.9748      0.265674       6.57601        100    
reco:pmtrajfittccalo:Calorimetry                       7.729e-05     0.020673       1.07225     0.00093274     0.110273        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.039e-05    0.000343241   0.00438161    0.000148602   0.000603702      100    
reco:blurredcluster:BlurredClustering                 0.000137882     1.06492       10.2009      0.507892       1.6094         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000122591     9.73829       189.45        1.85279       23.9989        100    
reco:emshower:EMShower                                0.000178552    0.0234764     0.551577      0.0032142     0.062302        100    
reco:cvnmap:CVNMapper                                 2.2061e-05     0.0119318      0.1198       0.0134076     0.0135035       100    
reco:cvneva:CVNEvaluator                               1.731e-05     0.632183       1.13089       0.81357      0.423298        100    
reco:energyrecnumu:EnergyReco                         0.000346253    0.056643       2.38958     0.00353401     0.270259        100    
reco:energyrecnue:EnergyReco                          0.000133892   0.00221202     0.0584759    0.000402174   0.00654033       100    
reco:energyrecnc:EnergyReco                           0.00167935    0.00475002     0.0625055    0.00324864    0.00687563       100    
reco:energyrecnumurange:EnergyReco                    0.000139302    0.0527275      2.27308     0.00213036     0.258511        100    
reco:energyrecnumumcs:EnergyReco                      0.000713078    0.0585195      2.76889     0.00340183     0.301874        100    
reco:opdec:Deconvolution                               0.0488371     0.264347      0.482562      0.265029      0.0891634       100    
reco:ophitspe:OpHitFinderDeco                          0.142536       1.74377       4.56655       1.5548        0.85442        100    
reco:opflash:OpFlashFinder                            0.000138941   0.00147799    0.00634489    0.00133028    0.000761702      100    
reco:rns:RandomNumberSaver                            1.8951e-05    4.61702e-05   0.000365504   4.18255e-05   3.41045e-05      100    
reco:anglereconue:NuAngularReco                       0.000183672   0.00195919     0.087943     0.000479295   0.00882716       100    
reco:anglereconumu:NuAngularReco                      7.4271e-05    0.00164328     0.0843088    0.000225022    0.0084768       100    
reco:anglereconuepfps:NuAngularReco                   0.000135821   0.00181088     0.0853709    0.000370724   0.00858229       100    
reco:anglereconumupfps:NuAngularReco                  0.000102651   0.00171311     0.086294     0.000271403   0.00865706       100    
[art]:TriggerResults:TriggerResultInserter             8.74e-06     1.4445e-05     6.288e-05     1.315e-05    6.67667e-06      100    
end_path:out1:RootOutput                               3.11e-06     6.60223e-06   2.1841e-05     6.295e-06    2.34653e-06      100    
end_path:out1:RootOutput(write)                        0.011722      0.228573       1.70399      0.208875      0.210301        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21854.2 MB
  Peak resident set size usage (VmHWM): 2439.05 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2_reco_2024-11-23T_045811Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_471_20231201T212740Z_gen_g4_detsim_hitreco__20240508T045559Z_reco2_reco_data_2024-11-23T_045811Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work