Jobsub ID 108593.87@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108593.87@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:25:00
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2024-11-23 01:48:08
From worker node	Hostname	cdr52.int.cedar.computecanada.ca
	cpuinfo	Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	84598 (23 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:36:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7824s)
	CPU time	1h (6712s = 85%)
Outputting started	2024-11-23 01:47:54
Output files
Finished	2024-11-23 01:48:08
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Run: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 17:46:37 PST run: 50610358 subRun: 1 event: 70900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 199, 96, 84
minwire 0: 2023
minwire 1: 555
minwire 2: 2329
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0012654, 0.000363634, 0.998371, 
Output 1: 0.00759396, 0.991677, 0.000722263, 7.01259e-06, 
Output 2: 0.999025, 0.000974908, 

339
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1004!
22-Nov-2024 17:46:48 PST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2_reco_data_2024-11-22T_233629Z.root"
22-Nov-2024 17:46:48 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/de/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4.18519       70.6012       2183.26       16.541        229.566        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0736218     0.160229      0.433304      0.163132      0.0573072       100    
reco:linecluster:LineCluster                           0.662967       1.28424       18.2441      0.734907       2.50475        100    
reco:trajcluster:TrajCluster                           0.217852       3.30526       102.929      0.248886       12.4699        100    
reco:pandora:StandardPandora                            1.76082       14.3318       1084.54       1.93378       107.69         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000172951    0.018309      0.751932     0.00278557     0.0769913       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000241685    0.0182589     0.381873     0.00554315     0.0439406       100    
reco:pandoracalo:Calorimetry                          0.00013916     0.0864306      4.70091     0.00194806     0.481291        100    
reco:pandorapid:Chi2ParticleID                        5.7814e-05     0.0002337    0.00213749    0.000174973   0.000255591      100    
reco:trkshowersplit:TrackShowerHits                   0.000126745    0.0563817      1.65021     0.00118781     0.205207        100    
reco:pmtrack:PMAlgTrackMaker                           0.0017257      4.95315       125.498      0.348504       15.7357        100    
reco:pmtrackcalo:Calorimetry                          0.000116565    0.0372247      1.62514     0.00133267     0.170905        100    
reco:pmtrackpid:Chi2ParticleID                        4.9536e-05    0.000519079   0.00676972    0.000184958   0.000917811      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000266058     4.42946       88.1246      0.552151       12.3256        100    
reco:pmtrajfitcalo:Calorimetry                        0.000108505    0.0888919      4.77663     0.00271048     0.489214        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.8381e-05    0.000273704   0.00169666    0.00019497    0.000245635      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00151517      6.9864        199.606       0.40406       23.5429        100    
reco:pmtracktccalo:Calorimetry                        0.000112928    0.0228347     0.825463     0.00150196     0.0895665       100    
reco:pmtracktcpid:Chi2ParticleID                      5.0136e-05    0.000465743   0.00576327    0.000194904   0.000750451      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000256275     3.93326       135.771      0.470818       14.3731        100    
reco:pmtrajfittccalo:Calorimetry                      0.000114753    0.0565222      2.65567     0.00209197     0.273984        100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.924e-05    0.000651005   0.00886409    0.00024784    0.00115931       100    
reco:blurredcluster:BlurredClustering                  0.0028952      1.76398       15.7387      0.675573       2.59889        100    
reco:emtrkmichelid:EmTrackMichelId                      0.1524        22.5486       379.507       5.72986       48.8823        100    
reco:emshower:EMShower                                0.00030441     0.0571358     0.767398      0.016783      0.105171        100    
reco:cvnmap:CVNMapper                                 3.3813e-05     0.0142112     0.0761193     0.0135643     0.0120014       100    
reco:cvneva:CVNEvaluator                              2.6259e-05     0.897552       7.92643       1.05619       0.87183        100    
reco:energyrecnumu:EnergyReco                         0.000452523    0.0750985      1.75989     0.00674124     0.230207        100    
reco:energyrecnue:EnergyReco                          0.000233681   0.00575157      0.22467     0.000881063    0.0231118       100    
reco:energyrecnc:EnergyReco                            0.0029546    0.00851827      0.22761     0.00364561     0.023173        100    
reco:energyrecnumurange:EnergyReco                    0.000304456    0.0726103      1.75016     0.00375369     0.229522        100    
reco:energyrecnumumcs:EnergyReco                      0.00252697     0.0747866      1.77837     0.00524017     0.231454        100    
reco:opdec:Deconvolution                               0.303924       2.47045       164.586      0.790221       16.295         100    
reco:ophitspe:OpHitFinderDeco                          0.736342       2.46759       3.92801       2.43342      0.742305        100    
reco:opflash:OpFlashFinder                            0.000291154   0.00182246    0.00563463    0.00170176    0.000856996      100    
reco:rns:RandomNumberSaver                            2.1242e-05    3.67698e-05   0.000335167   3.31325e-05   3.15124e-05      100    
reco:anglereconue:NuAngularReco                       0.000175587   0.00480373     0.222744     0.000534985    0.0226337       100    
reco:anglereconumu:NuAngularReco                      0.000115278     0.00462      0.220133     0.000422778    0.0223855       100    
reco:anglereconuepfps:NuAngularReco                   0.000170979   0.00481549     0.225099     0.000541661    0.0228532       100    
reco:anglereconumupfps:NuAngularReco                  0.000169501   0.00474783     0.221973     0.000526774    0.0225535       100    
[art]:TriggerResults:TriggerResultInserter             1.386e-05    1.62962e-05   5.6609e-05    1.5478e-05    4.29405e-06      100    
end_path:out1:RootOutput                               2.963e-06    5.45105e-06   1.9193e-05    3.9905e-06    2.61177e-06      100    
end_path:out1:RootOutput(write)                        0.0162891     0.351062       2.11078      0.314578      0.286167        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8637.13 MB
  Peak resident set size usage (VmHWM): 2653.67 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2_reco_2024-11-22T_233629Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_708_20231207T085811Z_gen_g4_detsim_hitreco__20240510T035505Z_reco2_reco_data_2024-11-22T_233629Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work