Single GPU benchmark scripts by ChSonnabend · Pull Request #15514 · AliceO2Group/AliceO2

ChSonnabend · 2026-06-12T07:28:27Z

This PR brings two scripts that benchmark the single GPU performance

gen_single_gpu_rtc_benchmark.sh generates the workflow from dpl-workflow.sh by setting environment variables and using early stops to avoid processing failures
analyze_gpu_benchmarks.py then analyzes the resulting log file for processing times, records then, histograms them and fits a gaussian to the result to determine the mean processing time per timeslice

…ithout accesibility to numactl

davidrohr

Didn't check anything in detail, but the things that immediately came to my mind

davidrohr · 2026-06-12T07:51:22Z

+
+# ROCm library injection is only useful for HIP runs. Keep it off by default for CUDA/NVIDIA containers,
+# because mixed AMD/NVIDIA hosts can otherwise leak ROCm libraries into LD_LIBRARY_PATH.
+if [[ "${GPUTYPE:-}" == "HIP" && "0$BENCH_AUTO_ROCM_LIBS" == "01" ]]; then


With new bash you can just use $BENCH_AUTO_ROCM_LIBS == 1

davidrohr · 2026-06-12T07:52:22Z

+
+export DPL_REPORT_PROCESSING="${DPL_REPORT_PROCESSING:-1}"
+
+export FST_TMUX_NO_EPN="${FST_TMUX_NO_EPN:-1}"


not needed, since start_tmux.sh is not used

davidrohr · 2026-06-12T07:52:39Z

+# ----------------------------------------------------------------------------------------------------------------------
+# Locate original workflow script. Keep the original untouched.
+
+: "${GEN_TOPO_MYDIR:=$(dirname "$(realpath "$0")")}"


Why don't you simple use $O2_ROOT/dpl-workflow.sh?

davidrohr · 2026-06-12T07:53:17Z

+export WORKFLOW_PARAMETERS="${WORKFLOW_PARAMETERS:-GPU,CTF}"
+export GPUTYPE="${GPUTYPE:-CUDA}"
+export NGPUS=1
+export NUMAGPUIDS=1


NUMAGPUIDS and NUMAID should not be set, if not using NUMA pinning

davidrohr · 2026-06-12T07:54:47Z

+export EPNSYNCMODE="${EPNSYNCMODE:-0}"
+export SYNCMODE="${SYNCMODE:-1}"
+export SYNCRAWMODE="${SYNCRAWMODE:-0}"
+
+export TIMEFRAME_RATE_LIMIT="${TIMEFRAME_RATE_LIMIT:-5}"
+export GEN_TOPO_NO_TF_RATE_UPSCALING="${GEN_TOPO_NO_TF_RATE_UPSCALING:-1}"
+
+export DISABLE_ROOT_OUTPUT="${DISABLE_ROOT_OUTPUT:-1}"
+
+# Double pipeline requires zsraw input. Therefore default to raw TF input, not CTF.
+export CTFINPUT="${CTFINPUT:-0}"
+export RAWTFINPUT="${RAWTFINPUT:-1}"
+export DIGITINPUT="${DIGITINPUT:-0}"
+export EXTINPUT="${EXTINPUT:-0}"


Why do you redefine all the defaults that come from setenv.sh?
I would only set those settings, which you need.
That should be
SYNCMODE=1
TIMEFRAME_RATE_LIMIT=5
RAWTFINPUT=1

davidrohr · 2026-06-12T07:55:45Z

+  source "$PWD/local_env.sh"
+fi
+
+export ALICE_O2_FST="${ALICE_O2_FST:-1}"


This is a hack for running on MI100, I would not put it in this script

davidrohr · 2026-06-12T07:56:00Z

+
+export ALICE_O2_FST="${ALICE_O2_FST:-1}"
+
+if [[ -f "$GEN_TOPO_MYDIR/setenv.sh" ]]; then


dpl-workflow.sh will source setenv.sh, why do you source it here?

davidrohr · 2026-06-12T07:56:33Z

+# Let O2/core dumps land in the benchmark run directory, not in the original working directory.
+export CORE_DUMP_DIR="${CORE_DUMP_DIR:-$RUNDIR}"
+export O2_CORE_DUMP_DIR="${O2_CORE_DUMP_DIR:-$RUNDIR}"
+export FAIRMQ_SHM_MONITOR_CONFIG="${FAIRMQ_SHM_MONITOR_CONFIG:-}"


We do not run the SHM MONITOR, why do you need this?

alibuild · 2026-06-12T19:08:09Z

Error while checking build/O2/fullCI_slc9 for 6353bf6 at 2026-06-12 21:08:

## sw/BUILD/O2-full-system-test-latest/log
command /sw/slc9_x86-64/O2/slc9_x86-64-slc9_x86-64-local1/prodtests/full-system-test/dpl-workflow.sh had nonzero exit code 1

Full log here.

davidrohr · 2026-06-13T07:24:25Z

 (has_detector_reco ITS && ! has_detector_gpu ITS) && ! has_detector_from_global_reader ITS && add_W o2-its-reco-workflow "$ITS_CONFIG $ITS_STAGGERED $DISABLE_MC ${DISABLE_DIGIT_CLUSTER_INPUT:-} $DISABLE_ROOT_OUTPUT --pipeline $(get_N its-tracker ITS REST 1 ITSTRK),$(get_N its-clusterer ITS REST 1 ITSCL)" "$ITS_CONFIG_KEY;$ITSMFT_STROBES;$ITSEXTRAERR"
 [[ ${DISABLE_DIGIT_CLUSTER_INPUT:-} =~ "--digits-from-upstream" ]]  && has_detector_gpu ITS && ! has_detector_from_global_reader ITS && add_W o2-its-reco-workflow "--disable-tracking ${DISABLE_DIGIT_CLUSTER_INPUT:-} $ITS_STAGGERED $DISABLE_MC $DISABLE_ROOT_OUTPUT --pipeline $(get_N its-clusterer ITS REST 1 ITSCL)" "$ITS_CONFIG_KEY;$ITSMFT_STROBES;$ITSEXTRAERR"
-(has_detector_reco TPC || has_detector_ctf TPC) && ! has_detector_from_global_reader TPC && add_W o2-gpu-reco-workflow "--gpu-reconstruction \"$GPU_CONFIG_SELF\" --input-type=$GPU_INPUT $DISABLE_MC --output-type $GPU_OUTPUT $([[ $TPC_CORR_OPT == *--disable-ctp-lumi-request* ]] && echo --disable-ctp-lumi-request) $ITS_STAGGERED --pipeline gpu-reconstruction:${N_TPCTRK:-1},gpu-reconstruction-prepare:${N_TPCTRK:-1} $GPU_CONFIG" "GPU_global.deviceType=$GPUTYPE;GPU_proc.debugLevel=0;$GPU_CONFIG_KEY;$TRACKTUNETPCINNER;"
+(has_detector_reco TPC || has_detector_ctf TPC) && ! has_detector_from_global_reader TPC && add_W o2-gpu-reco-workflow "--gpu-reconstruction \"$GPU_CONFIG_SELF\" $MSLOG --input-type=$GPU_INPUT $DISABLE_MC --output-type $GPU_OUTPUT $([[ $TPC_CORR_OPT == *--disable-ctp-lumi-request* ]] && echo --disable-ctp-lumi-request) $ITS_STAGGERED --pipeline gpu-reconstruction:${N_TPCTRK:-1},gpu-reconstruction-prepare:${N_TPCTRK:-1} $GPU_CONFIG" "GPU_global.deviceType=$GPUTYPE;GPU_proc.debugLevel=0;$GPU_CONFIG_KEY;$TRACKTUNETPCINNER;"


Instead of modifying dpl-workflow.sh, you can just set

ARGS_EXTRA_PROCESS_o2_gpu_reco_workflow="--log-timestamp-us"

in your benchmark script.

davidrohr · 2026-06-13T07:25:40Z

+
+export DPL_REPORT_PROCESSING="${DPL_REPORT_PROCESSING:-1}"
+export WORKFLOW_PARAMETERS="${WORKFLOW_PARAMETERS:-GPU,CTF}"
+export GPUTYPE="${GPUTYPE:-CUDA}"


Perhaps I would not set CUDA here, but would request the user to set it, since the script is supposed to work equally for CUDA and for HIP. Just to avoid user error, if the user doesn't provide it.

davidrohr · 2026-06-13T07:28:03Z

+export O2_GPU_DOUBLE_PIPELINE="${O2_GPU_DOUBLE_PIPELINE:-1}"
+export O2_GPU_RTC="${O2_GPU_RTC:-1}"
+export SYNCMODE="${SYNCMODE:-1}"
+export DISABLE_ROOT_OUTPUT="${DISABLE_ROOT_OUTPUT:-1}"


DISABLE_ROOT_OUTPUT is alrady enabled by default.
So you can remove it here.
(And btw, for this setting to correct should be DISABLE_ROOT_OUTPUT="--disable-root-output")

davidrohr · 2026-06-13T07:30:29Z

+
+export RUN_BENCHMARK="${RUN_BENCHMARK:-0}"
+
+echo "# Alien/JAliEn environment check:"


I still don't understand why we need this alien token magic. If alien-token-info finds the token before running this script, that should be all that is needed?

ChSonnabend added 4 commits June 8, 2026 14:59

Avoiding numactl execution to avoid crashes of FST in container env w…

dd89cae

…ithout accesibility to numactl

Adding GPU benchmark scripts and python analysis script

95f3190

Merge branch 'AliceO2Group:dev' into devel_fst_numactl

48b88e3

Resetting start_tumx.sh to upstream/dev

478d76b

ChSonnabend requested a review from a team as a code owner June 12, 2026 07:28

davidrohr requested changes Jun 12, 2026

View reviewed changes

ChSonnabend added 3 commits June 12, 2026 13:11

Updating scripts

f87246f

Update env variables

43e244b

Adding microsecond logging for dpl-workflow.sh

6353bf6

Avoiding unbound variable

d57d8ab

davidrohr reviewed Jun 13, 2026

View reviewed changes

Adjusting for comments

18943ef


		export DPL_REPORT_PROCESSING="${DPL_REPORT_PROCESSING:-1}"

		export FST_TMUX_NO_EPN="${FST_TMUX_NO_EPN:-1}"


		export ALICE_O2_FST="${ALICE_O2_FST:-1}"

		if [[ -f "$GEN_TOPO_MYDIR/setenv.sh" ]]; then


		export RUN_BENCHMARK="${RUN_BENCHMARK:-0}"

		echo "# Alien/JAliEn environment check:"

Conversation

ChSonnabend commented Jun 12, 2026

Uh oh!

davidrohr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alibuild commented Jun 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants