Skip to content

[test](pipeline) collect Cloud P0 resource usage #65125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hello-stephen merged 1 commit into from
Jul 3, 2026
Merged

[test](pipeline) collect Cloud P0 resource usage #65125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions regression-test/pipeline/cloud_p0/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ need_collect_log=false

# monitoring the log files in "${DORIS_HOME}"/regression-test/log/ for keyword 'Reach limit of connections'
_monitor_regression_log &
start_system_resource_monitor
trap 'stop_system_resource_monitor' EXIT

# shellcheck disable=SC2329
run() {
Expand Down Expand Up @@ -114,6 +116,7 @@ timeout_minutes=$((${repeat_times_from_trigger:-1} * ${BUILD_TIMEOUT_MINUTES:-18
timeout "${timeout_minutes}" bash -cx run
exit_flag="$?"
if print_running_pipeline_tasks; then :; fi
stop_system_resource_monitor
# shellcheck source=/dev/null
source "$(cd "${teamcity_build_checkoutDir}" && bash "${teamcity_build_checkoutDir}"/regression-test/pipeline/common/get-or-set-tmp-env.sh 'get')"
if get_jstack_and_jmap_of_fe; then echo "INFO: get_jstack_and_jmap_of_fe done."; fi
Expand Down
75 changes: 75 additions & 0 deletions regression-test/pipeline/common/doris-utils.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,6 +662,81 @@ _monitor_regression_log() {

}

_append_system_resource_snapshot() {
local output_file="$1"
local top_n="${SYSTEM_RESOURCE_MONITOR_TOP_N:-10}"
{
echo "==================== $(date '+%Y-%m-%d %H:%M:%S %z') ===================="
echo "[uptime]"
uptime || true
echo
echo "[free -h]"
free -h || true
echo
echo "[/proc/meminfo]"
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Slab|SReclaimable|SUnreclaim):' /proc/meminfo || true
echo
echo "[ps sorted by rss]"
ps -eo pid,ppid,rss,vsz,%mem,%cpu,comm,args --sort=-rss | head -n "$((top_n + 1))" || true
echo
echo "[ps sorted by cpu]"
ps -eo pid,ppid,rss,vsz,%mem,%cpu,comm,args --sort=-%cpu | head -n "$((top_n + 1))" || true
echo
if command -v top >/dev/null; then
echo "[top]"
COLUMNS=240 top -b -n 1 -w 240 | head -n "$((top_n + 7))" || true
echo
fi
} >>"${output_file}" 2>&1
}

start_system_resource_monitor() {
if [[ ! -d "${DORIS_HOME:-}" ]]; then return 1; fi
if [[ -n "${SYSTEM_RESOURCE_MONITOR_PID:-}" ]] && kill -0 "${SYSTEM_RESOURCE_MONITOR_PID}" 2>/dev/null; then
echo "INFO: system resource monitor already started, pid=${SYSTEM_RESOURCE_MONITOR_PID}"
return 0
fi

local interval="${SYSTEM_RESOURCE_MONITOR_INTERVAL_SECONDS:-60}"
local output_file="${DORIS_HOME}/be/log/system_resource_usage.log"
mkdir -p "$(dirname "${output_file}")"
echo "INFO: start system resource monitor, interval=${interval}s, output=${output_file}"
(
set +e
monitor_sleep_pid=""
monitor_cleanup() {
if [[ -n "${monitor_sleep_pid}" ]]; then
kill "${monitor_sleep_pid}" 2>/dev/null || true
wait "${monitor_sleep_pid}" 2>/dev/null || true
fi
exit 0
}
trap monitor_cleanup TERM INT
while true; do
_append_system_resource_snapshot "${output_file}"
sleep "${interval}" &
monitor_sleep_pid="$!"
wait "${monitor_sleep_pid}"
monitor_sleep_pid=""
done
) &
SYSTEM_RESOURCE_MONITOR_PID="$!"
export SYSTEM_RESOURCE_MONITOR_PID
}

stop_system_resource_monitor() {
if [[ -z "${SYSTEM_RESOURCE_MONITOR_PID:-}" ]]; then return 0; fi
if kill -0 "${SYSTEM_RESOURCE_MONITOR_PID}" 2>/dev/null; then
kill "${SYSTEM_RESOURCE_MONITOR_PID}" 2>/dev/null || true

@github-actions github-actions Bot Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stop_system_resource_monitor only kills and waits for the monitor subshell, but the loop is often blocked in its foreground sleep "${interval}" child. I can reproduce this with SYSTEM_RESOURCE_MONITOR_INTERVAL_SECONDS=123: before stop the monitor owns sleep 123, and after this function returns the monitor PID is gone but that sleep is still alive with PPID 1. In the normal Cloud P0 path this leaves a stray sleep 60 after cleanup, and a custom interval can leave it around much longer. Please have the monitor worker clean up its active child on TERM, or stop the whole monitor process tree/process group, before reporting that the monitor has stopped.

@shuke987 shuke987 Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0f7197f. The monitor worker now tracks the active sleep child and kills/waits it from a TERM/INT cleanup trap before exiting.

wait "${SYSTEM_RESOURCE_MONITOR_PID}" 2>/dev/null || true
echo "INFO: stop system resource monitor, pid=${SYSTEM_RESOURCE_MONITOR_PID}"
fi
if [[ -d "${DORIS_HOME:-}" ]]; then
_append_system_resource_snapshot "${DORIS_HOME}/be/log/system_resource_usage.log"
fi
unset SYSTEM_RESOURCE_MONITOR_PID
}

_redact_creds() {
local expr="" v escaped
for v in "${hwYunAk:-}" "${hwYunSk:-}" "${s3SourceAk:-}" "${s3SourceSk:-}" "${txYunAk:-}" "${txYunSk:-}"; do
Expand Down
Loading