[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail. by daviftorres · Pull Request #13424 · apache/cloudstack

daviftorres · 2026-06-15T19:57:40Z

Description

This PR tried to prevent the failure of the job at the statistics section of a backup that has actually succeeded.

Apparently, it also fixes some silent failures I previously reported in #11727

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

daviftorres · 2026-06-15T20:50:13Z

This is the equivalent command for applying the fix:

sed -i 's_du -sb $dest | cut -f1_du -sb $dest 2>/dev/null | cut -f1 || true_g' /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh

We haven't confirmed the exact root cause of the du failure yet. As a precaution, we applied this fix to all servers and will monitor backups over the next few days.

So, I am running tests with 2>>/var/log/cloudstack/agent/nasbackup.err so I can see what is the error message.

daviftorres · 2026-06-17T14:08:49Z

Proposed Changes Rationale

backup_size=$(du -sb "$dest" 2>/dev/null | cut -f1) || true

NFS issues may cause du command to fail.
A size retrieval failure should not invalidate a successful backup.

timeout 60 umount "$mount_point" 2>/dev/null || true
rmdir "$mount_point" 2>/dev/null || true

Another process may keep the device busy (e.g., parallel backups).
Network issues may cause hangs on NFS.
Cleanup failures should not invalidate a successful backup.

echo -n "$backup_size"

Outputs the size at the end to confirm the script completed past the potentially problematic commands.

daviftorres · 2026-06-17T18:59:55Z

Dear @abh1sar , do you think you can help me with this bug? Regards,

codecov · 2026-06-18T11:27:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.93%. Comparing base (5ed4894) to head (664b2d5).
⚠️ Report is 41 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13424      +/-   ##
============================================
- Coverage     18.94%   18.93%   -0.01%     
- Complexity    18376    18473      +97     
============================================
  Files          6192     6221      +29     
  Lines        556550   560039    +3489     
  Branches      67954    68285     +331     
============================================
+ Hits         105454   106066     +612     
- Misses       439517   442347    +2830     
- Partials      11579    11626      +47

Flag	Coverage Δ
uitests	`3.50% <ø> (-0.02%)`	⬇️
unittests	`20.14% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adjusts the KVM NAS backup script’s “statistics/cleanup” section so that failures while computing backup disk usage (and related cleanup commands) don’t cause an otherwise successful backup job to be marked as failed.

Changes:

Capture du output into backup_size and suppress du stderr to avoid failing the script during size calculation.
Add timeout around umount and suppress errors from umount/rmdir.
Emit the computed backup size at the end of backup_running_vm().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

DaanHoogland

clgtm

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

abh1sar

hi @daviftorres
would it be possible to test script changes in your env where it is reproducible?
Some log outputs would be nice.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

daviftorres · 2026-06-25T13:48:46Z

would it be possible to test script changes in your env where it i

2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Request:Seq 218-281474976710814:  { Cmd , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-84-6493-VM","backupPath":"i-84-6493-VM/2026.06.25.13.22.42","backupRepoType":"nfs","backupRepoAddress":"10.4.2.145:/mnt/VAN3-NAS01-STOR-POOL-A1/VAN3-NAS01-DS-01","wait":"0","bypassHostMaintenance":"false"}}] }
2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Processing command: org.apache.cloudstack.backup.TakeBackupCommand
2026-06-25 13:25:04,120 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Seq 218-281474976710814:  { Ans: , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 110, [{"org.apache.cloudstack.backup.BackupAnswer":{"size":"(47.67 GB) 51190241222","result":"true","details":"Job type:         Completed

Nothing else that I can find in the logs are related to the backup job.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

abh1sar · 2026-06-26T01:50:43Z

@blueorangutan package

blueorangutan · 2026-06-26T01:52:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2026-06-29T14:44:51Z

@daviftorres , can you look at https://github.com/apache/cloudstack/actions/runs/28373728590/job/84079353133?pr=13424#step:7:11412 ?

davift · 2026-06-29T15:14:09Z

@daviftorres , can you look at https://github.com/apache/cloudstack/actions/runs/28373728590/job/84079353133?pr=13424#step:7:11412 ?

Hey @DaanHoogland , sure!

trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

If I understand it right, it is caused by the echo command that produces a new line by default. I had -n in the past, but I accepted GitHub Copilot advise to remove it to prevent some error. Honestly, I am not sure if it is related or not. I am re-adding the argument now and will see if the build fails again.

Please point me in the right direction if you see that I am going the wrong way. Regards!

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

abh1sar · 2026-07-02T11:52:16Z

If I understand it right, it is caused by the echo command that produces a new line by default. I had -n in the past, but I accepted GitHub Copilot advise to remove it to prevent some error. Honestly, I am not sure if it is related or not. I am re-adding the argument now and will see if the build fails again.

Please point me in the right direction if you see that I am going the wrong way. Regards!

it's not due to echo
we should revert back to echo from echo -n

The extra whitespaces were at line 200 and 203.
I'm take the liberty to incorporate these as code suggestions and reruns GHA. Hope you don't mind

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

abh1sar

LGTM

abh1sar · 2026-07-02T14:35:39Z

@blueorangutan package

blueorangutan · 2026-07-02T14:38:05Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-07-02T15:25:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18442

abh1sar · 2026-07-02T15:53:07Z

@blueorangutan test

blueorangutan · 2026-07-02T15:54:03Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-07-03T07:01:48Z

[SF] Trillian test result (tid-16468)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13424-t16468-kvm-ol8.zip
Smoke tests completed. 152 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

boring-cyborg Bot added the component:kvm label Jun 15, 2026

github-actions Bot mentioned this pull request Jun 15, 2026

[repo-status] Daily Status Report — June 15, 2026 #13425

Closed

daviftorres marked this pull request as ready for review June 16, 2026 15:02

github-actions Bot mentioned this pull request Jun 16, 2026

[repo-status] Daily Status Report — June 16, 2026 #13435

Closed

Copilot AI review requested due to automatic review settings June 18, 2026 20:07

Copilot started reviewing on behalf of daviftorres June 18, 2026 20:07 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

DaanHoogland approved these changes Jun 19, 2026

View reviewed changes

DaanHoogland added the status:needs-testing label Jun 19, 2026

DaanHoogland requested a review from abh1sar June 23, 2026 06:53

Copilot AI review requested due to automatic review settings June 23, 2026 12:20

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilot started reviewing on behalf of daviftorres June 23, 2026 12:31 View session

abh1sar reviewed Jun 23, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilot AI review requested due to automatic review settings June 24, 2026 20:55

Copilot started reviewing on behalf of daviftorres June 24, 2026 20:55 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

abh1sar reviewed Jun 25, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilot AI review requested due to automatic review settings June 25, 2026 13:19

Copilot started reviewing on behalf of daviftorres June 25, 2026 13:21 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilot AI review requested due to automatic review settings June 26, 2026 01:49

Copilot AI reviewed Jun 26, 2026

Change echo to echo -n for backup size output

8dea747

Copilot AI review requested due to automatic review settings June 29, 2026 15:15

Copilot started reviewing on behalf of daviftorres June 29, 2026 15:17 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh

abh1sar reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

remove extra whitespace

5af17c2

Copilot AI review requested due to automatic review settings July 2, 2026 11:53

remove extra whitespace

830fbac

Copilot started reviewing on behalf of abh1sar July 2, 2026 11:53 View session

revert to echo

200c40e

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilot AI review requested due to automatic review settings July 2, 2026 11:56

Copilot started reviewing on behalf of daviftorres July 2, 2026 11:56 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

abh1sar reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Comment thread scripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Update scripts/vm/hypervisor/kvm/nasbackup.sh

cce95d4

Copilot AI review requested due to automatic review settings July 2, 2026 12:12

Update scripts/vm/hypervisor/kvm/nasbackup.sh

664b2d5

Copilot started reviewing on behalf of abh1sar July 2, 2026 12:14 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

abh1sar approved these changes Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

daviftorres commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

daviftorres commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daviftorres commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes Rationale

Uh oh!

daviftorres commented Jun 17, 2026

Uh oh!

codecov Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

daviftorres commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented Jun 26, 2026

Uh oh!

blueorangutan commented Jun 26, 2026

Uh oh!

DaanHoogland commented Jun 29, 2026

Uh oh!

davift commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

daviftorres commented Jun 15, 2026 •

edited

Loading

daviftorres commented Jun 15, 2026 •

edited

Loading

daviftorres commented Jun 17, 2026 •

edited

Loading

codecov Bot commented Jun 18, 2026 •

edited

Loading