Skip to content

Ap 722 copy files#78

Merged
steve-sullivan merged 9 commits into
mainfrom
AP-722-copy-files
Jun 30, 2026
Merged

Ap 722 copy files#78
steve-sullivan merged 9 commits into
mainfrom
AP-722-copy-files

Conversation

@steve-sullivan

Copy link
Copy Markdown
Contributor

No description provided.

@davezuckerman davezuckerman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm know versed enough in python to comment on any stylistic issues. For copying files (maybe in a round 2) we might want to use something like rsync where the checksum validation is baked in.

Comment thread mokelumne/util/file_transfer.py Outdated

@anarchivist anarchivist left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally looks good; a few minor refactoring things to cleanup, and otherwise a few questions. i think this is pretty close, so i'll give it a r+wc / deciding how we're going to address the questions.

Comment thread mokelumne/dags/copy_files.py Outdated
Comment thread mokelumne/dags/copy_files.py Outdated
Comment thread mokelumne/dags/copy_files.py
Comment thread test/unit/test_file_transfer.py Outdated
relative_path = item.relative_to(source_path)
manifest.append(
{
"path": str(relative_path),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason why you're preferring relative path here? my inclination would be to use the absolute path to ensure there's a lack of ambiguity, but happy to hear your thoughts otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the manifest is intended to describe the directory tree being transferred (not necessarily an absolute location). I'm not married to the idea, I can change it if you think it would be better...or I can add the source root separately:

{
"source_root": "/the/root/directory",
"files": [
{
"path": "test4.txt",
"size": 5,
"sha256": "..."
},
{
"path": "otherother/test.txt",
"size": 4,
"sha256": "..."
}
]
}

Think I should make one of those changes now or next iteration?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd add the source root info somewhere now - i think it's useful context to have in the manifest if we're trying to diagnose issues.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done - added "source_root":

{
  "source_root": "tmp/source",
  "files": [
    {
      "path": "test4.txt",
      "size": 5,
      "sha256": "b9cca56a720f2beee61f2e744ab3d20a95772a4315d18c5eee251a465f078012"
    }
  ]
}

Comment thread mokelumne/dags/copy_files.py
Comment thread mokelumne/dags/copy_files.py Outdated
Comment thread mokelumne/dags/copy_files.py Outdated

@awilfox awilfox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a few small things, but overall looking really nice.

Comment thread mokelumne/dags/copy_files.py Outdated
Comment thread mokelumne/dags/copy_files.py
Comment thread mokelumne/dags/copy_files.py Outdated
@steve-sullivan steve-sullivan merged commit 5f9bfc1 into main Jun 30, 2026
5 checks passed
@steve-sullivan steve-sullivan deleted the AP-722-copy-files branch June 30, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants