Bugfix: Pydantic deserialization for FlyteFile and FlyteDirectory #3339

gverkes · 2025-10-13T08:45:59Z

Fixes #6669

Tracking issue

Why are the changes needed?

When deserializing Pydantic models containing FlyteFile or FlyteDirectory fields using model_validate(), the deserialized objects were
missing private attributes (_remote_source, _downloader, etc.). This caused an AttributeError when attempting to re-serialize these
objects with model_dump(), breaking the serialize → deserialize → serialize cycle.

This is a critical bug that prevents users from using FlyteFile/FlyteDirectory within Pydantic BaseModel classes in a normal way.

What changes were proposed in this pull request?

Added Pydantic model validators to both FlyteFile and FlyteDirectory classes that ensure private attributes are properly initialized during
deserialization:

FlyteFile (flytekit/types/file/file.py):
- Enhanced deserialize_flyte_file validator to check if private attributes exist
- If missing, reconstructs the FlyteFile using dict_to_flyte_file() transformer
- If attributes already exist (e.g., when passing already-constructed FlyteFile), returns as-is
FlyteDirectory (flytekit/types/directory/types.py):
- Applied same fix to deserialize_flyte_dir validator
- Uses dict_to_flyte_directory() to properly reconstruct the object

How was this patch tested?

Added two new unit tests in test_pydantic_basemodel_transformer.py:

test_flytefile_pydantic_model_dump_validate_cycle - Verifies FlyteFile can be serialized, deserialized, and re-serialized without errors
test_flytedirectory_pydantic_model_dump_validate_cycle - Same for FlyteDirectory

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Summary by Bito

This pull request introduces a critical bug in the deserialization of Pydantic models containing FlyteFile and FlyteDirectory fields, which could lead to AttributeErrors during re-serialization.
Model validators are added to ensure that private attributes are properly initialized during deserialization.
New unit tests have been added to verify the functionality of these changes, ensuring that the serialize-deserialize-serialize cycle works without errors.
Overall, this pull request addresses deserialization issues in Pydantic models, introduces critical bugs, and adds unit tests.

welcome · 2025-10-13T08:46:03Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

fg91 · 2025-10-13T17:04:11Z

flytekit/types/directory/types.py

        if info.context is None or info.context.get("deserialize") is not True:
-            return self
+            # Check if all private attributes are already set up (e.g., from __init__)
+            if hasattr(self, "_downloader") and hasattr(self, "_remote_source"):


Nit:
Do we need to check for the other private vars like _downloaded too?

Or will they all together be set or not set anyways?

fg91 · 2025-10-13T17:08:12Z

flytekit/types/file/file.py

+
+            dict_obj = {"path": str(self.path)}
+            metadata = getattr(self, "metadata", None)
+            if metadata is not None:


Sanity check:
@gverkes did you test whether we need to do this for flyte directory too?

fg91

I tested and can confirm that the fix works, thanks for fixing 🙏

My approval appears to be not enough to merge, I'm not a code owner for those dirs. Could you please take a look @davidmirror-ops or @pingsutw ? 🙏

codecov · 2025-10-14T06:53:02Z

Codecov Report

❌ Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.03%. Comparing base (531d2aa) to head (f16d4c1).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/types/directory/types.py	75.00%	0 Missing and 1 partial ⚠️
flytekit/types/file/file.py	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3339      +/-   ##
==========================================
- Coverage   76.21%   76.03%   -0.18%     
==========================================
  Files         216      216              
  Lines       22700    22718      +18     
  Branches     2975     2981       +6     
==========================================
- Hits        17301    17274      -27     
- Misses       4540     4582      +42     
- Partials      859      862       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fixes #6669 Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: Govert Verkes <[email protected]>

flytekit/types/directory/types.py

…y validators Signed-off-by: Govert Verkes <[email protected]>

Extract hasattr check into _was_initialized_via_init() helper for clarity. Remove redundant dict_to_flyte_* calls - fall through to to_python_value instead. Signed-off-by: Govert Verkes <[email protected]>

machichima · 2025-12-09T07:40:27Z

HI @gverkes,
Feel free to ping me when this PR is ready for review. Thanks!

machichima

Overall looking good! Just few nits.
Thank you!

tests/flytekit/unit/extras/pydantic_transformer/test_pydantic_basemodel_transformer.py

flytekit/types/directory/types.py

flytekit/types/file/file.py

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: gverkes <[email protected]>

…basemodel_transformer.py Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: gverkes <[email protected]>

machichima

LGTM! Thank you!

machichima · 2025-12-22T02:42:22Z

@gverkes Could you please fix the CI? Thank you

gverkes requested review from cosmicBboy, davidmirror-ops, kumare3, machichima, pingsutw, samhita-alla and wild-endeavor as code owners October 13, 2025 08:46

fg91 reviewed Oct 13, 2025

View reviewed changes

fg91 previously approved these changes Oct 13, 2025

View reviewed changes

gverkes force-pushed the fix/pydantic-flytefile-deserialization branch from 82990a6 to 628cf96 Compare October 13, 2025 19:23

gverkes dismissed fg91’s stale review via e279566 October 18, 2025 13:46

gverkes force-pushed the fix/pydantic-flytefile-deserialization branch from e279566 to f4a16e4 Compare October 18, 2025 13:49

fg91 approved these changes Oct 23, 2025

View reviewed changes

Fix Pydantic deserialization for FlyteFile and FlyteDirectory

e68b4a1

Fixes #6669 Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: Govert Verkes <[email protected]>

gverkes force-pushed the fix/pydantic-flytefile-deserialization branch from f4a16e4 to 0b4bf36 Compare December 1, 2025 10:29

machichima reviewed Dec 3, 2025

View reviewed changes

flytekit/types/directory/types.py Outdated Show resolved Hide resolved

wild-endeavor reviewed Dec 3, 2025

View reviewed changes

flytekit/types/directory/types.py Outdated Show resolved Hide resolved

gverkes added 2 commits December 6, 2025 23:15

Add tests to improve codecov coverage for FlyteFile and FlyteDirector…

f160262

…y validators Signed-off-by: Govert Verkes <[email protected]>

Simplify Pydantic deserialization with helper method

38bf633

Extract hasattr check into _was_initialized_via_init() helper for clarity. Remove redundant dict_to_flyte_* calls - fall through to to_python_value instead. Signed-off-by: Govert Verkes <[email protected]>

gverkes force-pushed the fix/pydantic-flytefile-deserialization branch from 8d7c203 to 38bf633 Compare December 6, 2025 22:15

gverkes requested review from machichima and wild-endeavor December 9, 2025 07:42

machichima reviewed Dec 9, 2025

View reviewed changes

tests/flytekit/unit/extras/pydantic_transformer/test_pydantic_basemodel_transformer.py Outdated Show resolved Hide resolved

flytekit/types/directory/types.py Outdated Show resolved Hide resolved

flytekit/types/file/file.py Outdated Show resolved Hide resolved

gverkes and others added 3 commits December 18, 2025 10:53

Update flytekit/types/file/file.py

51424ec

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: gverkes <[email protected]>

Update flytekit/types/directory/types.py

0820f13

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: gverkes <[email protected]>

Update tests/flytekit/unit/extras/pydantic_transformer/test_pydantic_…

6bcdbef

…basemodel_transformer.py Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Govert Verkes <[email protected]> Signed-off-by: gverkes <[email protected]>

gverkes force-pushed the fix/pydantic-flytefile-deserialization branch from f16d4c1 to 6bcdbef Compare December 18, 2025 10:54

gverkes requested a review from machichima December 18, 2025 10:55

machichima approved these changes Dec 18, 2025

View reviewed changes

Merge branch 'master' into fix/pydantic-flytefile-deserialization

23c296f

Bugfix: Pydantic deserialization for FlyteFile and FlyteDirectory #3339

Are you sure you want to change the base?

Bugfix: Pydantic deserialization for FlyteFile and FlyteDirectory #3339

Uh oh!

Conversation

gverkes commented Oct 13, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Check all the applicable boxes

Summary by Bito

Uh oh!

welcome bot commented Oct 13, 2025

Uh oh!

fg91 Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

fg91 Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

fg91 Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

fg91 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

machichima commented Dec 9, 2025

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

machichima commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gverkes commented Oct 13, 2025 •

edited by flyte-bot

Loading

fg91 left a comment •

edited

Loading

codecov bot commented Oct 14, 2025 •

edited

Loading