Root module (non relative) import does not work well from submodules

[angerpointnerd]

First off, this package is really cool and I enjoy using it a lot :)

I think I found a bug in the 0.9 version of this package that I can reproduce with the setup below. The last cell of the example notebook errors when I run it with PlutoDevMacros version 0.9.1 but didn't error on 0.8.1. Or is what I'm doing not supposed to work in the first place?

What I'm trying to do is use an object whose type is defined directly in the top-level module of MyPackage, but which was constructed in a submodule MyPackage.SubModule. In 0.9.1, this object seems to be different to Pluto somehow so that methods with argument type restrictions don't accept it anymore. Also the test t2 == t returned true on 0.8.1, but false 0.9.1 (so the whole MethodError below is probably just the symptom of this).

Reproduce:

1. create a dummy package julia -e 'using Pkg; Pkg.generate("MyPackage")
2. add content to MyPackage/src/MyPackage.jl with some struct MyThing and a submodule Submodule

module MyPackage

struct MyThing
    a::Int
end

get_a(t::MyThing) = t.a

function do_stuff(t::MyThing)
    a = get_a(t)
    return a * rand()
end

module SubModule

    using MyPackage: MyThing

    function construct_thing(a)
        return MyThing(a)
    end

end

end

3. create test environments MyPackage/notebooks/PDM_0.8.1 and run inside julia --project=. -e 'using Pkg; Pkg.add(name="PlutoDevMacros", version="0.8.1")' (and analogous for version 0.9.1).
4. create a notebook MyPackage/notebooks/notebooks.jl with

### A Pluto.jl notebook ###
# v0.20.20

using Markdown
using InteractiveUtils

# ╔═╡ f6cd2c49-2318-46a1-a945-57b688a96735
begin
	using Pkg
	Pkg.activate("PDM_0.8.1")
end

# ╔═╡ ddaf67f2-e383-41c1-aa79-8f76d8871d1e
using PlutoDevMacros

# ╔═╡ e964e090-0ba5-4e1c-9126-499fc6718257
@frompackage ".." begin
	using MyPackage
end

# ╔═╡ 8542d7d4-1ef1-473f-aed5-984b5d43372b
t = MyPackage.MyThing(1)

# ╔═╡ 6469268a-65f9-494c-97c8-f2603df95296
t2 = MyPackage.SubModule.construct_thing(1)

# ╔═╡ 75f20939-9dd4-4cc6-bc3a-65488399d440
t2 == t

# ╔═╡ d698b4e7-08c8-4fbf-968f-56c1aed8991c
MyPackage.do_stuff(t)

# ╔═╡ ac91061c-e01c-489f-92e5-e6f016ff9c4f
MyPackage.do_stuff(t2)

# ╔═╡ Cell order:
# ╠═f6cd2c49-2318-46a1-a945-57b688a96735
# ╠═ddaf67f2-e383-41c1-aa79-8f76d8871d1e
# ╠═e964e090-0ba5-4e1c-9126-499fc6718257
# ╠═8542d7d4-1ef1-473f-aed5-984b5d43372b
# ╠═6469268a-65f9-494c-97c8-f2603df95296
# ╠═75f20939-9dd4-4cc6-bc3a-65488399d440
# ╠═d698b4e7-08c8-4fbf-968f-56c1aed8991c
# ╠═ac91061c-e01c-489f-92e5-e6f016ff9c4f

5. run the notebook and change between Pkg.activate("PDM_0.8.1") and Pkg.activate("PDM_0.9.1")

For 0.8.1 I get no errors, for 0.9.1 I get

MethodError: no method matching do_stuff(::MyPackage.MyThing)
The function `do_stuff` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  do_stuff(::Main.var"workspace#3".var"##TempModule#283".MyPackage.MyThing)
   @ Main.var"workspace#3" .../MyPackage/src/MyPackage.jl:11

I've tested this on Julia 1.12 (where 0.8.1 gives a warning and should probably not be used) but also on Julia 1.10 with the same result.

It might be a bit of a niche error, but it would be very useful if this would work again. For now, I can manually downgrade PlutoDevMacros to 0.8.1, but it's inconvenient to not use the internal Package manager of Pluto just for one package in a large notebook.

I would also be happy to think about a fix if you could point me in the right direction (at the moment, I have no idea how this package works internally though 😅 )

[disberd]

Hello!
Nice to hear that you find this package useful :).

At a first glance I don't really find an easy reason why this errors so I'd have to go a bit deeper to try understand this.

The current package code is a bit complicated and I wanted to try simplifying it for a bit, so not strange that you have no idea how this works internally :D.

When I get some time to try pick this up I'll check if its reasonable for you to help with a fix and let you know :)

[disberd]

Hi @angerpointnerd,

I actually started putting down the test package for solving this in #69 and it seems the problem is simpler than expected, and it just boils down to, currently you need to use relative imports from submodules.

I still want to fix this to support root module identifier directly without relative path but in the meantime you should be able to fix your code by adding the .. in the import of the submodule.

Hopefully #69 would be a relatively easy fix so you can also get some glimpse of internals by looking at the PR changes

[angerpointnerd]

Awesome, thank you for fixing this so quickly :)

For the notebook where I originally discovered this issue, I am still running Julia 1.10 (where the issue seems to persist) but I've been planning to transition that project to 1.12 for a while now anyway – this is a good occasion 😄

[disberd]

That is strange as CI also test with julia LTS and I added the test notebook explicitly for this. Are you sure that in your original notebook you had updated PlutoDevMacros?

[angerpointnerd]

I'm pretty sure I updated the package and restarted the notebook, but I will double-check later and let you know!

[angerpointnerd]

Yeah, sorry for the confusion: The test notebooks I made when creating a MWE are indeed working now also on Julia 1.10.

But for some reason, the original notebook where I stumbled upon this issue still seems to have the same kind of method error I mentioned in the original post (after updating to version 0.9.2) 🤔 I will try to figure out what else could be going on here... (the full notebook and the package which I'm importing are quite large so I don't think it will be useful to post them here).

[angerpointnerd]

OK, I think I narrowed down the problem: In my actual notebook, I don't just do @frompackage, but also directly include a script that defines some plotting functions and uses the structs from the main package (which is probably not good practice in the first place 😅 ).

So the basic structure looks like this

.
├── Manifest.toml
├── Project.toml
├── notebooks
│   └── notebook.jl
├── script.jl
└── src
    └── MyPackage.jl

And in script.jl I do something like

using MyPackage: do_stuff

function test_stuff(thing)
    return do_stuff(thing)
end

and finally include("../../script.jl") within the notebook. I get a MethodError:

MethodError: no method matching do_stuff(::MyPackage.MyThing)

Closest candidates are:

  do_stuff(::Main.var"workspace#3".var"##TempModule#231".MyPackage.MyThing)
   @ Main.var"workspace#3" .../MyPackage/src/MyPackage.jl:9

Come to think of it, I'm surprised that this even worked in the first place with 0.8 (the using MyPackage in the script is not wrapped in the macro 😅

What's also funny is that this works on 0.9.2 if explicitly call MyPackage.do_stuff instead of using only the method name:

using MyPackage

function test_stuff(thing)
    return MyPackage.do_stuff(thing)
end

For now I think I will just change the plotting script to use the second style (or just remove the using MyPackage from the script, it should not be necessary 🤔 ), but I feel like I'm not really thinking about my code structure right, so below are some thoughts of how one could avoid this. Feel free to ignore it though – I think this is not really in the scope of this package 😅

---

I guess possible solutions would be creating either a separate small package for the plotting code or an extension of the main package. Both feel kind of awkward when using them with the Pluto notebook and PlutoDevMacros. I've seen your page about using extensions https://disberd.github.io/PlutoDevMacros.jl/stable/frompackage/package_extensions/#Package-Extensions but my issue is that I want to define completely new functions in the plotting script and use them, which doesn't seem to be the intended use case for extensions.

Is it somehow possible to use @frompackage on multiple packages (e.g. the main package and the small plotting package? I tried it and it doesn't seem to work, which is probably intended?

Or is there another solution for when I just want to use the same couple of functions in different notebooks, but not copy-paste the code to each one (since the functions still change from time to time)?

[disberd]

Hi @angerpointnerd yeah in that case it kinda makes sense things don't work properly.

This is indeed the perfect use case for extensions, I wonder why you couldn't simply define a function in the main package (without methods) and then add a method to it within the extension.

Similarly to what I do in this other package https://github.com/JuliaSatcomFramework/GeoPlottingHelpers.jl/blob/main/src%2Fapi.jl#L129.

That way you can properly support plotting from your package without having the user manually including a script. Even if at the moment you are the only user of the package I find this approach to be oftentimes more convenient than creating a completely separate script unless it's for some very fast plots you only have to do once or twice, but in that case you are probably better suited to have it directly in the notebook.
If you have a script it means you probably already use it enough to justify an extension.