Serialization formats

[jaapschoutenalliander]

For PGM-DS we are looking into serialisation of Grid objects. PGM supports json & msgpack on which we could be build. However, we are also still considering parquet or numpy-binary. What were your considerations for supporting json and msgpack?

[TonyXiang8787]

Hi @jaapschoutenalliander,

numpy-binary is python/numpy specific, PGM core is in general language acoustic (with universal C-API).

As far as I know, parquet cannot store multiple tables with different length and width inside one file. Also it misses the function to store sparse batch dataset, metadata, etc.

We use json and msgpack because they are both well-known serialization format. We define our data model on top of the serialization.

[mgovers]

@jaapschoutenalliander,

We have also considered Arrow as a serialization format using a star schema structure. We did document our initial findings in https://power-grid-model-io.readthedocs.io/en/stable/examples/arrow_example.html as an example as to how to do this on the Python side, but we have not talked about officially supporting it yet.

Note that the original investigation happened before we had support for columnar data sets, which we now support in the PGM core as well.

[Thijss]

Things to consider for me:

• Storage effficiency
• Performance (write & read times)
• Required dependencies