layers repacker feature

[azr]

Description

TLDR: when you take images with big layers and split up these layers into smaller chunks. It makes download+unpacking way faster. Could Builkit do that for us ?

@co42 hacked together a test tool that takes an image with big layers, and splits up the layers into smaller ones. Just by moving files around.

To give you an example, if you take vllm/vllm-openai:v0.9.0 – GZIP layers. A few layers are very big ( ~2GB, 3GB, ~5GB ), some layers are smaller with a total compressed size of 10.31GB. A ctr i pull takes about 54.02s. Most of the time is spent on that one big layer, checksuming it + unpacking it, because only one cpu core can work on it.

Now if we 'repack' that image into smaller more numerous zstd layers, a ctr i pull takes ~15s with the same conf/machine. Because the unpacking and disk are just never taking a break and we are using more cores. The granularity of unpacking is great ( machine has many cores & parallel unpacking needs to be on )

# on a r6id.8xlarge instance.
# containerd config:
    max_concurrent_downloads = 32
    concurrent_layer_fetch_buffer = 8 388 608
    max_concurrent_unpacks = 10 # without concurrent unpack it's not such a great gain.

repacked vllm 0.9 layers layout :

Config:
  mediaType: application/vnd.oci.image.config.v1+json
  size:      12504 bytes (12.3KiB) [12.6kB]
Layers:
  application/vnd.oci.image.layer.v1.tar+zstd sha256:12b79940943b62ad18ba9b4f3228582f92c8d376b16dee0c284e9c89b5548496
    size:   32577664 bytes (31.1MiB) [32.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:b56aba350315d9f4f17ca0859b9e72792c415702fefbfb1e44348607224c69e5
    size:   4953980 bytes (4.8MiB) [5.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c13bdd4f4fe7fdd475985236db041284cfda77e7857ed7e7497b08b43b282387
    size:   64446055 bytes (61.5MiB) [64.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:9fec13fdb3e41f6aea33a78d8a2938da0142b17256fff21c26cefce27f3070e1
    size:   152 bytes (152.0B) [152.0B]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:cc4b78d3a10dc87a553b27ba3aa9b6b35ff5a0d8d73d820fe66310db32b481a2
    size:   7647 bytes (7.5KiB) [7.7kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:e298e992978e0baf5f4a20adcbb38d27bd3f75098f83f7941640b9e3dd1f825a
    size:   477594055 bytes (455.5MiB) [477.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:ec4e536fc14020f2e38a5eb186520ac075a2ae3ef4069f4dc5b8174e9f64c6ef
    size:   261543356 bytes (249.5MiB) [261.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c966edd080d6520e6b9421d1d6d5065af02a7d3afa7225d4829e1da635ff3f42
    size:   279645972 bytes (266.7MiB) [279.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:91758cc0d0fe25ef3e05efbe4e2eb8f6c3964798ee31dd9a75733dea4635ab22
    size:   197506986 bytes (188.4MiB) [197.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c74b483a7f47ae48c5a7dad62951b36db55fdcad8a8e739057a3c160c0c1aef5
    size:   156279670 bytes (149.1MiB) [156.3MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:0d90196578be380183fbbb192f11f3a4fd0c58f1b17406cec26397095f99e390
    size:   97410440 bytes (92.9MiB) [97.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:d97e6b0e7c82033e442458b5978486bba1f8b603d0b15b6240637bbc0e4e7034
    size:   59509846 bytes (56.8MiB) [59.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4834768b8d2df0f32efbb69c11bfdf34d6c9978a406ac1686d9f7bd1939c5b73
    size:   86736247 bytes (82.8MiB) [86.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:b72dda36c0ce246adc3cb0a77da0aeac6019c067da7135388c30c24c1277b7db
    size:   85460819 bytes (81.6MiB) [85.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:fa0017f6f132519219ce63797d7b91f3c1ec940c03d2c04200efdc609ad506a7
    size:   46725598 bytes (44.6MiB) [46.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:0d242eab9d3d514e3911c1a1cf0a94c1bed5f50c224b65b75e15b7db70844e95
    size:   46708493 bytes (44.6MiB) [46.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:7210718aeca99838809de454c071041b7e0671bb3493d21f827edf24651d59de
    size:   42974134 bytes (41.0MiB) [43.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4a532b0a7010860d19cf3ea550c4da8e49a6d1e95e98366b1c28d61a63c4f2b4
    size:   42583664 bytes (40.7MiB) [42.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:321a5073b7e3d5eefb02d1cc8fbb86d04c56097745ee26f4781a892aed22a216
    size:   41971082 bytes (40.1MiB) [42.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:2dfd75333e87bccdb8360d062c954e8e61a9b17499742f7c35a0f4e93a3a880e
    size:   32364212 bytes (30.9MiB) [32.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:f338886ebb34fba1945ca9cd482cf77df0a9b018f557165258f10f38a1f4551c
    size:   35950 bytes (35.2KiB) [36.0kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:2ed6aacb52b3fd843c237eb5b60351336547c053e8deda2b473bff1c31843ade
    size:   1758 bytes (1.8KiB) [1.8kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:614700e4690781a9c7e91ca31053f1a05cd55285f199d93bf6b83288c033991d
    size:   1615 bytes (1.6KiB) [1.7kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:cabcb8e68487edd3496fb182ea53b58144a1309aa6704b68f6d8b1159b1f7470
    size:   529623998 bytes (505.1MiB) [529.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:32cb26f45bff981c37dc620e0474f782de0ab4ce6e1d760a9423913b0a56e615
    size:   267918870 bytes (255.6MiB) [268.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:695937165f2c5395368f6eb64ef9f33889f5db2d93e6b0744693b5796bb57881
    size:   280193834 bytes (267.3MiB) [280.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:d321cd5f189ce314ef73f9d06a2ad5ea9f35b2120e007029145ceaf3c8e05fd2
    size:   198713138 bytes (189.6MiB) [198.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:a0aaab857713a5f6921b7c511b7f8157d954dcde90ee3a06df0b8fb996e6debe
    size:   198645828 bytes (189.5MiB) [198.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:a7862db23c551ee9d67e933b9cac7f03c254aa507fa79dd0fd0d118b1d599b79
    size:   159721279 bytes (152.4MiB) [159.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:3572c4c03d66781afc5c653ac59f5b6544cc50c192e98b23b19b8ee2e1f2f389
    size:   78675617 bytes (75.1MiB) [78.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:bcd7e94a4c83fce3c55d22318f94c63e42deb284493c2e0695150b22e2dc18c3
    size:   89801191 bytes (85.7MiB) [89.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:25167183b6946731977ca30c8b3fc5ead5660da39a26494b529093c6956faf02
    size:   61253759 bytes (58.5MiB) [61.3MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4a840d9be4d5e226612e8d1e1d73aba8d5522f042e16a64b1d32b2f7ebcd36af
    size:   42647672 bytes (40.7MiB) [42.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:57974174d78524c1c18bc53b1c361b59cecb15812ae2d482423fc04c85f4ebcd
    size:   89988201 bytes (85.9MiB) [90.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:56cfaac5a4b0088e5ee9e2bdaa4dfcd4aac347a836dc8bab2f59ad892c814fbf
    size:   46420306 bytes (44.3MiB) [46.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:e8f0d8c174d6156dfc298c1ff8f7492447cd3e300214e608d59ff606aa715ec4
    size:   46132729 bytes (44.0MiB) [46.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c9b188747e7f99b55f267c588cf5f5691679a2c9ecb88aa6c963e136c5bc6823
    size:   48682376 bytes (46.5MiB) [48.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c7d1a303ac2fe67a179d6fc0b1bacc02dd8bc32985f0e16442cc40e45c22b65f
    size:   63088711 bytes (60.2MiB) [63.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:128e7f7860dbc2a9bcc460d8819cd799be1fc466f50ec639caeb67d0b0d4ecc0
    size:   60312271 bytes (57.6MiB) [60.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:674c13d426b48861f77bc8396be3e143e3440743264dda943d8b88bac02ecbce
    size:   53935103 bytes (51.5MiB) [54.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:1d4e6452eeb87c09e00d8d88f2ef69f9306ca23669b26d47ab74a587708cbd09
    size:   55812195 bytes (53.3MiB) [55.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4badc8d7408ef08cc2ad2bf52ede05b5c01be4d8ee2f14db3488bb547e95382a
    size:   64475824 bytes (61.5MiB) [64.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:5fa718e8a3feeb3c00ba03a83480cb7c93c191523c3036b5a17826cf974b53b7
    size:   46471074 bytes (44.4MiB) [46.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:bebc9489ccb1caa5b08fb26af7db7678d53684f988235824e3176fd935075013
    size:   63251446 bytes (60.4MiB) [63.3MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:d89b84418724d014640fa145902720e8f27ab57f8a0d574cd1d36481199c5b7a
    size:   48969521 bytes (46.8MiB) [49.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:86e8790d1bd5a9fc3bdb6aee2530001d9fdd2214d8ddaa855b488677633a4a6a
    size:   36087998 bytes (34.5MiB) [36.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:ff4a77a203bdd56b6ec5c24a34e16e7211db555d4f0714849829f43193d3a4e3
    size:   51731522 bytes (49.4MiB) [51.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:52e0793c14e13f6a0452426fdb23cd3c6349f0e0e9a15cfa3f96b15c49fa7965
    size:   60852189 bytes (58.1MiB) [60.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c793e11e55446d6624b3d7192823d55f3c32ca76fd4012cd6ba127aa0a6d4075
    size:   53809149 bytes (51.4MiB) [53.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:9d26131a6fc014706e5f379012665024d56730d122ed2605f939ac6b9d183c36
    size:   48800187 bytes (46.6MiB) [48.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c7830aa4d715573ff02a3542582719f69b2ecdb2f5d1568e67be05eabb4568fc
    size:   44764886 bytes (42.7MiB) [44.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:b7cb3ed0b10dc8d71e013d5f3ba110434cbcd17b6726b5ea30edd7fb7e267b24
    size:   48615 bytes (47.5KiB) [48.7kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:5b43a7c83df3d39488128637f6889cad5b51e4e82ca746bc3596b1453053b154
    size:   92 bytes (92.0B) [92.0B]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:bc0180791c6b385a76115a320a30d574843d2ad28867a478709a00cf0c9bfc2e
    size:   204 bytes (204.0B) [204.0B]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:184a0255ddbea9be6af97da8e271c4f670658a05198639eef10ca9d9b2267bd0
    size:   66608074 bytes (63.6MiB) [66.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:b69ec30839b7cfa0ab5883bc70a428ba809e4e4c97739a16945906ab869eda53
    size:   74135934 bytes (70.8MiB) [74.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:6bed59606d0f39ba7be4701da7af9522350d761658e930b46d0a13c263c8beff
    size:   76695715 bytes (73.2MiB) [76.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:02cbf797a0874f6007e6c31d16fd74d2af277c61d61b15b55ccfc5bf2fae2d91
    size:   90166762 bytes (86.0MiB) [90.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:0c2fcd4f439346769fd462a74fee882036af789392afa8161c15497ad5bf5c50
    size:   92116960 bytes (87.9MiB) [92.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4c5458d279499bd77e8d617c64f5a0a0903525c60c29909d6b6ee7d5908a95ad
    size:   73065712 bytes (69.7MiB) [73.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:10793e4afcac7ee02b41e4831f1eef8cea84ac924622b66c5696eabaf172f718
    size:   77035422 bytes (73.5MiB) [77.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:1c232fd2cf247d128c273cba0f3a7f3ef8e4a981c303f831a9796ccb8df81af5
    size:   36611829 bytes (35.0MiB) [36.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c16952263d1cdbdb7482f410b0375ba27f8c3c63266dbe8e445a6bcc8d8fdad9
    size:   12720 bytes (12.5KiB) [12.8kB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:e4148e64308a66125a0845d33b66e7216a9f678d76e36a63855f57919b15e99f
    size:   98 bytes (98.0B) [98.0B]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:dff4dc2c3c5637e6ba4f93298c578c9125fe564364c0142f94bace2a4dbc8d26
    size:   817969244 bytes (780.1MiB) [818.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:5fc9b8e88e3a0cc1aeb98a05830ce2c52ad03339f119968fa7acbaa4fe76938d
    size:   489571002 bytes (466.9MiB) [489.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:aadc65755bf8463c42254b5ab9554b75745d89621aa45e396e022ce52ee0ee48
    size:   342994867 bytes (327.2MiB) [343.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:5a27adac4b2a12ffeb9c551b4f4ee6c909c5ad8cf1579991432e4cdeac329c1a
    size:   103872709 bytes (99.1MiB) [103.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:34c98e62a40dd45f1872b0b6e79e8ade47109e0cd40fc9dd1286d5d636bc19d6
    size:   112081206 bytes (106.9MiB) [112.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:3162d81a924ceafdf05676f7cec6bdcf036ab547537175d7b43eb1fd0e6df3bc
    size:   85993363 bytes (82.1MiB) [86.0MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:5e8ecec4fbef70f7902ccb1503b11beff28068c1e1c29100c7d2869374fafe3a
    size:   265383543 bytes (253.1MiB) [265.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:6fb943192336bd9f6e926fe3a227defeb085ba5f531058764948b6b0628bad5b
    size:   96045364 bytes (91.6MiB) [96.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:65a13315a1f3e15d8372b02a1efcb337a5969128aaa03a10e972fddd1f67148b
    size:   230379755 bytes (219.8MiB) [230.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:36114b2394585291d4c4c9f8e5ab41d240ef942556efa0de29fb4c168507b9b8
    size:   197513704 bytes (188.4MiB) [197.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:b4b950e7142fd64c1641ac21dd71a7413f864e4af2bb8d69c55dfed96dfe0b63
    size:   190787170 bytes (182.0MiB) [190.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:1d4599356f47917b5fba67309fa74e127aba80e284599ae5ee4c1be69dd07467
    size:   110700889 bytes (105.6MiB) [110.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:4a30773fe18183bf489f3deec5403f4fda9789521ab739c213446abb41344275
    size:   151161323 bytes (144.2MiB) [151.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c60ffcc915b35028074cf567a582e69b296681e7cdb29dda846b6cfcc0af6644
    size:   152581477 bytes (145.6MiB) [152.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:bc743d2b02179ef99ec91de8c433bbb059e3c31165d665b669e32d2b7d27090a
    size:   93898537 bytes (89.6MiB) [93.9MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:3d964fdc51a9911c265e51c84db4de4aa348ec3090e1cec83a4632ad69b4ae3a
    size:   59539533 bytes (56.8MiB) [59.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:87ec45fe34e160efd6f32515776e93e74b49b1fb1a7f30a2a5855c0f3178aa58
    size:   46478242 bytes (44.4MiB) [46.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:be836a1f5e1ae5586ca42bbbc86b45980cc60055c9472f46a7784c916a2e2356
    size:   77416625 bytes (73.9MiB) [77.5MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:56376bd4c0b3d276da780debe56a36493c73007aa70d3b8716b04830fa736936
    size:   90299503 bytes (86.2MiB) [90.3MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:c12a6db7f2fa71ecfb01896443becd89b965d2c8d855553e0bd2985d2fde4463
    size:   89592821 bytes (85.5MiB) [89.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:598c2c284c328b49238c787afee897b92d9129a59c984a7c5e3670deae964c26
    size:   54753696 bytes (52.3MiB) [54.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:541c9e6eb4a576192966d5787405a39726f9ba5d8802ecbf9c899ee374b91831
    size:   55108248 bytes (52.6MiB) [55.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:81408865fff875d6550baa38d2c64b0f2f4d58af02365d8a223a0501047bc8c8
    size:   39320462 bytes (37.5MiB) [39.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:984b84c809ed63e519a8e4e117645e2fa500bbaecd2feb6868e5ece8143dab2a
    size:   56045925 bytes (53.5MiB) [56.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:d7bdfcaaa5675e6c28bebb91d1695aaf1d59da2a2013455a70776f52f3607f63
    size:   49500246 bytes (47.3MiB) [49.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:a0c84d9f24c9a7e3eeb2e6dc364708a3f653442281f78dcc0c4dadd10e16cfaa
    size:   43639053 bytes (41.7MiB) [43.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:09344ced11bfa99b9b46ea2eea553b3fb7aacb9babf12cea7c818809e9717c40
    size:   36100331 bytes (34.5MiB) [36.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:77f25d00057010a06557597e315554c6c3b6b47efc8a2ff64a550cf390de015f
    size:   37664350 bytes (36.0MiB) [37.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:197c75ae2aa47b694c0fe7cbcd5ba52fd9012863d17389592631ef3a8d54df66
    size:   33557384 bytes (32.1MiB) [33.6MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:8532a2b070ff02edaefba70f03b1db736221025384ff19681da4a1b09f847063
    size:   62356448 bytes (59.5MiB) [62.4MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:ca2880c9880c2353952d0899baadc2a0b1207c4b83f81113f3ef40338baba408
    size:   33274154 bytes (31.8MiB) [33.3MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:98de6aeded445902443ecc7eeb26bf41d15bbf39e997eb7eb4e3e8cccbd6c199
    size:   57102204 bytes (54.5MiB) [57.2MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:f49051dcf2ac387d5f173902896044adc075ae3e8d1ef780d6a8ec01e5c452b6
    size:   48722964 bytes (46.5MiB) [48.8MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:006fc6eb9f4435bc049e8ffbb96873ab603e5fd77f075e465803aa5c82e46714
    size:   42022398 bytes (40.1MiB) [42.1MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:1d0b3818b6ce194dd64edbcf96cd752ed155adc7847b72c0cca9c328f67fe35f
    size:   32633073 bytes (31.2MiB) [32.7MB]
  application/vnd.oci.image.layer.v1.tar+zstd sha256:8a6e4376594bcc35e91c994e2afd1abb9352fb174b136305365369441248d91a
    size:   63034844 bytes (60.2MiB) [63.1MB]

Total:      10083427567 bytes (9.4GiB) [10.1GB]

This is somewhat a bit similar to what nix would be doing, but like, without having to deal with nix, and also it would allow to repack already built images.

[polarathene]

Most of the time is spent on that one big layer, checksuming it + unpacking it, because only one cpu core can work on it.

So your mainly interested in the ability for multi-threaded checksum and unpacking (decompression?), if that's specifically where you're seeing a perf issue, rather than splitting an image into smaller "repacked" chunks to reconstruct once pulled, wouldn't it be wiser to just improve the checksum and codec algorithms used that cause the issue for you in the first place? Especially since I assume this repack involves reprocessing a built image to (re)publish anyway?

ZSTD should already be capable of multi-threaded decompression which AFAIK you can already use for publishing (and thus subsequently pulling instead of gzip), while for checksums if whatever is presently used lacks support for such, if you can land support for BLAKE3 / b3sum (bao may also be of interest) that'd probably sort that out?

At a glance SHA256 can't parallelize checksum verification like BLAKE3 can. You could take an approach like this which is roughly what you're asking, but rather than split the file/layer into multiple pieces to reconstruct again it just creates multiple SHA256 over chunks of the file content directly, followed by a final SHA256 of those checksums.

Last I recall there is a limit to how many layers an image can have, so for your approach you would need to be mindful of that.

---

Existing solutions

It seems that you're probably better off with using existing third-party features? For example if you utilize the eStargz snapshotter (see ecosystem support):

https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md

The solution for fast image distribution is called Remote Snapshotter plugin. This prepares the container's rootfs layers by directly mounting from remote stores instead of downloading and unpacking the entire image contents.
The actual image contents can be fetched lazily so runtimes can start containers before the entire image contents are locally available. We call these remotely mounted layers remote snapshots.

Relevant docs on chunking / optimizing for larger files:

Large regular files in an eStargz blob MAY be chunked into several smaller gzip members. Each chunked member is called chunk in this doc.

eStargz contains a regular file called TOC which records metadata (e.g. name, file type, owners, offset etc) of all file entries in eStargz, except TOC itself. Container runtimes MAY use TOC to mount the container's filesystem without downloading the entire layer contents.

Lazy pulling can cause runtime performance overhead by on-demand fetching of each file. eStargz mitigates this by supporting prefetching of important files called prioritized files.

Stargz snapshotter provides an image converter command ctr-remote images optimize to create optimized eStargz images. When converting the image, this command runs the specified workload in a sandboxed environment and profiles all file accesses. This command treats all accessed files as prioritized files. Then it constructs eStargz by

For gzip they have support for using a helper to parallelize gzip with. There also appears to be support for zstd:

• https://github.com/containerd/nerdctl/blob/main/docs/stargz.md#tips-3-using-zstd-instead-of-gzip-aka-zstdchunked
• Document eStargz with zstd compression instead of only gzip containerd/stargz-snapshotter#1596
• NOTE: The zstd support isn't via CGO to the proper zstd lib, it uses a Go implementation which is apparently 50%-ish slower?.

For AI specific workloads like you're describing with large binaries involved, there is this recent issue with the same interest that has some info and comparison tables about how eStargz is performing vs the usual OverlayFS you'd have by default. They noticed a notable delay at runtime, probably due to how some large content was loaded on-demand.

This article should give you a better overview of eStargz and how to actually get started with it 👍

There's a few other alternatives available (see the "on-demand image pulling" section) if you want to look into those and compare to eStargz and see what works best for you.

[azr]

Hey @polarathene, thanks for these !!!

eStargz

We have explored eStargz, and it mostly made things similar and worse in some cases.
The thing with images like vllm or tgi, is that we can't predict what files will be used. Depending on the hardware/machine/model/etc. different files will be used, sort of making this optimisation a bit moot.

While 54.02s was 'too much' for vllm, 15s post-repack is so much more okay that - in our case - image pull times would no longer be an issue we saturate bandwidth and everything is there. Moreover if we can spend time optimising the layers by access patterns by booting the container, why not just make the layers sizes more balanced ? The solution is a bit more simple, and the rebalance wouldn't require a GPU ( in our case ).

Docker limit on layers count

As a mater of fact, I have - just for experimentation - built a nix image of TGI with 309 layers and everything was fine, pulling worked, starting worked, and things were fast. It also made the download drastically faster. I wonder if there could be some system limitations to this like, too many open files or something. So far things were fine 'in a lab'.

But yeah with that said limiting to ~100 layers was the fastest in most cases, and I think balancing the layer size would be the best thing to do (currently working on that in our repacker thing).

Conf'able checksumming

Would be such a great improvement, but, as fas as I can tell, would be a huge huge hard change, because looking at different codebases, a lot of things expect a sha256: and everyone would need to update, etc.

Moreover, for example with containerd, we get the layers from a store and io.Copy them with a io.Reader and write to a file, and the checksumming is piped in the middle of this, so I expect the parallelisation of blake3 to be sort of minimal here? Maybe not, have to try.

Here, having more layers really makes things fast because we use more cpu-cores, and everything just remains so simple and backwards compatible.

I'll see if I can PR something quickly, but I don't expect a lot from this.
Places in buildkit with hardcoded sha256: to look at:

buildkit/cache/contenthash/filehash.go

Line 65 in d5a40ad

tsh := &tarsumHash{hdr: hdr, Hash: sha256.New()}

buildkit/cache/contenthash/checksum.go

Line 1223 in d5a40ad

return digest.NewDigest(digest.SHA256, h), nil

Compression

We found zstd to be the fastest thing with containerd, most of the time, and managed to make it faster for containerd, here containerd/containerd#11335 by turning off the low mem decoder. Maybe this setting could be turned on to fix containerd/stargz-snapshotter#2168 (comment) ?

[azr]

Update here: I have briefly checked how to change the checksumming, and it would be a massive PR + it would require a change in a lot of places in the ecosystem because corners were taken and a lot of hot paths expect "sha256:", in a lot of places. Moreover, other priorities keep coming up. I wish I had more time ^^'.