Skip to content

Conversation

@norio-nomura
Copy link
Contributor

@norio-nomura norio-nomura commented Nov 22, 2025

VZVmnetNetworkDeviceAttachment support on macOS 26+

Depends on Code-Hex/vz#205

About what can be done and the restrictions with this API:

  • Supports using vmnet without root privilege
  • Supports VMNET_SHARED_MODE, and VMNET_HOST_MODE
  • Like vzNAT, the "bridge10x" interface is created on the host
  • Not support VMNET_BRIDGED_MODE
  • To share vmnet_network_ref with multiple processes, it must be the same executable file.
    • documented in - initWithNetwork:
    • This restriction seems to be imposed by VZVmnetNetworkDeviceAttachment instead of vmnet_network_create.
    • CDHash is probably used to determine "same executable file". TN3126: Inside Code Signing: Hashes
      • Redoing codesign with "Developer ID", where the hash of the executable file changes, does not interfere with the sharing of vmnet_network_ref.
      • CDHash does not change before and after redoing codesign with "Developer ID".
      • Even if it is codesign using the same "Developer ID", if CDHash is changed due to source code changes, etc., vmnet_network_ref cannot be shared.
  • The network created based on vmnet_network_ref is owned by the process that created vmnet_network_ref, not the process that started the network based on the shared vmnet_network_ref.
    • When the owner process terminates, it also terminates the owned network, regardless of the presence or absence of connected VMs.
  • The serialization of vmnet_network_ref is done in the form of xpc_object_t.
    • It is possible to share byte sequences included in the serialized xpc_object_t outside the XPC API, but to avoid relying on private information, use the XPC API for sharing.

This PR allows VMNET_SHARED_MODE and VMNET_HOST_MODE to be used under these conditions.

Functions to be added

networks.yaml

vz:
  shared:
    mode: shared
    dhcp: true
    dnsProxy: true
    mtu: 1500
    nat44: true
    nat66: true
    routerAdvertisement: true
    subnet: 192.168.107.0/24
  host:
    mode: host
    dhcp: true
    dnsProxy: true
    mtu: 1500
    nat44: true
    nat66: true
    # host mode ignores routerAdvertisement setting
    routerAdvertisement: false
    subnet: 192.168.108.0/24

lima.yaml

# requires `vmType: vz` and macOS 26.0 or later.
- vz: shared
- vz: host

limactl vz-vmnet

$ limactl vz-vmnet
Run vz-vmnet

Usage:
  limactl vz-vmnet [flags]

Flags:
  -h, --help                      help for vz-vmnet
      --unregister-mach-service   Unregister Mach service

Global Flags:
      --debug               Debug mode
      --log-format string   Set the logging format [text, json] (default "text")
      --log-level string    Set the logging level [trace, debug, info, warn, error]
      --tty                 Enable TUI interactions such as opening an editor. Defaults to true when stdout is a terminal. Set to false for automation. (default true)
  -y, --yes                 Alias of --tty=false
  • Registration as a Mach service is automatically done by VZ driver when the instance using - vz: * is started, so only unregistering is visible from the user.
  • The registered Mach service is displayed as "io.lima-vm.vz.vmnet.sh" in macOS System Settings (General > Login Items & Extensions).
  • As a Mach service process, it is started by launchd as appropriate at the request of the VZ driver.
  • The Mach service responds to requests only if the executable file of the VZ driver process that sent the request has the same CDHash as itself. It uses xpc_peer_requirement.
  • The Mach service process creates and retains vmnet_network_ref and provides serialization data to VZ driver.
  • When the Mach service detects the termination of the network started by the VZ driver based on the serialization data, it discards vmnet_network_ref.
  • When all the vmnet_network_ref to be retained is gone, the Mach service process will terminate.

Additional changes

  • Because shutdown takes longer on using VmnetNetwork:
    • Extend VZ driver's shutdown timeout from 5 seconds to 15 seconds
    • Add ExitTimeOut key with 20 seconds to autostart io.lima-vm.autostart.INSTANCE.plist

Benchmark: iperf3 between VMs on the same network

MacBook Pro 14 inch, 2023
CPU: Apple M2 Pro
Memory: 16GB
macOS Tahoe 26.2

$ ssh lima-debug ip a show scope global dev lima1|grep -E "^ +inet "
    inet 192.168.105.11/24 metric 100 brd 192.168.105.255 scope global dynamic lima1
$ ssh lima-debug ip a show scope global dev lima3|grep -E "^ +inet "
    inet 192.168.107.8/24 metric 100 brd 192.168.107.255 scope global dynamic lima3
$ ssh lima-docker ip a show scope global dev lima1|grep -E "^ +inet "
    inet 192.168.105.10/24 metric 100 brd 192.168.105.255 scope global dynamic lima1
$ ssh lima-docker ip a show scope global dev lima3|grep -E "^ +inet "
    inet 192.168.107.7/24 metric 100 brd 192.168.107.255 scope global dynamic lima3

socket_vmnet

$ ssh lima-docker iperf3 -c 192.168.105.11
Connecting to host 192.168.105.11, port 5201
[  5] local 192.168.105.10 port 47434 connected to 192.168.105.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   148 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   1.00-2.00   sec   147 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   2.00-3.00   sec   148 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   3.00-4.00   sec   149 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   4.00-5.00   sec   147 MBytes  1.23 Gbits/sec    0   3.97 MBytes       
[  5]   5.00-6.00   sec   149 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   6.00-7.00   sec   148 MBytes  1.24 Gbits/sec    0   3.97 MBytes       
[  5]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec    0   3.97 MBytes       
[  5]   8.00-9.00   sec   148 MBytes  1.25 Gbits/sec    0   3.97 MBytes       
[  5]   9.00-10.00  sec   147 MBytes  1.23 Gbits/sec    0   3.97 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.45 GBytes  1.24 Gbits/sec    0            sender
[  5]   0.00-10.02  sec  1.45 GBytes  1.24 Gbits/sec                  receiver

iperf Done.

vz: shared

$ ssh lima-docker iperf3 -c 192.168.107.8
Connecting to host 192.168.107.8, port 5201
[  5] local 192.168.107.7 port 54642 connected to 192.168.107.8 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  3.38 GBytes  28.8 Gbits/sec    0   4.07 MBytes       
[  5]   1.01-2.00   sec  3.47 GBytes  29.9 Gbits/sec    0   4.07 MBytes       
[  5]   2.00-3.00   sec  3.42 GBytes  29.4 Gbits/sec    0   4.07 MBytes       
[  5]   3.00-4.00   sec  3.45 GBytes  29.7 Gbits/sec    0   4.07 MBytes       
[  5]   4.00-5.00   sec  3.38 GBytes  29.0 Gbits/sec    0   4.07 MBytes       
[  5]   5.00-6.00   sec  3.42 GBytes  29.4 Gbits/sec    0   4.07 MBytes       
[  5]   6.00-7.00   sec  3.37 GBytes  28.9 Gbits/sec    0   4.07 MBytes       
[  5]   7.00-8.00   sec  3.29 GBytes  28.3 Gbits/sec    0   4.07 MBytes       
[  5]   8.00-9.00   sec  3.24 GBytes  27.9 Gbits/sec    0   4.07 MBytes       
[  5]   9.00-10.00  sec  3.45 GBytes  29.6 Gbits/sec    0   4.07 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  33.9 GBytes  29.1 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  33.9 GBytes  29.1 Gbits/sec                  receiver

iperf Done.

@norio-nomura

This comment was marked as outdated.

@AkihiroSuda
Copy link
Member

Currently, both vzShared and vzHost have almost the same functions as vzNAT.

Any advantage using them?

@norio-nomura
Copy link
Contributor Author

Currently, both vzShared and vzHost have almost the same functions as vzNAT.

Any advantage using them?

As far as the API is concerned, customization that is not supported by vzNAT should be possible.
https://github.com/Code-Hex/vz/pull/205/files#diff-d7ec6a7a97f55e264883085a4676c2b0a5466a657e0e87312accd40caa67295dR80-R106
I still don't know what can be done with those APIs (including unimplemented).

@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from 07eebec to 270556e Compare December 3, 2025 07:59
@norio-nomura

This comment was marked as outdated.

@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 2 times, most recently from 81d4cc6 to 57553f8 Compare December 4, 2025 01:54
@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 4 times, most recently from 433432d to adf5456 Compare December 15, 2025 05:32
@norio-nomura

This comment was marked as resolved.

@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from adf5456 to aa15ed7 Compare December 16, 2025 16:09
Based on `VMNET_SHARED_MODE`, and `VMNET_HOST_MODE`
```yaml
networks:
- vzShared: true
- vzHost: true
```
But, to sharing network between multiple VMs, `VZVmnetNetworkDeviceAttachment` requires VMs are launched by same process.

It depends on Code-Hex/vz#205

Signed-off-by: Norio Nomura <[email protected]>
@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch 7 times, most recently from d560fae to 4cd78ff Compare December 17, 2025 03:16
It provides `VmnetNetwork` serialization to VMs.
`limactl vz-vmnet` takes flags:
- `--unregister-mach-service`: unregister Mach service from `launchd`
  - There is no need to register manually because the VZ driver registers as appropriate.

- `--mach-service=<service name>`: launched as Mach server by `launchd` via `io.lima-vm.vz.vmnet.plist`
  - Launched on demand to connection from VZ driver by `launchd`.
  - Receives a request payload from VZ driver with fields:
    - `Network`: name of the network ("shared", "host", etc)
    - `Configuration`: `[]bytes@ representing `VzNetworkConfig` in JSON.
  - Validates clients are the same executable (cdhash) by using xpc_peer_requirement API.
  - Create `VmnetNetwork` from provided `Configuration` if cached one does not exist.
  - Replies to VZ driver with fields:
    - `Configuration`: If `VmnetNetwork` is cached, it may be created by different configuration.
    - `Serialization`: newly created or cached.
  - Monitors changes of networks
    - When the interface created by `VmnetNetwork` disappears from host, remove them from cache.
    - If all `VmnetNetwork` are removed, `limactl vz-vmnet` exits.

VZ driver (hostagent) does:
- Read `.vz` VzVmnetConfig from `networks.yaml`
- Use them on `- vz: <network>` fields; "shared" and "host" network are predefined.
- Register `limactl vz-vmnet` to `launchd` if not registered.
- Request serialization to the Mach service "io.lima-vm.vz.vmnet".
- Create `VmnetNetwork` by provided serialization, then use them.

Additional changes:
- Because shutdown takes longer on using `VmnetNetwork`:
  - Extend VZ driver's shutdown timeout from 5 seconds to 15 seconds
  - Add `ExitTimeOut` key with 20 seconds to autostart `io.lima-vm.autostart.INSTANCE.plist`
- `lima.yaml`: `- vzShared` and `- vzHost` are renamed to `- vz: shared` and `- vz: host`

Signed-off-by: Norio Nomura <[email protected]>
@norio-nomura norio-nomura force-pushed the use-vz-vmnet-network-device-attachment branch from 4cd78ff to 8cc5f26 Compare December 17, 2025 04:07
@norio-nomura
Copy link
Contributor Author

Updated PR description.

@norio-nomura
Copy link
Contributor Author

Most of the functions I wanted at the start of this PR have been realized.

@norio-nomura norio-nomura marked this pull request as ready for review December 17, 2025 04:52
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
)

replace github.com/Code-Hex/vz/v3 => github.com/norio-nomura/vz/v3 v3.7.2-0.20251217001012-3b512d7782b0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep this PR draft until the Code-Hex/vz PR gets merged and released

@AkihiroSuda AkihiroSuda added this to the v2.1.0 (?) milestone Dec 17, 2025
@AkihiroSuda AkihiroSuda marked this pull request as draft December 17, 2025 05:09
@norio-nomura norio-nomura marked this pull request as ready for review December 17, 2025 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants