Feature Request: Make `SecureSocket` TLS APIs More Flexible for Protocols with Encapsulated Handshakes (e.g., SQL Server TDS)

[insinfo]

I am implementing a pure TDS driver in Dart for Microsoft SQL Server, and I ran into a limitation in the current SecureSocket API that makes it very difficult (or practically impossible) to support protocols that encapsulate the TLS handshake inside their own application framing, such as SQL Server TDS 7.x.

This feature request asks for a more flexible TLS API, comparable to what Go, Node.js, Java, and .NET expose, so that Dart code can plug the TLS engine into a custom transport instead of being tied to the internal socket implementation.

Background: Protocols With TLS Inside Their Framing

Some protocols do not start with “raw” TLS on the wire. Instead, they:

1. Perform an initial negotiation using their own protocol messages (e.g., a PRELOGIN phase).
2. Negotiate whether encryption/TLS should be used, and at what level.
3. Encapsulate TLS handshake messages (ClientHello, ServerHello, etc.) inside protocol frames.
4. Optionally, after authentication, they may disable TLS and return to cleartext on the same TCP connection (for legacy “login only” encryption modes).

In SQL Server:

• In TDS 7.x, the TLS handshake is carried inside TDS PRELOGIN (0x12) messages.
• The client takes the bytes produced by the TLS engine and wraps them in TDS packets, and vice versa.
• This requires the client to have a finer-grained control over the TLS I/O than “just wrap a socket”.

In many runtimes, the TLS engine is intentionally designed to be “pluggable”:

• Go: tls.Client(conn net.Conn, ...) works with any type that implements net.Conn, so you can create a custom connection that performs custom framing and run crypto/tls on top of it.
• Node.js: tls.TLSSocket can be created on top of any Duplex stream, allowing a custom transport that wraps/unwraps bytes into protocol frames.
• .NET: SslStream(Stream innerStream, ...) can work with any Stream implementation, not only NetworkStream, enabling adapter streams for framed protocols.
• Java: SSLEngine / SSLSocket APIs allow similar integration with custom channels/streams.

This sort of flexibility is what makes it possible to implement protocols like “TDS + TLS” cleanly in those ecosystems.

Current Limitation in Dart (SecureSocket + Socket)

The Dart public API suggests one can pass any Socket to:

Future<SecureSocket> SecureSocket.secure(
  Socket socket, {
  String? host,
  SecurityContext? context,
  bool Function(X509Certificate certificate)? onBadCertificate,
  List<String>? supportedProtocols,
  ...
});

However, the current implementation of SecureSocket.secure internally casts the incoming Socket to the concrete _Socket implementation and calls a private method to detach the underlying RawSocket. In practice this means:

• If I create a wrapper like class TdsSocketAdapter implements Socket that intercepts reads/writes for framing/unframing TLS bytes inside TDS packets, it will not work, because it is not the concrete _Socket type that SecureSocket.secure expects.
• I cannot insert my own adapter between the TLS engine and the underlying RawSocket using only the public APIs.

In other words, there is currently no official extension point that allows:

• Running the TLS engine on top of a custom transport (protocol adapter).
• Feeding the TLS engine with arbitrary Stream<List<int>> + StreamSink<List<int>> instead of a concrete Socket/RawSocket implementation.

This makes it very hard, or impossible in pure Dart, to implement protocols like:

• SQL Server TDS 7.x with handshake encapsulated in PRELOGIN.
• Protocols that have STARTTLS-like semantics but with custom binary framing.
• Any protocol where TLS records must be handled explicitly before hitting the network.

What Other Runtimes Allow

For comparison, today I can:

• In Go: create a type that implements net.Conn, performing TDS framing internally, and then pass it to tls.Client; the TLS engine only cares about read/write of bytes, not about the underlying socket type.
• In Node.js: create a custom Duplex stream that intercepts data and pass it to tls.TLSSocket.
• In .NET: implement a custom Stream and pass it to SslStream.
• In Java: use SSLEngine with custom I/O loops over channels or streams.

The common pattern is: the TLS engine talks to a generic transport interface, not to a fixed, internal socket type, and the user is free to implement/adapt the transport to match protocol requirements.

Proposed Improvements for Dart’s TLS API

It would be extremely helpful for driver and protocol implementers if Dart exposed a similar level of flexibility. A few possible directions (not mutually exclusive):

1. Relax the Concrete Type Requirement in SecureSocket.secure(Socket)

Ensure that any object that implements Socket can be used, without requiring that it be the internal _Socket type.

Internally, instead of casting to the concrete class and calling private APIs, provide a documented, public way to obtain a generic “transport” from a Socket, or refactor the implementation to work purely through public interfaces.

This alone might be enough to allow implementers to create Socket adapters that perform protocol framing while still being usable with SecureSocket.

2. Introduce a Stream-Based TLS API

Alternatively (or additionally), provide a TLS API that works directly on streams, something conceptually like:

abstract class TlsTransport {
  Stream<List<int>> get incoming;
  StreamSink<List<int>> get outgoing;
}

Future<SecureSocket> SecureSocket.secureTransport(
  TlsTransport transport, {
    String? host,
    SecurityContext? context,
    bool Function(X509Certificate certificate)? onBadCertificate,
    List<String>? supportedProtocols,
    ...
  }
);

With an API like this, a protocol implementation could:

• Implement TlsTransport on top of TDS framing (or any other binary protocol).
• Let the TLS engine manage handshake, record protection, renegotiation, etc.
• Take care only of how TLS records are injected into/removed from the on-the-wire format.

3. Provide a Lower-Level “Record Engine” API

Another option (more advanced, but very powerful) would be a low-level API similar to Java’s SSLEngine or OpenSSL’s BIO-based APIs, where code can:

• Feed raw bytes from the network (or a framed protocol) into a method,
• Obtain decrypted application data from another,
• And generate TLS records to be sent out when given application data.

This is more work to design and implement, but it would solve not only TDS, but any protocol where TLS needs to be embedded in a non-trivial way.