Allow cancellation even when allow_interruptions is False

[chenghao-mou]

Feature Type

Nice to have

Feature Description

This stems from the conversations in #4418 and #4413.

The core issue is that when we create a speech that has allow_interruptions set to False, we don't have a way to reliably stop or cancel it before it's played, even if the customer continues to speak after it's being scheduled. This also makes the cancellation of a tool call with say/generate_reply invoked indeterministic, esp since the introduction of #4102

Scenario 1: (current behavior)

• user says something
• agent speech (allow_interruptions=False) scheduled
• user continues speaking, preventing the agent speech from being played
• agent speech plays when the user stops speaking, but new user input is skipped

In this case, the agent would appear missing input

Scenario 2: (old behavior)

• user says something
• agent speech (allow_interruptions=False) scheduled
• user continues speaking
• agent speech plays, overlapping with user's speech

Here, the agent would appear "impatient"

Scenario 3: (ideal behavior)

• user says something
• agent speech (allow_interruptions=False) scheduled
• user continues speaking, leading to cancellation of the scheduled agent speech
• new agent speech (allow_interruptions=False) scheduled and then played

Workarounds / Alternatives

• restore old behavior by checking speech handle flags; or
• introduce new ways of speech cancellation

Additional Context

No response

[davidzhao]

IMO, allow_interruptions should only apply to user interruptions.. if the user speaks before the agent starts speaking, then we should not consider it interruptions.

also if the developer uses interrupt(), it should always be allowed..

[longcw]

I think Scenario 2: (old behavior) makes more sense. when a developer schedule a uninterruptible speech I don't think they are expecting it will be cancelled by user speech in any case. If they really want it, they can call the session.interrupt(force=True) in a event callback when user state changed to speaking.

for the scenario 3, it's just postponed the speech, and if the speech was from session.say, the content won't be updated no matter what user said which feels like the agent ignored the user's input.

[chenghao-mou]

I created a PR #4434 to restore the old one for now.

In the examples, scenario 1 is intended for the delaying case, and scenario 3 would have a new user transcript, thus a new "turn".

Do we expect the same "expectations" when the speech is scheduled from both locations?

1. normal agent turns, internally calling generate_reply
2. user explicitly calls generate_reply or say

I feel like in the first case, allow_interruptions=False means that any course correction ("I will have the fish and chips, please... You know what, a Diet Coke too") wiil be missed even before the agent "responds" and repeated.

How about buffering the transcripts and scheduling them for the next turn? We already do something similar for barge-in.

[martin-purplefish]

Maybe it needs to be an "interruption behavior" instead of true/false?

Thinking to our use case, a few scenarios:

1. sometimes, we want the speech to be guaranteed to play and uninterrupted. For example, at the beginning of a call - if the user speaks first, we still want our speech to play.
2. sometimes, we just want it to be uninteruptable unless the user speaks. once it plays, it shouldn't be interruptable.
3. sometimes we just want it to be interruptable of course.

[davidzhao]

when a developer schedule a uninterruptible speech I don't think they are expecting it will be cancelled by user speech in any case

this is true in general, though the issue is deciding on who's turn it is. What I mean is that this should be treated similarly to preemptive synthesis, i.e. if we've started playing agent's speech, then I agree it should not be interruptible.. but if the user speaks before we are able to playout audio (say it was due to TTS synthesis), then to me we would need to yield to user speech

[davidzhao]

I think the tricky case comes in with 2. say if you have a function tool, that gets triggered and reads out a consent that you don't want to be interrupted.

then doing session.say(..., allow_interruptions=False), if the user isn't done with their turn, we probably do not want to interrupt that speech.. because there's no way to retrigger that function call. (currently)

[longcw]

@davidzhao

but if the user speaks before we are able to playout audio (say it was due to TTS synthesis), then to me we would need to yield to user speech

my main concern is for the first use case @martin-purplefish mentioned, the greeting message should be played without waiting for user's speech.

maybe it needs to be three options:

1. always allow interruption
2. never
3. before playing.

but for the third case do we want to reschedule the speech automatically after the user turn, if so it may conflict with the next agent turn?

[martin-purplefish]

Yeah I think that's right. Building on what you guys said, I wonder if there's a delay case? Just random thoughts, but some things we run into:

1. Our first message we make uninterruptible because some people answer the phone and don't stop talking (and then hang up when nobody interrupts them 🙃). So I want to ignore the user and just talk.
2. "This call will be recorded." - I need this always to play in full for compliance reasons.

At least for our use case though - i'm struggling to come up with an example where I'd want #3 - if we wanted to allow interruptions even before playing, we'd just make the message itself interruptable.