Kotlin coroutine instrumentation span hierarchy broken

[patschl]

Describe the bug

Under specific circumstances, when instrumenting Kotlin coroutines or suspending methods, the span hierarchy seems to be broken. Calls to suspending methods appear to not only end the current span but also the parent span, resulting in subsequent spans being created with the wrong parent.
To be specific this happens when calling suspending functions not annotated with @WithSpan and then calling functions annotated with @WithSpan. This issue apparently does not occur when annotating every function in the call stack with @WithSpan.

Steps to reproduce

A specific example is this call stack:

@WithSpan
fun main() {
    runBlocking {
        println("root - span: ${Span.current()}")
        first()
    }
}

@WithSpan
suspend fun first() {
    println("first - span: ${Span.current()}")
    second()
}

suspend fun second() {
    println("second 1 - span: ${Span.current()}")
    third()
    println("second 2 - span: ${Span.current()}")
    third()
}

@WithSpan
suspend fun third() {
    println("third - span: ${Span.current()}")
    delay(200)
}

When executing this code, the second invocation of third() will be a child span of the root span instead of the span created by the invocation of first().

Example code can be found here: https://github.com/patschl/otel-coroutines-reproduce

Expected behavior

The expected result is (simplified output):

root - span: spanId=rootSpan, parentSpanContext=ImmutableSpanContext{spanId=0000000000000000...}
first - span: spanId=intermediateSpan, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}
second 1 - span: spanId=intermediateSpan, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}
third - span: spanId=leafSpan1, parentSpanContext=ImmutableSpanContext{spanId=intermediateSpan...}
second 2 - span: spanId=intermediateSpan, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}
third - span: spanId=leafSpan2, parentSpanContext=ImmutableSpanContext{spanId=intermediateSpan...}

I expect one root span, which in turn has one child span, which in turn has two leaf spans as children.
Each invocation of third() should have the same parent, the intermediate span.

Actual behavior

The actual result is (simplified output):

root - span: spanId=rootSpan, parentSpanContext=ImmutableSpanContext{spanId=0000000000000000...}
first - span: spanId=intermediateSpan, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}
second 1 - span: spanId=intermediateSpan, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}
third - span: spanId=leafSpan1, parentSpanContext=ImmutableSpanContext{spanId=intermediateSpan...}
second 2 - span: spanId=rootSpan, parentSpanContext=ImmutableSpanContext{spanId=0000000000000000...}
third - span: spanId=leafSpan2, parentSpanContext=ImmutableSpanContext{spanId=rootSpan...}

Up until the first invocation of third() everything seems fine, but after returning to second() the current span is the root span instead of the intermediate span.

To clarify, adding @WithSpan to fun second() fixes this issue and everything works as expected.

Javaagent or library instrumentation version

opentelemetry-instrumentation-bom=2.10.0

Environment

No response

Additional context

No response