RTPListener doesn't work

[cwcsh95]

Thank you for your lib before writing issue.

i tried to make sample in window with this library and it make a successful.

so i saw your reference main.go
and i setup ffmpeg in window in order to use h264decoder.go

this link is ffmpeg library (it's good with window)
ffmpeg-master-latest-win64-gpl.zip

and i change the cgo flag like this

// #cgo CFLAGS: -I D:/Camera/ffmpeg-master-latest-win64-gpl-shared/include
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libavcodec.dll.a
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libavutil.dll.a
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libswscale.dll.a
// #include <libavcodec/avcodec.h>
// #include <libavutil/imgutils.h>
// #include <libswscale/swscale.h>
import "C"

until that, it is my development environment. and from down here, i think it is issue.

with your sample, it should have worked normally.
but, in my environment, RTCP callback function work well but RTP callback function doesn't called. :(

so i debuged with goland, i found the expected part.
in client.go with 713 - 716 Line, it call function start (clientudpl.go)
[client.go 713-716]
for _, ct := range c.tracks {
ct.udpRTPListener.start(true)
ct.udpRTCPListener.start(true)
}
[clientudpl.go start]
func (u *clientUDPListener) start(forPlay bool) {
u.running = true
u.pc.SetReadDeadline(time.Time{})
u.readerDone = make(chan struct{})
go u.runReader(forPlay)
}

in this start function, it called goroutine u.runReader.
I don't know the internal movement of goroutine, but it wasn't called. (only udpRTPListener.start)

so i set delay in client.go, it works well
[client.go 713-716]
for _, ct := range c.tracks {
ct.udpRTPListener.start(true)
++ time.Sleep(time.Second * time.Duration(3)) // at least 3 second. not worked in 2 second
ct.udpRTCPListener.start(true)
}

The exact cause was not found, but symptoms were found so i think it seemed to be reported.

i attach my sample code and i hope it helps to resolve that Symptom.

thank you.

package main

import (
"fmt"
"github.com/aler9/gortsplib"
"github.com/aler9/gortsplib/pkg/base"
"github.com/aler9/gortsplib/pkg/url"
"image"
"image/jpeg"
"os"
"runtime"
"strconv"
"time"
)

func saveToFile(img image.Image) error {
fname := "D://"+strconv.FormatInt(time.Now().UnixNano()/int64(time.Millisecond), 10) + ".jpg" //

f, err := os.Create(fname)
if err != nil {
	fmt.Println("os.Create fname fail ", err)
	os.Exit(1)
}
defer f.Close()

fmt.Println("saving", fname)

// convert to jpeg
return jpeg.Encode(f, img, &jpeg.Options{
	Quality: 60,
})

}

func main() {
runtime.GOMAXPROCS(8)
c := gortsplib.Client{}

// parse URL
u, err := url.Parse("rtsp://myid:[email protected]:554/live4.sdp")
if err != nil {
	fmt.Println("URL Parse ", err)
	os.Exit(1)
}

// connect to the server
err = c.Start(u.Scheme, u.Host)
if err != nil {
	fmt.Println("Start ", err)
	os.Exit(1)
}
defer c.Close()


// find published tracks
fmt.Println("Describe")
tracks, baseURL, _, err := c.Describe(u)
if err != nil {
	fmt.Println("Describe ", err)
	os.Exit(1)
}

// find the H264 track
h264TrackID, h264track := func() (int, *gortsplib.TrackH264) {
	for i, track := range tracks {
		if h264track, ok := track.(*gortsplib.TrackH264); ok {
			return i, h264track
		}
	}
	return -1, nil
}()

if h264TrackID < 0 {
	fmt.Println("H264 track not found")
	os.Exit(1)
}

// setup H264->raw frames decoder
h264dec, err := newH264Decoder()
if err != nil {
	fmt.Println("newH264Decoder ", err)
	os.Exit(1)
}
defer h264dec.close()

// if present, send SPS and PPS from the SDP to the decoder
sps := h264track.SafeSPS()
if sps != nil {
	h264dec.decode(sps)
}
pps := h264track.SafePPS()
if pps != nil {
	h264dec.decode(pps)
}


// Callback Function
saveCount := 0
// Callback Video Udp Data
c.OnPacketRTP = func(ctx *gortsplib.ClientOnPacketRTPCtx) {
	fmt.Printf("[RTP]\n%v, %v, %v\n", ctx.TrackID, ctx.H264PTS, ctx.PTSEqualsDTS) //ctx.Packet, ctx.H264NALUs, is frame Data
	if ctx.TrackID != h264TrackID {
		return
	}

	if ctx.H264NALUs == nil {
		fmt.Println("H264NALUs == nil")
		return
	}

	for _, nalu := range ctx.H264NALUs {
		// convert H264 NALUs to RGBA frames
		img, err := h264dec.decode(nalu)
		if err != nil {
			fmt.Println("h264dec.decode ", err)
			os.Exit(1)
		}

		// wait for a frame
		if img == nil {
			continue
		}
		// ==== Save Image Start ====
		err = saveToFile(img)
		if err != nil {
			fmt.Println("saveToFile ", err)
			os.Exit(1)
		}
		saveCount++
		if saveCount == 5 {
			fmt.Println("saved 5 images Succ")
			os.Exit(1)
		}
		// ==== Save Image End ====

		//fmt.Printf("%v", img) // img data
		fmt.Printf("decoded frame with size %v", img.Bounds().Max)


	}
}
// Callback Rtp Check Data
c.OnPacketRTCP = func(ctx *gortsplib.ClientOnPacketRTCPCtx) {
	fmt.Printf("[RTCP]\n%v, %v", ctx.TrackID, ctx.Packet)
}

// When Response msg from rtsp
c.OnResponse = func(resp *base.Response) {
	fmt.Printf("%v, %v, %v, %v\n", resp.StatusMessage, resp.StatusCode, resp.Header, resp.Body)
}

// When Send smg from rtsp
c.OnRequest = func(req *base.Request) {
	fmt.Printf("%v, %v, %v, %v\n", req.Method, req.URL, req.Header, req.Body)
}

// i wanna set Setup(false, t, baseURL, 0, 0) , so i didn't use SetupAndPlay Func
fmt.Println("Setup")
for _, t := range tracks {
	_, err := c.Setup(true, t, baseURL, 0, 0) // i don't know why i can't set (forPlay == false) in Setup Function.
	if err != nil {
		fmt.Println("hihi", err)
	}
}

fmt.Println("Play")
_, err = c.Play(nil)

// wait until a fatal error
panic(c.Wait())

}

//[decoder.go]
// https://github.com/aler9/gortsplib/blob/main/examples/client-read-h264/h264decoder.go
package main

import (
"fmt"
"image"
"unsafe"
)
// #cgo CFLAGS: -I D:/Camera/ffmpeg-master-latest-win64-gpl-shared/include
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libavcodec.dll.a
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libavutil.dll.a
// #cgo LDFLAGS: -Ld D:/Camera/ffmpeg-master-latest-win64-gpl-shared/lib/libswscale.dll.a
// #include <libavcodec/avcodec.h>
// #include <libavutil/imgutils.h>
// #include <libswscale/swscale.h>
import "C"

func frameData(frame *C.AVFrame) **C.uint8_t {
return (**C.uint8_t)(unsafe.Pointer(&frame.data[0]))
}

func frameLineSize(frame *C.AVFrame) *C.int {
return (*C.int)(unsafe.Pointer(&frame.linesize[0]))
}

// h264Decoder is a wrapper around ffmpeg's H264 decoder.
type h264Decoder struct {
codecCtx *C.AVCodecContext
srcFrame *C.AVFrame
swsCtx *C.struct_SwsContext
dstFrame *C.AVFrame
dstFramePtr []uint8
}

// newH264Decoder allocates a new h264Decoder.
func newH264Decoder() (*h264Decoder, error) {
codec := C.avcodec_find_decoder(C.AV_CODEC_ID_H264)
if codec == nil {
return nil, fmt.Errorf("avcodec_find_decoder() failed")
}

codecCtx := C.avcodec_alloc_context3(codec)
if codecCtx == nil {
	return nil, fmt.Errorf("avcodec_alloc_context3() failed")
}

res := C.avcodec_open2(codecCtx, codec, nil)
if res < 0 {
	C.avcodec_close(codecCtx)
	return nil, fmt.Errorf("avcodec_open2() failed")
}

srcFrame := C.av_frame_alloc()
if srcFrame == nil {
	C.avcodec_close(codecCtx)
	return nil, fmt.Errorf("av_frame_alloc() failed")
}

return &h264Decoder{
	codecCtx: codecCtx,
	srcFrame: srcFrame,
}, nil

}

// close closes the decoder.
func (d *h264Decoder) close() {
if d.dstFrame != nil {
C.av_frame_free(&d.dstFrame)
}

if d.swsCtx != nil {
	C.sws_freeContext(d.swsCtx)
}

C.av_frame_free(&d.srcFrame)
C.avcodec_close(d.codecCtx)

}

func (d *h264Decoder) decode(nalu []byte) (image.Image, error) {
nalu = append([]uint8{0x00, 0x00, 0x00, 0x01}, []uint8(nalu)...)

// send frame to decoder
var avPacket C.AVPacket
avPacket.data = (*C.uint8_t)(C.CBytes(nalu))
defer C.free(unsafe.Pointer(avPacket.data))
avPacket.size = C.int(len(nalu))
res := C.avcodec_send_packet(d.codecCtx, &avPacket)
if res < 0 {
	return nil, nil
}

// receive frame if available
res = C.avcodec_receive_frame(d.codecCtx, d.srcFrame)
if res < 0 {
	return nil, nil
}

// if frame size has changed, allocate needed objects
if d.dstFrame == nil || d.dstFrame.width != d.srcFrame.width || d.dstFrame.height != d.srcFrame.height {
	if d.dstFrame != nil {
		C.av_frame_free(&d.dstFrame)
	}

	if d.swsCtx != nil {
		C.sws_freeContext(d.swsCtx)
	}

	d.dstFrame = C.av_frame_alloc()
	d.dstFrame.format = C.AV_PIX_FMT_RGBA
	d.dstFrame.width = d.srcFrame.width
	d.dstFrame.height = d.srcFrame.height
	d.dstFrame.color_range = C.AVCOL_RANGE_JPEG
	res = C.av_frame_get_buffer(d.dstFrame, 1)
	if res < 0 {
		return nil, fmt.Errorf("av_frame_get_buffer() err")
	}

	d.swsCtx = C.sws_getContext(d.srcFrame.width, d.srcFrame.height, C.AV_PIX_FMT_YUV420P,
		d.dstFrame.width, d.dstFrame.height, (int32)(d.dstFrame.format), C.SWS_BILINEAR, nil, nil, nil)
	if d.swsCtx == nil {
		return nil, fmt.Errorf("sws_getContext() err")
	}

	dstFrameSize := C.av_image_get_buffer_size((int32)(d.dstFrame.format), d.dstFrame.width, d.dstFrame.height, 1)
	d.dstFramePtr = (*[1 << 30]uint8)(unsafe.Pointer(d.dstFrame.data[0]))[:dstFrameSize:dstFrameSize]
}

// convert frame from YUV420 to RGB
res = C.sws_scale(d.swsCtx, frameData(d.srcFrame), frameLineSize(d.srcFrame),
	0, d.srcFrame.height, frameData(d.dstFrame), frameLineSize(d.dstFrame))
if res < 0 {
	return nil, fmt.Errorf("sws_scale() err")
}

// embed frame into an image.Image
return &image.RGBA{
	Pix:    d.dstFramePtr,
	Stride: 4 * (int)(d.dstFrame.width),
	Rect: image.Rectangle{
		Max: image.Point{(int)(d.dstFrame.width), (int)(d.dstFrame.height)},
	},
}, nil

}

[aler9]

Hello, first of all, why did you do this?

// i wanna set Setup(false, t, baseURL, 0, 0) , so i didn't use SetupAndPlay Func
fmt.Println("Setup")
for _, t := range tracks {
	_, err := c.Setup(true, t, baseURL, 0, 0) // i don't know why i can't set (forPlay == false) in Setup Function.
	if err != nil {
		fmt.Println("hihi", err)
	}
}

Setup(false) can be called only when you're publishing a stream to a RTSP server, not when you're reading one. If you're using Describe() and Play() you're definitely reading a stream from a camera.

[cwcsh95]

thanks for your response. first i refered your example code in gortsplib/examples/client-read-h264 and during to study rtsp, i get to know that 'Setup' Request is Define the way how to send media stream. (like client port, server port) and in my question '// i wanna set Setup(false, t, baseURL, 0, 0) , so i didn't use SetupAndPlay Func', i misunderstand first pamameters (forPay) mean it is forPlay with streaming(pulbish) so i used that method with parameter 'true' /* _, err := c.Setup(true, t, baseURL, 0, 0) // i don't know why i can't set (forPlay == false) in Setup Function. */ and i tried to unuse Setup Method in your answer 'If you're using Describe() and Play() you're definitely reading a stream from a camera.' in this case the function doPlay in Client.go, it returns the error cauze Preplay Error Message [ Play must be in state [prePlay], while is in state initial ] in server.go error function (67~71 line) // Error implements the error interface. func (e ErrServerInvalidState) Error() string { return fmt.Sprintf("must be in state %v, while is in state %v", e.AllowedList, e.State) } i understand separate Setup & Play function is unnecessary so i change my code like err = c.SetupAndPlay(tracks, baseURL) if err != nil { fmt.Println("Setup And Play!") } but it doesn't work without sleep (in previous question) thanks for your attention.

…

-----Original Message----- From: "Alessandro ***@***.***> To: ***@***.***>; Cc: ***@***.***>; ***@***.***>; Sent: 2022-08-23 (화) 19:46:39 (GMT+09:00) Subject: Re: [aler9/gortsplib] RTPListener doesn't work (Issue #136) Hello, first of all, why did you do this? // i wanna set Setup(false, t, baseURL, 0, 0) , so i didn't use SetupAndPlay Func fmt.Println("Setup") for _, t := range tracks { _, err := c.Setup(true, t, baseURL, 0, 0) // i don't know why i can't set (forPlay == false) in Setup Function. if err != nil { fmt.Println("hihi", err) } } Setup(true) can be called only when you're publishing a stream to a RTSP server, not when you're reading one. If you're using Describe() and Play() you're definitely reading a stream from a camera. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>