q5-record video recording

[quinton-ashley]

I'd like there to be a small module, q5-record, for audio + video recording a q5 canvas.

Existing solutions for this include p5.capture, which is compatible with q5. It's a great library but it's very slow and it's 1.1MB minified due to its dependency on gif.js and h264-mp4-encoder (which has a base64 encoded wasm file inside it).

Since 2023, there's been widely available native support for in-browser video encoding via the WebCodecs API. The author of the canvas-record library says WebCodecs is 5-10x faster than the h264 mp4 encoder library.

Sensible defaults for the video encoding settings should enable users to get great results without needing to mess around. I'd like for recording video of q5 to be nearly just as easy as using saveCanvas (saving an image of the canvas).

My intention is to include q5-record in the default bundle, since cursory research on the WebCodecs api suggests that this module could be quite small.

Default Video Settings

• MP4 container
• frame rate that matches the sketch' frame rate (60 by default)
• h.264 SDR
• a high bitrate to prioritize quality

Currently it's not possible to encode HDR video.

Proposed API

• record(videoSettings) function that starts or resumes recording, optional "advanced mode" usage should enable users to change video settings via an options in an object input parameter, if a recording is already in progress repeatedly calling this function does nothing

• isRecording boolean is true if there's a recording in progress (EDIT: false if the recording has not begun or is paused)

• pauseRecording() function pauses a recording, which can be resumed

• deleteRecording() function stops a recording if one is in progress and deletes it

• saveRecording(fileName) function to stop and save a recording

• createRecorder(x, y) function that displays a recording options in an HTML section above the canvas so that it's not included in the recording, it shouldn't show up by default. Optionally users can set a position for the recorder.

Recorder UI

The recorder HTML will have two buttons: record/pause, delete/save.

It should also display recording time.

Buttons states before a recording has started or after saving or after deleting:

Button 1: Record
Button 2: Save (grayed out, unclickable)

Button states while recording:

Button 1: Pause
Button 2: Delete

Button states when recording is paused:

Button 1: Record (resumes recording)
Button 2: Save (stops and saves the recording)

Example Usage

Basic use:

function setup() {
  createCanvas();
  
  createRecorder();
}

Basic programmatic use:

// one click to start, next click stops and saves the recording
function mousePressed() {
  if (!isRecording) record();
  else saveRecording();
}

Clip taking programmatic use:

function mousePressed() {
  if (!isRecording) record();
  else pauseRecording();
}

function keyPressed() {
  saveRecording('mySketchVideo')
}

Advanced use:

record({
  type: 'mp4',
  codec: 'vp9',
  bitRate: 5000 // mb per second
});

[quinton-ashley]

@Tezumie I've done some research and apparently HEVC/H.265 encoding is not supported on any browser lol. HDR recording is also not yet supported, so sad.

I pulled your PR #88 and then did some updates to q5-record. For now let's support h.264 in a mp4 container as the default. No need for VP8 but VP9 is good. I also set the frame rate of the videos with getTargetFrameRate() instead of the current frame rate.

I think it'd be nice to have a button toggle for whether to include sound in the recording or not.

Also for video time code it's standard to have "hours:minutes:seconds:frames" and frames is a count of how many frames before the target frame rate is reached, then add one to the second counter. Use counter vars for the time code. Also only update the frame count when the draw loop finishes. You could add a post draw hook for this. The video timecode should show how much time has elapsed in the recording, not real time spent recording. That's cause the recorder can enable users to capture sketches that can't be run in realtime. In that case it'd be nice to know how much footage they've recorded.