Native GPU capture and WebRTC delivery for an owned Chromium browser on Linux. Rust composites the selected XComposite window and its owned popup surfaces into an OpenGL texture, encodes with NVIDIA NVENC, and sends encrypted RTP through str0m. TypeScript uses @push.rocks/smartrust for lifecycle, signaling and statistics. No raw video crosses JavaScript or JSON IPC.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Runtime

The shipped native target requires Linux x64 with glibc 2.39 or newer, NVIDIA OpenGL, NVENC API 13.0, Xorg, Xlib, XComposite, XTEST and GLX installed. The host must grant the running user access to the NVIDIA GPU. This path has been exercised on Ubuntu 24.04 with an NVIDIA A2 and driver 580.173.02. The driver and system graphics libraries are separate host dependencies. GStreamer is not used.

pnpm add @push.rocks/smartwebrtc

startDisplay() creates a private rootless Xorg display and returns its display name, disposable Xauthority path, selected GPU PCI address and actual screen dimensions. Keep the Xauthority file private and pass it only to the owned Chromium process. When several NVIDIA display GPUs are available, select one with gpuPciAddress.

An unavailable GPU, graphics library or encoder fails explicitly. Direct IPv4 ICE candidates support LAN and routed VPN connections; this package does not configure a public STUN or TURN service. The network must permit UDP between the advertised host addresses and viewers.

Control API

import { NativeWebRtcVideoEngine } from '@push.rocks/smartwebrtc';

const engine = new NativeWebRtcVideoEngine({
  onError: reason => console.error(reason),
  onPeerError: (peerId, reason) => console.error(peerId, reason),
  onDiagnostic: message => console.error(message),
});
const display = await engine.startDisplay();
// Launch owned, headed Chromium with DISPLAY=display.display and
// XAUTHORITY=display.xauthority, then obtain its root process PID.
await engine.startSource({
  display: display.display,
  browserPid,
  framesPerSecond: 30,
  minBitrate: 1_000_000,
  startBitrate: 8_000_000,
  maxBitrate: 14_000_000,
  maxViewers: 2,
});
await engine.pauseSource();
// Focus the owned page, obtain its CDP window ID as windowKey, and put the
// Chrome window in fullscreen. These are browser-controller operations.
await engine.bindFocusedWindow(windowKey);
await engine.resizeWindow(windowKey, 800, 600);
// Verify the page reports innerWidth=800 and innerHeight=600 before continuing.
await engine.setSourceWindow(windowKey, { x: 0, y: 0, width: 800, height: 600 });
const offer = await engine.openPeer('viewer');
// Exchange offer and gathered answer with a trusted browser controller.
await engine.setAnswer('viewer', answer);
const statistics = await engine.getStatistics('viewer');
await engine.closePeer('viewer');
await engine.stopSource();
// Close Chromium and confirm its process has exited before stopping its display.
await engine.stop();

The caller owns Chromium; this package owns its native media process, capture resources, UDP peers and optional display. stopSource() joins media workers while keeping the display alive. stop() releases the display and confirms native process termination. A stopped engine is not reusable. Failed process startup is never retried implicitly during shutdown.

Before changing the selected window or its geometry, await pauseSource(). After focusing the target browser page, call bindFocusedWindow(windowKey) with the opaque CDP window identifier. The native implementation resolves X11 focus to a root window, verifies its _NET_WM_PID against the owned browser PID, and maintains a one-to-one binding. The CDP identifier is not treated as an XID.

For the native Chromium surface, launch with defaultViewport: null, --app=<url>, --ozone-platform=x11 and --force-device-scale-factor=1. Exclude Puppeteer's --enable-automation default argument: its Chrome infobar occupies 56 pixels outside the page even in fullscreen. Remote debugging still identifies the browser as automated. Set the owned Chrome window fullscreen through CDP, then call resizeWindow(windowKey, width, height) while capture is paused and verify innerWidth/innerHeight match before resuming. Do not use kiosk mode; it disables browser interactions. Geometry is bounded by the actual owned display, which the installed NVIDIA A2 driver caps at 2560×1600. Call releaseWindow after closing an inactive window to retire its binding.

setSourceWindow(windowKey, { x, y, width, height }) selects an exact content rectangle within that window. The rectangle must come from the browser controller's verified layout. Source transitions advance an internal epoch and wait for the capture worker to acknowledge it. Frames still waiting in the native queue are fenced by epoch; the receiving controller must also fence video presentation across viewport revisions. Odd content dimensions extend the last content pixel into one extra pixel on the right or bottom for codec alignment; viewers must present the visible content rectangle rather than treating padding as page content.

The selection result includes rtpTimestampFloor, an unsigned 32-bit timestamp shared by all source viewers. Forward it with the committed viewport revision. In requestVideoFrameCallback, accept a frame only when metadata.rtpTimestamp is present and ((metadata.rtpTimestamp - rtpTimestampFloor) >>> 0) < 0x80000000. This comparison handles RTP clock wrap and rejects frames already in transit before the selection. It does not depend on optional browser captureTime metadata. Apply the timestamp check together with the expected coded dimensions and current source identity; a source restart creates a new clock and must retire its old peers.

getPointerSurface() returns 'content' or 'chrome-ui' for the selected source. Capture and input share exact browser-owned popup discovery. Tooltips, notifications and drag images are visual only; an interactive popup owns outside clicks too, allowing dismissal. The event-invalidated query runs on the dedicated input thread and rejects obsolete source epochs. Browser controllers should use CDP for content gestures, XTEST for Chrome UI, and pin the selected route from the first button press through the last release. Chromium's X11 XDND handler can discard its queued final position on release, so XTEST is not a reliable web-content drag transport.

dispatchPointer({ action, x, y, button, modifiers, timestampMs }) sends native move/down/up events into the selected viewport. XTEST follows Chrome's pointer grabs, so native context menus receive clicks. Coordinates are in visible content pixels; modifier bits follow CDP (Alt 1, Control 2, Meta 4, Shift 8). Events use a bounded ordered queue on a separate native input thread and are acknowledged after X server processing. A source pause releases held pointer buttons/modifiers and invalidates queued input from the old viewport. Wheel deltas and text remain the browser controller's CDP responsibility.

Pass the viewer event's monotonic occurrence time as optional timestampMs (nonnegative finite milliseconds). While a pointer button is held, the native thread restores source intervals compressed by transport batching, subtracting time already elapsed since the preceding injection. Ordinary live input incurs no extra wait. Each restored interval is capped at 32 ms so idle pauses are not replayed; hover and the start of a new gesture are immediate. Missing timestamps retain immediate dispatch, and timestamps must not decrease within a timed gesture. Pause, release and untimed input clear the timing history. The bounded wait is independent of capture, encoding and statistics, and shutdown interrupts it. This matters for Chromium's asynchronous X11 drop negotiation, which can cancel a gesture if every movement is compressed into one burst before the page's drop acceptance is available.

Codec and latency behavior

The sender offers H.264 constrained baseline and HEVC Main. It uses the parsed receiver answer to choose a codec that covers the requested dimensions, frame rate and configured minimum bitrate. The H.264 level in the sender's offer does not override a lower receiver answer. Each peer caps its bitrate at the smaller of the configured maximum and the negotiated codec limit. A typical H.264 Level 3.1 answer covers 1280×720 at 30 fps and at most 14 Mbit/s; higher modes need a receiver that advertises sufficient H.264 or HEVC support. Unsupported modes fail rather than silently downscaling or selecting software encoding.

NVENC uses a low-latency preset, no B-frames or lookahead, and a one-frame VBV. One source-owned media thread captures and composites the browser once for admitted viewers. Each viewer has an independent NVENC session, bitrate and network admission credit; ICE, DTLS, SRTP, retransmission and congestion control run independently. Per-peer capture admission follows RTP pacer credit and keeps at most one compressed frame waiting. Missed capture ticks are not replayed as bursts. Stale frames are discarded and the next frame is forced to an IDR.

getStatistics(peerId) reports encoder, codec, renderer, dimensions, sent/encoded/discarded frames, bitrate, transport RTT, queue size and capture-to-send timing. These are sender measurements, not an input-to-photon latency claim. Concurrent statistics calls for a peer share one request. A failed peer emits onPeerError and remains available for statistics until closePeer releases it; close failed peers before another source transition. Unexpected native process exit emits onError.

Build and verification

pnpm build
pnpm test
cargo test --manifest-path rust/Cargo.toml --locked

pnpm build runs the existing TypeScript and @git.zone/tsrust build tools. It requires a Rust toolchain and the native C build tools used by the locked cryptography dependencies. Driver APIs are loaded dynamically at runtime; compiling does not need the proprietary NVIDIA SDK or a GStreamer SDK image. rust/src/nvenc_sys.rs contains generated bindings from the pinned MIT-licensed NVENC header documented at its top.

The optional GPU integration test launches disposable Chromium processes on the owned display and a real Chrome WebRTC receiver:

SMARTWEBRTC_GPU_TEST=1 pnpm exec tstest test/test.gpu.node.ts --logfile

SMARTWEBRTC_CHROME selects the installed Chrome executable. SMARTWEBRTC_TEST_BINARY can select a directly built native executable for development. The Rust hevc_gpu_encode_decode test additionally requires an installed ffmpeg with the NVIDIA CUVID decoder and runs only with --ignored. It verifies native HEVC encoding and GPU decoding; it does not imply that every browser advertises HEVC support.

The published third-party notices preserve the exact Rust dependency and NVENC header license notices. Refresh them when the Cargo graph changes.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in license.md.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md Unescape Escape

@push.rocks/smartwebrtc