feat(streaming): Add streaming support: chunked stream transfers, file send/receive, stream events and helpers
This commit is contained in:
85
readme.md
85
readme.md
@@ -16,6 +16,7 @@ SmartIPC delivers bulletproof Inter-Process Communication for Node.js applicatio
|
||||
- **CI/Test Ready** - Built-in helpers and race condition prevention for testing
|
||||
- **Observable** - Real-time metrics, connection tracking, and health monitoring
|
||||
- **Multiple Patterns** - Request/Response, Pub/Sub, and Fire-and-Forget messaging
|
||||
- **Streaming Support** - Efficient, backpressure‑aware streaming for large data and files
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
@@ -184,6 +185,90 @@ await publisher.publish('user.login', {
|
||||
|
||||
## 💪 Advanced Features
|
||||
|
||||
### 📦 Streaming Large Data & Files
|
||||
|
||||
SmartIPC supports efficient, backpressure-aware streaming of large payloads using chunked messages. Streams work both directions and emit a high-level `stream` event for consumption.
|
||||
|
||||
Client → Server streaming:
|
||||
|
||||
```typescript
|
||||
// Server side: receive stream
|
||||
server.on('stream', async (info, readable) => {
|
||||
if (info.meta?.type === 'file') {
|
||||
console.log('Receiving file', info.meta.basename, 'from', info.clientId);
|
||||
}
|
||||
// Pipe to disk or process chunks
|
||||
await SmartIpc.pipeStreamToFile(readable, '/tmp/incoming.bin');
|
||||
});
|
||||
|
||||
// Client side: send a stream
|
||||
const readable = fs.createReadStream('/path/to/local.bin');
|
||||
await client.sendStream(readable, {
|
||||
meta: { type: 'file', basename: 'local.bin' },
|
||||
chunkSize: 64 * 1024 // optional, defaults to 64k
|
||||
});
|
||||
```
|
||||
|
||||
Server → Client streaming:
|
||||
|
||||
```typescript
|
||||
client.on('stream', async (info, readable) => {
|
||||
console.log('Got stream from server', info.meta);
|
||||
await SmartIpc.pipeStreamToFile(readable, '/tmp/from-server.bin');
|
||||
});
|
||||
|
||||
await server.sendStreamToClient(client.getClientId(), fs.createReadStream('/path/server.bin'), {
|
||||
meta: { type: 'file', basename: 'server.bin' }
|
||||
});
|
||||
```
|
||||
|
||||
High-level helpers for files:
|
||||
|
||||
```typescript
|
||||
// Client → Server
|
||||
await client.sendFile('/path/to/bigfile.iso');
|
||||
|
||||
// Server → Client
|
||||
await server.sendFileToClient(clientId, '/path/to/backup.tar');
|
||||
|
||||
// Save an incoming stream to a file (both sides)
|
||||
server.on('stream', async (info, readable) => {
|
||||
await SmartIpc.pipeStreamToFile(readable, '/data/uploaded/' + info.meta?.basename);
|
||||
});
|
||||
```
|
||||
|
||||
Events & metadata:
|
||||
|
||||
- `channel/server/client` emit `stream` with `(info, readable)`
|
||||
- `info` contains: `streamId`, `meta` (your metadata, e.g., filename/size), `headers`, and `clientId` (if available)
|
||||
|
||||
API summary:
|
||||
|
||||
- Client: `sendStream(readable, opts)`, `sendFile(filePath, opts)`, `cancelOutgoingStream(id)`, `cancelIncomingStream(id)`
|
||||
- Server: `sendStreamToClient(clientId, readable, opts)`, `sendFileToClient(clientId, filePath, opts)`, `cancelIncomingStreamFromClient(clientId, id)`, `cancelOutgoingStreamToClient(clientId, id)`
|
||||
- Utility: `SmartIpc.pipeStreamToFile(readable, filePath)`
|
||||
|
||||
Concurrency and cancelation:
|
||||
|
||||
```typescript
|
||||
// Limit concurrent streams per connection
|
||||
const server = SmartIpc.createServer({
|
||||
id: 'svc', socketPath: '/tmp/svc.sock', maxConcurrentStreams: 2
|
||||
});
|
||||
|
||||
// Cancel a stream from the receiver side
|
||||
server.on('stream', (info, readable) => {
|
||||
if (info.meta?.shouldCancel) {
|
||||
(server as any).primaryChannel.cancelIncomingStream(info.streamId, { clientId: info.clientId });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
Notes:
|
||||
- Streaming uses chunked messages under the hood and respects socket backpressure.
|
||||
- Include `meta` to share context like filename/size; it’s delivered with the `stream` event.
|
||||
- Configure `maxConcurrentStreams` (default: 32) to guard resources.
|
||||
|
||||
### 🏁 Server Readiness Detection
|
||||
|
||||
Eliminate race conditions in tests and production:
|
||||
|
Reference in New Issue
Block a user