Compare commits

...

10 Commits

12 changed files with 1657 additions and 93 deletions

View File

@@ -1,5 +1,47 @@
# Changelog
## 2026-02-15 - 6.1.1 - fix(tests)
improve buffered task tests: add chain, concurrency and queue behavior tests
- Replace tools.delayFor with @push.rocks/smartdelay for more deterministic timing in tests
- Add tests for afterTask chaining, bufferMax concurrency, queued-run limits, and re-trigger behavior
- Rename tasks to descriptive names and fix afterTask chaining order to avoid circular references
- Change test runner invocation to export default tap.start() instead of calling tap.start() directly
## 2026-02-15 - 6.1.0 - feat(taskbuffer)
add sliding-window rate limiting and result-sharing to TaskConstraintGroup and integrate with TaskManager
- Added IRateLimitConfig and TResultSharingMode types and exported them from the public index
- TaskConstraintGroup: added rateLimit and resultSharingMode options, internal completion timestamp tracking, and last-result storage
- TaskConstraintGroup: new helpers - pruneCompletionTimestamps, getRateLimitDelay, getNextAvailableDelay, recordResult, getLastResult, hasResultSharing
- TaskConstraintGroup: rate-limit logic enforces maxPerWindow (counts running + completions) and composes with cooldown/maxConcurrent
- TaskManager: records successful task results to constraint groups and resolves queued entries immediately when a shared result exists
- TaskManager: queue drain now considers unified next-available delay (cooldown + rate limit) when scheduling retries
- Documentation updated: README and hints with usage examples for sliding-window rate limiting and result sharing
- Comprehensive tests added for rate limiting, concurrency interaction, and result-sharing behavior
## 2026-02-15 - 6.0.1 - fix(taskbuffer)
no changes to commit
- Git diff shows no changes
- package.json current version is 6.0.0; no version bump required
## 2026-02-15 - 6.0.0 - BREAKING CHANGE(constraints)
make TaskConstraintGroup constraint matcher input-aware and add shouldExecute pre-execution hook
- Rename ITaskConstraintGroupOptions.constraintKeyForTask -> constraintKeyForExecution(task, input?) and update TaskConstraintGroup.getConstraintKey signature
- Add optional shouldExecute(task, input?) hook; TaskManager checks shouldExecute before immediate runs, after acquiring slots, and when draining the constraint queue (queued tasks are skipped when shouldExecute returns false)
- Export ITaskExecution type and store constraintKeys on queued entries (IConstrainedTaskEntry.constraintKeys)
- Documentation and tests updated to demonstrate input-aware constraint keys and shouldExecute pruning
## 2026-02-15 - 5.0.1 - fix(tests)
add and tighten constraint-related tests covering return values, error propagation, concurrency, cooldown timing, and constraint removal
- Tightened cooldown timing assertion from >=100ms to >=250ms to reflect 300ms cooldown with 50ms tolerance.
- Added tests for queued task return values, error propagation when catchErrors is false, and error swallowing behavior when catchErrors is true.
- Added concurrency and cooldown interaction tests to ensure maxConcurrent is respected and batch timing is correct.
- Added test verifying removing a constraint group unblocks queued tasks and drain behavior completes correctly.
## 2026-02-15 - 5.0.0 - BREAKING CHANGE(taskbuffer)
Introduce constraint-based concurrency with TaskConstraintGroup and TaskManager integration; remove legacy TaskRunner and several Task APIs (breaking); add typed Task.data and update exports and tests.

View File

@@ -1,6 +1,6 @@
{
"name": "@push.rocks/taskbuffer",
"version": "5.0.0",
"version": "6.1.1",
"private": false,
"description": "A flexible task management library supporting TypeScript, allowing for task buffering, scheduling, and execution with dependency management.",
"main": "dist_ts/index.js",

View File

@@ -12,11 +12,30 @@
- Typed data bag accessible as `task.data`
### TaskConstraintGroup
- `new TaskConstraintGroup<TData>({ name, constraintKeyForTask, maxConcurrent?, cooldownMs? })`
- `constraintKeyForTask(task)` returns a string key (constraint applies) or `null` (skip)
- `new TaskConstraintGroup<TData>({ name, constraintKeyForExecution, maxConcurrent?, cooldownMs?, shouldExecute?, rateLimit?, resultSharingMode? })`
- `constraintKeyForExecution(task, input?)` returns a string key (constraint applies) or `null` (skip). Receives both task and runtime input.
- `shouldExecute(task, input?)` — optional pre-execution check. Returns `false` to skip (deferred resolves `undefined`). Can be async.
- `maxConcurrent` (default: `Infinity`) — max concurrent tasks per key
- `cooldownMs` (default: `0`) — minimum ms gap between completions per key
- Methods: `canRun(key)`, `acquireSlot(key)`, `releaseSlot(key)`, `getCooldownRemaining(key)`, `getRunningCount(key)`, `reset()`
- `rateLimit` (optional) — `{ maxPerWindow: number, windowMs: number }` sliding window rate limiter. Counts both running + completed tasks in window.
- `resultSharingMode` (default: `'none'`) — `'none'` | `'share-latest'`. When `'share-latest'`, queued tasks for the same key resolve with the first task's result without executing.
- Methods: `getConstraintKey(task, input?)`, `checkShouldExecute(task, input?)`, `canRun(key)`, `acquireSlot(key)`, `releaseSlot(key)`, `getCooldownRemaining(key)`, `getRateLimitDelay(key)`, `getNextAvailableDelay(key)`, `getRunningCount(key)`, `recordResult(key, result)`, `getLastResult(key)`, `hasResultSharing()`, `reset()`
- `ITaskExecution<TData>` type exported from index — `{ task, input }` tuple
### Rate Limiting (v6.1.0+)
- Sliding window rate limiter: `rateLimit: { maxPerWindow: N, windowMs: ms }`
- Counts running + completed tasks against the window cap
- Per-key independence: saturating key A doesn't block key B
- Composable with `maxConcurrent` and `cooldownMs`
- `getNextAvailableDelay(key)` returns `Math.max(cooldownRemaining, rateLimitDelay)` — unified "how long until I can run" answer
- Drain timer auto-schedules based on shortest delay across all constraints
### Result Sharing (v6.1.0+)
- `resultSharingMode: 'share-latest'` — queued tasks for the same key get the first task's result without executing
- Only successful results are shared (errors from `catchErrors: true` or thrown errors are NOT shared)
- `shouldExecute` is NOT called for shared results (the task's purpose was already fulfilled)
- `lastResults` persists until `reset()` — for time-bounded sharing, use `shouldExecute` to control staleness
- Composable with rate limiting: rate-limited waiters get shared result without waiting for the window
### TaskManager Constraint Integration
- `manager.addConstraintGroup(group)` / `manager.removeConstraintGroup(name)`
@@ -26,7 +45,7 @@
### Exported from index.ts
- `TaskConstraintGroup` class
- `ITaskConstraintGroupOptions` type
- `ITaskConstraintGroupOptions`, `IRateLimitConfig`, `TResultSharingMode` types
## Error Handling (v3.6.0+)
- `Task` now has `catchErrors` constructor option (default: `false`)

239
readme.md
View File

@@ -13,7 +13,7 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
## 🌟 Features
- **🎯 Type-Safe Task Management** — Full TypeScript support with generics and type inference
- **🔒 Constraint-Based Concurrency** — Per-key mutual exclusion, group concurrency limits, and cooldown enforcement via `TaskConstraintGroup`
- **🔒 Constraint-Based Concurrency** — Per-key mutual exclusion, group concurrency limits, cooldown enforcement, sliding-window rate limiting, and result sharing via `TaskConstraintGroup`
- **📊 Real-Time Progress Tracking** — Step-based progress with percentage weights
- **⚡ Smart Buffering** — Intelligent request debouncing and batching
- **⏰ Cron Scheduling** — Schedule tasks with cron expressions
@@ -120,7 +120,7 @@ const manager = new TaskManager();
const domainMutex = new TaskConstraintGroup<{ domain: string }>({
name: 'domain-mutex',
maxConcurrent: 1,
constraintKeyForTask: (task) => task.data.domain,
constraintKeyForExecution: (task, input?) => task.data.domain,
});
manager.addConstraintGroup(domainMutex);
@@ -156,7 +156,7 @@ Cap how many tasks can run concurrently across a group:
const dnsLimit = new TaskConstraintGroup<{ group: string }>({
name: 'dns-concurrency',
maxConcurrent: 3,
constraintKeyForTask: (task) =>
constraintKeyForExecution: (task) =>
task.data.group === 'dns' ? 'dns' : null, // null = skip constraint
});
@@ -173,7 +173,7 @@ const rateLimiter = new TaskConstraintGroup<{ domain: string }>({
name: 'api-rate-limit',
maxConcurrent: 1,
cooldownMs: 11000,
constraintKeyForTask: (task) => task.data.domain,
constraintKeyForExecution: (task) => task.data.domain,
});
manager.addConstraintGroup(rateLimiter);
@@ -187,7 +187,7 @@ Limit total concurrent tasks system-wide:
const globalCap = new TaskConstraintGroup({
name: 'global-cap',
maxConcurrent: 10,
constraintKeyForTask: () => 'all', // same key = shared limit
constraintKeyForExecution: () => 'all', // same key = shared limit
});
manager.addConstraintGroup(globalCap);
@@ -208,26 +208,219 @@ await manager.triggerTask(dnsTask);
### Selective Constraints
Return `null` from `constraintKeyForTask` to exempt a task from a constraint group:
Return `null` from `constraintKeyForExecution` to exempt a task from a constraint group:
```typescript
const constraint = new TaskConstraintGroup<{ priority: string }>({
name: 'low-priority-limit',
maxConcurrent: 2,
constraintKeyForTask: (task) =>
constraintKeyForExecution: (task) =>
task.data.priority === 'low' ? 'low-priority' : null, // high priority tasks skip this constraint
});
```
### Input-Aware Constraints 🎯
The `constraintKeyForExecution` function receives both the **task** and the **runtime input** passed to `trigger(input)`. This means the same task triggered with different inputs can be constrained independently:
```typescript
const extractTLD = (domain: string) => {
const parts = domain.split('.');
return parts.slice(-2).join('.');
};
// Same TLD → serialized. Different TLDs → parallel.
const tldMutex = new TaskConstraintGroup({
name: 'tld-mutex',
maxConcurrent: 1,
constraintKeyForExecution: (task, input?: string) => {
if (!input) return null;
return extractTLD(input); // "example.com", "other.org", etc.
},
});
manager.addConstraintGroup(tldMutex);
// These two serialize (same TLD "example.com")
const p1 = manager.triggerTaskConstrained(getCert, 'app.example.com');
const p2 = manager.triggerTaskConstrained(getCert, 'api.example.com');
// This runs in parallel (different TLD "other.org")
const p3 = manager.triggerTaskConstrained(getCert, 'my.other.org');
```
You can also combine `task.data` and `input` for composite keys:
```typescript
const providerDomain = new TaskConstraintGroup<{ provider: string }>({
name: 'provider-domain',
maxConcurrent: 1,
constraintKeyForExecution: (task, input?: string) => {
return `${task.data.provider}:${input || 'default'}`;
},
});
```
### Pre-Execution Check with `shouldExecute` ✅
The `shouldExecute` callback runs right before a queued task executes. If it returns `false`, the task is skipped and its promise resolves with `undefined`. This is perfect for scenarios where a prior execution's outcome makes subsequent queued tasks unnecessary:
```typescript
const certCache = new Map<string, string>();
const certConstraint = new TaskConstraintGroup({
name: 'cert-mutex',
maxConcurrent: 1,
constraintKeyForExecution: (task, input?: string) => {
if (!input) return null;
return extractTLD(input);
},
shouldExecute: (task, input?: string) => {
if (!input) return true;
// Skip if a wildcard cert already covers this TLD
return certCache.get(extractTLD(input)) !== 'wildcard';
},
});
const getCert = new Task({
name: 'get-certificate',
taskFunction: async (domain: string) => {
const cert = await acme.getCert(domain);
if (cert.isWildcard) certCache.set(extractTLD(domain), 'wildcard');
return cert;
},
});
manager.addConstraintGroup(certConstraint);
manager.addTask(getCert);
const r1 = manager.triggerTaskConstrained(getCert, 'app.example.com'); // runs, gets wildcard
const r2 = manager.triggerTaskConstrained(getCert, 'api.example.com'); // queued → skipped!
const r3 = manager.triggerTaskConstrained(getCert, 'my.other.org'); // parallel (different TLD)
const [cert1, cert2, cert3] = await Promise.all([r1, r2, r3]);
// cert2 === undefined (skipped because wildcard already covers example.com)
```
**`shouldExecute` semantics:**
- Runs right before execution (after slot acquisition, before `trigger()`)
- Also checked on immediate (non-queued) triggers
- Returns `false` → skip execution, deferred resolves with `undefined`
- Can be async (return `Promise<boolean>`)
- Has closure access to external state modified by prior executions
- If multiple constraint groups have `shouldExecute`, **all** must return `true`
### Sliding Window Rate Limiting
Enforce "N completions per time window" with burst capability. Unlike `cooldownMs` (which forces even spacing between executions), `rateLimit` allows bursts up to the cap, then blocks until the window slides:
```typescript
// Let's Encrypt style: 300 new orders per 3 hours
const acmeRateLimit = new TaskConstraintGroup({
name: 'acme-rate',
constraintKeyForExecution: () => 'acme-account',
rateLimit: {
maxPerWindow: 300,
windowMs: 3 * 60 * 60 * 1000, // 3 hours
},
});
manager.addConstraintGroup(acmeRateLimit);
// All 300 can burst immediately. The 301st waits until the oldest
// completion falls out of the 3-hour window.
for (const domain of domains) {
manager.triggerTaskConstrained(certTask, { domain });
}
```
Compose multiple rate limits for layered protection:
```typescript
// Per-domain weekly cap AND global order rate
const perDomainWeekly = new TaskConstraintGroup({
name: 'per-domain-weekly',
constraintKeyForExecution: (task, input) => input.registeredDomain,
rateLimit: { maxPerWindow: 50, windowMs: 7 * 24 * 60 * 60 * 1000 },
});
const globalOrderRate = new TaskConstraintGroup({
name: 'global-order-rate',
constraintKeyForExecution: () => 'global',
rateLimit: { maxPerWindow: 300, windowMs: 3 * 60 * 60 * 1000 },
});
manager.addConstraintGroup(perDomainWeekly);
manager.addConstraintGroup(globalOrderRate);
```
Combine with `maxConcurrent` and `cooldownMs` for fine-grained control:
```typescript
const throttled = new TaskConstraintGroup({
name: 'acme-throttle',
constraintKeyForExecution: () => 'acme',
maxConcurrent: 5, // max 5 concurrent requests
cooldownMs: 1000, // 1s gap after each completion
rateLimit: {
maxPerWindow: 300,
windowMs: 3 * 60 * 60 * 1000,
},
});
```
### Result Sharing — Deduplication for Concurrent Requests
When multiple callers request the same resource concurrently, `resultSharingMode: 'share-latest'` ensures only one execution occurs. All queued waiters receive the same result:
```typescript
const certMutex = new TaskConstraintGroup({
name: 'cert-per-tld',
constraintKeyForExecution: (task, input) => extractTld(input.domain),
maxConcurrent: 1,
resultSharingMode: 'share-latest',
});
manager.addConstraintGroup(certMutex);
const certTask = new Task({
name: 'obtain-cert',
taskFunction: async (input) => {
return await acmeClient.obtainWildcard(input.domain);
},
});
manager.addTask(certTask);
// Three requests for *.example.com arrive simultaneously
const [cert1, cert2, cert3] = await Promise.all([
manager.triggerTaskConstrained(certTask, { domain: 'api.example.com' }),
manager.triggerTaskConstrained(certTask, { domain: 'www.example.com' }),
manager.triggerTaskConstrained(certTask, { domain: 'mail.example.com' }),
]);
// Only ONE ACME request was made.
// cert1 === cert2 === cert3 — all callers got the same cert object.
```
**Result sharing semantics:**
- `shouldExecute` is NOT called for shared results (the task's purpose was already fulfilled)
- Error results are NOT shared — queued tasks execute independently after a failure
- `lastResults` persists until `reset()` — for time-bounded sharing, use `shouldExecute` to control staleness
- Composable with rate limiting: rate-limited waiters get shared results without waiting for the window
### How It Works
When you trigger a task through `TaskManager` (via `triggerTask`, `triggerTaskByName`, `addExecuteRemoveTask`, or cron), the manager:
1. Evaluates all registered constraint groups against the task
2. If no constraints apply (all matchers return `null`) → runs immediately
3. If all applicable constraints have capacity → acquires slots and runs
1. Evaluates all registered constraint groups against the task and input
2. If no constraints apply (all matchers return `null`) → checks `shouldExecute` → runs or skips
3. If all applicable constraints have capacity → acquires slots → checks `shouldExecute` → runs or skips
4. If any constraint blocks → enqueues the task; when a running task completes, the queue is drained
5. Cooldown-blocked tasks auto-retry after the shortest remaining cooldown expires
5. Cooldown/rate-limit-blocked tasks auto-retry after the shortest remaining delay expires
6. Queued tasks check for shared results first (if any group has `resultSharingMode: 'share-latest'`)
7. Queued tasks re-check `shouldExecute` when their turn comes — stale work is automatically pruned
## 🎯 Core Concepts
@@ -732,7 +925,7 @@ const manager = new TaskManager();
const tenantLimit = new TaskConstraintGroup<{ tenantId: string }>({
name: 'tenant-concurrency',
maxConcurrent: 2,
constraintKeyForTask: (task) => task.data.tenantId,
constraintKeyForExecution: (task, input?) => task.data.tenantId,
});
manager.addConstraintGroup(tenantLimit);
@@ -829,21 +1022,30 @@ const acmeTasks = manager.getTasksMetadataByLabel('tenantId', 'acme');
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `name` | `string` | *required* | Constraint group identifier |
| `constraintKeyForTask` | `(task) => string \| null` | *required* | Returns key for grouping, or `null` to skip |
| `constraintKeyForExecution` | `(task, input?) => string \| null` | *required* | Returns key for grouping, or `null` to skip. Receives both the task and runtime input. |
| `maxConcurrent` | `number` | `Infinity` | Max concurrent tasks per key |
| `cooldownMs` | `number` | `0` | Minimum ms between completions per key |
| `shouldExecute` | `(task, input?) => boolean \| Promise<boolean>` | — | Pre-execution check. Return `false` to skip; deferred resolves `undefined`. |
| `rateLimit` | `IRateLimitConfig` | — | Sliding window: `{ maxPerWindow, windowMs }`. Counts running + completed tasks. |
| `resultSharingMode` | `TResultSharingMode` | `'none'` | `'none'` or `'share-latest'`. Queued tasks get first task's result without executing. |
### TaskConstraintGroup Methods
| Method | Returns | Description |
| --- | --- | --- |
| `getConstraintKey(task)` | `string \| null` | Get the constraint key for a task |
| `canRun(key)` | `boolean` | Check if a slot is available |
| `getConstraintKey(task, input?)` | `string \| null` | Get the constraint key for a task + input |
| `checkShouldExecute(task, input?)` | `Promise<boolean>` | Run the `shouldExecute` callback (defaults to `true`) |
| `canRun(key)` | `boolean` | Check if a slot is available (considers concurrency, cooldown, and rate limit) |
| `acquireSlot(key)` | `void` | Claim a running slot |
| `releaseSlot(key)` | `void` | Release a slot and record completion time |
| `releaseSlot(key)` | `void` | Release a slot and record completion time + rate-limit timestamp |
| `getCooldownRemaining(key)` | `number` | Milliseconds until cooldown expires |
| `getRateLimitDelay(key)` | `number` | Milliseconds until a rate-limit slot opens |
| `getNextAvailableDelay(key)` | `number` | Max of cooldown + rate-limit delay — unified "when can I run" |
| `getRunningCount(key)` | `number` | Current running count for key |
| `reset()` | `void` | Clear all state |
| `recordResult(key, result)` | `void` | Store result for sharing (no-op if mode is `'none'`) |
| `getLastResult(key)` | `{result, timestamp} \| undefined` | Get last shared result for key |
| `hasResultSharing()` | `boolean` | Whether result sharing is enabled |
| `reset()` | `void` | Clear all state (running counts, cooldowns, rate-limit timestamps, shared results) |
### TaskManager Methods
@@ -884,12 +1086,15 @@ const acmeTasks = manager.getTasksMetadataByLabel('tenantId', 'acme');
import type {
ITaskMetadata,
ITaskExecutionReport,
ITaskExecution,
IScheduledTaskInfo,
ITaskEvent,
TTaskEventType,
ITaskStep,
ITaskFunction,
ITaskConstraintGroupOptions,
IRateLimitConfig,
TResultSharingMode,
StepNames,
} from '@push.rocks/taskbuffer';
```

File diff suppressed because it is too large Load Diff

View File

@@ -1,52 +1,151 @@
import { tap, expect } from '@git.zone/tstest/tapbundle';
import * as taskbuffer from '../ts/index.js';
import * as smartdelay from '@push.rocks/smartdelay';
let counter1 = 0;
let counter2 = 0;
let counter3 = 0;
// Test 1: Basic buffered execution with afterTask chain
tap.test('should run buffered tasks with afterTask chain', async () => {
let counter1 = 0;
let counter2 = 0;
let counter3 = 0;
tap.test('should run buffered', async (tools) => {
const task = new taskbuffer.Task({
name: 'a buffered task',
taskFunction: async () => {
counter1++;
await tools.delayFor(2000);
console.log(`task 1 ran ${counter1} times`);
},
buffered: true,
bufferMax: 1,
afterTask: () => {
return task2;
},
});
const task2 = new taskbuffer.Task({
name: 'a buffered task',
taskFunction: async () => {
counter2++;
await tools.delayFor(2000);
console.log(`task2 ran ${counter2} times`);
},
buffered: true,
bufferMax: 1,
afterTask: () => {
return task3;
},
});
const task3 = new taskbuffer.Task({
name: 'a buffered task',
name: 'buffered-chain-3',
taskFunction: async () => {
counter3++;
await tools.delayFor(2000);
console.log(`task3 ran ${counter3} times`);
await smartdelay.delayFor(50);
},
buffered: true,
bufferMax: 1,
});
while (counter1 < 10) {
await tools.delayFor(5000);
task.trigger();
const task2 = new taskbuffer.Task({
name: 'buffered-chain-2',
taskFunction: async () => {
counter2++;
await smartdelay.delayFor(50);
},
buffered: true,
bufferMax: 1,
afterTask: () => task3,
});
const task1 = new taskbuffer.Task({
name: 'buffered-chain-1',
taskFunction: async () => {
counter1++;
await smartdelay.delayFor(50);
},
buffered: true,
bufferMax: 1,
afterTask: () => task2,
});
// Trigger 3 times with enough spacing for the chain to complete
for (let i = 0; i < 3; i++) {
task1.trigger();
await smartdelay.delayFor(250); // enough for chain of 3 x 50ms tasks
}
// Wait for final chain to finish
await smartdelay.delayFor(500);
// Each task in the chain should have run at least once
expect(counter1).toBeGreaterThanOrEqual(1);
expect(counter2).toBeGreaterThanOrEqual(1);
expect(counter3).toBeGreaterThanOrEqual(1);
// afterTask chain means task2 count should match task1 (each trigger chains)
expect(counter2).toEqual(counter1);
expect(counter3).toEqual(counter1);
});
tap.start();
// Test 2: bufferMax limits concurrent buffered executions
tap.test('should respect bufferMax for concurrent buffered calls', async () => {
let running = 0;
let maxRunning = 0;
let totalRuns = 0;
const task = new taskbuffer.Task({
name: 'buffer-max-test',
taskFunction: async () => {
running++;
maxRunning = Math.max(maxRunning, running);
totalRuns++;
await smartdelay.delayFor(100);
running--;
},
buffered: true,
bufferMax: 2,
});
// Fire many triggers rapidly — only bufferMax should run concurrently
for (let i = 0; i < 10; i++) {
task.trigger();
}
// Wait for all buffered executions to complete
await smartdelay.delayFor(1000);
expect(maxRunning).toBeLessThanOrEqual(2);
expect(totalRuns).toBeGreaterThanOrEqual(1);
});
// Test 3: bufferMax limits how many runs are queued during execution
tap.test('should limit queued runs to bufferMax during execution', async () => {
let runCount = 0;
const task = new taskbuffer.Task({
name: 'buffer-queue-test',
taskFunction: async () => {
runCount++;
await smartdelay.delayFor(100);
},
buffered: true,
bufferMax: 2,
});
// Rapid-fire 5 triggers — bufferMax:2 means counter caps at 2
// so only 2 runs will happen (the initial run + 1 buffered rerun)
task.trigger();
task.trigger();
task.trigger();
task.trigger();
task.trigger();
await smartdelay.delayFor(500);
expect(runCount).toEqual(2);
});
// Test 4: Triggers spaced after completion queue new runs
tap.test('should re-trigger after previous buffered run completes', async () => {
let runCount = 0;
const task = new taskbuffer.Task({
name: 'retrigger-test',
taskFunction: async () => {
runCount++;
await smartdelay.delayFor(50);
},
buffered: true,
bufferMax: 1,
});
// First trigger starts execution
task.trigger();
// Wait for it to complete
await smartdelay.delayFor(100);
// Second trigger starts a new execution (task is now idle)
task.trigger();
await smartdelay.delayFor(100);
// Third trigger
task.trigger();
await smartdelay.delayFor(100);
expect(runCount).toEqual(3);
});
export default tap.start();

View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/taskbuffer',
version: '5.0.0',
version: '6.1.1',
description: 'A flexible task management library supporting TypeScript, allowing for task buffering, scheduling, and execution with dependency management.'
}

View File

@@ -12,7 +12,7 @@ export { TaskStep } from './taskbuffer.classes.taskstep.js';
export type { ITaskStep } from './taskbuffer.classes.taskstep.js';
// Metadata interfaces
export type { ITaskMetadata, ITaskExecutionReport, IScheduledTaskInfo, ITaskEvent, TTaskEventType, ITaskConstraintGroupOptions } from './taskbuffer.interfaces.js';
export type { ITaskMetadata, ITaskExecutionReport, IScheduledTaskInfo, ITaskEvent, TTaskEventType, ITaskConstraintGroupOptions, ITaskExecution, IRateLimitConfig, TResultSharingMode } from './taskbuffer.interfaces.js';
import * as distributedCoordination from './taskbuffer.classes.distributedcoordinator.js';
export { distributedCoordination };

View File

@@ -1,27 +1,42 @@
import type { Task } from './taskbuffer.classes.task.js';
import type { ITaskConstraintGroupOptions } from './taskbuffer.interfaces.js';
import type { ITaskConstraintGroupOptions, IRateLimitConfig, TResultSharingMode } from './taskbuffer.interfaces.js';
export class TaskConstraintGroup<TData extends Record<string, unknown> = Record<string, unknown>> {
public name: string;
public maxConcurrent: number;
public cooldownMs: number;
private constraintKeyForTask: (task: Task<any, any, TData>) => string | null | undefined;
public rateLimit: IRateLimitConfig | null;
public resultSharingMode: TResultSharingMode;
private constraintKeyForExecution: (task: Task<any, any, TData>, input?: any) => string | null | undefined;
private shouldExecuteFn?: (task: Task<any, any, TData>, input?: any) => boolean | Promise<boolean>;
private runningCounts = new Map<string, number>();
private lastCompletionTimes = new Map<string, number>();
private completionTimestamps = new Map<string, number[]>();
private lastResults = new Map<string, { result: any; timestamp: number }>();
constructor(options: ITaskConstraintGroupOptions<TData>) {
this.name = options.name;
this.constraintKeyForTask = options.constraintKeyForTask;
this.constraintKeyForExecution = options.constraintKeyForExecution;
this.maxConcurrent = options.maxConcurrent ?? Infinity;
this.cooldownMs = options.cooldownMs ?? 0;
this.shouldExecuteFn = options.shouldExecute;
this.rateLimit = options.rateLimit ?? null;
this.resultSharingMode = options.resultSharingMode ?? 'none';
}
public getConstraintKey(task: Task<any, any, TData>): string | null {
const key = this.constraintKeyForTask(task);
public getConstraintKey(task: Task<any, any, TData>, input?: any): string | null {
const key = this.constraintKeyForExecution(task, input);
return key ?? null;
}
public async checkShouldExecute(task: Task<any, any, TData>, input?: any): Promise<boolean> {
if (!this.shouldExecuteFn) {
return true;
}
return this.shouldExecuteFn(task, input);
}
public canRun(subGroupKey: string): boolean {
const running = this.runningCounts.get(subGroupKey) ?? 0;
if (running >= this.maxConcurrent) {
@@ -38,6 +53,16 @@ export class TaskConstraintGroup<TData extends Record<string, unknown> = Record<
}
}
if (this.rateLimit) {
this.pruneCompletionTimestamps(subGroupKey);
const timestamps = this.completionTimestamps.get(subGroupKey);
const completedInWindow = timestamps ? timestamps.length : 0;
const running = this.runningCounts.get(subGroupKey) ?? 0;
if (completedInWindow + running >= this.rateLimit.maxPerWindow) {
return false;
}
}
return true;
}
@@ -55,6 +80,12 @@ export class TaskConstraintGroup<TData extends Record<string, unknown> = Record<
this.runningCounts.set(subGroupKey, next);
}
this.lastCompletionTimes.set(subGroupKey, Date.now());
if (this.rateLimit) {
const timestamps = this.completionTimestamps.get(subGroupKey) ?? [];
timestamps.push(Date.now());
this.completionTimestamps.set(subGroupKey, timestamps);
}
}
public getCooldownRemaining(subGroupKey: string): number {
@@ -73,8 +104,61 @@ export class TaskConstraintGroup<TData extends Record<string, unknown> = Record<
return this.runningCounts.get(subGroupKey) ?? 0;
}
// Rate limit helpers
private pruneCompletionTimestamps(subGroupKey: string): void {
const timestamps = this.completionTimestamps.get(subGroupKey);
if (!timestamps || !this.rateLimit) return;
const cutoff = Date.now() - this.rateLimit.windowMs;
let i = 0;
while (i < timestamps.length && timestamps[i] <= cutoff) {
i++;
}
if (i > 0) {
timestamps.splice(0, i);
}
}
public getRateLimitDelay(subGroupKey: string): number {
if (!this.rateLimit) return 0;
this.pruneCompletionTimestamps(subGroupKey);
const timestamps = this.completionTimestamps.get(subGroupKey);
const completedInWindow = timestamps ? timestamps.length : 0;
const running = this.runningCounts.get(subGroupKey) ?? 0;
if (completedInWindow + running < this.rateLimit.maxPerWindow) {
return 0;
}
// If only running tasks fill the window (no completions yet), we can't compute a delay
if (!timestamps || timestamps.length === 0) {
return 1; // minimal delay; drain will re-check after running tasks complete
}
// The oldest timestamp in the window determines when a slot opens
const oldestInWindow = timestamps[0];
const expiry = oldestInWindow + this.rateLimit.windowMs;
return Math.max(0, expiry - Date.now());
}
public getNextAvailableDelay(subGroupKey: string): number {
return Math.max(this.getCooldownRemaining(subGroupKey), this.getRateLimitDelay(subGroupKey));
}
// Result sharing helpers
public recordResult(subGroupKey: string, result: any): void {
if (this.resultSharingMode === 'none') return;
this.lastResults.set(subGroupKey, { result, timestamp: Date.now() });
}
public getLastResult(subGroupKey: string): { result: any; timestamp: number } | undefined {
return this.lastResults.get(subGroupKey);
}
public hasResultSharing(): boolean {
return this.resultSharingMode !== 'none';
}
public reset(): void {
this.runningCounts.clear();
this.lastCompletionTimes.clear();
this.completionTimestamps.clear();
this.lastResults.clear();
}
}

View File

@@ -80,14 +80,18 @@ export class TaskManager {
// Gather applicable constraints
const applicableGroups: Array<{ group: TaskConstraintGroup<any>; key: string }> = [];
for (const group of this.constraintGroups) {
const key = group.getConstraintKey(task);
const key = group.getConstraintKey(task, input);
if (key !== null) {
applicableGroups.push({ group, key });
}
}
// No constraints apply → trigger directly
// No constraints apply → check shouldExecute then trigger directly
if (applicableGroups.length === 0) {
const shouldRun = await this.checkAllShouldExecute(task, input);
if (!shouldRun) {
return undefined;
}
return task.trigger(input);
}
@@ -97,24 +101,56 @@ export class TaskManager {
return this.executeWithConstraintTracking(task, input, applicableGroups);
}
// Blocked → enqueue with deferred promise
// Blocked → enqueue with deferred promise and cached constraint keys
const deferred = plugins.smartpromise.defer<any>();
this.constraintQueue.push({ task, input, deferred });
const constraintKeys = new Map<string, string>();
for (const { group, key } of applicableGroups) {
constraintKeys.set(group.name, key);
}
this.constraintQueue.push({ task, input, deferred, constraintKeys });
return deferred.promise;
}
private async checkAllShouldExecute(task: Task<any, any, any>, input?: any): Promise<boolean> {
for (const group of this.constraintGroups) {
const shouldRun = await group.checkShouldExecute(task, input);
if (!shouldRun) {
return false;
}
}
return true;
}
private async executeWithConstraintTracking(
task: Task<any, any, any>,
input: any,
groups: Array<{ group: TaskConstraintGroup<any>; key: string }>,
): Promise<any> {
// Acquire slots
// Acquire slots synchronously to prevent race conditions
for (const { group, key } of groups) {
group.acquireSlot(key);
}
// Check shouldExecute after acquiring slots
const shouldRun = await this.checkAllShouldExecute(task, input);
if (!shouldRun) {
// Release slots and drain queue
for (const { group, key } of groups) {
group.releaseSlot(key);
}
this.drainConstraintQueue();
return undefined;
}
try {
return await task.trigger(input);
const result = await task.trigger(input);
// Record result for groups with result sharing (only on true success, not caught errors)
if (!task.lastError) {
for (const { group, key } of groups) {
group.recordResult(key, result);
}
}
return result;
} finally {
// Release slots
for (const { group, key } of groups) {
@@ -131,32 +167,51 @@ export class TaskManager {
for (const entry of this.constraintQueue) {
const applicableGroups: Array<{ group: TaskConstraintGroup<any>; key: string }> = [];
for (const group of this.constraintGroups) {
const key = group.getConstraintKey(entry.task);
const key = group.getConstraintKey(entry.task, entry.input);
if (key !== null) {
applicableGroups.push({ group, key });
}
}
// No constraints apply anymore (group removed?) → run directly
// No constraints apply anymore (group removed?) → check shouldExecute then run
if (applicableGroups.length === 0) {
entry.task.trigger(entry.input).then(
(result) => entry.deferred.resolve(result),
(err) => entry.deferred.reject(err),
);
this.checkAllShouldExecute(entry.task, entry.input).then((shouldRun) => {
if (!shouldRun) {
entry.deferred.resolve(undefined);
return;
}
entry.task.trigger(entry.input).then(
(result) => entry.deferred.resolve(result),
(err) => entry.deferred.reject(err),
);
});
continue;
}
// Check result sharing — if any applicable group has a shared result, resolve immediately
const sharingGroups = applicableGroups.filter(({ group }) => group.hasResultSharing());
if (sharingGroups.length > 0) {
const groupWithResult = sharingGroups.find(({ group, key }) =>
group.getLastResult(key) !== undefined
);
if (groupWithResult) {
entry.deferred.resolve(groupWithResult.group.getLastResult(groupWithResult.key)!.result);
continue;
}
}
const allCanRun = applicableGroups.every(({ group, key }) => group.canRun(key));
if (allCanRun) {
// executeWithConstraintTracking handles shouldExecute check internally
this.executeWithConstraintTracking(entry.task, entry.input, applicableGroups).then(
(result) => entry.deferred.resolve(result),
(err) => entry.deferred.reject(err),
);
} else {
stillQueued.push(entry);
// Track shortest cooldown for timer scheduling
// Track shortest delay for timer scheduling (cooldown + rate limit)
for (const { group, key } of applicableGroups) {
const remaining = group.getCooldownRemaining(key);
const remaining = group.getNextAvailableDelay(key);
if (remaining > 0 && remaining < shortestCooldown) {
shortestCooldown = remaining;
}

View File

@@ -1,17 +1,33 @@
import type { ITaskStep } from './taskbuffer.classes.taskstep.js';
import type { Task } from './taskbuffer.classes.task.js';
export interface IRateLimitConfig {
maxPerWindow: number; // max completions allowed within the sliding window
windowMs: number; // sliding window duration in ms
}
export type TResultSharingMode = 'none' | 'share-latest';
export interface ITaskConstraintGroupOptions<TData extends Record<string, unknown> = Record<string, unknown>> {
name: string;
constraintKeyForTask: (task: Task<any, any, TData>) => string | null | undefined;
constraintKeyForExecution: (task: Task<any, any, TData>, input?: any) => string | null | undefined;
maxConcurrent?: number; // default: Infinity
cooldownMs?: number; // default: 0
shouldExecute?: (task: Task<any, any, TData>, input?: any) => boolean | Promise<boolean>;
rateLimit?: IRateLimitConfig;
resultSharingMode?: TResultSharingMode; // default: 'none'
}
export interface ITaskExecution<TData extends Record<string, unknown> = Record<string, unknown>> {
task: Task<any, any, TData>;
input: any;
}
export interface IConstrainedTaskEntry {
task: Task<any, any, any>;
input: any;
deferred: import('@push.rocks/smartpromise').Deferred<any>;
constraintKeys: Map<string, string>; // groupName -> key
}
export interface ITaskMetadata {

View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/taskbuffer',
version: '5.0.0',
version: '6.1.1',
description: 'A flexible task management library supporting TypeScript, allowing for task buffering, scheduling, and execution with dependency management.'
}