push.rocks/taskbuffer
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(core): Add step-based progress tracking, task metadata and enhanced TaskManager scheduling/metadata APIs

Browse Source
This commit is contained in:
Jürgen Kunz
2025-09-06 13:36:04 +00:00
parent 6c9b975029
commit 9784a5eacf
12 changed files with 1191 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files

524
readme.md
View File

@@ -1,6 +1,6 @@
# @push.rocks/taskbuffer 🚀
A **powerful**, **flexible**, and **TypeScript-first** task management library for orchestrating asynchronous operations with style. From simple task execution to complex distributed workflows, taskbuffer has got you covered.
A **powerful**, **flexible**, and **TypeScript-first** task management library for orchestrating asynchronous operations with style. From simple task execution to complex distributed workflows with real-time progress tracking, taskbuffer has got you covered.
## Install 📦
@@ -23,35 +23,33 @@ In the modern JavaScript ecosystem, managing asynchronous tasks efficiently is c
- **🔄 Smart buffering**: Control concurrent executions with intelligent buffer management
- **⏰ Built-in scheduling**: Cron-based task scheduling without additional dependencies
- **🎭 Multiple paradigms**: Support for debounced, throttled, and one-time execution patterns
- **📊 Progress tracking**: Real-time step-by-step progress monitoring for UI integration
- **🔌 Extensible**: Clean architecture that's easy to extend and customize
- **🏃 Zero dependencies on external schedulers**: Everything you need is included
## Core Concepts 🎓
### Task
The fundamental unit of work. A task wraps an asynchronous function and provides powerful execution control.
The fundamental unit of work. A task wraps an asynchronous function and provides powerful execution control, now with step-by-step progress tracking.
### Taskchain
Sequential task execution - tasks run one after another, with results passed along the chain.
### Taskparallel
Parallel task execution - multiple tasks run simultaneously for maximum performance.
### TaskManager
Centralized task scheduling and management using cron expressions.
Centralized task scheduling and management using cron expressions, with rich metadata collection.
### TaskDebounced
Debounced task execution - prevents rapid repeated executions, only running after a quiet period.
### TaskOnce
Singleton task execution - ensures a task runs exactly once, perfect for initialization routines.
### TaskStep 🆕
Granular progress tracking - define named steps with percentage weights for real-time progress monitoring.
## Quick Start 🏁
### Basic Task Execution
@@ -72,6 +70,42 @@ const myTask = new Task({
const result = await myTask.trigger();
```
### Task with Progress Steps 🆕
Track granular progress for complex operations - perfect for UI progress bars:
```typescript
const dataProcessingTask = new Task({
name: 'DataProcessor',
steps: [
{ name: 'validate', description: 'Validating input data', percentage: 15 },
{ name: 'fetch', description: 'Fetching external resources', percentage: 25 },
{ name: 'transform', description: 'Transforming data', percentage: 35 },
{ name: 'save', description: 'Saving to database', percentage: 25 }
] as const, // Use 'as const' for full type safety
taskFunction: async (inputData) => {
// TypeScript knows these step names!
dataProcessingTask.notifyStep('validate');
const validated = await validateData(inputData);
dataProcessingTask.notifyStep('fetch');
const external = await fetchExternalData();
dataProcessingTask.notifyStep('transform');
const transformed = await transformData(validated, external);
dataProcessingTask.notifyStep('save');
const result = await saveToDatabase(transformed);
return result;
}
});
// Monitor progress in real-time
const result = await dataProcessingTask.trigger();
console.log(`Final progress: ${dataProcessingTask.getProgress()}%`); // 100%
```
## TypeScript Generics Support 🔬
TaskBuffer leverages TypeScript's powerful generics system for complete type safety across your task chains and workflows.
@@ -97,7 +131,7 @@ interface ProcessedUser {
}
// Create strongly typed tasks
const processUserTask = new Task<UserData, ProcessedUser>({
const processUserTask = new Task<ProcessedUser>({
name: 'ProcessUser',
taskFunction: async (user: UserData): Promise<ProcessedUser> => {
return {
@@ -129,7 +163,7 @@ interface TaskConfig {
const configuredTask = new Task<TaskConfig>({
name: 'ConfiguredTask',
taskSetup: async () => ({
taskSetup: async (): Promise<TaskConfig> => ({
apiEndpoint: 'https://api.example.com',
retryCount: 3,
timeout: 5000
@@ -156,21 +190,21 @@ Chain tasks with preserved type flow:
```typescript
// Each task knows its input and output types
const fetchTask = new Task<void, UserData[]>({
const fetchTask = new Task<void>({
name: 'FetchUsers',
taskFunction: async (): Promise<UserData[]> => {
return await api.getUsers();
}
});
const filterTask = new Task<UserData[], UserData[]>({
const filterTask = new Task<void>({
name: 'FilterActive',
taskFunction: async (users: UserData[]): Promise<UserData[]> => {
return users.filter(user => user.isActive);
}
});
const mapTask = new Task<UserData[], ProcessedUser[]>({
const mapTask = new Task<void>({
name: 'MapToProcessed',
taskFunction: async (users: UserData[]): Promise<ProcessedUser[]> => {
return users.map(transformUser);
@@ -186,6 +220,204 @@ const chain = new Taskchain({
const finalResult: ProcessedUser[] = await chain.trigger();
```
## Progress Tracking & Metadata 📊 🆕
TaskBuffer now provides comprehensive progress tracking and metadata collection, perfect for building dashboards and monitoring systems.
### Step-by-Step Progress
Define weighted steps for accurate progress calculation:
```typescript
const migrationTask = new Task({
name: 'DatabaseMigration',
steps: [
{ name: 'backup', description: 'Backing up database', percentage: 20 },
{ name: 'schema', description: 'Updating schema', percentage: 30 },
{ name: 'data', description: 'Migrating data', percentage: 40 },
{ name: 'validate', description: 'Validating integrity', percentage: 10 }
] as const,
taskFunction: async () => {
migrationTask.notifyStep('backup');
await backupDatabase();
console.log(`Progress: ${migrationTask.getProgress()}%`); // ~20%
migrationTask.notifyStep('schema');
await updateSchema();
console.log(`Progress: ${migrationTask.getProgress()}%`); // ~50%
migrationTask.notifyStep('data');
await migrateData();
console.log(`Progress: ${migrationTask.getProgress()}%`); // ~90%
migrationTask.notifyStep('validate');
await validateIntegrity();
console.log(`Progress: ${migrationTask.getProgress()}%`); // 100%
}
});
// Get detailed step information
const steps = migrationTask.getStepsMetadata();
steps.forEach(step => {
console.log(`${step.name}: ${step.status} (${step.percentage}%)`);
if (step.duration) {
console.log(` Duration: ${step.duration}ms`);
}
});
```
### Task Metadata Collection
Get comprehensive metadata about task execution:
```typescript
const task = new Task({
name: 'DataProcessor',
buffered: true,
bufferMax: 5,
steps: [
{ name: 'process', description: 'Processing', percentage: 100 }
] as const,
taskFunction: async () => {
task.notifyStep('process');
await processData();
}
});
// Get complete task metadata
const metadata = task.getMetadata();
console.log({
name: metadata.name,
status: metadata.status, // 'idle' | 'running' | 'completed' | 'failed'
progress: metadata.currentProgress, // 0-100
currentStep: metadata.currentStep,
runCount: metadata.runCount,
lastRun: metadata.lastRun,
buffered: metadata.buffered,
bufferMax: metadata.bufferMax
});
```
### TaskManager Enhanced Metadata
The TaskManager now provides rich metadata for monitoring and dashboards:
```typescript
const manager = new TaskManager();
// Add tasks with step tracking
manager.addAndScheduleTask(backupTask, '0 2 * * *'); // 2 AM daily
manager.addAndScheduleTask(cleanupTask, '0 */6 * * *'); // Every 6 hours
// Get metadata for all tasks
const allTasksMetadata = manager.getAllTasksMetadata();
allTasksMetadata.forEach(task => {
console.log(`Task: ${task.name}`);
console.log(` Status: ${task.status}`);
console.log(` Progress: ${task.currentProgress}%`);
console.log(` Run count: ${task.runCount}`);
console.log(` Schedule: ${task.cronSchedule}`);
});
// Get scheduled tasks with next run times
const scheduledTasks = manager.getScheduledTasks();
scheduledTasks.forEach(task => {
console.log(`${task.name}: Next run at ${task.nextRun}`);
if (task.steps) {
console.log(` Steps: ${task.steps.length}`);
}
});
// Get upcoming executions
const nextRuns = manager.getNextScheduledRuns(10);
console.log('Next 10 scheduled executions:', nextRuns);
```
### Execute and Track Tasks
Execute tasks with full lifecycle tracking and automatic cleanup:
```typescript
const manager = new TaskManager();
const analyticsTask = new Task({
name: 'Analytics',
steps: [
{ name: 'collect', description: 'Collecting metrics', percentage: 30 },
{ name: 'analyze', description: 'Analyzing data', percentage: 50 },
{ name: 'report', description: 'Generating report', percentage: 20 }
] as const,
taskFunction: async () => {
analyticsTask.notifyStep('collect');
const metrics = await collectMetrics();
analyticsTask.notifyStep('analyze');
const analysis = await analyzeData(metrics);
analyticsTask.notifyStep('report');
return await generateReport(analysis);
}
});
// Execute with automatic cleanup and metadata collection
const report = await manager.addExecuteRemoveTask(analyticsTask, {
trackProgress: true
});
console.log('Execution Report:', {
taskName: report.taskName,
duration: report.duration,
stepsCompleted: report.stepsCompleted,
finalProgress: report.progress,
result: report.result
});
```
### Frontend Integration Example
Perfect for building real-time progress UIs:
```typescript
// WebSocket server for real-time updates
io.on('connection', (socket) => {
socket.on('startTask', async (taskId) => {
const task = new Task({
name: taskId,
steps: [
{ name: 'start', description: 'Starting...', percentage: 10 },
{ name: 'process', description: 'Processing...', percentage: 70 },
{ name: 'finish', description: 'Finishing...', percentage: 20 }
] as const,
taskFunction: async () => {
task.notifyStep('start');
socket.emit('progress', {
step: 'start',
progress: task.getProgress(),
metadata: task.getStepsMetadata()
});
task.notifyStep('process');
socket.emit('progress', {
step: 'process',
progress: task.getProgress(),
metadata: task.getStepsMetadata()
});
task.notifyStep('finish');
socket.emit('progress', {
step: 'finish',
progress: task.getProgress(),
metadata: task.getStepsMetadata()
});
}
});
await task.trigger();
socket.emit('complete', task.getMetadata());
});
});
```
## Buffer Behavior Deep Dive 🌊
The buffer system in TaskBuffer provides intelligent control over concurrent executions, preventing system overload while maximizing throughput.
@@ -388,26 +620,7 @@ setInterval(() => {
});
```
### Buffered Execution (Rate Limiting)
Perfect for API calls or database operations that need throttling:
```typescript
const apiTask = new Task({
name: 'APICall',
taskFunction: async (endpoint: string) => {
return await fetch(endpoint);
},
buffered: true,
bufferMax: 3, // Maximum 3 concurrent executions
execDelay: 1000, // Wait 1 second between executions
});
// These will be automatically throttled
for (let i = 0; i < 10; i++) {
apiTask.trigger(`/api/data/${i}`);
}
```
## Common Patterns 🎨
### Task Chains - Sequential Workflows
@@ -488,8 +701,17 @@ import { Task, TaskManager } from '@push.rocks/taskbuffer';
const backupTask = new Task({
name: 'DatabaseBackup',
steps: [
{ name: 'dump', description: 'Creating dump', percentage: 70 },
{ name: 'upload', description: 'Uploading to S3', percentage: 30 }
] as const,
taskFunction: async () => {
backupTask.notifyStep('dump');
await performBackup();
backupTask.notifyStep('upload');
await uploadToS3();
console.log(`Backup completed at ${new Date().toISOString()}`);
},
});
@@ -498,11 +720,14 @@ const manager = new TaskManager();
// Add and schedule tasks
manager.addAndScheduleTask(backupTask, '0 0 * * *'); // Daily at midnight
manager.addAndScheduleTask(healthCheck, '*/5 * * * *'); // Every 5 minutes
// Start the scheduler
manager.start();
// Monitor scheduled tasks
const scheduled = manager.getScheduledTasks();
console.log('Scheduled tasks:', scheduled);
// Later... stop if needed
manager.stop();
```
@@ -598,45 +823,6 @@ runner.registerTask(imageResizeTask);
runner.start();
```
### Buffer Management Strategies
Fine-tune concurrent execution behavior:
```typescript
const task = new Task({
name: 'ResourceIntensive',
taskFunction: async () => {
/* ... */
},
buffered: true,
bufferMax: 5, // Max 5 concurrent
execDelay: 100, // 100ms between starts
timeout: 30000, // 30 second timeout
});
```
### Cycle Detection and Prevention
TaskBuffer automatically detects and prevents circular dependencies:
```typescript
const taskA = new Task({
name: 'TaskA',
taskFunction: async () => {
/* ... */
},
preTask: taskB, // This would create a cycle
});
const taskB = new Task({
name: 'TaskB',
taskFunction: async () => {
/* ... */
},
preTask: taskA, // Circular dependency detected!
});
```
### Dynamic Task Creation
Create tasks on-the-fly based on runtime conditions:
@@ -647,8 +833,17 @@ const dynamicWorkflow = async (config: Config) => {
(step) =>
new Task({
name: step.name,
steps: step.substeps?.map(s => ({
name: s.id,
description: s.label,
percentage: s.weight
})) as const,
taskFunction: async (input) => {
return await processStep(step, input);
for (const substep of step.substeps || []) {
task.notifyStep(substep.id);
await processStep(substep, input);
}
return input;
},
}),
);
@@ -666,26 +861,46 @@ const dynamicWorkflow = async (config: Config) => {
### Task Options
| Option | Type | Description |
| -------------- | ---------- | ------------------------------ |
| `name` | `string` | Unique identifier for the task |
| `taskFunction` | `Function` | Async function to execute |
| `buffered` | `boolean` | Enable buffer management |
| `bufferMax` | `number` | Maximum concurrent executions |
| `execDelay` | `number` | Delay between executions (ms) |
| `timeout` | `number` | Task timeout (ms) |
| `preTask` | `Task` | Task to run before |
| `afterTask` | `Task` | Task to run after |
| Option | Type | Description |
| -------------- | ---------- | -------------------------------------- |
| `name` | `string` | Unique identifier for the task |
| `taskFunction` | `Function` | Async function to execute |
| `steps` | `Array` | Step definitions with name, description, percentage |
| `buffered` | `boolean` | Enable buffer management |
| `bufferMax` | `number` | Maximum concurrent executions |
| `execDelay` | `number` | Delay between executions (ms) |
| `timeout` | `number` | Task timeout (ms) |
| `preTask` | `Task` | Task to run before |
| `afterTask` | `Task` | Task to run after |
### Task Methods
| Method | Description |
| ------------------------- | ---------------------------------------------- |
| `trigger(x?)` | Execute the task |
| `notifyStep(stepName)` | Mark a step as active (typed step names!) |
| `getProgress()` | Get current progress percentage (0-100) |
| `getStepsMetadata()` | Get all steps with their current status |
| `getMetadata()` | Get complete task metadata |
| `resetSteps()` | Reset all steps to pending state |
### TaskManager Methods
| Method | Description |
| ------------------------------- | ------------------------ |
| `addTask(task, cronExpression)` | Add and schedule a task |
| `removeTask(taskName)` | Remove a scheduled task |
| `start()` | Start the scheduler |
| `stop()` | Stop the scheduler |
| `getStats()` | Get execution statistics |
| Method | Description |
| ----------------------------------- | -------------------------------------- |
| `addTask(task)` | Add a task to the manager |
| `addAndScheduleTask(task, cron)` | Add and schedule a task |
| `getTaskByName(name)` | Get a specific task by name |
| `getTaskMetadata(name)` | Get metadata for a specific task |
| `getAllTasksMetadata()` | Get metadata for all tasks |
| `getScheduledTasks()` | Get all scheduled tasks with info |
| `getNextScheduledRuns(limit)` | Get upcoming scheduled executions |
| `addExecuteRemoveTask(task, opts)` | Execute task with lifecycle tracking |
| `triggerTaskByName(name)` | Trigger a task by its name |
| `scheduleTaskByName(name, cron)` | Schedule a task using cron expression |
| `descheduleTaskByName(name)` | Remove task from schedule |
| `start()` | Start the scheduler |
| `stop()` | Stop the scheduler |
### Taskchain Methods
@@ -703,14 +918,20 @@ const dynamicWorkflow = async (config: Config) => {
3. **Implement proper error handling**: Use try-catch in task functions
4. **Monitor task execution**: Use the built-in stats and logging
5. **Set appropriate timeouts**: Prevent hanging tasks from blocking your system
6. **Use step tracking wisely**: Don't create too many granular steps - aim for meaningful progress points
## Error Handling 🛡️
```typescript
const robustTask = new Task({
name: 'RobustOperation',
steps: [
{ name: 'try', description: 'Attempting operation', percentage: 80 },
{ name: 'retry', description: 'Retrying on failure', percentage: 20 }
] as const,
taskFunction: async (input) => {
try {
robustTask.notifyStep('try');
return await riskyOperation(input);
} catch (error) {
// Log error
@@ -718,6 +939,7 @@ const robustTask = new Task({
// Optionally retry
if (error.retryable) {
robustTask.notifyStep('retry');
return await riskyOperation(input);
}
@@ -731,12 +953,20 @@ const robustTask = new Task({
## Real-World Examples 🌍
### API Rate Limiting
### API Rate Limiting with Progress
```typescript
const apiClient = new Task({
name: 'RateLimitedAPI',
steps: [
{ name: 'wait', description: 'Rate limit delay', percentage: 10 },
{ name: 'call', description: 'API call', percentage: 90 }
] as const,
taskFunction: async (endpoint: string) => {
apiClient.notifyStep('wait');
await delay(100); // Rate limiting
apiClient.notifyStep('call');
return await fetch(`https://api.example.com${endpoint}`);
},
buffered: true,
@@ -745,22 +975,43 @@ const apiClient = new Task({
});
```
### Database Migration Pipeline
### Database Migration Pipeline with Progress
```typescript
const migrationChain = new Taskchain({
name: 'DatabaseMigration',
taskArray: [
backupTask,
schemaUpdateTask,
dataTransformTask,
validationTask,
cleanupTask,
new Task({
name: 'Backup',
steps: [{ name: 'backup', description: 'Creating backup', percentage: 100 }] as const,
taskFunction: async () => {
backupTask.notifyStep('backup');
return await createBackup();
}
}),
new Task({
name: 'SchemaUpdate',
steps: [
{ name: 'analyze', description: 'Analyzing changes', percentage: 30 },
{ name: 'apply', description: 'Applying migrations', percentage: 70 }
] as const,
taskFunction: async () => {
schemaTask.notifyStep('analyze');
const changes = await analyzeSchema();
schemaTask.notifyStep('apply');
return await applyMigrations(changes);
}
}),
// ... more tasks
],
});
// Execute with progress monitoring
const result = await migrationChain.trigger();
```
### Microservice Health Monitoring
### Microservice Health Monitoring Dashboard
```typescript
const healthMonitor = new TaskManager();
@@ -768,36 +1019,89 @@ const healthMonitor = new TaskManager();
services.forEach((service) => {
const healthCheck = new Task({
name: `HealthCheck:${service.name}`,
steps: [
{ name: 'ping', description: 'Pinging service', percentage: 30 },
{ name: 'check', description: 'Checking health', percentage: 50 },
{ name: 'report', description: 'Reporting status', percentage: 20 }
] as const,
taskFunction: async () => {
healthCheck.notifyStep('ping');
const responsive = await ping(service.url);
healthCheck.notifyStep('check');
const healthy = await checkHealth(service.url);
healthCheck.notifyStep('report');
if (!healthy) {
await alertOps(service);
}
return { service: service.name, healthy, timestamp: Date.now() };
},
});
healthMonitor.addAndScheduleTask(healthCheck, '*/1 * * * *'); // Every minute
});
// Dashboard endpoint
app.get('/api/health/dashboard', (req, res) => {
const metadata = healthMonitor.getAllTasksMetadata();
res.json({
services: metadata.map(task => ({
name: task.name.replace('HealthCheck:', ''),
status: task.status,
lastCheck: task.lastRun,
nextCheck: healthMonitor.getScheduledTasks()
.find(s => s.name === task.name)?.nextRun,
progress: task.currentProgress,
currentStep: task.currentStep
}))
});
});
```
## Testing 🧪
```typescript
import { expect, tap } from '@git.zone/tstest';
import { Task } from '@push.rocks/taskbuffer';
import { Task, TaskStep } from '@push.rocks/taskbuffer';
tap.test('should execute task successfully', async () => {
const result = await myTask.trigger();
expect(result).toEqual(expectedValue);
tap.test('should track task progress through steps', async () => {
const task = new Task({
name: 'TestTask',
steps: [
{ name: 'step1', description: 'First step', percentage: 50 },
{ name: 'step2', description: 'Second step', percentage: 50 }
] as const,
taskFunction: async () => {
task.notifyStep('step1');
expect(task.getProgress()).toBeLessThanOrEqual(50);
task.notifyStep('step2');
expect(task.getProgress()).toBeLessThanOrEqual(100);
}
});
await task.trigger();
expect(task.getProgress()).toEqual(100);
});
tap.test('should collect execution metadata', async () => {
const manager = new TaskManager();
const task = new Task({
name: 'MetadataTest',
taskFunction: async () => 'result'
});
const report = await manager.addExecuteRemoveTask(task);
expect(report.taskName).toEqual('MetadataTest');
expect(report.result).toEqual('result');
expect(report.duration).toBeGreaterThan(0);
});
tap.start();
```
## Contributing 🤝
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
## Support 💬
- 📧 Email: [hello@task.vc](mailto:hello@task.vc)
@@ -806,7 +1110,7 @@ We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) f
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
@@ -821,4 +1125,4 @@ Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.