Compare commits

...

10 Commits

Author SHA1 Message Date
1c4ffbb612 5.6.2
Some checks failed
Default (tags) / security (push) Successful in 50s
Default (tags) / test (push) Failing after 12m37s
Default (tags) / release (push) Has been cancelled
Default (tags) / metadata (push) Has been cancelled
2025-08-31 07:45:48 +00:00
0a75c4cf76 fix(processmanager): Improve process lifecycle handling and cleanup in daemon, monitors and wrappers 2025-08-31 07:45:47 +00:00
8f31672a67 5.6.1
Some checks failed
Default (tags) / security (push) Successful in 51s
Default (tags) / test (push) Failing after 3m57s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-08-31 00:01:50 +00:00
b3087831e2 fix(daemon): Ensure robust process shutdown and improve logs/subscriber diagnostics 2025-08-31 00:01:50 +00:00
4160b3f031 5.6.0
Some checks failed
Default (tags) / security (push) Successful in 38s
Default (tags) / test (push) Failing after 3m57s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-08-30 23:36:26 +00:00
fa50ce40c8 feat(processmonitor): Add CPU monitoring and display CPU in process list 2025-08-30 23:36:26 +00:00
8f96118e0c 5.5.0
Some checks failed
Default (tags) / security (push) Successful in 50s
Default (tags) / test (push) Failing after 3m58s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-08-30 23:26:59 +00:00
b210efde2a feat(logs): Improve logs streaming and backlog delivery; add CLI filters and ndjson output 2025-08-30 23:26:59 +00:00
d8709d8b94 5.4.2
Some checks failed
Default (tags) / security (push) Successful in 49s
Default (tags) / test (push) Failing after 3m58s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-08-30 22:16:44 +00:00
43799f3431 fix(cli/process/logs): Reset log sequence on process restart to avoid false log gap warnings 2025-08-30 22:16:44 +00:00
13 changed files with 457 additions and 78 deletions

View File

@@ -1,5 +1,48 @@
# Changelog # Changelog
## 2025-08-31 - 5.6.2 - fix(processmanager)
Improve process lifecycle handling and cleanup in daemon, monitors and wrappers
- StartAll: when a monitor exists but is not running, restart it instead of skipping — ensures saved processes are reliably brought online.
- ProcessMonitor.stop: cancel any pending restart timers to prevent stray restarts after explicit stop.
- ProcessWrapper: add killProcessTree helper and use it for graceful (SIGTERM) and force (SIGKILL) shutdowns to reliably signal child processes.
- Daemon stopAll: yield briefly after stopping processes and inspect monitors (not only processInfo) to accurately report stopped vs failed processes.
## 2025-08-31 - 5.6.1 - fix(daemon)
Ensure robust process shutdown and improve logs/subscriber diagnostics
- Make ProcessWrapper.stop asynchronous and awaitable to avoid race conditions when stopping processes
- Signal entire process groups on POSIX (kill by negative PID) and fall back to per-PID signalling; escalate to SIGKILL after a timeout
- Await processWrapper.stop() from ProcessMonitor when enforcing memory limits or handling exits/errors to ensure child processes are cleaned up
- Add logs:subscribers IPC endpoint and corresponding types to inspect current subscribers for a process log topic
- Add optional CLI debug output in logs command (enabled via TSPM_DEBUG=true) to print subscriber counts and details
- Support passing request.lines to getLogs handler in daemon to limit returned log entries
## 2025-08-30 - 5.6.0 - feat(processmonitor)
Add CPU monitoring and display CPU in process list
- CLI: show a CPU column in the `tspm list` output (adds formatting and placeholder name display)
- Daemon: ProcessMonitor now collects CPU usage for the process group in addition to memory
- Daemon: ProcessMonitor exposes getLastCpuUsage() and ProcessManager syncs CPU values into IProcessInfo
- Non-breaking: UI and internal stats enriched to surface CPU metrics for processes
## 2025-08-30 - 5.5.0 - feat(logs)
Improve logs streaming and backlog delivery; add CLI filters and ndjson output
- CLI: add new logs options: --since, --stderr-only, --stdout-only and --ndjson; enhance streaming output and gap detection
- CLI: fetch backlog conditionally (honoring --since) and print filtered results before live streaming
- Client: add TspmIpcClient.requestLogsBacklogStream, onStream and onBacklogTopic helpers to receive backlog chunks and streams
- Daemon: add logs:subscribe IPC handler to stream backlog entries to requesting client in small batches
- Protocol: extend IPC types with LogsSubscribeRequest/Response and register 'logs:subscribe' method
- Dependency: bump @push.rocks/smartipc to ^2.3.0 to support the streaming/IPC changes
## 2025-08-30 - 5.4.2 - fix(cli/process/logs)
Reset log sequence on process restart to avoid false log gap warnings
- Track process runId when streaming logs and initialize lastRunId from fetched logs
- When a new runId is detected, reset lastSeq so that subsequent streamed logs are accepted (prevents spurious gap warnings)
- Emit an informational message when a restart/runId change is detected to aid debugging of log streams
## 2025-08-30 - 5.4.1 - fix(processmonitor) ## 2025-08-30 - 5.4.1 - fix(processmonitor)
Bump tsbuild devDependency and relax ps-tree callback typing in ProcessMonitor Bump tsbuild devDependency and relax ps-tree callback typing in ProcessMonitor

View File

@@ -1,6 +1,6 @@
{ {
"name": "@git.zone/tspm", "name": "@git.zone/tspm",
"version": "5.4.1", "version": "5.6.2",
"private": false, "private": false,
"description": "a no fuzz process manager", "description": "a no fuzz process manager",
"main": "dist_ts/index.js", "main": "dist_ts/index.js",
@@ -38,7 +38,7 @@
"@push.rocks/smartdaemon": "^2.0.9", "@push.rocks/smartdaemon": "^2.0.9",
"@push.rocks/smartfile": "^11.2.7", "@push.rocks/smartfile": "^11.2.7",
"@push.rocks/smartinteract": "^2.0.16", "@push.rocks/smartinteract": "^2.0.16",
"@push.rocks/smartipc": "^2.2.2", "@push.rocks/smartipc": "^2.3.0",
"@push.rocks/smartpath": "^6.0.0", "@push.rocks/smartpath": "^6.0.0",
"@types/pidusage": "^2.0.5", "@types/pidusage": "^2.0.5",
"@types/ps-tree": "^1.1.6", "@types/ps-tree": "^1.1.6",

10
pnpm-lock.yaml generated
View File

@@ -27,8 +27,8 @@ importers:
specifier: ^2.0.16 specifier: ^2.0.16
version: 2.0.16 version: 2.0.16
'@push.rocks/smartipc': '@push.rocks/smartipc':
specifier: ^2.2.2 specifier: ^2.3.0
version: 2.2.2 version: 2.3.0
'@push.rocks/smartpath': '@push.rocks/smartpath':
specifier: ^6.0.0 specifier: ^6.0.0
version: 6.0.0 version: 6.0.0
@@ -877,8 +877,8 @@ packages:
'@push.rocks/smartinteract@2.0.16': '@push.rocks/smartinteract@2.0.16':
resolution: {integrity: sha512-eltvVRRUKBKd77DSFA4DPY2g4V4teZLNe8A93CDy/WglglYcUjxMoLY/b0DFTWCWKYT+yjk6Fe6p0FRrvX9Yvg==} resolution: {integrity: sha512-eltvVRRUKBKd77DSFA4DPY2g4V4teZLNe8A93CDy/WglglYcUjxMoLY/b0DFTWCWKYT+yjk6Fe6p0FRrvX9Yvg==}
'@push.rocks/smartipc@2.2.2': '@push.rocks/smartipc@2.3.0':
resolution: {integrity: sha512-pkWqp2nQH7p5zD9Efh5KNX2O0+gFWL6bxbdd6SdDh4gP8Gb0b3Sn87Tpedghpc/d+LCVql+1pUf6OlvMQpD5Yw==} resolution: {integrity: sha512-/btC/DHf+2PWF6Qiq0oHHP7XHzacgYfHAShIts2ZXS+nhpvSyjucNzB2ErNUPHLMITNXGUSu5Wpt7sfvIQzxJQ==}
'@push.rocks/smartjson@5.0.20': '@push.rocks/smartjson@5.0.20':
resolution: {integrity: sha512-ogGBLyOTluphZVwBYNyjhm5sziPGuiAwWihW07OSRxD4HQUyqj9Ek6r1pqH07JUG5EbtRYivM1Yt1cCwnu3JVQ==} resolution: {integrity: sha512-ogGBLyOTluphZVwBYNyjhm5sziPGuiAwWihW07OSRxD4HQUyqj9Ek6r1pqH07JUG5EbtRYivM1Yt1cCwnu3JVQ==}
@@ -6360,7 +6360,7 @@ snapshots:
'@push.rocks/smartpromise': 4.2.3 '@push.rocks/smartpromise': 4.2.3
inquirer: 11.1.0 inquirer: 11.1.0
'@push.rocks/smartipc@2.2.2': '@push.rocks/smartipc@2.3.0':
dependencies: dependencies:
'@push.rocks/smartdelay': 3.0.5 '@push.rocks/smartdelay': 3.0.5
'@push.rocks/smartrx': 3.0.10 '@push.rocks/smartrx': 3.0.10

View File

@@ -177,11 +177,15 @@ Watch: disabled
#### `tspm logs <id|id:N|name:LABEL> [options]` #### `tspm logs <id|id:N|name:LABEL> [options]`
View process logs (stdout and stderr combined). View and stream process logs (stdout, stderr, and system messages).
**Options:** **Options:**
- `--lines <n>` - Number of lines to display (default: 50) - `--lines <n>` Number of lines to show (default: 50)
- `--follow` - Stream logs in real-time (like `tail -f`) - `--since <dur>` Only show logs since duration (e.g., `10m`, `2h`, `1d`; units: `ms|s|m|h|d`)
- `--stderr-only` Only show stderr logs
- `--stdout-only` Only show stdout logs
- `--ndjson` Output each log as JSON line (timestamp in ms)
- `--follow` Stream logs in real-time (like `tail -f`)
```bash ```bash
# View last 50 lines # View last 50 lines
@@ -190,10 +194,20 @@ tspm logs name:my-server
# View last 100 lines # View last 100 lines
tspm logs name:my-server --lines 100 tspm logs name:my-server --lines 100
# Follow logs in real-time # Only stderr for the last 10 minutes (as NDJSON)
tspm logs name:my-server --since 10m --stderr-only --ndjson
# Follow logs in real time (prints recent lines, then streams backlog incrementally and live logs)
tspm logs name:my-server --follow tspm logs name:my-server --follow
# Follow only stdout since 2h ago
tspm logs name:my-server --follow --since 2h --stdout-only
``` ```
Notes:
- Follow mode prints a small recent backlog, then streams older entries incrementally (to avoid large payloads) and continues with live logs.
- Log sequences are restart-aware; TSPM detects run changes and keeps output consistent across restarts.
### Batch Operations ### Batch Operations
#### `tspm start-all` #### `tspm start-all`
@@ -285,6 +299,18 @@ Processes: 5
Socket: /home/user/.tspm/tspm.sock Socket: /home/user/.tspm/tspm.sock
``` ```
#### Version check and service refresh
Check CLI vs daemon versions and refresh the systemd service if they differ:
```bash
tspm -v
# tspm CLI: 5.x.y
# Daemon: running v5.x.z (pid 1234)
# Version mismatch detected → optionally refresh the systemd service (equivalent to `tspm disable && tspm enable`).
```
This is helpful after upgrades where the system service still references an older CLI path.
### System Service Management ### System Service Management
Run TSPM as a system service (systemd) for production deployments. Run TSPM as a system service (systemd) for production deployments.

View File

@@ -3,6 +3,6 @@
*/ */
export const commitinfo = { export const commitinfo = {
name: '@git.zone/tspm', name: '@git.zone/tspm',
version: '5.4.1', version: '5.6.2',
description: 'a no fuzz process manager' description: 'a no fuzz process manager'
} }

View File

@@ -20,13 +20,13 @@ export function registerListCommand(smartcli: plugins.smartcli.Smartcli) {
console.log('Process List:'); console.log('Process List:');
console.log( console.log(
'┌─────────┬─────────────┬───────────┬───────────┬──────────┬──────────┐', '┌─────────┬─────────────┬───────────┬───────────┬──────────┬──────────┬─────────┐',
); );
console.log( console.log(
'│ ID │ Name │ Status │ PID │ Memory │ Restarts │', '│ ID │ Name │ Status │ PID │ Memory │ CPU │ Restarts │',
); );
console.log( console.log(
'├─────────┼─────────────┼───────────┼───────────┼──────────┼──────────┤', '├─────────┼─────────────┼───────────┼───────────┼──────────┼──────────┼──────────┤',
); );
for (const proc of processes) { for (const proc of processes) {
@@ -38,13 +38,18 @@ export function registerListCommand(smartcli: plugins.smartcli.Smartcli) {
: '\x1b[33m'; : '\x1b[33m';
const resetColor = '\x1b[0m'; const resetColor = '\x1b[0m';
const cpuStr = typeof proc.cpu === 'number' && isFinite(proc.cpu)
? `${proc.cpu.toFixed(1)}%`
: '-';
// Name is not part of IProcessInfo; show ID as placeholder for now
const nameDisplay = String(proc.id);
console.log( console.log(
`${pad(String(proc.id), 7)}${pad(String(proc.id), 11)}${statusColor}${pad(proc.status, 9)}${resetColor}${pad((proc.pid || '-').toString(), 9)}${pad(formatMemory(proc.memory), 8)}${pad(proc.restarts.toString(), 8)}`, `${pad(String(proc.id), 7)}${pad(nameDisplay, 11)}${statusColor}${pad(proc.status, 9)}${resetColor}${pad((proc.pid || '-').toString(), 9)}${pad(formatMemory(proc.memory), 8)} ${pad(cpuStr, 8)} ${pad(proc.restarts.toString(), 8)}`,
); );
} }
console.log( console.log(
'└─────────┴─────────────┴───────────┴───────────┴──────────┴──────────┘', '└─────────┴─────────────┴───────────┴───────────┴──────────┴──────────┴──────────┘',
); );
}, },
{ actionLabel: 'list processes' }, { actionLabel: 'list processes' },

View File

@@ -2,7 +2,7 @@ import * as plugins from '../../plugins.js';
import { tspmIpcClient } from '../../../client/tspm.ipcclient.js'; import { tspmIpcClient } from '../../../client/tspm.ipcclient.js';
import type { CliArguments } from '../../types.js'; import type { CliArguments } from '../../types.js';
import { registerIpcCommand } from '../../registration/index.js'; import { registerIpcCommand } from '../../registration/index.js';
import { getBool, getNumber } from '../../helpers/argv.js'; import { getBool, getNumber, getString } from '../../helpers/argv.js';
import { formatLog } from '../../helpers/formatting.js'; import { formatLog } from '../../helpers/formatting.js';
import { withStreamingLifecycle } from '../../helpers/lifecycle.js'; import { withStreamingLifecycle } from '../../helpers/lifecycle.js';
@@ -17,22 +17,60 @@ export function registerLogsCommand(smartcli: plugins.smartcli.Smartcli) {
console.log('Usage: tspm logs <id | id:N | name:LABEL> [options]'); console.log('Usage: tspm logs <id | id:N | name:LABEL> [options]');
console.log('\nOptions:'); console.log('\nOptions:');
console.log(' --lines <n> Number of lines to show (default: 50)'); console.log(' --lines <n> Number of lines to show (default: 50)');
console.log(' --since <dur> Only show logs since duration (e.g., 10m, 2h, 1d)');
console.log(' --stderr-only Only show stderr logs');
console.log(' --stdout-only Only show stdout logs');
console.log(' --ndjson Output each log as JSON line');
console.log(' --follow Stream logs in real-time (like tail -f)'); console.log(' --follow Stream logs in real-time (like tail -f)');
return; return;
} }
const lines = getNumber(argvArg, 'lines', 50); const lines = getNumber(argvArg, 'lines', 50);
const follow = getBool(argvArg, 'follow', 'f'); const follow = getBool(argvArg, 'follow', 'f');
const sinceSpec = getString(argvArg, 'since');
const stderrOnly = getBool(argvArg, 'stderr-only');
const stdoutOnly = getBool(argvArg, 'stdout-only');
const ndjson = getBool(argvArg, 'ndjson');
const parseDuration = (spec?: string): number | undefined => {
if (!spec) return undefined;
const m = spec.trim().match(/^(\d+)(ms|s|m|h|d)?$/i);
if (!m) return undefined;
const val = Number(m[1]);
const unit = (m[2] || 'm').toLowerCase();
const mult = unit === 'ms' ? 1 : unit === 's' ? 1000 : unit === 'm' ? 60000 : unit === 'h' ? 3600000 : 86400000;
return Date.now() - val * mult;
};
const sinceTime = parseDuration(sinceSpec);
const typesFilter: Array<'stdout' | 'stderr' | 'system'> | undefined =
stderrOnly && !stdoutOnly
? ['stderr']
: stdoutOnly && !stderrOnly
? ['stdout']
: undefined; // all
const resolved = await tspmIpcClient.request('resolveTarget', { target: String(target) }); const resolved = await tspmIpcClient.request('resolveTarget', { target: String(target) });
const id = resolved.id; const id = resolved.id;
const response = await tspmIpcClient.request('getLogs', { id, lines }); const response = await tspmIpcClient.request('getLogs', { id, lines: sinceTime ? 0 : lines });
if (!follow) { if (!follow) {
// One-shot mode - auto-disconnect handled by registerIpcCommand // One-shot mode - auto-disconnect handled by registerIpcCommand
console.log(`Logs for process: ${id} (last ${lines} lines)`); const filtered = response.logs.filter((l) => {
if (typesFilter && !typesFilter.includes(l.type)) return false;
if (sinceTime && new Date(l.timestamp).getTime() < sinceTime) return false;
return true;
});
console.log(`Logs for process: ${id} (${sinceTime ? 'since ' + new Date(sinceTime).toLocaleString() : 'last ' + lines + ' lines'})`);
console.log('─'.repeat(60)); console.log('─'.repeat(60));
for (const log of response.logs) { for (const log of filtered) {
if (ndjson) {
console.log(
JSON.stringify({
...log,
timestamp: new Date(log.timestamp).getTime(),
}),
);
} else {
const timestamp = new Date(log.timestamp).toLocaleTimeString(); const timestamp = new Date(log.timestamp).toLocaleTimeString();
const prefix = const prefix =
log.type === 'stdout' log.type === 'stdout'
@@ -42,6 +80,7 @@ export function registerLogsCommand(smartcli: plugins.smartcli.Smartcli) {
: '[SYS]'; : '[SYS]';
console.log(`${timestamp} ${prefix} ${log.message}`); console.log(`${timestamp} ${prefix} ${log.message}`);
} }
}
return; return;
} }
@@ -49,8 +88,20 @@ export function registerLogsCommand(smartcli: plugins.smartcli.Smartcli) {
console.log(`Logs for process: ${resolved.name || id} (streaming...)`); console.log(`Logs for process: ${resolved.name || id} (streaming...)`);
console.log('─'.repeat(60)); console.log('─'.repeat(60));
// Prepare backlog printing state and stream handler
let lastSeq = 0; let lastSeq = 0;
for (const log of response.logs) { let lastRunId: string | undefined = undefined;
const printLog = (log: any) => {
if (typesFilter && !typesFilter.includes(log.type)) return;
if (sinceTime && new Date(log.timestamp).getTime() < sinceTime) return;
if (ndjson) {
console.log(
JSON.stringify({
...log,
timestamp: new Date(log.timestamp).getTime(),
}),
);
} else {
const timestamp = new Date(log.timestamp).toLocaleTimeString(); const timestamp = new Date(log.timestamp).toLocaleTimeString();
const prefix = const prefix =
log.type === 'stdout' log.type === 'stdout'
@@ -59,26 +110,61 @@ export function registerLogsCommand(smartcli: plugins.smartcli.Smartcli) {
? '[ERR]' ? '[ERR]'
: '[SYS]'; : '[SYS]';
console.log(`${timestamp} ${prefix} ${log.message}`); console.log(`${timestamp} ${prefix} ${log.message}`);
}
};
// Print initial backlog (already fetched via getLogs)
for (const log of response.logs) {
printLog(log);
if (log.seq !== undefined) lastSeq = Math.max(lastSeq, log.seq); if (log.seq !== undefined) lastSeq = Math.max(lastSeq, log.seq);
if ((log as any).runId) lastRunId = (log as any).runId;
} }
await withStreamingLifecycle( // Request additional backlog delivered as incremental messages to avoid large payloads
async () => { try {
await tspmIpcClient.subscribe(id, (log: any) => { const disposeBacklog = tspmIpcClient.onBacklogTopic(id, (log: any) => {
if (log.runId && log.runId !== lastRunId) {
console.log(`[INFO] Detected process restart (runId changed).`);
lastSeq = -1;
lastRunId = log.runId;
}
if (log.seq !== undefined && log.seq <= lastSeq) return; if (log.seq !== undefined && log.seq <= lastSeq) return;
if (log.seq !== undefined && log.seq > lastSeq + 1) { if (log.seq !== undefined && log.seq > lastSeq + 1) {
console.log( console.log(
`[WARNING] Log gap detected: expected seq ${lastSeq + 1}, got ${log.seq}`, `[WARNING] Log gap detected: expected seq ${lastSeq + 1}, got ${log.seq}`,
); );
} }
const timestamp = new Date(log.timestamp).toLocaleTimeString(); printLog({ ...log, timestamp: new Date(log.timestamp) });
const prefix = if (log.seq !== undefined) lastSeq = log.seq;
log.type === 'stdout' });
? '[OUT]' await tspmIpcClient.requestLogsBacklogStream(id, { lines: sinceTime ? undefined : lines, sinceTime, types: typesFilter });
: log.type === 'stderr' // Dispose backlog handler after a short grace (backlog is finite)
? '[ERR]' setTimeout(() => disposeBacklog(), 10000);
: '[SYS]'; } catch {}
console.log(`${timestamp} ${prefix} ${log.message}`);
await withStreamingLifecycle(
async () => {
// Optional: debug subscribers if requested via env (hidden)
if (process.env.TSPM_DEBUG === 'true') {
try {
const subInfo = await tspmIpcClient.request('logs:subscribers' as any, { id });
console.log(`[DEBUG] Subscribers for logs.${id}: ${subInfo.count} (${(subInfo.subscribers||[]).join(',')})`);
} catch {}
}
await tspmIpcClient.subscribe(id, (log: any) => {
// Reset sequence if runId changed (e.g., process restarted)
if (log.runId && log.runId !== lastRunId) {
console.log(`[INFO] Detected process restart (runId changed).`);
lastSeq = -1;
lastRunId = log.runId;
}
if (log.seq !== undefined && log.seq <= lastSeq) return;
if (log.seq !== undefined && log.seq > lastSeq + 1) {
console.log(
`[WARNING] Log gap detected: expected seq ${lastSeq + 1}, got ${log.seq}`,
);
}
printLog(log);
if (log.seq !== undefined) lastSeq = log.seq; if (log.seq !== undefined) lastSeq = log.seq;
}); });
}, },

View File

@@ -160,6 +160,55 @@ export class TspmIpcClient {
await this.ipcClient.subscribe(topic, handler); await this.ipcClient.subscribe(topic, handler);
} }
/**
* Request backlog logs as a stream from the daemon.
* The actual stream will be delivered via the 'stream' event.
*/
public async requestLogsBacklogStream(
processId: ProcessId | number | string,
opts: { lines?: number; sinceTime?: number; types?: Array<'stdout' | 'stderr' | 'system'> } = {},
): Promise<void> {
if (!this.ipcClient || !this.isConnected) {
throw new Error('Not connected to daemon');
}
const id = toProcessId(processId);
await this.request('logs:subscribe' as any, {
id,
lines: opts.lines,
sinceTime: opts.sinceTime,
types: opts.types,
} as any);
}
/**
* Register a handler for incoming streams (e.g., backlog logs)
*/
public onStream(
handler: (info: any, readable: NodeJS.ReadableStream) => void,
): void {
if (!this.ipcClient) throw new Error('Not connected to daemon');
// smartipc emits 'stream' with (info, readable)
(this.ipcClient as any).on('stream', handler);
}
/**
* Register a temporary handler for backlog topic messages for a specific process
*/
public onBacklogTopic(
processId: ProcessId | number | string,
handler: (log: any) => void,
): () => void {
if (!this.ipcClient) throw new Error('Not connected to daemon');
const id = toProcessId(processId);
const topicType = `topic:logs.backlog.${id}`;
(this.ipcClient as any).onMessage(topicType, handler);
return () => {
try {
(this.ipcClient as any).messageHandlers?.delete?.(topicType);
} catch {}
};
}
/** /**
* Unsubscribe from log updates for a specific process * Unsubscribe from log updates for a specific process
*/ */

View File

@@ -438,6 +438,13 @@ export class ProcessManager extends EventEmitter {
info.uptime = uptime; info.uptime = uptime;
} }
// Update memory and cpu from latest monitor readings
info.memory = monitor.getLastMemoryUsage();
const cpu = monitor.getLastCpuUsage();
if (Number.isFinite(cpu)) {
info.cpu = cpu;
}
// Update restart count // Update restart count
info.restarts = monitor.getRestartCount(); info.restarts = monitor.getRestartCount();
@@ -492,8 +499,12 @@ export class ProcessManager extends EventEmitter {
*/ */
public async startAll(): Promise<void> { public async startAll(): Promise<void> {
for (const [id, config] of this.processConfigs.entries()) { for (const [id, config] of this.processConfigs.entries()) {
if (!this.processes.has(id)) { const monitor = this.processes.get(id);
if (!monitor) {
await this.start(config); await this.start(config);
} else if (!monitor.isRunning()) {
// If a monitor exists but is not running, restart the process to ensure a clean start
await this.restart(id);
} }
} }
} }

View File

@@ -24,6 +24,8 @@ export class ProcessMonitor extends EventEmitter {
private lastRetryAt: number | null = null; private lastRetryAt: number | null = null;
private readonly MAX_RETRIES = 10; private readonly MAX_RETRIES = 10;
private readonly RESET_WINDOW_MS = 60 * 60 * 1000; // 1 hour private readonly RESET_WINDOW_MS = 60 * 60 * 1000; // 1 hour
private lastMemoryUsage: number = 0;
private lastCpuUsage: number = 0;
constructor(config: IMonitorConfig & { id?: ProcessId }) { constructor(config: IMonitorConfig & { id?: ProcessId }) {
super(); super();
@@ -260,12 +262,16 @@ export class ProcessMonitor extends EventEmitter {
memoryLimit: number, memoryLimit: number,
): Promise<void> { ): Promise<void> {
try { try {
const memoryUsage = await this.getProcessGroupMemory(pid); const { memory: memoryUsage, cpu: cpuUsage } = await this.getProcessGroupStats(pid);
this.logger.debug( this.logger.debug(
`Memory usage for PID ${pid}: ${this.humanReadableBytes(memoryUsage)} (${memoryUsage} bytes)`, `Memory usage for PID ${pid}: ${this.humanReadableBytes(memoryUsage)} (${memoryUsage} bytes)`,
); );
// Store latest readings
this.lastMemoryUsage = memoryUsage;
this.lastCpuUsage = cpuUsage;
// Only log memory usage in debug mode to avoid spamming // Only log memory usage in debug mode to avoid spamming
if (process.env.TSPM_DEBUG) { if (process.env.TSPM_DEBUG) {
this.log( this.log(
@@ -285,7 +291,7 @@ export class ProcessMonitor extends EventEmitter {
// Stop the process wrapper, which will trigger the exit handler and restart // Stop the process wrapper, which will trigger the exit handler and restart
if (this.processWrapper) { if (this.processWrapper) {
this.processWrapper.stop(); await this.processWrapper.stop();
} }
} }
} catch (error: Error | unknown) { } catch (error: Error | unknown) {
@@ -303,7 +309,7 @@ export class ProcessMonitor extends EventEmitter {
/** /**
* Get the total memory usage (in bytes) for the process group (the main process and its children). * Get the total memory usage (in bytes) for the process group (the main process and its children).
*/ */
private getProcessGroupMemory(pid: number): Promise<number> { private getProcessGroupStats(pid: number): Promise<{ memory: number; cpu: number }> {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
this.logger.debug( this.logger.debug(
`Getting memory usage for process group with PID ${pid}`, `Getting memory usage for process group with PID ${pid}`,
@@ -333,7 +339,7 @@ export class ProcessMonitor extends EventEmitter {
plugins.pidusage( plugins.pidusage(
pids, pids,
(err: Error | null, stats: Record<string, { memory: number }>) => { (err: Error | null, stats: Record<string, { memory: number; cpu: number }>) => {
if (err) { if (err) {
const processError = new ProcessError( const processError = new ProcessError(
`Failed to get process usage stats: ${err.message}`, `Failed to get process usage stats: ${err.message}`,
@@ -345,14 +351,16 @@ export class ProcessMonitor extends EventEmitter {
} }
let totalMemory = 0; let totalMemory = 0;
let totalCpu = 0;
for (const key in stats) { for (const key in stats) {
totalMemory += stats[key].memory; totalMemory += stats[key].memory;
totalCpu += Number.isFinite(stats[key].cpu) ? stats[key].cpu : 0;
} }
this.logger.debug( this.logger.debug(
`Total memory for process group: ${this.humanReadableBytes(totalMemory)}`, `Total memory for process group: ${this.humanReadableBytes(totalMemory)}`,
); );
resolve(totalMemory); resolve({ memory: totalMemory, cpu: totalCpu });
}, },
); );
}, },
@@ -392,6 +400,11 @@ export class ProcessMonitor extends EventEmitter {
if (this.intervalId) { if (this.intervalId) {
clearInterval(this.intervalId); clearInterval(this.intervalId);
} }
// Cancel any pending restart timer
if (this.restartTimer) {
clearTimeout(this.restartTimer);
this.restartTimer = null;
}
if (this.processWrapper) { if (this.processWrapper) {
// Clear pidusage state for current PID before stopping to avoid leaks // Clear pidusage state for current PID before stopping to avoid leaks
try { try {
@@ -400,7 +413,7 @@ export class ProcessMonitor extends EventEmitter {
(plugins.pidusage as any)?.clear?.(pidToClear); (plugins.pidusage as any)?.clear?.(pidToClear);
} }
} catch {} } catch {}
this.processWrapper.stop(); await this.processWrapper.stop();
} }
} }
@@ -448,6 +461,20 @@ export class ProcessMonitor extends EventEmitter {
return this.processWrapper?.isRunning() || false; return this.processWrapper?.isRunning() || false;
} }
/**
* Get last measured memory usage for the process group (bytes)
*/
public getLastMemoryUsage(): number {
return this.lastMemoryUsage;
}
/**
* Get last measured CPU usage for the process group (sum of group, percent)
*/
public getLastCpuUsage(): number {
return this.lastCpuUsage;
}
/** /**
* Helper method for logging messages with the instance name. * Helper method for logging messages with the instance name.
*/ */

View File

@@ -24,6 +24,26 @@ export class ProcessWrapper extends EventEmitter {
private stdoutRemainder: string = ''; private stdoutRemainder: string = '';
private stderrRemainder: string = ''; private stderrRemainder: string = '';
// Helper: send a signal to the process and all its children (best-effort)
private async killProcessTree(signal: NodeJS.Signals): Promise<void> {
if (!this.process || !this.process.pid) return;
const rootPid = this.process.pid;
await new Promise<void>((resolve) => {
plugins.psTree(rootPid, (err: any, children: ReadonlyArray<{ PID: string }>) => {
const pids: number[] = [rootPid, ...children.map((c) => Number(c.PID)).filter((n) => Number.isFinite(n))];
for (const pid of pids) {
try {
// Always signal individual PIDs to avoid accidentally targeting unrelated groups
process.kill(pid, signal);
} catch {
// ignore ESRCH/EPERM
}
}
resolve();
});
});
}
constructor(options: IProcessWrapperOptions) { constructor(options: IProcessWrapperOptions) {
super(); super();
this.options = options; this.options = options;
@@ -180,7 +200,7 @@ export class ProcessWrapper extends EventEmitter {
/** /**
* Stop the wrapped process * Stop the wrapped process
*/ */
public stop(): void { public async stop(): Promise<void> {
if (!this.process) { if (!this.process) {
this.logger.debug('Stop called but no process is running'); this.logger.debug('Stop called but no process is running');
this.addSystemLog('No process running'); this.addSystemLog('No process running');
@@ -193,30 +213,46 @@ export class ProcessWrapper extends EventEmitter {
// First try SIGTERM for graceful shutdown // First try SIGTERM for graceful shutdown
if (this.process.pid) { if (this.process.pid) {
try { try {
this.logger.debug(`Sending SIGTERM to process ${this.process.pid}`); this.logger.debug(`Sending SIGTERM to process tree rooted at ${this.process.pid}`);
process.kill(this.process.pid, 'SIGTERM'); await this.killProcessTree('SIGTERM');
// Give it 5 seconds to shut down gracefully // If the process already exited, return immediately
setTimeout((): void => { if (typeof this.process.exitCode === 'number') {
if (this.process && this.process.pid) { this.logger.debug('Process already exited, no need to wait');
return;
}
// Wait for exit or escalate
await new Promise<void>((resolve) => {
let settled = false;
const cleanup = () => {
if (settled) return;
settled = true;
resolve();
};
const onExit = () => cleanup();
this.process!.once('exit', onExit);
const killTimer = setTimeout(async () => {
if (!this.process || !this.process.pid) return cleanup();
this.logger.warn( this.logger.warn(
`Process ${this.process.pid} did not exit gracefully, force killing...`, `Process ${this.process.pid} did not exit gracefully, force killing tree...`,
);
this.addSystemLog(
'Process did not exit gracefully, force killing...',
); );
this.addSystemLog('Process did not exit gracefully, force killing...');
try { try {
process.kill(this.process.pid, 'SIGKILL'); await this.killProcessTree('SIGKILL');
} catch (error: Error | unknown) { } catch {}
// Process might have exited between checks // Give a short grace period after SIGKILL
this.logger.debug( setTimeout(() => cleanup(), 500);
`Failed to send SIGKILL, process probably already exited: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}
}, 5000); }, 5000);
// Safety cap in case neither exit nor timer fires (shouldn't happen)
setTimeout(() => {
clearTimeout(killTimer);
cleanup();
}, 10000);
});
} catch (error: Error | unknown) { } catch (error: Error | unknown) {
const processError = new ProcessError( const processError = new ProcessError(
error instanceof Error ? error.message : String(error), error instanceof Error ? error.message : String(error),

View File

@@ -293,11 +293,80 @@ export class TspmDaemon {
this.ipcServer.onMessage( this.ipcServer.onMessage(
'getLogs', 'getLogs',
async (request: RequestForMethod<'getLogs'>) => { async (request: RequestForMethod<'getLogs'>) => {
const logs = await this.tspmInstance.getLogs(toProcessId(request.id)); const id = toProcessId(request.id);
const logs = await this.tspmInstance.getLogs(id, request.lines);
return { logs }; return { logs };
}, },
); );
// Stream backlog logs and let client subscribe to live topic separately
this.ipcServer.onMessage(
'logs:subscribe',
async (
request: RequestForMethod<'logs:subscribe'>,
clientId: string,
) => {
const id = toProcessId(request.id);
// Determine backlog set
const allLogs = await this.tspmInstance.getLogs(id);
let filtered = allLogs;
if (request.types && request.types.length) {
filtered = filtered.filter((l) => request.types!.includes(l.type));
}
if (request.sinceTime && request.sinceTime > 0) {
filtered = filtered.filter(
(l) => new Date(l.timestamp).getTime() >= request.sinceTime!,
);
}
const lines = request.lines && request.lines > 0 ? request.lines : 0;
if (lines > 0 && filtered.length > lines) {
filtered = filtered.slice(-lines);
}
// Send backlog entries directly to the requesting client as topic messages
// in small batches to avoid overwhelming the transport or client.
const chunkSize = 200;
for (let i = 0; i < filtered.length; i += chunkSize) {
const chunk = filtered.slice(i, i + chunkSize);
await Promise.allSettled(
chunk.map((entry) =>
this.ipcServer.sendToClient(
clientId,
`topic:logs.backlog.${id}`,
{
...entry,
timestamp: new Date(entry.timestamp).getTime(),
},
),
),
);
// Yield a bit between chunks
await new Promise((r) => setTimeout(r, 5));
}
return { ok: true } as any;
},
);
// Inspect subscribers for a process log topic
this.ipcServer.onMessage(
'logs:subscribers',
async (
request: RequestForMethod<'logs:subscribers'>,
clientId: string,
) => {
const id = toProcessId(request.id);
const topic = `logs.${id}`;
try {
const topicIndex = (this.ipcServer as any).topicIndex as Map<string, Set<string>> | undefined;
const subs = Array.from(topicIndex?.get(topic) || []);
// Also include the requesting clientId if it has a local handler without subscription
return { topic, subscribers: subs, count: subs.length } as any;
} catch (err: any) {
return { topic, subscribers: [], count: 0 } as any;
}
},
);
// Resolve target (id:n | name:foo | numeric string) to ProcessId // Resolve target (id:n | name:foo | numeric string) to ProcessId
this.ipcServer.onMessage( this.ipcServer.onMessage(
'resolveTarget', 'resolveTarget',
@@ -381,10 +450,12 @@ export class TspmDaemon {
await this.tspmInstance.setDesiredStateForAll('stopped'); await this.tspmInstance.setDesiredStateForAll('stopped');
await this.tspmInstance.stopAll(); await this.tspmInstance.stopAll();
// Yield briefly to allow any pending exit events to settle
await new Promise((r) => setTimeout(r, 50));
// Get status of all processes // Determine which monitors are no longer running
for (const [id, processInfo] of this.tspmInstance.processInfo) { for (const [id, monitor] of this.tspmInstance.processes) {
if (processInfo.status === 'stopped') { if (!monitor.isRunning()) {
stopped.push(id); stopped.push(id);
} else { } else {
failed.push({ id, error: 'Failed to stop' }); failed.push({ id, error: 'Failed to stop' });

View File

@@ -139,6 +139,29 @@ export interface GetLogsResponse {
logs: IProcessLog[]; logs: IProcessLog[];
} }
// Subscribe and stream backlog logs
export interface LogsSubscribeRequest {
id: ProcessId;
lines?: number; // number of backlog lines
sinceTime?: number; // ms epoch
types?: Array<IProcessLog['type']>;
}
export interface LogsSubscribeResponse {
ok: boolean;
}
// Inspect current subscribers for a process log topic
export interface LogsSubscribersRequest {
id: ProcessId;
}
export interface LogsSubscribersResponse {
topic: string;
subscribers: string[];
count: number;
}
// Start all command // Start all command
export interface StartAllRequest { export interface StartAllRequest {
// No parameters needed // No parameters needed
@@ -274,6 +297,8 @@ export type IpcMethodMap = {
list: { request: ListRequest; response: ListResponse }; list: { request: ListRequest; response: ListResponse };
describe: { request: DescribeRequest; response: DescribeResponse }; describe: { request: DescribeRequest; response: DescribeResponse };
getLogs: { request: GetLogsRequest; response: GetLogsResponse }; getLogs: { request: GetLogsRequest; response: GetLogsResponse };
'logs:subscribe': { request: LogsSubscribeRequest; response: LogsSubscribeResponse };
'logs:subscribers': { request: LogsSubscribersRequest; response: LogsSubscribersResponse };
startAll: { request: StartAllRequest; response: StartAllResponse }; startAll: { request: StartAllRequest; response: StartAllResponse };
stopAll: { request: StopAllRequest; response: StopAllResponse }; stopAll: { request: StopAllRequest; response: StopAllResponse };
restartAll: { request: RestartAllRequest; response: RestartAllResponse }; restartAll: { request: RestartAllRequest; response: RestartAllResponse };