@foss.global/codefeed

Generate an activity feed from forge providers. The default provider scans a Gitea instance; CodeFeedGitManagerProvider adapts a local @foss.global/gitmanager instance. CodeFeed retrieves commits since a configurable timestamp and enriches them with tags, optional npm publish detection, and CHANGELOG snippets.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Install

pnpm add @foss.global/codefeed

Requires Node.js 18+ (global fetch/Request/Response) and ESM.

Read-only forge inventory

ForgeInventoryScanner observes repositories and namespaces on a Gitea or GitLab instance, with optional issue inventory, server version discovery and authenticated namespace directories. It returns operator-facing DTOs from @foss.global/interfaces, distinct observed counts, available Git size estimates, and request/response-byte/rate-limit metrics. The scanner uses its own bounded GET transport. It does not mutate the forge or persist runtime data.

import { ForgeInventoryScanner } from '@foss.global/codefeed';

const scanner = new ForgeInventoryScanner({
  kind: 'gitlab', // or 'gitea'
  instanceId: 'engineering-forge', // stable operator ID, independent of its URL
  baseUrl: 'https://gitlab.example.com', // optional deployment subpath supported
  token: process.env.FORGE_INVENTORY_TOKEN, // omit for an anonymous observation
  pageSize: 50,
  maxPages: 1000,
  requestTimeoutMs: 30_000,
  maxResponseBytes: 8 * 1024 * 1024,
});

const inventory = await scanner.scan({
  maxItems: 100_000, // per collection
  timeoutMs: 60_000, // all collection and version requests together
  maxScanResponseBytes: 64 * 1024 * 1024,
  inspectVersion: true, // optional fresh version probe; default false
  includeIssues: true, // optional issue metadata scan; default false
  // namespaceDirectory: true, // requires a token; Gitea requires admin access
  // signal: abortController.signal,
});

const repositories = inventory.report.collections.repositories.result;
if (repositories.status === 'enumerated') {
  console.log(repositories.count, repositories.observation.access);
} else {
  // Inspect the explicit incomplete/uninspected result before making decisions.
  console.log(repositories.status);
}

Treat the result as private operator data. It can contain private repository paths and descriptions. Tokens stay in request headers, HTTPS is required, redirects are refused, and diagnostics omit raw response bodies and network errors. Credential-bearing clone URLs are rejected. SSH transport usernames are retained. Response-byte limits count decoded body bytes consumed by the reader; the chunk that crosses a limit is counted and discarded before JSON decoding. They are not HTTP wire-byte measurements or exact JavaScript heap limits.

The scanner marks observations publicOnly without a token and callerVisible with one. It never claims entireScope or a frozen snapshot. Gitea uses repository search and organization listing; GitLab uses keyset project pagination and offset group pagination with all_available=true. Short offset pages trigger another request because a server can cap page size below the requested limit. GitLab keyset completion relies on its documented next-Link protocol. Invalid continuations, denied requests, repeated IDs, cancellations and resource limits yield incomplete results. No automatic retry occurs. A scanner permits one active scan at a time, and each scan starts with fresh counts and metrics.

Every canonical entity class is present in the report. By default, the canonical namespaces result remains incomplete, including after organization/group endpoints are exhausted, because personal namespaces and full private namespace coverage are not established. Endpoint counts appear separately as gitea.organizations or gitlab.groups in additionalCollections.

Set namespaceDirectory: true to include personal namespaces. Gitea enumerates /admin/orgs followed by /admin/users; GitLab enumerates /namespaces without owner or top-level filters. A configured token is required. Access failures remain explicit and never fall back to public organization/group listings. Gitea's two directory phases share one maxPages limit, including empty-page probes. Successful completion produces an enumerated canonical namespace result within callerVisible access; additionalCollections is empty in this mode. It does not prove instance-wide administrator coverage, a frozen export, complete users, or permissions. Only namespace metadata is retained from the directory responses; user email/login-source and billing fields are discarded. Entity classes without an enabled reader remain notInspected with unverified capabilities.

Set inspectVersion: true to probe /api/v1/version on Gitea or /api/v4/version on GitLab before collection requests. versionInspection records observed with its timestamped observation, unverified with a safe reason, or notInspected when disabled. report.instance.version is set only after a successful probe in that scan. Denied or malformed responses never reuse an earlier version. A failed probe allows collection reads to proceed while the shared deadline and byte budget permit. The reported version is provider metadata, not proof that a real server version has been qualified.

Set includeIssues: true to read all visible open and closed issues in every observed repository. Gitea uses state=all&type=issues and rejects unexpected pull-request records. GitLab uses the stable project ID with state=all&scope=all, without author, confidentiality or issue-type filters. The returned issues array uses the shared IForgeInventoryIssue contract: stable row ID, separate repository-local number, repository identity, title/body, state, confidentiality, provider issue type, reported author and original timestamp precision/offset. Empty body text remains empty. Missing metadata is unknown; Gitea deleted/built-in authors remain unresolved. This is an observation of reported authors, not proof of original attribution for previously imported issues.

Issue pagination is bounded per repository by maxPages. The issue maxItems limit and duplicate detection span all observed repositories; deadline and response-byte limits span the entire scan. Gitea checks the current repository ID at its mutable path before each issue page, including empty pages, and also validates each issue's repository ID. These identity probes count in metrics and consume the same scan/caller byte budget. GitLab verifies each issue's returned project ID. A denied issue page, changed repository identity, incomplete repository enumeration or reached limit leaves issue inventory incomplete. No deletion or missing-issue inference follows from a partial scan. Comments, labels, milestones, assignees, attachments and history remain uninspected; outbound issue creation/update is not implemented by this scanner. Scan choices and the cancellation signal are captured at the beginning of each scan.

For one repository, call adapter.listIssues({ kind: 'repository', identity: repository.identity, location: repository.location }, { cursor?, signal?, maxResponseBytes? }) using a previously observed repository. Its cursor is additionally bound to that exact repository identity and location. The response carries repository scope; a foreign, unresolved or malformed repository identity is rejected before network access.

Provider IDs remain namespaced decimal strings, separate from mutable paths. Gitea organizations and repository owners share gitea.user.id; GitLab groups use gitlab.group.id, while a personal project namespace uses gitlab.namespace.id rather than a user ID. Nested paths preserve literal hyphens. Numeric JSON IDs or sizes outside JavaScript's safe integer range fail closed; canonical decimal strings retain their precision. Missing metadata remains unknown. Gitea's rounded size field is not used as a separate Git byte measurement. Exposed GitLab repository statistics are estimates, and their sum is available only when every observed repository has a measurement. Neither quantity estimates bytes for a Git push.

The adapter's read-only baseUrl property exposes the validated, normalized HTTPS instance root, including its deployment subpath and without a trailing slash. Connection management can retain this value without copying URL validation or exposing the private token. It identifies a location, not an instance identity or verified server.

For incremental consumption, ForgeInventoryAdapter.listRepositories({ cursor?, signal?, maxResponseBytes? }), .listNamespaces(...) and .listNamespaceDirectory(...) return shared inventory pages. A more result contains a signed cursor bound to that adapter object, collection, directory phase and fixed query/page policy. Reuse it only with the same adapter; it is disposable process-local state, not a durable checkpoint. exhausted describes the selected endpoint or completed directory within the recorded visibility. Page methods throw ForgeInventoryError for request and payload failures, while malformed continuation metadata returns an incomplete page retaining its validated items. The scanner converts those failures into an incomplete collection report. .inspectVersion({ signal?, maxResponseBytes? }) returns a fresh observed/unverified result separately; it does not mutate the adapter's instance or page metadata, whose version remains unknown.

The implementation has synthetic protocol fixtures based on the Gitea 1.24.7 API schema, Gitea pagination documentation, and the GitLab projects, groups, namespaces, metadata, issues and pagination contracts. Real server/version qualification, native inventory, administrator completeness, full ref/object enumeration, collaboration, packages, collision analysis and migration manifests remain pending.

Quick Start

import { CodeFeed } from '@foss.global/codefeed';

// Fetch commits since one week ago (default), no caching
const feed = new CodeFeed('https://code.example.com', 'gitea_token');
const commits = await feed.fetchAllCommitsFromInstance();
console.log(commits);

With options

const thirtyDays = 30 * 24 * 60 * 60 * 1000;
const since = new Date(Date.now() - thirtyDays).toISOString();

const feed = new CodeFeed('https://code.example.com', 'gitea_token', since, {
  enableCache: true,          // keep results in memory
  cacheWindowMs: thirtyDays,  // trim cache to this window
  enableNpmCheck: true,       // check npm for published versions
  taggedOnly: false,          // return all commits (or only tagged)
  orgAllowlist: ['myorg'],    // only scan these orgs
  orgDenylist: ['archive'],   // skip these orgs
  repoAllowlist: ['myorg/app1', 'myorg/app2'], // only these repos
  repoDenylist: ['myorg/old-repo'],            // skip these repos
  untilTimestamp: new Date().toISOString(),    // optional upper bound
  verbose: true,               // print a short metrics summary
});

const commits = await feed.fetchAllCommitsFromInstance();

Advanced callers can provide a custom ICodeFeedProvider through the constructor options. The default provider keeps the existing Gitea HTTP behavior, while the provider contract lets adapters supply organizations, repositories, commits, tags, and changelog content from another backend.

With GitManager

import { CodeFeed, CodeFeedGitManagerProvider } from '@foss.global/codefeed';
import { GitManager } from '@foss.global/gitmanager';

const since = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();
const gitManager = new GitManager({
  dataDir: '/var/lib/foss-global/gitmanager',
});

await gitManager.start();

const feed = new CodeFeed('gitmanager://local', undefined, since, {
  provider: new CodeFeedGitManagerProvider({ gitManager }),
  enableNpmCheck: false,
});

const commits = await feed.fetchAllCommitsFromInstance();

await gitManager.stop();

Public GitManager read models

GitManagerPublicReadClient is an in-process adapter from a running GitManager to the shared public response models. It supports exactly five read methods:

• getOverview(query?)
• getOrg(org)
• getRepo(org, repo, path?, issueNumber?)
• getFile(org, repo, path)
• getCommits(org, repo, sha?)

Construction is default-deny. Provide an authoritative isRepositoryPublic predicate, or set allowAllRepositories: true only when every repository managed by the supplied GitManager is public:

import { GitManagerPublicReadClient } from '@foss.global/codefeed';

const publicRepositories = new Set([
  'foss.global/interfaces',
  'foss.global/gitmanager',
]);

const publicReads = new GitManagerPublicReadClient({
  gitManager,
  publicBaseUrl: 'https://foss.global',
  smartHttpBaseUrl: 'https://foss.global',
  isRepositoryPublic: async (repository) =>
    publicRepositories.has(`${repository.org}/${repository.repo}`),
});

const overview = await publicReads.getOverview();
const repository = await publicReads.getRepo('foss.global', 'gitmanager');

Construction fails when neither visibility option is supplied. Predicate failures reject the request, so visibility checks fail closed. When allowAllRepositories is true, the predicate is bypassed.

Public-model visibility and GitManager Smart HTTP visibility are separate policy surfaces. Synchronize both from the same authoritative source. A generated cloneUrl or sshUrl describes a URL; it does not prove that the corresponding transport is available or that the caller is authorized.

URL defaults:

• publicBaseUrl: https://foss.global
• smartHttpBaseUrl: the configured publicBaseUrl
• sshBaseUrl: ssh://git@code.foss.global:29419

Bounded safe-integer options:

• maxCommits: 40; accepted range 1–1,000.
• maxReleases: 50; accepted range 1–1,000.
• maxLanguageFiles: 1,000; accepted range 1–10,000.
• maxConcurrency: 8; accepted range 1–32.

Optional organizationMetadata, repositoryMetadata, and newsProvider callbacks enrich the mapped response data. They do not add forge capabilities. Models intentionally leave members, issues, issue comments, release assets, download URLs, forge metrics, identity fields, and verification fields empty when GitManager has no authoritative source for them. The optional issueNumber argument is accepted for public-request compatibility and currently ignored.

Individual GitManager read failures may map to empty or null fallback data so one unavailable tree, file, tag, commit list, or diff does not necessarily fail the whole public model. Visibility predicate failures are not converted to fallbacks.

This client does not start an HTTP server, configure Git Smart HTTP or SSH, terminate TLS, authenticate users, or implement write/admin APIs. A server layer can call these methods from its own authenticated typed handlers.

CodeFeed.fetchAllCommitsFromInstance() returns items with this shape:

interface ICommitResult {
  baseUrl: string;
  org: string;
  repo: string;
  timestamp: string;        // ISO date
  hash: string;             // commit SHA
  commitMessage: string;
  tagged: boolean;          // commit is pointed to by a tag
  publishedOnNpm: boolean;  // only when npm check enabled and tag matches
  prettyAgoTime: string;    // human-readable diff
  changelog: string | undefined; // snippet for matching tag version
}

The canonical public result contract is exported as ICodefeedCommitResult from @foss.global/interfaces. @foss.global/codefeed/interfaces keeps ICommitResult as a package-local alias for that contract.

Features

• Pagination for orgs, repos, commits, and tags (no missing pages)
• Retries with exponential backoff for 429/5xx and network errors
• CHANGELOG discovery with case variants (CHANGELOG.md, changelog.md, docs/CHANGELOG.md)
• Tag-to-version mapping based on tag names (vX.Y.Z → X.Y.Z)
• Optional npm publish detection via @org/repo package versions
• GitManager provider adapter for repositories managed by @foss.global/gitmanager
• Default-deny GitManager public read-model adapter for overview, organization, repository, file, and commit responses
• In-memory caching with window trimming and stable sorting
• Allow/deny filters for orgs and repos, optional time upper bound
• One-line metrics summary when verbose: true

Environment

• Gitea base URL and an optional token with read access
• For GitManager-backed feeds, a started GitManager instance with imported or local repositories
• For public GitManager reads, an authoritative repository-visibility source or an explicit all-public declaration
• Node.js 18+ (global fetch)

Testing

The repo contains:

• An opt-in integration test using a GITEA_TOKEN from .nogit/ via @push.rocks/qenv.
• A mocked pagination test that does not require network.
• A GitManager provider mapping test using a local test stub.
• A default-deny GitManager public-client mapping test using a local test stub.

Run tests:

pnpm test

For the integration test, set CODEFEED_RUN_LIVE=true and ensure GITEA_TOKEN is provided, e.g. via .nogit/ as used by test/test.live.node.ts.

Notes

• When taggedOnly is enabled, the feed includes only commits associated with tags.
• publishedOnNpm is computed by matching the tag-derived version against the npm registry for @org/repo.
• For very large instances, consider using allowlists/denylists and enabling caching for incremental runs.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.