@foss.global/codefeed

Generate an activity feed from forge providers. The default provider scans a Gitea instance; CodeFeedGitManagerProvider adapts a local @foss.global/gitmanager instance. CodeFeed retrieves commits since a configurable timestamp and enriches them with tags, optional npm publish detection, and CHANGELOG snippets.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Install

pnpm add @foss.global/codefeed

Requires Node.js 22+ and ESM.

Read-only forge inventory

ForgeInventoryScanner observes repositories and namespaces on a Gitea or GitLab instance, with optional issue inventory, server version discovery and authenticated namespace directories. It returns operator-facing DTOs from @foss.global/interfaces, distinct observed counts, available Git size estimates, and request/response-byte/rate-limit metrics. The scanner uses its own bounded GET transport. It does not mutate the forge or persist runtime data.

import { ForgeInventoryScanner } from '@foss.global/codefeed';

const scanner = new ForgeInventoryScanner({
  kind: 'gitlab', // or 'gitea'
  instanceId: 'engineering-forge', // stable operator ID, independent of its URL
  baseUrl: 'https://gitlab.example.com', // optional deployment subpath supported
  token: process.env.FORGE_INVENTORY_TOKEN, // omit for an anonymous observation
  pageSize: 50,
  maxPages: 1000,
  requestTimeoutMs: 30_000,
  maxResponseBytes: 8 * 1024 * 1024,
  // caCertificates: [privateForgeCaPem], // optional; replaces the default trust store for this forge
});

const inventory = await scanner.scan({
  maxItems: 100_000, // per collection
  timeoutMs: 60_000, // all collection and version requests together
  maxScanResponseBytes: 64 * 1024 * 1024,
  inspectVersion: true, // optional fresh version probe; default false
  includeIssues: true, // optional issue metadata scan; default false
  // namespaceDirectory: true, // requires a token; Gitea requires admin access
  // signal: abortController.signal,
});

const repositories = inventory.report.collections.repositories.result;
if (repositories.status === 'enumerated') {
  console.log(repositories.count, repositories.observation.access);
} else {
  // Inspect the explicit incomplete/uninspected result before making decisions.
  console.log(repositories.status);
}

Treat the result as private operator data. It can contain private repository paths and descriptions. Tokens stay in request headers, HTTPS is required, redirects are refused, and diagnostics omit raw response bodies and network errors. Credential-bearing clone URLs are rejected. SSH transport usernames are retained. Response-byte limits count decoded body bytes consumed by the reader; the chunk that crosses a limit is counted and discarded before JSON decoding. They are not HTTP wire-byte measurements or exact JavaScript heap limits.

Set caCertificates to reach a forge whose certificate chains to a private CA. Each entry is one PEM certificate; together they replace the default trust store for that adapter's (or scanner's) HTTPS connections, so include every anchor the forge's chain can end in. The adapter then sends its requests through node:https with its own keep-alive https.Agent instead of the global fetch. Nothing process-wide changes: NODE_TLS_REJECT_UNAUTHORIZED, the global fetch dispatcher and other connections keep their configuration. HTTPS-only base URLs, same-origin pagination links, redirect refusal, timeouts, response-byte limits and error codes apply unchanged. An empty list, more than 100 entries, a non-string entry, an entry larger than 64 KiB, a multi-certificate bundle in one entry or an unparseable entry fail construction with invalid-ca-certificates. Idle pooled connections do not keep the process alive.

The scanner marks observations publicOnly without a token and callerVisible with one. It never claims entireScope or a frozen snapshot. Gitea uses repository search and organization listing; GitLab uses keyset project pagination and offset group pagination with all_available=true. Short offset pages trigger another request because a server can cap page size below the requested limit. GitLab keyset completion relies on its documented next-Link protocol. Invalid continuations, denied requests, repeated IDs, cancellations and resource limits yield incomplete results. No automatic retry occurs. A scanner permits one active scan at a time, and each scan starts with fresh counts and metrics.

Every canonical entity class is present in the report. By default, the canonical namespaces result remains incomplete, including after organization/group endpoints are exhausted, because personal namespaces and full private namespace coverage are not established. Endpoint counts appear separately as gitea.organizations or gitlab.groups in additionalCollections.

Set namespaceDirectory: true to include personal namespaces. Gitea enumerates /admin/orgs followed by /admin/users; GitLab enumerates /namespaces without owner or top-level filters. A configured token is required. Access failures remain explicit and never fall back to public organization/group listings. Gitea's two directory phases share one maxPages limit, including empty-page probes. Successful completion produces an enumerated canonical namespace result within callerVisible access; additionalCollections is empty in this mode. It does not prove instance-wide administrator coverage, a frozen export, complete users, or permissions. Only namespace metadata is retained from the directory responses; user email/login-source and billing fields are discarded. Entity classes without an enabled reader remain notInspected with unverified capabilities.

Set inspectVersion: true to probe /api/v1/version on Gitea or /api/v4/version on GitLab before collection requests. versionInspection records observed with its timestamped observation, unverified with a safe reason, or notInspected when disabled. report.instance.version is set only after a successful probe in that scan. Denied or malformed responses never reuse an earlier version. A failed probe allows collection reads to proceed while the shared deadline and byte budget permit. The reported version is provider metadata, not proof that a real server version has been qualified.

Set includeIssues: true to read all visible open and closed issues in every observed repository. Gitea uses state=all&type=issues and rejects unexpected pull-request records. GitLab uses the stable project ID with state=all&scope=all, without author, confidentiality or issue-type filters. The returned issues array uses the shared IForgeInventoryIssue contract: stable row ID, separate repository-local number, repository identity, title/body, state, confidentiality, provider issue type, reported author and original timestamp precision/offset. Empty body text remains empty. Missing metadata is unknown; Gitea deleted/built-in authors remain unresolved. This is an observation of reported authors, not proof of original attribution for previously imported issues.

Issue pagination is bounded per repository by maxPages. The issue maxItems limit and duplicate detection span all observed repositories; deadline and response-byte limits span the entire scan. Gitea checks the current repository ID at its mutable path before each issue page, including empty pages, and also validates each issue's repository ID. These identity probes count in metrics and consume the same scan/caller byte budget. GitLab verifies each issue's returned project ID. A denied issue page, changed repository identity, incomplete repository enumeration or reached limit leaves issue inventory incomplete. No deletion or missing-issue inference follows from a partial scan. Comments, labels, milestones, assignees, attachments and history remain uninspected; outbound issue creation/update is not implemented by this scanner. Scan choices and the cancellation signal are captured at the beginning of each scan.

For one repository, call adapter.listIssues({ kind: 'repository', identity: repository.identity, location: repository.location }, { cursor?, signal?, maxResponseBytes? }) using a previously observed repository. Its cursor is additionally bound to that exact repository identity and location. The response carries repository scope; a foreign, unresolved or malformed repository identity is rejected before network access.

Provider IDs remain namespaced decimal strings, separate from mutable paths. Gitea organizations and repository owners share gitea.user.id; GitLab groups use gitlab.group.id, while a personal project namespace uses gitlab.namespace.id rather than a user ID. Nested paths preserve literal hyphens. Numeric JSON IDs or sizes outside JavaScript's safe integer range fail closed; canonical decimal strings retain their precision. Missing metadata remains unknown. A Gitea repository's visibility follows its owner's visibility, because Gitea's own internal flag is set only for a public repository of a private owner: a private repository, or a public one of a private owner, is private; a public repository of a limited owner is internal (signed-in users); a public repository of a public owner is public. A payload whose internal flag contradicts the owner visibility fails with inconsistent-repository-visibility. Gitea's rounded size field is not used as a separate Git byte measurement. Exposed GitLab repository statistics are estimates, and their sum is available only when every observed repository has a measurement. Neither quantity estimates bytes for a Git push.

The adapter's read-only baseUrl property exposes the validated, normalized HTTPS instance root, including its deployment subpath and without a trailing slash. Connection management can retain this value without copying URL validation or exposing the private token. It identifies a location, not an instance identity or verified server.

For incremental consumption, ForgeInventoryAdapter.listRepositories({ cursor?, signal?, maxResponseBytes? }), .listNamespaces(...) and .listNamespaceDirectory(...) return shared inventory pages. A more result contains a signed cursor bound to that adapter object, collection, directory phase and fixed query/page policy. Reuse it only with the same adapter; it is disposable process-local state, not a durable checkpoint. exhausted describes the selected endpoint or completed directory within the recorded visibility. Page methods throw ForgeInventoryError for request and payload failures, while malformed continuation metadata returns an incomplete page retaining its validated items. The scanner converts those failures into an incomplete collection report. .inspectVersion({ signal?, maxResponseBytes? }) returns a fresh observed/unverified result separately; it does not mutate the adapter's instance or page metadata, whose version remains unknown.

The implementation has synthetic protocol fixtures based on the Gitea 1.24.7 API schema, Gitea pagination documentation, and the GitLab projects, groups, namespaces, metadata, issues and pagination contracts. For an administrator's complete Gitea measurement, see the source inventory below. Native inventory, collision analysis and migration manifests remain pending.

Authenticated Gitea source inventory

GiteaSourceInventory measures a whole Gitea instance before its one-time migration, read with a site administrator's token. It returns the IForgeSourceInventory contract from @foss.global/interfaces: per-instance, per-owner and per-repository counts with totals, where every measure is complete, partial (attempted, did not finish) or notMeasured (not attempted). The inventory is complete only when discovery finished and every measure is complete; otherwise inventoryComplete is false and incompleteReasons names each gap. It never writes to the forge: it sends REST GET requests, clones each repository with Git, and asks the LFS batch API, as a download, which objects Gitea holds.

import { GiteaSourceInventory, composeGiteaSourceInventory } from '@foss.global/codefeed';

const reader = new GiteaSourceInventory({
  instanceId: 'code-foss-global',
  baseUrl: 'https://gitea.example.com', // HTTPS only
  token: siteAdminToken, // read it from a file; never pass it on a command line
  gitWorkDir: '/var/lib/inventory/mirrors', // absolute, private; holds one temporary mirror per repository in flight
  // caCertificates: [privateCaPem], gitTimeoutMs: 30 * 60 * 1000, pageSize: 50, maxPages: 10_000, requestTimeoutMs: 60_000, maxResponseBytes: 16 * 1024 * 1024,
});

const discovery = await reader.discover(); // rejects with `reader-not-site-admin` for other tokens
const results = await reader.measureUnits(reader.units(discovery), {
  concurrency: 4, // units measured at once; requests within a unit are sequential
  onResult: async (resultArg) => { /* persist each unit result to resume a run */ },
});
const inventory = composeGiteaSourceInventory(discovery, results, [{ package: '@foss.global/codefeed', version }]);

A run is split into units: the instance, each owner and each repository. measureUnit never rejects; a unit whose primary record cannot be read becomes failed with a sanitized reason. A resumed run measures failed units, and units holding a partial measure, again and passes all results to composeGiteaSourceInventory, where the last result per unit wins. The composed document is deterministic: it carries no timestamps, sorts owners, repositories, packages and reasons, and does not depend on the order of the results. An aborted signal or a rejected onResult aborts the units in flight, hands over no further result, and rejects measureUnits once every worker has stopped.

Discovery records each repository's Gitea id as a canonical decimal string, and units() passes it into the repository unit. A unit with an id fails with repository-identity-changed when its name answers with another id, as after a rename and a new repository under the old name; a unit without one, from a discovery stored by an earlier version, is checked by name only. composeGiteaSourceInventory counts a repository result for another id than discovery recorded as a failed unit (repository: unit failed: repository-identity-changed), and adds repository: duplicate-id when two measured repositories share one id, which a discovery without ids cannot rule out.

lookupRepository(id) reads one repository by its id through GET /api/v1/repositories/{id}, which keeps answering after a rename or transfer. It resolves { status: 'present', id, owner, name } with the name Gitea gives the repository now, or { status: 'absent', id } only when Gitea answers 404, which a site administrator gets only for a deleted repository. Every other failed read rejects with its code, so a timeout or a 500 never counts as a deletion; an id that is not a canonical decimal rejects with invalid-repository-id without a request.

What is measured: users with their account flags, two-factor and SSH/GPG keys; organizations, members, teams, team memberships and repository grants; repositories with their flags, Git refs split into branches, tags, pull request refs and other refs, Git objects and their uncompressed bytes, LFS objects referenced anywhere in history with the bytes Gitea holds and the number it lacks, issues, pull requests, comments, reviews, review comments, labels, milestones, reactions and attachments; releases including drafts, assets and asset bytes; wiki pages; collaborators, team grants, deploy keys, webhooks, branch and tag protections and push mirrors; packages per type with versions, files and bytes; system webhooks, Actions runners, workflows, runs, artifacts and the number of Actions secrets and variables; unadopted repositories.

Beyond counts, each repository keeps records of what the migration recreates. releaseRecords lists every release, drafts and prereleases included, sorted by numeric id: tag name, target commitish, name, body, the draft and prerelease flags, created and published times as Gitea reports them (a draft reports its creation time as published, and its tag does not exist as a Git ref yet), an author snapshot (account id and login; Gitea's placeholder for a deleted account has id -1) and its assets sorted by numeric id with id, storage uuid, name, size, download count and creation time. Gitea exposes no content digest, so a record carries none; a replaced file carries another id and uuid, and the download count moves with every download, so it never decides whether a release changed. protections holds the branch protection rules sorted by rule name (code-unit order) with every restriction Gitea reads back, allowlists included, and the tag protection rules sorted by numeric id with their name pattern and allowlist. releaseRecords is null while a release count is not complete, and protections is null while either protection count is not; the counts then say why.

Secrets are counted, never named: the report carries no tokens, Actions secret or variable values or names, webhook secrets or mirror remote URLs. Reasons are adapter codes or fixed text, never response bodies or URLs. Gitea has no read that lists user-scoped Actions secrets; that is listed in limitations, since they are reissued. User-scoped webhooks, variables and runners are read as that user through the Sudo header.

Git-level measures come from a mirror clone of every ref Gitea advertises, refs/pull/* included, made with @foss.global/gitmanager in a fresh directory below gitWorkDir and removed afterwards. The reader's login and token form Basic credentials; the credential reaches Git only through GIT_CONFIG_* entries in the child process environment; it is never in the URL, the command line or any file. The private CA list applies to the clone as well. The mirror's HEAD becomes the repository's head: the ref it names as target and whether it resolves. In a repository that is not empty, a HEAD that does not resolve names a missing branch, such as a default branch deleted at the source, and a target other than refs/heads/<defaultBranch> means Gitea's setting and its Git HEAD disagree. head is null when the mirror or its refs could not be read. A detached source HEAD is one such case: GitManager's mirror import requires a symbolic HEAD, so the clone fails with git-mirror-failed, and this reader never reports a target of null. The mirror's distinct LFS pointers, from every reachable revision, are checked against Gitea in batches of 100: an object with a download action counts as held, one answered with error 404 as missing, and any other answer makes the LFS measures partial. A clone that fails or exceeds gitTimeoutMs makes every Git and LFS measure of that repository partial; only the API-reported size stays complete. A clone starts only once its scratch reservation, twice the reported size plus 64 MiB, fits the free space of the file system that holds gitWorkDir, counting the reservations of the clones already in flight from every reader in the process. A clone that does not fit waits for them, in arrival order; one that does not fit even alone makes every Git and LFS measure of its repository partial with git-workdir-too-small, so the inventory stays incomplete. Size gitWorkDir for the largest repositories measured at once.

Known gaps that keep an inventory incomplete:

• A repository with its issues or pull requests unit disabled hides both through the API, so its issues, pull requests, comments, reviews, reactions and attachments are notMeasured. A disabled wiki hides its pages the same way. Labels, milestones, releases and Actions stay readable while their units are disabled.

Listings page until an empty page or the x-total-count total is reached. A total that changes mid-listing or disagrees with the rows read, a repeated page, a malformed page, a failed request or the page limit makes that measure partial and keeps the rows already read; only a listing that does not exist, such as the wiki of a repository without one, counts as empty. A repository search that lists one repository id twice has shifted under its pages and may have skipped a repository: discovery keeps the first entry and adds discovery: repositories: duplicate-id to incompleteReasons.

Gitea source transfer

GiteaSourceTransfer reads one repository's Git data, LFS objects and release assets for the one-time migration, with the same reader credentials as the inventory. It never writes to Gitea: Git is read only through git-upload-pack, LFS only through batch download requests and their download actions, release assets only through GETs of their attachment.

import { GitManager } from '@foss.global/gitmanager';
import { GiteaSourceTransfer } from '@foss.global/codefeed';

const source = new GiteaSourceTransfer({
  baseUrl: 'https://gitea.example.com', // HTTPS only
  login: readerLogin,
  token: readerToken, // read it from a file; never pass it on a command line
  // caCertificates: [privateCaPem], gitTimeoutMs: 30 * 60 * 1000, requestTimeoutMs: 60_000, maxResponseBytes: 16 * 1024 * 1024,
  // lfsTransferTimeoutMs: 30 * 60 * 1000, releaseAssetTimeoutMs: 30 * 60 * 1000,
});

// A started GitManager rooted in the caller's own scratch space; codefeed creates no scratch roots.
const scratch = new GitManager({ dataDir: scratchDir });
await scratch.start();
const { repository, reportedSizeKiB } = await source.importMirror({ owner: 'team', name: 'app', id: '11' }, scratch, {
  target: { org: 'team', repo: 'app' },
  signal,
});

const download = await source.downloadLfsObject({ owner: 'team', name: 'app', id: '11' }, oid, size, { signal });
if (download.status === 'missing') {
  // Gitea answered error 404: the object is not held at the source.
} else {
  await pipeline(download.stream, destination); // errors unless the bytes are exactly the object
}

// A release asset as the inventory recorded it: { id, uuid, name, size } from releaseRecords[].assets[].
const asset = await source.downloadReleaseAsset({ owner: 'team', name: 'app', id: '11' }, release.id, recordedAsset, { signal });
await pipeline(asset.stream, destination); // errors unless exactly `size` bytes of the recorded asset arrived

A source repository is named by owner, name and its Gitea repository id, a canonical decimal string as the inventory records it; anything else rejects with invalid-repository. Git and the LFS API reach a repository only by name, and a name can move to another repository between two reads: Gitea answers a renamed repository's old path with a redirect, and a new repository can take the old name. Every source read is therefore bracketed by an identity read of GET /api/v1/repositories/{id}, which must still name owner/name (compared case-insensitively, as Gitea does). A repository that was renamed, or whose name now belongs to another repository, rejects with repository-identity-changed; a deleted one rejects with http-404. A transfer that fails is explained the same way when the identity read shows one of those two causes.

importMirror reads the repository's size from the REST API by id, reserves its scratch in the process-wide GitScratchBudget of the file system holding the GitManager's data directory (twice the reported size plus 64 MiB, admitted in arrival order), and then clones every ref into target with importMirror of @foss.global/gitmanager and reads the identity again. When that last read rejects, the clone has finished and stays at target; the caller discards it. The clone URL is formed from baseUrl and the repository's owner and name; the reader's login and token reach Git as a Basic credential only through GIT_CONFIG_* entries in the child process environment, never in the URL, the command line or any file. A repository that does not fit even alone rejects with git-workdir-too-small without cloning. The reservation is released when the clone ends. Failures reject with a fixed ForgeInventoryError code (invalid-repository, invalid-target, gitmanager-not-running, repository-identity-changed, a REST code such as http-404, git-workdir-failed, git-workdir-too-small, git-mirror-failed, request-aborted); GitManager error text is never passed on. An abort rejects with request-aborted. While the clone runs, the abort reaches the GitManager import through its own signal (@foss.global/gitmanager 2.4.0 or later): Git is ended, the call settles once it has exited, no repository is left at target, and the reservation is released then. The GitManager keeps serving its other work, so one engine can serve several imports and other commands at once; a stop() of it during the clone also rejects the import with request-aborted.

While the import runs, GitManager holds target: writes to it, such as initBareRepository, restoreRepositoryBundle, setDefaultBranch or a Smart HTTP push, fail with a GitManagerError whose code is repositoryBusy, and a second import into the same target is refused at once (codefeed rejects it with git-mirror-failed). Reads, of target as of other repositories, proceed.

downloadLfsObject sends one batch download request with the reader's Basic credentials, then a GET of the action's href with the action's headers only, and then reads the identity; when that read rejects, the stream is destroyed and the call rejects with its code. A missing object resolves as { status: 'missing' }, so a caller can list the missing oids; any other object error rejects with lfs-object-error-<code>, such as lfs-object-error-422 for a size Gitea disagrees with. The returned stream never holds the whole object: it counts and hashes the bytes as they pass and holds back the last chunk until the body ended with exactly size bytes whose SHA-256 is oid. Otherwise it errors with lfs-size-overrun, lfs-short-body or lfs-digest-mismatch; a content-length other than size rejects with lfs-size-mismatch before streaming. An abort errors it with request-aborted, a transfer longer than lfsTransferTimeoutMs or idle for requestTimeoutMs with lfs-transfer-timed-out. Destroying the stream ends the download. The stream fails internally with lfs-transfer-failed when body bytes still arrive after the download has ended, which happens only once the stream is destroyed, so a reader of the stream does not see it.

downloadReleaseAsset(repository, releaseId, asset) takes the release id and the asset's id, uuid, name and size as the inventory recorded them; anything else rejects with release-asset-invalid before any request. It reads the repository by id, then GET /api/v1/repos/{owner}/{name}/releases/{releaseId}, which must still carry an asset with the same id, uuid, name and size; otherwise it rejects with release-asset-identity-changed (a deleted release with http-404). The download count is not compared. It then GETs {baseUrl}/attachments/{uuid} with the reader's token authorization: no redirect is followed and no range is asked for, so any answer but 200 rejects, with http-<status> or, for a refused redirect, release-asset-download-failed, and a content-length other than size rejects with release-asset-size-mismatch. No digest is computed while the bytes stream: the stream counts them and holds back the last chunk until exactly size bytes have arrived and both identity reads have passed once more, so a consumer never receives a complete asset that is short, long or no longer the recorded one. Otherwise it errors with release-asset-size-overrun, release-asset-short-body, repository-identity-changed or release-asset-identity-changed. Content digests come from the content itself, such as a publisher's SHA256SUMS asset or the importer's own hash. An abort errors it with request-aborted, a download longer than releaseAssetTimeoutMs (default 30 minutes, at most six hours) or idle for requestTimeoutMs with release-asset-transfer-timed-out. A failed request is explained by the identity reads as for downloadLfsObject. Destroying the stream ends the download; the download's connections close with the stream. The stream fails internally with release-asset-transfer-failed when body bytes still arrive after the download has ended, which happens only once the stream is destroyed, so a reader of the stream does not see it.

Git LFS batch client

LfsBatchClient is a provider-neutral Git LFS batch API client with the basic transfer adapter, bound to one LFS endpoint and its credential. GiteaSourceTransfer downloads through it; the migration uploads to the native Git service through it.

import { LfsBatchClient } from '@foss.global/codefeed';

const lfs = new LfsBatchClient({
  endpoint: 'https://git.example.com/team/app.git/info/lfs', // https, or http only to 127.0.0.0/8 or [::1]
  credential: { kind: 'authorization', value: `Bearer ${grantToken}` }, // or { kind: 'basic', username, password }
  // caCertificates, requestTimeoutMs: 60_000, transferTimeoutMs: 30 * 60 * 1000, maxResponseBytes: 4 * 1024 * 1024, batchSize: 100,
});
const uploaded = await lfs.upload([{ oid, size, open: (signalArg) => openObjectStream(oid, signalArg) }], { signal });
// [{ oid, size, outcome: 'uploaded' | 'present' }]
const presence = await lfs.checkPresence([{ oid, size }], { signal }); // [{ oid, size, status: 'held' | 'missing' }]
const download = await lfs.download({ oid, size }, { signal }); // as for downloadLfsObject
lfs.close(); // optional: idle pooled connections close on their own

• upload sends one batch upload request per batchSize objects (at most 100). An object without an upload action is already present; for each other one it opens the source, PUTs exactly size bytes with the action's headers and Content-Length: size, and then calls the verify action when the server sent one. The bytes are hashed while they stream and the last chunk is held back, so a source that is not the object is never sent complete (lfs-size-overrun, lfs-short-body, lfs-digest-mismatch). Any object error rejects with lfs-object-error-<code>. A source that still yields bytes after its upload has ended fails internally with lfs-transfer-failed; that happens only once the upload has settled, so the caller does not see it.
• Action expiry: expires_in (whole seconds after the batch answer arrived) is preferred over expires_at (RFC 3339), as the batch API specifies; like the git-lfs client, an expires_in of 0 and Go's zero time mean no expiry, and any other malformed value fails with invalid-lfs-batch. An upload, verify or download action is used only while it stays valid for the expiry margin, 30 s or requestTimeoutMs when that is shorter. Objects are transferred one at a time after their batch answer, so a later object whose action would expire within the margin is asked for again in a batch of its own, once; when the fresh answer is again too short-lived the call fails with lfs-action-expired. A verify action that expires during its PUT is asked for the same way; a fresh answer without actions means the server holds the object.
• checkPresence sends batch download requests and downloads nothing: a download action means held, error 404 missing. It uses no action, so expiry is validated but not enforced there.
• The credential is sent only as the Authorization header of batch requests to the endpoint, never to action hrefs and never in a URL, error or result; hrefs get the action's headers only. Action headers may not set connection-level fields (host, content-length, transfer-encoding and similar).
• URL policy, enforced for the endpoint and every action href: https:, or http: only to an IP literal in 127.0.0.0/8 or [::1]; localhost and every other host name over HTTP are refused (unsafe-lfs-url), as is user information. An href must keep the endpoint's scheme, so an HTTPS server cannot move a transfer to plain HTTP (unsafe-lfs-href). Redirects are never followed.
• Every request is bounded: batch answers by maxResponseBytes, batch and verify requests by requestTimeoutMs, object transfers by transferTimeoutMs and an idle limit of requestTimeoutMs, everything by the caller's AbortSignal. Errors are fixed ForgeInventoryError codes.

Shared scratch budget

GitScratchBudget and mirrorScratchBytes are exported so that every scratch GitManager in a process admits its work through the same budget as the inventory and the source transfer. GitScratchBudget.forDirectory(directory) returns the process-wide budget of the file system holding directory; reserve(directory, bytes, signal) resolves with a release function once the bytes fit its free space next to the reservations in flight, in arrival order, or with null when they do not fit even alone. mirrorScratchBytes(reportedSizeKiB) is the reservation of one mirror clone: twice the reported size plus 64 MiB.

Quick Start

import { CodeFeed } from '@foss.global/codefeed';

// Fetch commits since one week ago (default), no caching
const feed = new CodeFeed('https://code.example.com', 'gitea_token');
const commits = await feed.fetchAllCommitsFromInstance();
console.log(commits);

With options

const thirtyDays = 30 * 24 * 60 * 60 * 1000;
const since = new Date(Date.now() - thirtyDays).toISOString();

const feed = new CodeFeed('https://code.example.com', 'gitea_token', since, {
  enableCache: true,          // keep results in memory
  cacheWindowMs: thirtyDays,  // trim cache to this window
  enableNpmCheck: true,       // check npm for published versions
  taggedOnly: false,          // return all commits (or only tagged)
  orgAllowlist: ['myorg'],    // only scan these orgs
  orgDenylist: ['archive'],   // skip these orgs
  repoAllowlist: ['myorg/app1', 'myorg/app2'], // only these repos
  repoDenylist: ['myorg/old-repo'],            // skip these repos
  untilTimestamp: new Date().toISOString(),    // optional upper bound
  verbose: true,               // print a short metrics summary
});

const commits = await feed.fetchAllCommitsFromInstance();

Advanced callers can provide a custom ICodeFeedProvider through the constructor options. The default provider keeps the existing Gitea HTTP behavior, while the provider contract lets adapters supply organizations, repositories, commits, tags, and changelog content from another backend.

With GitManager

import { CodeFeed, CodeFeedGitManagerProvider } from '@foss.global/codefeed';
import { GitManager } from '@foss.global/gitmanager';

const since = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();
const gitManager = new GitManager({
  dataDir: '/var/lib/foss-global/gitmanager',
});

await gitManager.start();

const feed = new CodeFeed('gitmanager://local', undefined, since, {
  provider: new CodeFeedGitManagerProvider({ gitManager }),
  enableNpmCheck: false,
});

const commits = await feed.fetchAllCommitsFromInstance();

await gitManager.stop();

Public GitManager read models

GitManagerPublicReadClient is an in-process adapter from a running GitManager to the shared public response models. It supports exactly five read methods:

• getOverview(query?)
• getOrg(org)
• getRepo(org, repo, path?, issueNumber?)
• getFile(org, repo, path)
• getCommits(org, repo, sha?)

Construction is default-deny. Provide an authoritative isRepositoryPublic predicate, or set allowAllRepositories: true only when every repository managed by the supplied GitManager is public:

import { GitManagerPublicReadClient } from '@foss.global/codefeed';

const publicRepositories = new Set([
  'foss.global/interfaces',
  'foss.global/gitmanager',
]);

const publicReads = new GitManagerPublicReadClient({
  gitManager,
  publicBaseUrl: 'https://foss.global',
  smartHttpBaseUrl: 'https://foss.global',
  isRepositoryPublic: async (repository) =>
    publicRepositories.has(`${repository.org}/${repository.repo}`),
});

const overview = await publicReads.getOverview();
const repository = await publicReads.getRepo('foss.global', 'gitmanager');

Construction fails when neither visibility option is supplied. Predicate failures reject the request, so visibility checks fail closed. When allowAllRepositories is true, the predicate is bypassed.

Public-model visibility and GitManager Smart HTTP visibility are separate policy surfaces. Synchronize both from the same authoritative source. A generated cloneUrl or sshUrl describes a URL; it does not prove that the corresponding transport is available or that the caller is authorized.

URL defaults:

• publicBaseUrl: https://foss.global
• smartHttpBaseUrl: the configured publicBaseUrl
• sshBaseUrl: ssh://git@code.foss.global:29419

Bounded safe-integer options:

• maxCommits: 40; accepted range 1–1,000.
• maxReleases: 50; accepted range 1–1,000.
• maxLanguageFiles: 1,000; accepted range 1–10,000.
• maxConcurrency: 8; accepted range 1–32.

Optional organizationMetadata, repositoryMetadata, and newsProvider callbacks enrich the mapped response data. They do not add forge capabilities. Models intentionally leave members, issues, issue comments, release assets, download URLs, forge metrics, identity fields, and verification fields empty when GitManager has no authoritative source for them. The optional issueNumber argument is accepted for public-request compatibility and currently ignored.

Individual GitManager read failures may map to empty or null fallback data so one unavailable tree, file, tag, commit list, or diff does not necessarily fail the whole public model. Visibility predicate failures are not converted to fallbacks.

This client does not start an HTTP server, configure Git Smart HTTP or SSH, terminate TLS, authenticate users, or implement write/admin APIs. A server layer can call these methods from its own authenticated typed handlers.

CodeFeed.fetchAllCommitsFromInstance() returns items with this shape:

interface ICommitResult {
  baseUrl: string;
  org: string;
  repo: string;
  timestamp: string;        // ISO date
  hash: string;             // commit SHA
  commitMessage: string;
  tagged: boolean;          // commit is pointed to by a tag
  publishedOnNpm: boolean;  // only when npm check enabled and tag matches
  prettyAgoTime: string;    // human-readable diff
  changelog: string | undefined; // snippet for matching tag version
}

The canonical public result contract is exported as ICodefeedCommitResult from @foss.global/interfaces. @foss.global/codefeed/interfaces keeps ICommitResult as a package-local alias for that contract.

Features

• Pagination for orgs, repos, commits, and tags (no missing pages)
• Retries with exponential backoff for 429/5xx and network errors
• CHANGELOG discovery with case variants (CHANGELOG.md, changelog.md, docs/CHANGELOG.md)
• Tag-to-version mapping based on tag names (vX.Y.Z → X.Y.Z)
• Optional npm publish detection via @org/repo package versions
• GitManager provider adapter for repositories managed by @foss.global/gitmanager
• Default-deny GitManager public read-model adapter for overview, organization, repository, file, and commit responses
• In-memory caching with window trimming and stable sorting
• Allow/deny filters for orgs and repos, optional time upper bound
• One-line metrics summary when verbose: true

Environment

• Gitea base URL and an optional token with read access
• For GitManager-backed feeds, a started @foss.global/gitmanager 2.x GitManager instance with imported or local repositories
• For GiteaSourceTransfer.importMirror, a started @foss.global/gitmanager 2.4.0 or later GitManager, whose transfers take an abort signal
• For public GitManager reads, an authoritative repository-visibility source or an explicit all-public declaration
• Node.js 22+ (global fetch; dependencies such as file-type 22 through @push.rocks/smartnpm declare Node.js 22)

Testing

The repo contains:

• An opt-in integration test using a GITEA_TOKEN from .nogit/ via @push.rocks/qenv.
• A mocked pagination test that does not require network.
• A GitManager provider mapping test using a local test stub.
• A default-deny GitManager public-client mapping test using a local test stub.
• A mocked GiteaSourceInventory test, Git-level GiteaSourceInventory and GiteaSourceTransfer tests and a release asset download test against a local HTTPS Git, LFS batch, REST and attachment server (needs openssl and git http-backend), and an opt-in one against a Docker Gitea from @foss.global/forgefixtures, run with FORGEFIXTURES=gitea pnpm test (Docker required).
• An LfsBatchClient test against local loopback LFS servers over HTTP and over HTTPS with a private CA.

Run tests:

pnpm test

For the integration test, set CODEFEED_RUN_LIVE=true and ensure GITEA_TOKEN is provided, e.g. via .nogit/ as used by test/test.live.node.ts.

Notes

• When taggedOnly is enabled, the feed includes only commits associated with tags.
• publishedOnNpm is computed by matching the tag-derived version against the npm registry for @org/repo.
• For very large instances, consider using allowlists/denylists and enabling caching for incremental runs.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.