Generate an activity feed from forge providers. The default provider scans a Gitea instance; CodeFeedGitManagerProvider adapts a local @foss.global/gitmanager instance. CodeFeed retrieves commits since a configurable timestamp and enriches them with tags, optional npm publish detection, and CHANGELOG snippets.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Install

pnpm add @foss.global/codefeed

Requires Node.js 22+ and ESM.

Read-only forge inventory

ForgeInventoryScanner observes repositories and namespaces on a Gitea or GitLab instance, with optional issue inventory, server version discovery and authenticated namespace directories. It returns operator-facing DTOs from @foss.global/interfaces, distinct observed counts, available Git size estimates, and request/response-byte/rate-limit metrics. The scanner uses its own bounded GET transport. It does not mutate the forge or persist runtime data.

import { ForgeInventoryScanner } from '@foss.global/codefeed';

const scanner = new ForgeInventoryScanner({
  kind: 'gitlab', // or 'gitea'
  instanceId: 'engineering-forge', // stable operator ID, independent of its URL
  baseUrl: 'https://gitlab.example.com', // optional deployment subpath supported
  token: process.env.FORGE_INVENTORY_TOKEN, // omit for an anonymous observation
  pageSize: 50,
  maxPages: 1000,
  requestTimeoutMs: 30_000,
  maxResponseBytes: 8 * 1024 * 1024,
  // caCertificates: [privateForgeCaPem], // optional; replaces the default trust store for this forge
});

const inventory = await scanner.scan({
  maxItems: 100_000, // per collection
  timeoutMs: 60_000, // all collection and version requests together
  maxScanResponseBytes: 64 * 1024 * 1024,
  inspectVersion: true, // optional fresh version probe; default false
  includeIssues: true, // optional issue metadata scan; default false
  // namespaceDirectory: true, // requires a token; Gitea requires admin access
  // signal: abortController.signal,
});

const repositories = inventory.report.collections.repositories.result;
if (repositories.status === 'enumerated') {
  console.log(repositories.count, repositories.observation.access);
} else {
  // Inspect the explicit incomplete/uninspected result before making decisions.
  console.log(repositories.status);
}

Treat the result as private operator data. It can contain private repository paths and descriptions. Tokens stay in request headers, HTTPS is required, redirects are refused, and diagnostics omit raw response bodies and network errors. Credential-bearing clone URLs are rejected. SSH transport usernames are retained. Response-byte limits count decoded body bytes consumed by the reader; the chunk that crosses a limit is counted and discarded before JSON decoding. They are not HTTP wire-byte measurements or exact JavaScript heap limits.

Set caCertificates to reach a forge whose certificate chains to a private CA. Each entry is one PEM certificate; together they replace the default trust store for that adapter's (or scanner's) HTTPS connections, so include every anchor the forge's chain can end in. The adapter then sends its requests through node:https with its own keep-alive https.Agent instead of the global fetch. Nothing process-wide changes: NODE_TLS_REJECT_UNAUTHORIZED, the global fetch dispatcher and other connections keep their configuration. HTTPS-only base URLs, same-origin pagination links, redirect refusal, timeouts, response-byte limits and error codes apply unchanged. An empty list, more than 100 entries, a non-string entry, an entry larger than 64 KiB, a multi-certificate bundle in one entry or an unparseable entry fail construction with invalid-ca-certificates. Idle pooled connections do not keep the process alive.

The scanner marks observations publicOnly without a token and callerVisible with one. It never claims entireScope or a frozen snapshot. Gitea uses repository search and organization listing; GitLab uses keyset project pagination and offset group pagination with all_available=true. Short offset pages trigger another request because a server can cap page size below the requested limit. GitLab keyset completion relies on its documented next-Link protocol. Invalid continuations, denied requests, repeated IDs, cancellations and resource limits yield incomplete results. No automatic retry occurs. A scanner permits one active scan at a time, and each scan starts with fresh counts and metrics.

Every canonical entity class is present in the report. By default, the canonical namespaces result remains incomplete, including after organization/group endpoints are exhausted, because personal namespaces and full private namespace coverage are not established. Endpoint counts appear separately as gitea.organizations or gitlab.groups in additionalCollections.

Set namespaceDirectory: true to include personal namespaces. Gitea enumerates /admin/orgs followed by /admin/users; GitLab enumerates /namespaces without owner or top-level filters. A configured token is required. Access failures remain explicit and never fall back to public organization/group listings. Gitea's two directory phases share one maxPages limit, including empty-page probes. Successful completion produces an enumerated canonical namespace result within callerVisible access; additionalCollections is empty in this mode. It does not prove instance-wide administrator coverage, a frozen export, complete users, or permissions. Only namespace metadata is retained from the directory responses; user email/login-source and billing fields are discarded. Entity classes without an enabled reader remain notInspected with unverified capabilities.

Set inspectVersion: true to probe /api/v1/version on Gitea or /api/v4/version on GitLab before collection requests. versionInspection records observed with its timestamped observation, unverified with a safe reason, or notInspected when disabled. report.instance.version is set only after a successful probe in that scan. Denied or malformed responses never reuse an earlier version. A failed probe allows collection reads to proceed while the shared deadline and byte budget permit. The reported version is provider metadata, not proof that a real server version has been qualified.

Set includeIssues: true to read all visible open and closed issues in every observed repository. Gitea uses state=all&type=issues and rejects unexpected pull-request records. GitLab uses the stable project ID with state=all&scope=all, without author, confidentiality or issue-type filters. The returned issues array uses the shared IForgeInventoryIssue contract: stable row ID, separate repository-local number, repository identity, title/body, state, confidentiality, provider issue type, reported author and original timestamp precision/offset. Empty body text remains empty. Missing metadata is unknown; Gitea deleted/built-in authors remain unresolved. This is an observation of reported authors, not proof of original attribution for previously imported issues.

Issue pagination is bounded per repository by maxPages. The issue maxItems limit and duplicate detection span all observed repositories; deadline and response-byte limits span the entire scan. Gitea checks the current repository ID at its mutable path before each issue page, including empty pages, and also validates each issue's repository ID. These identity probes count in metrics and consume the same scan/caller byte budget. GitLab verifies each issue's returned project ID. A denied issue page, changed repository identity, incomplete repository enumeration or reached limit leaves issue inventory incomplete. No deletion or missing-issue inference follows from a partial scan. Comments, labels, milestones, assignees, attachments and history remain uninspected; outbound issue creation/update is not implemented by this scanner. Scan choices and the cancellation signal are captured at the beginning of each scan.

For one repository, call adapter.listIssues({ kind: 'repository', identity: repository.identity, location: repository.location }, { cursor?, signal?, maxResponseBytes? }) using a previously observed repository. Its cursor is additionally bound to that exact repository identity and location. The response carries repository scope; a foreign, unresolved or malformed repository identity is rejected before network access.

Provider IDs remain namespaced decimal strings, separate from mutable paths. Gitea organizations and repository owners share gitea.user.id; GitLab groups use gitlab.group.id, while a personal project namespace uses gitlab.namespace.id rather than a user ID. Nested paths preserve literal hyphens. Numeric JSON IDs or sizes outside JavaScript's safe integer range fail closed; canonical decimal strings retain their precision. Missing metadata remains unknown. A Gitea repository's visibility follows its owner's visibility, because Gitea's own internal flag is set only for a public repository of a private owner: a private repository, or a public one of a private owner, is private; a public repository of a limited owner is internal (signed-in users); a public repository of a public owner is public. A payload whose internal flag contradicts the owner visibility fails with inconsistent-repository-visibility. Gitea's rounded size field is not used as a separate Git byte measurement. Exposed GitLab repository statistics are estimates, and their sum is available only when every observed repository has a measurement. Neither quantity estimates bytes for a Git push.

The adapter's read-only baseUrl property exposes the validated, normalized HTTPS instance root, including its deployment subpath and without a trailing slash. Connection management can retain this value without copying URL validation or exposing the private token. It identifies a location, not an instance identity or verified server.

For incremental consumption, ForgeInventoryAdapter.listRepositories({ cursor?, signal?, maxResponseBytes? }), .listNamespaces(...) and .listNamespaceDirectory(...) return shared inventory pages. A more result contains a signed cursor bound to that adapter object, collection, directory phase and fixed query/page policy. Reuse it only with the same adapter; it is disposable process-local state, not a durable checkpoint. exhausted describes the selected endpoint or completed directory within the recorded visibility. Page methods throw ForgeInventoryError for request and payload failures, while malformed continuation metadata returns an incomplete page retaining its validated items. The scanner converts those failures into an incomplete collection report. .inspectVersion({ signal?, maxResponseBytes? }) returns a fresh observed/unverified result separately; it does not mutate the adapter's instance or page metadata, whose version remains unknown.

The implementation has synthetic protocol fixtures based on the Gitea 1.24.7 API schema, Gitea pagination documentation, and the GitLab projects, groups, namespaces, metadata, issues and pagination contracts. For an administrator's complete Gitea measurement, see the source inventory below. Native inventory, collision analysis and migration manifests remain pending.

Authenticated Gitea source inventory

GiteaSourceInventory measures a whole Gitea instance before its one-time migration, read with a site administrator's token. It returns the IForgeSourceInventory contract from @foss.global/interfaces: per-instance, per-owner and per-repository counts with totals, where every measure is complete, partial (attempted, did not finish) or notMeasured (not attempted). The inventory is complete only when discovery finished and every measure is complete; otherwise inventoryComplete is false and incompleteReasons names each gap. It never writes to the forge: it sends REST GET requests, clones each repository with Git, and asks the LFS batch API, as a download, which objects Gitea holds.

import { GiteaSourceInventory, composeGiteaSourceInventory } from '@foss.global/codefeed';

const reader = new GiteaSourceInventory({
  instanceId: 'code-foss-global',
  baseUrl: 'https://gitea.example.com', // HTTPS only
  token: siteAdminToken, // read it from a file; never pass it on a command line
  gitWorkDir: '/var/lib/inventory/mirrors', // absolute, private; holds one temporary mirror per repository in flight
  // caCertificates: [privateCaPem], gitTimeoutMs: 30 * 60 * 1000, pageSize: 50, maxPages: 10_000, requestTimeoutMs: 60_000, maxResponseBytes: 16 * 1024 * 1024,
});

const discovery = await reader.discover(); // rejects with `reader-not-site-admin` for other tokens
const results = await reader.measureUnits(reader.units(discovery), {
  concurrency: 4, // units measured at once; requests within a unit are sequential
  onResult: async (resultArg) => { /* persist each unit result to resume a run */ },
});
const inventory = composeGiteaSourceInventory(discovery, results, [{ package: '@foss.global/codefeed', version }]);

A run is split into units: the instance, each owner and each repository. measureUnit never rejects; a unit whose primary record cannot be read becomes failed with a sanitized reason. A resumed run measures failed units, and units holding a partial measure, again and passes all results to composeGiteaSourceInventory, where the last result per unit wins. The composed document is deterministic: it carries no timestamps, sorts owners, repositories, packages and reasons, and does not depend on the order of the results. An aborted signal or a rejected onResult aborts the units in flight, hands over no further result, and rejects measureUnits once every worker has stopped.

Discovery records each repository's Gitea id as a canonical decimal string, and units() passes it into the repository unit. A unit with an id fails with repository-identity-changed when its name answers with another id, as after a rename and a new repository under the old name; a unit without one, from a discovery stored by an earlier version, is checked by name only. composeGiteaSourceInventory counts a repository result for another id than discovery recorded as a failed unit (repository: unit failed: repository-identity-changed), and adds repository: duplicate-id when two measured repositories share one id, which a discovery without ids cannot rule out.

lookupRepository(id) reads one repository by its id through GET /api/v1/repositories/{id}, which keeps answering after a rename or transfer. It resolves { status: 'present', id, owner, name } with the name Gitea gives the repository now, or { status: 'absent', id } only when Gitea answers 404, which a site administrator gets only for a deleted repository. Every other failed read rejects with its code, so a timeout or a 500 never counts as a deletion; an id that is not a canonical decimal rejects with invalid-repository-id without a request.

What is measured: users with their account flags, two-factor and SSH/GPG keys; organizations, members, teams, team memberships and repository grants; repositories with their flags, Git refs split into branches, tags, pull request refs and other refs, Git objects and their uncompressed bytes, LFS objects referenced anywhere in history with the bytes Gitea holds and the number it lacks, issues, pull requests, comments, reviews, review comments, labels, milestones, reactions and attachments; releases including drafts, assets and asset bytes; wiki pages; collaborators, team grants, SSH deploy keys, deploy tokens, webhooks, branch and tag protections and push mirrors; packages per type with versions, files and bytes; system webhooks, Actions runners, workflows, runs, artifacts and the number of Actions secrets and variables; unadopted repositories.

Beyond counts, each repository keeps records of what the migration recreates. description is the repository's description exactly as Gitea's API reports it, without trimming or normalising; '' means the source has none. A description that is not a string fails the repository unit with invalid-string, so the inventory stays incomplete; reports from readers before 4.4.0 carry no description, which a consumer treats as unread, never as ''. releaseRecords lists every release, drafts and prereleases included, sorted by numeric id, as TForgeSourceRelease records (narrow on draft before reading publishedAt as a string): tag name, target commitish, name, body, the draft and prerelease flags, created and published times as Gitea reports them (a published release must carry a valid publication time; a draft's is recorded as reported, null on Gitea 28 and never replaced by its creation time, and a draft's tag does not exist as a Git ref yet; on Gitea 28 a new release's creation time is the date of the commit it points at), an author snapshot (account id and login; Gitea's placeholder for a deleted account has id -1) and its assets sorted by numeric id with id, storage uuid, name, size, download count and creation time. Gitea exposes no content digest, so a record carries none; a replaced file carries another id and uuid, and the download count moves with every download, so it never decides whether a release changed. protections holds the branch protection rules sorted by rule name (code-unit order) with every restriction Gitea reads back, allowlists included, and the tag protection rules sorted by numeric id with their name pattern and allowlist. releaseRecords is null while a release count is not complete, and protections is null while either protection count is not; the counts then say why.

Identity and access are recorded too, as Gitea reads them and without a verdict, all with the administrator's token:

Each user owner's userRecord holds its id, login, the active, administrator, restricted and sign-in-prohibited flags, its profile visibility and every SSH key from /users/{user}/keys (id, title, fingerprint, the OpenSSH key type, which is the first field of the public key, and the public key), sorted by numeric id. It is null while sshKeys is not complete.
Each organization's teamRecords holds every team from /orgs/{org}/teams with its permission, units map (null for a whole-repository team), includes-all and create-repository flags, visibility, the members from /teams/{id}/members by id and login, and the ids of the repositories /teams/{id}/repos lists (an includes-all team lists every repository it holds). Gitea 28 reports the Owners team as owner with a units map. It is null while teams, teamMemberships or teamRepositoryGrants is not complete.
Each repository's accessRecords holds its collaborators, each with one /collaborators/{user}/permission read, and the entries of /repos/{owner}/{repo}/keys split by key_type: SSH deploy keys (binding id, shared key row id, title, fingerprint, public key, read-only flag) and deploy tokens (id, title and read-only flag only; neither the token nor the mask Gitea reports as its fingerprint is read). It is null while access.collaborators, access.deployKeys or access.deployTokens is not complete.

A collaborator's effectivePermission is what Gitea reports: the strongest of the collaboration, teams, ownership, site administration and public access, never the collaboration's own mode, which no REST endpoint reports. The first permission read that fails, answers for another user or reports an unknown mode makes access.collaborators partial with the reason collaborator-permission: <code>, and the records null; a record is never a partial list. From 5.1.0 access.deployKeys counts SSH deploy keys only and access.deployTokens counts the deploy tokens Gitea 28 lists beside them; 5.0.0 and earlier counted both as deploy keys. These identity reads target Gitea 28 only (the release code.foss.global runs): a deploy key entry without a known key_type (every deploy key on Gitea before 28), or a team without Gitea 28's visibility, fails its unit. Composing a stored repository result from a reader before 5.1.0 counts its deployTokens as not measured, so the inventory stays incomplete until that repository is measured again.

Secrets are counted, never named: the report carries no tokens, Actions secret or variable values or names, webhook secrets or mirror remote URLs. Reasons are adapter codes or fixed text, never response bodies or URLs. Gitea has no read that lists user-scoped Actions secrets; that is listed in limitations, since they are reissued. User-scoped webhooks, variables and runners are read as that user through the Sudo header.

Git-level measures come from a mirror clone of every ref Gitea advertises, refs/pull/* included, made with @foss.global/gitmanager in a fresh directory below gitWorkDir and removed afterwards. The reader's login and token form Basic credentials; the credential reaches Git only through GIT_CONFIG_* entries in the child process environment; it is never in the URL, the command line or any file. The private CA list applies to the clone as well. The mirror's HEAD becomes the repository's head: the ref it names as target and whether it resolves. In a repository that is not empty, a HEAD that does not resolve names a missing branch, such as a default branch deleted at the source, and a target other than refs/heads/<defaultBranch> means Gitea's setting and its Git HEAD disagree. head is null when the mirror or its refs could not be read. A detached source HEAD is one such case: GitManager's mirror import requires a symbolic HEAD, so the clone fails with git-mirror-failed, and this reader never reports a target of null. The same mirror records refDigests: for each of refs/heads/, refs/tags/ and refs/notes/, GitManager's ref inventoryDigest (HEAD and the refs under that prefix) and the refCount of that selection, read as one single-ref listRefs page per prefix; codefeed computes no digest of its own. Any other GitManager repository with the same HEAD and refs under a prefix reports the same digest, so a migration compares them with its native copy. Pull request refs have no digest, and git.otherRefs - refDigests.notes.refCount refs lie outside every digested namespace. refDigests is null whenever head is, or when any of the three reads fails; it is never partial. The mirror's distinct LFS pointers, from every reachable revision, are checked against Gitea in batches of 100: an object with a download action counts as held, one answered with error 404 as missing, and any other answer makes the LFS measures partial. A clone that fails or exceeds gitTimeoutMs makes every Git and LFS measure of that repository partial; only the API-reported size stays complete. A clone starts only once its scratch reservation, twice the reported size plus 64 MiB, fits the free space of the file system that holds gitWorkDir, counting the reservations of the clones already in flight from every reader in the process. A clone that does not fit waits for them, in arrival order; one that does not fit even alone makes every Git and LFS measure of its repository partial with git-workdir-too-small, so the inventory stays incomplete. Size gitWorkDir for the largest repositories measured at once.

Known gaps that keep an inventory incomplete:

A repository with its issues or pull requests unit disabled hides both through the API, so its issues, pull requests, comments, reviews, reactions and attachments are notMeasured. A disabled wiki hides its pages the same way. Labels, milestones, releases and Actions stay readable while their units are disabled.
Gitea 28 answers 404 to every pull request route of an archived repository, reads included. The reviews of each pull request are still requested; when an archived repository answers 404 there, reviews and reviewComments are partial with the reason repository-archived (in incompleteReasons as repository.collaboration.reviews: repository-archived). Its issues, pull requests, comments and refs/pull/* stay measured. A 404 in a repository that is not archived stays http-404.

Packages

Packages are measured only when discovery selects their class: reader.discover(signal, { classes: ['packages'] }) lists every package from the administrator package listing (/admin/packages, user-owned packages included) as discovery.packages, and units() then adds one unit per package. Without the class, discovery carries no packages, no package unit is measured, and the composed inventory has neither packages nor totals.packageContent, so the package class is unread; a stored discovery from an earlier reader composes as before. An unknown class rejects with invalid-options.

A package unit returns an IForgeSourcePackageInventory (@foss.global/interfaces 3.2): the owner's id, the linked repository's id or null, every version Gitea's v1 package API lists (id, version, created_at verbatim) and every distinct file content once, by SHA-256 with its size, with the files holding it (version id, file id, name). A unit is read whole or fails, never as a partial record:

Container: a version whose file list is exactly manifest.json is an index candidate, because Gitea stores an index's manifest and nothing else. Each candidate is read by digest from the registry (/v2/<owner>/<image>/manifests/<digest>) with the reader's Basic credentials and the index media types; the bytes must hash to the listed SHA-256 and the response must be an OCI index or a Docker manifest list, else the unit fails with unexpected-manifest-shape. Its references are the digests its manifests name, sorted, each once. No other manifest is read; a tag is a version whose version does not start with sha256:.
npm: distTags is every dist-tag Gitea serves (/-/package/{name}/dist-tags), a latest Gitea derives when none is stored included, since no read tells the two apart; each version's deprecated comes from the package document ('' when not deprecated). A version missing from the document fails the unit with npm-version-missing.
Cargo: each version's yanked comes from the crate's sparse index; a version missing from it fails the unit with cargo-index-incomplete.
A listing that does not finish fails the unit with the listing's reason, rows of another package with package-identity-changed, and a record over forgeSourceInventoryUnitMaxBytes (4 MiB of UTF-8 JSON) with unit-over-bound; it is never truncated.

Every read is a GET. Reading an index counts a download of its version in Gitea (package_version.download_count), an accepted read stamp (S10 ruling, Addendum 1); every other package read counts nothing.

totals.packageContent comes from the exported pure function composeGiteaPackageContent(discoveredPackages, packageRecords), which a consumer can recompose and compare: distinct SHA-256 digests and their bytes for every owner together and per owner, each by type, over all versions and over the tagged versions plus the versions holding a manifest a tagged index references (every version of other types). It sizes stores that keep each digest once, Gitea globally and the native container registry per owner. A measure is complete only when every discovered package in its scope has a record; a failed unit leaves its package out, adds package: unit failed: <reason> to incompleteReasons and makes the measures of its owner, its type and the whole instance partial. Discovery with an empty packages list composes packages: [] and complete zeros.

Listings page until an empty page or the x-total-count total is reached. A total that changes mid-listing or disagrees with the rows read, a repeated page, a malformed page, a failed request or the page limit makes that measure partial and keeps the rows already read; only a listing that does not exist, such as the wiki of a repository without one, counts as empty. A repository search that lists one repository id twice has shifted under its pages and may have skipped a repository: discovery keeps the first entry and adds discovery: repositories: duplicate-id to incompleteReasons.

Gitea source transfer

GiteaSourceTransfer reads one repository's Git data, LFS objects and release assets, and package content, for the one-time migration, with the same reader credentials as the inventory. It never writes to Gitea: Git is read only through git-upload-pack, LFS only through batch download requests and their download actions, release assets only through GETs of their attachment, packages only through GETs of the registries and the v1 package API.

import { GitManager } from '@foss.global/gitmanager';
import { GiteaSourceTransfer } from '@foss.global/codefeed';

const source = new GiteaSourceTransfer({
  baseUrl: 'https://gitea.example.com', // HTTPS only
  login: readerLogin,
  token: readerToken, // read it from a file; never pass it on a command line
  // caCertificates: [privateCaPem], gitTimeoutMs: 30 * 60 * 1000, requestTimeoutMs: 60_000, maxResponseBytes: 16 * 1024 * 1024,
  // lfsTransferTimeoutMs: 30 * 60 * 1000, releaseAssetTimeoutMs: 30 * 60 * 1000,
});

// A started GitManager rooted in the caller's own scratch space; codefeed creates no scratch roots.
const scratch = new GitManager({ dataDir: scratchDir });
await scratch.start();
const { repository, reportedSizeKiB } = await source.importMirror({ owner: 'team', name: 'app', id: '11' }, scratch, {
  target: { org: 'team', repo: 'app' },
  signal,
});

const download = await source.downloadLfsObject({ owner: 'team', name: 'app', id: '11' }, oid, size, { signal });
if (download.status === 'missing') {
  // Gitea answered error 404: the object is not held at the source.
} else {
  await pipeline(download.stream, destination); // errors unless the bytes are exactly the object
}

// A release asset as the inventory recorded it: { id, uuid, name, size } from releaseRecords[].assets[].
const asset = await source.downloadReleaseAsset({ owner: 'team', name: 'app', id: '11' }, release.id, recordedAsset, { signal });
await pipeline(asset.stream, destination); // errors unless exactly `size` bytes of the recorded asset arrived

A source repository is named by owner, name and its Gitea repository id, a canonical decimal string as the inventory records it; anything else rejects with invalid-repository. Git and the LFS API reach a repository only by name, and a name can move to another repository between two reads: Gitea answers a renamed repository's old path with a redirect, and a new repository can take the old name. Every source read is therefore bracketed by an identity read of GET /api/v1/repositories/{id}, which must still name owner/name (compared case-insensitively, as Gitea does). A repository that was renamed, or whose name now belongs to another repository, rejects with repository-identity-changed; a deleted one rejects with http-404. A transfer that fails is explained the same way when the identity read shows one of those two causes.

importMirror reads the repository's size from the REST API by id, reserves its scratch in the process-wide GitScratchBudget of the file system holding the GitManager's data directory (twice the reported size plus 64 MiB, admitted in arrival order), and then clones every ref into target with importMirror of @foss.global/gitmanager and reads the identity again. When that last read rejects, the clone has finished and stays at target; the caller discards it. The clone URL is formed from baseUrl and the repository's owner and name; the reader's login and token reach Git as a Basic credential only through GIT_CONFIG_* entries in the child process environment, never in the URL, the command line or any file. A repository that does not fit even alone rejects with git-workdir-too-small without cloning. The reservation is released when the clone ends. Failures reject with a fixed ForgeInventoryError code (invalid-repository, invalid-target, gitmanager-not-running, repository-identity-changed, a REST code such as http-404, git-workdir-failed, git-workdir-too-small, git-mirror-failed, request-aborted); GitManager error text is never passed on. An abort rejects with request-aborted. While the clone runs, the abort reaches the GitManager import through its own signal (@foss.global/gitmanager 2.4.0 or later): Git is ended, the call settles once it has exited, no repository is left at target, and the reservation is released then. The GitManager keeps serving its other work, so one engine can serve several imports and other commands at once; a stop() of it during the clone also rejects the import with request-aborted.

While the import runs, GitManager holds target: writes to it, such as initBareRepository, restoreRepositoryBundle, setDefaultBranch or a Smart HTTP push, fail with a GitManagerError whose code is repositoryBusy, and a second import into the same target is refused at once (codefeed rejects it with git-mirror-failed). Reads, of target as of other repositories, proceed.

downloadLfsObject sends one batch download request with the reader's Basic credentials, then a GET of the action's href with the action's headers only, and then reads the identity; when that read rejects, the stream is destroyed and the call rejects with its code. A missing object resolves as { status: 'missing' }, so a caller can list the missing oids; any other object error rejects with lfs-object-error-<code>, such as lfs-object-error-422 for a size Gitea disagrees with. The returned stream never holds the whole object: it counts and hashes the bytes as they pass and holds back the last chunk until the body ended with exactly size bytes whose SHA-256 is oid. Otherwise it errors with lfs-size-overrun, lfs-short-body or lfs-digest-mismatch; a content-length other than size rejects with lfs-size-mismatch before streaming. An abort errors it with request-aborted, a transfer longer than lfsTransferTimeoutMs or idle for requestTimeoutMs with lfs-transfer-timed-out. Destroying the stream ends the download. The stream fails internally with lfs-transfer-failed when body bytes still arrive after the download has ended, which happens only once the stream is destroyed, so a reader of the stream does not see it.

downloadReleaseAsset(repository, releaseId, asset) takes the release id and the asset's id, uuid, name and size as the inventory recorded them; anything else rejects with release-asset-invalid before any request. It reads the repository by id, then GET /api/v1/repos/{owner}/{name}/releases/{releaseId}, which must still carry an asset with the same id, uuid, name and size; otherwise it rejects with release-asset-identity-changed (a deleted release with http-404). The download count is not compared. It then GETs {baseUrl}/attachments/{uuid} with the reader's token authorization: no redirect is followed and no range is asked for, so any answer but 200 rejects, with http-<status> or, for a refused redirect, release-asset-download-failed, and a content-length other than size rejects with release-asset-size-mismatch. No digest is computed while the bytes stream: the stream counts them and holds back the last chunk until exactly size bytes have arrived and both identity reads have passed once more, so a consumer never receives a complete asset that is short, long or no longer the recorded one. Otherwise it errors with release-asset-size-overrun, release-asset-short-body, repository-identity-changed or release-asset-identity-changed. Content digests come from the content itself, such as a publisher's SHA256SUMS asset or the importer's own hash. An abort errors it with request-aborted, a download longer than releaseAssetTimeoutMs (default 30 minutes, at most six hours) or idle for requestTimeoutMs with release-asset-transfer-timed-out. A failed request is explained by the identity reads as for downloadLfsObject. Destroying the stream ends the download; the download's connections close with the stream. The stream fails internally with release-asset-transfer-failed when body bytes still arrive after the download has ended, which happens only once the stream is destroyed, so a reader of the stream does not see it.

Package content

The package reads take selectors from the inventory's package records: the package (owner, ownerId, type, name), a version (id, version) and content (sha256, size, with the file name for a version's file). A malformed selector rejects with package-invalid, package-version-invalid or package-content-invalid before any request.

readContainerManifest(package, digest) reads a manifest by digest from the registry with the reader's Basic credentials and returns its exact bytes and its Content-Type verbatim, the media type it was pushed with. The bytes must hash to digest, and a Docker-Content-Digest must name it (package-manifest-digest-mismatch); a type that is no OCI or Docker image manifest, index or manifest list rejects with package-manifest-media-type-invalid.
downloadContainerBlob(package, content) streams a config or layer by digest. Manifests and blobs are content-addressed, so no identity read accompanies them.
downloadPackageFile(package, version, file) streams a generic file, an npm tarball ({name}/-/{version}/{file}), a Swift source archive (with Accept: application/vnd.swift.registry.v1+zip) or a crate, with the reader's token authorization; container content uses the two reads above (package-type-unsupported).
readPackageDocument(package, version, kind, options?) returns a version's metadata with the SHA-256 of the returned bytes, which a consumer records beside the version id:
- npm-version: the version document.
- swift-release: the release metadata.
- swift-manifest: Package.swift, which must answer 200. With { swiftVersion } (5.8, 5.9.1: trimmed as Gitea keys alternate manifests, two or three segments without leading zeros, the patch only when it is not zero, each segment at most 9223372036854775807; anything else rejects with swift-version-invalid before any request) it reads Package@swift-<v>.swift through ?swift-version= without following a redirect, and resolves null only when Gitea answers 303 to the release's own Package.swift URL, its answer for a version it holds no manifest for. Any other redirect rejects with swift-manifest-redirect-unexpected, any other status with http-<status>.
- cargo-index-entry: the version's line of the sparse index; a missing line rejects with package-document-missing.
- cargo-description: {"description":…} from the crate's entry in the Cargo search (api/v1/crates?q=<name>, the entry with exactly that name, read page by page, at most 200 pages of 50). Gitea's search describes only the version with the highest id, so its max_version must be the requested version; otherwise, or when no entry has the name, it rejects with package-document-missing, and a search that has not run out after 200 pages with cargo-search-over-bound.
A manifest comes back verbatim. Every JSON document comes back in its canonical form (canonicalPackageDocument, below), never the bytes Gitea served: Gitea 28 writes a JSON object held in a map, such as a crate's features, in a new key order on every read, so the served bytes of one unchanged version differ from read to read. A kind of another type, or swiftVersion with another kind, rejects with package-document-kind-invalid before any request, another media type with invalid-content-type. None of these reads counts a download.

canonicalPackageDocument(kind, bytes) returns { bytes, sha256 } for the JSON kinds (npm-version, swift-release, cargo-index-entry, cargo-description), in this order: decode as UTF-8 (fatal) and parse a JSON object; remove the top-level member holding state the forge changes without a new version, which the inventory records apart (yanked of a Cargo index entry, deprecated of an npm version document); write it compact, with object keys sorted by Unicode code point at every depth, arrays in their order, and strings and integers as JSON.stringify writes them. A number that is not a safe integer, and anything that is not a JSON object, rejects with document-not-canonical. The same document gives the same bytes whatever order its keys arrived in, so a consumer computes the canonical form of its own document with the same function and compares the digests.

A version's file and documents are read between two identity reads of GET /api/v1/packages/{owner}/{type}/{name}/{version}, which must still carry the recorded version id, owner id, type and name: a version deleted and published again rejects with package-version-identity-changed, a deleted one with http-404, also when the read itself failed. Streams verify while they pass: the last chunk is held back until exactly size bytes have arrived, their SHA-256 is sha256 and, for a version's file, the closing identity read has passed; otherwise they error with package-content-size-overrun, package-content-short-body, package-content-digest-mismatch or the identity code, and a content-length other than size rejects with package-content-size-mismatch. An abort errors with request-aborted, a download longer than packageTransferTimeoutMs (default 30 minutes, at most six hours) or idle for requestTimeoutMs with package-content-transfer-timed-out. Reading a manifest or a version's file counts a download of its version in Gitea, an accepted read stamp (S10 ruling, item 8); blobs and documents count nothing.

Git LFS batch client

LfsBatchClient is a provider-neutral Git LFS batch API client with the basic transfer adapter, bound to one LFS endpoint and its credential. GiteaSourceTransfer downloads through it; the migration uploads to the native Git service through it.

import { LfsBatchClient } from '@foss.global/codefeed';

const lfs = new LfsBatchClient({
  endpoint: 'https://git.example.com/team/app.git/info/lfs', // https, or http only to 127.0.0.0/8 or [::1]
  credential: { kind: 'authorization', value: `Bearer ${grantToken}` }, // or { kind: 'basic', username, password }
  // caCertificates, requestTimeoutMs: 60_000, transferTimeoutMs: 30 * 60 * 1000, maxResponseBytes: 4 * 1024 * 1024, batchSize: 100,
});
const uploaded = await lfs.upload([{ oid, size, open: (signalArg) => openObjectStream(oid, signalArg) }], { signal });
// [{ oid, size, outcome: 'uploaded' | 'present' }]
const presence = await lfs.checkPresence([{ oid, size }], { signal }); // [{ oid, size, status: 'held' | 'missing' }]
const download = await lfs.download({ oid, size }, { signal }); // as for downloadLfsObject
lfs.close(); // optional: idle pooled connections close on their own

upload sends one batch upload request per batchSize objects (at most 100). An object without an upload action is already present; for each other one it opens the source, PUTs exactly size bytes with the action's headers and Content-Length: size, and then calls the verify action when the server sent one. The bytes are hashed while they stream and the last chunk is held back, so a source that is not the object is never sent complete (lfs-size-overrun, lfs-short-body, lfs-digest-mismatch). Any object error rejects with lfs-object-error-<code>. A source that still yields bytes after its upload has ended fails internally with lfs-transfer-failed; that happens only once the upload has settled, so the caller does not see it.
Action expiry: expires_in (whole seconds after the batch answer arrived) is preferred over expires_at (RFC 3339), as the batch API specifies; like the git-lfs client, an expires_in of 0 and Go's zero time mean no expiry, and any other malformed value fails with invalid-lfs-batch. An upload, verify or download action is used only while it stays valid for the expiry margin, 30 s or requestTimeoutMs when that is shorter. Objects are transferred one at a time after their batch answer, so a later object whose action would expire within the margin is asked for again in a batch of its own, once; when the fresh answer is again too short-lived the call fails with lfs-action-expired. A verify action that expires during its PUT is asked for the same way; a fresh answer without actions means the server holds the object.
checkPresence sends batch download requests and downloads nothing: a download action means held, error 404 missing. It uses no action, so expiry is validated but not enforced there.
The credential is sent only as the Authorization header of batch requests to the endpoint, never to action hrefs and never in a URL, error or result; hrefs get the action's headers only. Action headers may not set connection-level fields (host, content-length, transfer-encoding and similar).
URL policy, enforced for the endpoint and every action href: https:, or http: only to an IP literal in 127.0.0.0/8 or [::1]; localhost and every other host name over HTTP are refused (unsafe-lfs-url), as is user information. An href must keep the endpoint's scheme, so an HTTPS server cannot move a transfer to plain HTTP (unsafe-lfs-href). Redirects are never followed.
Every request is bounded: batch answers by maxResponseBytes, batch and verify requests by requestTimeoutMs, object transfers by transferTimeoutMs and an idle limit of requestTimeoutMs, everything by the caller's AbortSignal. Errors are fixed ForgeInventoryError codes.

Shared scratch budget

GitScratchBudget and mirrorScratchBytes are exported so that every scratch GitManager in a process admits its work through the same budget as the inventory and the source transfer. GitScratchBudget.forDirectory(directory) returns the process-wide budget of the file system holding directory; reserve(directory, bytes, signal) resolves with a release function once the bytes fit its free space next to the reservations in flight, in arrival order, or with null when they do not fit even alone. mirrorScratchBytes(reportedSizeKiB) is the reservation of one mirror clone: twice the reported size plus 64 MiB.

Quick Start

import { CodeFeed } from '@foss.global/codefeed';

// Fetch commits since one week ago (default), no caching
const feed = new CodeFeed('https://code.example.com', 'gitea_token');
const commits = await feed.fetchAllCommitsFromInstance();
console.log(commits);

With options

const thirtyDays = 30 * 24 * 60 * 60 * 1000;
const since = new Date(Date.now() - thirtyDays).toISOString();

const feed = new CodeFeed('https://code.example.com', 'gitea_token', since, {
  enableCache: true,          // keep results in memory
  cacheWindowMs: thirtyDays,  // trim cache to this window
  enableNpmCheck: true,       // check npm for published versions
  taggedOnly: false,          // return all commits (or only tagged)
  orgAllowlist: ['myorg'],    // only scan these orgs
  orgDenylist: ['archive'],   // skip these orgs
  repoAllowlist: ['myorg/app1', 'myorg/app2'], // only these repos
  repoDenylist: ['myorg/old-repo'],            // skip these repos
  untilTimestamp: new Date().toISOString(),    // optional upper bound
  verbose: true,               // print a short metrics summary
});

const commits = await feed.fetchAllCommitsFromInstance();

Advanced callers can provide a custom ICodeFeedProvider through the constructor options. The default provider keeps the existing Gitea HTTP behavior, while the provider contract lets adapters supply organizations, repositories, commits, tags, and changelog content from another backend.

With GitManager

import { CodeFeed, CodeFeedGitManagerProvider } from '@foss.global/codefeed';
import { GitManager } from '@foss.global/gitmanager';

const since = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();
const gitManager = new GitManager({
  dataDir: '/var/lib/foss-global/gitmanager',
});

await gitManager.start();

const feed = new CodeFeed('gitmanager://local', undefined, since, {
  provider: new CodeFeedGitManagerProvider({ gitManager }),
  enableNpmCheck: false,
});

const commits = await feed.fetchAllCommitsFromInstance();

await gitManager.stop();

Public GitManager read models

GitManagerPublicReadClient is an in-process adapter from a running GitManager to the shared public response models. It supports exactly five read methods:

getOverview(query?)
getOrg(org)
getRepo(org, repo, path?, issueNumber?)
getFile(org, repo, path)
getCommits(org, repo, sha?)

Construction is default-deny. Provide an authoritative isRepositoryPublic predicate, or set allowAllRepositories: true only when every repository managed by the supplied GitManager is public:

import { GitManagerPublicReadClient } from '@foss.global/codefeed';

const publicRepositories = new Set([
  'foss.global/interfaces',
  'foss.global/gitmanager',
]);

const publicReads = new GitManagerPublicReadClient({
  gitManager,
  publicBaseUrl: 'https://foss.global',
  smartHttpBaseUrl: 'https://foss.global',
  isRepositoryPublic: async (repository) =>
    publicRepositories.has(`${repository.org}/${repository.repo}`),
});

const overview = await publicReads.getOverview();
const repository = await publicReads.getRepo('foss.global', 'gitmanager');

Construction fails when neither visibility option is supplied. Predicate failures reject the request, so visibility checks fail closed. When allowAllRepositories is true, the predicate is bypassed.

Public-model visibility and GitManager Smart HTTP visibility are separate policy surfaces. Synchronize both from the same authoritative source. A generated cloneUrl or sshUrl describes a URL; it does not prove that the corresponding transport is available or that the caller is authorized.

URL defaults:

publicBaseUrl: https://foss.global
smartHttpBaseUrl: the configured publicBaseUrl
sshBaseUrl: ssh://git@code.foss.global:29419

Bounded safe-integer options:

maxCommits: 40; accepted range 1–1,000.
maxReleases: 50; accepted range 1–1,000.
maxLanguageFiles: 1,000; accepted range 1–10,000.
maxConcurrency: 8; accepted range 1–32.

Optional organizationMetadata, repositoryMetadata, and newsProvider callbacks enrich the mapped response data. They do not add forge capabilities. Models intentionally leave members, issues, issue comments, release assets, download URLs, forge metrics, identity fields, and verification fields empty when GitManager has no authoritative source for them. The optional issueNumber argument is accepted for public-request compatibility and currently ignored.

Individual GitManager read failures may map to empty or null fallback data so one unavailable tree, file, tag, commit list, or diff does not necessarily fail the whole public model. Visibility predicate failures are not converted to fallbacks.

This client does not start an HTTP server, configure Git Smart HTTP or SSH, terminate TLS, authenticate users, or implement write/admin APIs. A server layer can call these methods from its own authenticated typed handlers.

CodeFeed.fetchAllCommitsFromInstance() returns items with this shape:

interface ICommitResult {
  baseUrl: string;
  org: string;
  repo: string;
  timestamp: string;        // ISO date
  hash: string;             // commit SHA
  commitMessage: string;
  tagged: boolean;          // commit is pointed to by a tag
  publishedOnNpm: boolean;  // only when npm check enabled and tag matches
  prettyAgoTime: string;    // human-readable diff
  changelog: string | undefined; // snippet for matching tag version
}

The canonical public result contract is exported as ICodefeedCommitResult from @foss.global/interfaces. @foss.global/codefeed/interfaces keeps ICommitResult as a package-local alias for that contract.

Features

Pagination for orgs, repos, commits, and tags (no missing pages)
Retries with exponential backoff for 429/5xx and network errors
CHANGELOG discovery with case variants (CHANGELOG.md, changelog.md, docs/CHANGELOG.md)
Tag-to-version mapping based on tag names (vX.Y.Z → X.Y.Z)
Optional npm publish detection via @org/repo package versions
GitManager provider adapter for repositories managed by @foss.global/gitmanager
Default-deny GitManager public read-model adapter for overview, organization, repository, file, and commit responses
In-memory caching with window trimming and stable sorting
Allow/deny filters for orgs and repos, optional time upper bound
One-line metrics summary when verbose: true

Environment

Gitea base URL and an optional token with read access
For GitManager-backed feeds, a started @foss.global/gitmanager 2.x GitManager instance with imported or local repositories
For GiteaSourceTransfer.importMirror, a started @foss.global/gitmanager 2.4.0 or later GitManager, whose transfers take an abort signal
For public GitManager reads, an authoritative repository-visibility source or an explicit all-public declaration
Node.js 22+ (global fetch; dependencies such as file-type 22 through @push.rocks/smartnpm declare Node.js 22)

Testing

The repo contains:

An opt-in integration test using a GITEA_TOKEN from .nogit/ via @push.rocks/qenv.
A mocked pagination test that does not require network.
A GitManager provider mapping test using a local test stub.
A default-deny GitManager public-client mapping test using a local test stub.
A mocked GiteaSourceInventory test and a mocked package inventory test, Git-level GiteaSourceInventory and GiteaSourceTransfer tests, a release asset download test and a package transfer test against local HTTPS servers (needs openssl and git http-backend), and an opt-in one against a Docker Gitea from @foss.global/forgefixtures, run with FORGEFIXTURES=gitea pnpm test (Docker required). It checks every package record field for field against the fixture's manifest, the distinct content against an independent computation, every transfer read, and, with the fixture's state snapshots, that the package inventory and transfer reads change nothing in Gitea but package_version.download_count (once per lead-file read) and the administrator token's access_token.updated_unix.
An LfsBatchClient test against local loopback LFS servers over HTTP and over HTTPS with a private CA.

Run tests:

pnpm test

For the integration test, set CODEFEED_RUN_LIVE=true and ensure GITEA_TOKEN is provided, e.g. via .nogit/ as used by test/test.live.node.ts.

Notes

When taggedOnly is enabled, the feed includes only commits associated with tags.
publishedOnNpm is computed by matching the tag-derived version against the npm registry for @org/repo.
For very large instances, consider using allowlists/denylists and enabling caching for incremental runs.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md Unescape Escape

@foss.global/codefeed