opendata/readme.md

# @fin.cx/opendata

`@fin.cx/opendata` is a TypeScript toolbox for serious data plumbing: market data, SEC fundamentals, German company registry workflows, and locally synced law corpora. It is built for programmers who want scriptable APIs, local persistence, and real workflows instead of a thin wrapper around one remote endpoint.

It currently gives you four practical lanes in one package:

- `StockPriceService` for stock and crypto prices with caching, retries, failover, historical ranges, and intraday support
- `FundamentalsService` and `StockDataService` for SEC fundamentals and combined price + fundamentals payloads
- `OpenData` for German company ingestion plus browser-driven Handelsregister lookup and document download
- `LawService` for syncing and searching German, EU, and US law texts in a local embedded database

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Installation

```bash
pnpm add @fin.cx/opendata
```

## What You Get

### Market Data

- `StockPriceService`
  Fetches current, batch, historical, and intraday price data.
- `FundamentalsService`
  Fetches SEC EDGAR fundamentals for US-listed companies.
- `StockDataService`
  Combines price + fundamentals and enriches fundamentals with metrics like `marketCap`, `priceToEarnings`, and `priceToBook`.
- `MarketstackProvider`
  Authenticated stock market provider with current, batch, historical, and intraday support.
- `CoinGeckoProvider`
  Crypto market provider behind the same stock-price interface.
- `SecEdgarProvider`
  Fundamentals provider backed by SEC EDGAR company facts.

### German Company Data

- `OpenData`
  Starts a local embedded database, imports the public German company JSONL dump, and exposes a `handelsregister` helper for company lookup and file download.

### Laws

- `LawService`
  Syncs laws into a local embedded database and lets you search them later.
- `LawRecord`
  The stored law document model returned by sync and search operations.

## Before You Start

- No stock or fundamentals providers are pre-registered. You must register them before calling the services.
- `MarketstackProvider` requires an API key.
- `SecEdgarProvider` requires a valid `User-Agent` string in the format `Company Name email@example.com`.
- `CoinGeckoProvider` works without an API key, but a key gives you better rate limits.
- `OpenData` requires `nogitDir`, `downloadDir`, and `germanBusinessDataDir`.
- `OpenData.start()` is required before using `OpenData.handelsregister` or `buildInitialDb()`.
- `LawService` stores data locally. It auto-starts on demand, but explicit `start()` and `stop()` keep lifecycle handling cleaner.
- EU law syncing and Handelsregister access use a headless browser under the hood.
- `StockDataService` defaults to a 24 hour price cache. If you want fresher quote caching, pass `cache.priceTTL` explicitly or use `StockPriceService` directly.

## Quick Start

### Combined Stock Data

Use `StockDataService` when you want one call that returns both the latest price and SEC fundamentals.

```ts
import {
  MarketstackProvider,
  SecEdgarProvider,
  StockDataService,
} from '@fin.cx/opendata';

const stocks = new StockDataService({
  cache: {
    priceTTL: 60_000,
  },
});

stocks.registerPriceProvider(
  new MarketstackProvider(process.env.MARKETSTACK_API_KEY!)
);

stocks.registerFundamentalsProvider(
  new SecEdgarProvider({
    userAgent: 'Acme Inc. dev@acme.dev',
  })
);

const apple = await stocks.getStockData('AAPL');

console.log({
  ticker: apple.ticker,
  price: apple.price.price,
  provider: apple.price.provider,
  companyName: apple.fundamentals?.companyName,
  marketCap: apple.fundamentals?.marketCap,
  priceToEarnings: apple.fundamentals?.priceToEarnings,
  fetchedAt: apple.fetchedAt,
});
```

You can also batch this:

```ts
const batch = await stocks.getBatchStockData(['AAPL', 'MSFT', 'NVDA']);
```

### Price-Only Stocks And Intraday Data

Use `StockPriceService` if you want smarter per-request caching and direct access to current, historical, and intraday request types.

```ts
import { MarketstackProvider, StockPriceService } from '@fin.cx/opendata';

const prices = new StockPriceService();
prices.register(new MarketstackProvider(process.env.MARKETSTACK_API_KEY!));

const current = await prices.getPrice({ ticker: 'AAPL' });

const historical = await prices.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2025-01-01'),
  to: new Date('2025-01-31'),
  sort: 'DESC',
});

const intraday = await prices.getData({
  type: 'intraday',
  ticker: 'AAPL',
  interval: '1hour',
  limit: 24,
});

console.log({
  live: current.price,
  candles: intraday.length,
  historicalRows: historical.length,
});
```

Supported stock request shapes:

- `{ type: 'current', ticker, exchange? }`
- `{ type: 'batch', tickers, exchange? }`
- `{ type: 'historical', ticker, from, to, exchange?, sort?, limit?, offset? }`
- `{ type: 'intraday', ticker, interval, exchange?, limit?, date? }`

`StockPriceService` also exposes operational helpers like `checkProvidersHealth()`, `getProviderStats()`, `clearCache()`, and `getCacheStats()`.

### Crypto Prices With The Same Interface

`CoinGeckoProvider` plugs into `StockPriceService`, so crypto lookups use the same request model as stock lookups.

```ts
import { CoinGeckoProvider, StockPriceService } from '@fin.cx/opendata';

const prices = new StockPriceService({ ttl: 30_000 });
prices.register(new CoinGeckoProvider(process.env.COINGECKO_API_KEY));

const btc = await prices.getPrice({ ticker: 'BTC' });
const top = await prices.getPrices({ tickers: ['BTC', 'ETH', 'SOL'] });

console.log(btc.price, top.map((item) => item.ticker));
```

You can also pass CoinGecko ids like `bitcoin` or `ethereum` instead of ticker symbols.

### Fundamentals Only

Use `FundamentalsService` when you only care about SEC filing data.

```ts
import { FundamentalsService, SecEdgarProvider } from '@fin.cx/opendata';

const fundamentals = new FundamentalsService();
fundamentals.register(
  new SecEdgarProvider({
    userAgent: 'Acme Inc. dev@acme.dev',
  })
);

const apple = await fundamentals.getFundamentals('AAPL');

console.log({
  companyName: apple.companyName,
  revenue: apple.revenue,
  netIncome: apple.netIncome,
  earningsPerShareDiluted: apple.earningsPerShareDiluted,
  filingDate: apple.filingDate,
});
```

### German Company Data And Handelsregister Automation

`OpenData` is the local workflow layer for German company data. It creates its directories, boots an embedded SmartDB-backed database, and starts a browser automation session for Handelsregister access.

```ts
import { OpenData } from '@fin.cx/opendata';
import * as path from 'node:path';

const nogitDir = path.join(process.cwd(), '.nogit');

const openData = new OpenData({
  nogitDir,
  downloadDir: path.join(nogitDir, 'downloads'),
  germanBusinessDataDir: path.join(nogitDir, 'germanbusinessdata'),
});

await openData.start();

const matches = await openData.handelsregister.searchCompany('Siemens AG', 20);

const company = await openData.handelsregister.getSpecificCompany({
  court: 'Munich',
  type: 'HRB',
  number: '6684',
});

console.log({
  searchHits: matches.length,
  downloadedFiles: company.files.map((file) => file.path),
});

await openData.stop();
```

Useful `OpenData` workflows today:

- `await openData.start()`
- `await openData.buildInitialDb()` to ingest the public German company JSONL dataset
- `await openData.handelsregister.searchCompany(name, limit)`
- `await openData.handelsregister.getSpecificCompany({ court, type, number })`
- `await openData.handelsregister.getSpecificCompanyByName(name)`

The download workflow currently fetches the Handelsregister files into a temporary folder and returns the files to you as `SmartFile` objects.

### Law Syncing And Search

`LawService` stores synced laws in a local embedded database and supports three jurisdictions:

- Germany via `gesetze-im-internet`
- EU via `EUR-Lex`
- US via GovInfo and Cornell LII

```ts
import { LawService } from '@fin.cx/opendata';

const laws = new LawService({
  dbFolderPath: '.nogit/law-smartdb',
  dbName: 'laws',
  govInfoApiKey: process.env.GOVINFO_API_KEY,
});

await laws.start();

const germanLaw = await laws.syncLaw({
  jurisdiction: 'de',
  identifier: 'aeg',
});

const euLaw = await laws.syncLaw({
  jurisdiction: 'eu',
  identifier: '32024R1689',
  language: 'EN',
});

const usCode = await laws.syncLaw({
  jurisdiction: 'us',
  identifier: '8 U.S.C. § 1226',
});

const results = await laws.searchLaws({
  jurisdiction: 'eu',
  query: 'Artificial Intelligence Act',
  limit: 5,
});

console.log({
  germany: germanLaw.title,
  eu: euLaw.title,
  us: usCode.title,
  hits: results.length,
});

await laws.stop();
```

`LawService` also supports bulk sync:

```ts
await laws.syncLaws({
  jurisdiction: 'us',
  usCollection: 'PLAW',
  limit: 25,
  since: new Date('2025-01-01'),
});
```

Useful request patterns:

- Germany: `identifier: 'aeg'`
- EU: `identifier: '32024R1689'`, `language: 'EN'`
- US public law package: `identifier: 'PLAW-119publ1'`, `usCollection: 'PLAW'`
- US Code citation: `identifier: '8 U.S.C. § 1226'`

## API At A Glance

### `StockPriceService`

- `register(provider, config?)`
- `unregister(providerName)`
- `getPrice({ ticker })`
- `getPrices({ tickers })`
- `getData(request)`
- `checkProvidersHealth()`
- `getProviderStats()`
- `clearCache()`
- `getCacheStats()`

### `FundamentalsService`

- `register(provider, config?)`
- `getFundamentals(ticker)`
- `getBatchFundamentals(tickers)`
- `getData(request)`
- `enrichWithPrice(fundamentals, price)`
- `checkProvidersHealth()`
- `getProviderStats()`
- `clearCache()`

### `StockDataService`

- `registerPriceProvider(provider, config?)`
- `registerFundamentalsProvider(provider, config?)`
- `getPrice(ticker)`
- `getPrices(tickers)`
- `getFundamentals(ticker)`
- `getBatchFundamentals(tickers)`
- `getStockData(ticker | request)`
- `getBatchStockData(tickers | request)`
- `checkProvidersHealth()`
- `getProviderStats()`
- `clearCache()`

### `OpenData`

- `start()`
- `stop()`
- `buildInitialDb()`
- `handelsregister.searchCompany(name, limit?)`
- `handelsregister.getSpecificCompany({ court, type, number })`
- `handelsregister.getSpecificCompanyByName(name)`

### `LawService`

- `start()`
- `stop()`
- `getLaw(request)`
- `syncLaw(request)`
- `syncLaws(request)`
- `searchLaws({ query, jurisdiction?, limit? })`

## Behavior And Caveats

- Historical market data is cached aggressively because it is treated as immutable.
- Intraday data uses interval-aware caching and tries incremental refreshes for recent data.
- `StockDataService` is convenience-first. `StockPriceService` is the better low-level choice if you care about request-shape-specific cache behavior.
- `OpenData` is a real workflow layer, not just types. It writes local data, launches a browser, and manages downloads.
- Handelsregister and EUR-Lex integrations depend on current site structure. They are practical automation workflows, not guaranteed stable official APIs.
- The most useful public `OpenData` features today are startup, JSONL import, and Handelsregister operations. Some extra validation/search helper methods exist in source but are still stubs.
- `LawService` persists synced laws locally, so repeated searches can stay offline once the data has been synced.
- There is no hosted backend in this package. Everything is meant to run from your own TypeScript/Node process.

## Testing

```bash
pnpm test
```

Some tests hit live remote services, so they may depend on network access, optional API credentials, or browser automation support.

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./license) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH  
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.