opendata/readme.md

# @fin.cx/opendata

🚀 **Real-time financial data and German business intelligence in one powerful TypeScript library**

Access live stock prices, cryptocurrencies, forex, commodities AND comprehensive German company data - all through a single, unified API.

## ⚠️ Breaking Changes

### v2.1 - Enhanced Stock Market API (Current)

**The stock market API has been significantly enhanced with a new unified request system.**

**What Changed:**
- New discriminated union request types (`IStockDataRequest`)
- Enhanced `IStockPrice` interface with OHLCV data and metadata
- New `getData()` method replaces legacy `getPrice()` and `getPrices()`
- Historical data support with date ranges
- Exchange filtering via MIC codes
- Smart caching with data-type aware TTL

**Migration:**
```typescript
// OLD (v2.0 and earlier)
const price = await service.getPrice({ ticker: 'AAPL' });
const prices = await service.getPrices({ tickers: ['AAPL', 'MSFT'] });

// NEW (v2.1+) - Unified API
const price = await service.getData({ type: 'current', ticker: 'AAPL' });
const prices = await service.getData({ type: 'batch', tickers: ['AAPL', 'MSFT'] });

// NEW - Historical data
const history = await service.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2024-01-01'),
  to: new Date('2024-12-31')
});

// NEW - Exchange filtering
const price = await service.getData({
  type: 'current',
  ticker: 'VOD',
  exchange: 'XLON' // London Stock Exchange
});
```

**Note:** Legacy `getPrice()` and `getPrices()` methods still work but are deprecated.

### v2.0 - Directory Configuration

**Directory paths are now MANDATORY when using German business data features.** The package no longer creates `.nogit/` directories automatically. You must explicitly configure all directory paths when instantiating `OpenData`:

```typescript
const openData = new OpenData({
  nogitDir: '/path/to/your/data',
  downloadDir: '/path/to/your/data/downloads',
  germanBusinessDataDir: '/path/to/your/data/germanbusinessdata'
});
```

This change enables the package to work in read-only filesystems (like Deno compiled binaries) and gives you full control over where data is stored.

## Installation

```bash
npm install @fin.cx/opendata
# or
pnpm add @fin.cx/opendata
```

## Quick Start

### 📈 Stock Market Data (v2.1+ Enhanced)

Get comprehensive market data with EOD, historical, and OHLCV data:

```typescript
import { StockPriceService, MarketstackProvider } from '@fin.cx/opendata';

// Initialize the service with smart caching
const stockService = new StockPriceService({
  ttl: 60000,       // Default cache TTL (historical data cached forever)
  maxEntries: 10000  // Increased for historical data
});

// Register Marketstack provider with API key
stockService.register(new MarketstackProvider('YOUR_API_KEY'), {
  priority: 100,
  retryAttempts: 3
});

// Get current price (new unified API)
const apple = await stockService.getData({ type: 'current', ticker: 'AAPL' });
console.log(`Apple: $${apple.price} (${apple.changePercent.toFixed(2)}%)`);
console.log(`OHLCV: O=${apple.open} H=${apple.high} L=${apple.low} V=${apple.volume}`);

// Get historical data (1 year of daily prices)
const history = await stockService.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2024-01-01'),
  to: new Date('2024-12-31'),
  sort: 'DESC' // Newest first
});
console.log(`Fetched ${history.length} trading days`);

// Exchange-specific data (London vs NYSE)
const vodLondon = await stockService.getData({
  type: 'current',
  ticker: 'VOD',
  exchange: 'XLON' // London Stock Exchange
});

const vodNYSE = await stockService.getData({
  type: 'current',
  ticker: 'VOD',
  exchange: 'XNYS' // New York Stock Exchange
});

// Batch current prices
const prices = await stockService.getData({
  type: 'batch',
  tickers: ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA']
});
```

### 🏢 German Business Data

Access comprehensive data on German companies:

```typescript
import { OpenData } from '@fin.cx/opendata';
import * as path from 'path';

// REQUIRED: Configure directory paths
const openData = new OpenData({
  nogitDir: path.join(process.cwd(), '.nogit'),
  downloadDir: path.join(process.cwd(), '.nogit', 'downloads'),
  germanBusinessDataDir: path.join(process.cwd(), '.nogit', 'germanbusinessdata')
});
await openData.start();

// Create a business record
const company = new openData.CBusinessRecord();
company.data = {
  name: "TechStart GmbH",
  city: "Berlin",
  registrationId: "HRB 123456",
  // ... more fields
};
await company.save();

// Search companies by city
const berlinCompanies = await openData.db
  .collection('businessrecords')
  .find({ city: "Berlin" })
  .toArray();

// Import bulk data from official sources
await openData.buildInitialDb();
```

## Features

### 🎯 Stock Market Module (v2.1 Enhanced)

- **Historical Data** - Up to 15 years of daily EOD prices with automatic pagination
- **Exchange Filtering** - Query specific exchanges via MIC codes (XNAS, XLON, XNYS, etc.)
- **OHLCV Data** - Open, High, Low, Close, Volume for comprehensive analysis
- **Smart Caching** - Data-type aware TTL (historical cached forever, EOD 24h, live 30s)
- **Marketstack API** - 500,000+ tickers across 72+ exchanges worldwide
- **Batch Operations** - Fetch 100+ symbols in one request
- **Unified API** - Discriminated union types for type-safe requests
- **Extensible Providers** - Easy to add new data sources
- **Retry Logic** - Configurable attempts and delays
- **Type-Safe** - Full TypeScript support with detailed interfaces

### 🇩🇪 German Business Intelligence

- **MongoDB integration** for scalable data storage
- **Bulk JSONL import** from official German data sources
- **Handelsregister automation** - automated document retrieval
- **CRUD operations** with validation
- **Streaming processing** for multi-GB datasets

## Advanced Examples

### Phase 1: Historical Data Analysis

Fetch and analyze historical price data:

```typescript
// Get 1 year of historical data
const history = await stockService.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2024-01-01'),
  to: new Date('2024-12-31'),
  sort: 'DESC' // Newest first (default)
});

// Calculate statistics
const prices = history.map(p => p.price);
const high52Week = Math.max(...prices);
const low52Week = Math.min(...prices);
const avgPrice = prices.reduce((a, b) => a + b) / prices.length;

console.log(`52-Week Analysis for AAPL:`);
console.log(`High:    $${high52Week.toFixed(2)}`);
console.log(`Low:     $${low52Week.toFixed(2)}`);
console.log(`Average: $${avgPrice.toFixed(2)}`);
console.log(`Days:    ${history.length}`);

// Calculate daily returns
for (let i = 0; i < history.length - 1; i++) {
  const todayPrice = history[i].price;
  const yesterdayPrice = history[i + 1].price;
  const dailyReturn = ((todayPrice - yesterdayPrice) / yesterdayPrice) * 100;
  console.log(`${history[i].timestamp.toISOString().split('T')[0]}: ${dailyReturn.toFixed(2)}%`);
}
```

### Phase 1: Exchange-Specific Trading

Compare prices across different exchanges:

```typescript
// Vodafone trades on both London and NYSE
const exchanges = [
  { mic: 'XLON', name: 'London Stock Exchange' },
  { mic: 'XNYS', name: 'New York Stock Exchange' }
];

for (const exchange of exchanges) {
  try {
    const price = await stockService.getData({
      type: 'current',
      ticker: 'VOD',
      exchange: exchange.mic
    });

    console.log(`${exchange.name}:`);
    console.log(`  Price: ${price.price} ${price.currency}`);
    console.log(`  Volume: ${price.volume?.toLocaleString()}`);
    console.log(`  Exchange: ${price.exchangeName}`);
  } catch (error) {
    console.log(`${exchange.name}: Not available`);
  }
}
```

### Phase 1: OHLCV Technical Analysis

Use OHLCV data for technical indicators:

```typescript
const history = await stockService.getData({
  type: 'historical',
  ticker: 'TSLA',
  from: new Date('2024-11-01'),
  to: new Date('2024-11-30')
});

// Calculate daily trading range
for (const day of history) {
  const range = day.high - day.low;
  const rangePercent = (range / day.low) * 100;

  console.log(`${day.timestamp.toISOString().split('T')[0]}:`);
  console.log(`  Open:   $${day.open}`);
  console.log(`  High:   $${day.high}`);
  console.log(`  Low:    $${day.low}`);
  console.log(`  Close:  $${day.price}`);
  console.log(`  Volume: ${day.volume?.toLocaleString()}`);
  console.log(`  Range:  $${range.toFixed(2)} (${rangePercent.toFixed(2)}%)`);
}

// Calculate Simple Moving Average (SMA)
const calculateSMA = (data: IStockPrice[], period: number) => {
  const sma: number[] = [];
  for (let i = period - 1; i < data.length; i++) {
    const sum = data.slice(i - period + 1, i + 1)
      .reduce((acc, p) => acc + p.price, 0);
    sma.push(sum / period);
  }
  return sma;
};

const sma20 = calculateSMA(history, 20);
const sma50 = calculateSMA(history, 50);

console.log(`20-day SMA: $${sma20[0].toFixed(2)}`);
console.log(`50-day SMA: $${sma50[0].toFixed(2)}`);
```

### Phase 1: Smart Caching Performance

Leverage smart caching for efficiency:

```typescript
// Historical data is cached FOREVER (never changes)
console.time('First historical fetch');
const history1 = await stockService.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2024-01-01'),
  to: new Date('2024-12-31')
});
console.timeEnd('First historical fetch');
// Output: First historical fetch: 2341ms

console.time('Second historical fetch (cached)');
const history2 = await stockService.getData({
  type: 'historical',
  ticker: 'AAPL',
  from: new Date('2024-01-01'),
  to: new Date('2024-12-31')
});
console.timeEnd('Second historical fetch (cached)');
// Output: Second historical fetch (cached): 2ms (1000x faster!)

// EOD data cached for 24 hours
const currentPrice = await stockService.getData({
  type: 'current',
  ticker: 'MSFT'
});
// Subsequent calls within 24h served from cache

// Cache statistics
console.log(`Cache size: ${stockService['cache'].size} entries`);
```

### Market Dashboard

Create an EOD market overview:

```typescript
const indicators = [
  // Indices
  { ticker: '^GSPC', name: 'S&P 500' },
  { ticker: '^DJI', name: 'DOW Jones' },

  // Tech Giants
  { ticker: 'AAPL', name: 'Apple' },
  { ticker: 'MSFT', name: 'Microsoft' },
  { ticker: 'GOOGL', name: 'Alphabet' },
  { ticker: 'AMZN', name: 'Amazon' },
  { ticker: 'TSLA', name: 'Tesla' }
];

const prices = await stockService.getPrices({
  tickers: indicators.map(i => i.ticker)
});

// Display with color-coded changes
prices.forEach(price => {
  const indicator = indicators.find(i => i.ticker === price.ticker);
  const arrow = price.change >= 0 ? '↑' : '↓';
  const color = price.change >= 0 ? '\x1b[32m' : '\x1b[31m';

  console.log(
    `${indicator.name.padEnd(15)} ${price.price.toFixed(2).padStart(10)} ` +
    `${color}${arrow} ${price.changePercent.toFixed(2)}%\x1b[0m`
  );
});
```

### Provider Health and Statistics

Monitor your provider health and track usage:

```typescript
// Check provider health
const health = await stockService.checkProvidersHealth();
console.log(`Marketstack: ${health.get('Marketstack') ? '✅' : '❌'}`);

// Get provider statistics
const stats = stockService.getProviderStats();
const marketstackStats = stats.get('Marketstack');
console.log('Marketstack Stats:', {
  successCount: marketstackStats.successCount,
  errorCount: marketstackStats.errorCount,
  lastError: marketstackStats.lastError
});
```

### Handelsregister Integration

Automate German company data retrieval:

```typescript
import { OpenData } from '@fin.cx/opendata';
import * as path from 'path';

// Configure paths first
const openData = new OpenData({
  nogitDir: path.join(process.cwd(), '.nogit'),
  downloadDir: path.join(process.cwd(), '.nogit', 'downloads'),
  germanBusinessDataDir: path.join(process.cwd(), '.nogit', 'germanbusinessdata')
});
await openData.start();

// Search for a company
const results = await openData.handelsregister.searchCompany("Siemens AG");

// Get detailed information and documents
const details = await openData.handelsregister.getSpecificCompany({
  court: "Munich",
  type: "HRB",
  number: "6684"
});

// Downloaded files include:
// - XML data (SI files)
// - PDF documents (AD files)
for (const file of details.files) {
  await file.writeToDir('./downloads');
}
```

### Combined Data Analysis

Merge financial and business data:

```typescript
import { OpenData, StockPriceService, MarketstackProvider } from '@fin.cx/opendata';
import * as path from 'path';

// Configure OpenData with paths
const openData = new OpenData({
  nogitDir: path.join(process.cwd(), '.nogit'),
  downloadDir: path.join(process.cwd(), '.nogit', 'downloads'),
  germanBusinessDataDir: path.join(process.cwd(), '.nogit', 'germanbusinessdata')
});
await openData.start();

// Setup stock service
const stockService = new StockPriceService({ ttl: 60000, maxEntries: 1000 });
stockService.register(new MarketstackProvider('YOUR_API_KEY'));

// Find all public German companies (AG)
const publicCompanies = await openData.db
  .collection('businessrecords')
  .find({ legalForm: 'AG' })
  .toArray();

// Enrich with stock data
for (const company of publicCompanies) {
  try {
    // Map company to ticker (custom logic needed)
    const ticker = mapCompanyToTicker(company.data.name);

    if (ticker) {
      const stock = await stockService.getPrice({ ticker });

      // Add financial metrics
      company.data.stockPrice = stock.price;
      company.data.marketCap = stock.price * getSharesOutstanding(ticker);
      company.data.priceChange = stock.changePercent;

      await company.save();
    }
  } catch (error) {
    // Handle missing tickers gracefully
  }
}
```

## Configuration

### Directory Configuration (Required for German Business Data)

**All directory paths are mandatory when using `OpenData`.** Here are examples for different environments:

#### Development Environment
```typescript
import { OpenData } from '@fin.cx/opendata';
import * as path from 'path';

const openData = new OpenData({
  nogitDir: path.join(process.cwd(), '.nogit'),
  downloadDir: path.join(process.cwd(), '.nogit', 'downloads'),
  germanBusinessDataDir: path.join(process.cwd(), '.nogit', 'germanbusinessdata')
});
```

#### Production Environment
```typescript
import { OpenData } from '@fin.cx/opendata';
import * as path from 'path';

const openData = new OpenData({
  nogitDir: '/var/lib/myapp/data',
  downloadDir: '/var/lib/myapp/data/downloads',
  germanBusinessDataDir: '/var/lib/myapp/data/germanbusinessdata'
});
```

#### Deno Compiled Binaries (or other read-only filesystems)
```typescript
import { OpenData } from '@fin.cx/opendata';

// Use OS temp directory or user data directory
const dataDir = Deno.env.get('HOME') + '/.myapp/data';

const openData = new OpenData({
  nogitDir: dataDir,
  downloadDir: dataDir + '/downloads',
  germanBusinessDataDir: dataDir + '/germanbusinessdata'
});
```

### Stock Service Options

```typescript
const stockService = new StockPriceService({
  ttl: 60000,         // Cache for 1 minute
  maxEntries: 1000    // Max cached symbols
});

// Marketstack - EOD data, requires API key
stockService.register(new MarketstackProvider('YOUR_API_KEY'), {
  enabled: true,
  priority: 100,
  timeout: 10000,
  retryAttempts: 3,
  retryDelay: 1000
});
```

### MongoDB Setup

Set environment variables for German business data:

```env
MONGODB_URL=mongodb://localhost:27017
MONGODB_NAME=opendata
MONGODB_USER=myuser
MONGODB_PASS=mypass
```

### Marketstack API Key

Get your free API key at [marketstack.com](https://marketstack.com) and set it in your environment:

```env
MARKETSTACK_COM_TOKEN=your_api_key_here
```

## API Reference

### Stock Types

```typescript
interface IStockPrice {
  ticker: string;
  price: number;
  currency: string;
  change: number;
  changePercent: number;
  previousClose: number;
  timestamp: Date;
  provider: string;
  marketState: 'PRE' | 'REGULAR' | 'POST' | 'CLOSED';
  exchange?: string;
  exchangeName?: string;
}
```

### Key Methods

**StockPriceService**
- `getPrice(request)` - Single stock price with automatic provider selection
- `getPrices(request)` - Batch prices (100+ symbols in one request)
- `register(provider, config)` - Add data provider with priority and retry config
- `checkProvidersHealth()` - Test all providers and return health status
- `getProviderStats()` - Get success/error statistics for each provider
- `clearCache()` - Clear price cache
- `setCacheTTL(ttl)` - Update cache TTL dynamically

**MarketstackProvider**
- ✅ End-of-Day (EOD) data
- ✅ 125,000+ tickers across 72+ exchanges worldwide
- ✅ Batch fetching support (multiple symbols in one request)
- ✅ Comprehensive data: open, high, low, close, volume, splits, dividends
- ⚠️ Requires API key (free tier: 100 requests/month)
- ⚠️ EOD data only (not real-time)

**OpenData**
- `start()` - Initialize MongoDB connection
- `buildInitialDb()` - Import bulk data
- `CBusinessRecord` - Business record class
- `handelsregister` - Registry automation

## Provider Architecture

The library uses a flexible provider system that makes it easy to add new data sources:

```typescript
class MyCustomProvider implements IStockProvider {
  name = 'My Provider';
  priority = 50;
  requiresAuth = true;

  async fetchPrice(request: IStockQuoteRequest): Promise<IStockPrice> {
    // Your implementation
  }

  async fetchPrices(request: IStockBatchQuoteRequest): Promise<IStockPrice[]> {
    // Batch implementation
  }

  async isAvailable(): Promise<boolean> {
    // Health check
  }

  supportsMarket(market: string): boolean {
    // Market validation
  }

  supportsTicker(ticker: string): boolean {
    // Ticker validation
  }
}

stockService.register(new MyCustomProvider());
```

## Performance

- **Batch fetching**: Get 100+ EOD prices in one API request
- **Smart caching**: Instant repeated queries with configurable TTL
- **Rate limit aware**: Automatic retry logic for API limits
- **Concurrent processing**: Handle 1000+ business records/second
- **Streaming**: Process GB-sized datasets without memory issues

## Testing

Run the comprehensive test suite:

```bash
npm test
```

Test stock provider:

```bash
npx tstest test/test.marketstack.node.ts --verbose
```

Test German business data:

```bash
npx tstest test/test.handelsregister.ts --verbose
```

## License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

### Company Information

Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.