opendata/readme.md

# @fin.cx/opendata

🚀 **Real-time financial data and German business intelligence in one powerful TypeScript library**

Access live stock prices, cryptocurrencies, forex, commodities AND comprehensive German company data - all through a single, unified API.

## Installation

```bash
npm install @fin.cx/opendata
# or
pnpm add @fin.cx/opendata
```

## Quick Start

### 📈 Stock Market Data

Get market data with EOD (End-of-Day) pricing:

```typescript
import { StockPriceService, MarketstackProvider } from '@fin.cx/opendata';

// Initialize the service with caching
const stockService = new StockPriceService({
  ttl: 60000,      // Cache for 1 minute
  maxEntries: 1000 // Max cached symbols
});

// Register Marketstack provider with API key
stockService.register(new MarketstackProvider('YOUR_API_KEY'), {
  priority: 100,
  retryAttempts: 3
});

// Get single stock price
const apple = await stockService.getPrice({ ticker: 'AAPL' });
console.log(`Apple: $${apple.price} (${apple.changePercent.toFixed(2)}%)`);

// Get multiple prices at once (batch fetching)
const prices = await stockService.getPrices({
  tickers: ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA']
});

// 125,000+ tickers across 72+ exchanges worldwide
const internationalStocks = await stockService.getPrices({
  tickers: ['AAPL', 'VOD.LON', 'SAP.DEX', 'TM', 'BABA']
});
```

### 🏢 German Business Data

Access comprehensive data on German companies:

```typescript
import { OpenData } from '@fin.cx/opendata';

const openData = new OpenData();
await openData.start();

// Create a business record
const company = new openData.CBusinessRecord();
company.data = {
  name: "TechStart GmbH",
  city: "Berlin",
  registrationId: "HRB 123456",
  // ... more fields
};
await company.save();

// Search companies by city
const berlinCompanies = await openData.db
  .collection('businessrecords')
  .find({ city: "Berlin" })
  .toArray();

// Import bulk data from official sources
await openData.buildInitialDb();
```

## Features

### 🎯 Stock Market Module

- **Marketstack API** - End-of-Day (EOD) data for 125,000+ tickers across 72+ exchanges
- **Stock prices** for stocks, ETFs, indices, and more
- **Batch operations** - fetch 100+ symbols in one request
- **Smart caching** - configurable TTL, automatic invalidation
- **Extensible provider system** - easily add new data sources
- **Retry logic** - configurable retry attempts and delays
- **Type-safe** - full TypeScript support with detailed interfaces

### 🇩🇪 German Business Intelligence

- **MongoDB integration** for scalable data storage
- **Bulk JSONL import** from official German data sources
- **Handelsregister automation** - automated document retrieval
- **CRUD operations** with validation
- **Streaming processing** for multi-GB datasets

## Advanced Examples

### Market Dashboard

Create an EOD market overview:

```typescript
const indicators = [
  // Indices
  { ticker: '^GSPC', name: 'S&P 500' },
  { ticker: '^DJI', name: 'DOW Jones' },

  // Tech Giants
  { ticker: 'AAPL', name: 'Apple' },
  { ticker: 'MSFT', name: 'Microsoft' },
  { ticker: 'GOOGL', name: 'Alphabet' },
  { ticker: 'AMZN', name: 'Amazon' },
  { ticker: 'TSLA', name: 'Tesla' }
];

const prices = await stockService.getPrices({
  tickers: indicators.map(i => i.ticker)
});

// Display with color-coded changes
prices.forEach(price => {
  const indicator = indicators.find(i => i.ticker === price.ticker);
  const arrow = price.change >= 0 ? '↑' : '↓';
  const color = price.change >= 0 ? '\x1b[32m' : '\x1b[31m';

  console.log(
    `${indicator.name.padEnd(15)} ${price.price.toFixed(2).padStart(10)} ` +
    `${color}${arrow} ${price.changePercent.toFixed(2)}%\x1b[0m`
  );
});
```

### Provider Health and Statistics

Monitor your provider health and track usage:

```typescript
// Check provider health
const health = await stockService.checkProvidersHealth();
console.log(`Marketstack: ${health.get('Marketstack') ? '✅' : '❌'}`);

// Get provider statistics
const stats = stockService.getProviderStats();
const marketstackStats = stats.get('Marketstack');
console.log('Marketstack Stats:', {
  successCount: marketstackStats.successCount,
  errorCount: marketstackStats.errorCount,
  lastError: marketstackStats.lastError
});
```

### Handelsregister Integration

Automate German company data retrieval:

```typescript
// Search for a company
const results = await openData.handelsregister.searchCompany("Siemens AG");

// Get detailed information and documents
const details = await openData.handelsregister.getSpecificCompany({
  court: "Munich",
  type: "HRB",
  number: "6684"
});

// Downloaded files include:
// - XML data (SI files)
// - PDF documents (AD files)
for (const file of details.files) {
  await file.writeToDir('./downloads');
}
```

### Combined Data Analysis

Merge financial and business data:

```typescript
// Find all public German companies (AG)
const publicCompanies = await openData.db
  .collection('businessrecords')
  .find({ legalForm: 'AG' })
  .toArray();

// Enrich with stock data
for (const company of publicCompanies) {
  try {
    // Map company to ticker (custom logic needed)
    const ticker = mapCompanyToTicker(company.data.name);

    if (ticker) {
      const stock = await stockService.getPrice({ ticker });

      // Add financial metrics
      company.data.stockPrice = stock.price;
      company.data.marketCap = stock.price * getSharesOutstanding(ticker);
      company.data.priceChange = stock.changePercent;

      await company.save();
    }
  } catch (error) {
    // Handle missing tickers gracefully
  }
}
```

## Configuration

### Stock Service Options

```typescript
const stockService = new StockPriceService({
  ttl: 60000,         // Cache for 1 minute
  maxEntries: 1000    // Max cached symbols
});

// Marketstack - EOD data, requires API key
stockService.register(new MarketstackProvider('YOUR_API_KEY'), {
  enabled: true,
  priority: 100,
  timeout: 10000,
  retryAttempts: 3,
  retryDelay: 1000
});
```

### MongoDB Setup

Set environment variables for German business data:

```env
MONGODB_URL=mongodb://localhost:27017
MONGODB_NAME=opendata
MONGODB_USER=myuser
MONGODB_PASS=mypass
```

### Marketstack API Key

Get your free API key at [marketstack.com](https://marketstack.com) and set it in your environment:

```env
MARKETSTACK_COM_TOKEN=your_api_key_here
```

## API Reference

### Stock Types

```typescript
interface IStockPrice {
  ticker: string;
  price: number;
  currency: string;
  change: number;
  changePercent: number;
  previousClose: number;
  timestamp: Date;
  provider: string;
  marketState: 'PRE' | 'REGULAR' | 'POST' | 'CLOSED';
  exchange?: string;
  exchangeName?: string;
}
```

### Key Methods

**StockPriceService**
- `getPrice(request)` - Single stock price with automatic provider selection
- `getPrices(request)` - Batch prices (100+ symbols in one request)
- `register(provider, config)` - Add data provider with priority and retry config
- `checkProvidersHealth()` - Test all providers and return health status
- `getProviderStats()` - Get success/error statistics for each provider
- `clearCache()` - Clear price cache
- `setCacheTTL(ttl)` - Update cache TTL dynamically

**MarketstackProvider**
- ✅ End-of-Day (EOD) data
- ✅ 125,000+ tickers across 72+ exchanges worldwide
- ✅ Batch fetching support (multiple symbols in one request)
- ✅ Comprehensive data: open, high, low, close, volume, splits, dividends
- ⚠️ Requires API key (free tier: 100 requests/month)
- ⚠️ EOD data only (not real-time)

**OpenData**
- `start()` - Initialize MongoDB connection
- `buildInitialDb()` - Import bulk data
- `CBusinessRecord` - Business record class
- `handelsregister` - Registry automation

## Provider Architecture

The library uses a flexible provider system that makes it easy to add new data sources:

```typescript
class MyCustomProvider implements IStockProvider {
  name = 'My Provider';
  priority = 50;
  requiresAuth = true;

  async fetchPrice(request: IStockQuoteRequest): Promise<IStockPrice> {
    // Your implementation
  }

  async fetchPrices(request: IStockBatchQuoteRequest): Promise<IStockPrice[]> {
    // Batch implementation
  }

  async isAvailable(): Promise<boolean> {
    // Health check
  }

  supportsMarket(market: string): boolean {
    // Market validation
  }

  supportsTicker(ticker: string): boolean {
    // Ticker validation
  }
}

stockService.register(new MyCustomProvider());
```

## Performance

- **Batch fetching**: Get 100+ EOD prices in one API request
- **Smart caching**: Instant repeated queries with configurable TTL
- **Rate limit aware**: Automatic retry logic for API limits
- **Concurrent processing**: Handle 1000+ business records/second
- **Streaming**: Process GB-sized datasets without memory issues

## Testing

Run the comprehensive test suite:

```bash
npm test
```

Test stock provider:

```bash
npx tstest test/test.marketstack.node.ts --verbose
```

Test German business data:

```bash
npx tstest test/test.handelsregister.ts --verbose
```

## License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

### Company Information

Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.