opendata/changelog.md

# Changelog

## 2025-11-02 - 3.3.0 - feat(stocks/CoinGeckoProvider)
Add CoinGecko provider for cryptocurrency prices, export and tests, and update documentation

- Implemented CoinGeckoProvider with rate limiting, coin-id resolution, and support for current, batch, historical and intraday price endpoints
- Added unit/integration tests for CoinGecko: test/test.coingecko.node+bun+deno.ts
- Exported CoinGeckoProvider from ts/stocks/index.ts
- Updated README and readme.hints.md with CoinGecko usage, provider notes and examples
- Added .claude/settings.local.json with webfetch and bash permissions required for testing and CI

## 2025-11-01 - 3.2.2 - fix(handelsregister)
Correct screenshot path handling in HandelsRegister and add local tool permissions

- ts/classes.handelsregister.ts: Replace string concatenation for screenshot path with a template literal and explicit string assertion to ensure the path is formed correctly for page.screenshot() and avoid type issues.
- Add .claude/settings.local.json: Introduce local Claude settings that grant specific tool permissions used during development and testing (bash commands, web fetches, pnpm build, tstest, etc.).

## 2025-11-01 - 3.2.1 - fix(stocks/providers/provider.secedgar)
Improve SEC EDGAR provider networking and error handling, update plugin path import, bump dev deps and add/refresh tests and lockfile

- SEC EDGAR provider: switch from SmartRequest to native fetch for ticker list and company facts, add AbortController-based timeouts, handle gzip automatically, improve response validation and error messages, and keep CIK/ticker-list caching
- Improve timeout and rate-limit handling in SecEdgarProvider (uses native fetch + explicit timeout clear), plus clearer logging on failures
- Update ts/plugins import to use node:path for Node compatibility
- Bump devDependencies: @git.zone/tsrun to ^1.6.2 and @git.zone/tstest to ^2.7.0; bump @push.rocks/smartrequest to ^4.3.4
- Add and refresh comprehensive test files (node/bun/deno variants) for fundamentals, marketstack, secedgar and stockdata services
- Add deno.lock (dependency lock) and a local .claude/settings.local.json for CI/permissions

## 2025-11-01 - 3.2.0 - feat(StockDataService)
Add unified StockDataService and BaseProviderService with new stockdata interfaces, provider integrations, tests and README updates

- Introduce StockDataService: unified API to fetch prices and fundamentals with automatic enrichment and caching
- Add IStockData and IStockDataServiceConfig interfaces to define combined price+fundamentals payloads and configuration
- Implement BaseProviderService abstraction to share provider registration, health, stats and caching logic
- Add classes.stockdataservice.ts implementing batch/single fetch, enrichment, caching, health checks and provider stats
- Export new stockdata module and classes from ts/stocks/index.ts
- Add comprehensive tests: test/test.stockdata.service.node.ts to cover setup, provider registration, fetching, caching, enrichment, health and error handling
- Update README with Unified Stock Data API examples, usage, and documentation reflecting new unified service
- Minor infra: add .claude/settings.local.json permissions for local tooling and web fetch domains

## 2025-11-01 - 3.1.0 - feat(fundamentals)
Add FundamentalsService and SEC EDGAR provider with caching, rate-limiting, tests, and docs updates

- Introduce FundamentalsService to manage fundamentals providers, caching, retry logic and provider statistics
- Add SecEdgarProvider to fetch SEC EDGAR company facts (CIK lookup, company facts parsing) with rate limiting and local caches
- Expose fundamentals interfaces and services from ts/stocks (exports updated)
- Add comprehensive tests for FundamentalsService and SecEdgarProvider (new test files)
- Update README with new Fundamentals module documentation, usage examples, and configuration guidance
- Implement caching and TTL handling for fundamentals data and provider-specific cache TTL support
- Add .claude/settings.local.json (local permissions) and various test improvements

## 2025-10-31 - 3.0.0 - BREAKING CHANGE(stocks)
Unify stock provider API to discriminated IStockDataRequest and add company name/fullname enrichment

- Replace legacy provider methods (fetchPrice/fetchPrices) with a single fetchData(request: IStockDataRequest) on IStockProvider — providers must be migrated to the new signature.
- Migrate StockPriceService to the unified getData(request: IStockDataRequest) API. Convenience helpers getPrice/getPrices now wrap getData.
- Add companyName and companyFullName fields to IStockPrice and populate them in provider mappings (Marketstack mapping updated; Yahoo provider updated to support the unified API).
- MarketstackProvider: added buildCompanyFullName helper and improved mapping to include company identification fields and full name formatting.
- YahooFinanceProvider: updated to implement fetchData and to route current/batch requests through the new unified request types; historical/intraday throw explicit errors.
- Updated tests to exercise the new unified API, company-name enrichment, caching behavior, and provider direct methods.
- Note: This is a breaking change for external providers and integrations that implemented the old fetchPrice/fetchPrices API. Bump major version.

## 2025-10-31 - 2.1.0 - feat(stocks)
Add unified stock data API (getData) with historical/OHLCV support, smart caching and provider enhancements

- Introduce discriminated union request types (IStockDataRequest) and a unified getData() method (replaces legacy getPrice/getPrices for new use cases)
- Add OHLCV fields (open, high, low, volume, adjusted) and metadata (dataType, fetchedAt) to IStockPrice
- Implement data-type aware smart caching with TTLs (historical = never expire, EOD = 24h, live = 30s, intraday matches interval)
- Extend StockPriceService: new getData(), data-specific cache keys, cache maxEntries increased (default 10000), and TTL-aware add/get cache logic
- Enhance Marketstack provider: unified fetchData(), historical date-range retrieval with pagination, exchange filtering, batch current fetch, OHLCV mapping, and intraday placeholder
- Update Yahoo provider to include dataType and fetchedAt (live data) and maintain legacy fetchPrice/fetchPrices compatibility
- Add/adjust tests to cover unified API, historical retrieval, OHLCV presence and smart caching behavior; test setup updated to require explicit OpenData directory paths
- Update README to document v2.1 changes, migration examples, and new stock provider capabilities

## 2025-10-31 - 2.0.0 - BREAKING CHANGE(OpenData)
Require explicit directory paths for OpenData (nogit/download/germanBusinessData); remove automatic .nogit creation; update HandelsRegister, JsonlDataProcessor, tests and README.

- Breaking: OpenData constructor now requires a config object with nogitDir, downloadDir and germanBusinessDataDir. The constructor will throw if these paths are not provided.
- Removed automatic creation/export of .nogit/download/germanBusinessData from ts/paths. OpenData.start now ensures the required directories exist.
- HandelsRegister API changed: constructor now accepts downloadDir and manages its own unique download folder; screenshot and download paths now use the configured downloadDir.
- JsonlDataProcessor now accepts a germanBusinessDataDir parameter and uses it when ensuring/storing data instead of relying on global paths.
- Updated tests to provide explicit path configuration (tests now set testNogitDir, testDownloadDir, testGermanBusinessDataDir and write outputs accordingly) and to use updated constructors and qenv usage.
- Documentation updated (README) to document the breaking change and show examples for required directory configuration when instantiating OpenData.
- Added .claude/settings.local.json for local permissions/config used in development/CI environments.

## 2025-10-11 - 1.7.0 - feat(stocks)
Add Marketstack provider (EOD) with tests, exports and documentation updates

- Add MarketstackProvider implementation (ts/stocks/providers/provider.marketstack.ts) providing EOD single and batch fetching, availability checks and mapping to IStockPrice.
- Export MarketstackProvider from ts/stocks/index.ts so it is available via the public API.
- Add comprehensive Marketstack tests (test/test.marketstack.node.ts) covering registration, health checks, single/batch fetches, caching, ticker/market validation, provider stats and sample output.
- Update README with Marketstack usage examples, configuration, API key instructions and provider/health documentation.
- Bump dev dependency @git.zone/tstest to ^2.4.2 in package.json.
- Add project helper/config files (.claude/settings.local.json, .serena/project.yml and .serena/.gitignore) to support CI/tooling.

## 2025-09-24 - 1.6.1 - fix(stocks)
Fix Yahoo provider request handling and bump dependency versions

- Refactored Yahoo Finance provider to use SmartRequest.create() builder and await response.json() for HTTP responses (replaces direct getJson usage).
- Improved batch and single-price fetching to use the SmartRequest API, keeping User-Agent header and timeouts.
- Added a compile-time type-check alias to ensure IStockPrice matches tsclass.finance.IStockPrice.
- Bumped development and runtime dependency versions (notable bumps include @git.zone/tsbuild, @git.zone/tstest, @push.rocks/qenv, @push.rocks/smartarchive, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartlog, @push.rocks/smartpath, @push.rocks/smartrequest, @tsclass/tsclass).
- Added .claude/settings.local.json to grant local CI permissions for a few Bash commands.

## 2025-07-12 - 1.6.0 - feat(readme)
Revamp documentation and package description for enhanced clarity

- Restructured README to highlight real-time stock data and German business data, streamlining quick start and advanced examples
- Updated package.json description to better reflect library capabilities
- Added .claude/settings.local.json to define permissions for external tools
- Refined code examples in tests and documentation for improved clarity and consistency

## 2025-04-09 - 1.5.3 - fix(test)
Await file writes in Handelsregister tests to ensure all downloads complete before test end

- Replaced array.map with await Promise.all to properly await asynchronous file writes in test/test.handelsregister.ts
- Improved robustness of asynchronous operations in test suite

## 2025-04-09 - 1.5.2 - fix(readme)
Improve .env configuration code block formatting in documentation

- Wrap the .env variables block in triple backticks for clarity
- Ensure consistency in the Markdown styling of code snippets

## 2025-04-09 - 1.5.1 - fix(core)
No changes detected in project files or documentation. This commit is a placeholder to record that nothing was updated.


## 2025-04-09 - 1.5.0 - feat(documentation)
Enhance project metadata and documentation with comprehensive usage examples, updated descriptions, and improved keywords.

- Updated npmextra.json and package.json to refine the project description and keyword list.
- Expanded readme.md with detailed sections on environment setup, CRUD operations, bulk JSONL processing, and advanced Handelsregister integrations.
- Included advanced workflow examples and error handling strategies in the documentation.
- Adjusted test cases (e.g. in test/test.handelsregister.ts) to reflect changes in company name usage.

## 2025-04-08 - 1.4.6 - fix(tests & jsonl)
Improve test structure and refine JSONL parsing for incomplete data

- Refactored test files to remove redundant get-specific-company tests in test.ts and added missing tests in test.handelsregister.ts
- Updated JSONL data processor to conditionally parse remaining data when available

## 2025-04-05 - 1.4.5 - fix(metadata)
Update repository, bugs, and homepage URLs to code.foss.global

- Repository URL updated from gitlab.com to code.foss.global
- Bugs URL updated from gitlab.com to code.foss.global
- Homepage URL updated to code.foss.global

## 2025-04-05 - 1.4.4 - fix(dependencies & tests)
Update dependency versions and refine test search query

- Bumped versions for several dependencies in package.json, including @git.zone/tsbuild, @git.zone/tsbundle, @git.zone/tstest, @push.rocks/tapbundle, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartpromise, @push.rocks/smartrequest, and @tsclass/tsclass
- Updated test file to replace the search query 'Volkswagen' with 'LADR'
- Re-enabled the build initial data test by removing tap.skip

## 2025-01-07 - 1.4.3 - fix(test)
Corrected index value in test for fetching specific company data

- Updated the index from 8 to 7 for the germanParsedRegistration fetch in test

## 2025-01-07 - 1.4.2 - fix(core)
Fix concurrency and download handling in HandelsRegister class and adjust test cases

- Improved the clickFindButton function to include an argument for results limit.
- Enhanced the downloadFile function to rename and ensure files are correctly handled.
- Updated searchCompany method to allow specifying a limit on the number of search results.
- Adjusted test cases to select specific test data indices and output test files to a dedicated directory.

## 2025-01-04 - 1.4.1 - fix(core)
Fix issues with JSONL data processing and improve error handling in business record validation

- Fixed JSONL data processing by adding concurrent processing for each JSON line to enhance performance.
- Added validation logic in BusinessRecord class to ensure that the mandatory fields are checked.
- Adjusted environment variable loading in OpenData class to ensure correct database initialization.
- Included missing dependencies and exports in the project files to ensure proper functionality.

## 2025-01-04 - 1.4.0 - feat(HandelsRegister)
Add file download functionality to HandelsRegister

- Implemented file download feature in the HandelsRegister class.
- Configured pages in Puppeteer to allow downloads and set download paths.
- Parsed German registration information with more robust error handling.
- Added specific methods for downloading and handling 'SI' and 'AD' files.

## 2025-01-03 - 1.3.1 - fix(HandelsRegister)
Refined HandelsRegister functionality for better error handling and response capture.

- Improved parsing logic in parseGermanRegistration function.
- Enhanced navigateToPage and clickFindButton methods with error messages for clarity.
- Implemented a new responseListener to handle and log HTTP responses correctly.

## 2025-01-03 - 1.3.0 - feat(core)
Enhanced data handling capabilities and improved company search functionalities.

- Updated business record handling to support more registration types.
- Improved search capabilities for fetching company data with refined registration type matching.
- Added robust logging for JSONL data processing with early exit on successful parse.
- Reorganized test cases to include specific company data retrieval.

## 2025-01-02 - 1.2.1 - fix(BusinessRecord)
Add missing field registrationType to BusinessRecord data

- Introduced the 'registrationType' field to the BusinessRecord data schema with possible values 'HRA' or 'HRB'.

## 2025-01-02 - 1.2.0 - feat(core)
Integrate Handelsregister search for company data retrieval

- Added support for searching company data via Handelsregister.
- Replaced GermanBusinessData functionality with JsonlDataProcessor.
- Included smartbrowser dependency for handling web requests to Handelsregister.

## 2025-01-01 - 1.1.5 - fix(GermanBusinessData)
Add console log for total records processed at the end of the stream.

- Ensure that the number of records processed is logged at the end of data stream processing.

## 2024-12-31 - 1.1.4 - fix(documentation)
Update description and keywords in package.json

- Corrected the package description to reflect the focus on managing, accessing, and updating open data with MongoDB integration.
- Expanded the keywords in the package metadata to include data integration and processing terms.
- Improved README.md with more extensive setup, usage, and introduction of the library's functionalities.

## 2024-12-31 - 1.1.3 - fix(core)
Added missing license file for project completeness.

- Introduced a LICENSE file to the project, ensuring clarity on software usage permissions.

## 2024-12-31 - 1.1.2 - fix(GermanBusinessData)
Ensure unique ID generation for BusinessRecord

- Added generation of a new ID for each BusinessRecord in GermanBusinessData.
- Ensures each business record has a unique identifier.

## 2024-12-31 - 1.1.1 - fix(dependencies)
Update dependencies and devDependencies to newer versions.

- @git.zone/tsbuild from ^2.1.25 to ^2.2.0
- @git.zone/tsbundle from ^2.0.5 to ^2.1.0
- @git.zone/tsrun from ^1.2.46 to ^1.3.3
- @git.zone/tstest from ^1.0.84 to ^1.0.90
- @push.rocks/tapbundle from ^5.0.15 to ^5.5.4
- @types/node from ^20.9.0 to ^22.10.2
- @push.rocks/qenv from ^6.0.4 to ^6.1.0
- @push.rocks/smartarchive from ^4.0.19 to ^4.0.39
- @push.rocks/smartdata from ^5.0.33 to ^5.2.10
- @push.rocks/smartfile from ^11.0.0 to ^11.0.23
- @push.rocks/smartpath from ^5.0.11 to ^5.0.18
- @push.rocks/smartpromise from ^4.0.3 to ^4.0.4
- @push.rocks/smartrequest from ^2.0.21 to ^2.0.23
- @push.rocks/smartstream from ^3.0.30 to ^3.2.5

## 2024-12-31 - 1.1.0 - feat(core)
Enhanced data handling and retrieval features, improved usage documentation

- Updated package description and added project keywords in package.json.
- Extended readme with detailed usage examples and class structures.
- Added getBusinessRecordByName function for dynamic business record retrieval.

## 2024-07-02 - 1.0.3 - fix(core)
No new changes detected. Preparing for patch release.


## 2023-11-14 - 1.0.1 to 1.0.2 - General Updates
Minor version updates and fixes.

- fix(core): update