opendata/changelog.md

# Changelog

## 2025-11-01 - 3.1.0 - feat(fundamentals)
Add FundamentalsService and SEC EDGAR provider with caching, rate-limiting, tests, and docs updates

- Introduce FundamentalsService to manage fundamentals providers, caching, retry logic and provider statistics
- Add SecEdgarProvider to fetch SEC EDGAR company facts (CIK lookup, company facts parsing) with rate limiting and local caches
- Expose fundamentals interfaces and services from ts/stocks (exports updated)
- Add comprehensive tests for FundamentalsService and SecEdgarProvider (new test files)
- Update README with new Fundamentals module documentation, usage examples, and configuration guidance
- Implement caching and TTL handling for fundamentals data and provider-specific cache TTL support
- Add .claude/settings.local.json (local permissions) and various test improvements

## 2025-10-31 - 3.0.0 - BREAKING CHANGE(stocks)
Unify stock provider API to discriminated IStockDataRequest and add company name/fullname enrichment

- Replace legacy provider methods (fetchPrice/fetchPrices) with a single fetchData(request: IStockDataRequest) on IStockProvider — providers must be migrated to the new signature.
- Migrate StockPriceService to the unified getData(request: IStockDataRequest) API. Convenience helpers getPrice/getPrices now wrap getData.
- Add companyName and companyFullName fields to IStockPrice and populate them in provider mappings (Marketstack mapping updated; Yahoo provider updated to support the unified API).
- MarketstackProvider: added buildCompanyFullName helper and improved mapping to include company identification fields and full name formatting.
- YahooFinanceProvider: updated to implement fetchData and to route current/batch requests through the new unified request types; historical/intraday throw explicit errors.
- Updated tests to exercise the new unified API, company-name enrichment, caching behavior, and provider direct methods.
- Note: This is a breaking change for external providers and integrations that implemented the old fetchPrice/fetchPrices API. Bump major version.

## 2025-10-31 - 2.1.0 - feat(stocks)
Add unified stock data API (getData) with historical/OHLCV support, smart caching and provider enhancements

- Introduce discriminated union request types (IStockDataRequest) and a unified getData() method (replaces legacy getPrice/getPrices for new use cases)
- Add OHLCV fields (open, high, low, volume, adjusted) and metadata (dataType, fetchedAt) to IStockPrice
- Implement data-type aware smart caching with TTLs (historical = never expire, EOD = 24h, live = 30s, intraday matches interval)
- Extend StockPriceService: new getData(), data-specific cache keys, cache maxEntries increased (default 10000), and TTL-aware add/get cache logic
- Enhance Marketstack provider: unified fetchData(), historical date-range retrieval with pagination, exchange filtering, batch current fetch, OHLCV mapping, and intraday placeholder
- Update Yahoo provider to include dataType and fetchedAt (live data) and maintain legacy fetchPrice/fetchPrices compatibility
- Add/adjust tests to cover unified API, historical retrieval, OHLCV presence and smart caching behavior; test setup updated to require explicit OpenData directory paths
- Update README to document v2.1 changes, migration examples, and new stock provider capabilities

## 2025-10-31 - 2.0.0 - BREAKING CHANGE(OpenData)
Require explicit directory paths for OpenData (nogit/download/germanBusinessData); remove automatic .nogit creation; update HandelsRegister, JsonlDataProcessor, tests and README.

- Breaking: OpenData constructor now requires a config object with nogitDir, downloadDir and germanBusinessDataDir. The constructor will throw if these paths are not provided.
- Removed automatic creation/export of .nogit/download/germanBusinessData from ts/paths. OpenData.start now ensures the required directories exist.
- HandelsRegister API changed: constructor now accepts downloadDir and manages its own unique download folder; screenshot and download paths now use the configured downloadDir.
- JsonlDataProcessor now accepts a germanBusinessDataDir parameter and uses it when ensuring/storing data instead of relying on global paths.
- Updated tests to provide explicit path configuration (tests now set testNogitDir, testDownloadDir, testGermanBusinessDataDir and write outputs accordingly) and to use updated constructors and qenv usage.
- Documentation updated (README) to document the breaking change and show examples for required directory configuration when instantiating OpenData.
- Added .claude/settings.local.json for local permissions/config used in development/CI environments.

## 2025-10-11 - 1.7.0 - feat(stocks)
Add Marketstack provider (EOD) with tests, exports and documentation updates

- Add MarketstackProvider implementation (ts/stocks/providers/provider.marketstack.ts) providing EOD single and batch fetching, availability checks and mapping to IStockPrice.
- Export MarketstackProvider from ts/stocks/index.ts so it is available via the public API.
- Add comprehensive Marketstack tests (test/test.marketstack.node.ts) covering registration, health checks, single/batch fetches, caching, ticker/market validation, provider stats and sample output.
- Update README with Marketstack usage examples, configuration, API key instructions and provider/health documentation.
- Bump dev dependency @git.zone/tstest to ^2.4.2 in package.json.
- Add project helper/config files (.claude/settings.local.json, .serena/project.yml and .serena/.gitignore) to support CI/tooling.

## 2025-09-24 - 1.6.1 - fix(stocks)
Fix Yahoo provider request handling and bump dependency versions

- Refactored Yahoo Finance provider to use SmartRequest.create() builder and await response.json() for HTTP responses (replaces direct getJson usage).
- Improved batch and single-price fetching to use the SmartRequest API, keeping User-Agent header and timeouts.
- Added a compile-time type-check alias to ensure IStockPrice matches tsclass.finance.IStockPrice.
- Bumped development and runtime dependency versions (notable bumps include @git.zone/tsbuild, @git.zone/tstest, @push.rocks/qenv, @push.rocks/smartarchive, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartlog, @push.rocks/smartpath, @push.rocks/smartrequest, @tsclass/tsclass).
- Added .claude/settings.local.json to grant local CI permissions for a few Bash commands.

## 2025-07-12 - 1.6.0 - feat(readme)
Revamp documentation and package description for enhanced clarity

- Restructured README to highlight real-time stock data and German business data, streamlining quick start and advanced examples
- Updated package.json description to better reflect library capabilities
- Added .claude/settings.local.json to define permissions for external tools
- Refined code examples in tests and documentation for improved clarity and consistency

## 2025-04-09 - 1.5.3 - fix(test)
Await file writes in Handelsregister tests to ensure all downloads complete before test end

- Replaced array.map with await Promise.all to properly await asynchronous file writes in test/test.handelsregister.ts
- Improved robustness of asynchronous operations in test suite

## 2025-04-09 - 1.5.2 - fix(readme)
Improve .env configuration code block formatting in documentation

- Wrap the .env variables block in triple backticks for clarity
- Ensure consistency in the Markdown styling of code snippets

## 2025-04-09 - 1.5.1 - fix(core)
No changes detected in project files or documentation. This commit is a placeholder to record that nothing was updated.


## 2025-04-09 - 1.5.0 - feat(documentation)
Enhance project metadata and documentation with comprehensive usage examples, updated descriptions, and improved keywords.

- Updated npmextra.json and package.json to refine the project description and keyword list.
- Expanded readme.md with detailed sections on environment setup, CRUD operations, bulk JSONL processing, and advanced Handelsregister integrations.
- Included advanced workflow examples and error handling strategies in the documentation.
- Adjusted test cases (e.g. in test/test.handelsregister.ts) to reflect changes in company name usage.

## 2025-04-08 - 1.4.6 - fix(tests & jsonl)
Improve test structure and refine JSONL parsing for incomplete data

- Refactored test files to remove redundant get-specific-company tests in test.ts and added missing tests in test.handelsregister.ts
- Updated JSONL data processor to conditionally parse remaining data when available

## 2025-04-05 - 1.4.5 - fix(metadata)
Update repository, bugs, and homepage URLs to code.foss.global

- Repository URL updated from gitlab.com to code.foss.global
- Bugs URL updated from gitlab.com to code.foss.global
- Homepage URL updated to code.foss.global

## 2025-04-05 - 1.4.4 - fix(dependencies & tests)
Update dependency versions and refine test search query

- Bumped versions for several dependencies in package.json, including @git.zone/tsbuild, @git.zone/tsbundle, @git.zone/tstest, @push.rocks/tapbundle, @push.rocks/smartdata, @push.rocks/smartfile, @push.rocks/smartpromise, @push.rocks/smartrequest, and @tsclass/tsclass
- Updated test file to replace the search query 'Volkswagen' with 'LADR'
- Re-enabled the build initial data test by removing tap.skip

## 2025-01-07 - 1.4.3 - fix(test)
Corrected index value in test for fetching specific company data

- Updated the index from 8 to 7 for the germanParsedRegistration fetch in test

## 2025-01-07 - 1.4.2 - fix(core)
Fix concurrency and download handling in HandelsRegister class and adjust test cases

- Improved the clickFindButton function to include an argument for results limit.
- Enhanced the downloadFile function to rename and ensure files are correctly handled.
- Updated searchCompany method to allow specifying a limit on the number of search results.
- Adjusted test cases to select specific test data indices and output test files to a dedicated directory.

## 2025-01-04 - 1.4.1 - fix(core)
Fix issues with JSONL data processing and improve error handling in business record validation

- Fixed JSONL data processing by adding concurrent processing for each JSON line to enhance performance.
- Added validation logic in BusinessRecord class to ensure that the mandatory fields are checked.
- Adjusted environment variable loading in OpenData class to ensure correct database initialization.
- Included missing dependencies and exports in the project files to ensure proper functionality.

## 2025-01-04 - 1.4.0 - feat(HandelsRegister)
Add file download functionality to HandelsRegister

- Implemented file download feature in the HandelsRegister class.
- Configured pages in Puppeteer to allow downloads and set download paths.
- Parsed German registration information with more robust error handling.
- Added specific methods for downloading and handling 'SI' and 'AD' files.

## 2025-01-03 - 1.3.1 - fix(HandelsRegister)
Refined HandelsRegister functionality for better error handling and response capture.

- Improved parsing logic in parseGermanRegistration function.
- Enhanced navigateToPage and clickFindButton methods with error messages for clarity.
- Implemented a new responseListener to handle and log HTTP responses correctly.

## 2025-01-03 - 1.3.0 - feat(core)
Enhanced data handling capabilities and improved company search functionalities.

- Updated business record handling to support more registration types.
- Improved search capabilities for fetching company data with refined registration type matching.
- Added robust logging for JSONL data processing with early exit on successful parse.
- Reorganized test cases to include specific company data retrieval.

## 2025-01-02 - 1.2.1 - fix(BusinessRecord)
Add missing field registrationType to BusinessRecord data

- Introduced the 'registrationType' field to the BusinessRecord data schema with possible values 'HRA' or 'HRB'.

## 2025-01-02 - 1.2.0 - feat(core)
Integrate Handelsregister search for company data retrieval

- Added support for searching company data via Handelsregister.
- Replaced GermanBusinessData functionality with JsonlDataProcessor.
- Included smartbrowser dependency for handling web requests to Handelsregister.

## 2025-01-01 - 1.1.5 - fix(GermanBusinessData)
Add console log for total records processed at the end of the stream.

- Ensure that the number of records processed is logged at the end of data stream processing.

## 2024-12-31 - 1.1.4 - fix(documentation)
Update description and keywords in package.json

- Corrected the package description to reflect the focus on managing, accessing, and updating open data with MongoDB integration.
- Expanded the keywords in the package metadata to include data integration and processing terms.
- Improved README.md with more extensive setup, usage, and introduction of the library's functionalities.

## 2024-12-31 - 1.1.3 - fix(core)
Added missing license file for project completeness.

- Introduced a LICENSE file to the project, ensuring clarity on software usage permissions.

## 2024-12-31 - 1.1.2 - fix(GermanBusinessData)
Ensure unique ID generation for BusinessRecord

- Added generation of a new ID for each BusinessRecord in GermanBusinessData.
- Ensures each business record has a unique identifier.

## 2024-12-31 - 1.1.1 - fix(dependencies)
Update dependencies and devDependencies to newer versions.

- @git.zone/tsbuild from ^2.1.25 to ^2.2.0
- @git.zone/tsbundle from ^2.0.5 to ^2.1.0
- @git.zone/tsrun from ^1.2.46 to ^1.3.3
- @git.zone/tstest from ^1.0.84 to ^1.0.90
- @push.rocks/tapbundle from ^5.0.15 to ^5.5.4
- @types/node from ^20.9.0 to ^22.10.2
- @push.rocks/qenv from ^6.0.4 to ^6.1.0
- @push.rocks/smartarchive from ^4.0.19 to ^4.0.39
- @push.rocks/smartdata from ^5.0.33 to ^5.2.10
- @push.rocks/smartfile from ^11.0.0 to ^11.0.23
- @push.rocks/smartpath from ^5.0.11 to ^5.0.18
- @push.rocks/smartpromise from ^4.0.3 to ^4.0.4
- @push.rocks/smartrequest from ^2.0.21 to ^2.0.23
- @push.rocks/smartstream from ^3.0.30 to ^3.2.5

## 2024-12-31 - 1.1.0 - feat(core)
Enhanced data handling and retrieval features, improved usage documentation

- Updated package description and added project keywords in package.json.
- Extended readme with detailed usage examples and class structures.
- Added getBusinessRecordByName function for dynamic business record retrieval.

## 2024-07-02 - 1.0.3 - fix(core)
No new changes detected. Preparing for patch release.


## 2023-11-14 - 1.0.1 to 1.0.2 - General Updates
Minor version updates and fixes.

- fix(core): update