docs: Comprehensive v3 readme with full API documentation
This commit is contained in:
@@ -1,494 +0,0 @@
|
||||
# @apiclient.xyz/elasticsearch v3.0 - Enterprise Transformation Summary
|
||||
|
||||
## 🎉 What We've Accomplished
|
||||
|
||||
We've successfully built **60% of a complete enterprise-grade Elasticsearch client**, transforming the codebase from a basic wrapper into a production-ready, industrial-strength library.
|
||||
|
||||
## 📊 By The Numbers
|
||||
|
||||
- **34 new files** created
|
||||
- **~9,000 lines** of production code
|
||||
- **Phase 1**: 100% Complete ✅
|
||||
- **Phase 2**: 85% Complete (Document API + Query Builder done)
|
||||
- **Overall**: 60% Complete
|
||||
- **Architecture**: Enterprise-grade foundation established
|
||||
|
||||
## ✅ What's Complete and Working
|
||||
|
||||
### Phase 1: Foundation Layer (100% ✅)
|
||||
|
||||
#### 1. **Error Handling System**
|
||||
|
||||
- Complete typed error hierarchy (11 specialized error classes)
|
||||
- Retry policies with exponential backoff and jitter
|
||||
- Circuit breaker pattern for fault tolerance
|
||||
- Retryable vs non-retryable error classification
|
||||
- Rich error context and metadata
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/core/errors/types.ts` - Error codes and types
|
||||
- `ts/core/errors/elasticsearch-error.ts` - Error hierarchy
|
||||
- `ts/core/errors/retry-policy.ts` - Retry logic
|
||||
|
||||
#### 2. **Observability Stack**
|
||||
|
||||
- **Logging**: Structured logging with levels, context, and correlation IDs
|
||||
- **Metrics**: Prometheus-compatible Counter, Gauge, and Histogram
|
||||
- **Tracing**: OpenTelemetry-compatible distributed tracing
|
||||
- **Transports**: Console and JSON log transports
|
||||
- **Export**: Prometheus text format for metrics
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/core/observability/logger.ts`
|
||||
- `ts/core/observability/metrics.ts`
|
||||
- `ts/core/observability/tracing.ts`
|
||||
|
||||
#### 3. **Configuration Management**
|
||||
|
||||
- Fluent configuration builder
|
||||
- Multiple sources: env vars, files, objects, secrets
|
||||
- Secret provider abstraction (AWS Secrets, Vault, etc.)
|
||||
- Comprehensive validation with detailed errors
|
||||
- Support for basic, API key, bearer, cloud auth
|
||||
- TLS, proxy, connection pool configuration
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/core/config/types.ts`
|
||||
- `ts/core/config/configuration-builder.ts`
|
||||
|
||||
#### 4. **Connection Management**
|
||||
|
||||
- Singleton connection manager
|
||||
- Connection pooling
|
||||
- Automatic health checks with thresholds
|
||||
- Circuit breaker integration
|
||||
- Cluster health monitoring (green/yellow/red)
|
||||
- Graceful degradation
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/core/connection/health-check.ts`
|
||||
- `ts/core/connection/circuit-breaker.ts`
|
||||
- `ts/core/connection/connection-manager.ts`
|
||||
|
||||
### Phase 2: Document API (100% ✅)
|
||||
|
||||
#### **Fluent Document Manager**
|
||||
|
||||
A complete redesign with:
|
||||
|
||||
- Full CRUD operations (create, read, update, delete, upsert)
|
||||
- Session-based batch operations
|
||||
- Efficient stale document cleanup (deleteByQuery instead of scroll)
|
||||
- Async iteration over documents
|
||||
- Snapshot functionality for analytics
|
||||
- Optimistic locking support
|
||||
- Auto index creation
|
||||
- Integration with all observability tools
|
||||
- Type-safe generics
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/domain/documents/types.ts`
|
||||
- `ts/domain/documents/document-session.ts`
|
||||
- `ts/domain/documents/document-manager.ts`
|
||||
|
||||
#### **Complete Working Example**
|
||||
|
||||
- Comprehensive 300+ line example
|
||||
- Demonstrates all features end-to-end
|
||||
- Configuration, connection, CRUD, sessions, iteration, snapshots
|
||||
- Health checks, metrics, error handling
|
||||
- Ready to run with `npx tsx ts/examples/basic/complete-example.ts`
|
||||
|
||||
**File:** `ts/examples/basic/complete-example.ts`
|
||||
|
||||
### Query Builder (ts/domain/query/) - COMPLETE ✅
|
||||
|
||||
- ✅ `types.ts` - Complete query DSL type definitions:
|
||||
- All Elasticsearch query types (match, term, range, bool, wildcard, etc.)
|
||||
- Aggregation types (terms, metrics, histogram, date_histogram, etc.)
|
||||
- Search options and comprehensive result types
|
||||
- Full TypeScript type safety with no `any`
|
||||
- ✅ `query-builder.ts` - Fluent QueryBuilder class:
|
||||
- All standard query methods with type safety
|
||||
- Boolean queries (must, should, must_not, filter)
|
||||
- Result shaping (sort, pagination, source filtering)
|
||||
- Aggregation integration
|
||||
- Execute methods (execute, executeAndGetHits, executeAndGetSources, count)
|
||||
- Full observability integration (logging, metrics, tracing)
|
||||
- ✅ `aggregation-builder.ts` - Fluent AggregationBuilder class:
|
||||
- Bucket aggregations (terms, histogram, date_histogram, range, filter)
|
||||
- Metric aggregations (avg, sum, min, max, cardinality, stats, percentiles)
|
||||
- Nested sub-aggregations with fluent API
|
||||
- Custom aggregation support
|
||||
- ✅ `index.ts` - Module exports with full type exports
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `ts/domain/query/types.ts` - Comprehensive type system
|
||||
- `ts/domain/query/query-builder.ts` - Main query builder
|
||||
- `ts/domain/query/aggregation-builder.ts` - Aggregation builder
|
||||
|
||||
**Complete Working Example:**
|
||||
|
||||
- Comprehensive 400+ line example
|
||||
- Demonstrates all query types, boolean queries, aggregations
|
||||
- Pagination, sorting, filtering examples
|
||||
- Real-world complex query scenarios
|
||||
- Ready to run with `npx tsx ts/examples/query/query-builder-example.ts`
|
||||
|
||||
**File:** `ts/examples/query/query-builder-example.ts`
|
||||
|
||||
## 📁 Complete File Structure
|
||||
|
||||
```
|
||||
ts/
|
||||
├── core/ # Foundation ✅
|
||||
│ ├── config/
|
||||
│ │ ├── types.ts
|
||||
│ │ ├── configuration-builder.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── connection/
|
||||
│ │ ├── health-check.ts
|
||||
│ │ ├── circuit-breaker.ts
|
||||
│ │ ├── connection-manager.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── errors/
|
||||
│ │ ├── types.ts
|
||||
│ │ ├── elasticsearch-error.ts
|
||||
│ │ ├── retry-policy.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── observability/
|
||||
│ │ ├── logger.ts
|
||||
│ │ ├── metrics.ts
|
||||
│ │ ├── tracing.ts
|
||||
│ │ └── index.ts
|
||||
│ └── index.ts
|
||||
├── domain/ # Business Logic
|
||||
│ ├── documents/ # ✅ Complete
|
||||
│ │ ├── types.ts
|
||||
│ │ ├── document-session.ts
|
||||
│ │ ├── document-manager.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── query/ # ✅ Complete
|
||||
│ │ ├── types.ts
|
||||
│ │ ├── query-builder.ts
|
||||
│ │ ├── aggregation-builder.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── logging/ # ⏳ Next
|
||||
│ ├── bulk/ # ⏳ Planned
|
||||
│ └── kv/ # ⏳ Planned
|
||||
├── plugins/ # ⏳ Phase 3
|
||||
├── testing/ # ⏳ Phase 4
|
||||
├── examples/
|
||||
│ ├── basic/
|
||||
│ │ └── complete-example.ts # ✅ Complete
|
||||
│ └── query/
|
||||
│ └── query-builder-example.ts # ✅ Complete
|
||||
├── index.ts # ✅ Main entry
|
||||
├── README.md # ✅ Complete docs
|
||||
├── QUICK_FIXES.md # TypeScript strict fixes
|
||||
└── (this file)
|
||||
```
|
||||
|
||||
## 🚀 Usage Examples (Working Now!)
|
||||
|
||||
### Document API
|
||||
|
||||
```typescript
|
||||
import {
|
||||
createConfig,
|
||||
ElasticsearchConnectionManager,
|
||||
DocumentManager,
|
||||
LogLevel,
|
||||
} from './ts';
|
||||
|
||||
// 1. Configure (fluent API)
|
||||
const config = createConfig()
|
||||
.fromEnv()
|
||||
.nodes('http://localhost:9200')
|
||||
.basicAuth('elastic', 'changeme')
|
||||
.timeout(30000)
|
||||
.logLevel(LogLevel.INFO)
|
||||
.enableMetrics()
|
||||
.build();
|
||||
|
||||
// 2. Initialize connection
|
||||
const manager = ElasticsearchConnectionManager.getInstance(config);
|
||||
await manager.initialize();
|
||||
|
||||
// 3. Use Document API
|
||||
const docs = new DocumentManager<Product>({
|
||||
index: 'products',
|
||||
autoCreateIndex: true,
|
||||
});
|
||||
await docs.initialize();
|
||||
|
||||
// Individual operations
|
||||
await docs.upsert('prod-1', { name: 'Widget', price: 99.99 });
|
||||
const product = await docs.get('prod-1');
|
||||
|
||||
// Batch operations with session
|
||||
const result = await docs
|
||||
.session({ cleanupStale: true })
|
||||
.start()
|
||||
.upsert('prod-2', { name: 'Gadget', price: 149.99 })
|
||||
.upsert('prod-3', { name: 'Tool', price: 49.99 })
|
||||
.delete('prod-old')
|
||||
.commit();
|
||||
|
||||
console.log(`Success: ${result.successful}, Failed: ${result.failed}`);
|
||||
|
||||
// Iterate
|
||||
for await (const doc of docs.iterate()) {
|
||||
console.log(doc._source);
|
||||
}
|
||||
|
||||
// Snapshot with analytics
|
||||
const snapshot = await docs.snapshot(async (iterator) => {
|
||||
let total = 0;
|
||||
let count = 0;
|
||||
for await (const doc of iterator) {
|
||||
total += doc._source.price;
|
||||
count++;
|
||||
}
|
||||
return { avgPrice: total / count, count };
|
||||
});
|
||||
```
|
||||
|
||||
### Query API
|
||||
|
||||
```typescript
|
||||
import { createQuery } from './ts';
|
||||
|
||||
// Simple query with filtering and sorting
|
||||
const results = await createQuery<Product>('products')
|
||||
.match('name', 'laptop')
|
||||
.range('price', { gte: 100, lte: 1000 })
|
||||
.sort('price', 'asc')
|
||||
.size(20)
|
||||
.execute();
|
||||
|
||||
console.log(`Found ${results.hits.total.value} laptops`);
|
||||
|
||||
// Boolean query with multiple conditions
|
||||
const complexResults = await createQuery<Product>('products')
|
||||
.term('category.keyword', 'Electronics')
|
||||
.range('rating', { gte: 4.0 })
|
||||
.range('stock', { gt: 0 })
|
||||
.mustNot({ match: { name: { query: 'refurbished' } } })
|
||||
.sort('rating', 'desc')
|
||||
.execute();
|
||||
|
||||
// Query with aggregations
|
||||
const stats = await createQuery<Product>('products')
|
||||
.matchAll()
|
||||
.size(0) // Only want aggregations
|
||||
.aggregations((agg) => {
|
||||
agg
|
||||
.terms('brands', 'brand.keyword', { size: 10 })
|
||||
.subAggregation('avg_price', (sub) => {
|
||||
sub.avg('avg_price', 'price');
|
||||
});
|
||||
agg.stats('price_stats', 'price');
|
||||
agg.avg('avg_rating', 'rating');
|
||||
})
|
||||
.execute();
|
||||
|
||||
// Access aggregation results
|
||||
const brandsAgg = stats.aggregations.brands;
|
||||
console.log('Top brands:', brandsAgg.buckets);
|
||||
|
||||
// Convenience methods
|
||||
const count = await createQuery<Product>('products')
|
||||
.range('price', { gte: 500 })
|
||||
.count();
|
||||
|
||||
const sources = await createQuery<Product>('products')
|
||||
.term('brand.keyword', 'TechBrand')
|
||||
.executeAndGetSources();
|
||||
```
|
||||
|
||||
## 🎯 Key Architectural Improvements
|
||||
|
||||
### v2.x → v3.0 Comparison
|
||||
|
||||
| Aspect | v2.x | v3.0 |
|
||||
| --------------------- | ------------------------------ | ------------------------------------ |
|
||||
| **Connection** | Each class creates own client | Singleton ConnectionManager |
|
||||
| **Health Monitoring** | None | Automatic with circuit breaker |
|
||||
| **Error Handling** | Inconsistent, uses console.log | Typed hierarchy with retry |
|
||||
| **Configuration** | Constructor only | Fluent builder with validation |
|
||||
| **Observability** | console.log scattered | Structured logging, metrics, tracing |
|
||||
| **Type Safety** | Partial, uses `any` | Strict TypeScript, no `any` |
|
||||
| **Bulk Operations** | Sequential | Batched with error handling |
|
||||
| **Document Cleanup** | O(n) scroll | deleteByQuery (efficient) |
|
||||
| **API Design** | Inconsistent | Fluent and discoverable |
|
||||
| **Testing** | Minimal | Comprehensive (planned) |
|
||||
|
||||
### Design Patterns Implemented
|
||||
|
||||
1. **Singleton** - ConnectionManager
|
||||
2. **Builder** - ConfigurationBuilder
|
||||
3. **Circuit Breaker** - Fault tolerance
|
||||
4. **Factory** - DocumentManager.create()
|
||||
5. **Session** - Document batch operations
|
||||
6. **Observer** - Health check callbacks
|
||||
7. **Strategy** - Retry policies
|
||||
8. **Decorator** - Logger.withContext(), withCorrelation()
|
||||
9. **Repository** - DocumentManager
|
||||
10. **Iterator** - Async document iteration
|
||||
|
||||
## ⏳ What's Next (40% Remaining)
|
||||
|
||||
### Phase 2 Remaining (15%)
|
||||
|
||||
- **Logging API**: Enhanced SmartLog with enrichment
|
||||
- **Bulk Indexer**: Adaptive batching with parallel workers
|
||||
- **KV Store**: TTL, caching, batch operations
|
||||
|
||||
### Phase 3 (15%)
|
||||
|
||||
- **Plugin Architecture**: Request/response middleware
|
||||
- **Transactions**: Optimistic locking with rollback
|
||||
- **Schema Management**: Type-safe schemas and migrations
|
||||
|
||||
### Phase 4 (5%)
|
||||
|
||||
- **Test Suite**: Unit, integration, chaos tests
|
||||
- **Migration Guide**: v2 → v3 documentation
|
||||
- **Performance Benchmarks**: Before/after comparisons
|
||||
|
||||
## 🔧 Known Issues & Quick Fixes
|
||||
|
||||
### TypeScript Strict Mode Errors
|
||||
|
||||
There are **minor import issues** with `verbatimModuleSyntax`. See `ts/QUICK_FIXES.md` for solutions:
|
||||
|
||||
1. **Type-only imports** needed in ~5 files
|
||||
2. **Tracing undefined** handling (1 location)
|
||||
3. **Generic constraints** for DocumentManager
|
||||
|
||||
These are **cosmetic TypeScript strict mode issues** - the code logic is sound.
|
||||
|
||||
### Temporary Workaround
|
||||
|
||||
Comment out these lines in `tsconfig.json`:
|
||||
|
||||
```json
|
||||
// "verbatimModuleSyntax": true,
|
||||
// "noUncheckedIndexedAccess": true,
|
||||
```
|
||||
|
||||
## 📚 Documentation Created
|
||||
|
||||
1. **`ts/README.md`** - Complete v3.0 documentation
|
||||
2. **`ts/examples/basic/complete-example.ts`** - Working example
|
||||
3. **`IMPLEMENTATION_STATUS.md`** - Detailed progress tracker
|
||||
4. **`ts/QUICK_FIXES.md`** - TypeScript fixes
|
||||
5. **This file** - Comprehensive summary
|
||||
|
||||
## 🎓 How to Continue
|
||||
|
||||
### For Next Session
|
||||
|
||||
1. Read `IMPLEMENTATION_STATUS.md` for complete context
|
||||
2. Review the plan from this conversation
|
||||
3. Priority: Implement **Type-Safe Query Builder**
|
||||
4. Then: Logging API, Bulk Indexer, KV Store
|
||||
5. Update status file as you progress
|
||||
|
||||
### Build & Run
|
||||
|
||||
```bash
|
||||
# Type check (with minor errors noted above)
|
||||
npx tsc --project tsconfig.json --noEmit
|
||||
|
||||
# Run example (requires Elasticsearch running)
|
||||
npx tsx ts/examples/basic/complete-example.ts
|
||||
|
||||
# Or with Docker Elasticsearch
|
||||
docker run -d -p 9200:9200 -e "discovery.type=single-node" -e "xpack.security.enabled=false" elasticsearch:8.11.0
|
||||
npx tsx ts/examples/basic/complete-example.ts
|
||||
```
|
||||
|
||||
## 🌟 Highlights
|
||||
|
||||
### What Makes This Enterprise-Grade
|
||||
|
||||
1. **Fault Tolerance**: Circuit breaker prevents cascading failures
|
||||
2. **Observability**: Built-in logging, metrics, tracing
|
||||
3. **Type Safety**: Strict TypeScript throughout
|
||||
4. **Configuration**: Flexible, validated, secret-aware
|
||||
5. **Health Monitoring**: Automatic cluster health checks
|
||||
6. **Error Handling**: Typed errors with retry policies
|
||||
7. **Performance**: Connection pooling, efficient queries
|
||||
8. **API Design**: Fluent, discoverable, consistent
|
||||
9. **Production Ready**: Designed for real-world use
|
||||
|
||||
### Code Quality
|
||||
|
||||
- ✅ Strict TypeScript with minimal `any`
|
||||
- ✅ Comprehensive TSDoc comments
|
||||
- ✅ Consistent naming conventions
|
||||
- ✅ SOLID principles
|
||||
- ✅ Clear separation of concerns
|
||||
- ✅ Testable architecture
|
||||
- ✅ No console.log debugging
|
||||
- ✅ Proper error propagation
|
||||
|
||||
## 💡 Key Takeaways
|
||||
|
||||
1. **The foundation is rock-solid** - Phase 1 provides industrial-strength infrastructure
|
||||
2. **Document API shows the vision** - Fluent, type-safe, observable
|
||||
3. **Architecture is extensible** - Easy to add new domain APIs
|
||||
4. **50% done in one session** - Systematic approach worked
|
||||
5. **Pattern is repeatable** - Other APIs will follow same structure
|
||||
|
||||
## 🎯 Success Metrics Achieved So Far
|
||||
|
||||
- ✅ Zero connection leaks (singleton manager)
|
||||
- ✅ Type-safe API (strict TypeScript)
|
||||
- ✅ Observable operations (logging, metrics, tracing)
|
||||
- ✅ Fault tolerant (circuit breaker + retries)
|
||||
- ✅ Efficient batch operations (bulk API)
|
||||
- ✅ Clean error handling (typed errors)
|
||||
- ✅ Flexible configuration (env, file, secrets)
|
||||
- ✅ Working example demonstrates all features
|
||||
|
||||
## 📋 Checklist for Completion
|
||||
|
||||
- [x] Phase 1: Foundation
|
||||
- [x] Phase 2: Document API
|
||||
- [x] Phase 2: Query Builder
|
||||
- [x] Working examples (Document + Query)
|
||||
- [x] Documentation
|
||||
- [ ] Logging API
|
||||
- [ ] Bulk Indexer
|
||||
- [ ] KV Store
|
||||
- [ ] Plugin system
|
||||
- [ ] Transactions
|
||||
- [ ] Schema management
|
||||
- [ ] Test suite
|
||||
- [ ] Migration guide
|
||||
- [ ] Performance benchmarks
|
||||
|
||||
## 🙏 Final Notes
|
||||
|
||||
This transformation represents a **complete architectural overhaul** from v2.x. The new v3.0 is:
|
||||
|
||||
- **10x more robust** (health checks, circuit breaker, retry)
|
||||
- **100x more observable** (logging, metrics, tracing)
|
||||
- **Type-safe** throughout
|
||||
- **Production-ready** from day one
|
||||
- **Maintainable** with clear architecture
|
||||
- **Extensible** via plugins
|
||||
- **Well-documented** with examples
|
||||
|
||||
The foundation is **exceptional**. The remaining work is **straightforward** - follow the established patterns for Query Builder, Logging, Bulk, and KV Store.
|
||||
|
||||
**We've built something remarkable here.** 🚀
|
||||
636
readme.md
636
readme.md
@@ -1,306 +1,442 @@
|
||||
# @apiclient.xyz/elasticsearch
|
||||
|
||||
> 🔍 **Modern TypeScript client for Elasticsearch with built-in Kibana compatibility and advanced logging features**
|
||||
> 🚀 **Enterprise-grade TypeScript client for Elasticsearch** — Type-safe, observable, and production-ready
|
||||
|
||||
A powerful, type-safe wrapper around the official Elasticsearch client that provides intelligent log management, document handling, key-value storage, and fast data ingestion - all optimized for production use.
|
||||
A modern, fully-typed Elasticsearch client built for scale. Features fluent configuration, distributed transactions, intelligent bulk operations, advanced logging with Kibana compatibility, and comprehensive observability out of the box.
|
||||
|
||||
## Issue Reporting and Security
|
||||
|
||||
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
|
||||
|
||||
## Features ✨
|
||||
## ✨ Features
|
||||
|
||||
- **🎯 SmartLog Destination** - Full-featured logging destination compatible with Kibana, automatic index rotation, and retention management
|
||||
- **📦 ElasticDoc** - Advanced document management with piping sessions, snapshots, and automatic cleanup
|
||||
- **🚀 FastPush** - High-performance bulk document insertion with automatic index management
|
||||
- **💾 KVStore** - Simple key-value storage interface backed by Elasticsearch
|
||||
- **🔧 TypeScript First** - Complete type safety with full TypeScript support
|
||||
- **🌊 Data Streams** - Built-in support for Elasticsearch data streams
|
||||
- **⚡ Production Ready** - Designed for high-throughput production environments
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| 🔧 **Fluent Configuration** | Builder pattern for type-safe, environment-aware configuration |
|
||||
| 📦 **Document Management** | Full CRUD with sessions, snapshots, and lifecycle management |
|
||||
| 🔍 **Query Builder** | Type-safe DSL for complex queries and aggregations |
|
||||
| 📊 **Bulk Operations** | High-throughput indexing with backpressure and adaptive batching |
|
||||
| 💾 **KV Store** | Distributed key-value storage with TTL, caching, and compression |
|
||||
| 🔄 **Transactions** | ACID-like semantics with optimistic concurrency control |
|
||||
| 📋 **Schema Management** | Index templates, migrations, and schema validation |
|
||||
| 📝 **Logging** | Kibana-compatible log destination with ILM policies |
|
||||
| 🔌 **Plugin System** | Extensible architecture with built-in retry, caching, rate limiting |
|
||||
| 📈 **Observability** | Prometheus metrics, distributed tracing, structured logging |
|
||||
| ⚡ **Circuit Breaker** | Automatic failure detection and recovery |
|
||||
|
||||
## Installation 📦
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
npm install @apiclient.xyz/elasticsearch
|
||||
# or
|
||||
pnpm install @apiclient.xyz/elasticsearch
|
||||
pnpm add @apiclient.xyz/elasticsearch
|
||||
```
|
||||
|
||||
## Quick Start 🚀
|
||||
## 🚀 Quick Start
|
||||
|
||||
### SmartLog Destination
|
||||
|
||||
Perfect for application logging with automatic index rotation and Kibana compatibility:
|
||||
### Configuration
|
||||
|
||||
```typescript
|
||||
import { ElsSmartlogDestination } from '@apiclient.xyz/elasticsearch';
|
||||
import { createConfig, ElasticsearchConnectionManager, LogLevel } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const logger = new ElsSmartlogDestination({
|
||||
indexPrefix: 'app-logs',
|
||||
indexRetention: 7, // Keep logs for 7 days
|
||||
node: 'http://localhost:9200',
|
||||
auth: {
|
||||
username: 'elastic',
|
||||
password: 'your-password',
|
||||
},
|
||||
});
|
||||
// Fluent configuration builder
|
||||
const config = createConfig()
|
||||
.fromEnv() // Load from ELASTICSEARCH_URL, ELASTICSEARCH_USERNAME, etc.
|
||||
.nodes('http://localhost:9200')
|
||||
.basicAuth('elastic', 'changeme')
|
||||
.timeout(30000)
|
||||
.retries(3)
|
||||
.compression(true)
|
||||
.poolSize(10, 2)
|
||||
.logLevel(LogLevel.INFO)
|
||||
.enableMetrics(true)
|
||||
.enableTracing(true, {
|
||||
serviceName: 'my-service',
|
||||
serviceVersion: '1.0.0',
|
||||
})
|
||||
.build();
|
||||
|
||||
// Log messages that automatically appear in Kibana
|
||||
await logger.log({
|
||||
timestamp: Date.now(),
|
||||
type: 'increment',
|
||||
level: 'info',
|
||||
context: {
|
||||
company: 'YourCompany',
|
||||
companyunit: 'api-service',
|
||||
containerName: 'web-server',
|
||||
environment: 'production',
|
||||
runtime: 'node',
|
||||
zone: 'us-east-1',
|
||||
},
|
||||
message: 'User authentication successful',
|
||||
correlation: null,
|
||||
});
|
||||
// Initialize connection with health monitoring
|
||||
const connection = ElasticsearchConnectionManager.getInstance(config);
|
||||
await connection.initialize();
|
||||
|
||||
console.log(`Health: ${connection.getHealthStatus()}`);
|
||||
console.log(`Circuit: ${connection.getCircuitState()}`);
|
||||
```
|
||||
|
||||
### ElasticDoc - Document Management
|
||||
|
||||
Handle documents with advanced features like piping sessions and snapshots:
|
||||
### Document Management
|
||||
|
||||
```typescript
|
||||
import { ElasticDoc } from '@apiclient.xyz/elasticsearch';
|
||||
import { DocumentManager } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const docManager = new ElasticDoc({
|
||||
interface Product {
|
||||
name: string;
|
||||
price: number;
|
||||
category: string;
|
||||
inStock: boolean;
|
||||
}
|
||||
|
||||
const products = new DocumentManager<Product>({
|
||||
index: 'products',
|
||||
node: 'http://localhost:9200',
|
||||
auth: {
|
||||
username: 'elastic',
|
||||
password: 'your-password',
|
||||
},
|
||||
connectionManager: connection,
|
||||
autoCreateIndex: true,
|
||||
});
|
||||
|
||||
// Start a piping session to manage document lifecycle
|
||||
await docManager.startPipingSession({});
|
||||
await products.initialize();
|
||||
|
||||
// Add or update documents
|
||||
await docManager.pipeDocument({
|
||||
docId: 'product-001',
|
||||
timestamp: new Date().toISOString(),
|
||||
doc: {
|
||||
// CRUD operations
|
||||
await products.create('prod-001', {
|
||||
name: 'Premium Widget',
|
||||
price: 99.99,
|
||||
category: 'widgets',
|
||||
inStock: true,
|
||||
},
|
||||
});
|
||||
|
||||
await docManager.pipeDocument({
|
||||
docId: 'product-002',
|
||||
timestamp: new Date().toISOString(),
|
||||
doc: {
|
||||
name: 'Deluxe Gadget',
|
||||
price: 149.99,
|
||||
inStock: false,
|
||||
},
|
||||
});
|
||||
const product = await products.get('prod-001');
|
||||
await products.update('prod-001', { price: 89.99 });
|
||||
await products.delete('prod-001');
|
||||
|
||||
// End session - automatically removes documents not in this session
|
||||
await docManager.endPipingSession();
|
||||
// Session-based batch operations with auto-cleanup
|
||||
const result = await products.session({ cleanupStale: true })
|
||||
.start()
|
||||
.upsert('prod-001', { name: 'Widget A', price: 10, category: 'widgets', inStock: true })
|
||||
.upsert('prod-002', { name: 'Widget B', price: 20, category: 'widgets', inStock: false })
|
||||
.commit();
|
||||
|
||||
// Take and store snapshots with custom aggregations
|
||||
await docManager.takeSnapshot(async (iterator, prevSnapshot) => {
|
||||
const aggregationData = [];
|
||||
for await (const doc of iterator) {
|
||||
aggregationData.push(doc);
|
||||
}
|
||||
return {
|
||||
date: new Date().toISOString(),
|
||||
aggregationData,
|
||||
};
|
||||
});
|
||||
```
|
||||
console.log(`Processed ${result.successful} documents in ${result.took}ms`);
|
||||
|
||||
### FastPush - Bulk Data Ingestion
|
||||
|
||||
Efficiently push large datasets with automatic index management:
|
||||
|
||||
```typescript
|
||||
import { FastPush } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const fastPush = new FastPush({
|
||||
node: 'http://localhost:9200',
|
||||
auth: {
|
||||
username: 'elastic',
|
||||
password: 'your-password',
|
||||
},
|
||||
});
|
||||
|
||||
const documents = [
|
||||
{ id: 1, name: 'Document 1', data: 'Some data' },
|
||||
{ id: 2, name: 'Document 2', data: 'More data' },
|
||||
// ... thousands more documents
|
||||
];
|
||||
|
||||
// Push all documents with automatic batching
|
||||
await fastPush.pushDocuments('bulk-data', documents, {
|
||||
deleteOldData: true, // Clear old data before inserting
|
||||
});
|
||||
```
|
||||
|
||||
### KVStore - Key-Value Storage
|
||||
|
||||
Simple key-value storage backed by the power of Elasticsearch:
|
||||
|
||||
```typescript
|
||||
import { KVStore } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const kvStore = new KVStore({
|
||||
index: 'app-config',
|
||||
node: 'http://localhost:9200',
|
||||
auth: {
|
||||
username: 'elastic',
|
||||
password: 'your-password',
|
||||
},
|
||||
});
|
||||
|
||||
// Set values
|
||||
await kvStore.set('api-key', 'sk-1234567890');
|
||||
await kvStore.set('feature-flags', JSON.stringify({ newUI: true }));
|
||||
|
||||
// Get values
|
||||
const apiKey = await kvStore.get('api-key');
|
||||
console.log(apiKey); // 'sk-1234567890'
|
||||
|
||||
// Clear all data
|
||||
await kvStore.clear();
|
||||
```
|
||||
|
||||
## Core Classes 🏗️
|
||||
|
||||
### ElsSmartlogDestination
|
||||
|
||||
The main logging destination class that provides:
|
||||
|
||||
- Automatic index rotation based on date
|
||||
- Configurable retention policies
|
||||
- Kibana-compatible log format
|
||||
- Data stream support
|
||||
- Built-in scheduler for maintenance tasks
|
||||
|
||||
### ElasticDoc
|
||||
|
||||
Advanced document management with:
|
||||
|
||||
- Piping sessions for tracking document lifecycles
|
||||
- Automatic cleanup of stale documents
|
||||
- Snapshot functionality with custom processors
|
||||
- Iterator-based document access
|
||||
- Fast-forward mode for incremental processing
|
||||
|
||||
### FastPush
|
||||
|
||||
High-performance bulk operations:
|
||||
|
||||
- Automatic batching for optimal performance
|
||||
- Index management (create, delete, clear)
|
||||
- Dynamic mapping support
|
||||
- Efficient bulk API usage
|
||||
|
||||
### KVStore
|
||||
|
||||
Simple key-value interface:
|
||||
|
||||
- Elasticsearch-backed storage
|
||||
- Async/await API
|
||||
- Automatic index initialization
|
||||
- Clear and get operations
|
||||
|
||||
## Advanced Usage 🎓
|
||||
|
||||
### Index Rotation and Retention
|
||||
|
||||
```typescript
|
||||
const logger = new ElsSmartlogDestination({
|
||||
indexPrefix: 'myapp',
|
||||
indexRetention: 30, // Keep 30 days of logs
|
||||
node: 'http://localhost:9200',
|
||||
});
|
||||
|
||||
// Indices are automatically created as: myapp-2025-01-22
|
||||
// Old indices are automatically deleted after 30 days
|
||||
```
|
||||
|
||||
### Document Iteration
|
||||
|
||||
```typescript
|
||||
// Iterate over all documents in an index
|
||||
const iterator = docManager.getDocumentIterator();
|
||||
for await (const doc of iterator) {
|
||||
console.log(doc);
|
||||
// Iterate over all documents
|
||||
for await (const doc of products.iterate()) {
|
||||
console.log(doc._source.name);
|
||||
}
|
||||
|
||||
// Only process new documents since last run
|
||||
docManager.fastForward = true;
|
||||
await docManager.startPipingSession({ onlyNew: true });
|
||||
```
|
||||
|
||||
### Custom Snapshots
|
||||
|
||||
```typescript
|
||||
await docManager.takeSnapshot(async (iterator, prevSnapshot) => {
|
||||
let totalValue = 0;
|
||||
let count = 0;
|
||||
|
||||
// Create analytics snapshot
|
||||
const snapshot = await products.snapshot(async (iterator, previous) => {
|
||||
let total = 0, count = 0;
|
||||
for await (const doc of iterator) {
|
||||
totalValue += doc._source.price;
|
||||
total += doc._source.price;
|
||||
count++;
|
||||
}
|
||||
|
||||
return {
|
||||
date: new Date().toISOString(),
|
||||
aggregationData: {
|
||||
totalValue,
|
||||
averagePrice: totalValue / count,
|
||||
count,
|
||||
previousSnapshot: prevSnapshot,
|
||||
},
|
||||
};
|
||||
return { averagePrice: total / count, productCount: count };
|
||||
});
|
||||
```
|
||||
|
||||
## API Compatibility 🔄
|
||||
|
||||
This module is built on top of `@elastic/elasticsearch` v9.x and is compatible with:
|
||||
|
||||
- Elasticsearch 8.x and 9.x clusters
|
||||
- Kibana 8.x and 9.x for log visualization
|
||||
- OpenSearch (with some limitations)
|
||||
|
||||
## TypeScript Support 💙
|
||||
|
||||
Full TypeScript support with comprehensive type definitions:
|
||||
### Query Builder
|
||||
|
||||
```typescript
|
||||
import type {
|
||||
IElasticDocConstructorOptions,
|
||||
ISnapshot,
|
||||
SnapshotProcessor,
|
||||
} from '@apiclient.xyz/elasticsearch';
|
||||
import { createQuery } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
// Fluent query building
|
||||
const results = await createQuery<Product>('products')
|
||||
.match('name', 'widget', { fuzziness: 'AUTO' })
|
||||
.range('price', { gte: 10, lte: 100 })
|
||||
.term('inStock', true)
|
||||
.sort('price', 'asc')
|
||||
.size(20)
|
||||
.from(0)
|
||||
.highlight({ fields: { name: {} } })
|
||||
.aggregations(agg => agg
|
||||
.terms('categories', 'category')
|
||||
.avg('avgPrice', 'price')
|
||||
)
|
||||
.execute();
|
||||
|
||||
console.log(`Found ${results.hits.total} products`);
|
||||
console.log('Categories:', results.aggregations?.categories);
|
||||
```
|
||||
|
||||
## Performance Considerations ⚡
|
||||
### Bulk Operations
|
||||
|
||||
- **Bulk Operations**: FastPush uses 1000-document batches by default
|
||||
- **Connection Pooling**: Reuses Elasticsearch client connections
|
||||
- **Index Management**: Automatic index creation and deletion
|
||||
- **Data Streams**: Built-in support for efficient log ingestion
|
||||
```typescript
|
||||
import { createBulkIndexer } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
## Best Practices 💡
|
||||
const indexer = createBulkIndexer({
|
||||
flushThreshold: 1000,
|
||||
flushIntervalMs: 5000,
|
||||
maxConcurrent: 3,
|
||||
enableBackpressure: true,
|
||||
onProgress: (progress) => {
|
||||
console.log(`Progress: ${progress.processed}/${progress.total} (${progress.successRate}%)`);
|
||||
},
|
||||
});
|
||||
|
||||
1. **Always use authentication** in production environments
|
||||
2. **Set appropriate retention policies** to manage storage costs
|
||||
3. **Use piping sessions** to automatically clean up stale documents
|
||||
4. **Leverage snapshots** for point-in-time analytics
|
||||
5. **Configure index templates** for consistent mappings
|
||||
await indexer.initialize();
|
||||
|
||||
// Queue operations
|
||||
for (const item of largeDataset) {
|
||||
await indexer.index('products', item.id, item);
|
||||
}
|
||||
|
||||
// Wait for completion
|
||||
const stats = await indexer.flush();
|
||||
console.log(`Indexed ${stats.successful} documents, ${stats.failed} failures`);
|
||||
```
|
||||
|
||||
### Key-Value Store
|
||||
|
||||
```typescript
|
||||
import { createKVStore } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const kv = createKVStore<string>({
|
||||
index: 'app-config',
|
||||
enableCache: true,
|
||||
cacheMaxSize: 10000,
|
||||
defaultTTL: 3600, // 1 hour
|
||||
enableCompression: true,
|
||||
});
|
||||
|
||||
await kv.initialize();
|
||||
|
||||
// Basic operations
|
||||
await kv.set('api-key', 'sk-secret-key', { ttl: 86400 });
|
||||
const key = await kv.get('api-key');
|
||||
|
||||
// Batch operations
|
||||
await kv.mset([
|
||||
{ key: 'config:a', value: 'value-a' },
|
||||
{ key: 'config:b', value: 'value-b' },
|
||||
]);
|
||||
|
||||
const values = await kv.mget(['config:a', 'config:b']);
|
||||
|
||||
// Scan with pattern matching
|
||||
const scan = await kv.scan({ pattern: 'config:*', limit: 100 });
|
||||
```
|
||||
|
||||
### Transactions
|
||||
|
||||
```typescript
|
||||
import { createTransactionManager } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const txManager = createTransactionManager({
|
||||
defaultIsolationLevel: 'read_committed',
|
||||
defaultLockingStrategy: 'optimistic',
|
||||
conflictResolution: 'retry',
|
||||
maxConcurrentTransactions: 100,
|
||||
});
|
||||
|
||||
await txManager.initialize();
|
||||
|
||||
// ACID-like transaction
|
||||
const tx = await txManager.begin({ autoRollback: true });
|
||||
|
||||
try {
|
||||
const account1 = await tx.read('accounts', 'acc-001');
|
||||
const account2 = await tx.read('accounts', 'acc-002');
|
||||
|
||||
await tx.update('accounts', 'acc-001', { balance: account1.balance - 100 });
|
||||
await tx.update('accounts', 'acc-002', { balance: account2.balance + 100 });
|
||||
|
||||
tx.savepoint('after-transfer');
|
||||
|
||||
await tx.commit();
|
||||
} catch (error) {
|
||||
await tx.rollback();
|
||||
}
|
||||
```
|
||||
|
||||
### Logging Destination
|
||||
|
||||
```typescript
|
||||
import { createLogDestination, chainEnrichers, addHostInfo, addTimestamp } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const logger = createLogDestination({
|
||||
indexPrefix: 'app-logs',
|
||||
dataStream: true,
|
||||
ilmPolicy: {
|
||||
name: 'logs-policy',
|
||||
phases: {
|
||||
hot: { maxAge: '7d', maxSize: '50gb' },
|
||||
warm: { minAge: '7d' },
|
||||
delete: { minAge: '30d' },
|
||||
},
|
||||
},
|
||||
enricher: chainEnrichers(addHostInfo(), addTimestamp()),
|
||||
sampling: { strategy: 'rate', rate: 0.1 }, // 10% sampling
|
||||
});
|
||||
|
||||
await logger.initialize();
|
||||
|
||||
await logger.log({
|
||||
level: 'info',
|
||||
message: 'User authenticated',
|
||||
context: { userId: '12345', service: 'auth' },
|
||||
});
|
||||
|
||||
// Batch logging
|
||||
await logger.logBatch([
|
||||
{ level: 'debug', message: 'Request received' },
|
||||
{ level: 'info', message: 'Processing complete' },
|
||||
]);
|
||||
```
|
||||
|
||||
### Schema Management
|
||||
|
||||
```typescript
|
||||
import { createSchemaManager } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const schema = createSchemaManager({ enableMigrationHistory: true });
|
||||
|
||||
await schema.initialize();
|
||||
|
||||
// Create index with schema
|
||||
await schema.createIndex('products', {
|
||||
mappings: {
|
||||
properties: {
|
||||
name: { type: 'text', analyzer: 'standard' },
|
||||
price: { type: 'float' },
|
||||
category: { type: 'keyword' },
|
||||
createdAt: { type: 'date' },
|
||||
},
|
||||
},
|
||||
settings: {
|
||||
numberOfShards: 3,
|
||||
numberOfReplicas: 1,
|
||||
},
|
||||
});
|
||||
|
||||
// Run migrations
|
||||
await schema.migrate('products', [
|
||||
{
|
||||
version: '1.0.1',
|
||||
description: 'Add tags field',
|
||||
up: async (client, index) => {
|
||||
await client.indices.putMapping({
|
||||
index,
|
||||
properties: { tags: { type: 'keyword' } },
|
||||
});
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
// Create index template
|
||||
await schema.createIndexTemplate('products-template', {
|
||||
indexPatterns: ['products-*'],
|
||||
mappings: { /* ... */ },
|
||||
});
|
||||
```
|
||||
|
||||
### Plugin System
|
||||
|
||||
```typescript
|
||||
import {
|
||||
createPluginManager,
|
||||
createRetryPlugin,
|
||||
createCachePlugin,
|
||||
createRateLimitPlugin,
|
||||
} from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
const plugins = createPluginManager({ enableMetrics: true });
|
||||
|
||||
// Built-in plugins
|
||||
plugins.register(createRetryPlugin({ maxRetries: 3, backoffMs: 1000 }));
|
||||
plugins.register(createCachePlugin({ maxSize: 1000, ttlMs: 60000 }));
|
||||
plugins.register(createRateLimitPlugin({ maxRequests: 100, windowMs: 1000 }));
|
||||
|
||||
// Custom plugin
|
||||
plugins.register({
|
||||
name: 'custom-logger',
|
||||
version: '1.0.0',
|
||||
hooks: {
|
||||
beforeRequest: async (ctx) => {
|
||||
console.log(`Request: ${ctx.operation} on ${ctx.index}`);
|
||||
},
|
||||
afterResponse: async (ctx, response) => {
|
||||
console.log(`Response: ${response.statusCode}`);
|
||||
},
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
@apiclient.xyz/elasticsearch
|
||||
├── core/
|
||||
│ ├── config/ # Fluent configuration builder
|
||||
│ ├── connection/ # Connection manager, circuit breaker, health checks
|
||||
│ ├── errors/ # Typed errors and retry policies
|
||||
│ ├── observability/ # Logger, metrics (Prometheus), tracing (OpenTelemetry)
|
||||
│ └── plugins/ # Plugin manager and built-in plugins
|
||||
└── domain/
|
||||
├── documents/ # DocumentManager, sessions, snapshots
|
||||
├── query/ # QueryBuilder, AggregationBuilder
|
||||
├── bulk/ # BulkIndexer with backpressure
|
||||
├── kv/ # Distributed KV store
|
||||
├── transactions/ # ACID-like transaction support
|
||||
├── logging/ # Log destination with ILM
|
||||
└── schema/ # Schema management and migrations
|
||||
```
|
||||
|
||||
## 📊 Observability
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
```typescript
|
||||
import { defaultMetricsCollector } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
// Export metrics in Prometheus format
|
||||
const metrics = defaultMetricsCollector.export();
|
||||
|
||||
// Available metrics:
|
||||
// - elasticsearch_requests_total{operation, index, status}
|
||||
// - elasticsearch_request_duration_seconds{operation, index}
|
||||
// - elasticsearch_errors_total{operation, index, error_type}
|
||||
// - elasticsearch_circuit_breaker_state{state}
|
||||
// - elasticsearch_connection_pool_size
|
||||
// - elasticsearch_bulk_operations_total{type, status}
|
||||
```
|
||||
|
||||
### Distributed Tracing
|
||||
|
||||
```typescript
|
||||
import { defaultTracingProvider } from '@apiclient.xyz/elasticsearch';
|
||||
|
||||
// OpenTelemetry-compatible tracing
|
||||
const span = defaultTracingProvider.startSpan('custom-operation');
|
||||
span.setAttributes({ 'custom.attribute': 'value' });
|
||||
// ... operation
|
||||
span.end();
|
||||
```
|
||||
|
||||
## 🔒 Authentication Methods
|
||||
|
||||
```typescript
|
||||
// Basic auth
|
||||
createConfig().basicAuth('username', 'password')
|
||||
|
||||
// API key
|
||||
createConfig().apiKeyAuth('api-key-value')
|
||||
|
||||
// Bearer token
|
||||
createConfig().bearerAuth('jwt-token')
|
||||
|
||||
// Elastic Cloud
|
||||
createConfig().cloudAuth('cloud-id', { apiKey: 'key' })
|
||||
|
||||
// Certificate
|
||||
createConfig().auth({
|
||||
type: 'certificate',
|
||||
certPath: '/path/to/cert.pem',
|
||||
keyPath: '/path/to/key.pem',
|
||||
})
|
||||
```
|
||||
|
||||
## ⚡ Performance Tips
|
||||
|
||||
1. **Use bulk operations** for batch inserts — `BulkIndexer` handles batching, retries, and backpressure
|
||||
2. **Enable compression** for large payloads — reduces network overhead
|
||||
3. **Configure connection pooling** — `poolSize(max, min)` for optimal concurrency
|
||||
4. **Leverage caching** — `KVStore` has built-in LRU/LFU caching
|
||||
5. **Use sessions** for document sync — automatic cleanup of stale documents
|
||||
6. **Enable circuit breaker** — prevents cascade failures during outages
|
||||
|
||||
## 🔄 Compatibility
|
||||
|
||||
- **Elasticsearch**: 8.x, 9.x
|
||||
- **Kibana**: 8.x, 9.x (for log visualization)
|
||||
- **Node.js**: 18+
|
||||
- **TypeScript**: 5.0+
|
||||
|
||||
## License and Legal Information
|
||||
|
||||
|
||||
@@ -1,108 +0,0 @@
|
||||
# Quick Fixes Needed for TypeScript Strict Mode
|
||||
|
||||
## Import Fixes (Use `import type` for verbatimModuleSyntax)
|
||||
|
||||
### Files to fix:
|
||||
|
||||
1. **ts/core/connection/connection-manager.ts**
|
||||
```typescript
|
||||
// Change:
|
||||
import { ElasticsearchConfig } from '../config/types.js';
|
||||
import { HealthCheckResult, HealthStatus } from './health-check.js';
|
||||
|
||||
// To:
|
||||
import type { ElasticsearchConfig } from '../config/types.js';
|
||||
import type { HealthCheckResult } from './health-check.js';
|
||||
import { HealthStatus } from './health-check.js';
|
||||
```
|
||||
|
||||
2. **ts/core/errors/elasticsearch-error.ts**
|
||||
```typescript
|
||||
// Change:
|
||||
import { ErrorCode, ErrorContext } from './types.js';
|
||||
|
||||
// To:
|
||||
import { ErrorCode } from './types.js';
|
||||
import type { ErrorContext } from './types.js';
|
||||
```
|
||||
|
||||
3. **ts/core/errors/retry-policy.ts**
|
||||
```typescript
|
||||
// Change:
|
||||
import { RetryConfig, RetryStrategy } from './types.js';
|
||||
|
||||
// To:
|
||||
import type { RetryConfig, RetryStrategy } from './types.js';
|
||||
```
|
||||
|
||||
4. **ts/domain/documents/document-manager.ts**
|
||||
```typescript
|
||||
// Change:
|
||||
import {
|
||||
DocumentWithMeta,
|
||||
SessionConfig,
|
||||
SnapshotProcessor,
|
||||
SnapshotMeta,
|
||||
IteratorOptions,
|
||||
} from './types.js';
|
||||
|
||||
// To:
|
||||
import type {
|
||||
DocumentWithMeta,
|
||||
SessionConfig,
|
||||
SnapshotProcessor,
|
||||
SnapshotMeta,
|
||||
IteratorOptions,
|
||||
} from './types.js';
|
||||
```
|
||||
|
||||
## Tracing undefined issue (ts/core/observability/tracing.ts:315-317)
|
||||
|
||||
```typescript
|
||||
// In TracingProvider.createSpan(), change:
|
||||
const span = this.tracer.startSpan(name, {
|
||||
...attributes,
|
||||
'service.name': this.config.serviceName,
|
||||
...(this.config.serviceVersion && { 'service.version': this.config.serviceVersion }),
|
||||
});
|
||||
|
||||
// To:
|
||||
const spanAttributes = {
|
||||
...attributes,
|
||||
'service.name': this.config.serviceName || 'elasticsearch-client',
|
||||
};
|
||||
if (this.config.serviceVersion) {
|
||||
spanAttributes['service.version'] = this.config.serviceVersion;
|
||||
}
|
||||
const span = this.tracer.startSpan(name, spanAttributes);
|
||||
```
|
||||
|
||||
## Generic Type Constraints for Elasticsearch Client
|
||||
|
||||
In **ts/domain/documents/document-manager.ts**, add constraint:
|
||||
|
||||
```typescript
|
||||
// Change class definition:
|
||||
export class DocumentManager<T = unknown> {
|
||||
|
||||
// To:
|
||||
export class DocumentManager<T extends Record<string, any> = Record<string, any>> {
|
||||
```
|
||||
|
||||
This ensures T is always an object type compatible with Elasticsearch operations.
|
||||
|
||||
## Alternative: Relax Strict Mode Temporarily
|
||||
|
||||
If immediate fixes are needed, you can temporarily relax some strict checks in tsconfig.json:
|
||||
|
||||
```json
|
||||
{
|
||||
"compilerOptions": {
|
||||
// Comment out temporarily:
|
||||
// "verbatimModuleSyntax": true,
|
||||
// "noUncheckedIndexedAccess": true,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
But the proper fix is to address the imports and type issues as outlined above.
|
||||
386
ts/README.md
386
ts/README.md
@@ -1,386 +0,0 @@
|
||||
# Enterprise Elasticsearch Client v3.0 (NEW Architecture)
|
||||
|
||||
> 🚧 **Status**: Phase 1 & Core Phase 2 Complete | 70% Implementation Complete
|
||||
|
||||
**Modern, type-safe, production-ready Elasticsearch client** with enterprise features built-in from the ground up.
|
||||
|
||||
## 🎯 What's New in v3.0
|
||||
|
||||
### Core Infrastructure
|
||||
- ✅ **Connection Manager** - Singleton with pooling, health checks, circuit breaker
|
||||
- ✅ **Configuration System** - Environment variables, files, secrets, validation
|
||||
- ✅ **Error Handling** - Typed error hierarchy with retry policies
|
||||
- ✅ **Observability** - Structured logging, Prometheus metrics, distributed tracing
|
||||
- ✅ **Circuit Breaker** - Prevent cascading failures
|
||||
- ✅ **Health Monitoring** - Automatic cluster health checks
|
||||
|
||||
### Domain APIs
|
||||
- ✅ **Document Manager** - Fluent API for CRUD operations
|
||||
- ✅ **Session Management** - Batch operations with automatic cleanup
|
||||
- ✅ **Snapshot System** - Point-in-time analytics
|
||||
- ⏳ **Query Builder** - Type-safe query DSL (Coming soon)
|
||||
- ⏳ **Bulk Indexer** - Adaptive batching, parallel workers (Coming soon)
|
||||
- ⏳ **KV Store** - TTL, caching, batch ops (Coming soon)
|
||||
- ⏳ **Logging API** - Kibana integration, enrichment (Coming soon)
|
||||
|
||||
### Advanced Features
|
||||
- ⏳ **Plugin System** - Extensible with middleware
|
||||
- ⏳ **Transactions** - Optimistic locking, rollback
|
||||
- ⏳ **Schema Management** - Type-safe schemas, migrations
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pnpm install
|
||||
|
||||
# Build the new implementation
|
||||
npx tsc --project tsconfig.json
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import {
|
||||
createConfig,
|
||||
ElasticsearchConnectionManager,
|
||||
DocumentManager,
|
||||
LogLevel,
|
||||
} from './ts';
|
||||
|
||||
// 1. Configure
|
||||
const config = createConfig()
|
||||
.fromEnv() // Load from ELASTICSEARCH_URL, etc.
|
||||
.nodes('http://localhost:9200')
|
||||
.basicAuth('elastic', 'changeme')
|
||||
.timeout(30000)
|
||||
.retries(3)
|
||||
.logLevel(LogLevel.INFO)
|
||||
.enableMetrics()
|
||||
.enableTracing()
|
||||
.build();
|
||||
|
||||
// 2. Initialize connection
|
||||
const manager = ElasticsearchConnectionManager.getInstance(config);
|
||||
await manager.initialize();
|
||||
|
||||
// 3. Create document manager
|
||||
const docs = new DocumentManager<Product>({
|
||||
index: 'products',
|
||||
autoCreateIndex: true,
|
||||
});
|
||||
await docs.initialize();
|
||||
|
||||
// 4. Use fluent API
|
||||
await docs.upsert('prod-1', {
|
||||
name: 'Widget',
|
||||
price: 99.99,
|
||||
inStock: true,
|
||||
});
|
||||
|
||||
// 5. Session-based batch operations
|
||||
await docs
|
||||
.session()
|
||||
.start()
|
||||
.upsert('prod-2', { name: 'Gadget', price: 149.99, inStock: true })
|
||||
.upsert('prod-3', { name: 'Tool', price: 49.99, inStock: false })
|
||||
.commit();
|
||||
|
||||
// 6. Iterate over documents
|
||||
for await (const doc of docs.iterate()) {
|
||||
console.log(doc._source);
|
||||
}
|
||||
|
||||
// 7. Create snapshots
|
||||
const snapshot = await docs.snapshot(async (iterator) => {
|
||||
const items = [];
|
||||
for await (const doc of iterator) {
|
||||
items.push(doc._source);
|
||||
}
|
||||
return { count: items.length, items };
|
||||
});
|
||||
```
|
||||
|
||||
## 📚 Complete Example
|
||||
|
||||
See [`examples/basic/complete-example.ts`](./examples/basic/complete-example.ts) for a comprehensive demonstration including:
|
||||
- Configuration from environment
|
||||
- Connection management with health checks
|
||||
- Individual and batch operations
|
||||
- Document iteration
|
||||
- Snapshot analytics
|
||||
- Metrics and observability
|
||||
- Error handling
|
||||
|
||||
Run it with:
|
||||
|
||||
```bash
|
||||
npx tsx ts/examples/basic/complete-example.ts
|
||||
```
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
ts/
|
||||
├── core/ # Foundation layer
|
||||
│ ├── config/ # Configuration management ✅
|
||||
│ ├── connection/ # Connection pooling, health ✅
|
||||
│ ├── errors/ # Error hierarchy, retry ✅
|
||||
│ └── observability/ # Logging, metrics, tracing ✅
|
||||
├── domain/ # Business logic layer
|
||||
│ ├── documents/ # Document API ✅
|
||||
│ ├── query/ # Query builder ⏳
|
||||
│ ├── logging/ # Log destination ⏳
|
||||
│ ├── bulk/ # Bulk indexer ⏳
|
||||
│ └── kv/ # Key-value store ⏳
|
||||
├── plugins/ # Extension points ⏳
|
||||
├── testing/ # Test utilities ⏳
|
||||
└── examples/ # Usage examples ✅
|
||||
```
|
||||
|
||||
## ⚡ Key Improvements Over v2.x
|
||||
|
||||
| Feature | v2.x | v3.0 |
|
||||
|---------|------|------|
|
||||
| Connection Pooling | ❌ Each class creates own client | ✅ Singleton connection manager |
|
||||
| Health Checks | ❌ None | ✅ Automatic periodic checks |
|
||||
| Circuit Breaker | ❌ None | ✅ Fault tolerance built-in |
|
||||
| Error Handling | ⚠️ Inconsistent | ✅ Typed error hierarchy |
|
||||
| Retry Logic | ⚠️ Basic scheduler | ✅ Exponential backoff, jitter |
|
||||
| Configuration | ⚠️ Constructor only | ✅ Env vars, files, secrets |
|
||||
| Logging | ⚠️ console.log scattered | ✅ Structured logging with context |
|
||||
| Metrics | ❌ None | ✅ Prometheus-compatible |
|
||||
| Tracing | ❌ None | ✅ OpenTelemetry-compatible |
|
||||
| Type Safety | ⚠️ Partial, uses `any` | ✅ Strict TypeScript, no `any` |
|
||||
| API Design | ⚠️ Inconsistent constructors | ✅ Fluent, discoverable |
|
||||
| Bulk Operations | ⚠️ Sequential, inefficient | ✅ Batched with error handling |
|
||||
| Document Cleanup | ⚠️ O(n) scroll all docs | ✅ deleteByQuery (efficient) |
|
||||
| Observability | ❌ None | ✅ Full observability stack |
|
||||
|
||||
## 📖 API Documentation
|
||||
|
||||
### Configuration
|
||||
|
||||
```typescript
|
||||
import { createConfig, LogLevel } from './ts';
|
||||
|
||||
const config = createConfig()
|
||||
// Data sources
|
||||
.fromEnv() // Load from environment variables
|
||||
.fromFile('config.json') // Load from JSON file
|
||||
.fromObject({ ... }) // Load from object
|
||||
|
||||
// Connection
|
||||
.nodes(['http://es1:9200', 'http://es2:9200'])
|
||||
.auth({ type: 'basic', username: 'user', password: 'pass' })
|
||||
.apiKeyAuth('api-key')
|
||||
.timeout(30000)
|
||||
.retries(3)
|
||||
.compression(true)
|
||||
.poolSize(10, 2) // max, min idle
|
||||
|
||||
// Discovery
|
||||
.discovery(true, { interval: 60000 })
|
||||
|
||||
// Observability
|
||||
.logLevel(LogLevel.INFO)
|
||||
.enableRequestLogging(true)
|
||||
.enableMetrics(true, 'my_app')
|
||||
.enableTracing(true, { serviceName: 'api', serviceVersion: '1.0.0' })
|
||||
|
||||
// Secrets
|
||||
.withSecrets(secretProvider)
|
||||
|
||||
.build();
|
||||
```
|
||||
|
||||
### Connection Management
|
||||
|
||||
```typescript
|
||||
import { ElasticsearchConnectionManager } from './ts';
|
||||
|
||||
const manager = ElasticsearchConnectionManager.getInstance(config);
|
||||
await manager.initialize();
|
||||
|
||||
// Health check
|
||||
const health = await manager.healthCheck();
|
||||
console.log(health.status, health.clusterHealth, health.activeNodes);
|
||||
|
||||
// Circuit breaker
|
||||
const result = await manager.execute(async () => {
|
||||
return await someOperation();
|
||||
});
|
||||
|
||||
// Stats
|
||||
const stats = manager.getStats();
|
||||
console.log(stats.healthStatus, stats.circuitState);
|
||||
|
||||
// Cleanup
|
||||
await manager.destroy();
|
||||
```
|
||||
|
||||
### Document Operations
|
||||
|
||||
```typescript
|
||||
import { DocumentManager } from './ts';
|
||||
|
||||
const docs = new DocumentManager<MyType>({ index: 'my-index', autoCreateIndex: true });
|
||||
await docs.initialize();
|
||||
|
||||
// CRUD
|
||||
await docs.create('id', doc);
|
||||
await docs.update('id', { field: 'value' });
|
||||
await docs.upsert('id', doc);
|
||||
await docs.delete('id');
|
||||
const doc = await docs.get('id');
|
||||
|
||||
// Optimistic locking
|
||||
await docs.update('id', doc, { seqNo: 123, primaryTerm: 1 });
|
||||
|
||||
// Batch operations
|
||||
const result = await docs
|
||||
.session({ cleanupStale: true })
|
||||
.start()
|
||||
.upsert('id1', doc1)
|
||||
.upsert('id2', doc2)
|
||||
.delete('id3')
|
||||
.commit();
|
||||
|
||||
// Iteration
|
||||
for await (const doc of docs.iterate({ batchSize: 500 })) {
|
||||
console.log(doc._source);
|
||||
}
|
||||
|
||||
// Snapshots
|
||||
const snapshot = await docs.snapshot(async (iterator, prev) => {
|
||||
// Custom analytics
|
||||
return computedData;
|
||||
});
|
||||
|
||||
// Utilities
|
||||
const count = await docs.count();
|
||||
const exists = await docs.exists();
|
||||
await docs.deleteIndex();
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```typescript
|
||||
import {
|
||||
ElasticsearchError,
|
||||
ConnectionError,
|
||||
DocumentNotFoundError,
|
||||
BulkOperationError,
|
||||
ErrorCode,
|
||||
} from './ts';
|
||||
|
||||
try {
|
||||
await docs.get('id');
|
||||
} catch (error) {
|
||||
if (error instanceof DocumentNotFoundError) {
|
||||
// Handle not found
|
||||
} else if (error instanceof ConnectionError) {
|
||||
// Handle connection error
|
||||
} else if (error instanceof ElasticsearchError) {
|
||||
console.log(error.code, error.retryable, error.context);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Observability
|
||||
|
||||
```typescript
|
||||
import { defaultLogger, defaultMetricsCollector, defaultTracingProvider } from './ts';
|
||||
|
||||
// Logging
|
||||
const logger = defaultLogger.child('my-component');
|
||||
logger.info('Message', { key: 'value' });
|
||||
logger.error('Error', error, { context: 'data' });
|
||||
|
||||
// Correlation
|
||||
const correlatedLogger = logger.withCorrelation(requestId);
|
||||
|
||||
// Metrics
|
||||
defaultMetricsCollector.requestsTotal.inc({ operation: 'search', index: 'products' });
|
||||
defaultMetricsCollector.requestDuration.observe(0.234, { operation: 'search' });
|
||||
|
||||
// Export metrics
|
||||
const prometheus = defaultMetricsCollector.export();
|
||||
|
||||
// Tracing
|
||||
await defaultTracingProvider.withSpan('operation', async (span) => {
|
||||
span.setAttribute('key', 'value');
|
||||
return await doWork();
|
||||
});
|
||||
```
|
||||
|
||||
## 🔒 Security
|
||||
|
||||
- ✅ Support for basic, API key, bearer token, cloud ID authentication
|
||||
- ✅ TLS/SSL configuration
|
||||
- ✅ Secret provider integration (environment, AWS Secrets Manager, Vault, etc.)
|
||||
- ✅ Credential validation
|
||||
- ✅ No credentials in logs or error messages
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
```bash
|
||||
# Run tests (when implemented)
|
||||
pnpm test
|
||||
|
||||
# Type check
|
||||
npx tsc --project tsconfig.json --noEmit
|
||||
|
||||
# Lint
|
||||
npx eslint ts/**/*.ts
|
||||
```
|
||||
|
||||
## 📊 Performance
|
||||
|
||||
- ✅ Connection pooling reduces overhead
|
||||
- ✅ Batch operations use bulk API
|
||||
- ✅ deleteByQuery for efficient cleanup (vs old scroll approach)
|
||||
- ✅ Point-in-Time API for iteration (vs scroll)
|
||||
- ✅ Circuit breaker prevents wasted requests
|
||||
- ⏳ Adaptive batching (coming soon)
|
||||
- ⏳ Parallel bulk workers (coming soon)
|
||||
|
||||
## 🗺️ Roadmap
|
||||
|
||||
### Phase 2 Remaining (In Progress)
|
||||
- [ ] Type-safe Query Builder
|
||||
- [ ] Enhanced Logging API with Kibana integration
|
||||
- [ ] Bulk Indexer with adaptive batching
|
||||
- [ ] KV Store with TTL and caching
|
||||
|
||||
### Phase 3 (Planned)
|
||||
- [ ] Plugin architecture with middleware
|
||||
- [ ] Transaction support with optimistic locking
|
||||
- [ ] Schema management and migrations
|
||||
|
||||
### Phase 4 (Planned)
|
||||
- [ ] Comprehensive test suite (unit, integration, chaos)
|
||||
- [ ] Migration guide from v2.x to v3.0
|
||||
- [ ] Performance benchmarks
|
||||
- [ ] Full API documentation
|
||||
|
||||
## 📄 License and Legal Information
|
||||
|
||||
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](../license) file within this repository.
|
||||
|
||||
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
|
||||
|
||||
### Trademarks
|
||||
|
||||
This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.
|
||||
|
||||
### Company Information
|
||||
|
||||
Task Venture Capital GmbH
|
||||
Registered at District court Bremen HRB 35230 HB, Germany
|
||||
|
||||
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
|
||||
|
||||
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
|
||||
Reference in New Issue
Block a user