✅ Connection Manager - Singleton with pooling, health checks, circuit breaker
✅ Configuration System - Environment variables, files, secrets, validation
✅ Error Handling - Typed error hierarchy with retry policies
✅ Observability - Structured logging, Prometheus metrics, distributed tracing
✅ Circuit Breaker - Prevent cascading failures
✅ Health Monitoring - Automatic cluster health checks

Domain APIs

✅ Document Manager - Fluent API for CRUD operations
✅ Session Management - Batch operations with automatic cleanup
✅ Snapshot System - Point-in-time analytics
⏳ Query Builder - Type-safe query DSL (Coming soon)
⏳ Bulk Indexer - Adaptive batching, parallel workers (Coming soon)
⏳ KV Store - TTL, caching, batch ops (Coming soon)
⏳ Logging API - Kibana integration, enrichment (Coming soon)

Advanced Features

⏳ Plugin System - Extensible with middleware
⏳ Transactions - Optimistic locking, rollback
⏳ Schema Management - Type-safe schemas, migrations

🚀 Quick Start

Installation

# Install dependencies
pnpm install

# Build the new implementation
npx tsc --project tsconfig.json

Basic Usage

import {
  createConfig,
  ElasticsearchConnectionManager,
  DocumentManager,
  LogLevel,
} from './ts';

// 1. Configure
const config = createConfig()
  .fromEnv() // Load from ELASTICSEARCH_URL, etc.
  .nodes('http://localhost:9200')
  .basicAuth('elastic', 'changeme')
  .timeout(30000)
  .retries(3)
  .logLevel(LogLevel.INFO)
  .enableMetrics()
  .enableTracing()
  .build();

// 2. Initialize connection
const manager = ElasticsearchConnectionManager.getInstance(config);
await manager.initialize();

// 3. Create document manager
const docs = new DocumentManager<Product>({
  index: 'products',
  autoCreateIndex: true,
});
await docs.initialize();

// 4. Use fluent API
await docs.upsert('prod-1', {
  name: 'Widget',
  price: 99.99,
  inStock: true,
});

// 5. Session-based batch operations
await docs
  .session()
  .start()
  .upsert('prod-2', { name: 'Gadget', price: 149.99, inStock: true })
  .upsert('prod-3', { name: 'Tool', price: 49.99, inStock: false })
  .commit();

// 6. Iterate over documents
for await (const doc of docs.iterate()) {
  console.log(doc._source);
}

// 7. Create snapshots
const snapshot = await docs.snapshot(async (iterator) => {
  const items = [];
  for await (const doc of iterator) {
    items.push(doc._source);
  }
  return { count: items.length, items };
});

📚 Complete Example

See examples/basic/complete-example.ts for a comprehensive demonstration including:

Configuration from environment
Connection management with health checks
Individual and batch operations
Document iteration
Snapshot analytics
Metrics and observability
Error handling

Run it with:

npx tsx ts/examples/basic/complete-example.ts

🏗️ Architecture

ts/
├── core/                    # Foundation layer
│   ├── config/             # Configuration management ✅
│   ├── connection/         # Connection pooling, health ✅
│   ├── errors/             # Error hierarchy, retry ✅
│   └── observability/      # Logging, metrics, tracing ✅
├── domain/                  # Business logic layer
│   ├── documents/          # Document API ✅
│   ├── query/              # Query builder ⏳
│   ├── logging/            # Log destination ⏳
│   ├── bulk/               # Bulk indexer ⏳
│   └── kv/                 # Key-value store ⏳
├── plugins/                 # Extension points ⏳
├── testing/                 # Test utilities ⏳
└── examples/               # Usage examples ✅

⚡ Key Improvements Over v2.x

Feature	v2.x	v3.0
Connection Pooling	❌ Each class creates own client	✅ Singleton connection manager
Health Checks	❌ None	✅ Automatic periodic checks
Circuit Breaker	❌ None	✅ Fault tolerance built-in
Error Handling	⚠️ Inconsistent	✅ Typed error hierarchy
Retry Logic	⚠️ Basic scheduler	✅ Exponential backoff, jitter
Configuration	⚠️ Constructor only	✅ Env vars, files, secrets
Logging	⚠️ console.log scattered	✅ Structured logging with context
Metrics	❌ None	✅ Prometheus-compatible
Tracing	❌ None	✅ OpenTelemetry-compatible
Type Safety	⚠️ Partial, uses `any`	✅ Strict TypeScript, no `any`
API Design	⚠️ Inconsistent constructors	✅ Fluent, discoverable
Bulk Operations	⚠️ Sequential, inefficient	✅ Batched with error handling
Document Cleanup	⚠️ O(n) scroll all docs	✅ deleteByQuery (efficient)
Observability	❌ None	✅ Full observability stack

📖 API Documentation

Configuration

import { createConfig, LogLevel } from './ts';

const config = createConfig()
  // Data sources
  .fromEnv()                    // Load from environment variables
  .fromFile('config.json')       // Load from JSON file
  .fromObject({ ... })           // Load from object

  // Connection
  .nodes(['http://es1:9200', 'http://es2:9200'])
  .auth({ type: 'basic', username: 'user', password: 'pass' })
  .apiKeyAuth('api-key')
  .timeout(30000)
  .retries(3)
  .compression(true)
  .poolSize(10, 2)              // max, min idle

  // Discovery
  .discovery(true, { interval: 60000 })

  // Observability
  .logLevel(LogLevel.INFO)
  .enableRequestLogging(true)
  .enableMetrics(true, 'my_app')
  .enableTracing(true, { serviceName: 'api', serviceVersion: '1.0.0' })

  // Secrets
  .withSecrets(secretProvider)

  .build();

Connection Management

import { ElasticsearchConnectionManager } from './ts';

const manager = ElasticsearchConnectionManager.getInstance(config);
await manager.initialize();

// Health check
const health = await manager.healthCheck();
console.log(health.status, health.clusterHealth, health.activeNodes);

// Circuit breaker
const result = await manager.execute(async () => {
  return await someOperation();
});

// Stats
const stats = manager.getStats();
console.log(stats.healthStatus, stats.circuitState);

// Cleanup
await manager.destroy();

Document Operations

import { DocumentManager } from './ts';

const docs = new DocumentManager<MyType>({ index: 'my-index', autoCreateIndex: true });
await docs.initialize();

// CRUD
await docs.create('id', doc);
await docs.update('id', { field: 'value' });
await docs.upsert('id', doc);
await docs.delete('id');
const doc = await docs.get('id');

// Optimistic locking
await docs.update('id', doc, { seqNo: 123, primaryTerm: 1 });

// Batch operations
const result = await docs
  .session({ cleanupStale: true })
  .start()
  .upsert('id1', doc1)
  .upsert('id2', doc2)
  .delete('id3')
  .commit();

// Iteration
for await (const doc of docs.iterate({ batchSize: 500 })) {
  console.log(doc._source);
}

// Snapshots
const snapshot = await docs.snapshot(async (iterator, prev) => {
  // Custom analytics
  return computedData;
});

// Utilities
const count = await docs.count();
const exists = await docs.exists();
await docs.deleteIndex();

Error Handling

import {
  ElasticsearchError,
  ConnectionError,
  DocumentNotFoundError,
  BulkOperationError,
  ErrorCode,
} from './ts';

try {
  await docs.get('id');
} catch (error) {
  if (error instanceof DocumentNotFoundError) {
    // Handle not found
  } else if (error instanceof ConnectionError) {
    // Handle connection error
  } else if (error instanceof ElasticsearchError) {
    console.log(error.code, error.retryable, error.context);
  }
}

Observability

import { defaultLogger, defaultMetricsCollector, defaultTracingProvider } from './ts';

// Logging
const logger = defaultLogger.child('my-component');
logger.info('Message', { key: 'value' });
logger.error('Error', error, { context: 'data' });

// Correlation
const correlatedLogger = logger.withCorrelation(requestId);

// Metrics
defaultMetricsCollector.requestsTotal.inc({ operation: 'search', index: 'products' });
defaultMetricsCollector.requestDuration.observe(0.234, { operation: 'search' });

// Export metrics
const prometheus = defaultMetricsCollector.export();

// Tracing
await defaultTracingProvider.withSpan('operation', async (span) => {
  span.setAttribute('key', 'value');
  return await doWork();
});

🔒 Security

✅ Support for basic, API key, bearer token, cloud ID authentication
✅ TLS/SSL configuration
✅ Secret provider integration (environment, AWS Secrets Manager, Vault, etc.)
✅ Credential validation
✅ No credentials in logs or error messages

🧪 Testing

# Run tests (when implemented)
pnpm test

# Type check
npx tsc --project tsconfig.json --noEmit

# Lint
npx eslint ts/**/*.ts

📊 Performance

✅ Connection pooling reduces overhead
✅ Batch operations use bulk API
✅ deleteByQuery for efficient cleanup (vs old scroll approach)
✅ Point-in-Time API for iteration (vs scroll)
✅ Circuit breaker prevents wasted requests
⏳ Adaptive batching (coming soon)
⏳ Parallel bulk workers (coming soon)

🗺️ Roadmap

Phase 2 Remaining (In Progress)

Type-safe Query Builder
Enhanced Logging API with Kibana integration
Bulk Indexer with adaptive batching
KV Store with TTL and caching

Phase 3 (Planned)

Plugin architecture with middleware
Transaction support with optimistic locking
Schema management and migrations

Phase 4 (Planned)

Comprehensive test suite (unit, integration, chaos)
Migration guide from v2.x to v3.0
Performance benchmarks
Full API documentation

📄 License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

README.md

Enterprise Elasticsearch Client v3.0 (NEW Architecture)

🎯 What's New in v3.0

Core Infrastructure