A powerful, type-safe wrapper around the official Elasticsearch client that provides intelligent log management, document handling, key-value storage, and fast data ingestion - all optimized for production use.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Features ✨

🎯 SmartLog Destination - Full-featured logging destination compatible with Kibana, automatic index rotation, and retention management
📦 ElasticDoc - Advanced document management with piping sessions, snapshots, and automatic cleanup
🚀 FastPush - High-performance bulk document insertion with automatic index management
💾 KVStore - Simple key-value storage interface backed by Elasticsearch
🔧 TypeScript First - Complete type safety with full TypeScript support
🌊 Data Streams - Built-in support for Elasticsearch data streams
⚡ Production Ready - Designed for high-throughput production environments

Installation 📦

npm install @apiclient.xyz/elasticsearch
# or
pnpm install @apiclient.xyz/elasticsearch

Quick Start 🚀

SmartLog Destination

Perfect for application logging with automatic index rotation and Kibana compatibility:

import { ElsSmartlogDestination } from '@apiclient.xyz/elasticsearch';

const logger = new ElsSmartlogDestination({
  indexPrefix: 'app-logs',
  indexRetention: 7, // Keep logs for 7 days
  node: 'http://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'your-password',
  },
});

// Log messages that automatically appear in Kibana
await logger.log({
  timestamp: Date.now(),
  type: 'increment',
  level: 'info',
  context: {
    company: 'YourCompany',
    companyunit: 'api-service',
    containerName: 'web-server',
    environment: 'production',
    runtime: 'node',
    zone: 'us-east-1',
  },
  message: 'User authentication successful',
  correlation: null,
});

ElasticDoc - Document Management

Handle documents with advanced features like piping sessions and snapshots:

import { ElasticDoc } from '@apiclient.xyz/elasticsearch';

const docManager = new ElasticDoc({
  index: 'products',
  node: 'http://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'your-password',
  },
});

// Start a piping session to manage document lifecycle
await docManager.startPipingSession({});

// Add or update documents
await docManager.pipeDocument({
  docId: 'product-001',
  timestamp: new Date().toISOString(),
  doc: {
    name: 'Premium Widget',
    price: 99.99,
    inStock: true
  },
});

await docManager.pipeDocument({
  docId: 'product-002',
  timestamp: new Date().toISOString(),
  doc: {
    name: 'Deluxe Gadget',
    price: 149.99,
    inStock: false
  },
});

// End session - automatically removes documents not in this session
await docManager.endPipingSession();

// Take and store snapshots with custom aggregations
await docManager.takeSnapshot(async (iterator, prevSnapshot) => {
  const aggregationData = [];
  for await (const doc of iterator) {
    aggregationData.push(doc);
  }
  return {
    date: new Date().toISOString(),
    aggregationData,
  };
});

FastPush - Bulk Data Ingestion

Efficiently push large datasets with automatic index management:

import { FastPush } from '@apiclient.xyz/elasticsearch';

const fastPush = new FastPush({
  node: 'http://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'your-password',
  },
});

const documents = [
  { id: 1, name: 'Document 1', data: 'Some data' },
  { id: 2, name: 'Document 2', data: 'More data' },
  // ... thousands more documents
];

// Push all documents with automatic batching
await fastPush.pushDocuments('bulk-data', documents, {
  deleteOldData: true, // Clear old data before inserting
});

KVStore - Key-Value Storage

Simple key-value storage backed by the power of Elasticsearch:

import { KVStore } from '@apiclient.xyz/elasticsearch';

const kvStore = new KVStore({
  index: 'app-config',
  node: 'http://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'your-password',
  },
});

// Set values
await kvStore.set('api-key', 'sk-1234567890');
await kvStore.set('feature-flags', JSON.stringify({ newUI: true }));

// Get values
const apiKey = await kvStore.get('api-key');
console.log(apiKey); // 'sk-1234567890'

// Clear all data
await kvStore.clear();

Core Classes 🏗️

ElsSmartlogDestination

The main logging destination class that provides:

Automatic index rotation based on date
Configurable retention policies
Kibana-compatible log format
Data stream support
Built-in scheduler for maintenance tasks

ElasticDoc

Advanced document management with:

Piping sessions for tracking document lifecycles
Automatic cleanup of stale documents
Snapshot functionality with custom processors
Iterator-based document access
Fast-forward mode for incremental processing

FastPush

High-performance bulk operations:

Automatic batching for optimal performance
Index management (create, delete, clear)
Dynamic mapping support
Efficient bulk API usage

KVStore

Simple key-value interface:

Elasticsearch-backed storage
Async/await API
Automatic index initialization
Clear and get operations

Advanced Usage 🎓

Index Rotation and Retention

const logger = new ElsSmartlogDestination({
  indexPrefix: 'myapp',
  indexRetention: 30, // Keep 30 days of logs
  node: 'http://localhost:9200',
});

// Indices are automatically created as: myapp-2025-01-22
// Old indices are automatically deleted after 30 days

Document Iteration

// Iterate over all documents in an index
const iterator = docManager.getDocumentIterator();
for await (const doc of iterator) {
  console.log(doc);
}

// Only process new documents since last run
docManager.fastForward = true;
await docManager.startPipingSession({ onlyNew: true });

Custom Snapshots

await docManager.takeSnapshot(async (iterator, prevSnapshot) => {
  let totalValue = 0;
  let count = 0;

  for await (const doc of iterator) {
    totalValue += doc._source.price;
    count++;
  }

  return {
    date: new Date().toISOString(),
    aggregationData: {
      totalValue,
      averagePrice: totalValue / count,
      count,
      previousSnapshot: prevSnapshot,
    },
  };
});

API Compatibility 🔄

This module is built on top of @elastic/elasticsearch v9.x and is compatible with:

Elasticsearch 8.x and 9.x clusters
Kibana 8.x and 9.x for log visualization
OpenSearch (with some limitations)

TypeScript Support 💙

Full TypeScript support with comprehensive type definitions:

import type {
  IElasticDocConstructorOptions,
  ISnapshot,
  SnapshotProcessor
} from '@apiclient.xyz/elasticsearch';

Performance Considerations ⚡

Bulk Operations: FastPush uses 1000-document batches by default
Connection Pooling: Reuses Elasticsearch client connections
Index Management: Automatic index creation and deletion
Data Streams: Built-in support for efficient log ingestion

Best Practices 💡

Always use authentication in production environments
Set appropriate retention policies to manage storage costs
Use piping sessions to automatically clean up stale documents
Leverage snapshots for point-in-time analytics
Configure index templates for consistent mappings

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

readme.md

@apiclient.xyz/elasticsearch