push.rocks/levelcache

Fork 0

Philipp Kunz 8bf1af08d3 fix(core): Fix S3 cache manager methods for better path encoding consistency

2024-11-26 22:41:10 +01:00

6.8 KiB

Raw Permalink Blame History

@push.rocks/levelcache

A cache that utilizes memory, disk, and S3 for data storage and backup.

Install

To install @push.rocks/levelcache, you can use npm or yarn:

npm install @push.rocks/levelcache --save

yarn add @push.rocks/levelcache

This installs @push.rocks/levelcache and adds it to your project's dependencies.

Usage

@push.rocks/levelcache provides a comprehensive solution for multi-level caching that takes advantage of memory, disk, and Amazon S3 storage, making it a versatile tool for data caching and backup. The package is built with TypeScript, enabling strict type checks and better development experience. Below, we'll explore how to effectively employ @push.rocks/levelcache in your projects, discussing its features and demonstrating its usage with code examples.

1. Overview

The LevelCache class handles all cache operations. It decides where to store data based on pre-configured thresholds corresponding to the data size and the total storage capacity allocated for each storage type (memory/disk/S3). This mechanism optimizes both speed and persistence, allowing for efficient data storage and retrieval.

2. Getting Started: Initialization

To use @push.rocks/levelcache, you'll need to import the main classes: LevelCache and CacheEntry. LevelCache is the primary class, while CacheEntry represents individual pieces of cached data.

import { LevelCache, CacheEntry } from '@push.rocks/levelcache';

Initialization with Optional Configurations

To create a cache, instantiate the LevelCache class with desired configurations. You can specify the limits for memory and disk storage, setup S3 configurations if needed, and more.

const myCache = new LevelCache({
  cacheId: 'myUniqueCacheId', // Unique ID for cache delineation
  maxMemoryStorageInMB: 10, // Maximum memory use in MB (default 0.5 MB)
  maxDiskStorageInMB: 100, // Maximum disk space in MB (default 10 MB)
  diskStoragePath: './myCache', // Path for storing disk cache; default is '.nogit'
  s3Config: {
    accessKeyId: 'yourAccessKeyId', // AWS S3 access key
    secretAccessKey: 'yourSecretAccessKey', // Corresponding secret key
    region: 'us-west-2' // AWS region, e.g., 'us-west-2'
  },
  s3BucketName: 'myBucketName', // Designated name for S3 bucket
  immutableCache: false, // Whether stored cache entries should remain unaltered
  persistentCache: true, // Should the cache persist upon restarts
});

3. Storing and Retrieving Data

LevelCache methods enable seamless data storage and retrieval, handling complexity under the hood.

Storing Data

Create a CacheEntry specifying the data content and time-to-live (ttl). Use storeCacheEntryByKey to add this entry to the cache.

async function storeData() {
  // Wait for cache to be ready before operations
  await myCache.ready;

  const entryContents = Buffer.from('Caching this data');
  const myCacheEntry = new CacheEntry({
    ttl: 7200000, // Time-to-live in milliseconds (2 hours)
    contents: entryContents,
  });

  // Storing the cache entry associated with a specific key
  await myCache.storeCacheEntryByKey('someDataKey', myCacheEntry);
}

Retrieving Data

Retrieve stored data using retrieveCacheEntryByKey. The retrieved CacheEntry will give access to the original data.

async function retrieveData() {
  const retrievedEntry = await myCache.retrieveCacheEntryByKey('someDataKey');
  if (retrievedEntry) {
    const data = retrievedEntry.contents.toString();
    console.log(data); // Expected output: Caching this data
  } else {
    console.log('Data not found or expired.');
  }
}

4. Key Management: Updating and Deleting

Deleting Cache Entries

Remove entries with deleteCacheEntryByKey, enabling clean cache management.

async function deleteData() {
  // Removes an entry using its unique key identifier
  await myCache.deleteCacheEntryByKey('someDataKey');
}

5. Cache Cleaning

Often, managing storage limits or removing outdated data becomes essential. The library supports these scenarios.

Automated Cleaning

While cache entries will naturally expire with ttl values, you can force-remove outdated entries.

// Clean outdated or expired entries
await myCache.cleanOutdated();

Full Cache Reset

Clear all entries, efficiently resetting your cache storage.

// Flush entire cache content
await myCache.cleanAll();

6. Configuring and Managing Advanced Use Cases

The flexible nature of @push.rocks/levelcache grants additional customization suited for more advanced requirements.

Custom Route Management

For certain demands, you might want to specify distinct data handling policies or routing logic.

Adjust S3 handling, size thresholds, or immutability options dynamically.
Utilize internal API expansions defined within the library for fine-grained operations.

Handling Large Datasets

Tailor the cache levels (memory, disk, S3) to accommodate higher loads:

const largeDatasetCache = new LevelCache({
  cacheId: 'largeDatasetCache',
  // Customize limits and behavior for particular patterns
  maxMemoryStorageInMB: 1024, // 1 GB memory allocation
  maxDiskStorageInMB: 2048, // 2 GB disk space allowance
  maxS3StorageInMB: 10240, // 10 GB S3 backup buffering
});

With intelligent routing and management embedded, LevelCache ensures optimal trade-offs between speed and stability.

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

6.8 KiB Raw Permalink Blame History