push.rocks/npmextra
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(appdata): Fix iteration over overwriteObject in AppData and update configuration for dependency and path handling

Browse Source
This commit is contained in:
Jürgen Kunz
2025-08-15 12:12:26 +00:00
parent 5e0edecf18
commit 62e61168a0
25 changed files with 5821 additions and 2835 deletions
Download Patch File Download Diff File Expand all files Collapse all files

604
readme.md
View File

@@ -1,295 +1,463 @@
# @push.rocks/npmextra
A utility to enhance npm with additional configuration, tool management capabilities, and a key-value store for project setups.
# @push.rocks/npmextra 🚀
## Install
To install `@push.rocks/npmextra`, use the following npm command:
**Supercharge your npm projects with powerful configuration management, tool orchestration, and persistent key-value storage.**
## Install 📦
```bash
# Using npm
npm install @push.rocks/npmextra --save
# Using pnpm (recommended)
pnpm add @push.rocks/npmextra
```
This package is available on [npm](https://www.npmjs.com/package/@push.rocks/npmextra) and can be installed into your project as a dependency to enhance npm with additional configuration and tool management capabilities.
## Overview 🎯
## Usage
`@push.rocks/npmextra` is designed to supplement npm functionalities with enhanced configuration and tool management. It facilitates the management of project configurations and tool setups in a consolidated manner, enabling a smoother workflow and maintenance process. Below are detailed use cases and examples implemented with ESM syntax and TypeScript.
`@push.rocks/npmextra` is your Swiss Army knife for npm project configuration. It eliminates configuration sprawl by centralizing tool settings, providing intelligent key-value storage, and offering powerful environment variable mapping with automatic type conversions.
### Initial Setup and Configuration
To start using `npmextra` in your project, first include it with an import statement:
### Why npmextra?
- **🎛️ Centralized Configuration**: Manage all your tool configs in one `npmextra.json` file
- **💾 Persistent Storage**: Smart key-value store with multiple storage strategies
- **🔐 Environment Mapping**: Sophisticated env var handling with automatic type conversion
- **🏗️ TypeScript First**: Full type safety and IntelliSense support
- **⚡ Zero Config**: Works out of the box with sensible defaults
- **🔄 Reactive**: Built-in change detection and observables
## Core Concepts 🏗️
### 1. Npmextra Configuration Management
Stop scattering configuration across dozens of files. Centralize everything in `npmextra.json`:
```typescript
import { Npmextra } from '@push.rocks/npmextra';
// Initialize with current directory
const npmextra = new Npmextra();
// Or specify a custom path
const npmextra = new Npmextra('/path/to/project');
// Get merged configuration for any tool
const eslintConfig = npmextra.dataFor<EslintConfig>('eslint', {
// Default values if not in npmextra.json
extends: 'standard',
rules: {}
});
```
Instantiate the `Npmextra` class optionally with a custom path to your project's working directory. If no path is provided, the current working directory (`process.cwd()`) is used.
```typescript
const npmExtraInstance = new Npmextra('/path/to/your/project');
```
### Managing Tool Configurations with `npmextra.json`
`@push.rocks/npmextra` excels in unifying tool configurations through a single `npmextra.json` file. Instead of scattering configurations across multiple files, `npmextra` enables you to define tool-specific settings within this centralized configuration file, which can then be accessed programmatically.
#### Creating and Utilizing `npmextra.json`
Create a `npmextra.json` in your project root with the following structure:
**npmextra.json example:**
```json
{
"toolname": {
"setting1": "value1",
"setting2": "value2"
"eslint": {
"extends": "@company/eslint-config",
"rules": {
"no-console": "warn"
}
},
"prettier": {
"semi": false,
"singleQuote": true
}
}
```
For example, to configure a hypothetical tool named `toolname`, define its settings as shown above.
### 2. KeyValueStore - Persistent Data Storage
#### Accessing Configuration in Your Project
With the configuration defined, you can easily access these settings in your TypeScript code as follows:
```typescript
// Import the npmextra module
import { Npmextra } from '@push.rocks/npmextra';
// Create an instance pointing at the current directory
const npmExtraInstance = new Npmextra();
// Retrieve the configuration for 'toolname', merging defaults with any found in npmextra.json
const toolConfig = npmExtraInstance.dataFor<{ setting1: string, setting2: string }>('toolname', {
setting1: 'defaultValue1',
setting2: 'defaultValue2'
});
// toolConfig now contains the merged settings from npmextra.json and provided defaults
console.log(toolConfig);
```
### Key-Value Store Management
`@push.rocks/npmextra` also includes a Key-Value Store (KeyValueStore) functionality enabling persistent storage of key-value pairs between script executions.
#### Setting Up KeyValueStore
To utilize the KeyValueStore, create an instance specifying its scope (e.g., 'userHomeDir') and a unique identity for your application or tool:
A flexible key-value store that persists data between script executions:
```typescript
import { KeyValueStore } from '@push.rocks/npmextra';
const kvStore = new KeyValueStore<'userHomeDir'>({
typeArg: 'userHomeDir',
identityArg: 'myUniqueAppName'
interface UserSettings {
username: string;
apiKey: string;
preferences: {
theme: 'light' | 'dark';
notifications: boolean;
};
}
// Different storage strategies
const kvStore = new KeyValueStore<UserSettings>({
typeArg: 'userHomeDir', // Store in user's home directory
identityArg: 'myApp',
mandatoryKeys: ['username', 'apiKey']
});
```
You can then use the `writeKey`, `readKey`, `writeAll`, and `readAll` methods to manage your store respectively.
// Write operations
await kvStore.writeKey('username', 'john_doe');
await kvStore.writeKey('preferences', {
theme: 'dark',
notifications: true
});
#### Example: Storing and Retrieving Data
```typescript
// Write a single key-value pair
await kvStore.writeKey('username', 'johnDoe');
// Read a single key
// Read operations
const username = await kvStore.readKey('username');
console.log(username); // Outputs: johnDoe
// Write multiple key-value pairs
await kvStore.writeAll({
email: 'john@example.com',
isAdmin: true
});
// Read all key-value pairs
const allData = await kvStore.readAll();
console.log(allData); // Outputs the entire store's contents
```
### Advanced Key-Value Store Management
In addition to basic read/write operations, `npmextra`’s `KeyValueStore` supports advanced scenarios like mandatory keys and custom file paths.
#### Example: Mandatory Keys and Custom Paths
Consider a scenario where your application requires specific keys to be present in the KeyValueStore. You can define mandatory keys and use a custom path for your store like this:
```typescript
import { KeyValueStore } from '@push.rocks/npmextra';
interface CustomData {
key1: string;
key2: number;
key3?: boolean;
}
const kvStore = new KeyValueStore<CustomData>({
typeArg: 'custom',
identityArg: 'customApp',
customPath: '/custom/path/to/store.json',
mandatoryKeys: ['key1', 'key2']
});
// Ensure all mandatory keys are present
// Check for missing mandatory keys
const missingKeys = await kvStore.getMissingMandatoryKeys();
if (missingKeys.length) {
console.log(`Missing mandatory keys: ${missingKeys.join(', ')}`);
if (missingKeys.length > 0) {
console.log('Missing required configuration:', missingKeys);
}
// Use the KeyValueStore
await kvStore.writeKey('key1', 'value1');
await kvStore.writeKey('key2', 123);
const key1Value = await kvStore.readKey('key1');
const allData = await kvStore.readAll();
console.log(key1Value); // Outputs: value1
console.log(allData); // Outputs: { key1: 'value1', key2: 123 }
// Wait for keys to be present
await kvStore.waitForKeysPresent(['apiKey']);
```
### Combining AppData and KeyValueStore
**Storage Types:**
- `userHomeDir`: Store in user's home directory
- `custom`: Specify your own path
- `ephemeral`: In-memory only (perfect for testing)
The `AppData` class extends the functionality of `KeyValueStore` by integrating environmental variables, specifying additional configurations, and providing a more structured approach to data management.
### 3. AppData - Advanced Environment Management 🌟
#### Example: AppData Usage
The crown jewel of npmextra - sophisticated environment variable mapping with automatic type conversion:
```typescript
import { AppData } from '@push.rocks/npmextra';
interface AppSettings {
settingA: string;
settingB: number;
nestedSetting: {
innerSetting: boolean;
}
interface AppConfig {
apiUrl: string;
apiKey: string;
port: number;
features: {
analytics: boolean;
payment: boolean;
};
cache: {
ttl: number;
redis: {
host: string;
password: string;
};
};
}
const appDataInstance = await AppData.createAndInit<AppSettings>({
dirPath: '/custom/path/to/appdata',
requiredKeys: ['settingA', 'settingB'],
const appData = await AppData.createAndInit<AppConfig>({
dirPath: '/app/config', // Optional: defaults to smart path selection
requiredKeys: ['apiKey', 'apiUrl'],
envMapping: {
settingA: 'MY_ENV_A',
settingB: 'hard:42',
nestedSetting: {
innerSetting: 'MY_ENV_INNER'
apiUrl: 'API_URL', // Simple mapping
apiKey: 'hard:development-key-123', // Hardcoded value
port: 'hard:3000', // Hardcoded number
features: {
analytics: 'boolean:ENABLE_ANALYTICS', // Force boolean conversion
payment: 'hard_boolean:true' // Hardcoded boolean
},
cache: {
ttl: 'json:CACHE_CONFIG', // Parse JSON from env var
redis: {
host: 'REDIS_HOST',
password: 'base64:REDIS_PASSWORD_B64' // Decode base64
}
}
},
overwriteObject: {
// Force these values regardless of env vars
port: 8080
}
});
const store = await appData.getKvStore();
const apiUrl = await store.readKey('apiUrl');
```
## AppData Special Cases & Conversions 🎯
### Environment Variable Prefixes
AppData supports sophisticated type conversion through prefixes:
| Prefix | Description | Example | Result |
|--------|-------------|---------|--------|
| `hard:` | Hardcoded value | `hard:myvalue` | `"myvalue"` |
| `hard_boolean:` | Hardcoded boolean | `hard_boolean:true` | `true` |
| `hard_json:` | Hardcoded JSON | `hard_json:{"key":"value"}` | `{key: "value"}` |
| `hard_base64:` | Hardcoded base64 | `hard_base64:SGVsbG8=` | `"Hello"` |
| `boolean:` | Env var as boolean | `boolean:FEATURE_FLAG` | `true/false` |
| `json:` | Parse env var as JSON | `json:CONFIG_JSON` | Parsed object |
| `base64:` | Decode env var from base64 | `base64:SECRET_B64` | Decoded string |
### Automatic Suffix Detection
Variables ending with certain suffixes get automatic conversion:
```typescript
{
envMapping: {
// Automatically parsed as JSON if MY_CONFIG_JSON="{"enabled":true}"
config: 'MY_CONFIG_JSON',
// Automatically decoded from base64 if SECRET_KEY_BASE64="SGVsbG8="
secret: 'SECRET_KEY_BASE64'
}
}
```
### Complex Examples
```typescript
const appData = await AppData.createAndInit({
envMapping: {
// Simple environment variable
apiUrl: 'API_URL',
// Hardcoded values with type conversion
debugMode: 'hard_boolean:false',
maxRetries: 'hard:5',
defaultConfig: 'hard_json:{"timeout":30,"retries":3}',
// Environment variables with conversion
features: 'json:FEATURE_FLAGS', // Expects: {"feature1":true,"feature2":false}
isProduction: 'boolean:IS_PROD', // Expects: "true" or "false"
apiSecret: 'base64:API_SECRET', // Expects: base64 encoded string
// Nested structures
database: {
host: 'DB_HOST',
port: 'hard:5432',
credentials: {
user: 'DB_USER',
password: 'base64:DB_PASSWORD_ENCODED',
ssl: 'boolean:DB_USE_SSL'
}
}
},
// Override any env mappings
overwriteObject: {
debugMode: true, // Force debug mode regardless of env
database: {
host: 'localhost' // Force localhost for development
}
}
});
const appDataKvStore = await appDataInstance.getKvStore();
// Writing values
await appDataKvStore.writeKey('settingA', 'exampleValue');
await appDataKvStore.writeKey('settingB', 100);
await appDataKvStore.writeKey('nestedSetting', { innerSetting: true });
// Reading values
const settingA = await appDataKvStore.readKey('settingA');
const allSettings = await appDataKvStore.readAll();
console.log(settingA); // Outputs: 'exampleValue'
console.log(allSettings); // Outputs: { settingA: 'exampleValue', settingB: 100, nestedSetting: { innerSetting: true } }
```
### Error Handling and Debugging
### Boolean Conversion Rules
Proper error handling ensures your integrations with `npmextra` are robust and stable. Below are some strategies for error handling and debugging potential issues.
AppData intelligently handles boolean conversions:
#### Example: Error Handling in KeyValueStore
1. **String "true"/"false"**: Converted to boolean
2. **With `boolean:` prefix**: Any env var value is converted (`"true"` → `true`, anything else → `false`)
3. **With `hard_boolean:` prefix**: Hardcoded boolean value
4. **Regular env vars**: Strings remain strings unless prefixed
```typescript
import { KeyValueStore } from '@push.rocks/npmextra';
const kvStore = new KeyValueStore('userHomeDir', 'errorHandlingApp');
try {
await kvStore.writeKey('importantKey', 'importantValue');
const value = await kvStore.readKey('importantKey');
console.log(value); // Outputs: importantValue
} catch (error) {
console.error('Error managing key-value store:', error);
// Environment: FEATURE_A="true", FEATURE_B="yes", FEATURE_C="1"
{
envMapping: {
featureA: 'FEATURE_A', // Result: "true" (string)
featureB: 'boolean:FEATURE_B', // Result: false (only "true" → true)
featureC: 'boolean:FEATURE_C', // Result: false (only "true" → true)
featureD: 'hard_boolean:true' // Result: true (hardcoded)
}
}
```
#### Debugging Configuration Issues in `npmextra.json`
## Advanced Patterns 🎨
To debug configuration issues, you can utilize conditional logging and checks:
### Reactive Configuration
Subscribe to configuration changes:
```typescript
import { Npmextra } from '@push.rocks/npmextra';
const npmExtraInstance = new Npmextra();
const toolConfig = npmExtraInstance.dataFor('toolname', {
configKey1: 'defaultValue1',
configKey2: 'defaultValue2'
const kvStore = new KeyValueStore<Config>({
typeArg: 'custom',
identityArg: 'myApp'
});
if (!toolConfig.configKey1) {
console.error('configKey1 is missing in npmextra.json');
// Subscribe to changes
kvStore.changeSubject.subscribe((newData) => {
console.log('Configuration changed:', newData);
});
// Changes trigger notifications
await kvStore.writeKey('theme', 'dark');
```
### Testing with Ephemeral Storage
Perfect for unit tests - no file system pollution:
```typescript
const testStore = new KeyValueStore<TestData>({
typeArg: 'ephemeral',
identityArg: 'test'
});
// All operations work normally, but nothing persists to disk
await testStore.writeKey('testKey', 'testValue');
```
### Smart Path Resolution
AppData automatically selects the best storage location:
1. Checks for `/app/data` (containerized environments)
2. Falls back to `/data` (alternate container path)
3. Uses `.nogit/appdata` (local development)
```typescript
// Automatic path selection
const appData = await AppData.createAndInit({
// No dirPath specified - smart detection
requiredKeys: ['apiKey']
});
// Or force ephemeral for testing
const testData = await AppData.createAndInit({
ephemeral: true, // No disk persistence
requiredKeys: ['testKey']
});
```
### Waiting for Configuration
Block until required configuration is available:
```typescript
const appData = await AppData.createAndInit<Config>({
requiredKeys: ['apiKey', 'apiUrl']
});
// Wait for specific key
const apiKey = await appData.waitForAndGetKey('apiKey');
// Check missing keys
const missingKeys = await appData.logMissingKeys();
// Logs: "The following mandatory keys are missing in the appdata:
// -> apiKey,
// -> apiUrl"
```
## Real-World Example 🌍
Here's a complete example of a CLI tool using npmextra:
```typescript
import { Npmextra, AppData, KeyValueStore } from '@push.rocks/npmextra';
interface CliConfig {
githubToken: string;
openaiKey: string;
model: 'gpt-3' | 'gpt-4';
cache: {
enabled: boolean;
ttl: number;
};
}
console.log(toolConfig);
class MyCLI {
private npmextra: Npmextra;
private appData: AppData<CliConfig>;
private cache: KeyValueStore<{[key: string]: any}>;
async initialize() {
// Load tool configuration
this.npmextra = new Npmextra();
const config = this.npmextra.dataFor<any>('mycli', {
defaultModel: 'gpt-3'
});
// Setup app data with env mapping
this.appData = await AppData.createAndInit<CliConfig>({
requiredKeys: ['githubToken', 'openaiKey'],
envMapping: {
githubToken: 'GITHUB_TOKEN',
openaiKey: 'base64:OPENAI_KEY_ENCODED',
model: 'hard:gpt-4',
cache: {
enabled: 'boolean:ENABLE_CACHE',
ttl: 'hard:3600'
}
}
});
// Initialize cache
this.cache = new KeyValueStore({
typeArg: 'userHomeDir',
identityArg: 'mycli-cache'
});
// Check for missing configuration
const missingKeys = await this.appData.logMissingKeys();
if (missingKeys.length > 0) {
console.error('Please configure the missing keys');
process.exit(1);
}
}
async run() {
await this.initialize();
const config = await this.appData.getKvStore();
const settings = await config.readAll();
console.log(`Using model: ${settings.model}`);
console.log(`Cache enabled: ${settings.cache.enabled}`);
// Use the configuration...
}
}
// Run the CLI
const cli = new MyCLI();
cli.run();
```
### Integration Tests
## API Reference 📚
Writing tests ensures that your integration with `npmextra` works as expected. Below are examples of integration tests for both `Npmextra` and `KeyValueStore`.
#### Example: Testing `Npmextra` Class
### Npmextra Class
```typescript
import { expect, tap } from '@push.rocks/tapbundle';
import { Npmextra } from '@push.rocks/npmextra';
let npmExtraInstance: Npmextra;
tap.test('should create an instance of Npmextra', async () => {
npmExtraInstance = new Npmextra();
expect(npmExtraInstance).toBeInstanceOf(Npmextra);
});
tap.test('should load configuration from npmextra.json', async () => {
const config = npmExtraInstance.dataFor('toolname', {
defaultKey1: 'defaultValue1',
});
expect(config).toHaveProperty('defaultKey1');
});
tap.start();
new Npmextra(cwdArg?: string)
```
- `cwdArg`: Optional working directory path
#### Example: Testing `KeyValueStore` Class
**Methods:**
- `dataFor<T>(toolName: string, defaultOptions: T): T` - Get merged configuration
### KeyValueStore Class
```typescript
import { expect, tap } from '@push.rocks/tapbundle';
import { KeyValueStore } from '@push.rocks/npmextra';
let kvStore: KeyValueStore<{ key1: string, key2: number }>;
tap.test('should create a KeyValueStore instance', async () => {
kvStore = new KeyValueStore({
typeArg: 'userHomeDir',
identityArg: 'testApp'
});
expect(kvStore).toBeInstanceOf(KeyValueStore);
});
tap.test('should write and read back a value', async () => {
await kvStore.writeKey('key1', 'value1');
const result = await kvStore.readKey('key1');
expect(result).toEqual('value1');
});
tap.test('should write and read back multiple values', async () => {
await kvStore.writeAll({ key1: 'updatedValue1', key2: 2 });
const result = await kvStore.readAll();
expect(result).toEqual({ key1: 'updatedValue1', key2: 2 });
});
tap.start();
new KeyValueStore<T>(options: {
typeArg: 'custom' | 'userHomeDir' | 'ephemeral';
identityArg: string;
customPath?: string;
mandatoryKeys?: Array<keyof T>;
})
```
### Summary
**Methods:**
- `readKey(key: string): Promise<T>` - Read single value
- `writeKey(key: string, value: T): Promise<void>` - Write single value
- `readAll(): Promise<T>` - Read all values
- `writeAll(data: T): Promise<void>` - Write all values
- `deleteKey(key: string): Promise<void>` - Delete a key
- `getMissingMandatoryKeys(): Promise<string[]>` - Check missing required keys
- `waitForKeysPresent(keys: string[]): Promise<void>` - Wait for keys
By centralizing configuration management and offering a versatile key-value store, `@push.rocks/npmextra` significantly simplifies the setup and management of tools and settings in modern JavaScript and TypeScript projects. Whether you're managing project-wide configurations or need persistent storage for key-value pairs, `npmextra` provides an efficient and streamlined solution. Leveraging these robust features will ensure your project is well-configured and maintainable.
### AppData Class
```typescript
await AppData.createAndInit<T>(options: {
dirPath?: string;
requiredKeys?: Array<keyof T>;
ephemeral?: boolean;
envMapping?: PartialDeep<T>;
overwriteObject?: PartialDeep<T>;
})
```
**Methods:**
- `getKvStore(): Promise<KeyValueStore<T>>` - Get underlying store
- `logMissingKeys(): Promise<Array<keyof T>>` - Log and return missing keys
- `waitForAndGetKey<K>(key: K): Promise<T[K]>` - Wait for and retrieve key
## License and Legal Information
@@ -308,4 +476,4 @@ Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.