push.rocks/smartfile
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

BREAKING CHANGE(SmartFileFactory): Refactor to in-memory file API and introduce SmartFileFactory; delegate filesystem operations to @push.rocks/smartfs; bump to 12.0.0

Browse Source
This commit is contained in:
Jürgen Kunz
2025-11-22 13:18:32 +00:00
parent 16d47ea348
commit ad33cb6d73
24 changed files with 1552 additions and 1789 deletions
Download Patch File Download Diff File Expand all files Collapse all files

589
readme.md
View File

@@ -1,318 +1,495 @@
# @push.rocks/smartfile 📁
> **A powerful, TypeScript-based file management library for Node.js**
> **High-level file representation classes for Node.js**
## 🚀 What is smartfile?
`@push.rocks/smartfile` is your go-to solution for file operations in Node.js. It offers a clean, promise-based API for handling files, directories, streams, and even virtual filesystems - all while maintaining maximum performance and reliability.
`@push.rocks/smartfile` provides powerful **in-memory file representations** for Node.js applications. It offers clean, TypeScript-first classes for working with files (`SmartFile`), streams (`StreamFile`), and virtual file collections (`VirtualDirectory`).
Think of it as `fs` on steroids, with TypeScript superpowers! 💪
Think of it as your go-to solution for **content manipulation**, **file transformations**, and **in-memory file operations** - all while seamlessly integrating with [@push.rocks/smartfs](https://code.foss.global/push.rocks/smartfs) for actual filesystem operations.
## 💾 Installation
```bash
npm install @push.rocks/smartfile
pnpm install @push.rocks/smartfile
# Optional: Install smartfs for filesystem operations
pnpm install @push.rocks/smartfs
```
## ✨ Features
## ✨ Key Features
- 🔥 **Streaming Support** - Handle massive files with ease using `StreamFile`
- 📦 **Virtual Directories** - Work with in-memory file structures
- 🌐 **URL Support** - Directly work with files from URLs
- 🎯 **TypeScript First** - Full type safety and IntelliSense support
- ⚡ **Promise-based API** - Modern async/await patterns throughout
- 🛠️ **Comprehensive Toolset** - From basic CRUD to advanced operations
- 🎯 **Factory Pattern** - Clean, consistent API for creating file instances
- 🔥 **Streaming Support** - Handle massive files efficiently with `StreamFile`
- 📦 **Virtual Directories** - Work with in-memory file collections
- 🌐 **URL Support** - Directly fetch files from URLs
- 🎨 **Content Manipulation** - Edit, transform, and parse file content
- ⚡ **TypeScript First** - Full type safety and IntelliSense support
- 🛠️ **Comprehensive Collection API** - Filter, map, find files in virtual directories
## 📚 Quick Start
### Using the Factory
```typescript
import * as smartfile from '@push.rocks/smartfile';
import { SmartFileFactory } from '@push.rocks/smartfile';
// Read a file
const content = await smartfile.fs.toStringSync('./my-file.txt');
// Create factory (uses Node.js filesystem by default)
const factory = SmartFileFactory.nodeFs();
// Write a file
await smartfile.memory.toFs('Hello World!', './output.txt');
// Load a file into memory
const file = await factory.fromFilePath('./config.json');
// Work with JSON
const data = await smartfile.fs.toObjectSync('./data.json');
// Edit content
await file.editContentAsString(async (content) => {
return content.toUpperCase();
});
// Save back to disk
await file.write();
```
### With SmartFs Integration
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
// Create SmartFs instance with provider
const smartFs = new SmartFs(new SmartFsProviderNode());
// Create factory bound to this filesystem
const factory = new SmartFileFactory(smartFs);
// Now all file operations use the smartfs instance
const file = await factory.fromFilePath('./data.json');
await file.write(); // Uses smartfs under the hood
```
## 🎨 Core Components
### SmartFile Class
### SmartFileFactory
The `SmartFile` class represents a single file with powerful manipulation capabilities:
The factory is your entry point for creating all file instances:
```typescript
import { SmartFile } from '@push.rocks/smartfile';
import { SmartFileFactory } from '@push.rocks/smartfile';
// Create from file path
const fileFromPath = await SmartFile.fromFilePath('./data.json');
const factory = SmartFileFactory.nodeFs();
// Create from URL
const fileFromUrl = await SmartFile.fromUrl('https://example.com/config.json');
// Create from various sources
const fileFromPath = await factory.fromFilePath('./data.json');
const fileFromUrl = await factory.fromUrl('https://example.com/config.json');
const fileFromBuffer = factory.fromBuffer('./file.txt', Buffer.from('content'));
const fileFromString = factory.fromString('./file.txt', 'Hello World', 'utf8');
// Create from text
const fileFromText = await SmartFile.fromString(
'./my-file.txt',
'This is my content',
'utf8'
);
// Create StreamFile instances
const stream = await factory.streamFromPath('./large-file.zip');
const streamFromUrl = await factory.streamFromUrl('https://example.com/video.mp4');
// Create from Buffer
const fileFromBuffer = await SmartFile.fromBuffer(
'./binary.dat',
Buffer.from([0x00, 0x01, 0x02])
);
// Edit content
await fileFromPath.editContentAsString(async (content) => {
return content.replace(/old/g, 'new');
});
// Write to disk
await fileFromPath.write();
// Get content
const contentString = fileFromPath.parseContentAsString();
const contentBuffer = fileFromPath.parseContentAsBuffer();
// Create VirtualDirectory instances
const vdir = await factory.virtualDirectoryFromPath('./src');
const emptyVdir = factory.virtualDirectoryEmpty();
```
### StreamFile Class 🌊
### SmartFile Class
Represents a single file loaded in memory:
```typescript
// Created via factory
const file = await factory.fromFilePath('./data.json');
// Content access
const asString = file.parseContentAsString();
const asBuffer = file.parseContentAsBuffer();
// Content manipulation
await file.editContentAsString(async (content) => {
const data = JSON.parse(content);
data.updated = new Date().toISOString();
return JSON.stringify(data, null, 2);
});
// File operations
await file.write(); // Save to original location
await file.writeToDiskAtPath('./output.json'); // Save to specific path
await file.writeToDir('./dist'); // Save to directory
await file.read(); // Reload from disk
await file.delete(); // Delete from disk
// Metadata
const size = await file.getSize(); // File size in bytes
const hash = await file.getHash('content'); // SHA256 hash
const stream = file.getStream(); // Get as Node.js stream
// Path information
console.log(file.path); // Relative path
console.log(file.absolutePath); // Absolute path
console.log(file.parsedPath); // Parsed path components
```
### StreamFile Class
Perfect for handling large files without memory overhead:
```typescript
import { StreamFile } from '@push.rocks/smartfile';
// Created via factory
const streamFile = await factory.streamFromPath('./bigfile.zip');
// Create from path
const streamFile = await StreamFile.fromPath('./bigfile.zip');
// Or from URL
const urlStream = await factory.streamFromUrl('https://example.com/large.mp4');
// Create from URL
const urlStream = await StreamFile.fromUrl('https://example.com/large.mp4');
// Or from buffer
const bufferStream = factory.streamFromBuffer(Buffer.from('content'));
// Create from buffer
const bufferStream = StreamFile.fromBuffer(
Buffer.from('streaming content'),
'stream.txt'
);
// Write to disk
// Write to disk (streams the content)
await streamFile.writeToDisk('./output/bigfile.zip');
await streamFile.writeToDir('./output');
// Get as buffer (careful with large files!)
// Get content (loads into memory - use carefully!)
const buffer = await streamFile.getContentAsBuffer();
const string = await streamFile.getContentAsString('utf8');
// Get as stream
// Get as Node.js stream for piping
const readStream = await streamFile.createReadStream();
// Convert to SmartFile (loads into memory)
const smartFile = await streamFile.toSmartFile();
// Get file size
const size = await streamFile.getSize();
```
### VirtualDirectory Class 📂
### VirtualDirectory Class
Manage collections of files as virtual filesystems:
Manage collections of SmartFiles in memory:
```typescript
import { VirtualDirectory } from '@push.rocks/smartfile';
// Created via factory
const vdir = await factory.virtualDirectoryFromPath('./src');
// Create from filesystem
const vDir = await VirtualDirectory.fromFsDirPath('./src');
// Or create empty
const emptyVdir = factory.virtualDirectoryEmpty();
// Create from file array
const vDirFromFiles = await VirtualDirectory.fromFileArray([
await SmartFile.fromFilePath('./file1.txt'),
await SmartFile.fromFilePath('./file2.txt')
]);
// Or from file array
const files = [file1, file2, file3];
const vdirFromFiles = factory.virtualDirectoryFromFileArray(files);
// ============================================
// Collection Queries (in-memory operations)
// ============================================
// Check existence in collection
if (vdir.exists('components/Button.tsx')) {
console.log('File exists in virtual directory');
}
// Get file from collection
const file = await vdir.getFileByPath('utils/helpers.ts');
// List all files
const allFiles = vdir.listFiles();
// List directory paths represented in collection
const dirs = vdir.listDirectories();
// Filter files
const tsFiles = vdir.filter(f => f.path.endsWith('.ts'));
const largeFiles = vdir.filter(f => f.contentBuffer.length > 10000);
// Map/transform files
const uppercased = vdir.map(f => {
f.contentBuffer = Buffer.from(f.parseContentAsString().toUpperCase());
return f;
});
// Find specific file
const configFile = vdir.find(f => f.path.includes('config'));
// Collection info
const fileCount = vdir.size();
const empty = vdir.isEmpty();
// ============================================
// Collection Mutations
// ============================================
// Add files
vDir.addSmartfiles([
await SmartFile.fromString('./virtual/new.txt', 'content')
]);
vdir.addSmartfile(newFile);
vdir.addSmartfiles([file1, file2, file3]);
// List files
const files = vDir.listFiles();
const directories = vDir.listDirectories();
// Remove file
vdir.removeByPath('old-file.ts');
// Get file
const file = vDir.getFileByPath('./some/path.txt');
// Clear all files
vdir.clear();
// Save to disk
await vDir.saveToDisk('./output');
// Merge another virtual directory
vdir.merge(otherVirtualDir);
// Load from disk
await vDir.loadFromDisk('./source');
// ============================================
// Load/Save (filesystem bridge operations)
// ============================================
// Save all files to disk
await vdir.saveToDisk('./dist');
// Reload from disk
await vdir.loadFromDisk('./src');
// Work with subdirectories
const subVdir = await vdir.shiftToSubdirectory('components');
await vdir.addVirtualDirectory(otherVdir, 'lib');
```
## 🛠️ File Operations
## 🔄 Integration with SmartFs
### Basic Operations
For filesystem operations beyond loading/saving content, use [@push.rocks/smartfs](https://code.foss.global/push.rocks/smartfs):
```typescript
// Check existence
const exists = await smartfile.fs.fileExists('./file.txt');
const existsSync = smartfile.fs.fileExistsSync('./file.txt');
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
// Read operations
const content = await smartfile.fs.toStringSync('./file.txt');
const buffer = await smartfile.fs.toBuffer('./file.txt');
const object = await smartfile.fs.toObjectSync('./data.json');
const smartFs = new SmartFs(new SmartFsProviderNode());
const factory = new SmartFileFactory(smartFs);
// Write operations
await smartfile.memory.toFs('content', './output.txt');
smartfile.memory.toFsSync('content', './output-sync.txt');
// Use smartfile for content manipulation
const file = await factory.fromFilePath('./config.json');
await file.editContentAsString(async (s) => s.toUpperCase());
await file.write();
// Copy operations
await smartfile.fs.copy('./source.txt', './dest.txt');
await smartfile.fs.copy('./src-dir', './dest-dir');
// Use smartfs for filesystem operations
const exists = await smartFs.file('./config.json').exists();
await smartFs.file('./config.json').copy('./config.backup.json');
const stats = await smartFs.file('./config.json').stat();
// Delete operations
await smartfile.fs.remove('./file.txt');
await smartfile.fs.removeSync('./file-sync.txt');
await smartfile.fs.removeMany(['./file1.txt', './file2.txt']);
// Ensure operations (create if not exists)
await smartfile.fs.ensureDir('./my/deep/directory');
await smartfile.fs.ensureFile('./my/file.txt');
await smartfile.fs.ensureEmptyDir('./empty-dir');
// List directory with smartfs
const entries = await smartFs.directory('./src').list();
```
### Directory Operations
## 🌟 Common Use Cases
### Configuration File Management
```typescript
// List contents
const files = await smartfile.fs.listFiles('./directory');
const folders = await smartfile.fs.listFolders('./directory');
const items = await smartfile.fs.listAllItems('./directory');
const factory = SmartFileFactory.nodeFs();
// Get file tree
const tree = await smartfile.fs.listFileTree('./src', '**/*.ts');
// Directory checks
const isDir = await smartfile.fs.isDirectory('./path');
const isFile = await smartfile.fs.isFile('./path');
```
### Advanced Features
```typescript
// Wait for file to be ready
await smartfile.fs.waitForFileToBeReady('./file.txt');
// Stream operations
const readStream = smartfile.fsStream.createReadStream('./input.txt');
const writeStream = smartfile.fsStream.createWriteStream('./output.txt');
// File type detection
const fileType = smartfile.interpreter.filetype('./document.pdf');
// Returns: 'pdf'
// Smart read stream (with custom processing)
const smartStream = new smartfile.fsStream.SmartReadStream('./data.txt');
smartStream.on('data', (chunk) => {
// Process chunk
console.log(chunk.toString());
// Load, modify, and save config
const config = await factory.fromFilePath('./package.json');
await config.editContentAsString(async (content) => {
const pkg = JSON.parse(content);
pkg.version = '2.0.0';
return JSON.stringify(pkg, null, 2);
});
await config.write();
```
## 🔄 Working with Multiple Files
### Batch File Processing
```typescript
// Process multiple SmartFiles
const files = await smartfile.fs.fileTreeToObject(
'./src',
'**/*.{ts,js}'
);
const factory = SmartFileFactory.nodeFs();
// Write array to disk
const smartfiles = [
await SmartFile.fromString('file1.txt', 'content1'),
await SmartFile.fromString('file2.txt', 'content2')
];
await smartfile.memory.smartfileArrayToFs(smartfiles, './output');
```
// Load directory into virtual collection
const vdir = await factory.virtualDirectoryFromPath('./content');
## 🎯 Real-World Examples
### Website Bundler
```typescript
// Bundle website assets
const website = await VirtualDirectory.fromFsDirPath('./website');
const bundle = await website.smartfileArray;
// Process all CSS files
for (const file of bundle.filter(f => f.path.endsWith('.css'))) {
await file.editContentAsString(async (css) => {
// Minify CSS here
return css.replace(/\s+/g, ' ');
// Process all markdown files
const mdFiles = vdir.filter(f => f.path.endsWith('.md'));
for (const file of mdFiles.listFiles()) {
await file.editContentAsString(async (content) => {
// Add frontmatter, transform links, etc.
return `---\nprocessed: true\n---\n\n${content}`;
});
}
// Save processed bundle
await website.saveToDisk('./dist');
// Save processed files
await vdir.saveToDisk('./dist/content');
```
### File Watcher & Processor
### Download and Process Remote Files
```typescript
// Watch for new files and process them
import { SmartFile, StreamFile } from '@push.rocks/smartfile';
const factory = SmartFileFactory.nodeFs();
async function processLargeFile(filePath: string) {
const streamFile = await StreamFile.fromPath(filePath);
// Stream to processed location
await streamFile.writeToDisk(`./processed/${path.basename(filePath)}`);
// Clean up original
await smartfile.fs.remove(filePath);
}
// Fetch from URL
const remoteFile = await factory.fromUrl('https://api.example.com/data.json');
// Process content
await remoteFile.editContentAsString(async (content) => {
const data = JSON.parse(content);
// Transform data
return JSON.stringify(data.results, null, 2);
});
// Save locally
await remoteFile.writeToDiskAtPath('./cache/data.json');
```
### Configuration Manager
### Large File Streaming
```typescript
// Load and merge config files
const defaultConfig = await smartfile.fs.toObjectSync('./config.default.json');
const userConfig = await smartfile.fs.toObjectSync('./config.user.json');
const factory = SmartFileFactory.nodeFs();
const merged = { ...defaultConfig, ...userConfig };
// Download large file as stream
const largeFile = await factory.streamFromUrl('https://example.com/large-dataset.csv');
await smartfile.memory.toFs(
JSON.stringify(merged, null, 2),
'./config.final.json'
);
// Save to disk (streams, doesn't load all into memory)
await largeFile.writeToDisk('./data/dataset.csv');
// Or get size without downloading entire file
const size = await largeFile.getSize();
console.log(`File size: ${size} bytes`);
```
## 🌟 API Reference
### Virtual File System for Testing
### Core Modules
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderMemory } from '@push.rocks/smartfs';
- `fs` - File system operations
- `fsStream` - Streaming operations
- `memory` - Memory/buffer operations
- `interpreter` - File type detection
// Use in-memory filesystem for tests
const memoryFs = new SmartFs(new SmartFsProviderMemory());
const factory = new SmartFileFactory(memoryFs);
### Main Classes
// Create virtual files
const testFile = factory.fromString('test.txt', 'test content');
await testFile.write(); // Writes to in-memory filesystem
- `SmartFile` - Single file representation
- `StreamFile` - Streaming file operations
- `VirtualDirectory` - Virtual filesystem management
// Test your code without touching real filesystem
```
## 🏗️ TypeScript Support
## 🏗️ Architecture
### Responsibility Split
**@push.rocks/smartfile** (this package):
- ✅ In-memory file representations (SmartFile, StreamFile, VirtualDirectory)
- ✅ Content manipulation and transformation
- ✅ Loading content FROM sources (disk, URL, buffer, string)
- ✅ Saving content TO destinations (disk, stream)
- ✅ Collection operations (filter, map, find on VirtualDirectory)
**@push.rocks/smartfs**:
- ✅ Low-level filesystem operations (exists, stat, copy, move, delete)
- ✅ Directory operations (list, create, remove)
- ✅ Provider abstraction (Node.js fs, in-memory, S3, etc.)
- ✅ Streaming (readStream, writeStream)
- ✅ Transactions and file watching
## 📖 API Reference
### SmartFileFactory
| Method | Description |
|--------|-------------|
| `SmartFileFactory.nodeFs()` | Create factory with Node.js filesystem provider |
| `new SmartFileFactory(smartFs)` | Create factory with custom SmartFs instance |
| `factory.fromFilePath(path, base?)` | Load file from disk into SmartFile |
| `factory.fromUrl(url)` | Fetch file from URL into SmartFile |
| `factory.fromBuffer(path, buffer, base?)` | Create SmartFile from Buffer |
| `factory.fromString(path, content, encoding, base?)` | Create SmartFile from string |
| `factory.streamFromPath(path)` | Create StreamFile from disk |
| `factory.streamFromUrl(url)` | Create StreamFile from URL |
| `factory.streamFromBuffer(buffer, path?)` | Create StreamFile from Buffer |
| `factory.virtualDirectoryFromPath(path)` | Load directory into VirtualDirectory |
| `factory.virtualDirectoryEmpty()` | Create empty VirtualDirectory |
| `factory.virtualDirectoryFromFileArray(files)` | Create VirtualDirectory from SmartFiles |
### SmartFile Instance Methods
| Method | Description |
|--------|-------------|
| `file.write()` | Save to original location |
| `file.writeToDiskAtPath(path)` | Save to specific path |
| `file.writeToDir(dir)` | Save to directory (preserves relative path) |
| `file.read()` | Reload content from disk |
| `file.delete()` | Delete file from disk |
| `file.editContentAsString(fn)` | Transform content as string |
| `file.parseContentAsString(encoding?)` | Get content as string |
| `file.parseContentAsBuffer()` | Get content as Buffer |
| `file.getHash(type?)` | Get SHA256 hash ('path', 'content', 'all') |
| `file.getSize()` | Get content size in bytes |
| `file.getStream()` | Get content as Node.js Readable stream |
### StreamFile Instance Methods
| Method | Description |
|--------|-------------|
| `stream.writeToDisk(path)` | Stream content to disk |
| `stream.writeToDir(dir)` | Stream to directory |
| `stream.createReadStream()` | Get as Node.js Readable stream |
| `stream.getContentAsBuffer()` | Load entire content into Buffer |
| `stream.getContentAsString(encoding?)` | Load entire content as string |
| `stream.getSize()` | Get content size in bytes |
| `stream.toSmartFile()` | Convert to SmartFile (loads into memory) |
### VirtualDirectory Instance Methods
**Collection Queries:**
| Method | Description |
|--------|-------------|
| `vdir.exists(path)` | Check if file exists in collection |
| `vdir.has(path)` | Alias for exists() |
| `vdir.getFileByPath(path)` | Get SmartFile by path |
| `vdir.listFiles()` | Get all SmartFiles |
| `vdir.listDirectories()` | Get all directory paths |
| `vdir.filter(predicate)` | Filter files, returns new VirtualDirectory |
| `vdir.map(fn)` | Transform files, returns new VirtualDirectory |
| `vdir.find(predicate)` | Find first matching file |
| `vdir.size()` | Get file count |
| `vdir.isEmpty()` | Check if empty |
**Collection Mutations:**
| Method | Description |
|--------|-------------|
| `vdir.addSmartfile(file)` | Add single file |
| `vdir.addSmartfiles(files)` | Add multiple files |
| `vdir.removeByPath(path)` | Remove file by path |
| `vdir.clear()` | Remove all files |
| `vdir.merge(otherVdir)` | Merge another VirtualDirectory |
**Load/Save:**
| Method | Description |
|--------|-------------|
| `vdir.saveToDisk(dir)` | Write all files to disk |
| `vdir.loadFromDisk(dir)` | Load files from disk (replaces collection) |
## 🔧 TypeScript Support
Full TypeScript support with comprehensive type definitions:
```typescript
import type { SmartFile, StreamFile, VirtualDirectory } from '@push.rocks/smartfile';
import type { SmartFile, StreamFile, VirtualDirectory, SmartFileFactory } from '@push.rocks/smartfile';
// All methods are fully typed
const processFile = async (file: SmartFile): Promise<void> => {
const content = file.parseContentAsString();
// TypeScript knows content is string
};
```
## 📦 Backward Compatibility
Version 12.0.0 introduces the factory pattern. Legacy exports are deprecated but still functional:
```typescript
// ⚠️ Deprecated (still works, but will be removed)
import * as smartfile from '@push.rocks/smartfile';
const file = await smartfile.SmartFile.fromFilePath('./file.txt');
await smartfile.fs.copy('./a.txt', './b.txt');
// ✅ Recommended (new factory pattern)
import { SmartFileFactory } from '@push.rocks/smartfile';
const factory = SmartFileFactory.nodeFs();
const file = await factory.fromFilePath('./file.txt');
// For filesystem operations, use smartfs:
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
const smartFs = new SmartFs(new SmartFsProviderNode());
await smartFs.file('./a.txt').copy('./b.txt');
```
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
@@ -322,9 +499,9 @@ This project is owned and maintained by Task Venture Capital GmbH. The names and
### Company Information
Task Venture Capital GmbH
Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.