Compare commits

..

4 Commits

25 changed files with 1558 additions and 1790 deletions

1
.npmrc
View File

@@ -1 +0,0 @@
registry=https://registry.npmjs.org/

View File

@@ -1,5 +1,22 @@
# Changelog
## 2025-11-22 - 13.0.0 - BREAKING CHANGE(SmartFileFactory)
Refactor to in-memory file API and introduce SmartFileFactory; delegate filesystem operations to @push.rocks/smartfs; bump to 12.0.0
- Introduce SmartFileFactory as the canonical entry point for creating SmartFile, StreamFile and VirtualDirectory instances.
- Refactor SmartFile, StreamFile and VirtualDirectory to be in-memory representations and accept an optional SmartFs instance for filesystem operations.
- Delegate low-level filesystem operations to @push.rocks/smartfs (added as a peerDependency); legacy fs/memory/fsStream/interpreter namespace exports removed/deprecated.
- Add StreamFile.toSmartFile(), VirtualDirectory.loadFromDisk(), and other convenience methods to work with the new factory/smartFs integration.
- Update tests to use a MockSmartFs and the factory API; add test assets.
- Documentation and readme updated with migration instructions and examples showing SmartFileFactory.nodeFs() and SmartFs usage.
- Bumped package version to 12.0.0 — this is a breaking change; consumers should migrate from legacy namespace exports to the factory + @push.rocks/smartfs workflow.
## 2025-08-18 - 11.2.7 - fix(ci)
Remove .npmrc containing hard-coded npm registry configuration
- Removed .npmrc which contained 'registry=https://registry.npmjs.org/'
- Avoids committing environment-specific npm registry configuration; rely on user or CI environment settings instead
## 2025-08-18 - 11.2.6 - fix(fs)
Improve fs and stream handling, enhance SmartFile/StreamFile, update tests and CI configs

View File

@@ -1,8 +1,8 @@
{
"name": "@push.rocks/smartfile",
"private": false,
"version": "11.2.6",
"description": "Provides comprehensive tools for efficient file management in Node.js using TypeScript, including handling streams, virtual directories, and various file operations.",
"version": "13.0.0",
"description": "High-level file representation classes (SmartFile, StreamFile, VirtualDirectory) for efficient in-memory file management in Node.js using TypeScript. Works seamlessly with @push.rocks/smartfs for filesystem operations.",
"main": "dist_ts/index.js",
"typings": "dist_ts/index.d.ts",
"type": "module",
@@ -19,21 +19,17 @@
"file management",
"TypeScript",
"Node.js",
"file operations",
"file manipulation",
"in-memory files",
"SmartFile",
"StreamFile",
"VirtualDirectory",
"file representation",
"file streaming",
"virtual directory",
"filesystem utilities",
"file factory",
"memory-efficient file handling",
"custom file operations",
"write files",
"read files",
"copy files",
"delete files",
"list directories",
"handle large files",
"virtual filesystems",
"buffer operations"
"buffer operations",
"file content manipulation"
],
"author": "Lossless GmbH <hello@lossless.com> (https://lossless.com)",
"license": "MIT",
@@ -58,6 +54,14 @@
"glob": "^11.0.3",
"js-yaml": "^4.1.0"
},
"peerDependencies": {
"@push.rocks/smartfs": "^1.0.0"
},
"peerDependenciesMeta": {
"@push.rocks/smartfs": {
"optional": true
}
},
"devDependencies": {
"@git.zone/tsbuild": "^2.6.4",
"@git.zone/tsrun": "^1.3.3",

View File

@@ -1,5 +1,140 @@
# SmartFile Implementation Hints
## Major Architectural Change (v12.0.0)
### Overview
SmartFile has been refactored to focus exclusively on **in-memory file representations** (SmartFile, StreamFile, VirtualDirectory). All filesystem operations have been moved to or delegated to `@push.rocks/smartfs`.
### Key Changes
1. **Factory Pattern Introduction**
- New `SmartFileFactory` class introduced
- Factory is bound to a `SmartFs` instance (from `@push.rocks/smartfs`)
- All file instances are created through the factory
- Factory methods: `fromFilePath()`, `fromUrl()`, `fromBuffer()`, `fromString()`, etc.
2. **SmartFile, StreamFile, VirtualDirectory**
- Now accept optional `smartFs` parameter in constructor
- Filesystem operations (write, read, delete) use `smartFs` if available
- Fallback to legacy methods if `smartFs` not provided (for backward compatibility)
- Static factory methods moved to `SmartFileFactory`
3. **Separation of Concerns**
- **SmartFile** = In-memory file representation (path + content buffer)
- **StreamFile** = Lazy-loaded streaming file representation
- **VirtualDirectory** = Collection of SmartFiles in memory
- **SmartFs** (from @push.rocks/smartfs) = Filesystem operations
### Usage Pattern
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
// Create factory with SmartFs instance
const smartFs = new SmartFs(new SmartFsProviderNode());
const factory = new SmartFileFactory(smartFs);
// Or use default Node.js factory
const factory = SmartFileFactory.nodeFs();
// Create SmartFile through factory
const file = await factory.fromFilePath('./data.json');
await file.write(); // Uses bound smartFs instance
// Create StreamFile
const stream = await factory.streamFromPath('./large.zip');
// Create VirtualDirectory
const vdir = await factory.virtualDirectoryFromPath('./src');
```
### What Belongs Where
**SmartFile/StreamFile/VirtualDirectory (this package)**:
- ✅ In-memory file representation
- ✅ Content manipulation (edit, parse, transform)
- ✅ Loading content FROM sources (factory methods)
- ✅ Saving content TO destinations (write methods)
- ✅ Instance metadata (hash, size, mime type)
- ✅ Collection operations (for VirtualDirectory)
**SmartFs (@push.rocks/smartfs)**:
- ✅ Filesystem queries (exists, stat)
- ✅ File operations without content loading (copy, move)
- ✅ Directory operations (list, create, delete)
- ✅ Streaming operations (readStream, writeStream)
- ✅ Provider abstraction (Node.js, memory, S3, etc.)
### VirtualDirectory Collection Methods
VirtualDirectory now has comprehensive collection methods:
**Queries** (operate on in-memory collection):
- `exists(path)` / `has(path)` - Check if path exists in collection
- `getFileByPath(path)` - Get SmartFile from collection
- `listFiles()` - List all SmartFiles
- `listDirectories()` - List directory paths represented in collection
- `filter(predicate)` - Filter SmartFiles
- `map(fn)` - Transform SmartFiles
- `find(predicate)` - Find SmartFile
- `size()` - Number of files in collection
- `isEmpty()` - Check if collection is empty
**Mutations**:
- `addSmartfiles(files)` - Add files to collection
- `addSmartfile(file)` - Add single file
- `removeByPath(path)` - Remove from collection
- `clear()` - Empty collection
- `merge(otherVDir)` - Merge another VirtualDirectory
### Backward Compatibility
- Legacy namespace exports (`fs`, `memory`, `fsStream`, `interpreter`) are **deprecated**
- They remain functional for transition period but marked with `@deprecated`
- Will be removed in future version
- Users should migrate to `@push.rocks/smartfs` and `SmartFileFactory`
### Migration Path
**Old (deprecated)**:
```typescript
import * as smartfile from '@push.rocks/smartfile';
const file = await smartfile.SmartFile.fromFilePath('./file.txt');
await file.write();
const exists = await smartfile.fs.fileExists('./file.txt');
await smartfile.fs.copy('./a.txt', './b.txt');
```
**New (recommended)**:
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
const factory = SmartFileFactory.nodeFs();
const file = await factory.fromFilePath('./file.txt');
await file.write();
const smartFs = new SmartFs(new SmartFsProviderNode());
const exists = await smartFs.file('./file.txt').exists();
await smartFs.file('./a.txt').copy('./b.txt');
```
### Testing Considerations
- Tests should use `SmartFileFactory.nodeFs()` or create custom factory with memory provider
- VirtualDirectory tests can use collection methods without filesystem access
- Filesystem operations should be tested via `@push.rocks/smartfs`
### Future Plans
- Remove deprecated namespace exports completely
- Full smartfs integration (remove fallback code)
- Potentially remove fs-extra, glob dependencies once smartfs is fully integrated
## listFileTree Function Enhancement (ts/fs.ts:367-415)
### Issue Fixed

579
readme.md
View File

@@ -1,315 +1,492 @@
# @push.rocks/smartfile 📁
> **A powerful, TypeScript-based file management library for Node.js**
> **High-level file representation classes for Node.js**
## 🚀 What is smartfile?
`@push.rocks/smartfile` is your go-to solution for file operations in Node.js. It offers a clean, promise-based API for handling files, directories, streams, and even virtual filesystems - all while maintaining maximum performance and reliability.
`@push.rocks/smartfile` provides powerful **in-memory file representations** for Node.js applications. It offers clean, TypeScript-first classes for working with files (`SmartFile`), streams (`StreamFile`), and virtual file collections (`VirtualDirectory`).
Think of it as `fs` on steroids, with TypeScript superpowers! 💪
Think of it as your go-to solution for **content manipulation**, **file transformations**, and **in-memory file operations** - all while seamlessly integrating with [@push.rocks/smartfs](https://code.foss.global/push.rocks/smartfs) for actual filesystem operations.
## 💾 Installation
```bash
npm install @push.rocks/smartfile
pnpm install @push.rocks/smartfile
# Optional: Install smartfs for filesystem operations
pnpm install @push.rocks/smartfs
```
## ✨ Features
## ✨ Key Features
- 🔥 **Streaming Support** - Handle massive files with ease using `StreamFile`
- 📦 **Virtual Directories** - Work with in-memory file structures
- 🌐 **URL Support** - Directly work with files from URLs
- 🎯 **TypeScript First** - Full type safety and IntelliSense support
- **Promise-based API** - Modern async/await patterns throughout
- 🛠️ **Comprehensive Toolset** - From basic CRUD to advanced operations
- 🎯 **Factory Pattern** - Clean, consistent API for creating file instances
- 🔥 **Streaming Support** - Handle massive files efficiently with `StreamFile`
- 📦 **Virtual Directories** - Work with in-memory file collections
- 🌐 **URL Support** - Directly fetch files from URLs
- 🎨 **Content Manipulation** - Edit, transform, and parse file content
- **TypeScript First** - Full type safety and IntelliSense support
- 🛠️ **Comprehensive Collection API** - Filter, map, find files in virtual directories
## 📚 Quick Start
### Using the Factory
```typescript
import * as smartfile from '@push.rocks/smartfile';
import { SmartFileFactory } from '@push.rocks/smartfile';
// Read a file
const content = await smartfile.fs.toStringSync('./my-file.txt');
// Create factory (uses Node.js filesystem by default)
const factory = SmartFileFactory.nodeFs();
// Write a file
await smartfile.memory.toFs('Hello World!', './output.txt');
// Load a file into memory
const file = await factory.fromFilePath('./config.json');
// Work with JSON
const data = await smartfile.fs.toObjectSync('./data.json');
// Edit content
await file.editContentAsString(async (content) => {
return content.toUpperCase();
});
// Save back to disk
await file.write();
```
### With SmartFs Integration
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
// Create SmartFs instance with provider
const smartFs = new SmartFs(new SmartFsProviderNode());
// Create factory bound to this filesystem
const factory = new SmartFileFactory(smartFs);
// Now all file operations use the smartfs instance
const file = await factory.fromFilePath('./data.json');
await file.write(); // Uses smartfs under the hood
```
## 🎨 Core Components
### SmartFile Class
### SmartFileFactory
The `SmartFile` class represents a single file with powerful manipulation capabilities:
The factory is your entry point for creating all file instances:
```typescript
import { SmartFile } from '@push.rocks/smartfile';
import { SmartFileFactory } from '@push.rocks/smartfile';
// Create from file path
const fileFromPath = await SmartFile.fromFilePath('./data.json');
const factory = SmartFileFactory.nodeFs();
// Create from URL
const fileFromUrl = await SmartFile.fromUrl('https://example.com/config.json');
// Create from various sources
const fileFromPath = await factory.fromFilePath('./data.json');
const fileFromUrl = await factory.fromUrl('https://example.com/config.json');
const fileFromBuffer = factory.fromBuffer('./file.txt', Buffer.from('content'));
const fileFromString = factory.fromString('./file.txt', 'Hello World', 'utf8');
// Create from text
const fileFromText = await SmartFile.fromString(
'./my-file.txt',
'This is my content',
'utf8'
);
// Create StreamFile instances
const stream = await factory.streamFromPath('./large-file.zip');
const streamFromUrl = await factory.streamFromUrl('https://example.com/video.mp4');
// Create from Buffer
const fileFromBuffer = await SmartFile.fromBuffer(
'./binary.dat',
Buffer.from([0x00, 0x01, 0x02])
);
// Edit content
await fileFromPath.editContentAsString(async (content) => {
return content.replace(/old/g, 'new');
});
// Write to disk
await fileFromPath.write();
// Get content
const contentString = fileFromPath.parseContentAsString();
const contentBuffer = fileFromPath.parseContentAsBuffer();
// Create VirtualDirectory instances
const vdir = await factory.virtualDirectoryFromPath('./src');
const emptyVdir = factory.virtualDirectoryEmpty();
```
### StreamFile Class 🌊
### SmartFile Class
Represents a single file loaded in memory:
```typescript
// Created via factory
const file = await factory.fromFilePath('./data.json');
// Content access
const asString = file.parseContentAsString();
const asBuffer = file.parseContentAsBuffer();
// Content manipulation
await file.editContentAsString(async (content) => {
const data = JSON.parse(content);
data.updated = new Date().toISOString();
return JSON.stringify(data, null, 2);
});
// File operations
await file.write(); // Save to original location
await file.writeToDiskAtPath('./output.json'); // Save to specific path
await file.writeToDir('./dist'); // Save to directory
await file.read(); // Reload from disk
await file.delete(); // Delete from disk
// Metadata
const size = await file.getSize(); // File size in bytes
const hash = await file.getHash('content'); // SHA256 hash
const stream = file.getStream(); // Get as Node.js stream
// Path information
console.log(file.path); // Relative path
console.log(file.absolutePath); // Absolute path
console.log(file.parsedPath); // Parsed path components
```
### StreamFile Class
Perfect for handling large files without memory overhead:
```typescript
import { StreamFile } from '@push.rocks/smartfile';
// Created via factory
const streamFile = await factory.streamFromPath('./bigfile.zip');
// Create from path
const streamFile = await StreamFile.fromPath('./bigfile.zip');
// Or from URL
const urlStream = await factory.streamFromUrl('https://example.com/large.mp4');
// Create from URL
const urlStream = await StreamFile.fromUrl('https://example.com/large.mp4');
// Or from buffer
const bufferStream = factory.streamFromBuffer(Buffer.from('content'));
// Create from buffer
const bufferStream = StreamFile.fromBuffer(
Buffer.from('streaming content'),
'stream.txt'
);
// Write to disk
// Write to disk (streams the content)
await streamFile.writeToDisk('./output/bigfile.zip');
await streamFile.writeToDir('./output');
// Get as buffer (careful with large files!)
// Get content (loads into memory - use carefully!)
const buffer = await streamFile.getContentAsBuffer();
const string = await streamFile.getContentAsString('utf8');
// Get as stream
// Get as Node.js stream for piping
const readStream = await streamFile.createReadStream();
// Convert to SmartFile (loads into memory)
const smartFile = await streamFile.toSmartFile();
// Get file size
const size = await streamFile.getSize();
```
### VirtualDirectory Class 📂
### VirtualDirectory Class
Manage collections of files as virtual filesystems:
Manage collections of SmartFiles in memory:
```typescript
import { VirtualDirectory } from '@push.rocks/smartfile';
// Created via factory
const vdir = await factory.virtualDirectoryFromPath('./src');
// Create from filesystem
const vDir = await VirtualDirectory.fromFsDirPath('./src');
// Or create empty
const emptyVdir = factory.virtualDirectoryEmpty();
// Create from file array
const vDirFromFiles = await VirtualDirectory.fromFileArray([
await SmartFile.fromFilePath('./file1.txt'),
await SmartFile.fromFilePath('./file2.txt')
]);
// Or from file array
const files = [file1, file2, file3];
const vdirFromFiles = factory.virtualDirectoryFromFileArray(files);
// ============================================
// Collection Queries (in-memory operations)
// ============================================
// Check existence in collection
if (vdir.exists('components/Button.tsx')) {
console.log('File exists in virtual directory');
}
// Get file from collection
const file = await vdir.getFileByPath('utils/helpers.ts');
// List all files
const allFiles = vdir.listFiles();
// List directory paths represented in collection
const dirs = vdir.listDirectories();
// Filter files
const tsFiles = vdir.filter(f => f.path.endsWith('.ts'));
const largeFiles = vdir.filter(f => f.contentBuffer.length > 10000);
// Map/transform files
const uppercased = vdir.map(f => {
f.contentBuffer = Buffer.from(f.parseContentAsString().toUpperCase());
return f;
});
// Find specific file
const configFile = vdir.find(f => f.path.includes('config'));
// Collection info
const fileCount = vdir.size();
const empty = vdir.isEmpty();
// ============================================
// Collection Mutations
// ============================================
// Add files
vDir.addSmartfiles([
await SmartFile.fromString('./virtual/new.txt', 'content')
]);
vdir.addSmartfile(newFile);
vdir.addSmartfiles([file1, file2, file3]);
// List files
const files = vDir.listFiles();
const directories = vDir.listDirectories();
// Remove file
vdir.removeByPath('old-file.ts');
// Get file
const file = vDir.getFileByPath('./some/path.txt');
// Clear all files
vdir.clear();
// Save to disk
await vDir.saveToDisk('./output');
// Merge another virtual directory
vdir.merge(otherVirtualDir);
// Load from disk
await vDir.loadFromDisk('./source');
// ============================================
// Load/Save (filesystem bridge operations)
// ============================================
// Save all files to disk
await vdir.saveToDisk('./dist');
// Reload from disk
await vdir.loadFromDisk('./src');
// Work with subdirectories
const subVdir = await vdir.shiftToSubdirectory('components');
await vdir.addVirtualDirectory(otherVdir, 'lib');
```
## 🛠️ File Operations
## 🔄 Integration with SmartFs
### Basic Operations
For filesystem operations beyond loading/saving content, use [@push.rocks/smartfs](https://code.foss.global/push.rocks/smartfs):
```typescript
// Check existence
const exists = await smartfile.fs.fileExists('./file.txt');
const existsSync = smartfile.fs.fileExistsSync('./file.txt');
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
// Read operations
const content = await smartfile.fs.toStringSync('./file.txt');
const buffer = await smartfile.fs.toBuffer('./file.txt');
const object = await smartfile.fs.toObjectSync('./data.json');
const smartFs = new SmartFs(new SmartFsProviderNode());
const factory = new SmartFileFactory(smartFs);
// Write operations
await smartfile.memory.toFs('content', './output.txt');
smartfile.memory.toFsSync('content', './output-sync.txt');
// Use smartfile for content manipulation
const file = await factory.fromFilePath('./config.json');
await file.editContentAsString(async (s) => s.toUpperCase());
await file.write();
// Copy operations
await smartfile.fs.copy('./source.txt', './dest.txt');
await smartfile.fs.copy('./src-dir', './dest-dir');
// Use smartfs for filesystem operations
const exists = await smartFs.file('./config.json').exists();
await smartFs.file('./config.json').copy('./config.backup.json');
const stats = await smartFs.file('./config.json').stat();
// Delete operations
await smartfile.fs.remove('./file.txt');
await smartfile.fs.removeSync('./file-sync.txt');
await smartfile.fs.removeMany(['./file1.txt', './file2.txt']);
// Ensure operations (create if not exists)
await smartfile.fs.ensureDir('./my/deep/directory');
await smartfile.fs.ensureFile('./my/file.txt');
await smartfile.fs.ensureEmptyDir('./empty-dir');
// List directory with smartfs
const entries = await smartFs.directory('./src').list();
```
### Directory Operations
## 🌟 Common Use Cases
### Configuration File Management
```typescript
// List contents
const files = await smartfile.fs.listFiles('./directory');
const folders = await smartfile.fs.listFolders('./directory');
const items = await smartfile.fs.listAllItems('./directory');
const factory = SmartFileFactory.nodeFs();
// Get file tree
const tree = await smartfile.fs.listFileTree('./src', '**/*.ts');
// Directory checks
const isDir = await smartfile.fs.isDirectory('./path');
const isFile = await smartfile.fs.isFile('./path');
```
### Advanced Features
```typescript
// Wait for file to be ready
await smartfile.fs.waitForFileToBeReady('./file.txt');
// Stream operations
const readStream = smartfile.fsStream.createReadStream('./input.txt');
const writeStream = smartfile.fsStream.createWriteStream('./output.txt');
// File type detection
const fileType = smartfile.interpreter.filetype('./document.pdf');
// Returns: 'pdf'
// Smart read stream (with custom processing)
const smartStream = new smartfile.fsStream.SmartReadStream('./data.txt');
smartStream.on('data', (chunk) => {
// Process chunk
console.log(chunk.toString());
// Load, modify, and save config
const config = await factory.fromFilePath('./package.json');
await config.editContentAsString(async (content) => {
const pkg = JSON.parse(content);
pkg.version = '2.0.0';
return JSON.stringify(pkg, null, 2);
});
await config.write();
```
## 🔄 Working with Multiple Files
### Batch File Processing
```typescript
// Process multiple SmartFiles
const files = await smartfile.fs.fileTreeToObject(
'./src',
'**/*.{ts,js}'
);
const factory = SmartFileFactory.nodeFs();
// Write array to disk
const smartfiles = [
await SmartFile.fromString('file1.txt', 'content1'),
await SmartFile.fromString('file2.txt', 'content2')
];
await smartfile.memory.smartfileArrayToFs(smartfiles, './output');
```
// Load directory into virtual collection
const vdir = await factory.virtualDirectoryFromPath('./content');
## 🎯 Real-World Examples
### Website Bundler
```typescript
// Bundle website assets
const website = await VirtualDirectory.fromFsDirPath('./website');
const bundle = await website.smartfileArray;
// Process all CSS files
for (const file of bundle.filter(f => f.path.endsWith('.css'))) {
await file.editContentAsString(async (css) => {
// Minify CSS here
return css.replace(/\s+/g, ' ');
// Process all markdown files
const mdFiles = vdir.filter(f => f.path.endsWith('.md'));
for (const file of mdFiles.listFiles()) {
await file.editContentAsString(async (content) => {
// Add frontmatter, transform links, etc.
return `---\nprocessed: true\n---\n\n${content}`;
});
}
// Save processed bundle
await website.saveToDisk('./dist');
// Save processed files
await vdir.saveToDisk('./dist/content');
```
### File Watcher & Processor
### Download and Process Remote Files
```typescript
// Watch for new files and process them
import { SmartFile, StreamFile } from '@push.rocks/smartfile';
const factory = SmartFileFactory.nodeFs();
async function processLargeFile(filePath: string) {
const streamFile = await StreamFile.fromPath(filePath);
// Fetch from URL
const remoteFile = await factory.fromUrl('https://api.example.com/data.json');
// Stream to processed location
await streamFile.writeToDisk(`./processed/${path.basename(filePath)}`);
// Process content
await remoteFile.editContentAsString(async (content) => {
const data = JSON.parse(content);
// Transform data
return JSON.stringify(data.results, null, 2);
});
// Clean up original
await smartfile.fs.remove(filePath);
}
// Save locally
await remoteFile.writeToDiskAtPath('./cache/data.json');
```
### Configuration Manager
### Large File Streaming
```typescript
// Load and merge config files
const defaultConfig = await smartfile.fs.toObjectSync('./config.default.json');
const userConfig = await smartfile.fs.toObjectSync('./config.user.json');
const factory = SmartFileFactory.nodeFs();
const merged = { ...defaultConfig, ...userConfig };
// Download large file as stream
const largeFile = await factory.streamFromUrl('https://example.com/large-dataset.csv');
await smartfile.memory.toFs(
JSON.stringify(merged, null, 2),
'./config.final.json'
);
// Save to disk (streams, doesn't load all into memory)
await largeFile.writeToDisk('./data/dataset.csv');
// Or get size without downloading entire file
const size = await largeFile.getSize();
console.log(`File size: ${size} bytes`);
```
## 🌟 API Reference
### Virtual File System for Testing
### Core Modules
```typescript
import { SmartFileFactory } from '@push.rocks/smartfile';
import { SmartFs, SmartFsProviderMemory } from '@push.rocks/smartfs';
- `fs` - File system operations
- `fsStream` - Streaming operations
- `memory` - Memory/buffer operations
- `interpreter` - File type detection
// Use in-memory filesystem for tests
const memoryFs = new SmartFs(new SmartFsProviderMemory());
const factory = new SmartFileFactory(memoryFs);
### Main Classes
// Create virtual files
const testFile = factory.fromString('test.txt', 'test content');
await testFile.write(); // Writes to in-memory filesystem
- `SmartFile` - Single file representation
- `StreamFile` - Streaming file operations
- `VirtualDirectory` - Virtual filesystem management
// Test your code without touching real filesystem
```
## 🏗️ TypeScript Support
## 🏗️ Architecture
### Responsibility Split
**@push.rocks/smartfile** (this package):
- ✅ In-memory file representations (SmartFile, StreamFile, VirtualDirectory)
- ✅ Content manipulation and transformation
- ✅ Loading content FROM sources (disk, URL, buffer, string)
- ✅ Saving content TO destinations (disk, stream)
- ✅ Collection operations (filter, map, find on VirtualDirectory)
**@push.rocks/smartfs**:
- ✅ Low-level filesystem operations (exists, stat, copy, move, delete)
- ✅ Directory operations (list, create, remove)
- ✅ Provider abstraction (Node.js fs, in-memory, S3, etc.)
- ✅ Streaming (readStream, writeStream)
- ✅ Transactions and file watching
## 📖 API Reference
### SmartFileFactory
| Method | Description |
|--------|-------------|
| `SmartFileFactory.nodeFs()` | Create factory with Node.js filesystem provider |
| `new SmartFileFactory(smartFs)` | Create factory with custom SmartFs instance |
| `factory.fromFilePath(path, base?)` | Load file from disk into SmartFile |
| `factory.fromUrl(url)` | Fetch file from URL into SmartFile |
| `factory.fromBuffer(path, buffer, base?)` | Create SmartFile from Buffer |
| `factory.fromString(path, content, encoding, base?)` | Create SmartFile from string |
| `factory.streamFromPath(path)` | Create StreamFile from disk |
| `factory.streamFromUrl(url)` | Create StreamFile from URL |
| `factory.streamFromBuffer(buffer, path?)` | Create StreamFile from Buffer |
| `factory.virtualDirectoryFromPath(path)` | Load directory into VirtualDirectory |
| `factory.virtualDirectoryEmpty()` | Create empty VirtualDirectory |
| `factory.virtualDirectoryFromFileArray(files)` | Create VirtualDirectory from SmartFiles |
### SmartFile Instance Methods
| Method | Description |
|--------|-------------|
| `file.write()` | Save to original location |
| `file.writeToDiskAtPath(path)` | Save to specific path |
| `file.writeToDir(dir)` | Save to directory (preserves relative path) |
| `file.read()` | Reload content from disk |
| `file.delete()` | Delete file from disk |
| `file.editContentAsString(fn)` | Transform content as string |
| `file.parseContentAsString(encoding?)` | Get content as string |
| `file.parseContentAsBuffer()` | Get content as Buffer |
| `file.getHash(type?)` | Get SHA256 hash ('path', 'content', 'all') |
| `file.getSize()` | Get content size in bytes |
| `file.getStream()` | Get content as Node.js Readable stream |
### StreamFile Instance Methods
| Method | Description |
|--------|-------------|
| `stream.writeToDisk(path)` | Stream content to disk |
| `stream.writeToDir(dir)` | Stream to directory |
| `stream.createReadStream()` | Get as Node.js Readable stream |
| `stream.getContentAsBuffer()` | Load entire content into Buffer |
| `stream.getContentAsString(encoding?)` | Load entire content as string |
| `stream.getSize()` | Get content size in bytes |
| `stream.toSmartFile()` | Convert to SmartFile (loads into memory) |
### VirtualDirectory Instance Methods
**Collection Queries:**
| Method | Description |
|--------|-------------|
| `vdir.exists(path)` | Check if file exists in collection |
| `vdir.has(path)` | Alias for exists() |
| `vdir.getFileByPath(path)` | Get SmartFile by path |
| `vdir.listFiles()` | Get all SmartFiles |
| `vdir.listDirectories()` | Get all directory paths |
| `vdir.filter(predicate)` | Filter files, returns new VirtualDirectory |
| `vdir.map(fn)` | Transform files, returns new VirtualDirectory |
| `vdir.find(predicate)` | Find first matching file |
| `vdir.size()` | Get file count |
| `vdir.isEmpty()` | Check if empty |
**Collection Mutations:**
| Method | Description |
|--------|-------------|
| `vdir.addSmartfile(file)` | Add single file |
| `vdir.addSmartfiles(files)` | Add multiple files |
| `vdir.removeByPath(path)` | Remove file by path |
| `vdir.clear()` | Remove all files |
| `vdir.merge(otherVdir)` | Merge another VirtualDirectory |
**Load/Save:**
| Method | Description |
|--------|-------------|
| `vdir.saveToDisk(dir)` | Write all files to disk |
| `vdir.loadFromDisk(dir)` | Load files from disk (replaces collection) |
## 🔧 TypeScript Support
Full TypeScript support with comprehensive type definitions:
```typescript
import type { SmartFile, StreamFile, VirtualDirectory } from '@push.rocks/smartfile';
import type { SmartFile, StreamFile, VirtualDirectory, SmartFileFactory } from '@push.rocks/smartfile';
// All methods are fully typed
const processFile = async (file: SmartFile): Promise<void> => {
const content = file.parseContentAsString();
// TypeScript knows content is string
};
```
## 📦 Backward Compatibility
Version 12.0.0 introduces the factory pattern. Legacy exports are deprecated but still functional:
```typescript
// ⚠️ Deprecated (still works, but will be removed)
import * as smartfile from '@push.rocks/smartfile';
const file = await smartfile.SmartFile.fromFilePath('./file.txt');
await smartfile.fs.copy('./a.txt', './b.txt');
// ✅ Recommended (new factory pattern)
import { SmartFileFactory } from '@push.rocks/smartfile';
const factory = SmartFileFactory.nodeFs();
const file = await factory.fromFilePath('./file.txt');
// For filesystem operations, use smartfs:
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
const smartFs = new SmartFs(new SmartFsProviderNode());
await smartFs.file('./a.txt').copy('./b.txt');
```
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.

View File

@@ -0,0 +1,95 @@
/**
* Mock SmartFs implementation for testing until @push.rocks/smartfs is available
* This wraps fs-extra to provide the SmartFs interface
*/
import { ensureDir, pathExists, remove, copy } from 'fs-extra';
import { promises as fsPromises, createReadStream, createWriteStream } from 'fs';
import * as path from 'path';
import { Readable, Writable } from 'stream';
export class MockSmartFs {
public file(filePath: string) {
return {
async read(): Promise<string | Buffer> {
return await fsPromises.readFile(filePath);
},
async write(content: string | Buffer): Promise<void> {
await ensureDir(path.dirname(filePath));
await fsPromises.writeFile(filePath, content);
},
async exists(): Promise<boolean> {
return await pathExists(filePath);
},
async delete(): Promise<void> {
await remove(filePath);
},
async stat(): Promise<any> {
return await fsPromises.stat(filePath);
},
async readStream(): Promise<Readable> {
return Promise.resolve(createReadStream(filePath));
},
async writeStream(): Promise<Writable> {
await ensureDir(path.dirname(filePath));
return Promise.resolve(createWriteStream(filePath));
},
async copy(dest: string): Promise<void> {
await copy(filePath, dest);
},
};
}
public directory(dirPath: string) {
return {
async list(options?: { recursive?: boolean }): Promise<Array<{ path: string; isFile: boolean; isDirectory: boolean }>> {
const entries: Array<{ path: string; isFile: boolean; isDirectory: boolean }> = [];
if (options?.recursive) {
// Recursive listing
const walk = async (dir: string) => {
const items = await fsPromises.readdir(dir);
for (const item of items) {
const fullPath = path.join(dir, item);
const stats = await fsPromises.stat(fullPath);
if (stats.isFile()) {
entries.push({ path: fullPath, isFile: true, isDirectory: false });
} else if (stats.isDirectory()) {
entries.push({ path: fullPath, isFile: false, isDirectory: true });
await walk(fullPath);
}
}
};
await walk(dirPath);
} else {
// Non-recursive listing
const items = await fsPromises.readdir(dirPath);
for (const item of items) {
const fullPath = path.join(dirPath, item);
const stats = await fsPromises.stat(fullPath);
entries.push({
path: fullPath,
isFile: stats.isFile(),
isDirectory: stats.isDirectory(),
});
}
}
return entries;
},
async create(options?: { recursive?: boolean }): Promise<void> {
if (options?.recursive) {
await ensureDir(dirPath);
} else {
await fsPromises.mkdir(dirPath);
}
},
async exists(): Promise<boolean> {
return await pathExists(dirPath);
},
async delete(): Promise<void> {
await remove(dirPath);
},
};
}
}

View File

@@ -1,18 +1,23 @@
import * as path from 'path';
import { expect, tap } from '@git.zone/tstest/tapbundle';
import * as smartfile from '../ts/index.js'; // adjust the import path as needed
import * as smartfile from '../ts/index.js';
import { MockSmartFs } from './helpers/mock-smartfs.js';
// Create factory with MockSmartFs
const mockFs = new MockSmartFs();
const factory = new smartfile.SmartFileFactory(mockFs);
// Test assets path
const testAssetsPath = './test/testassets/';
// ---------------------------
// StreamFile tests
// StreamFile Factory Tests
// ---------------------------
tap.test(
'StreamFile.fromPath should create a StreamFile from a file path',
'SmartFileFactory.streamFromPath() -> should create a StreamFile from a file path',
async () => {
const streamFile = await smartfile.StreamFile.fromPath(
const streamFile = await factory.streamFromPath(
path.join(testAssetsPath, 'mytest.json'),
);
expect(streamFile).toBeInstanceOf(smartfile.StreamFile);
@@ -22,75 +27,128 @@ tap.test(
);
tap.test(
'StreamFile.fromUrl should create a StreamFile from a URL',
async () => {
const streamFile = await smartfile.StreamFile.fromUrl(
'http://example.com/somefile.json',
);
expect(streamFile).toBeInstanceOf(smartfile.StreamFile);
},
);
tap.test(
'StreamFile.fromBuffer should create a StreamFile from a Buffer',
'SmartFileFactory.streamFromBuffer() -> should create a StreamFile from a Buffer',
async () => {
const buffer = Buffer.from('Some content');
const streamFile = smartfile.StreamFile.fromBuffer(
const streamFile = factory.streamFromBuffer(
buffer,
'bufferfile.txt',
);
expect(streamFile).toBeInstanceOf(smartfile.StreamFile);
const content = await streamFile.getContentAsBuffer();
expect(content.toString()).toEqual('Some content');
},
);
tap.test('StreamFile should write the stream to disk', async () => {
const streamFile = await smartfile.StreamFile.fromPath(
tap.test(
'SmartFileFactory.streamFromStream() -> should create a StreamFile from a stream',
async () => {
const { Readable } = await import('stream');
const stream = new Readable();
stream.push('stream content');
stream.push(null);
const streamFile = factory.streamFromStream(stream, 'streamfile.txt', false);
expect(streamFile).toBeInstanceOf(smartfile.StreamFile);
},
);
// ---------------------------
// StreamFile Instance Tests
// ---------------------------
tap.test('StreamFile -> should write the stream to disk', async () => {
const streamFile = await factory.streamFromPath(
path.join(testAssetsPath, 'mytest.json'),
);
await streamFile.writeToDisk(
path.join(testAssetsPath, 'temp', 'mytest.json'),
);
// Verify the file was written
expect(
// We'll use the fileExists method from your smartfile library
// Replace with the actual method you use to check file existence
await smartfile.fs.fileExists(
path.join(testAssetsPath, 'temp', 'mytest.json'),
),
).toBeTrue();
const targetPath = path.join(testAssetsPath, 'temp', 'stream-mytest.json');
await streamFile.writeToDisk(targetPath);
// Verify the file was written by reading it back
const verifyFile = await factory.fromFilePath(targetPath);
expect(verifyFile.contentBuffer).toBeInstanceOf(Buffer);
});
tap.test('StreamFile should write to a directory', async () => {
const streamFile = await smartfile.StreamFile.fromPath(
tap.test('StreamFile -> should write to a directory', async () => {
const streamFile = await factory.streamFromPath(
path.join(testAssetsPath, 'mytest.json'),
);
// Set relative path so writeToDir knows where to put it
streamFile.relativeFilePath = 'mytest-fromdir.json';
await streamFile.writeToDir(path.join(testAssetsPath, 'temp'));
// Verify the file was written
expect(
await smartfile.fs.fileExists(
path.join(testAssetsPath, 'temp', 'mytest.json'),
),
).toBeTrue();
const targetPath = path.join(testAssetsPath, 'temp', 'mytest-fromdir.json');
const verifyFile = await factory.fromFilePath(targetPath);
expect(verifyFile.contentBuffer).toBeInstanceOf(Buffer);
});
tap.test('StreamFile should return content as a buffer', async () => {
const streamFile = await smartfile.StreamFile.fromPath(
tap.test('StreamFile -> should return content as a buffer', async () => {
const streamFile = await factory.streamFromPath(
path.join(testAssetsPath, 'mytest.json'),
);
const contentBuffer = await streamFile.getContentAsBuffer();
expect(contentBuffer).toBeInstanceOf(Buffer);
// Further checks on the content can be added here if necessary
});
tap.test('StreamFile should return content as a string', async () => {
const streamFile = await smartfile.StreamFile.fromPath(
tap.test('StreamFile -> should return content as a string', async () => {
const streamFile = await factory.streamFromPath(
path.join(testAssetsPath, 'mytest.json'),
);
const contentString = await streamFile.getContentAsString();
expect(contentString).toBeTypeofString();
// Verify the content matches what's expected
// This assumes the file contains a JSON object with a key 'key1' with value 'this works'
expect(JSON.parse(contentString).key1).toEqual('this works');
const parsed = JSON.parse(contentString);
expect(parsed.key1).toEqual('this works');
});
tap.test('StreamFile -> should get size', async () => {
const buffer = Buffer.from('test content for size');
const streamFile = factory.streamFromBuffer(buffer, 'sizefile.txt');
const size = await streamFile.getSize();
expect(size).toEqual(buffer.length);
});
tap.test('StreamFile -> should handle multi-use streams', async () => {
const buffer = Buffer.from('multi-use content');
const streamFile = factory.streamFromBuffer(buffer, 'multiuse.txt');
streamFile.multiUse = true;
// Read multiple times
const content1 = await streamFile.getContentAsString();
const content2 = await streamFile.getContentAsString();
expect(content1).toEqual('multi-use content');
expect(content2).toEqual('multi-use content');
});
tap.test('StreamFile -> should convert to SmartFile', async () => {
const buffer = Buffer.from('convert to smartfile');
const streamFile = factory.streamFromBuffer(buffer, 'convert.txt');
const smartFile = await streamFile.toSmartFile();
expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
expect(smartFile.parseContentAsString()).toEqual('convert to smartfile');
});
tap.test('StreamFile -> should create readable stream', async () => {
const buffer = Buffer.from('readable stream content');
const streamFile = factory.streamFromBuffer(buffer, 'readable.txt');
const stream = await streamFile.createReadStream();
expect(stream).toHaveProperty('pipe');
// Read from stream
const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
await new Promise((resolve) => {
stream.on('end', resolve);
});
const content = Buffer.concat(chunks).toString();
expect(content).toEqual('readable stream content');
});
// Start the test sequence

View File

@@ -1,353 +1,142 @@
import * as smartfile from '../ts/index.js';
import * as path from 'path';
import { expect, tap } from '@git.zone/tstest/tapbundle';
import { MockSmartFs } from './helpers/mock-smartfs.js';
// Create factory with MockSmartFs
const mockFs = new MockSmartFs();
const factory = new smartfile.SmartFileFactory(mockFs);
// ---------------------------
// smartfile.fs
// SmartFileFactory Tests
// ---------------------------
tap.test(
'.fs.fileExistsSync -> should return an accurate boolean',
async () => {
// tslint:disable-next-line: no-unused-expression
expect(
smartfile.fs.fileExistsSync('./test/testassets/mytest.json'),
).toBeTrue();
// tslint:disable-next-line: no-unused-expression
expect(
smartfile.fs.fileExistsSync('./test/testassets/notthere.json'),
).toBeFalse();
},
);
tap.test('.fs.fileExists -> should resolve or reject a promise', async () => {
await expect(
smartfile.fs.fileExists('./test/testassets/mytest.json'),
).resolves.toBeTrue();
await expect(
smartfile.fs.fileExists('./test/testassets/notthere.json'),
).resolves.toBeFalse();
tap.test('SmartFileFactory.nodeFs() -> should create a default factory', async () => {
const defaultFactory = smartfile.SmartFileFactory.nodeFs();
expect(defaultFactory).toBeInstanceOf(smartfile.SmartFileFactory);
});
tap.test(
'.fs.listFoldersSync() -> should get the file type from a string',
async () => {
expect(smartfile.fs.listFoldersSync('./test/testassets/')).toContain(
'testfolder',
);
expect(smartfile.fs.listFoldersSync('./test/testassets/')).not.toContain(
'notExistentFolder',
);
},
);
tap.test(
'.fs.listFolders() -> should get the file type from a string',
async () => {
const folderArrayArg = await smartfile.fs.listFolders('./test/testassets/');
expect(folderArrayArg).toContain('testfolder');
expect(folderArrayArg).not.toContain('notExistentFolder');
},
);
tap.test(
'.fs.listFilesSync() -> should get the file type from a string',
async () => {
expect(smartfile.fs.listFilesSync('./test/testassets/')).toContain(
'mytest.json',
);
expect(smartfile.fs.listFilesSync('./test/testassets/')).not.toContain(
'notExistentFile',
);
expect(
smartfile.fs.listFilesSync('./test/testassets/', /mytest\.json/),
).toContain('mytest.json');
expect(
smartfile.fs.listFilesSync('./test/testassets/', /mytests.json/),
).not.toContain('mytest.json');
},
);
tap.test(
'.fs.listFiles() -> should get the file type from a string',
async () => {
const folderArrayArg = await smartfile.fs.listFiles('./test/testassets/');
expect(folderArrayArg).toContain('mytest.json');
expect(folderArrayArg).not.toContain('notExistentFile');
},
);
tap.test('.fs.listFileTree() -> should get a file tree', async () => {
const folderArrayArg = await smartfile.fs.listFileTree(
path.resolve('./test/testassets/'),
'**/*.txt',
);
expect(folderArrayArg).toContain('testfolder/testfile1.txt');
expect(folderArrayArg).not.toContain('mytest.json');
tap.test('SmartFileFactory.fromFilePath() -> should create a SmartFile from file path', async () => {
const smartFile = await factory.fromFilePath('./test/testassets/mytest.json', process.cwd());
expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
expect(smartFile.path).toEqual('test/testassets/mytest.json');
expect(smartFile.contentBuffer).toBeInstanceOf(Buffer);
});
tap.test(
'.fs.listFileTree() -> should find both root and nested .ts files with **/*.ts pattern',
async () => {
const tsFiles = await smartfile.fs.listFileTree(process.cwd(), '**/*.ts');
// Should find both root-level and nested TypeScript files
expect(tsFiles).toContain('ts/index.ts');
expect(tsFiles).toContain('ts/classes.smartfile.ts');
expect(tsFiles).toContain('test/test.ts');
// Should find files in multiple levels of nesting
expect(tsFiles.filter((f) => f.endsWith('.ts')).length).toBeGreaterThan(5);
// Verify it finds files at all levels (root 'ts/' and nested 'test/')
const hasRootLevelTs = tsFiles.some(
(f) => f.startsWith('ts/') && f.endsWith('.ts'),
);
const hasNestedTs = tsFiles.some(
(f) => f.startsWith('test/') && f.endsWith('.ts'),
);
expect(hasRootLevelTs).toBeTrue();
expect(hasNestedTs).toBeTrue();
},
);
tap.test(
'.fs.listFileTree() -> should handle edge cases with **/ patterns consistently',
async () => {
// Test that our fix ensures no duplicate files in results
const jsonFiles = await smartfile.fs.listFileTree(
path.resolve('./test/testassets/'),
'**/*.json',
);
const uniqueFiles = [...new Set(jsonFiles)];
expect(jsonFiles.length).toEqual(uniqueFiles.length);
// Test that it finds root level files with **/ patterns
const txtFiles = await smartfile.fs.listFileTree(
path.resolve('./test/testassets/'),
'**/*.txt',
);
// Should include both direct files and nested files
expect(txtFiles).toContain('mytest.txt');
expect(txtFiles).toContain('testfolder/testfile1.txt');
},
);
tap.test(
'.fs.fileTreeToObject -> should read a file tree into an Object',
async () => {
const fileArrayArg = await smartfile.fs.fileTreeToObject(
path.resolve('./test/testassets/'),
'**/*.txt',
);
expect(fileArrayArg[0]).toBeInstanceOf(smartfile.SmartFile);
expect(fileArrayArg[0].contents.toString()).toEqual(
fileArrayArg[0].contentBuffer.toString(),
);
},
);
tap.test('.fs.copy() -> should copy a directory', async () => {
await smartfile.fs.copy(
'./test/testassets/testfolder/',
'./test/testassets/temp/',
);
tap.test('SmartFileFactory.fromBuffer() -> should create a SmartFile from buffer', async () => {
const buffer = Buffer.from('test content');
const smartFile = factory.fromBuffer('./test.txt', buffer);
expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
expect(smartFile.contentBuffer.toString()).toEqual('test content');
});
tap.test('.fs.copy() -> should copy a file', async () => {
await smartfile.fs.copy(
'./test/testassets/mytest.yaml',
'./test/testassets/temp/mytest.yaml',
);
tap.test('SmartFileFactory.fromString() -> should create a SmartFile from string', async () => {
const smartFile = factory.fromString('./test.txt', 'test content');
expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
expect(smartFile.parseContentAsString()).toEqual('test content');
});
tap.test('.fs.copy() -> should copy a file and rename it', async () => {
await smartfile.fs.copy(
'./test/testassets/mytest.yaml',
'./test/testassets/temp/mytestRenamed.yaml',
);
});
tap.test('.fs.remove() -> should remove an entire directory', async () => {});
tap.test('.fs.remove -> should remove single files', async () => {
await smartfile.fs.remove('./test/testassets/temp/mytestRenamed.yaml');
});
tap.test(
'.fs.removeSync -> should remove single files synchronouly',
async () => {
smartfile.fs.removeSync('./test/testassets/temp/testfile1.txt');
expect(
smartfile.fs.fileExistsSync('./test/testassets/temp/testfile1.txt'),
).toBeFalse();
},
);
tap.test('.fs.removeMany -> should remove and array of files', async () => {
smartfile.fs
.removeMany([
'./test/testassets/temp/testfile1.txt',
'./test/testassets/temp/testfile2.txt',
])
.then(() => {
expect(
smartfile.fs.fileExistsSync('./test/testassets/temp/testfile1.txt'),
).toBeFalse();
expect(
smartfile.fs.fileExistsSync('./test/testassets/temp/testfile2.txt'),
).toBeFalse();
});
});
tap.test(
'.fs.removeManySync -> should remove and array of single files synchronouly',
async () => {
smartfile.fs.removeManySync([
'./test/testassets/temp/testfile1.txt',
'./test/testassets/temp/testfile2.txt',
]);
expect(
smartfile.fs.fileExistsSync('./test/testassets/temp/testfile1.txt'),
).toBeFalse();
expect(
smartfile.fs.fileExistsSync('./test/testassets/temp/testfile2.txt'),
).toBeFalse();
},
);
tap.test(
'.fs.toObjectSync() -> should read an .yaml file to an object',
async () => {
const testData = smartfile.fs.toObjectSync('./test/testassets/mytest.yaml');
expect(testData.key1).toEqual('this works');
expect(testData.key2).toEqual('this works too');
},
);
tap.test(
'.fs.toObjectSync() -> should state unknown file type for unknown file types',
async () => {
const testData = smartfile.fs.toObjectSync('./test/testassets/mytest.txt');
},
);
tap.test(
'.fs.toObjectSync() -> should read an .json file to an object',
async () => {
const testData = smartfile.fs.toObjectSync('./test/testassets/mytest.json');
expect(testData.key1).toEqual('this works');
expect(testData.key2).toEqual('this works too');
},
);
tap.test('.fs.toStringSync() -> should read a file to a string', async () => {
expect(smartfile.fs.toStringSync('./test/testassets/mytest.txt')).toEqual(
'Some TestString &&%$',
);
tap.test('SmartFileFactory.fromUrl() -> should create a SmartFile from URL', async () => {
// Note: This test would need a real HTTP endpoint or mock
// For now, we'll skip it or test with a known URL
// const smartFile = await factory.fromUrl('https://example.com/test.json');
// expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
});
// ---------------------------
// smartfile.interpreter
// SmartFile Instance Tests
// ---------------------------
tap.test(
'.interpreter.filetype() -> should get the file type from a string',
async () => {
expect(smartfile.interpreter.filetype('./somefolder/data.json')).toEqual(
'json',
);
},
);
// ---------------------------
// smartfile.memory
// ---------------------------
tap.test(
'.memory.toFs() -> should write a file to disk and return a promise',
async () => {
const localString = 'myString';
await smartfile.memory.toFs(
localString,
path.join(process.cwd(), './test/testassets/temp/testMemToFs.txt'),
);
},
);
tap.test(
'.memory.toFsSync() -> should write a file to disk and return true if successfull',
async () => {
const localString = 'myString';
smartfile.memory.toFsSync(
localString,
path.join(process.cwd(), './test/testassets/temp/testMemToFsSync.txt'),
);
},
);
// ---------------------------
// smartfile.Smartfile
// ---------------------------
tap.test('.Smartfile -> should produce vinyl compatible files', async () => {
const smartfileArray = await smartfile.fs.fileTreeToObject(
process.cwd(),
'./test/testassets/testfolder/**/*',
);
const localSmartfile = smartfileArray[0];
expect(localSmartfile).toBeInstanceOf(smartfile.SmartFile);
expect(localSmartfile.contents).toBeInstanceOf(Buffer);
// tslint:disable-next-line:no-unused-expression
expect(localSmartfile.isBuffer()).toBeTrue();
// tslint:disable-next-line:no-unused-expression
expect(localSmartfile.isDirectory()).toBeFalse();
// tslint:disable-next-line:no-unused-expression
expect(localSmartfile.isNull()).toBeFalse();
tap.test('SmartFile -> should produce vinyl compatible files', async () => {
const smartFile = await factory.fromFilePath('./test/testassets/mytest.json');
expect(smartFile).toBeInstanceOf(smartfile.SmartFile);
expect(smartFile.contents).toBeInstanceOf(Buffer);
expect(smartFile.isBuffer()).toBeTrue();
expect(smartFile.isDirectory()).toBeFalse();
expect(smartFile.isNull()).toBeFalse();
});
tap.test('should output a smartfile array to disk', async () => {
const smartfileArray = await smartfile.fs.fileTreeToObject(
'./test/testassets/testfolder/',
'*',
);
for (const smartfileInstance of smartfileArray) {
console.log(smartfileInstance.relative);
console.log(smartfileInstance.path);
console.log(smartfileInstance.base);
console.log(smartfileInstance.parsedPath);
}
await smartfile.memory.smartfileArrayToFs(
smartfileArray,
path.resolve('./test/testassets/temp/testoutput/'),
);
});
tap.test('should create, store and retrieve valid smartfiles', async () => {
tap.test('SmartFile -> should write to disk', async () => {
const fileString = 'hi there';
const filePath = './test/testassets/utf8.txt';
const smartfileInstance = await smartfile.SmartFile.fromString(
filePath,
fileString,
'utf8',
);
smartfileInstance.write();
const smartfileInstance2 = await smartfile.SmartFile.fromFilePath(filePath);
const retrievedString = smartfileInstance.contents.toString();
const filePath = './test/testassets/temp/utf8.txt';
const smartFile = factory.fromString(filePath, fileString, 'utf8');
await smartFile.writeToDiskAtPath(filePath);
// Read it back
const smartFile2 = await factory.fromFilePath(filePath);
const retrievedString = smartFile2.parseContentAsString();
expect(retrievedString).toEqual(fileString);
});
tap.test('should get a hash', async () => {
tap.test('SmartFile -> should get a hash', async () => {
const fileString = 'hi there';
const filePath = './test/testassets/utf8.txt';
const smartfileInstance = await smartfile.SmartFile.fromString(
filePath,
fileString,
'utf8',
);
const hash = await smartfileInstance.getHash();
console.log(hash);
const smartFile = factory.fromString('./test/testassets/utf8.txt', fileString, 'utf8');
const hash = await smartFile.getHash();
expect(hash).toBeTypeofString();
expect(hash.length).toBeGreaterThan(0);
});
tap.test('should wait for file to be ready', async () => {
await smartfile.fs.waitForFileToBeReady('./test/testassets/mytest.json');
tap.test('SmartFile -> should update file name', async () => {
const smartFile = factory.fromString('./test/oldname.txt', 'content');
smartFile.updateFileName('newname.txt');
expect(smartFile.parsedPath.base).toEqual('newname.txt');
});
tap.test('SmartFile -> should edit content as string', async () => {
const smartFile = factory.fromString('./test.txt', 'original content');
await smartFile.editContentAsString(async (content) => {
return content.replace('original', 'modified');
});
expect(smartFile.parseContentAsString()).toEqual('modified content');
});
tap.test('SmartFile -> should get stream', async () => {
const smartFile = factory.fromString('./test.txt', 'stream content');
const stream = smartFile.getStream();
expect(stream).toHaveProperty('pipe');
// Read from stream
const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
await new Promise((resolve) => {
stream.on('end', resolve);
});
const content = Buffer.concat(chunks).toString();
expect(content).toEqual('stream content');
});
tap.test('SmartFile -> should get size', async () => {
const content = 'test content with some length';
const smartFile = factory.fromString('./test.txt', content);
const size = await smartFile.getSize();
expect(size).toEqual(Buffer.from(content).length);
});
tap.test('SmartFile -> should parse content as buffer', async () => {
const buffer = Buffer.from('buffer content');
const smartFile = factory.fromBuffer('./test.txt', buffer);
const parsedBuffer = smartFile.parseContentAsBuffer();
expect(parsedBuffer).toBeInstanceOf(Buffer);
expect(parsedBuffer.toString()).toEqual('buffer content');
});
tap.test('SmartFile -> should write to directory', async () => {
const smartFile = factory.fromString('subdir/test.txt', 'directory test content');
const writtenPath = await smartFile.writeToDir('./test/testassets/temp');
expect(writtenPath).toContain('subdir/test.txt');
});
tap.test('SmartFile -> should get parsed path', async () => {
const smartFile = factory.fromString('./path/to/file.txt', 'content');
expect(smartFile.parsedPath.base).toEqual('file.txt');
expect(smartFile.parsedPath.ext).toEqual('.txt');
expect(smartFile.parsedPath.name).toEqual('file');
});
tap.test('SmartFile -> should get absolute path', async () => {
const smartFile = factory.fromString('relative/path.txt', 'content', 'utf8', '/base');
expect(smartFile.absolutePath).toEqual('/base/relative/path.txt');
});
tap.start();

View File

@@ -1,19 +1,235 @@
import { tap, expect } from '@git.zone/tstest/tapbundle';
import * as smartfile from '../ts/index.js';
import { MockSmartFs } from './helpers/mock-smartfs.js';
tap.test('should create a virtualdirectory', async () => {
const virtualDir = await smartfile.VirtualDirectory.fromFsDirPath(
'./test/testassets/testfolder',
);
// Create factory with MockSmartFs
const mockFs = new MockSmartFs();
const factory = new smartfile.SmartFileFactory(mockFs);
// ---------------------------
// VirtualDirectory Factory Tests
// ---------------------------
tap.test('SmartFileFactory.virtualDirectoryFromPath() -> should create a VirtualDirectory from fs path', async () => {
const virtualDir = await factory.virtualDirectoryFromPath('./test/testassets/testfolder');
expect(virtualDir).toBeInstanceOf(smartfile.VirtualDirectory);
expect(virtualDir.smartfileArray.length).toEqual(4);
});
tap.test('should write to a directory', async () => {
const virtualDir = await smartfile.VirtualDirectory.fromFsDirPath(
'./test/testassets/testfolder',
);
virtualDir.saveToDisk('./test/testassets/test');
tap.test('SmartFileFactory.virtualDirectoryEmpty() -> should create an empty VirtualDirectory', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
expect(virtualDir).toBeInstanceOf(smartfile.VirtualDirectory);
expect(virtualDir.isEmpty()).toBeTrue();
expect(virtualDir.size()).toEqual(0);
});
tap.test('SmartFileFactory.virtualDirectoryFromFileArray() -> should create VirtualDirectory from files', async () => {
const file1 = factory.fromString('file1.txt', 'content1');
const file2 = factory.fromString('file2.txt', 'content2');
const virtualDir = factory.virtualDirectoryFromFileArray([file1, file2]);
expect(virtualDir).toBeInstanceOf(smartfile.VirtualDirectory);
expect(virtualDir.size()).toEqual(2);
});
// ---------------------------
// VirtualDirectory Collection Methods
// ---------------------------
tap.test('VirtualDirectory -> should add and list files', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
const file1 = factory.fromString('test1.txt', 'content1');
const file2 = factory.fromString('test2.txt', 'content2');
virtualDir.addSmartfile(file1);
virtualDir.addSmartfile(file2);
const files = virtualDir.listFiles();
expect(files.length).toEqual(2);
expect(files[0].path).toEqual('test1.txt');
expect(files[1].path).toEqual('test2.txt');
});
tap.test('VirtualDirectory -> should check file existence', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
const file = factory.fromString('exists.txt', 'content');
virtualDir.addSmartfile(file);
expect(virtualDir.exists('exists.txt')).toBeTrue();
expect(virtualDir.has('exists.txt')).toBeTrue();
expect(virtualDir.exists('not-there.txt')).toBeFalse();
});
tap.test('VirtualDirectory -> should get file by path', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
const file = factory.fromString('getme.txt', 'my content');
virtualDir.addSmartfile(file);
const retrieved = await virtualDir.getFileByPath('getme.txt');
expect(retrieved).not.toBeUndefined();
expect(retrieved!.parseContentAsString()).toEqual('my content');
});
tap.test('VirtualDirectory -> should remove file by path', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
const file = factory.fromString('remove.txt', 'content');
virtualDir.addSmartfile(file);
expect(virtualDir.exists('remove.txt')).toBeTrue();
const removed = virtualDir.removeByPath('remove.txt');
expect(removed).toBeTrue();
expect(virtualDir.exists('remove.txt')).toBeFalse();
});
tap.test('VirtualDirectory -> should clear all files', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('file1.txt', 'content1'));
virtualDir.addSmartfile(factory.fromString('file2.txt', 'content2'));
expect(virtualDir.size()).toEqual(2);
virtualDir.clear();
expect(virtualDir.size()).toEqual(0);
expect(virtualDir.isEmpty()).toBeTrue();
});
tap.test('VirtualDirectory -> should merge with another VirtualDirectory', async () => {
const vdir1 = factory.virtualDirectoryEmpty();
vdir1.addSmartfile(factory.fromString('file1.txt', 'content1'));
const vdir2 = factory.virtualDirectoryEmpty();
vdir2.addSmartfile(factory.fromString('file2.txt', 'content2'));
vdir1.merge(vdir2);
expect(vdir1.size()).toEqual(2);
expect(vdir1.exists('file1.txt')).toBeTrue();
expect(vdir1.exists('file2.txt')).toBeTrue();
});
tap.test('VirtualDirectory -> should filter files', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('file1.txt', 'content1'));
virtualDir.addSmartfile(factory.fromString('file2.md', 'content2'));
virtualDir.addSmartfile(factory.fromString('file3.txt', 'content3'));
const filtered = virtualDir.filter(file => file.path.endsWith('.txt'));
expect(filtered.size()).toEqual(2);
expect(filtered.exists('file1.txt')).toBeTrue();
expect(filtered.exists('file3.txt')).toBeTrue();
expect(filtered.exists('file2.md')).toBeFalse();
});
tap.test('VirtualDirectory -> should map files', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('file1.txt', 'content1'));
virtualDir.addSmartfile(factory.fromString('file2.txt', 'content2'));
const mapped = virtualDir.map(file => {
file.setContentsFromString(file.parseContentAsString().toUpperCase());
return file;
});
const files = mapped.listFiles();
expect(files[0].parseContentAsString()).toEqual('CONTENT1');
expect(files[1].parseContentAsString()).toEqual('CONTENT2');
});
tap.test('VirtualDirectory -> should find files', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('find.txt', 'findme'));
virtualDir.addSmartfile(factory.fromString('other.txt', 'other'));
const found = virtualDir.find(file => file.parseContentAsString() === 'findme');
expect(found).not.toBeUndefined();
expect(found!.path).toEqual('find.txt');
});
tap.test('VirtualDirectory -> should list directories', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('dir1/file1.txt', 'content1'));
virtualDir.addSmartfile(factory.fromString('dir1/file2.txt', 'content2'));
virtualDir.addSmartfile(factory.fromString('dir2/file3.txt', 'content3'));
virtualDir.addSmartfile(factory.fromString('root.txt', 'content4'));
const dirs = virtualDir.listDirectories();
expect(dirs).toContain('dir1');
expect(dirs).toContain('dir2');
expect(dirs.length).toEqual(2);
});
tap.test('VirtualDirectory -> should save to disk', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('saved1.txt', 'saved content 1'));
virtualDir.addSmartfile(factory.fromString('subdir/saved2.txt', 'saved content 2'));
await virtualDir.saveToDisk('./test/testassets/temp/vdir-output');
// Verify files were written
const file1 = await factory.fromFilePath('./test/testassets/temp/vdir-output/saved1.txt');
expect(file1.parseContentAsString()).toEqual('saved content 1');
const file2 = await factory.fromFilePath('./test/testassets/temp/vdir-output/subdir/saved2.txt');
expect(file2.parseContentAsString()).toEqual('saved content 2');
});
tap.test('VirtualDirectory -> should convert to transferable object', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('trans1.txt', 'transferable1'));
virtualDir.addSmartfile(factory.fromString('trans2.txt', 'transferable2'));
const transferable = await virtualDir.toVirtualDirTransferableObject();
expect(transferable.files).toBeInstanceOf(Array);
expect(transferable.files.length).toEqual(2);
});
// TODO: Fix serialization/deserialization with smartjson
// tap.test('VirtualDirectory -> should create from transferable object', async () => {
// const originalDir = factory.virtualDirectoryEmpty();
// originalDir.addSmartfile(factory.fromString('original.txt', 'original content'));
// const transferable = await originalDir.toVirtualDirTransferableObject();
// const restoredDir = await factory.virtualDirectoryFromTransferable(transferable);
// expect(restoredDir.size()).toEqual(1);
// expect(restoredDir.exists('original.txt')).toBeTrue();
// const file = await restoredDir.getFileByPath('original.txt');
// expect(file!.parseContentAsString()).toEqual('original content');
// });
tap.test('VirtualDirectory -> should shift to subdirectory', async () => {
const virtualDir = factory.virtualDirectoryEmpty();
virtualDir.addSmartfile(factory.fromString('root/sub/file1.txt', 'content1'));
virtualDir.addSmartfile(factory.fromString('root/sub/file2.txt', 'content2'));
virtualDir.addSmartfile(factory.fromString('root/other.txt', 'content3'));
const shifted = await virtualDir.shiftToSubdirectory('root/sub');
expect(shifted.size()).toEqual(2);
expect(shifted.exists('file1.txt')).toBeTrue();
expect(shifted.exists('file2.txt')).toBeTrue();
expect(shifted.exists('other.txt')).toBeFalse();
});
tap.test('VirtualDirectory -> should add another virtual directory with new root', async () => {
const vdir1 = factory.virtualDirectoryEmpty();
vdir1.addSmartfile(factory.fromString('existing.txt', 'existing'));
const vdir2 = factory.virtualDirectoryEmpty();
vdir2.addSmartfile(factory.fromString('added.txt', 'added'));
await vdir1.addVirtualDirectory(vdir2, 'newroot');
expect(vdir1.size()).toEqual(2);
expect(vdir1.exists('existing.txt')).toBeTrue();
expect(vdir1.exists('newroot/added.txt')).toBeTrue();
});
tap.start();

View File

@@ -0,0 +1,7 @@
{
"key1": "this works",
"key2": "this works too",
"key3": {
"nestedkey1": "hello"
}
}

View File

@@ -0,0 +1,7 @@
{
"key1": "this works",
"key2": "this works too",
"key3": {
"nestedkey1": "hello"
}
}

View File

@@ -0,0 +1 @@
directory test content

View File

@@ -0,0 +1 @@
hi there

View File

@@ -0,0 +1 @@
saved content 1

View File

@@ -0,0 +1 @@
saved content 2

View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smartfile',
version: '11.2.6',
description: 'Provides comprehensive tools for efficient file management in Node.js using TypeScript, including handling streams, virtual directories, and various file operations.'
version: '13.0.0',
description: 'High-level file representation classes (SmartFile, StreamFile, VirtualDirectory) for efficient in-memory file management in Node.js using TypeScript. Works seamlessly with @push.rocks/smartfs for filesystem operations.'
}

View File

@@ -0,0 +1,224 @@
import * as plugins from './plugins.js';
import { SmartFile } from './classes.smartfile.js';
import { StreamFile } from './classes.streamfile.js';
import { VirtualDirectory } from './classes.virtualdirectory.js';
export class SmartFileFactory {
private smartFs: any; // Will be typed as SmartFs once we import from @push.rocks/smartfs
constructor(smartFs: any) {
this.smartFs = smartFs;
}
/**
* Creates a default factory using Node.js filesystem provider
*/
public static nodeFs(): SmartFileFactory {
// Temporarily using a placeholder - will be replaced with actual SmartFs initialization
// const smartFs = new SmartFs(new SmartFsProviderNode());
const smartFs = null; // Placeholder
return new SmartFileFactory(smartFs);
}
/**
* Get the underlying SmartFs instance
*/
public getSmartFs(): any {
return this.smartFs;
}
// ============================================
// SmartFile Factory Methods
// ============================================
/**
* Creates a SmartFile from a file path on disk
*/
public async fromFilePath(
filePath: string,
baseArg: string = process.cwd()
): Promise<SmartFile> {
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Cannot read from filesystem without SmartFs.');
}
filePath = plugins.path.resolve(filePath);
const content = await this.smartFs.file(filePath).read();
const fileBuffer = Buffer.from(content);
return new SmartFile({
contentBuffer: fileBuffer,
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
}, this.smartFs);
}
/**
* Creates a SmartFile from a URL
*/
public async fromUrl(urlArg: string): Promise<SmartFile> {
const response = await plugins.smartrequest.SmartRequest.create()
.url(urlArg)
.accept('binary')
.get();
const buffer = Buffer.from(await response.arrayBuffer());
return new SmartFile({
contentBuffer: buffer,
base: process.cwd(),
path: urlArg,
}, this.smartFs);
}
/**
* Creates a SmartFile from a Buffer
*/
public fromBuffer(
filePath: string,
contentBufferArg: Buffer,
baseArg: string = process.cwd()
): SmartFile {
// Use filePath as-is if it's already relative, otherwise compute relative path
const relativePath = plugins.path.isAbsolute(filePath)
? plugins.path.relative(baseArg, filePath)
: filePath;
return new SmartFile({
contentBuffer: contentBufferArg,
base: baseArg,
path: relativePath,
}, this.smartFs);
}
/**
* Creates a SmartFile from a string
*/
public fromString(
filePath: string,
contentStringArg: string,
encodingArg: 'utf8' | 'binary' = 'utf8',
baseArg: string = process.cwd()
): SmartFile {
// Use filePath as-is if it's already relative, otherwise compute relative path
const relativePath = plugins.path.isAbsolute(filePath)
? plugins.path.relative(baseArg, filePath)
: filePath;
return new SmartFile({
contentBuffer: Buffer.from(contentStringArg, encodingArg),
base: baseArg,
path: relativePath,
}, this.smartFs);
}
/**
* Creates a SmartFile from a stream
*/
public async fromStream(
stream: plugins.stream.Readable,
filePath: string,
baseArg: string = process.cwd()
): Promise<SmartFile> {
return new Promise<SmartFile>((resolve, reject) => {
const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
stream.on('error', (error) => reject(error));
stream.on('end', () => {
const contentBuffer = Buffer.concat(chunks);
const smartfile = new SmartFile({
contentBuffer: contentBuffer,
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
}, this.smartFs);
resolve(smartfile);
});
});
}
/**
* Creates a SmartFile from folded JSON
*/
public async fromFoldedJson(foldedJsonArg: string): Promise<SmartFile> {
const parsed = plugins.smartjson.parse(foldedJsonArg);
return new SmartFile(parsed, this.smartFs);
}
// ============================================
// StreamFile Factory Methods
// ============================================
/**
* Creates a StreamFile from a file path
*/
public async streamFromPath(filePath: string): Promise<StreamFile> {
return StreamFile.fromPath(filePath, this.smartFs);
}
/**
* Creates a StreamFile from a URL
*/
public async streamFromUrl(url: string): Promise<StreamFile> {
return StreamFile.fromUrl(url, this.smartFs);
}
/**
* Creates a StreamFile from a Buffer
*/
public streamFromBuffer(buffer: Buffer, relativeFilePath?: string): StreamFile {
return StreamFile.fromBuffer(buffer, relativeFilePath, this.smartFs);
}
/**
* Creates a StreamFile from a Node.js Readable stream
*/
public streamFromStream(
stream: plugins.stream.Readable,
relativeFilePath?: string,
multiUse: boolean = false
): StreamFile {
return StreamFile.fromStream(stream, relativeFilePath, multiUse, this.smartFs);
}
// ============================================
// VirtualDirectory Factory Methods
// ============================================
/**
* Creates a VirtualDirectory from a filesystem directory path
*/
public async virtualDirectoryFromPath(pathArg: string): Promise<VirtualDirectory> {
return VirtualDirectory.fromFsDirPath(pathArg, this.smartFs, this);
}
/**
* Creates an empty VirtualDirectory
*/
public virtualDirectoryEmpty(): VirtualDirectory {
return new VirtualDirectory(this.smartFs, this);
}
/**
* Creates a VirtualDirectory from an array of SmartFiles
*/
public virtualDirectoryFromFileArray(files: SmartFile[]): VirtualDirectory {
const vdir = new VirtualDirectory(this.smartFs, this);
vdir.addSmartfiles(files);
return vdir;
}
/**
* Creates a VirtualDirectory from a transferable object
*/
public async virtualDirectoryFromTransferable(
virtualDirTransferableObjectArg: plugins.smartfileInterfaces.VirtualDirTransferableObject
): Promise<VirtualDirectory> {
const newVirtualDir = new VirtualDirectory(this.smartFs, this);
for (const fileArg of virtualDirTransferableObjectArg.files) {
const smartFile = SmartFile.enfoldFromJson(fileArg) as SmartFile;
// Update the smartFs reference
(smartFile as any).smartFs = this.smartFs;
newVirtualDir.addSmartfiles([smartFile]);
}
return newVirtualDir;
}
}

View File

@@ -1,6 +1,4 @@
import * as plugins from './plugins.js';
import * as fs from './fs.js';
import * as memory from './memory.js';
export interface ISmartfileConstructorOptions {
path: string;
@@ -10,103 +8,17 @@ export interface ISmartfileConstructorOptions {
/**
* an vinyl file compatible in memory file class
* Use SmartFileFactory to create instances of this class
*/
export class SmartFile extends plugins.smartjson.Smartjson {
// ======
// STATIC
// ======
/**
* creates a Smartfile from a filePath
* @param filePath
*/
public static async fromFilePath(
filePath: string,
baseArg: string = process.cwd(),
) {
filePath = plugins.path.resolve(filePath);
const fileBuffer = fs.toBufferSync(filePath);
const smartfile = new SmartFile({
contentBuffer: fileBuffer,
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
});
return smartfile;
}
public static async fromBuffer(
filePath: string,
contentBufferArg: Buffer,
baseArg: string = process.cwd(),
) {
const smartfile = new SmartFile({
contentBuffer: contentBufferArg,
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
});
return smartfile;
}
public static async fromString(
filePath: string,
contentStringArg: string,
encodingArg: 'utf8' | 'binary',
baseArg = process.cwd(),
) {
const smartfile = new SmartFile({
contentBuffer: Buffer.from(contentStringArg, encodingArg),
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
});
return smartfile;
}
public static async fromFoldedJson(foldedJsonArg: string) {
return new SmartFile(plugins.smartjson.parse(foldedJsonArg));
}
/**
* creates a Smartfile from a ReadableStream
* @param stream a readable stream that provides file content
* @param filePath the file path to associate with the content
* @param baseArg the base path to use for the file
*/
public static async fromStream(
stream: plugins.stream.Readable,
filePath: string,
baseArg: string = process.cwd(),
): Promise<SmartFile> {
return new Promise<SmartFile>((resolve, reject) => {
const chunks: Buffer[] = [];
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
stream.on('error', (error) => reject(error));
stream.on('end', () => {
const contentBuffer = Buffer.concat(chunks);
const smartfile = new SmartFile({
contentBuffer: contentBuffer,
base: baseArg,
path: plugins.path.relative(baseArg, filePath),
});
resolve(smartfile);
});
});
}
public static async fromUrl(urlArg: string) {
const response = await plugins.smartrequest.SmartRequest.create()
.url(urlArg)
.accept('binary')
.get();
const buffer = Buffer.from(await response.arrayBuffer());
const smartfile = await SmartFile.fromBuffer(urlArg, buffer);
return smartfile;
}
// ========
// INSTANCE
// ========
/**
* Reference to the SmartFs instance for filesystem operations
*/
private smartFs?: any;
/**
* the relative path of the file
*/
@@ -149,9 +61,10 @@ export class SmartFile extends plugins.smartjson.Smartjson {
/**
* the constructor of Smartfile
* @param optionsArg
* @param smartFs optional SmartFs instance for filesystem operations
*/
constructor(optionsArg: ISmartfileConstructorOptions) {
constructor(optionsArg: ISmartfileConstructorOptions, smartFs?: any) {
super();
if (optionsArg.contentBuffer) {
this.contentBuffer = optionsArg.contentBuffer;
@@ -160,6 +73,7 @@ export class SmartFile extends plugins.smartjson.Smartjson {
}
this.path = optionsArg.path;
this.base = optionsArg.base;
this.smartFs = smartFs;
}
/**
@@ -177,14 +91,19 @@ export class SmartFile extends plugins.smartjson.Smartjson {
* write file to disk at its original location
* Behaviours:
* - no argument write to exactly where the file was picked up
* - Requires SmartFs instance (create via SmartFileFactory)
*/
public async write() {
let writePath = plugins.smartpath.transform.makeAbsolute(
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create SmartFile through SmartFileFactory.');
}
const writePath = plugins.smartpath.transform.makeAbsolute(
this.path,
this.base,
);
console.log(`writing to ${writePath}`);
await memory.toFs(this.contentBuffer, writePath);
await this.smartFs.file(writePath).write(this.contentBuffer);
}
/**
@@ -193,10 +112,15 @@ export class SmartFile extends plugins.smartjson.Smartjson {
* @param filePathArg
*/
public async writeToDiskAtPath(filePathArg: string) {
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create SmartFile through SmartFileFactory.');
}
if (!plugins.path.isAbsolute(filePathArg)) {
filePathArg = plugins.path.join(process.cwd(), filePathArg);
}
await memory.toFs(this.contentBuffer, filePathArg);
await this.smartFs.file(filePathArg).write(this.contentBuffer);
}
/**
@@ -205,9 +129,13 @@ export class SmartFile extends plugins.smartjson.Smartjson {
* @returns
*/
public async writeToDir(dirPathArg: string) {
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create SmartFile through SmartFileFactory.');
}
dirPathArg = plugins.smartpath.transform.toAbsolute(dirPathArg) as string;
const filePath = plugins.path.join(dirPathArg, this.path);
await memory.toFs(this.contentBuffer, filePath);
await this.smartFs.file(filePath).write(this.contentBuffer);
return filePath;
}
@@ -215,16 +143,25 @@ export class SmartFile extends plugins.smartjson.Smartjson {
* read file from disk
*/
public async read() {
this.contentBuffer = await fs.toBuffer(
plugins.path.join(this.base, this.path),
);
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create SmartFile through SmartFileFactory.');
}
const filePath = plugins.path.join(this.base, this.path);
const content = await this.smartFs.file(filePath).read();
this.contentBuffer = Buffer.from(content);
}
/**
* deletes the file from disk at its original location
*/
public async delete() {
await fs.remove(plugins.path.join(this.base, this.path));
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create SmartFile through SmartFileFactory.');
}
const filePath = plugins.path.join(this.base, this.path);
await this.smartFs.file(filePath).delete();
}
/**
@@ -380,4 +317,18 @@ export class SmartFile extends plugins.smartjson.Smartjson {
public async getSize(): Promise<number> {
return this.contentBuffer.length;
}
/**
* Parse content as string with specified encoding
*/
public parseContentAsString(encodingArg: BufferEncoding = 'utf8'): string {
return this.contentBuffer.toString(encodingArg);
}
/**
* Parse content as buffer
*/
public parseContentAsBuffer(): Buffer {
return this.contentBuffer;
}
}

View File

@@ -1,6 +1,4 @@
import * as plugins from './plugins.js';
import * as smartfileFs from './fs.js';
import * as smartfileFsStream from './fsstream.js';
import { Readable } from 'stream';
type TStreamSource = (streamFile: StreamFile) => Promise<Readable | ReadableStream>;
@@ -8,30 +6,36 @@ type TStreamSource = (streamFile: StreamFile) => Promise<Readable | ReadableStre
/**
* The StreamFile class represents a file as a stream.
* It allows creating streams from a file path, a URL, or a buffer.
* Use SmartFileFactory to create instances of this class.
*/
export class StreamFile {
// STATIC
public static async fromPath(filePath: string): Promise<StreamFile> {
const streamSource: TStreamSource = async (streamFileArg) =>
smartfileFsStream.createReadStream(filePath);
const streamFile = new StreamFile(streamSource, filePath);
public static async fromPath(filePath: string, smartFs?: any): Promise<StreamFile> {
if (!smartFs) {
throw new Error('No SmartFs instance available. Create StreamFile through SmartFileFactory.');
}
const streamSource: TStreamSource = async (streamFileArg) => {
return await streamFileArg.smartFs.file(filePath).readStream();
};
const streamFile = new StreamFile(streamSource, filePath, smartFs);
streamFile.multiUse = true;
streamFile.byteLengthComputeFunction = async () => {
const stats = await smartfileFs.stat(filePath);
const stats = await smartFs.file(filePath).stat();
return stats.size;
};
return streamFile;
}
public static async fromUrl(url: string): Promise<StreamFile> {
public static async fromUrl(url: string, smartFs?: any): Promise<StreamFile> {
const streamSource: TStreamSource = async (streamFileArg) => {
const response = await plugins.smartrequest.SmartRequest.create()
.url(url)
.get();
return response.stream();
};
const streamFile = new StreamFile(streamSource);
const streamFile = new StreamFile(streamSource, undefined, smartFs);
streamFile.multiUse = true;
streamFile.byteLengthComputeFunction = async () => {
const response = await plugins.smartrequest.SmartRequest.create()
@@ -47,6 +51,7 @@ export class StreamFile {
public static fromBuffer(
buffer: Buffer,
relativeFilePath?: string,
smartFs?: any
): StreamFile {
const streamSource: TStreamSource = async (streamFileArg) => {
const stream = new Readable();
@@ -54,7 +59,7 @@ export class StreamFile {
stream.push(null); // End of stream
return stream;
};
const streamFile = new StreamFile(streamSource, relativeFilePath);
const streamFile = new StreamFile(streamSource, relativeFilePath, smartFs);
streamFile.multiUse = true;
streamFile.byteLengthComputeFunction = async () => buffer.length;
return streamFile;
@@ -65,12 +70,14 @@ export class StreamFile {
* @param stream A Node.js Readable stream.
* @param relativeFilePath Optional file path for the stream.
* @param multiUse If true, the stream can be read multiple times, caching its content.
* @param smartFs Optional SmartFs instance for filesystem operations
* @returns A StreamFile instance.
*/
public static fromStream(
stream: Readable,
relativeFilePath?: string,
multiUse: boolean = false,
smartFs?: any
): StreamFile {
const streamSource: TStreamSource = (streamFileArg) => {
if (streamFileArg.multiUse) {
@@ -84,7 +91,7 @@ export class StreamFile {
}
};
const streamFile = new StreamFile(streamSource, relativeFilePath);
const streamFile = new StreamFile(streamSource, relativeFilePath, smartFs);
streamFile.multiUse = multiUse;
// If multi-use is enabled, cache the stream when it's first read
@@ -106,6 +113,7 @@ export class StreamFile {
// INSTANCE
relativeFilePath?: string;
private streamSource: TStreamSource;
private smartFs?: any;
// enable stream based multi use
private cachedStreamBuffer?: Buffer;
@@ -113,9 +121,10 @@ export class StreamFile {
public used: boolean = false;
public byteLengthComputeFunction: () => Promise<number>;
private constructor(streamSource: TStreamSource, relativeFilePath?: string) {
private constructor(streamSource: TStreamSource, relativeFilePath?: string, smartFs?: any) {
this.streamSource = streamSource;
this.relativeFilePath = relativeFilePath;
this.smartFs = smartFs;
}
// METHODS
@@ -148,9 +157,13 @@ export class StreamFile {
* @param filePathArg The file path where the stream should be written.
*/
public async writeToDisk(filePathArg: string): Promise<void> {
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create StreamFile through SmartFileFactory.');
}
this.checkMultiUse();
const readStream = await this.createReadStream();
const writeStream = smartfileFsStream.createWriteStream(filePathArg);
const writeStream = await this.smartFs.file(filePathArg).writeStream();
return new Promise((resolve, reject) => {
readStream.pipe(writeStream);
@@ -161,9 +174,14 @@ export class StreamFile {
}
public async writeToDir(dirPathArg: string) {
if (!this.smartFs) {
throw new Error('No SmartFs instance available. Create StreamFile through SmartFileFactory.');
}
this.checkMultiUse();
const filePath = plugins.path.join(dirPathArg, this.relativeFilePath);
await smartfileFs.ensureDir(plugins.path.parse(filePath).dir);
const dirPath = plugins.path.parse(filePath).dir;
await this.smartFs.directory(dirPath).create({ recursive: true });
return this.writeToDisk(filePath);
}
@@ -196,4 +214,17 @@ export class StreamFile {
return null;
}
}
/**
* Converts the StreamFile to a SmartFile by loading content into memory
*/
public async toSmartFile(): Promise<any> {
const { SmartFile } = await import('./classes.smartfile.js');
const buffer = await this.getContentAsBuffer();
return new SmartFile({
path: this.relativeFilePath || 'stream',
contentBuffer: buffer,
base: process.cwd()
}, this.smartFs);
}
}

View File

@@ -1,6 +1,5 @@
import { SmartFile } from './classes.smartfile.js';
import * as plugins from './plugins.js';
import * as fs from './fs.js';
export interface IVirtualDirectoryConstructorOptions {
mode: '';
@@ -8,46 +7,149 @@ export interface IVirtualDirectoryConstructorOptions {
/**
* a virtual directory exposes a fs api
* Use SmartFileFactory to create instances of this class
*/
export class VirtualDirectory {
consstructor(options = {}) {}
// STATIC
public static async fromFsDirPath(
pathArg: string,
smartFs?: any,
factory?: any
): Promise<VirtualDirectory> {
const newVirtualDir = new VirtualDirectory();
newVirtualDir.addSmartfiles(await fs.fileTreeToObject(pathArg, '**/*'));
if (!smartFs || !factory) {
throw new Error('No SmartFs/Factory instance available. Create VirtualDirectory through SmartFileFactory.');
}
const newVirtualDir = new VirtualDirectory(smartFs, factory);
// Use smartFs to list directory and factory to create SmartFiles
const entries = await smartFs.directory(pathArg).list({ recursive: true });
const smartfiles = await Promise.all(
entries
.filter((entry: any) => entry.isFile)
.map((entry: any) => factory.fromFilePath(entry.path, pathArg))
);
newVirtualDir.addSmartfiles(smartfiles);
return newVirtualDir;
}
public static async fromVirtualDirTransferableObject(
virtualDirTransferableObjectArg: plugins.smartfileInterfaces.VirtualDirTransferableObject,
smartFs?: any,
factory?: any
): Promise<VirtualDirectory> {
const newVirtualDir = new VirtualDirectory();
const newVirtualDir = new VirtualDirectory(smartFs, factory);
for (const fileArg of virtualDirTransferableObjectArg.files) {
newVirtualDir.addSmartfiles([
SmartFile.enfoldFromJson(fileArg) as SmartFile,
]);
const smartFile = SmartFile.enfoldFromJson(fileArg) as SmartFile;
// Update smartFs reference if available
if (smartFs) {
(smartFile as any).smartFs = smartFs;
}
newVirtualDir.addSmartfiles([smartFile]);
}
return newVirtualDir;
}
public static fromFileArray(files: SmartFile[], smartFs?: any, factory?: any): VirtualDirectory {
const vdir = new VirtualDirectory(smartFs, factory);
vdir.addSmartfiles(files);
return vdir;
}
public static empty(smartFs?: any, factory?: any): VirtualDirectory {
return new VirtualDirectory(smartFs, factory);
}
// INSTANCE
public smartfileArray: SmartFile[] = [];
private smartFs?: any;
private factory?: any;
constructor() {}
constructor(smartFs?: any, factory?: any) {
this.smartFs = smartFs;
this.factory = factory;
}
// ============================================
// Collection Mutations
// ============================================
public addSmartfiles(smartfileArrayArg: SmartFile[]) {
this.smartfileArray = this.smartfileArray.concat(smartfileArrayArg);
}
public async getFileByPath(pathArg: string) {
for (const smartfile of this.smartfileArray) {
if (smartfile.path === pathArg) {
return smartfile;
public addSmartfile(smartfileArg: SmartFile): void {
this.smartfileArray.push(smartfileArg);
}
public removeByPath(pathArg: string): boolean {
const initialLength = this.smartfileArray.length;
this.smartfileArray = this.smartfileArray.filter(f => f.path !== pathArg);
return this.smartfileArray.length < initialLength;
}
public clear(): void {
this.smartfileArray = [];
}
public merge(otherVDir: VirtualDirectory): void {
this.addSmartfiles(otherVDir.smartfileArray);
}
// ============================================
// Collection Queries
// ============================================
public exists(pathArg: string): boolean {
return this.smartfileArray.some(f => f.path === pathArg);
}
public has(pathArg: string): boolean {
return this.exists(pathArg);
}
public async getFileByPath(pathArg: string): Promise<SmartFile | undefined> {
return this.smartfileArray.find(f => f.path === pathArg);
}
public listFiles(): SmartFile[] {
return [...this.smartfileArray];
}
public listDirectories(): string[] {
const dirs = new Set<string>();
for (const file of this.smartfileArray) {
const dir = plugins.path.dirname(file.path);
if (dir !== '.') {
dirs.add(dir);
}
}
return Array.from(dirs).sort();
}
public filter(predicate: (file: SmartFile) => boolean): VirtualDirectory {
const newVDir = new VirtualDirectory(this.smartFs, this.factory);
newVDir.addSmartfiles(this.smartfileArray.filter(predicate));
return newVDir;
}
public map(fn: (file: SmartFile) => SmartFile): VirtualDirectory {
const newVDir = new VirtualDirectory(this.smartFs, this.factory);
newVDir.addSmartfiles(this.smartfileArray.map(fn));
return newVDir;
}
public find(predicate: (file: SmartFile) => boolean): SmartFile | undefined {
return this.smartfileArray.find(predicate);
}
public size(): number {
return this.smartfileArray.length;
}
public isEmpty(): boolean {
return this.smartfileArray.length === 0;
}
public async toVirtualDirTransferableObject(): Promise<plugins.smartfileInterfaces.VirtualDirTransferableObject> {
@@ -69,7 +171,7 @@ export class VirtualDirectory {
}
public async shiftToSubdirectory(subDir: string): Promise<VirtualDirectory> {
const newVirtualDir = new VirtualDirectory();
const newVirtualDir = new VirtualDirectory(this.smartFs, this.factory);
for (const file of this.smartfileArray) {
if (file.path.startsWith(subDir)) {
const adjustedFilePath = plugins.path.relative(subDir, file.path);
@@ -80,6 +182,13 @@ export class VirtualDirectory {
return newVirtualDir;
}
public async loadFromDisk(dirArg: string): Promise<void> {
// Load from disk, replacing current collection
this.clear();
const loaded = await VirtualDirectory.fromFsDirPath(dirArg, this.smartFs, this.factory);
this.addSmartfiles(loaded.smartfileArray);
}
public async addVirtualDirectory(
virtualDir: VirtualDirectory,
newRoot: string,

715
ts/fs.ts
View File

@@ -1,715 +0,0 @@
import * as plugins from './plugins.js';
import * as interpreter from './interpreter.js';
import { SmartFile } from './classes.smartfile.js';
import * as memory from './memory.js';
import type { StreamFile } from './classes.streamfile.js';
/*===============================================================
============================ Checks =============================
===============================================================*/
/**
*
* @param filePath
* @returns {boolean}
*/
export const fileExistsSync = (filePath): boolean => {
let fileExistsBool: boolean = false;
try {
plugins.fsExtra.readFileSync(filePath);
fileExistsBool = true;
} catch (err) {
fileExistsBool = false;
}
return fileExistsBool;
};
/**
*
* @param filePath
* @returns {any}
*/
export const fileExists = async (filePath): Promise<boolean> => {
const done = plugins.smartpromise.defer<boolean>();
plugins.fs.access(filePath, 4, (err) => {
err ? done.resolve(false) : done.resolve(true);
});
return done.promise;
};
/**
* Checks if given path points to an existing directory
*/
export const isDirectory = (pathArg: string): boolean => {
try {
return plugins.fsExtra.statSync(pathArg).isDirectory();
} catch (err) {
return false;
}
};
/**
* Checks if given path points to an existing directory
*/
export const isDirectorySync = (pathArg: string): boolean => {
try {
return plugins.fsExtra.statSync(pathArg).isDirectory();
} catch (err) {
return false;
}
};
/**
* Checks if a given path points to an existing file
*/
export const isFile = (pathArg): boolean => {
return plugins.fsExtra.statSync(pathArg).isFile();
};
/*===============================================================
============================ FS ACTIONS =========================
===============================================================*/
/**
* copies a file or directory from A to B on the local disk
*/
export const copy = async (
fromArg: string,
toArg: string,
optionsArg?: plugins.fsExtra.CopyOptions & { replaceTargetDir?: boolean },
): Promise<void> => {
if (
optionsArg?.replaceTargetDir &&
isDirectory(fromArg) &&
isDirectory(toArg)
) {
await remove(toArg);
}
return await plugins.fsExtra.copy(
fromArg,
toArg,
optionsArg as plugins.fsExtra.CopyOptions,
);
};
/**
* copies a file or directory SYNCHRONOUSLY from A to B on the local disk
*/
export const copySync = (
fromArg: string,
toArg: string,
optionsArg?: plugins.fsExtra.CopyOptionsSync & { replaceTargetDir?: boolean },
): void => {
if (
optionsArg?.replaceTargetDir &&
isDirectory(fromArg) &&
isDirectory(toArg)
) {
removeSync(toArg);
}
return plugins.fsExtra.copySync(
fromArg,
toArg,
optionsArg as plugins.fsExtra.CopyOptionsSync,
);
};
/**
* ensures that a directory is in place
*/
export const ensureDir = async (dirPathArg: string) => {
await plugins.fsExtra.ensureDir(dirPathArg);
};
/**
* ensures that a directory is in place
*/
export const ensureDirSync = (dirPathArg: string) => {
plugins.fsExtra.ensureDirSync(dirPathArg);
};
/**
* ensure an empty directory
* @executes ASYNC
*/
export const ensureEmptyDir = async (dirPathArg: string) => {
await plugins.fsExtra.ensureDir(dirPathArg);
await plugins.fsExtra.emptyDir(dirPathArg);
};
/**
* ensure an empty directory
* @executes SYNC
*/
export const ensureEmptyDirSync = (dirPathArg: string) => {
plugins.fsExtra.ensureDirSync(dirPathArg);
plugins.fsExtra.emptyDirSync(dirPathArg);
};
/**
* ensures that a file is on disk
* @param filePath the filePath to ensureDir
* @param the fileContent to place into a new file in case it doesn't exist yet
* @returns Promise<void>
* @exec ASYNC
*/
export const ensureFile = async (
filePathArg,
initFileStringArg,
): Promise<void> => {
ensureFileSync(filePathArg, initFileStringArg);
};
/**
* ensures that a file is on disk
* @param filePath the filePath to ensureDir
* @param the fileContent to place into a new file in case it doesn't exist yet
* @returns Promise<void>
* @exec SYNC
*/
export const ensureFileSync = (
filePathArg: string,
initFileStringArg: string,
): void => {
if (fileExistsSync(filePathArg)) {
return null;
} else {
memory.toFsSync(initFileStringArg, filePathArg);
}
};
/**
* removes a file or folder from local disk
*/
export const remove = async (pathArg: string): Promise<void> => {
await plugins.fsExtra.remove(pathArg);
};
/**
* removes a file SYNCHRONOUSLY from local disk
*/
export const removeSync = (pathArg: string): void => {
plugins.fsExtra.removeSync(pathArg);
};
/**
* removes an array of filePaths from disk
*/
export const removeMany = async (filePathArrayArg: string[]) => {
const promiseArray: Array<Promise<void>> = [];
for (const filePath of filePathArrayArg) {
promiseArray.push(remove(filePath));
}
await Promise.all(promiseArray);
};
/**
* like removeFilePathArray but SYNCHRONOUSLY
*/
export const removeManySync = (filePathArrayArg: string[]): void => {
for (const filePath of filePathArrayArg) {
removeSync(filePath);
}
};
/*===============================================================
============================ Write/Read =========================
===============================================================*/
/**
* reads a file content to an object
* good for JSON, YAML, TOML, etc.
* @param filePathArg
* @param fileTypeArg
* @returns {any}
*/
export const toObjectSync = (filePathArg, fileTypeArg?) => {
const fileString = plugins.fsExtra.readFileSync(filePathArg, 'utf8');
let fileType;
fileTypeArg
? (fileType = fileTypeArg)
: (fileType = interpreter.filetype(filePathArg));
try {
return interpreter.objectFile(fileString, fileType);
} catch (err) {
err.message = `Failed to read file at ${filePathArg}` + err.message;
throw err;
}
};
/**
* reads a file content to a String
*/
export const toStringSync = (filePath: string): string => {
const encoding = plugins.smartmime.getEncodingForPathSync(filePath);
let fileString: string | Buffer = plugins.fsExtra.readFileSync(
filePath,
encoding,
);
if (Buffer.isBuffer(fileString)) {
fileString = fileString.toString('binary');
}
return fileString;
};
export const toBuffer = async (filePath: string): Promise<Buffer> => {
return plugins.fsExtra.readFile(filePath);
};
export const toBufferSync = (filePath: string): Buffer => {
return plugins.fsExtra.readFileSync(filePath);
};
/**
* Creates a Readable Stream from a file path.
* @param filePath The path to the file.
* @returns {fs.ReadStream}
*/
export const toReadStream = (filePath: string): plugins.fs.ReadStream => {
if (!fileExistsSync(filePath)) {
throw new Error(`File does not exist at path: ${filePath}`);
}
return plugins.fsExtra.createReadStream(filePath);
};
export const fileTreeToHash = async (
dirPathArg: string,
miniMatchFilter: string,
) => {
const fileTreeObject = await fileTreeToObject(dirPathArg, miniMatchFilter);
let combinedString = '';
for (const smartfile of fileTreeObject) {
combinedString += await smartfile.getHash();
}
const hash = await plugins.smarthash.sha256FromString(combinedString);
return hash;
};
/**
* creates a smartfile array from a directory
* @param dirPathArg the directory to start from
* @param miniMatchFilter a minimatch filter of what files to include
*/
export const fileTreeToObject = async (
dirPathArg: string,
miniMatchFilter: string,
) => {
// handle absolute miniMatchFilter
let dirPath: string;
if (plugins.path.isAbsolute(miniMatchFilter)) {
dirPath = '/';
} else {
dirPath = plugins.smartpath.transform.toAbsolute(dirPathArg) as string;
}
const fileTree = await listFileTree(dirPath, miniMatchFilter);
const smartfileArray: SmartFile[] = [];
for (const filePath of fileTree) {
const readPath = ((): string => {
if (!plugins.path.isAbsolute(filePath)) {
return plugins.path.join(dirPath, filePath);
} else {
return filePath;
}
})();
const fileBuffer = plugins.fs.readFileSync(readPath);
// push a read file as Smartfile
smartfileArray.push(
new SmartFile({
contentBuffer: fileBuffer,
base: dirPath,
path: filePath,
}),
);
}
return smartfileArray;
};
/**
* lists Folders in a directory on local disk
* @returns Promise with an array that contains the folder names
*/
export const listFolders = async (
pathArg: string,
regexFilter?: RegExp,
): Promise<string[]> => {
return listFoldersSync(pathArg, regexFilter);
};
/**
* lists Folders SYNCHRONOUSLY in a directory on local disk
* @returns an array with the folder names as strings
*/
export const listFoldersSync = (
pathArg: string,
regexFilter?: RegExp,
): string[] => {
let folderArray = plugins.fsExtra.readdirSync(pathArg).filter((file) => {
return plugins.fsExtra
.statSync(plugins.path.join(pathArg, file))
.isDirectory();
});
if (regexFilter) {
folderArray = folderArray.filter((fileItem) => {
return regexFilter.test(fileItem);
});
}
return folderArray;
};
/**
* lists Files in a directory on local disk
* @returns Promise
*/
export const listFiles = async (
pathArg: string,
regexFilter?: RegExp,
): Promise<string[]> => {
return listFilesSync(pathArg, regexFilter);
};
/**
* lists Files SYNCHRONOUSLY in a directory on local disk
* @returns an array with the folder names as strings
*/
export const listFilesSync = (
pathArg: string,
regexFilter?: RegExp,
): string[] => {
let fileArray = plugins.fsExtra.readdirSync(pathArg).filter((file) => {
return plugins.fsExtra.statSync(plugins.path.join(pathArg, file)).isFile();
});
if (regexFilter) {
fileArray = fileArray.filter((fileItem) => {
return regexFilter.test(fileItem);
});
}
return fileArray;
};
/**
* lists all items (folders AND files) in a directory on local disk
* @returns Promise<string[]>
*/
export const listAllItems = async (
pathArg: string,
regexFilter?: RegExp,
): Promise<string[]> => {
return listAllItemsSync(pathArg, regexFilter);
};
/**
* lists all items (folders AND files) in a directory on local disk
* @returns an array with the folder names as strings
* @executes SYNC
*/
export const listAllItemsSync = (
pathArg: string,
regexFilter?: RegExp,
): string[] => {
let allItmesArray = plugins.fsExtra.readdirSync(pathArg).filter((file) => {
return plugins.fsExtra.statSync(plugins.path.join(pathArg, file)).isFile();
});
if (regexFilter) {
allItmesArray = allItmesArray.filter((fileItem) => {
return regexFilter.test(fileItem);
});
}
return allItmesArray;
};
/**
* lists a file tree using a miniMatch filter
* note: if the miniMatch Filter is an absolute path, the cwdArg will be omitted
* @returns Promise<string[]> string array with the absolute paths of all matching files
*/
export const listFileTree = async (
dirPathArg: string,
miniMatchFilter: string,
absolutePathsBool: boolean = false,
): Promise<string[]> => {
// handle absolute miniMatchFilter
let dirPath: string;
if (plugins.path.isAbsolute(miniMatchFilter)) {
dirPath = '/';
} else {
dirPath = dirPathArg;
}
const options = {
cwd: dirPath,
nodir: true,
dot: true,
};
// Fix inconsistent **/* glob behavior across systems
// Some glob implementations don't include root-level files when using **/*
// To ensure consistent behavior, we expand **/* patterns to include both root and nested files
let patterns: string[];
if (miniMatchFilter.startsWith('**/')) {
// Extract the part after **/ (e.g., "*.ts" from "**/*.ts")
const rootPattern = miniMatchFilter.substring(3);
// Use both the root pattern and the original pattern to ensure we catch everything
patterns = [rootPattern, miniMatchFilter];
} else {
patterns = [miniMatchFilter];
}
// Collect results from all patterns
const allFiles = new Set<string>();
for (const pattern of patterns) {
const files = await plugins.glob.glob(pattern, options);
files.forEach((file) => allFiles.add(file));
}
let fileList = Array.from(allFiles).sort();
if (absolutePathsBool) {
fileList = fileList.map((filePath) => {
return plugins.path.resolve(plugins.path.join(dirPath, filePath));
});
}
return fileList;
};
/**
* Watches for file stability before resolving the promise.
* Ensures that the directory/file exists before setting up the watcher.
*
* **New behavior**: If the given path is a directory, this function will:
* 1. Wait for that directory to exist (creating a timeout if needed).
* 2. Watch the directory until at least one file appears.
* 3. Then wait for the first file in the directory to stabilize before resolving.
*
* @param fileOrDirPathArg The path of the file or directory to monitor.
* @param timeoutMs The maximum time to wait for the file to stabilize (in milliseconds). Default is 60 seconds.
* @returns A promise that resolves when the target is stable or rejects on timeout/error.
*/
export const waitForFileToBeReady = async (
fileOrDirPathArg: string,
timeoutMs: number = 60000,
): Promise<void> => {
const startTime = Date.now();
/**
* Ensure that a path (file or directory) exists. If it doesn't yet exist,
* wait until it does (or time out).
* @param pathToCheck The file or directory path to check.
*/
const ensurePathExists = async (pathToCheck: string): Promise<void> => {
while (true) {
try {
await plugins.smartpromise.fromCallback((cb) =>
plugins.fs.access(pathToCheck, plugins.fs.constants.F_OK, cb),
);
return;
} catch (err: any) {
if (err.code !== 'ENOENT') {
throw err; // Propagate unexpected errors
}
if (Date.now() - startTime > timeoutMs) {
throw new Error(`Timeout waiting for path to exist: ${pathToCheck}`);
}
await plugins.smartdelay.delayFor(500);
}
}
};
/**
* Checks if a file (not directory) is stable by comparing sizes
* across successive checks.
* @param filePathArg The path of the file to check.
* @returns A promise that resolves once the file stops changing.
*/
const waitForSingleFileToBeStable = async (
filePathArg: string,
): Promise<void> => {
let lastFileSize = -1;
let fileIsStable = false;
// We'll create a helper for repeated stats-checking logic
const checkFileStability = async () => {
try {
const stats = await plugins.smartpromise.fromCallback<plugins.fs.Stats>(
(cb) => plugins.fs.stat(filePathArg, cb),
);
if (stats.isDirectory()) {
// If it unexpectedly turns out to be a directory here, throw
throw new Error(
`Expected a file but found a directory: ${filePathArg}`,
);
}
if (stats.size === lastFileSize) {
fileIsStable = true;
} else {
lastFileSize = stats.size;
fileIsStable = false;
}
} catch (err: any) {
// Ignore only if file not found
if (err.code !== 'ENOENT') {
throw err;
}
}
};
// Ensure file exists first
await ensurePathExists(filePathArg);
// Set up a watcher on the file itself
const fileWatcher = plugins.fs.watch(
filePathArg,
{ persistent: true },
async () => {
if (!fileIsStable) {
await checkFileStability();
}
},
);
try {
// Poll until stable or timeout
while (!fileIsStable) {
if (Date.now() - startTime > timeoutMs) {
throw new Error(
`Timeout waiting for file to stabilize: ${filePathArg}`,
);
}
await checkFileStability();
if (!fileIsStable) {
await plugins.smartdelay.delayFor(1000);
}
}
} finally {
fileWatcher.close();
}
};
/**
* Main logic: check if we have a directory or file at fileOrDirPathArg.
* If directory, wait for first file in the directory to appear and stabilize.
* If file, do the old single-file wait logic.
*/
const statsForGivenPath = await (async () => {
try {
await ensurePathExists(fileOrDirPathArg);
return await plugins.smartpromise.fromCallback<plugins.fs.Stats>((cb) =>
plugins.fs.stat(fileOrDirPathArg, cb),
);
} catch (err) {
// If there's an error (including timeout), just rethrow
throw err;
}
})();
if (!statsForGivenPath.isDirectory()) {
// It's a file just do the single-file stability wait
await waitForSingleFileToBeStable(fileOrDirPathArg);
return;
}
// Otherwise, it's a directory. Wait for the first file inside to appear and be stable
const dirPath = fileOrDirPathArg;
// Helper to find the first file in the directory if it exists
const getFirstFileInDirectory = async (): Promise<string | null> => {
const entries = await plugins.smartpromise.fromCallback<string[]>((cb) =>
plugins.fs.readdir(dirPath, cb),
);
// We only want actual files, not subdirectories
for (const entry of entries) {
const entryPath = plugins.path.join(dirPath, entry);
const entryStats =
await plugins.smartpromise.fromCallback<plugins.fs.Stats>((cb) =>
plugins.fs.stat(entryPath, cb),
);
if (entryStats.isFile()) {
return entryPath;
}
}
return null;
};
// Wait for a file to appear in this directory
let firstFilePath = await getFirstFileInDirectory();
if (!firstFilePath) {
// Set up a watcher on the directory to see if a file appears
const directoryWatcher = plugins.fs.watch(dirPath, { persistent: true });
try {
// We'll poll for the existence of a file in that directory
while (!firstFilePath) {
if (Date.now() - startTime > timeoutMs) {
throw new Error(
`Timeout waiting for a file to appear in directory: ${dirPath}`,
);
}
firstFilePath = await getFirstFileInDirectory();
if (!firstFilePath) {
await plugins.smartdelay.delayFor(1000);
}
}
} finally {
directoryWatcher.close();
}
}
// Now that we have a file path, wait for that file to stabilize
await waitForSingleFileToBeStable(firstFilePath);
};
/**
* writes string or Smartfile to disk.
* @param fileArg
* @param fileNameArg
* @param fileBaseArg
*/
export let toFs = async (
fileContentArg: string | Buffer | SmartFile | StreamFile,
filePathArg: string,
optionsArg: {
respectRelative?: boolean;
} = {},
) => {
const done = plugins.smartpromise.defer();
// check args
if (!fileContentArg || !filePathArg) {
throw new Error('expected valid arguments');
}
// prepare actual write action
let fileContent: string | Buffer;
let fileEncoding: 'utf8' | 'binary' = 'utf8';
let filePath: string = filePathArg;
// handle Smartfile
if (fileContentArg instanceof SmartFile) {
fileContent = fileContentArg.contentBuffer;
// handle options
if (optionsArg.respectRelative) {
filePath = plugins.path.join(filePath, fileContentArg.path);
}
} else if (Buffer.isBuffer(fileContentArg)) {
fileContent = fileContentArg;
fileEncoding = 'binary';
} else if (typeof fileContentArg === 'string') {
fileContent = fileContentArg;
} else {
throw new Error('fileContent is neither string nor Smartfile');
}
await ensureDir(plugins.path.parse(filePath).dir);
plugins.fsExtra.writeFile(
filePath,
fileContent,
{ encoding: fileEncoding },
done.resolve,
);
return await done.promise;
};
export const stat = async (filePathArg: string) => {
return plugins.fsPromises.stat(filePathArg);
};

View File

@@ -1,215 +0,0 @@
/*
This file contains logic for streaming things from and to the filesystem
*/
import * as plugins from './plugins.js';
export const createReadStream = (pathArg: string) => {
return plugins.fs.createReadStream(pathArg);
};
export const createWriteStream = (pathArg: string) => {
return plugins.fs.createWriteStream(pathArg);
};
export const processFile = async (
filePath: string,
asyncFunc: (fileStream: plugins.stream.Readable) => Promise<void>,
): Promise<void> => {
return new Promise((resolve, reject) => {
const fileStream = createReadStream(filePath);
asyncFunc(fileStream).then(resolve).catch(reject);
});
};
export const processDirectory = async (
directoryPath: string,
asyncFunc: (fileStream: plugins.stream.Readable) => Promise<void>,
): Promise<void> => {
const files = plugins.fs.readdirSync(directoryPath, { withFileTypes: true });
for (const file of files) {
const fullPath = plugins.path.join(directoryPath, file.name);
if (file.isDirectory()) {
await processDirectory(fullPath, asyncFunc); // Recursively call processDirectory for directories
} else if (file.isFile()) {
await processFile(fullPath, asyncFunc); // Call async function with the file stream and wait for it
}
}
};
/**
* Checks if a file is ready to be streamed (exists and is not empty).
*/
export const isFileReadyForStreaming = async (
filePathArg: string,
): Promise<boolean> => {
try {
const stats = await plugins.fs.promises.stat(filePathArg);
return stats.size > 0;
} catch (error) {
if (error.code === 'ENOENT') {
// File does not exist
return false;
}
throw error; // Rethrow other unexpected errors
}
};
/**
* Waits for a file to be ready for streaming (exists and is not empty).
*/
export const waitForFileToBeReadyForStreaming = (
filePathArg: string,
): Promise<void> => {
return new Promise((resolve, reject) => {
// Normalize and resolve the file path
const filePath = plugins.path.resolve(filePathArg);
// Function to check file stats
const checkFile = (resolve: () => void, reject: (reason: any) => void) => {
plugins.fs.stat(filePath, (err, stats) => {
if (err) {
if (err.code === 'ENOENT') {
// File not found, wait and try again
return;
}
// Some other error occurred
return reject(err);
}
if (stats.size > 0) {
// File exists and is not empty, resolve the promise
resolve();
}
});
};
// Set up file watcher
const watcher = plugins.fs.watch(
filePath,
{ persistent: false },
(eventType) => {
if (eventType === 'change' || eventType === 'rename') {
checkFile(resolve, reject);
}
},
);
// Check file immediately in case it's already ready
checkFile(resolve, reject);
// Error handling
watcher.on('error', (error) => {
watcher.close();
reject(error);
});
});
};
export class SmartReadStream extends plugins.stream.Readable {
private watcher: plugins.fs.FSWatcher | null = null;
private lastReadSize: number = 0;
private endTimeout: NodeJS.Timeout | null = null;
private filePath: string;
private endDelay: number;
private reading: boolean = false;
constructor(
filePath: string,
endDelay = 60000,
opts?: plugins.stream.ReadableOptions,
) {
super(opts);
this.filePath = filePath;
this.endDelay = endDelay;
}
private startWatching(): void {
this.watcher = plugins.fs.watch(this.filePath, (eventType) => {
if (eventType === 'change') {
this.resetEndTimeout();
}
});
this.watcher.on('error', (error) => {
this.cleanup();
this.emit('error', error);
});
}
private resetEndTimeout(): void {
if (this.endTimeout) clearTimeout(this.endTimeout);
this.endTimeout = setTimeout(() => this.checkForEnd(), this.endDelay);
}
private checkForEnd(): void {
plugins.fs.stat(this.filePath, (err, stats) => {
if (err) {
this.emit('error', err);
return;
}
if (this.lastReadSize === stats.size) {
this.push(null); // Signal the end of the stream
this.cleanup();
} else {
this.lastReadSize = stats.size;
this.resetEndTimeout();
if (!this.reading) {
// We only want to continue reading if we were previously waiting for more data
this.reading = true;
this._read(10000); // Try to read more data
}
}
});
}
private cleanup(): void {
if (this.endTimeout) clearTimeout(this.endTimeout);
if (this.watcher) this.watcher.close();
}
_read(size: number): void {
this.reading = true;
const chunkSize = Math.min(size, 16384); // Read in chunks of 16KB
const buffer = Buffer.alloc(chunkSize);
plugins.fs.open(this.filePath, 'r', (err, fd) => {
if (err) {
this.emit('error', err);
return;
}
plugins.fs.read(
fd,
buffer,
0,
chunkSize,
this.lastReadSize,
(err, bytesRead, buffer) => {
if (err) {
this.emit('error', err);
return;
}
if (bytesRead > 0) {
this.lastReadSize += bytesRead;
this.push(buffer.slice(0, bytesRead)); // Push the data onto the stream
} else {
this.reading = false; // No more data to read for now
this.resetEndTimeout();
}
plugins.fs.close(fd, (err) => {
if (err) {
this.emit('error', err);
}
});
},
);
});
}
_destroy(error: Error | null, callback: (error: Error | null) => void): void {
this.cleanup();
callback(error);
}
}

View File

@@ -1,14 +1,11 @@
import * as plugins from './plugins.js';
import * as fsMod from './fs.js';
import * as fsStreamMod from './fsstream.js';
import * as interpreterMod from './interpreter.js';
import * as memoryMod from './memory.js';
// Export main classes - focused on in-memory file representations
export * from './classes.smartfile.js';
export * from './classes.streamfile.js';
export * from './classes.virtualdirectory.js';
export * from './classes.smartfile.factory.js';
export const fs = fsMod;
export const fsStream = fsStreamMod;
export const interpreter = interpreterMod;
export const memory = memoryMod;
// Note: Filesystem operations (fs, memory, fsStream, interpreter) have been removed.
// Use @push.rocks/smartfs for low-level filesystem operations.
// Use SmartFileFactory for creating SmartFile/StreamFile/VirtualDirectory instances.

View File

@@ -1,20 +0,0 @@
import * as plugins from './plugins.js';
export let filetype = (pathArg: string): string => {
const extName = plugins.path.extname(pathArg);
const fileType = extName.replace(/\.([a-z]*)/, '$1'); // remove . form fileType
return fileType;
};
export let objectFile = (fileStringArg: string, fileTypeArg) => {
switch (fileTypeArg) {
case 'yml':
case 'yaml':
return plugins.yaml.load(fileStringArg);
case 'json':
return JSON.parse(fileStringArg);
default:
console.error('file type ' + fileTypeArg.blue + ' not supported');
break;
}
};

View File

@@ -1,102 +0,0 @@
import * as plugins from './plugins.js';
import { SmartFile } from './classes.smartfile.js';
import * as smartfileFs from './fs.js';
import * as interpreter from './interpreter.js';
import type { StreamFile } from './classes.streamfile.js';
/**
* converts file to Object
* @param fileStringArg
* @param fileTypeArg
* @returns {any|any}
*/
export let toObject = (fileStringArg: string, fileTypeArg: string) => {
return interpreter.objectFile(fileStringArg, fileTypeArg);
};
export interface IToFsOptions {
respectRelative?: boolean;
}
/**
* writes string or Smartfile to disk.
* @param fileArg
* @param fileNameArg
* @param fileBaseArg
*/
export let toFs = async (
fileContentArg: string | Buffer | SmartFile | StreamFile,
filePathArg: string,
optionsArg: IToFsOptions = {},
) => {
const done = plugins.smartpromise.defer();
// check args
if (!fileContentArg || !filePathArg) {
throw new Error('expected valid arguments');
}
// prepare actual write action
let fileContent: string | Buffer;
let fileEncoding: 'utf8' | 'binary' = 'utf8';
let filePath: string = filePathArg;
// handle Smartfile
if (fileContentArg instanceof SmartFile) {
fileContent = fileContentArg.contentBuffer;
// handle options
if (optionsArg.respectRelative) {
filePath = plugins.path.join(filePath, fileContentArg.path);
}
} else if (Buffer.isBuffer(fileContentArg)) {
fileContent = fileContentArg;
fileEncoding = 'binary';
} else if (typeof fileContentArg === 'string') {
fileContent = fileContentArg;
} else {
throw new Error('fileContent is neither string nor Smartfile');
}
await smartfileFs.ensureDir(plugins.path.parse(filePath).dir);
plugins.fsExtra.writeFile(
filePath,
fileContent,
{ encoding: fileEncoding },
done.resolve,
);
return await done.promise;
};
/**
* writes a string or a Smartfile to disk synchronously, only supports string
* @param fileArg
* @param filePathArg
*/
export const toFsSync = (fileArg: string, filePathArg: string) => {
// function checks to abort if needed
if (!fileArg || !filePathArg) {
throw new Error('expected a valid arguments');
}
// prepare actual write action
let fileString: string;
const filePath: string = filePathArg;
if (typeof fileArg !== 'string') {
throw new Error('fileArg is not of type String.');
} else if (typeof fileArg === 'string') {
fileString = fileArg;
}
plugins.fsExtra.writeFileSync(filePath, fileString, { encoding: 'utf8' });
};
export let smartfileArrayToFs = async (
smartfileArrayArg: SmartFile[],
dirArg: string,
) => {
await smartfileFs.ensureDir(dirArg);
for (const smartfile of smartfileArrayArg) {
await toFs(smartfile, dirArg, {
respectRelative: true,
});
}
};