fix(fs): Improve fs and stream handling, enhance SmartFile/StreamFile, update tests and CI configs
This commit is contained in:
429
readme.md
429
readme.md
@@ -1,226 +1,315 @@
|
||||
# @push.rocks/smartfile
|
||||
# @push.rocks/smartfile 📁
|
||||
|
||||
> Provides a robust suite of tools for managing files in Node.js using TypeScript.
|
||||
> **A powerful, TypeScript-based file management library for Node.js**
|
||||
|
||||
## Install
|
||||
## 🚀 What is smartfile?
|
||||
|
||||
To integrate `@push.rocks/smartfile` into your project, run:
|
||||
`@push.rocks/smartfile` is your go-to solution for file operations in Node.js. It offers a clean, promise-based API for handling files, directories, streams, and even virtual filesystems - all while maintaining maximum performance and reliability.
|
||||
|
||||
Think of it as `fs` on steroids, with TypeScript superpowers! 💪
|
||||
|
||||
## 💾 Installation
|
||||
|
||||
```bash
|
||||
npm install @push.rocks/smartfile
|
||||
```
|
||||
|
||||
## Usage
|
||||
## ✨ Features
|
||||
|
||||
`@push.rocks/smartfile` offers extensive file management utilities, enabling seamless file processing with TypeScript in a Node.js environment. Below are detailed examples showcasing various features of the module.
|
||||
- 🔥 **Streaming Support** - Handle massive files with ease using `StreamFile`
|
||||
- 📦 **Virtual Directories** - Work with in-memory file structures
|
||||
- 🌐 **URL Support** - Directly work with files from URLs
|
||||
- 🎯 **TypeScript First** - Full type safety and IntelliSense support
|
||||
- ⚡ **Promise-based API** - Modern async/await patterns throughout
|
||||
- 🛠️ **Comprehensive Toolset** - From basic CRUD to advanced operations
|
||||
|
||||
### Quick Start
|
||||
|
||||
First, ensure you're working in an environment that supports ECMAScript modules (ESM) and TypeScript. Here’s how you’d generally import and use `@push.rocks/smartfile`:
|
||||
## 📚 Quick Start
|
||||
|
||||
```typescript
|
||||
import { SmartFile, StreamFile, VirtualDirectory, fs, memory, interpreter } from '@push.rocks/smartfile';
|
||||
import * as smartfile from '@push.rocks/smartfile';
|
||||
|
||||
// Read a file
|
||||
const content = await smartfile.fs.toStringSync('./my-file.txt');
|
||||
|
||||
// Write a file
|
||||
await smartfile.memory.toFs('Hello World!', './output.txt');
|
||||
|
||||
// Work with JSON
|
||||
const data = await smartfile.fs.toObjectSync('./data.json');
|
||||
```
|
||||
|
||||
### Working with `SmartFile`
|
||||
## 🎨 Core Components
|
||||
|
||||
#### Reading Files
|
||||
### SmartFile Class
|
||||
|
||||
To read from a file and convert it to a `SmartFile` instance:
|
||||
The `SmartFile` class represents a single file with powerful manipulation capabilities:
|
||||
|
||||
```typescript
|
||||
const myJsonSmartFile: SmartFile = await SmartFile.fromFilePath('./somePath/data.json');
|
||||
const jsonData = JSON.parse(myJsonSmartFile.contents.toString());
|
||||
console.log(jsonData); // Assuming the file contains JSON content
|
||||
import { SmartFile } from '@push.rocks/smartfile';
|
||||
|
||||
// Create from file path
|
||||
const fileFromPath = await SmartFile.fromFilePath('./data.json');
|
||||
|
||||
// Create from URL
|
||||
const fileFromUrl = await SmartFile.fromUrl('https://example.com/config.json');
|
||||
|
||||
// Create from text
|
||||
const fileFromText = await SmartFile.fromString(
|
||||
'./my-file.txt',
|
||||
'This is my content',
|
||||
'utf8'
|
||||
);
|
||||
|
||||
// Create from Buffer
|
||||
const fileFromBuffer = await SmartFile.fromBuffer(
|
||||
'./binary.dat',
|
||||
Buffer.from([0x00, 0x01, 0x02])
|
||||
);
|
||||
|
||||
// Edit content
|
||||
await fileFromPath.editContentAsString(async (content) => {
|
||||
return content.replace(/old/g, 'new');
|
||||
});
|
||||
|
||||
// Write to disk
|
||||
await fileFromPath.write();
|
||||
|
||||
// Get content
|
||||
const contentString = fileFromPath.parseContentAsString();
|
||||
const contentBuffer = fileFromPath.parseContentAsBuffer();
|
||||
```
|
||||
|
||||
#### Writing Files
|
||||
### StreamFile Class 🌊
|
||||
|
||||
To write data to a file through a `SmartFile`:
|
||||
Perfect for handling large files without memory overhead:
|
||||
|
||||
```typescript
|
||||
const filePath: string = './output/outputData.json';
|
||||
const content: string = JSON.stringify({ key: 'value' });
|
||||
await memory.toFs(content, filePath);
|
||||
import { StreamFile } from '@push.rocks/smartfile';
|
||||
|
||||
// Create from path
|
||||
const streamFile = await StreamFile.fromPath('./bigfile.zip');
|
||||
|
||||
// Create from URL
|
||||
const urlStream = await StreamFile.fromUrl('https://example.com/large.mp4');
|
||||
|
||||
// Create from buffer
|
||||
const bufferStream = StreamFile.fromBuffer(
|
||||
Buffer.from('streaming content'),
|
||||
'stream.txt'
|
||||
);
|
||||
|
||||
// Write to disk
|
||||
await streamFile.writeToDisk('./output/bigfile.zip');
|
||||
|
||||
// Get as buffer (careful with large files!)
|
||||
const buffer = await streamFile.getContentAsBuffer();
|
||||
|
||||
// Get as stream
|
||||
const readStream = await streamFile.createReadStream();
|
||||
```
|
||||
|
||||
### Streaming Large Files with `StreamFile`
|
||||
### VirtualDirectory Class 📂
|
||||
|
||||
When dealing with large files, you can use `StreamFile` to handle such files efficiently, minimizing memory usage:
|
||||
Manage collections of files as virtual filesystems:
|
||||
|
||||
```typescript
|
||||
const largeFile: StreamFile = await StreamFile.fromPath('./largeInput/largeFile.mp4');
|
||||
await largeFile.writeToDisk('./largeOutput/largeFileCopy.mp4');
|
||||
import { VirtualDirectory } from '@push.rocks/smartfile';
|
||||
|
||||
// Create from filesystem
|
||||
const vDir = await VirtualDirectory.fromFsDirPath('./src');
|
||||
|
||||
// Create from file array
|
||||
const vDirFromFiles = await VirtualDirectory.fromFileArray([
|
||||
await SmartFile.fromFilePath('./file1.txt'),
|
||||
await SmartFile.fromFilePath('./file2.txt')
|
||||
]);
|
||||
|
||||
// Add files
|
||||
vDir.addSmartfiles([
|
||||
await SmartFile.fromString('./virtual/new.txt', 'content')
|
||||
]);
|
||||
|
||||
// List files
|
||||
const files = vDir.listFiles();
|
||||
const directories = vDir.listDirectories();
|
||||
|
||||
// Get file
|
||||
const file = vDir.getFileByPath('./some/path.txt');
|
||||
|
||||
// Save to disk
|
||||
await vDir.saveToDisk('./output');
|
||||
|
||||
// Load from disk
|
||||
await vDir.loadFromDisk('./source');
|
||||
```
|
||||
|
||||
### Working with Virtual Directories
|
||||
## 🛠️ File Operations
|
||||
|
||||
Handling multiple files as if they were part of a file system:
|
||||
### Basic Operations
|
||||
|
||||
```typescript
|
||||
const virtualDir = await VirtualDirectory.fromFsDirPath('./data/inputDir');
|
||||
await virtualDir.saveToDisk('./data/outputDir');
|
||||
// Check existence
|
||||
const exists = await smartfile.fs.fileExists('./file.txt');
|
||||
const existsSync = smartfile.fs.fileExistsSync('./file.txt');
|
||||
|
||||
// Read operations
|
||||
const content = await smartfile.fs.toStringSync('./file.txt');
|
||||
const buffer = await smartfile.fs.toBuffer('./file.txt');
|
||||
const object = await smartfile.fs.toObjectSync('./data.json');
|
||||
|
||||
// Write operations
|
||||
await smartfile.memory.toFs('content', './output.txt');
|
||||
smartfile.memory.toFsSync('content', './output-sync.txt');
|
||||
|
||||
// Copy operations
|
||||
await smartfile.fs.copy('./source.txt', './dest.txt');
|
||||
await smartfile.fs.copy('./src-dir', './dest-dir');
|
||||
|
||||
// Delete operations
|
||||
await smartfile.fs.remove('./file.txt');
|
||||
await smartfile.fs.removeSync('./file-sync.txt');
|
||||
await smartfile.fs.removeMany(['./file1.txt', './file2.txt']);
|
||||
|
||||
// Ensure operations (create if not exists)
|
||||
await smartfile.fs.ensureDir('./my/deep/directory');
|
||||
await smartfile.fs.ensureFile('./my/file.txt');
|
||||
await smartfile.fs.ensureEmptyDir('./empty-dir');
|
||||
```
|
||||
|
||||
### File System Operations
|
||||
|
||||
`@push.rocks/smartfile` provides a suite of utilities for common file system operations such as copying, deleting, and listing files or directories.
|
||||
|
||||
#### Copying a File
|
||||
### Directory Operations
|
||||
|
||||
```typescript
|
||||
await fs.copy('./sourceFile.txt', './destinationFile.txt');
|
||||
// List contents
|
||||
const files = await smartfile.fs.listFiles('./directory');
|
||||
const folders = await smartfile.fs.listFolders('./directory');
|
||||
const items = await smartfile.fs.listAllItems('./directory');
|
||||
|
||||
// Get file tree
|
||||
const tree = await smartfile.fs.listFileTree('./src', '**/*.ts');
|
||||
|
||||
// Directory checks
|
||||
const isDir = await smartfile.fs.isDirectory('./path');
|
||||
const isFile = await smartfile.fs.isFile('./path');
|
||||
```
|
||||
|
||||
#### Deleting a Directory
|
||||
### Advanced Features
|
||||
|
||||
```typescript
|
||||
await fs.remove('./directoryToDelete');
|
||||
// Wait for file to be ready
|
||||
await smartfile.fs.waitForFileToBeReady('./file.txt');
|
||||
|
||||
// Stream operations
|
||||
const readStream = smartfile.fsStream.createReadStream('./input.txt');
|
||||
const writeStream = smartfile.fsStream.createWriteStream('./output.txt');
|
||||
|
||||
// File type detection
|
||||
const fileType = smartfile.interpreter.filetype('./document.pdf');
|
||||
// Returns: 'pdf'
|
||||
|
||||
// Smart read stream (with custom processing)
|
||||
const smartStream = new smartfile.fsStream.SmartReadStream('./data.txt');
|
||||
smartStream.on('data', (chunk) => {
|
||||
// Process chunk
|
||||
console.log(chunk.toString());
|
||||
});
|
||||
```
|
||||
|
||||
#### Listing Files in a Directory
|
||||
## 🔄 Working with Multiple Files
|
||||
|
||||
```typescript
|
||||
const fileList: string[] = await fs.listFiles('./someDirectory');
|
||||
console.log(fileList);
|
||||
```
|
||||
// Process multiple SmartFiles
|
||||
const files = await smartfile.fs.fileTreeToObject(
|
||||
'./src',
|
||||
'**/*.{ts,js}'
|
||||
);
|
||||
|
||||
### Advanced File Management
|
||||
|
||||
For specialized file operations, such as editing the contents of a file or streaming files from URLs, `@push.rocks/smartfile` includes advanced management features.
|
||||
|
||||
#### Editing a File’s Contents
|
||||
|
||||
```typescript
|
||||
const smartFileToEdit: SmartFile = await SmartFile.fromFilePath('./editableFile.txt');
|
||||
await smartFileToEdit.editContentAsString(async (content) => content.replace(/originalText/g, 'newText'));
|
||||
await smartFileToEdit.write();
|
||||
```
|
||||
|
||||
#### Streaming a File from a URL
|
||||
|
||||
```typescript
|
||||
const streamedFile: StreamFile = await StreamFile.fromUrl('https://example.com/file.pdf');
|
||||
await streamedFile.writeToDisk('./downloadedFiles/file.pdf');
|
||||
```
|
||||
|
||||
### Working with File Buffers and Streams
|
||||
|
||||
`@push.rocks/smartfile` allows you to easily work with files using Buffers and Streams, facilitating operations like file transformations, uploads, and downloads.
|
||||
|
||||
#### Creating a SmartFile from a Buffer
|
||||
|
||||
```typescript
|
||||
const buffer: Buffer = Buffer.from('Sample data');
|
||||
const bufferSmartFile: SmartFile = await SmartFile.fromBuffer('./bufferFile.txt', buffer);
|
||||
await bufferSmartFile.write();
|
||||
```
|
||||
|
||||
### Using `VirtualDirectory` for Complex File Management
|
||||
|
||||
`VirtualDirectory` simplifies the management of multiple files that are otherwise scattered across different directories or created at runtime.
|
||||
|
||||
#### Creating a `VirtualDirectory`
|
||||
|
||||
To create a `VirtualDirectory` from an existing file directory:
|
||||
|
||||
```typescript
|
||||
const virtualDirectory = await VirtualDirectory.fromFsDirPath('./someDirectory');
|
||||
```
|
||||
|
||||
#### Adding More Files
|
||||
|
||||
You can add more `SmartFile` instances to your `VirtualDirectory`:
|
||||
|
||||
```typescript
|
||||
const additionalFiles = [
|
||||
await SmartFile.fromFilePath('./anotherDirectory/file1.txt'),
|
||||
await SmartFile.fromFilePath('./anotherDirectory/file2.txt')
|
||||
// Write array to disk
|
||||
const smartfiles = [
|
||||
await SmartFile.fromString('file1.txt', 'content1'),
|
||||
await SmartFile.fromString('file2.txt', 'content2')
|
||||
];
|
||||
virtualDirectory.addSmartfiles(additionalFiles);
|
||||
await smartfile.memory.smartfileArrayToFs(smartfiles, './output');
|
||||
```
|
||||
|
||||
#### Saving `VirtualDirectory` to Disk
|
||||
## 🎯 Real-World Examples
|
||||
|
||||
Save your entire `VirtualDirectory` to disk:
|
||||
### Website Bundler
|
||||
```typescript
|
||||
// Bundle website assets
|
||||
const website = await VirtualDirectory.fromFsDirPath('./website');
|
||||
const bundle = await website.smartfileArray;
|
||||
|
||||
// Process all CSS files
|
||||
for (const file of bundle.filter(f => f.path.endsWith('.css'))) {
|
||||
await file.editContentAsString(async (css) => {
|
||||
// Minify CSS here
|
||||
return css.replace(/\s+/g, ' ');
|
||||
});
|
||||
}
|
||||
|
||||
// Save processed bundle
|
||||
await website.saveToDisk('./dist');
|
||||
```
|
||||
|
||||
### File Watcher & Processor
|
||||
```typescript
|
||||
// Watch for new files and process them
|
||||
import { SmartFile, StreamFile } from '@push.rocks/smartfile';
|
||||
|
||||
async function processLargeFile(filePath: string) {
|
||||
const streamFile = await StreamFile.fromPath(filePath);
|
||||
|
||||
// Stream to processed location
|
||||
await streamFile.writeToDisk(`./processed/${path.basename(filePath)}`);
|
||||
|
||||
// Clean up original
|
||||
await smartfile.fs.remove(filePath);
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Manager
|
||||
```typescript
|
||||
// Load and merge config files
|
||||
const defaultConfig = await smartfile.fs.toObjectSync('./config.default.json');
|
||||
const userConfig = await smartfile.fs.toObjectSync('./config.user.json');
|
||||
|
||||
const merged = { ...defaultConfig, ...userConfig };
|
||||
|
||||
await smartfile.memory.toFs(
|
||||
JSON.stringify(merged, null, 2),
|
||||
'./config.final.json'
|
||||
);
|
||||
```
|
||||
|
||||
## 🌟 API Reference
|
||||
|
||||
### Core Modules
|
||||
|
||||
- `fs` - File system operations
|
||||
- `fsStream` - Streaming operations
|
||||
- `memory` - Memory/buffer operations
|
||||
- `interpreter` - File type detection
|
||||
|
||||
### Main Classes
|
||||
|
||||
- `SmartFile` - Single file representation
|
||||
- `StreamFile` - Streaming file operations
|
||||
- `VirtualDirectory` - Virtual filesystem management
|
||||
|
||||
## 🏗️ TypeScript Support
|
||||
|
||||
Full TypeScript support with comprehensive type definitions:
|
||||
|
||||
```typescript
|
||||
await virtualDirectory.saveToDisk('./outputDirectory');
|
||||
import type { SmartFile, StreamFile, VirtualDirectory } from '@push.rocks/smartfile';
|
||||
|
||||
// All methods are fully typed
|
||||
const processFile = async (file: SmartFile): Promise<void> => {
|
||||
const content = file.parseContentAsString();
|
||||
// TypeScript knows content is string
|
||||
};
|
||||
```
|
||||
|
||||
### Utilizing StreamFile for Efficient File Handling
|
||||
|
||||
Using `StreamFile` can be especially beneficial for large files or when performing streaming operations.
|
||||
|
||||
#### Streaming from a URL
|
||||
|
||||
`StreamFile` provides capabilities to stream files directly from URLs, making it easier to work with remote content.
|
||||
|
||||
```typescript
|
||||
const urlStreamFile: StreamFile = await StreamFile.fromUrl('https://example.com/largefile.zip');
|
||||
await urlStreamFile.writeToDisk('./downloadedFiles/largefile.zip');
|
||||
```
|
||||
|
||||
### Combining Buffer and Stream Approaches
|
||||
|
||||
Create `StreamFile` from a buffer for efficient, multi-use streams.
|
||||
|
||||
```typescript
|
||||
const buffer = Buffer.from('Streaming buffer content');
|
||||
const bufferStreamFile = StreamFile.fromBuffer(buffer, 'bufferBasedStream.txt');
|
||||
await bufferStreamFile.writeToDisk('./streams/bufferBasedStream.txt');
|
||||
```
|
||||
|
||||
### Read and Write Operations with StreamFile
|
||||
|
||||
Perform read and write operations efficiently using `StreamFile`.
|
||||
|
||||
```typescript
|
||||
const fileStream = await StreamFile.fromPath('./inputData/largeFile.data');
|
||||
await fileStream.writeToDisk('./outputData/largeFileCopy.data');
|
||||
```
|
||||
|
||||
Check for completeness of your read and write operations, ensuring the integrity of file content.
|
||||
|
||||
```typescript
|
||||
const readBuffer = await fileStream.getContentAsBuffer();
|
||||
console.log(readBuffer.toString());
|
||||
```
|
||||
|
||||
### Ensuring File Readiness for Streaming
|
||||
|
||||
Ensure a file is ready for streaming or create a custom readable stream incorporating data transformation.
|
||||
|
||||
```typescript
|
||||
const smartReadStream = new SmartReadStream('./incomingData/sample.data');
|
||||
smartReadStream.on('data', (chunk) => {
|
||||
console.log('New Data Chunk:', chunk.toString());
|
||||
});
|
||||
```
|
||||
|
||||
### File Transformation with SmartReadStream
|
||||
|
||||
Perform transformations on the stream of data while reading:
|
||||
|
||||
```typescript
|
||||
smartReadStream.on('data', (chunk) => {
|
||||
// Perform some transformation
|
||||
const transformedChunk = chunk.toString().toUpperCase();
|
||||
console.log('Transformed Data Chunk:', transformedChunk);
|
||||
});
|
||||
```
|
||||
|
||||
### Streaming with SmartReadStream
|
||||
|
||||
Stream data from a `SmartReadStream` to a file efficiently managing large datasets.
|
||||
|
||||
```typescript
|
||||
const transformedWriteStream = fs.createWriteStream('./processedData/transformed.data');
|
||||
smartReadStream.pipe(transformedWriteStream);
|
||||
```
|
||||
|
||||
`@push.rocks/smartfile` significantly simplifies the handling of complex file operations in Node.js projects, making these tasks straightforward while maintaining efficiency and ease of use. Explore and leverage these features to enhance your project's file management capabilities.
|
||||
|
||||
## License and Legal Information
|
||||
|
||||
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
|
||||
@@ -238,4 +327,4 @@ Registered at District court Bremen HRB 35230 HB, Germany
|
||||
|
||||
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
|
||||
|
||||
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
|
||||
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
|
||||
Reference in New Issue
Block a user