push.rocks/smartfile

Fork 0

Philipp Kunz 255cc844ad fix(core): update

2024-05-17 17:49:57 +02:00

8.2 KiB

Raw Permalink Blame History

@push.rocks/smartfile

Provides a robust suite of tools for managing files in Node.js using TypeScript.

Install

To integrate @push.rocks/smartfile into your project, run:

npm install @push.rocks/smartfile

Usage

@push.rocks/smartfile offers extensive file management utilities, enabling seamless file processing with TypeScript in a Node.js environment. Below are detailed examples showcasing various features of the module.

Quick Start

First, ensure you're working in an environment that supports ECMAScript modules (ESM) and TypeScript. Here’s how you’d generally import and use @push.rocks/smartfile:

import { SmartFile, StreamFile, VirtualDirectory, fs, memory, interpreter } from '@push.rocks/smartfile';

Working with `SmartFile`

Reading Files

To read from a file and convert it to a SmartFile instance:

const myJsonSmartFile: SmartFile = await SmartFile.fromFilePath('./somePath/data.json');
const jsonData = JSON.parse(myJsonSmartFile.contents.toString());
console.log(jsonData);  // Assuming the file contains JSON content

Writing Files

To write data to a file through a SmartFile:

const filePath: string = './output/outputData.json';
const content: string = JSON.stringify({ key: 'value' });
await memory.toFs(content, filePath);

Streaming Large Files with `StreamFile`

When dealing with large files, you can use StreamFile to handle such files efficiently, minimizing memory usage:

const largeFile: StreamFile = await StreamFile.fromPath('./largeInput/largeFile.mp4');
await largeFile.writeToDisk('./largeOutput/largeFileCopy.mp4');

Working with Virtual Directories

Handling multiple files as if they were part of a file system:

const virtualDir = await VirtualDirectory.fromFsDirPath('./data/inputDir');
await virtualDir.saveToDisk('./data/outputDir');

File System Operations

@push.rocks/smartfile provides a suite of utilities for common file system operations such as copying, deleting, and listing files or directories.

Copying a File

await fs.copy('./sourceFile.txt', './destinationFile.txt');

Deleting a Directory

await fs.remove('./directoryToDelete');

Listing Files in a Directory

const fileList: string[] = await fs.listFiles('./someDirectory');
console.log(fileList);

Advanced File Management

For specialized file operations, such as editing the contents of a file or streaming files from URLs, @push.rocks/smartfile includes advanced management features.

Editing a File’s Contents

const smartFileToEdit: SmartFile = await SmartFile.fromFilePath('./editableFile.txt');
await smartFileToEdit.editContentAsString(async (content) => content.replace(/originalText/g, 'newText'));
await smartFileToEdit.write();

Streaming a File from a URL

const streamedFile: StreamFile = await StreamFile.fromUrl('https://example.com/file.pdf');
await streamedFile.writeToDisk('./downloadedFiles/file.pdf');

Working with File Buffers and Streams

@push.rocks/smartfile allows you to easily work with files using Buffers and Streams, facilitating operations like file transformations, uploads, and downloads.

Creating a SmartFile from a Buffer

const buffer: Buffer = Buffer.from('Sample data');
const bufferSmartFile: SmartFile = await SmartFile.fromBuffer('./bufferFile.txt', buffer);
await bufferSmartFile.write();

Using `VirtualDirectory` for Complex File Management

VirtualDirectory simplifies the management of multiple files that are otherwise scattered across different directories or created at runtime.

Creating a `VirtualDirectory`

To create a VirtualDirectory from an existing file directory:

const virtualDirectory = await VirtualDirectory.fromFsDirPath('./someDirectory');

Adding More Files

You can add more SmartFile instances to your VirtualDirectory:

const additionalFiles = [
  await SmartFile.fromFilePath('./anotherDirectory/file1.txt'),
  await SmartFile.fromFilePath('./anotherDirectory/file2.txt')
];
virtualDirectory.addSmartfiles(additionalFiles);

Saving `VirtualDirectory` to Disk

Save your entire VirtualDirectory to disk:

await virtualDirectory.saveToDisk('./outputDirectory');

Utilizing StreamFile for Efficient File Handling

Using StreamFile can be especially beneficial for large files or when performing streaming operations.

Streaming from a URL

StreamFile provides capabilities to stream files directly from URLs, making it easier to work with remote content.

const urlStreamFile: StreamFile = await StreamFile.fromUrl('https://example.com/largefile.zip');
await urlStreamFile.writeToDisk('./downloadedFiles/largefile.zip');

Combining Buffer and Stream Approaches

Create StreamFile from a buffer for efficient, multi-use streams.

const buffer = Buffer.from('Streaming buffer content');
const bufferStreamFile = StreamFile.fromBuffer(buffer, 'bufferBasedStream.txt');
await bufferStreamFile.writeToDisk('./streams/bufferBasedStream.txt');

Read and Write Operations with StreamFile

Perform read and write operations efficiently using StreamFile.

const fileStream = await StreamFile.fromPath('./inputData/largeFile.data');
await fileStream.writeToDisk('./outputData/largeFileCopy.data');

Check for completeness of your read and write operations, ensuring the integrity of file content.

const readBuffer = await fileStream.getContentAsBuffer();
console.log(readBuffer.toString());

Ensuring File Readiness for Streaming

Ensure a file is ready for streaming or create a custom readable stream incorporating data transformation.

const smartReadStream = new SmartReadStream('./incomingData/sample.data');
smartReadStream.on('data', (chunk) => {
  console.log('New Data Chunk:', chunk.toString());
});

File Transformation with SmartReadStream

Perform transformations on the stream of data while reading:

smartReadStream.on('data', (chunk) => {
  // Perform some transformation
  const transformedChunk = chunk.toString().toUpperCase();
  console.log('Transformed Data Chunk:', transformedChunk);
});

Streaming with SmartReadStream

Stream data from a SmartReadStream to a file efficiently managing large datasets.

const transformedWriteStream = fs.createWriteStream('./processedData/transformed.data');
smartReadStream.pipe(transformedWriteStream);

@push.rocks/smartfile significantly simplifies the handling of complex file operations in Node.js projects, making these tasks straightforward while maintaining efficiency and ease of use. Explore and leverage these features to enhance your project's file management capabilities.

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

8.2 KiB Raw Permalink Blame History Unescape Escape