feat(tartools): add streaming TAR support (tar-stream), Node.js streaming APIs for TarTools, and browser / web bundle docs

2026-01-01 23:40:13 +00:00
parent d97e9c1dce
commit db48fcd455
13 changed files with 456 additions and 96 deletions
--- a/readme.md
+++ b/readme.md
@@ -1,6 +1,6 @@
 # @push.rocks/smartarchive 📦

-A powerful, streaming-first archive manipulation library with a fluent builder API. Works seamlessly in Node.js and Deno.
+A powerful, streaming-first archive manipulation library with a fluent builder API. Works seamlessly in **Node.js**, **Deno**, and **browsers**.

 ## Issue Reporting and Security

@@ -12,11 +12,12 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
 - 🌊 **Streaming-first architecture** – Process large archives without memory constraints
 - ✨ **Fluent builder API** – Chain methods for readable, expressive code
 - 🎯 **Smart detection** – Automatically identifies archive types via magic bytes
- ⚡ **High performance** – Built on `tar-stream` and `fflate` for speed
+- ⚡ **High performance** – Built on `modern-tar` and `fflate` for speed
 - 🔧 **Flexible I/O** – Work with files, URLs, streams, and buffers seamlessly
 - 🛠️ **Modern TypeScript** – Full type safety and excellent IDE support
 - 🔄 **Dual-mode operation** – Extract existing archives OR create new ones
- 🦕 **Cross-runtime** – Works in both Node.js and Deno environments
+- 🦕 **Cross-runtime** – Works in Node.js, Deno, and browsers
+- 🌐 **Browser-ready** – Dedicated browser bundle with zero Node.js dependencies

 ## Installation 📥

@@ -71,6 +72,59 @@ await SmartArchive.create()
  .extract('./node_modules/lodash');
 ```

+## Browser Usage 🌐
+
+smartarchive provides a dedicated browser-compatible bundle with no Node.js dependencies:
+
+```typescript
+// Import from the /web subpath for browser environments
+import { TarTools, ZipTools, GzipTools, Bzip2Tools } from '@push.rocks/smartarchive/web';
+
+// Create a TAR archive in the browser
+const tarTools = new TarTools();
+const tarBuffer = await tarTools.packFiles([
+  { archivePath: 'hello.txt', content: 'Hello from the browser!' },
+  { archivePath: 'data.json', content: JSON.stringify({ browser: true }) }
+]);
+
+// Create a TAR.GZ archive
+const tgzBuffer = await tarTools.packFilesToTarGz([
+  { archivePath: 'file.txt', content: 'Compressed!' }
+], 6);
+
+// Extract a TAR archive
+const entries = await tarTools.extractTar(tarBuffer);
+for (const entry of entries) {
+  console.log(`${entry.path}: ${entry.content.length} bytes`);
+}
+
+// Work with ZIP files
+const zipTools = new ZipTools();
+const zipBuffer = await zipTools.createZip([
+  { archivePath: 'doc.txt', content: 'Document content' }
+], 6);
+
+const zipEntries = await zipTools.extractZip(zipBuffer);
+
+// GZIP compression
+const gzipTools = new GzipTools();
+const compressed = gzipTools.compressSync(new TextEncoder().encode('Hello World'), 6);
+const decompressed = gzipTools.decompressSync(compressed);
+```
+
+### Browser Bundle Exports
+
+The `/web` subpath exports these browser-compatible tools:
+
+| Export | Description |
+|--------|-------------|
+| `TarTools` | Create and extract TAR and TAR.GZ archives |
+| `ZipTools` | Create and extract ZIP archives |
+| `GzipTools` | GZIP compression and decompression |
+| `Bzip2Tools` | BZIP2 decompression (extraction only) |
+
+> 💡 **Note:** The browser bundle does **not** include `SmartArchive` (which requires filesystem access). Use the individual tool classes for browser applications.
+
 ## Core Concepts 💡

 ### Fluent Builder Pattern
@@ -294,21 +348,14 @@ await SmartArchive.create()
 // Use GzipTools directly for compression/decompression
 const gzipTools = new GzipTools();

-// Compress a buffer
-const compressed = await gzipTools.compress(Buffer.from('Hello World'), 9);
-const decompressed = await gzipTools.decompress(compressed);
+// Compress a buffer (sync and async available)
+const input = new TextEncoder().encode('Hello World');
+const compressed = gzipTools.compressSync(input, 9);
+const decompressed = gzipTools.decompressSync(compressed);

-// Synchronous operations
-const compressedSync = gzipTools.compressSync(inputBuffer, 6);
-const decompressedSync = gzipTools.decompressSync(compressedSync);
-
-// Streaming
-const compressStream = gzipTools.getCompressionStream(6);
-const decompressStream = gzipTools.getDecompressionStream();
-
-createReadStream('./input.txt')
-  .pipe(compressStream)
-  .pipe(createWriteStream('./output.gz'));
+// Async versions (internally use sync for cross-runtime compatibility)
+const compressedAsync = await gzipTools.compress(input, 6);
+const decompressedAsync = await gzipTools.decompress(compressedAsync);
 ```

 ### Working with TAR archives directly
@@ -318,27 +365,90 @@ import { TarTools } from '@push.rocks/smartarchive';

 const tarTools = new TarTools();

-// Create a TAR archive manually
-const pack = await tarTools.getPackStream();
+// Create a TAR archive from entries (buffer-based, good for small files)
+const tarBuffer = await tarTools.packFiles([
+  { archivePath: 'hello.txt', content: 'Hello, World!' },
+  { archivePath: 'data.json', content: JSON.stringify({ foo: 'bar' }) }
+]);

+// Create a TAR.GZ archive
+const tgzBuffer = await tarTools.packFilesToTarGz([
+  { archivePath: 'file.txt', content: 'Compressed content' }
+], 6);
+
+// Extract a TAR archive
+const entries = await tarTools.extractTar(tarBuffer);
+for (const entry of entries) {
+  console.log(`${entry.path}: ${entry.isDirectory ? 'dir' : 'file'}`);
+}
+
+// Extract a TAR.GZ archive
+const tgzEntries = await tarTools.extractTarGz(tgzBuffer);
+
+// Node.js only: Pack a directory (buffer-based)
+const dirBuffer = await tarTools.packDirectory('./src');
+const dirTgzBuffer = await tarTools.packDirectoryToTarGz('./src', 9);
+```
+
+### Streaming TAR for Large Files (Node.js only) 🚀
+
+For large files that don't fit in memory, use the streaming APIs:
+
+```typescript
+import { TarTools } from '@push.rocks/smartarchive';
+import * as fs from 'fs';
+
+const tarTools = new TarTools();
+
+// ===== STREAMING PACK =====
+// Create a TAR pack stream - files are processed one at a time
+const pack = tarTools.getPackStream();
+
+// Add files with streaming content (requires size for streams)
 await tarTools.addFileToPack(pack, {
-  fileName: 'hello.txt',
-  content: 'Hello, World!'
+  fileName: 'small.txt',
+  content: 'Hello World'  // Strings and buffers auto-detect size
 });

 await tarTools.addFileToPack(pack, {
-  fileName: 'data.json',
-  content: Buffer.from(JSON.stringify({ foo: 'bar' }))
+  fileName: 'large-video.mp4',
+  content: fs.createReadStream('./video.mp4'),
+  size: fs.statSync('./video.mp4').size  // Size required for streams
 });

 pack.finalize();
-pack.pipe(createWriteStream('./output.tar'));
+pack.pipe(fs.createWriteStream('output.tar'));

-// Pack a directory to TAR.GZ buffer
-const tgzBuffer = await tarTools.packDirectoryToTarGz('./src', 6);
+// ===== STREAMING DIRECTORY PACK =====
+// Pack entire directory with true streaming (no buffering)
+const tarStream = await tarTools.getDirectoryPackStream('./large-folder');
+tarStream.pipe(fs.createWriteStream('backup.tar'));

-// Pack a directory to TAR.GZ stream
-const tgzStream = await tarTools.packDirectoryToTarGzStream('./src');
+// With GZIP compression
+const tgzStream = await tarTools.getDirectoryPackStreamGz('./large-folder', 6);
+tgzStream.pipe(fs.createWriteStream('backup.tar.gz'));
+
+// ===== STREAMING EXTRACT =====
+// Extract large archives without loading into memory
+const extract = tarTools.getExtractStream();
+
+extract.on('entry', (header, stream, next) => {
+  console.log(`Extracting: ${header.name} (${header.size} bytes)`);
+
+  const writeStream = fs.createWriteStream(`./out/${header.name}`);
+  stream.pipe(writeStream);
+  writeStream.on('finish', next);
+});
+
+extract.on('finish', () => console.log('Extraction complete'));
+
+fs.createReadStream('large-archive.tar').pipe(extract);
+
+// Or use the convenient directory extraction
+await tarTools.extractToDirectory(
+  fs.createReadStream('archive.tar'),
+  './output-folder'
+);
 ```

 ### Working with ZIP archives directly
@@ -351,7 +461,7 @@ const zipTools = new ZipTools();
 // Create a ZIP archive from entries
 const zipBuffer = await zipTools.createZip([
  { archivePath: 'readme.txt', content: 'Hello!' },
-  { archivePath: 'data.bin', content: Buffer.from([0x00, 0x01, 0x02]) }
+  { archivePath: 'data.bin', content: new Uint8Array([0x00, 0x01, 0x02]) }
 ], 6);

 // Extract a ZIP buffer
@@ -448,13 +558,13 @@ fileStream.on('data', async (file) => {

 ## Supported Formats 📋

-| Format | Extension(s) | Extract | Create |
-|--------|--------------|---------|--------|
-| TAR | `.tar` | ✅ | ✅ |
-| TAR.GZ / TGZ | `.tar.gz`, `.tgz` | ✅ | ✅ |
-| ZIP | `.zip` | ✅ | ✅ |
-| GZIP | `.gz` | ✅ | ✅ |
-| BZIP2 | `.bz2` | ✅ | ❌ |
+| Format | Extension(s) | Extract | Create | Browser |
+|--------|--------------|---------|--------|---------|
+| TAR | `.tar` | ✅ | ✅ | ✅ |
+| TAR.GZ / TGZ | `.tar.gz`, `.tgz` | ✅ | ✅ | ✅ |
+| ZIP | `.zip` | ✅ | ✅ | ✅ |
+| GZIP | `.gz` | ✅ | ✅ | ✅ |
+| BZIP2 | `.bz2` | ✅ | ❌ | ✅ |

 ## Type Definitions

@@ -468,7 +578,7 @@ type TCompressionLevel = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9;
 // Entry for creating archives
 interface IArchiveEntry {
  archivePath: string;
-  content: string | Buffer | Readable | SmartFile | StreamFile;
+  content: string | Buffer | Uint8Array | SmartFile | StreamFile;
  size?: number;
  mode?: number;
  mtime?: Date;
@@ -496,9 +606,9 @@ interface IArchiveInfo {
 ## Performance Tips 🏎️

 1. **Use streaming for large files** – `.toStreamFiles()` processes entries one at a time without loading the entire archive
-2. **Provide byte lengths when known** – When using TarTools directly, provide `byteLength` for better performance
-3. **Choose appropriate compression** – Use 1-3 for speed, 6 (default) for balance, 9 for maximum compression
-4. **Filter early** – Use `.include()`/`.exclude()` to skip unwanted entries before processing
+2. **Choose appropriate compression** – Use 1-3 for speed, 6 (default) for balance, 9 for maximum compression
+3. **Filter early** – Use `.include()`/`.exclude()` to skip unwanted entries before processing
+4. **Use Uint8Array in browsers** – The browser bundle works with `Uint8Array` for optimal performance

 ## Error Handling 🛡️

@@ -524,23 +634,21 @@ try {

 ## License and Legal Information

-This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
+This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.

 **Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

 ### Trademarks

-This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.
+This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

-### Issue Reporting and Security
-
-For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
+Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

 ### Company Information

 Task Venture Capital GmbH
-Registered at District court Bremen HRB 35230 HB, Germany
+Registered at District Court Bremen HRB 35230 HB, Germany

-For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
+For any legal inquiries or further information, please contact us via email at hello@task.vc.

 By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.